23 | (a) Monolingual training on a low-resource language Korean (KOR) results in incorrect predictions, whereas (b) CLL from a high-resource language English (ENG) to KOR leads to correct predictions. 24 |
22 |
23 | (a) Monolingual training on a low-resource language Korean (KOR) results in incorrect predictions, whereas (b) CLL from a high-resource language English (ENG) to KOR leads to correct predictions.
24 |
32 | 
33 | Two general insights about CLL related to (a) joint learning and (b) cascade learning (CL).
34 |
| اﻟﺤﻢ | اﻟﺒﺎب | الباب |
81 | | 
| বান | রাখিবেন | রাখিবেন |
82 | | 
| か本 | 北京西 | 北京西 |
83 | | 
| ज | मंडप | मंडप |
84 | | 
| ٥り | くすり | くすり |
85 | | 
| 상온 | 냉면은 | 냉면은 |
86 | | 
| 디센다 | CUISINE | CUISINE |
87 |
88 |
89 | ### Download preprocessed lmdb datasets for training and evaluation
90 | We use [MLT19](https://rrc.cvc.uab.es/?ch=15&com=introduction) and [SynthMLT](https://github.com/MichalBusta/E2E-MLT) datasets.
91 | MLT19 and SynthMLT consist of 7 scripts: Arabic (ARA), Bengali (BEN), Chinese (CHI), Hindi (HIN), Japanese (JPN), Korean (KOR), and Latin (LAT).
92 | LAT consists of English, French, German, and Italian.
93 |
94 | Download the preprocessed lmdb datasets from [here](https://www.dropbox.com/scl/fi/js1y8dd6y0sethbuge1zl/CLL-data.zip?rlkey=bca3631p91mw1f98qhh79qprg&dl=0), or use wget as follows.
95 | ```
96 | wget -O CLL-data.zip https://www.dropbox.com/scl/fi/js1y8dd6y0sethbuge1zl/CLL-data.zip?rlkey=bca3631p91mw1f98qhh79qprg
97 | unzip CLL-data.zip -d data/
98 | ```
99 | The structure of `data/` is as follows:
100 | ```
101 | data
102 | ├── training
103 | │ ├── ARA
104 | │ ├── BEN
105 | │ ├── CHI
106 | │ ├── HIN
107 | │ ├── JPN
108 | │ ├── KOR
109 | │ ├── LAT
110 | │ ├── SynthMLT_ARA
111 | │ ├── SynthMLT_BEN
112 | │ ├── SynthMLT_CHI
113 | │ ├── SynthMLT_HIN
114 | │ ├── SynthMLT_JPN
115 | │ ├── SynthMLT_KOR
116 | │ └── SynthMLT_LAT
117 | ├── validation
118 | │ ├── ARA
119 | │ ├── BEN
120 | │ ├── CHI
121 | │ ├── HIN
122 | │ ├── JPN
123 | │ ├── KOR
124 | │ └── LAT
125 | └── evaluation
126 | ├── ARA
127 | ├── BEN
128 | ├── CHI
129 | ├── HIN
130 | ├── JPN
131 | ├── KOR
132 | └── LAT
133 | ```
134 |
135 | ### Training
136 | 1. Train SVTR with only one language data (LAT). For detail, check [--select_data part](https://github.com/ku21fan/CLL-STR/blob/e65ac0288ea41c0cba7191141b838e7ab0d1eb4f/train.py#L532-L573) in train.py.
137 | ```
138 | CUDA_VISIBLE_DEVICES=0 python3 train.py --model_name SVTR --exp_name SVTR_lang_L --select_data lang_L
139 | ```
140 |
141 | 2. Joint learning with all seven languages (Related to Table 2 in the paper)
142 | ```
143 | CUDA_VISIBLE_DEVICES=0 python3 train.py --model_name SVTR --exp_name SVTR_joint_All --select_data lang_CBHJAKL
144 | ```
145 |
146 | 3. Cascase learning: Pre-train SVTR on LAT then HIN (Related to Table 3 in the paper)
147 | ```
148 | CUDA_VISIBLE_DEVICES=0 python3 train.py --model_name SVTR --exp_name SVTR_L_to_H \
149 | --num_iter 10000 --saved_model saved_models/SVTR_lang_L/best_score.pth --select_data lang_H
150 | ```
151 |
152 | 4. Joint learning with all seven languages but using only 2K LAT instead of all LAT = _Base_ setting of Table 4 in the paper.
153 | ```
154 | CUDA_VISIBLE_DEVICES=0 python3 train.py --model_name SVTR --exp_name SVTR_joint_Base \
155 | --data_usage_LAT 2000 --select_data lang_CBHJAKL
156 | ```
157 |
158 | Joint learning with _Base_ setting + 2K SynthMLT KOR
159 | ```
160 | CUDA_VISIBLE_DEVICES=0 python3 train.py --model_name SVTR --exp_name SVTR_joint_Base_2KSynthMLT_K \
161 | --data_usage_LAT 2000 --data_usage_SynthMLT 2000 --select_data lang_CBHJAKL-SynthMLT_K
162 | ```
163 |
164 | Joint learning with _Base_ setting + all SynthMLT KOR
165 | ```
166 | CUDA_VISIBLE_DEVICES=0 python3 train.py --model_name SVTR --exp_name SVTR_joint_Base_allSynthMLT_K \
167 | --data_usage_LAT 2000 --select_data lang_CBHJAKL-SynthMLT_K
168 | ```
169 |
170 |
171 | ### Evaluation
172 | Test SVTR model.
173 | ```
174 | CUDA_VISIBLE_DEVICES=0 python3 test.py --model_name SVTR --saved_model saved_models/SVTR_joint_All/best_score.pth
175 | ```
176 |
177 |
178 | ### Main arguments
179 | train.py
180 | * `--train_data`: folder path for training lmdb dataset. default: `data/training/`
181 | * `--valid_data`: folder path for validation lmdb dataset. default: `data/validation/`
182 | * `--eval_data`: folder path for evaluation lmdb dataset. default: `data/evaluation/`
183 | * `--model_name`: select model 'CRNN' or 'SVTR'.
184 | * `--saved_model`: assign the saved model for fine-tuning
185 |
186 | test.py
187 | * `--eval_data`: folder path for evaluation lmdb dataset. default: `data/evaluation/`
188 | * `--model_name`: select model 'CRNN' or 'SVTR'.
189 | * `--saved_model`: assign the saved model for evaluation.
190 |
191 | demo.py
192 | * `--image_folder`: path for image_folder which contains text images. default: `demo_image/`
193 | * `--model_name`: select model 'CRNN' or 'SVTR'.
194 | * `--saved_model`: assign the saved model to use.
195 |
196 |
197 | ## Citation
198 | Please consider citing this work in your publications if it helps your research.
199 | ```
200 | @inproceedings{baek2024CLLSTR,
201 | title={Cross-Lingual Learning in Multilingual Scene Text Recognition},
202 | author={Baek, Jeonghun and Matsui, Yusuke and Aizawa, Kiyoharu},
203 | booktitle={IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
204 | year={2024}
205 | }
206 | ```
207 |
208 | ## Contact
209 | Feel free to contact us if there is any question: Jeonghun Baek ku21fang@gmail.com
210 |
211 | ## Acknowledgements
212 | This implementation has been based on the repositories [STR-Fewer-Labels](https://github.com/ku21fan/STR-Fewer-Labels) and [MRN](https://github.com/simplify23/MRN).
213 |
214 | ## License
215 | MIT
216 |
--------------------------------------------------------------------------------
/dataset.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import re
4 | import six
5 | import time
6 | import math
7 | import random
8 |
9 | import lmdb
10 | import regex
11 | from tqdm import tqdm
12 | from natsort import natsorted
13 | import PIL
14 | import numpy as np
15 | import torch
16 | from torch.utils.data import Dataset, ConcatDataset, Subset
17 | from torch._utils import _accumulate
18 | import torchvision.transforms as transforms
19 |
20 |
21 | def hierarchical_dataset(root, opt, select_data="/", mode="train", data_type="label"):
22 | """ select_data='/' contains all sub-directory of root directory """
23 | dataset_list = []
24 | dataset_log = f"dataset_root: {root}\t dataset: {select_data[0]}"
25 | print(dataset_log)
26 | dataset_log += "\n"
27 | for dirpath, dirnames, filenames in os.walk(root + "/"):
28 | if not dirnames:
29 | select_flag = False
30 | for selected_d in select_data:
31 | if selected_d in dirpath:
32 | select_flag = True
33 | break
34 |
35 | if select_flag:
36 | dataset = LmdbDataset(dirpath, opt, mode=mode)
37 |
38 | sub_dataset_log = f"sub-directory:\t/{os.path.relpath(dirpath, root)}\t num samples: {len(dataset)}"
39 | print(sub_dataset_log)
40 | dataset_log += f"{sub_dataset_log}\n"
41 | dataset_list.append(dataset)
42 |
43 | concatenated_dataset = ConcatDataset(dataset_list)
44 |
45 | return concatenated_dataset, dataset_log
46 |
47 |
48 | def concat_dataset(root, opt, dataset_list, mode="train"):
49 | """ select_data='/' contains all sub-directory of root directory """
50 | concat_dataset_list = []
51 | for data in dataset_list:
52 | dataset_log = f"dataset_root: {root}\tsub-directory:\t{data}"
53 | print(dataset_log)
54 | dataset_log += "\n"
55 |
56 | dataset = LmdbDataset(os.path.join(root, data), opt, mode=mode)
57 | tmp_log = f"\t num samples: {len(dataset)}"
58 | print(tmp_log)
59 | dataset_log += f"{tmp_log}\n"
60 |
61 | if mode == "train" and data == "LAT" and opt.data_usage_LAT != "all":
62 | total_number_dataset = len(dataset)
63 | number_dataset = int(opt.data_usage_LAT)
64 |
65 | dataset_split = [number_dataset, total_number_dataset - number_dataset]
66 | indices = range(total_number_dataset)
67 | dataset, _ = [
68 | Subset(dataset, indices[offset - length : offset])
69 | for offset, length in zip(_accumulate(dataset_split), dataset_split)
70 | ]
71 | tmp_log = f"{len(dataset)} {data} samples are used for the experiment"
72 | print(tmp_log)
73 | dataset_log += f"{tmp_log}\n"
74 |
75 | elif (
76 | mode == "train" and "SynthMLT" in data and opt.data_usage_SynthMLT != "all"
77 | ):
78 | total_number_dataset = len(dataset)
79 | number_dataset = int(opt.data_usage_SynthMLT)
80 |
81 | dataset_split = [number_dataset, total_number_dataset - number_dataset]
82 | indices = range(total_number_dataset)
83 | dataset, _ = [
84 | Subset(dataset, indices[offset - length : offset])
85 | for offset, length in zip(_accumulate(dataset_split), dataset_split)
86 | ]
87 | tmp_log = f"{len(dataset)} {data} samples are used for the experiment"
88 | print(tmp_log)
89 | dataset_log += f"{tmp_log}\n"
90 |
91 | concat_dataset_list.append(dataset)
92 |
93 | concatenated_dataset = ConcatDataset(concat_dataset_list)
94 | print(f"num samples of concatenated_dataset: {len(concatenated_dataset)}")
95 |
96 | if mode == "train" and opt.data_usage != "all":
97 | total_number_dataset = len(concatenated_dataset)
98 | number_dataset = int(opt.data_usage)
99 |
100 | dataset_split = [number_dataset, total_number_dataset - number_dataset]
101 | indices = range(total_number_dataset)
102 | concatenated_dataset, _ = [
103 | Subset(concatenated_dataset, indices[offset - length : offset])
104 | for offset, length in zip(_accumulate(dataset_split), dataset_split)
105 | ]
106 | tmp_log = f"{len(concatenated_dataset)} samples are used for the experiment"
107 | print(tmp_log)
108 | dataset_log += f"{tmp_log}\n"
109 |
110 | # for faster training, we multiply small datasets itself.
111 | if mode == "train" and len(concatenated_dataset) < 100000:
112 | print(
113 | f"{len(concatenated_dataset)} is too small, it is multiplied to over 100K"
114 | )
115 | print(
116 | "CAUTION: If you use 'epoch' for training, you have to remove this multiplication part"
117 | )
118 | multiple_times = int(100000 / len(concatenated_dataset))
119 | dataset_self_multiple = [concatenated_dataset] * multiple_times
120 | concatenated_dataset = ConcatDataset(dataset_self_multiple)
121 |
122 | return concatenated_dataset, dataset_log
123 |
124 |
125 | class LmdbDataset(Dataset):
126 | def __init__(self, root, opt, mode="train"):
127 |
128 | self.root = root
129 | self.opt = opt
130 | self.mode = mode
131 | self.env = lmdb.open(
132 | root,
133 | max_readers=32,
134 | readonly=True,
135 | lock=False,
136 | readahead=False,
137 | meminit=False,
138 | )
139 |
140 | # print(self.env.info())
141 | self.txn = self.env.begin(write=False)
142 |
143 | if not self.env:
144 | print("cannot open lmdb from %s" % (root))
145 | sys.exit(0)
146 |
147 | self.nSamples = int(self.txn.get("num-samples".encode()))
148 |
149 | self.filtered_index_list = []
150 | for index in tqdm(range(self.nSamples), total=self.nSamples):
151 | index += 1 # lmdb starts with 1
152 | label_key = "label-%09d".encode() % index
153 | label = self.txn.get(label_key).decode("utf-8")
154 |
155 | # length filtering
156 | length_of_label = len(label)
157 |
158 | # For HIN/BEN/ARA, HIN/BEN/ARA chars are delimited with \t
159 | if regex.findall("[\p{InDevanagari}\p{InBengali}\p{Arabic}]", label):
160 | if (
161 | length_of_label > 2 * opt.batch_max_length - 1
162 | ): # to count "\t" between hindi chars.
163 | # print("length check", label)
164 | continue
165 |
166 | elif length_of_label > opt.batch_max_length:
167 | # print("length check", label)
168 | continue
169 |
170 | self.filtered_index_list.append(index)
171 |
172 | self.nSamples = len(self.filtered_index_list)
173 |
174 | def __len__(self):
175 | return self.nSamples
176 |
177 | def __getitem__(self, index):
178 | assert index <= len(self), "index range error"
179 | index = self.filtered_index_list[index]
180 |
181 | label_key = "label-%09d".encode() % index
182 | label = self.txn.get(label_key).decode("utf-8")
183 | img_key = "image-%09d".encode() % index
184 | imgbuf = self.txn.get(img_key)
185 |
186 | buf = six.BytesIO()
187 | buf.write(imgbuf)
188 | buf.seek(0)
189 |
190 | try:
191 | img = PIL.Image.open(buf).convert("RGB")
192 |
193 | except IOError:
194 | print(f"Corrupted image for {index}")
195 | # make dummy image and dummy label for corrupted image.
196 | img = PIL.Image.new("RGB", (self.opt.imgW, self.opt.imgH))
197 | label = "[dummy_label]"
198 |
199 | return (img, label)
200 |
201 |
202 | class RawDataset(Dataset):
203 | def __init__(self, root, opt):
204 | self.opt = opt
205 | self.image_path_list = []
206 | for dirpath, dirnames, filenames in os.walk(root):
207 | for name in filenames:
208 | _, ext = os.path.splitext(name)
209 | ext = ext.lower()
210 | if ext == ".jpg" or ext == ".jpeg" or ext == ".png":
211 | self.image_path_list.append(os.path.join(dirpath, name))
212 |
213 | self.image_path_list = natsorted(self.image_path_list)
214 | self.nSamples = len(self.image_path_list)
215 |
216 | def __len__(self):
217 | return self.nSamples
218 |
219 | def __getitem__(self, index):
220 |
221 | try:
222 | img = PIL.Image.open(self.image_path_list[index]).convert("RGB")
223 |
224 | except IOError:
225 | print(f"Corrupted image for {index}")
226 | # make dummy image and dummy label for corrupted image.
227 | img = PIL.Image.new("RGB", (self.opt.imgW, self.opt.imgH))
228 |
229 | return (img, self.image_path_list[index])
230 |
231 |
232 | class AlignCollate(object):
233 | def __init__(self, opt, mode="train"):
234 | self.opt = opt
235 | self.mode = mode
236 |
237 | if opt.PAD:
238 | self.transform = NormalizePAD(opt)
239 | else:
240 | if opt.Aug == "None" or mode != "train":
241 | self.transform = ResizeNormalize((opt.imgW, opt.imgH))
242 | else:
243 | self.transform = Text_augment(opt)
244 |
245 | def __call__(self, batch):
246 | images, labels = zip(*batch)
247 |
248 | image_tensors = [self.transform(image) for image in images]
249 | image_tensors = torch.cat([t.unsqueeze(0) for t in image_tensors], 0)
250 |
251 | return image_tensors, labels
252 |
253 |
254 | class ResizeNormalize(object):
255 | def __init__(self, size, interpolation=PIL.Image.BICUBIC):
256 | # CAUTION: it should be (width, height). different from size of transforms.Resize (height, width)
257 | self.size = size
258 | self.interpolation = interpolation
259 | self.toTensor = transforms.ToTensor()
260 |
261 | def __call__(self, image):
262 | image = image.resize(self.size, self.interpolation)
263 | image = self.toTensor(image)
264 | image.sub_(0.5).div_(0.5)
265 | return image
266 |
267 |
268 | class NormalizePAD(object):
269 | def __init__(self, opt, interpolation=PIL.Image.BICUBIC):
270 | self.opt = opt
271 | self.interpolation = interpolation
272 | self.padded_size = (3, self.opt.imgH, self.opt.imgW) # 3 for RGB input channel
273 | self.toTensor = transforms.ToTensor()
274 |
275 | def __call__(self, img):
276 | w, h = img.size
277 | ratio = w / float(h)
278 | if math.ceil(self.opt.imgH * ratio) > self.opt.imgW:
279 | resized_w = self.opt.imgW
280 | else:
281 | resized_w = math.ceil(self.opt.imgH * ratio)
282 |
283 | img = img.resize((resized_w, self.opt.imgH), self.interpolation)
284 |
285 | img = self.toTensor(img)
286 | img.sub_(0.5).div_(0.5)
287 | c, h, w = img.size()
288 | Pad_img = torch.FloatTensor(*self.padded_size).fill_(0)
289 | Pad_img[:, :, :w] = img # right pad
290 | if self.padded_size[2] != w: # add border Pad
291 | Pad_img[:, :, w:] = (
292 | img[:, :, w - 1].unsqueeze(2).expand(c, h, self.padded_size[2] - w)
293 | )
294 |
295 | return Pad_img
296 |
--------------------------------------------------------------------------------
/modules/feature_extraction.py:
--------------------------------------------------------------------------------
1 | import torch.nn as nn
2 | import torch.nn.functional as F
3 |
4 | from modules.svtr import SVTR
5 |
6 |
7 | class VGG_FeatureExtractor(nn.Module):
8 | """ FeatureExtractor of CRNN (https://arxiv.org/pdf/1507.05717.pdf) """
9 |
10 | def __init__(self, input_channel, output_channel=512):
11 | super(VGG_FeatureExtractor, self).__init__()
12 | self.output_channel = [
13 | int(output_channel / 8),
14 | int(output_channel / 4),
15 | int(output_channel / 2),
16 | output_channel,
17 | ] # [64, 128, 256, 512]
18 | self.ConvNet = nn.Sequential(
19 | nn.Conv2d(input_channel, self.output_channel[0], 3, 1, 1),
20 | nn.ReLU(True),
21 | nn.MaxPool2d(2, 2), # 64x16x50
22 | nn.Conv2d(self.output_channel[0], self.output_channel[1], 3, 1, 1),
23 | nn.ReLU(True),
24 | nn.MaxPool2d(2, 2), # 128x8x25
25 | nn.Conv2d(self.output_channel[1], self.output_channel[2], 3, 1, 1),
26 | nn.ReLU(True), # 256x8x25
27 | nn.Conv2d(self.output_channel[2], self.output_channel[2], 3, 1, 1),
28 | nn.ReLU(True),
29 | nn.MaxPool2d((2, 1), (2, 1)), # 256x4x25
30 | nn.Conv2d(
31 | self.output_channel[2], self.output_channel[3], 3, 1, 1, bias=False
32 | ),
33 | nn.BatchNorm2d(self.output_channel[3]),
34 | nn.ReLU(True), # 512x4x25
35 | nn.Conv2d(
36 | self.output_channel[3], self.output_channel[3], 3, 1, 1, bias=False
37 | ),
38 | nn.BatchNorm2d(self.output_channel[3]),
39 | nn.ReLU(True),
40 | nn.MaxPool2d((2, 1), (2, 1)), # 512x2x25
41 | nn.Conv2d(self.output_channel[3], self.output_channel[3], 2, 1, 0),
42 | nn.ReLU(True),
43 | ) # 512x1x24
44 |
45 | def forward(self, input):
46 | return self.ConvNet(input)
47 |
48 |
49 | class RCNN_FeatureExtractor(nn.Module):
50 | """ FeatureExtractor of GRCNN (https://papers.nips.cc/paper/6637-gated-recurrent-convolution-neural-network-for-ocr.pdf) """
51 |
52 | def __init__(self, input_channel, output_channel=512):
53 | super(RCNN_FeatureExtractor, self).__init__()
54 | self.output_channel = [
55 | int(output_channel / 8),
56 | int(output_channel / 4),
57 | int(output_channel / 2),
58 | output_channel,
59 | ] # [64, 128, 256, 512]
60 | self.ConvNet = nn.Sequential(
61 | nn.Conv2d(input_channel, self.output_channel[0], 3, 1, 1),
62 | nn.ReLU(True),
63 | nn.MaxPool2d(2, 2), # 64 x 16 x 50
64 | GRCL(
65 | self.output_channel[0],
66 | self.output_channel[0],
67 | num_iteration=5,
68 | kernel_size=3,
69 | pad=1,
70 | ),
71 | nn.MaxPool2d(2, 2), # 64 x 8 x 25
72 | GRCL(
73 | self.output_channel[0],
74 | self.output_channel[1],
75 | num_iteration=5,
76 | kernel_size=3,
77 | pad=1,
78 | ),
79 | nn.MaxPool2d(2, (2, 1), (0, 1)), # 128 x 4 x 26
80 | GRCL(
81 | self.output_channel[1],
82 | self.output_channel[2],
83 | num_iteration=5,
84 | kernel_size=3,
85 | pad=1,
86 | ),
87 | nn.MaxPool2d(2, (2, 1), (0, 1)), # 256 x 2 x 27
88 | nn.Conv2d(
89 | self.output_channel[2], self.output_channel[3], 2, 1, 0, bias=False
90 | ),
91 | nn.BatchNorm2d(self.output_channel[3]),
92 | nn.ReLU(True),
93 | ) # 512 x 1 x 26
94 |
95 | def forward(self, input):
96 | return self.ConvNet(input)
97 |
98 |
99 | class ResNet_FeatureExtractor(nn.Module):
100 | """ FeatureExtractor of FAN (http://openaccess.thecvf.com/content_ICCV_2017/papers/Cheng_Focusing_Attention_Towards_ICCV_2017_paper.pdf) """
101 |
102 | def __init__(self, input_channel, output_channel=512):
103 | super(ResNet_FeatureExtractor, self).__init__()
104 | self.ConvNet = ResNet(input_channel, output_channel, BasicBlock, [1, 2, 5, 3])
105 |
106 | def forward(self, input):
107 | return self.ConvNet(input)
108 |
109 |
110 | # For Gated RCNN
111 | class GRCL(nn.Module):
112 | def __init__(self, input_channel, output_channel, num_iteration, kernel_size, pad):
113 | super(GRCL, self).__init__()
114 | self.wgf_u = nn.Conv2d(input_channel, output_channel, 1, 1, 0, bias=False)
115 | self.wgr_x = nn.Conv2d(output_channel, output_channel, 1, 1, 0, bias=False)
116 | self.wf_u = nn.Conv2d(
117 | input_channel, output_channel, kernel_size, 1, pad, bias=False
118 | )
119 | self.wr_x = nn.Conv2d(
120 | output_channel, output_channel, kernel_size, 1, pad, bias=False
121 | )
122 |
123 | self.BN_x_init = nn.BatchNorm2d(output_channel)
124 |
125 | self.num_iteration = num_iteration
126 | self.GRCL = [GRCL_unit(output_channel) for _ in range(num_iteration)]
127 | self.GRCL = nn.Sequential(*self.GRCL)
128 |
129 | def forward(self, input):
130 | """ The input of GRCL is consistant over time t, which is denoted by u(0)
131 | thus wgf_u / wf_u is also consistant over time t.
132 | """
133 | wgf_u = self.wgf_u(input)
134 | wf_u = self.wf_u(input)
135 | x = F.relu(self.BN_x_init(wf_u))
136 |
137 | for i in range(self.num_iteration):
138 | x = self.GRCL[i](wgf_u, self.wgr_x(x), wf_u, self.wr_x(x))
139 |
140 | return x
141 |
142 |
143 | class GRCL_unit(nn.Module):
144 | def __init__(self, output_channel):
145 | super(GRCL_unit, self).__init__()
146 | self.BN_gfu = nn.BatchNorm2d(output_channel)
147 | self.BN_grx = nn.BatchNorm2d(output_channel)
148 | self.BN_fu = nn.BatchNorm2d(output_channel)
149 | self.BN_rx = nn.BatchNorm2d(output_channel)
150 | self.BN_Gx = nn.BatchNorm2d(output_channel)
151 |
152 | def forward(self, wgf_u, wgr_x, wf_u, wr_x):
153 | G_first_term = self.BN_gfu(wgf_u)
154 | G_second_term = self.BN_grx(wgr_x)
155 | G = F.sigmoid(G_first_term + G_second_term)
156 |
157 | x_first_term = self.BN_fu(wf_u)
158 | x_second_term = self.BN_Gx(self.BN_rx(wr_x) * G)
159 | x = F.relu(x_first_term + x_second_term)
160 |
161 | return x
162 |
163 |
164 | class BasicBlock(nn.Module):
165 | expansion = 1
166 |
167 | def __init__(self, inplanes, planes, stride=1, downsample=None):
168 | super(BasicBlock, self).__init__()
169 | self.conv1 = self._conv3x3(inplanes, planes)
170 | self.bn1 = nn.BatchNorm2d(planes)
171 | self.conv2 = self._conv3x3(planes, planes)
172 | self.bn2 = nn.BatchNorm2d(planes)
173 | self.relu = nn.ReLU(inplace=True)
174 | self.downsample = downsample
175 | self.stride = stride
176 |
177 | def _conv3x3(self, in_planes, out_planes, stride=1):
178 | "3x3 convolution with padding"
179 | return nn.Conv2d(
180 | in_planes, out_planes, kernel_size=3, stride=stride, padding=1, bias=False
181 | )
182 |
183 | def forward(self, x):
184 | residual = x
185 |
186 | out = self.conv1(x)
187 | out = self.bn1(out)
188 | out = self.relu(out)
189 |
190 | out = self.conv2(out)
191 | out = self.bn2(out)
192 |
193 | if self.downsample is not None:
194 | residual = self.downsample(x)
195 | out += residual
196 | out = self.relu(out)
197 |
198 | return out
199 |
200 |
201 | class ResNet(nn.Module):
202 | def __init__(self, input_channel, output_channel, block, layers):
203 | super(ResNet, self).__init__()
204 |
205 | self.output_channel_block = [
206 | int(output_channel / 4),
207 | int(output_channel / 2),
208 | output_channel,
209 | output_channel,
210 | ]
211 |
212 | self.inplanes = int(output_channel / 8)
213 | self.conv0_1 = nn.Conv2d(
214 | input_channel,
215 | int(output_channel / 16),
216 | kernel_size=3,
217 | stride=1,
218 | padding=1,
219 | bias=False,
220 | )
221 | self.bn0_1 = nn.BatchNorm2d(int(output_channel / 16))
222 | self.conv0_2 = nn.Conv2d(
223 | int(output_channel / 16),
224 | self.inplanes,
225 | kernel_size=3,
226 | stride=1,
227 | padding=1,
228 | bias=False,
229 | )
230 | self.bn0_2 = nn.BatchNorm2d(self.inplanes)
231 | self.relu = nn.ReLU(inplace=True)
232 |
233 | self.maxpool1 = nn.MaxPool2d(kernel_size=2, stride=2, padding=0)
234 | self.layer1 = self._make_layer(block, self.output_channel_block[0], layers[0])
235 | self.conv1 = nn.Conv2d(
236 | self.output_channel_block[0],
237 | self.output_channel_block[0],
238 | kernel_size=3,
239 | stride=1,
240 | padding=1,
241 | bias=False,
242 | )
243 | self.bn1 = nn.BatchNorm2d(self.output_channel_block[0])
244 |
245 | self.maxpool2 = nn.MaxPool2d(kernel_size=2, stride=2, padding=0)
246 | self.layer2 = self._make_layer(
247 | block, self.output_channel_block[1], layers[1], stride=1
248 | )
249 | self.conv2 = nn.Conv2d(
250 | self.output_channel_block[1],
251 | self.output_channel_block[1],
252 | kernel_size=3,
253 | stride=1,
254 | padding=1,
255 | bias=False,
256 | )
257 | self.bn2 = nn.BatchNorm2d(self.output_channel_block[1])
258 |
259 | self.maxpool3 = nn.MaxPool2d(kernel_size=2, stride=(2, 1), padding=(0, 1))
260 | self.layer3 = self._make_layer(
261 | block, self.output_channel_block[2], layers[2], stride=1
262 | )
263 | self.conv3 = nn.Conv2d(
264 | self.output_channel_block[2],
265 | self.output_channel_block[2],
266 | kernel_size=3,
267 | stride=1,
268 | padding=1,
269 | bias=False,
270 | )
271 | self.bn3 = nn.BatchNorm2d(self.output_channel_block[2])
272 |
273 | self.layer4 = self._make_layer(
274 | block, self.output_channel_block[3], layers[3], stride=1
275 | )
276 | self.conv4_1 = nn.Conv2d(
277 | self.output_channel_block[3],
278 | self.output_channel_block[3],
279 | kernel_size=2,
280 | stride=(2, 1),
281 | padding=(0, 1),
282 | bias=False,
283 | )
284 | self.bn4_1 = nn.BatchNorm2d(self.output_channel_block[3])
285 | self.conv4_2 = nn.Conv2d(
286 | self.output_channel_block[3],
287 | self.output_channel_block[3],
288 | kernel_size=2,
289 | stride=1,
290 | padding=0,
291 | bias=False,
292 | )
293 | self.bn4_2 = nn.BatchNorm2d(self.output_channel_block[3])
294 |
295 | def _make_layer(self, block, planes, blocks, stride=1):
296 | downsample = None
297 | if stride != 1 or self.inplanes != planes * block.expansion:
298 | downsample = nn.Sequential(
299 | nn.Conv2d(
300 | self.inplanes,
301 | planes * block.expansion,
302 | kernel_size=1,
303 | stride=stride,
304 | bias=False,
305 | ),
306 | nn.BatchNorm2d(planes * block.expansion),
307 | )
308 |
309 | layers = []
310 | layers.append(block(self.inplanes, planes, stride, downsample))
311 | self.inplanes = planes * block.expansion
312 | for i in range(1, blocks):
313 | layers.append(block(self.inplanes, planes))
314 |
315 | return nn.Sequential(*layers)
316 |
317 | def forward(self, x):
318 | x = self.conv0_1(x)
319 | x = self.bn0_1(x)
320 | x = self.relu(x)
321 | x = self.conv0_2(x)
322 | x = self.bn0_2(x)
323 | x = self.relu(x)
324 |
325 | x = self.maxpool1(x)
326 | x = self.layer1(x)
327 | x = self.conv1(x)
328 | x = self.bn1(x)
329 | x = self.relu(x)
330 |
331 | x = self.maxpool2(x)
332 | x = self.layer2(x)
333 | x = self.conv2(x)
334 | x = self.bn2(x)
335 | x = self.relu(x)
336 |
337 | x = self.maxpool3(x)
338 | x = self.layer3(x)
339 | x = self.conv3(x)
340 | x = self.bn3(x)
341 | x = self.relu(x)
342 |
343 | x = self.layer4(x)
344 | x = self.conv4_1(x)
345 | x = self.bn4_1(x)
346 | x = self.relu(x)
347 | x = self.conv4_2(x)
348 | x = self.bn4_2(x)
349 | x = self.relu(x)
350 |
351 | return x
352 |
353 |
354 | class SVTR_FeatureExtractor(nn.Module):
355 | def __init__(self, input_channel, output_channel=512):
356 | super(SVTR_FeatureExtractor, self).__init__()
357 | self.ConvNet = SVTR(in_channels=input_channel, out_channels=output_channel)
358 | # self.neck = Im2Seq()
359 |
360 | def forward(self, input):
361 | return self.ConvNet(input)
362 |
--------------------------------------------------------------------------------
/test.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import time
4 | import string
5 | import argparse
6 | import re
7 | from datetime import date
8 |
9 | import torch
10 | import torch.backends.cudnn as cudnn
11 | import torch.utils.data
12 | import torch.nn.functional as F
13 | import numpy as np
14 | from nltk.metrics.distance import edit_distance
15 | from tqdm import tqdm
16 |
17 | from utils import CTCLabelConverter, AttnLabelConverter, Averager
18 | from dataset import hierarchical_dataset, AlignCollate
19 | from model import Model
20 |
21 | device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
22 |
23 |
24 | def benchmark_all_eval(model, criterion, converter, opt, calculate_infer_time=False):
25 | """evaluation on 7 scripts"""
26 | if opt.dataset_name == "MLT19":
27 | eval_data_list = [
28 | "CHI",
29 | "BEN",
30 | "HIN",
31 | "JPN",
32 | "ARA",
33 | "KOR",
34 | "LAT",
35 | ]
36 |
37 | if calculate_infer_time:
38 | eval_batch_size = (
39 | 1 # batch_size should be 1 to calculate the GPU inference time per image.
40 | )
41 | else:
42 | eval_batch_size = opt.batch_size
43 |
44 | accuracy_list = []
45 | total_forward_time = []
46 | total_eval_data_number = []
47 | total_correct_number = []
48 | log = open(f"./result/{opt.exp_name}/log_all_evaluation.txt", "a")
49 | dashed_line = "-" * 80
50 | print(dashed_line)
51 | log.write(dashed_line + "\n")
52 | for eval_data in eval_data_list:
53 | eval_data_path = os.path.join(opt.eval_data, eval_data)
54 | AlignCollate_eval = AlignCollate(opt, mode="test")
55 | eval_data, eval_data_log = hierarchical_dataset(
56 | root=eval_data_path, opt=opt, mode="test"
57 | )
58 | eval_loader = torch.utils.data.DataLoader(
59 | eval_data,
60 | batch_size=eval_batch_size,
61 | shuffle=False,
62 | num_workers=int(opt.workers),
63 | collate_fn=AlignCollate_eval,
64 | pin_memory=False,
65 | )
66 |
67 | (_, accuracy_by_best_model, _, _, _, infer_time, length_of_data,) = validation(
68 | model, criterion, eval_loader, converter, opt, tqdm_position=0
69 | )
70 | accuracy_list.append(f"{accuracy_by_best_model:0.2f}")
71 | total_forward_time.append(infer_time)
72 | total_eval_data_number.append(len(eval_data))
73 | total_correct_number.append(accuracy_by_best_model * length_of_data)
74 | log.write(eval_data_log)
75 | print(f"Acc {accuracy_by_best_model:0.2f}")
76 | log.write(f"Acc {accuracy_by_best_model:0.2f}\n")
77 | print(dashed_line)
78 | log.write(dashed_line + "\n")
79 |
80 | averaged_forward_time = sum(total_forward_time) / sum(total_eval_data_number) * 1000
81 | total_accuracy = sum(total_correct_number) / sum(total_eval_data_number)
82 | params_num = sum([np.prod(p.size()) for p in model.parameters()])
83 |
84 | eval_log = "accuracy: "
85 | for name, accuracy in zip(eval_data_list, accuracy_list):
86 | eval_log += f"{name}: {accuracy}\t"
87 | eval_log += f"total_accuracy: {total_accuracy:0.2f}\t"
88 | eval_log += f"averaged_infer_time: {averaged_forward_time:0.3f}\t# parameters: {params_num/1e6:0.2f}"
89 | print(eval_log)
90 | log.write(eval_log + "\n")
91 |
92 | # for convenience
93 | tab_sep_accuracy_list = "\t".join(accuracy_list)
94 | print(f"{total_accuracy:0.2f}\t{tab_sep_accuracy_list}")
95 | # print(f"Total_accuracy:{total_accuracy:0.2f}")
96 | log.write(f"{tab_sep_accuracy_list}\n")
97 | log.write(f"Total_accuracy:{total_accuracy:0.2f}\n")
98 | log.close()
99 |
100 | # for convenience
101 | today = date.today()
102 | if opt.log_multiple_test:
103 | log_all_model = open(f"./evaluation_log/log_multiple_test_{today}.txt", "a")
104 | log_all_model.write(f"{total_accuracy:0.2f}\t{tab_sep_accuracy_list}\n")
105 | else:
106 | log_all_model = open(
107 | f"./evaluation_log/log_all_model_evaluation_{today}.txt", "a"
108 | )
109 | log_all_model.write(
110 | f"./result/{opt.exp_name}\tTotal_accuracy:{total_accuracy:0.2f}\n"
111 | )
112 | log_all_model.write(f"{total_accuracy:0.2f}\t{tab_sep_accuracy_list}\n")
113 | log_all_model.close()
114 |
115 | return total_accuracy, eval_data_list, accuracy_list
116 |
117 |
118 | def validation(model, criterion, eval_loader, converter, opt, tqdm_position=1):
119 | """validation or evaluation"""
120 | n_correct = 0
121 | norm_ED = 0
122 | length_of_data = 0
123 | infer_time = 0
124 | valid_loss_avg = Averager()
125 |
126 | for i, (image_tensors, labels) in tqdm(
127 | enumerate(eval_loader),
128 | total=len(eval_loader),
129 | position=tqdm_position,
130 | leave=False,
131 | ):
132 | batch_size = image_tensors.size(0)
133 | length_of_data = length_of_data + batch_size
134 | image = image_tensors.to(device)
135 |
136 | # For max length prediction
137 | labels_index, labels_length = converter.encode(
138 | labels, batch_max_length=opt.batch_max_length
139 | )
140 |
141 | if "CTC" in opt.Prediction:
142 | start_time = time.time()
143 | preds = model(image)
144 | forward_time = time.time() - start_time
145 |
146 | # Calculate evaluation loss for CTC deocder.
147 | preds_size = torch.IntTensor([preds.size(1)] * batch_size)
148 | # permute 'preds' to use CTCloss format
149 | cost = criterion(
150 | preds.log_softmax(2).permute(1, 0, 2),
151 | labels_index,
152 | preds_size,
153 | labels_length,
154 | )
155 |
156 | else:
157 | text_for_pred = (
158 | torch.LongTensor(batch_size).fill_(converter.dict["[SOS]"]).to(device)
159 | )
160 |
161 | start_time = time.time()
162 | preds = model(image, text_for_pred, is_train=False)
163 | forward_time = time.time() - start_time
164 |
165 | target = labels_index[:, 1:] # without [SOS] Symbol
166 | cost = criterion(
167 | preds.contiguous().view(-1, preds.shape[-1]),
168 | target.contiguous().view(-1),
169 | )
170 |
171 | # make unknown chracter to [UNK] token.
172 | if "Attn" in opt.Prediction:
173 | labels_string = converter.decode(
174 | labels_index[:, 1:], labels_length
175 | ) # without [SOS] Symbol
176 | elif "CTC" in opt.Prediction:
177 | labels_string = converter.decode_gt(labels_index, labels_length)
178 |
179 | # select max probabilty (greedy decoding) then decode index to character
180 | _, preds_index = preds.max(2)
181 | preds_size = torch.IntTensor([preds.size(1)] * preds_index.size(0)).to(device)
182 | preds_str = converter.decode(preds_index, preds_size)
183 |
184 | infer_time += forward_time
185 | valid_loss_avg.add(cost)
186 |
187 | # calculate accuracy & confidence score
188 | preds_prob = F.softmax(preds, dim=2)
189 | preds_max_prob, _ = preds_prob.max(dim=2)
190 | confidence_score_list = []
191 | for gt, prd, prd_max_prob in zip(labels_string, preds_str, preds_max_prob):
192 |
193 | if "Attn" in opt.Prediction:
194 | gt = gt[: gt.find("[EOS]")]
195 | prd_EOS = prd.find("[EOS]")
196 | prd = prd[:prd_EOS] # prune after "end of sentence" token ([EOS])
197 | prd_max_prob = prd_max_prob[:prd_EOS]
198 |
199 | """we only evaluate the characters included in char_set (=opt.character)."""
200 | gt = gt.replace("[UNK]", "")
201 | prd = prd.replace("[UNK]", "")
202 |
203 | if opt.NED:
204 | # ICDAR2019 Normalized Edit Distance
205 | if len(gt) == 0 or len(prd) == 0:
206 | norm_ED += 0
207 | elif len(gt) > len(prd):
208 | norm_ED += 1 - edit_distance(prd, gt) / len(gt)
209 | else:
210 | norm_ED += 1 - edit_distance(prd, gt) / len(prd)
211 |
212 | else:
213 | if prd == gt:
214 | n_correct += 1
215 |
216 | # calculate confidence score (= multiply of prd_max_prob)
217 | try:
218 | confidence_score = prd_max_prob.cumprod(dim=0)[-1]
219 | except:
220 | confidence_score = 0 # for empty pred case, when prune after "end of sentence" token ([EOS])
221 | confidence_score_list.append(confidence_score)
222 |
223 | if opt.NED:
224 | # ICDAR2019 Normalized Edit Distance. In web page, they report % of norm_ED (= norm_ED * 100).
225 | score = norm_ED / float(length_of_data) * 100
226 | else:
227 | score = n_correct / float(length_of_data) * 100 # accuracy
228 |
229 | return (
230 | valid_loss_avg.val(),
231 | score,
232 | preds_str,
233 | confidence_score_list,
234 | labels_string,
235 | infer_time,
236 | length_of_data,
237 | )
238 |
239 |
240 | def test(opt):
241 | """ model configuration """
242 | if "CTC" in opt.Prediction:
243 | converter = CTCLabelConverter(opt)
244 | else:
245 | converter = AttnLabelConverter(opt)
246 | opt.sos_token_index = converter.dict["[SOS]"]
247 | opt.eos_token_index = converter.dict["[EOS]"]
248 | opt.num_class = len(converter.character)
249 |
250 | model = Model(opt)
251 | print(
252 | "model input parameters",
253 | opt.imgH,
254 | opt.imgW,
255 | opt.num_fiducial,
256 | opt.input_channel,
257 | opt.output_channel,
258 | opt.hidden_size,
259 | opt.num_class,
260 | opt.batch_max_length,
261 | opt.Transformation,
262 | opt.FeatureExtraction,
263 | opt.SequenceModeling,
264 | opt.Prediction,
265 | )
266 | model = torch.nn.DataParallel(model).to(device)
267 |
268 | # load model
269 | print("loading pretrained model from %s" % opt.saved_model)
270 | try:
271 | model.load_state_dict(torch.load(opt.saved_model, map_location=device))
272 | except:
273 | print(
274 | "*** pretrained model not match strictly *** and thus load_state_dict with strict=False mode"
275 | )
276 | # pretrained_state_dict = torch.load(opt.saved_model)
277 | # for name in pretrained_state_dict:
278 | # print(name)
279 | model.load_state_dict(
280 | torch.load(opt.saved_model, map_location=device), strict=False
281 | )
282 |
283 | opt.exp_name = "_".join(opt.saved_model.split("/")[1:])
284 | # print(model)
285 |
286 | """ keep evaluation model and result logs """
287 | os.makedirs(f"./result/{opt.exp_name}", exist_ok=True)
288 | # os.system(f'cp {opt.saved_model} ./result/{opt.exp_name}/')
289 |
290 | """ setup loss """
291 | if "CTC" in opt.Prediction:
292 | criterion = torch.nn.CTCLoss(zero_infinity=True).to(device)
293 | else:
294 | # ignore [PAD] token = ignore index 0
295 | criterion = torch.nn.CrossEntropyLoss(ignore_index=converter.dict["[PAD]"]).to(
296 | device
297 | )
298 |
299 | """ evaluation """
300 | model.eval()
301 | with torch.no_grad():
302 | # evaluate 7 benchmark evaluation datasets
303 | benchmark_all_eval(model, criterion, converter, opt)
304 |
305 |
306 | if __name__ == "__main__":
307 | parser = argparse.ArgumentParser()
308 | parser.add_argument(
309 | "--eval_data", default="data/evaluation/", help="path to evaluation dataset"
310 | )
311 | parser.add_argument(
312 | "--workers", type=int, help="number of data loading workers", default=4
313 | )
314 | parser.add_argument("--batch_size", type=int, default=512, help="input batch size")
315 | parser.add_argument(
316 | "--saved_model", required=True, help="path to saved_model to evaluation"
317 | )
318 | parser.add_argument(
319 | "--log_multiple_test", action="store_true", help="log_multiple_test"
320 | )
321 | """ Data processing """
322 | parser.add_argument(
323 | "--batch_max_length", type=int, default=25, help="maximum-label-length"
324 | )
325 | parser.add_argument(
326 | "--imgH", type=int, default=32, help="the height of the input image"
327 | )
328 | parser.add_argument(
329 | "--imgW", type=int, default=100, help="the width of the input image"
330 | )
331 | parser.add_argument("--NED", default=True, help="For Normalized edit_distance")
332 | parser.add_argument(
333 | "--PAD",
334 | action="store_true",
335 | help="whether to keep ratio then pad for image resize",
336 | )
337 | parser.add_argument(
338 | "--Aug",
339 | type=str,
340 | default="None",
341 | help="whether to use augmentation |None|Crop|Rot|",
342 | )
343 |
344 | """ Model Architecture """
345 | parser.add_argument("--model_name", type=str, required=True, help="CRNN|SVTR")
346 | parser.add_argument(
347 | "--num_fiducial",
348 | type=int,
349 | default=20,
350 | help="number of fiducial points of TPS-STN",
351 | )
352 | parser.add_argument(
353 | "--input_channel",
354 | type=int,
355 | default=3,
356 | help="the number of input channel of Feature extractor",
357 | )
358 | parser.add_argument(
359 | "--output_channel",
360 | type=int,
361 | default=512,
362 | help="the number of output channel of Feature extractor",
363 | )
364 | parser.add_argument(
365 | "--hidden_size", type=int, default=256, help="the size of the LSTM hidden state"
366 | )
367 | parser.add_argument(
368 | "--dataset_name", default="MLT19", help="dataset name |MLT19|",
369 | )
370 | opt = parser.parse_args()
371 |
372 | with open(f"charset/MLT19_charset.txt", "r", encoding="utf-8") as file:
373 | opt.character = file.read().splitlines()
374 |
375 | # for convenience
376 | if opt.model_name == "CRNN": # CRNN
377 | opt.Transformation = "None"
378 | opt.FeatureExtraction = "VGG"
379 | opt.SequenceModeling = "BiLSTM"
380 | opt.Prediction = "CTC"
381 |
382 | elif opt.model_name == "SVTR": # SVTR
383 | opt.Transformation = "None"
384 | opt.FeatureExtraction = "SVTR"
385 | opt.SequenceModeling = "None"
386 | opt.Prediction = "CTC"
387 |
388 | os.makedirs(f"./evaluation_log", exist_ok=True)
389 |
390 | cudnn.benchmark = True
391 | cudnn.deterministic = True
392 | opt.num_gpu = torch.cuda.device_count()
393 |
394 | if opt.num_gpu > 1:
395 | print(
396 | "For lab setting, check your GPU number, you should be missed CUDA_VISIBLE_DEVICES=0 or typo"
397 | )
398 | sys.exit()
399 |
400 | test(opt)
401 |
--------------------------------------------------------------------------------
/modules/svtr.py:
--------------------------------------------------------------------------------
1 | # from https://github.com/simplify23/MRN/blob/23b24806219d66722c916db49c823f96368beb49/modules/svtr.py
2 | import numpy as np
3 | import torch
4 | from torch import nn
5 | from timm.models.layers import trunc_normal_
6 | from functools import partial
7 |
8 |
9 | def drop_path(
10 | x, drop_prob: float = 0.0, training: bool = False, scale_by_keep: bool = True
11 | ):
12 | """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).
13 | This is the same as the DropConnect impl I created for EfficientNet, etc networks, however,
14 | the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper...
15 | See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for
16 | changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use
17 | 'survival rate' as the argument.
18 | """
19 | if drop_prob == 0.0 or not training:
20 | return x
21 | keep_prob = 1 - drop_prob
22 | shape = (x.shape[0],) + (1,) * (
23 | x.ndim - 1
24 | ) # work with diff dim tensors, not just 2D ConvNets
25 | random_tensor = x.new_empty(shape).bernoulli_(keep_prob)
26 | if keep_prob > 0.0 and scale_by_keep:
27 | random_tensor.div_(keep_prob)
28 | return x * random_tensor
29 |
30 |
31 | class DropPath(nn.Module):
32 | """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).
33 | """
34 |
35 | def __init__(self, drop_prob: float = 0.0, scale_by_keep: bool = True):
36 | super(DropPath, self).__init__()
37 | self.drop_prob = drop_prob
38 | self.scale_by_keep = scale_by_keep
39 |
40 | def forward(self, x):
41 | return drop_path(x, self.drop_prob, self.training, self.scale_by_keep)
42 |
43 |
44 | class Identity(nn.Module):
45 | def __init__(self):
46 | super(Identity, self).__init__()
47 |
48 | def forward(self, input):
49 | return input
50 |
51 |
52 | class Mlp(nn.Module):
53 | def __init__(
54 | self,
55 | in_features,
56 | hidden_features=None,
57 | out_features=None,
58 | act_layer=nn.GELU,
59 | drop=0.0,
60 | ):
61 | super().__init__()
62 | out_features = out_features or in_features
63 | hidden_features = hidden_features or in_features
64 | self.fc1 = nn.Linear(in_features, hidden_features)
65 | self.act = act_layer()
66 | self.fc2 = nn.Linear(hidden_features, out_features)
67 | self.drop = nn.Dropout(drop)
68 |
69 | def forward(self, x):
70 | x = self.fc1(x)
71 | x = self.act(x)
72 | x = self.drop(x)
73 | x = self.fc2(x)
74 | x = self.drop(x)
75 | return x
76 |
77 |
78 | class ConvMixer(nn.Module):
79 | def __init__(
80 | self, dim, num_heads=8, HW=[8, 25], local_k=[3, 3],
81 | ):
82 | super().__init__()
83 | self.HW = HW
84 | self.dim = dim
85 | self.local_mixer = nn.Conv2d(
86 | dim, dim, local_k, 1, [local_k[0] // 2, local_k[1] // 2], groups=num_heads
87 | )
88 |
89 | def forward(self, x):
90 | h = self.HW[0]
91 | w = self.HW[1]
92 | x = x.transpose(1, 2).reshape(-1, self.dim, h, w)
93 | x = self.local_mixer(x)
94 | x = x.reshape(-1, self.dim, h * w).transpose(1, 2)
95 | return x
96 |
97 |
98 | class Attention(nn.Module):
99 | def __init__(
100 | self,
101 | dim,
102 | num_heads=8,
103 | mixer="Global",
104 | HW=(8, 25),
105 | local_k=[7, 11],
106 | qkv_bias=False,
107 | qk_scale=None,
108 | attn_drop=0.0,
109 | proj_drop=0.0,
110 | ):
111 | super().__init__()
112 | self.num_heads = num_heads
113 | head_dim = dim // num_heads
114 | self.scale = qk_scale or head_dim ** -0.5
115 |
116 | self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias)
117 | self.attn_drop = nn.Dropout(attn_drop)
118 | self.proj = nn.Linear(dim, dim)
119 | self.proj_drop = nn.Dropout(proj_drop)
120 | self.HW = HW
121 | if HW is not None:
122 | H = HW[0]
123 | W = HW[1]
124 | self.N = H * W
125 | self.C = dim
126 | if mixer == "Local" and HW is not None:
127 | hk = local_k[0]
128 | wk = local_k[1]
129 | mask = torch.ones([H * W, H + hk - 1, W + wk - 1], dtype=torch.float32)
130 | for h in range(0, H):
131 | for w in range(0, W):
132 | mask[h * W + w, h : h + hk, w : w + wk] = 0.0
133 | mask_torch = torch.flatten(
134 | mask[:, hk // 2 : H + hk // 2, wk // 2 : W + wk // 2], 1
135 | )
136 | mask_inf = torch.full([H * W, H * W], -np.inf, dtype=torch.float32)
137 | mask = torch.where(mask_torch < 1, mask_torch, mask_inf)
138 | self.mask = mask.unsqueeze(0)
139 | self.mask = self.mask.unsqueeze(0)
140 | # print(self.mask.size())
141 |
142 | self.mixer = mixer
143 |
144 | def forward(self, x):
145 | if self.HW is not None:
146 | N = self.N
147 | C = self.C
148 | else:
149 | _, N, C = x.size()
150 | qkv = self.qkv(x)
151 | qkv = qkv.reshape((-1, N, 3, self.num_heads, C // self.num_heads))
152 | qkv = qkv.permute(2, 0, 3, 1, 4)
153 | q, k, v = qkv[0] * self.scale, qkv[1], qkv[2]
154 | attn = q.matmul(k.permute(0, 1, 3, 2))
155 | if self.mixer == "Local":
156 | attn += self.mask.to(attn.device)
157 | attn = nn.functional.softmax(attn, dim=-1)
158 | attn = self.attn_drop(attn)
159 |
160 | x = (attn.matmul(v)).permute(0, 2, 1, 3).reshape((-1, N, C))
161 | x = self.proj(x)
162 | x = self.proj_drop(x)
163 | return x
164 |
165 |
166 | class Block(nn.Module):
167 | def __init__(
168 | self,
169 | dim,
170 | num_heads,
171 | mixer="Global",
172 | local_mixer=[7, 11],
173 | HW=[8, 25],
174 | mlp_ratio=4.0,
175 | qkv_bias=False,
176 | qk_scale=None,
177 | drop=0.0,
178 | attn_drop=0.0,
179 | drop_path=0.0,
180 | act_layer=nn.GELU,
181 | norm_layer="nn.LayerNorm",
182 | epsilon=1e-6,
183 | ):
184 | super().__init__()
185 |
186 | self.norm1 = norm_layer(dim)
187 |
188 | if mixer == "Global" or mixer == "Local":
189 | self.mixer = Attention(
190 | dim,
191 | num_heads=num_heads,
192 | mixer=mixer,
193 | HW=HW,
194 | local_k=local_mixer,
195 | qkv_bias=qkv_bias,
196 | qk_scale=qk_scale,
197 | attn_drop=attn_drop,
198 | proj_drop=drop,
199 | )
200 | if mixer == "Conv":
201 | self.mixer = ConvMixer(dim, num_heads=num_heads, HW=HW, local_k=local_mixer)
202 |
203 | # NOTE: drop path for stochastic depth, we shall see if this is better than dropout here
204 | self.drop_path = DropPath(drop_path) if drop_path > 0.0 else Identity()
205 |
206 | self.norm2 = norm_layer(dim)
207 |
208 | mlp_hidden_dim = int(dim * mlp_ratio)
209 | self.mlp_ratio = mlp_ratio
210 | self.mlp = Mlp(
211 | in_features=dim,
212 | hidden_features=mlp_hidden_dim,
213 | act_layer=act_layer,
214 | drop=drop,
215 | )
216 |
217 | def forward(self, x):
218 | self.N, self.C = x.shape[1:]
219 | x = x + self.drop_path(self.mixer(self.norm1(x)))
220 | x = x + self.drop_path(self.mlp(self.norm2(x)))
221 | return x
222 |
223 | def flops(self):
224 | flops = (
225 | self.attn.flops()
226 | + self.N * self.C * 2
227 | + 2 * self.N * self.C * self.C * self.mlp_ratio
228 | )
229 | return flops
230 |
231 |
232 | class PatchEmbed(nn.Module):
233 | """ Image to Patch Embedding
234 | """
235 |
236 | def __init__(self, img_size=[32, 100], in_channels=3, embed_dim=768, sub_num=2):
237 | super().__init__()
238 | # img_size = to_2tuple(img_size)
239 | num_patches = (img_size[1] // (2 ** sub_num)) * (img_size[0] // (2 ** sub_num))
240 | # num_patches = (img_size[1] ) * \
241 | # (img_size[0] )
242 | self.img_size = img_size
243 | self.num_patches = num_patches
244 | self.embed_dim = embed_dim
245 | self.norm = None
246 | if sub_num == 2:
247 | self.proj = nn.Sequential(
248 | nn.Conv2d(in_channels, embed_dim // 2, 3, 2, 1),
249 | nn.BatchNorm2d(embed_dim // 2),
250 | nn.GELU(),
251 | nn.Conv2d(embed_dim // 2, embed_dim, 3, 2, 1),
252 | nn.BatchNorm2d(embed_dim),
253 | nn.GELU(),
254 | )
255 | if sub_num == 3:
256 | self.proj = nn.Sequential(
257 | nn.Conv2d(in_channels, embed_dim // 4, 3, 2, 1),
258 | nn.BatchNorm2d(embed_dim // 4),
259 | nn.GELU(),
260 | nn.Conv2d(embed_dim // 4, embed_dim // 2, 3, 2, 1),
261 | nn.BatchNorm2d(embed_dim // 2),
262 | nn.GELU(),
263 | nn.Conv2d(embed_dim // 2, embed_dim, 3, 2, 1),
264 | nn.BatchNorm2d(embed_dim),
265 | )
266 |
267 | def forward(self, x):
268 | B, C, H, W = x.shape
269 | assert (
270 | H == self.img_size[0] and W == self.img_size[1]
271 | ), f"Input image size ({H}*{W}) doesn't match model ({self.img_size[0]}*{self.img_size[1]})."
272 | # x1 = self.proj1(x)
273 | # x2 = self.proj2(x1).flatten(2).transpose(1, 2)
274 | # x1 = x1.flatten(2).transpose(1, 2)
275 | x = self.proj(x).flatten(2).transpose(1, 2)
276 | return x
277 | # return x1,x2
278 |
279 | def flops(self):
280 | Ho, Wo = self.img_size
281 | flops = (
282 | Ho // 2 * Wo // 2 * 3 * self.embed_dim // 2 * (3 * 3)
283 | + Ho // 4 * Wo // 4 * self.embed_dim // 2 * self.embed_dim * (3 * 3)
284 | + Ho * Wo * self.embed_dim * 2
285 | )
286 | return flops
287 |
288 |
289 | class SubSample(nn.Module):
290 | def __init__(
291 | self,
292 | in_channels,
293 | out_channels,
294 | types="Pool",
295 | stride=[2, 1],
296 | sub_norm="nn.LayerNorm",
297 | act=None,
298 | ):
299 | super().__init__()
300 | self.types = types
301 | if types == "Pool":
302 | self.avgpool = nn.AvgPool2d(
303 | kernel_size=[3, 5], stride=stride, padding=[1, 2]
304 | )
305 | self.maxpool = nn.MaxPool2d(
306 | kernel_size=[3, 5], stride=stride, padding=[1, 2]
307 | )
308 | self.proj = nn.Linear(in_channels, out_channels)
309 | elif types == "Unet":
310 | # For mini-Unet
311 | attn_mode = "nearest"
312 | else:
313 | self.conv = nn.Conv2d(
314 | in_channels, out_channels, kernel_size=3, stride=stride, padding=1
315 | )
316 | self.norm = eval(sub_norm)(out_channels)
317 | if act is not None:
318 | self.act = act()
319 | else:
320 | self.act = None
321 |
322 | def forward(self, x):
323 |
324 | if self.types == "Pool":
325 | # x = x.transpose((0, 2, 1))
326 | x1 = self.avgpool(x)
327 | x2 = self.maxpool(x)
328 | x = (x1 + x2) * 0.5
329 | out = self.proj(x.flatten(2).transpose(1, 2))
330 | elif self.types == "Unet":
331 | # Apply mini U-Net on k
332 | features = []
333 | for i in range(len(self.encoder)):
334 | x = self.encoder[i](x)
335 | features.append(x)
336 | x = self.block(x)
337 | for i in range(len(self.decoder) - 1):
338 | x = self.k_decoder[i](x)
339 | x = x + features[len(self.decoder) - 2 - i]
340 | out = self.decoder[-1](x)
341 | else:
342 | # self.H, self.W = x.shape[2:]
343 | x = self.conv(x)
344 | out = x.flatten(2).transpose(1, 2)
345 | out = self.norm(out)
346 | if self.act is not None:
347 | out = self.act(out)
348 |
349 | return out
350 |
351 |
352 | class SVTR(nn.Module):
353 | def __init__(
354 | self,
355 | # img_size=[32, 256],
356 | img_size=[32, 100],
357 | in_channels=3,
358 | embed_dim=[64, 128, 256],
359 | depth=[3, 6, 3],
360 | num_heads=[2, 4, 8],
361 | mixer=[
362 | "Local",
363 | "Local",
364 | "Local",
365 | "Local",
366 | "Local",
367 | "Local",
368 | "Global",
369 | "Global",
370 | "Global",
371 | "Global",
372 | "Global",
373 | "Global",
374 | ], # Local atten, Global atten, Conv
375 | local_mixer=[[7, 11], [7, 11], [7, 11]],
376 | patch_merging="Conv", # Conv, Pool, None
377 | mlp_ratio=4,
378 | qkv_bias=True,
379 | qk_scale=None,
380 | drop_rate=0.0,
381 | last_drop=0.1,
382 | attn_drop_rate=0.0,
383 | drop_path_rate=0.1,
384 | norm_layer="nn.LayerNorm",
385 | sub_norm="nn.LayerNorm",
386 | epsilon=1e-6,
387 | out_channels=192,
388 | out_char_num=25,
389 | block_unit="Block",
390 | act="nn.GELU",
391 | last_stage=True,
392 | sub_num=2,
393 | **kwargs,
394 | ):
395 | super().__init__()
396 | self.img_size = img_size
397 | self.num_features = self.embed_dim = embed_dim
398 | self.out_channels = out_channels
399 | norm_layer = partial(eval(norm_layer), eps=epsilon)
400 | self.patch_embed = PatchEmbed(
401 | img_size=img_size,
402 | in_channels=in_channels,
403 | embed_dim=embed_dim[0],
404 | sub_num=sub_num,
405 | )
406 | num_patches = self.patch_embed.num_patches
407 | # self.HW = [img_size[0], img_size[1]]
408 | self.HW = [img_size[0] // (2 ** sub_num), img_size[1] // (2 ** sub_num)]
409 | self.pos_embed = nn.Parameter(torch.zeros(1, num_patches, embed_dim[0]))
410 | # self.add_parameter("pos_embed", self.pos_embed)
411 | # self.cls_token = self.create_parameter(
412 | # shape=(1, 1, embed_dim), default_initializer=zeros_)
413 | # self.add_parameter("cls_token", self.cls_token)
414 | self.pos_drop = nn.Dropout(p=drop_rate)
415 | # self.up_linear = nn.Linear(256, 512)
416 | Block_unit = eval(block_unit)
417 | if block_unit == "CSWinBlock":
418 | split_size_h = [1, 2, 2]
419 | split_size_w = [5, 5, 25]
420 | ex_arg = [
421 | {
422 | "reso": [img_size[0] // 4, img_size[1] // 4],
423 | "split_size_h": split_size_h[0],
424 | "split_size_w": split_size_w[0],
425 | },
426 | {
427 | "reso": [img_size[0] // 8, img_size[1] // 4],
428 | "split_size_h": split_size_h[1],
429 | "split_size_w": split_size_w[1],
430 | },
431 | {
432 | "reso": [img_size[0] // 16, img_size[1] // 4],
433 | "split_size_h": split_size_h[2],
434 | "split_size_w": split_size_w[2],
435 | },
436 | ]
437 | else:
438 | ex_arg = [{"epsilon": epsilon}, {"epsilon": epsilon}, {"epsilon": epsilon}]
439 | dpr = np.linspace(0, drop_path_rate, sum(depth))
440 | self.blocks1 = nn.ModuleList(
441 | [
442 | Block_unit(
443 | dim=embed_dim[0],
444 | num_heads=num_heads[0],
445 | mixer=mixer[0 : depth[0]][i],
446 | HW=self.HW,
447 | local_mixer=local_mixer[0],
448 | mlp_ratio=mlp_ratio,
449 | qkv_bias=qkv_bias,
450 | qk_scale=qk_scale,
451 | drop=drop_rate,
452 | act_layer=eval(act),
453 | attn_drop=attn_drop_rate,
454 | drop_path=dpr[0 : depth[0]][i],
455 | norm_layer=norm_layer,
456 | **ex_arg[0],
457 | )
458 | for i in range(depth[0])
459 | ]
460 | )
461 | if patch_merging is not None:
462 | self.sub_sample1 = SubSample(
463 | embed_dim[0],
464 | embed_dim[1],
465 | sub_norm=sub_norm,
466 | stride=[2, 1],
467 | types=patch_merging,
468 | ) # ConvBNLayer(embed_dim[0], embed_dim[1], kernel_size=3, stride=[2, 1], sub_norm=sub_norm)
469 | HW = [self.HW[0] // 2, self.HW[1]]
470 | # self.sub_sample1_0 = SubSample(embed_dim[0], embed_dim[0], sub_norm=sub_norm, stride=[2, 2],
471 | # types=patch_merging) # ConvBNLayer(embed_dim[0], embed_dim[1], kernel_size=3, stride=[2, 1], sub_norm=sub_norm)
472 | # HW = [self.HW[0] // 2, self.HW[1] // 2]
473 | # self.sub_sample1_1 = SubSample(embed_dim[0], embed_dim[1], sub_norm=sub_norm, stride=[1, 1],
474 | # types=patch_merging) # ConvBNLayer(embed_dim[0], embed_dim[1], kernel_size=3, stride=[2, 1], sub_norm=sub_norm)
475 |
476 | else:
477 | HW = self.HW
478 | self.patch_merging = patch_merging
479 | self.blocks2 = nn.ModuleList(
480 | [
481 | Block_unit(
482 | dim=embed_dim[1],
483 | num_heads=num_heads[1],
484 | mixer=mixer[depth[0] : depth[0] + depth[1]][i],
485 | HW=HW,
486 | local_mixer=local_mixer[1],
487 | mlp_ratio=mlp_ratio,
488 | qkv_bias=qkv_bias,
489 | qk_scale=qk_scale,
490 | drop=drop_rate,
491 | act_layer=eval(act),
492 | attn_drop=attn_drop_rate,
493 | drop_path=dpr[depth[0] : depth[0] + depth[1]][i],
494 | norm_layer=norm_layer,
495 | **ex_arg[1],
496 | )
497 | for i in range(depth[1])
498 | ]
499 | )
500 | if patch_merging is not None:
501 | self.sub_sample2 = SubSample(
502 | embed_dim[1],
503 | embed_dim[2],
504 | sub_norm=sub_norm,
505 | stride=[2, 1],
506 | types=patch_merging,
507 | ) # ConvBNLayer(embed_dim[1], embed_dim[2], kernel_size=3, stride=[2, 1], sub_norm=sub_norm)
508 | HW = [self.HW[0] // 4, self.HW[1]]
509 | # self.sub_sample2 = SubSample(embed_dim[1], embed_dim[2], sub_norm=sub_norm, stride=[2, 2],
510 | # types=patch_merging) # ConvBNLayer(embed_dim[1], embed_dim[2], kernel_size=3, stride=[2, 1], sub_norm=sub_norm)
511 | # HW = [self.HW[0] // 4, self.HW[1] // 4]
512 | else:
513 | HW = self.HW
514 | self.blocks3 = nn.ModuleList(
515 | [
516 | Block_unit(
517 | dim=embed_dim[2],
518 | num_heads=num_heads[2],
519 | mixer=mixer[depth[0] + depth[1] :][i],
520 | HW=HW,
521 | local_mixer=local_mixer[2],
522 | mlp_ratio=mlp_ratio,
523 | qkv_bias=qkv_bias,
524 | qk_scale=qk_scale,
525 | drop=drop_rate,
526 | act_layer=eval(act),
527 | attn_drop=attn_drop_rate,
528 | drop_path=dpr[depth[0] + depth[1] :][i],
529 | norm_layer=norm_layer,
530 | **ex_arg[2],
531 | )
532 | for i in range(depth[2])
533 | ]
534 | )
535 | if patch_merging is not None:
536 | # self.sub_sample1 = SubSample(embed_dim[0], embed_dim[1], sub_norm=sub_norm, stride=[2, 1],
537 | # types=patch_merging) # ConvBNLayer(embed_dim[0], embed_dim[1], kernel_size=3, stride=[2, 1], sub_norm=sub_norm)
538 | # HW = [self.HW[0] // 2, self.HW[1]]
539 | self.sub_sample3 = SubSample(
540 | embed_dim[2],
541 | out_channels,
542 | sub_norm=sub_norm,
543 | stride=[2, 1],
544 | types=patch_merging,
545 | ) # ConvBNLayer(embed_dim[0], embed_dim[1], kernel_size=3, stride=[2, 1], sub_norm=sub_norm)
546 |
547 | self.last_stage = last_stage
548 | if last_stage:
549 | self.avg_pool = nn.AdaptiveAvgPool2d([1, out_char_num])
550 | self.linear = nn.Linear(384, 512)
551 |
552 | self.last_conv = nn.Conv2d(
553 | in_channels=embed_dim[2],
554 | out_channels=self.out_channels,
555 | kernel_size=(1, 1),
556 | stride=(1, 1),
557 | padding=0,
558 | bias=False,
559 | )
560 |
561 | self.hardswish = nn.Hardswish()
562 | self.dropout = nn.Dropout(p=last_drop)
563 | self.norm = norm_layer(embed_dim[-1])
564 |
565 | # Classifier head
566 | # self.head = nn.Linear(embed_dim,
567 | # class_num) if class_num > 0 else Identity()
568 |
569 | trunc_normal_(self.pos_embed, std=0.02)
570 | # trunc_normal_(self.cls_token)
571 | self.apply(self._init_weights)
572 |
573 | def _init_weights(self, m):
574 | if isinstance(m, nn.Linear):
575 | trunc_normal_(m.weight, std=0.02)
576 | if isinstance(m, nn.Linear) and m.bias is not None:
577 | nn.init.constant_(m.bias, 0)
578 | elif isinstance(m, nn.LayerNorm):
579 | nn.init.constant_(m.bias, 0)
580 | nn.init.constant_(m.bias, 1.0)
581 | elif isinstance(m, nn.Conv2d):
582 | nn.init.kaiming_normal_(m.weight, mode="fan_in")
583 |
584 | def forward_features(self, x, tpsnet=None):
585 | # B = x.shape[0]
586 | # out = []
587 |
588 | B = x.shape[0]
589 | x = self.patch_embed(x)
590 | # out.append(rearrange(x1, 'b (h w) c -> b c h w',h=32,w=128))
591 | # out.append(rearrange(x, 'b (h w) c -> b c h w',h=32,w=128))
592 | # cls_tokens = self.cls_token.expand((B, -1, -1))
593 | # x = paddle.concat((cls_tokens, x), axis=1)
594 | x = x + self.pos_embed
595 | x = self.pos_drop(x)
596 | for blk in self.blocks1:
597 | x = blk(x)
598 | if self.patch_merging is not None:
599 | x = self.sub_sample1(
600 | x.transpose(1, 2).reshape(B, self.embed_dim[0], self.HW[0], self.HW[1])
601 | )
602 |
603 | for blk in self.blocks2:
604 | x = blk(x)
605 | if self.patch_merging is not None:
606 | x = self.sub_sample2(
607 | x.transpose(1, 2).reshape(
608 | B, self.embed_dim[1], self.HW[0] // 2, self.HW[1]
609 | )
610 | )
611 | # x = self.sub_sample2(x.transpose(1, 2).reshape(B, self.embed_dim[1], self.HW[0] // 2, self.HW[1] // 2))
612 | for blk in self.blocks3:
613 | x = blk(x)
614 | if self.patch_merging is not None:
615 | # x = self.sub_sample2(x.transpose(1, 2).reshape(B, self.embed_dim[1], self.HW[0] // 2, self.HW[1]))
616 | x = self.sub_sample3(
617 | x.transpose(1, 2).reshape(
618 | B, self.embed_dim[2], self.HW[0] // 4, self.HW[1]
619 | )
620 | )
621 | x = x.permute(0, 2, 1).reshape(
622 | [-1, self.out_channels, self.HW[0] // 8, self.HW[1]]
623 | )
624 | return x
625 |
626 | def forward(self, x):
627 | x = self.forward_features(x)
628 | return x
629 |
--------------------------------------------------------------------------------
/train.py:
--------------------------------------------------------------------------------
1 | import os
2 | import sys
3 | import time
4 | import random
5 | import string
6 | import argparse
7 | from datetime import date
8 |
9 | import torch
10 | import torch.backends.cudnn as cudnn
11 | import torch.nn.init as init
12 | import torch.utils.data
13 | import torch.nn.functional as F
14 | import numpy as np
15 | from torch.utils.tensorboard import SummaryWriter
16 | from tqdm import tqdm
17 |
18 | from utils import CTCLabelConverter, AttnLabelConverter, Averager, adjust_learning_rate
19 | from dataset import (
20 | hierarchical_dataset,
21 | AlignCollate,
22 | concat_dataset,
23 | )
24 | from model import Model
25 | from test import validation, benchmark_all_eval
26 |
27 | device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
28 |
29 |
30 | def train(opt, log):
31 | """ model configuration """
32 | if "CTC" in opt.Prediction:
33 | converter = CTCLabelConverter(opt)
34 | else:
35 | converter = AttnLabelConverter(opt)
36 | opt.sos_token_index = converter.dict["[SOS]"]
37 | opt.eos_token_index = converter.dict["[EOS]"]
38 | opt.num_class = len(converter.character)
39 |
40 | model = Model(opt)
41 | print(
42 | "model input parameters",
43 | opt.imgH,
44 | opt.imgW,
45 | opt.num_fiducial,
46 | opt.input_channel,
47 | opt.output_channel,
48 | opt.hidden_size,
49 | opt.num_class,
50 | opt.batch_max_length,
51 | opt.Transformation,
52 | opt.FeatureExtraction,
53 | opt.SequenceModeling,
54 | opt.Prediction,
55 | )
56 |
57 | # weight initialization
58 | for name, param in model.named_parameters():
59 | if "localization_fc2" in name:
60 | print(f"Skip {name} as it is already initialized")
61 | continue
62 |
63 | if opt.FeatureExtraction == "Swin" and "FeatureExtraction" in name:
64 | continue
65 |
66 | try:
67 | if "bias" in name:
68 | init.constant_(param, 0.0)
69 | elif "weight" in name:
70 | init.kaiming_normal_(param)
71 | except Exception as e: # for batchnorm.
72 | if "weight" in name:
73 | param.data.fill_(1)
74 | continue
75 |
76 | # data parallel for multi-GPU
77 | model = torch.nn.DataParallel(model).to(device)
78 | model.train()
79 | if opt.saved_model != "":
80 | fine_tuning_log = f"### loading pretrained model from {opt.saved_model}\n"
81 | pretrained_state_dict = torch.load(opt.saved_model)
82 |
83 | for name, param in model.named_parameters():
84 | try:
85 | param.data.copy_(
86 | pretrained_state_dict[name].data
87 | ) # load from pretrained model
88 | if opt.FT == "freeze":
89 | param.requires_grad = False # Freeze
90 | fine_tuning_log += f"pretrained layer (freezed): {name}\n"
91 | else:
92 | fine_tuning_log += f"pretrained layer: {name}\n"
93 | except:
94 | fine_tuning_log += f"non-pretrained layer: {name}\n"
95 |
96 | print(fine_tuning_log)
97 | log.write(fine_tuning_log + "\n")
98 |
99 | # print("Model:")
100 | # print(model)
101 | log.write(repr(model) + "\n")
102 |
103 | """ dataset preparation """ # for long data loading time, make model then load dataset!
104 | train_dataset, train_dataset_log = concat_dataset(
105 | root=opt.train_data, opt=opt, dataset_list=opt.select_data, mode="train"
106 | )
107 | AlignCollate_train = AlignCollate(opt)
108 | train_loader = torch.utils.data.DataLoader(
109 | train_dataset,
110 | batch_size=opt.batch_size,
111 | shuffle=True,
112 | num_workers=int(opt.workers),
113 | collate_fn=AlignCollate_train,
114 | pin_memory=False,
115 | drop_last=False,
116 | )
117 | train_loader_iter = iter(train_loader)
118 | log.write(train_dataset_log)
119 | print("-" * 80)
120 | log.write("-" * 80 + "\n")
121 |
122 | tmp_valid_data = []
123 | for val_select_d in opt.select_data:
124 | tmp_valid_data.append(val_select_d.replace("SynthMLT_", ""))
125 | valid_select_data = set(tmp_valid_data)
126 | # print(valid_select_data)
127 |
128 | valid_dataset, valid_dataset_log = concat_dataset(
129 | root=opt.valid_data, opt=opt, dataset_list=valid_select_data, mode="test"
130 | )
131 |
132 | AlignCollate_valid = AlignCollate(opt, mode="test")
133 | valid_loader = torch.utils.data.DataLoader(
134 | valid_dataset,
135 | batch_size=opt.batch_size,
136 | shuffle=True, # 'True' to check training progress with validation function.
137 | num_workers=int(opt.workers),
138 | collate_fn=AlignCollate_valid,
139 | pin_memory=False,
140 | )
141 | log.write(valid_dataset_log)
142 | print("-" * 80)
143 | log.write("-" * 80 + "\n")
144 |
145 | """ setup loss """
146 | if "CTC" in opt.Prediction:
147 | criterion = torch.nn.CTCLoss(zero_infinity=True).to(device)
148 | else:
149 | # ignore [PAD] token = ignore index 0
150 | criterion = torch.nn.CrossEntropyLoss(ignore_index=converter.dict["[PAD]"]).to(
151 | device
152 | )
153 |
154 | # loss averager
155 | train_loss_avg = Averager()
156 |
157 | # filter that only require gradient descent
158 | filtered_parameters = []
159 | params_num = []
160 | for p in filter(lambda p: p.requires_grad, model.parameters()):
161 | filtered_parameters.append(p)
162 | params_num.append(np.prod(p.size()))
163 | print(f"Trainable params num: {sum(params_num)}")
164 | log.write(f"Trainable params num: {sum(params_num)}\n")
165 | # [print(name, p.numel()) for name, p in filter(lambda p: p[1].requires_grad, model.named_parameters())]
166 |
167 | # setup optimizer
168 | if opt.optimizer == "sgd":
169 | optimizer = torch.optim.SGD(
170 | filtered_parameters,
171 | lr=opt.lr,
172 | momentum=opt.sgd_momentum,
173 | weight_decay=opt.sgd_weight_decay,
174 | )
175 | elif opt.optimizer == "adadelta":
176 | optimizer = torch.optim.Adadelta(
177 | filtered_parameters, lr=opt.lr, rho=opt.rho, eps=opt.eps
178 | )
179 | elif opt.optimizer == "adam":
180 | optimizer = torch.optim.Adam(filtered_parameters, lr=opt.lr)
181 |
182 | elif opt.optimizer == "adamw":
183 | optimizer = torch.optim.AdamW(filtered_parameters, lr=opt.lr)
184 | print("Optimizer:")
185 | print(optimizer)
186 | log.write(repr(optimizer) + "\n")
187 |
188 | if "super" in opt.schedule:
189 | if opt.optimizer == "sgd":
190 | cycle_momentum = True
191 | else:
192 | cycle_momentum = False
193 |
194 | scheduler = torch.optim.lr_scheduler.OneCycleLR(
195 | optimizer,
196 | max_lr=opt.lr,
197 | cycle_momentum=cycle_momentum,
198 | div_factor=20,
199 | final_div_factor=1000,
200 | total_steps=opt.num_iter,
201 | )
202 | print("Scheduler:")
203 | print(scheduler)
204 | log.write(repr(scheduler) + "\n")
205 |
206 | """ final options """
207 | # print(opt)
208 | opt_log = "------------ Options -------------\n"
209 | args = vars(opt)
210 | for k, v in args.items():
211 | if str(k) == "character" and len(v) > 500:
212 | opt_log += f"{str(k)}: So many characters to show all: number of characters: {len(v)}\n"
213 | else:
214 | opt_log += f"{str(k)}: {str(v)}\n"
215 | opt_log += "---------------------------------------\n"
216 | print(opt_log)
217 | log.write(opt_log)
218 | log.close()
219 |
220 | """ start training """
221 | start_iter = 0
222 | if opt.saved_model != "":
223 | try:
224 | start_iter = int(opt.saved_model.split("_")[-1].split(".")[0])
225 | print(f"continue to train, start_iter: {start_iter}")
226 | except:
227 | pass
228 |
229 | start_time = time.time()
230 | best_score = -1
231 |
232 | # training loop
233 | for iteration in tqdm(
234 | range(start_iter + 1, opt.num_iter + 1),
235 | total=opt.num_iter,
236 | position=0,
237 | leave=True,
238 | ):
239 |
240 | try:
241 | image_tensors, labels = train_loader_iter.next()
242 | except StopIteration:
243 | train_loader_iter = iter(train_loader)
244 | image_tensors, labels = train_loader_iter.next()
245 |
246 | image = image_tensors.to(device)
247 | labels_index, labels_length = converter.encode(
248 | labels, batch_max_length=opt.batch_max_length
249 | )
250 | batch_size = image.size(0)
251 |
252 | # default recognition loss part
253 | if "CTC" in opt.Prediction:
254 | preds = model(image)
255 | preds_size = torch.IntTensor([preds.size(1)] * batch_size)
256 | preds_log_softmax = preds.log_softmax(2).permute(1, 0, 2)
257 | loss = criterion(preds_log_softmax, labels_index, preds_size, labels_length)
258 | else:
259 | preds = model(image, labels_index[:, :-1]) # align with Attention.forward
260 | target = labels_index[:, 1:] # without [SOS] Symbol
261 | loss = criterion(
262 | preds.view(-1, preds.shape[-1]), target.contiguous().view(-1)
263 | )
264 |
265 | model.zero_grad()
266 | loss.backward()
267 | torch.nn.utils.clip_grad_norm_(model.parameters(), opt.grad_clip)
268 | optimizer.step()
269 | train_loss_avg.add(loss)
270 |
271 | if "super" in opt.schedule:
272 | scheduler.step()
273 | else:
274 | adjust_learning_rate(optimizer, iteration, opt)
275 |
276 | # validation part.
277 | # To see training progress, we also conduct validation when 'iteration == 1'
278 | if iteration % opt.val_interval == 0 or iteration == 1:
279 | # for validation log
280 | with open(f"./saved_models/{opt.exp_name}/log_train.txt", "a") as log:
281 | model.eval()
282 | with torch.no_grad():
283 | (
284 | valid_loss,
285 | current_score,
286 | preds,
287 | confidence_score,
288 | labels,
289 | infer_time,
290 | length_of_data,
291 | ) = validation(model, criterion, valid_loader, converter, opt)
292 | model.train()
293 |
294 | # keep best accuracy model (on valid dataset)
295 | # Do not use this on test datasets. It would be an unfair comparison (training should be done without referring test set).
296 | if current_score > best_score:
297 | best_score = current_score
298 | torch.save(
299 | model.state_dict(),
300 | f"./saved_models/{opt.exp_name}/best_score.pth",
301 | )
302 |
303 | # validation log: loss, lr, accuracy, time.
304 | lr = optimizer.param_groups[0]["lr"]
305 | elapsed_time = time.time() - start_time
306 | valid_log = f"\n[{iteration}/{opt.num_iter}] Train_loss: {train_loss_avg.val():0.5f}, Valid_loss: {valid_loss:0.5f}\n"
307 | valid_log += f'{"Current_score":17s}: {current_score:0.2f}, Current_lr: {lr:0.7f}\n'
308 | valid_log += f'{"Best_score":17s}: {best_score:0.2f}, Infer_time: {infer_time:0.1f}, Elapsed_time: {elapsed_time:0.1f}'
309 |
310 | # show some predicted results
311 | dashed_line = "-" * 80
312 | head = f'{"Ground Truth":25s} | {"Prediction":25s} | Confidence Score & T/F'
313 | predicted_result_log = f"{dashed_line}\n{head}\n{dashed_line}\n"
314 | for gt, pred, confidence in zip(
315 | labels[:5], preds[:5], confidence_score[:5]
316 | ):
317 | if "Attn" in opt.Prediction:
318 | gt = gt[: gt.find("[EOS]")]
319 | pred = pred[: pred.find("[EOS]")]
320 |
321 | predicted_result_log += f"{gt:25s} | {pred:25s} | {confidence:0.4f}\t{str(pred == gt)}\n"
322 | predicted_result_log += f"{dashed_line}"
323 | valid_log = f"{valid_log}\n{predicted_result_log}"
324 | print(valid_log)
325 | log.write(valid_log + "\n")
326 |
327 | opt.writer.add_scalar(
328 | "train/train_loss", float(f"{train_loss_avg.val():0.5f}"), iteration
329 | )
330 | opt.writer.add_scalar("train/lr", float(f"{lr:0.7f}"), iteration)
331 | opt.writer.add_scalar(
332 | "train/elapsed_time", float(f"{elapsed_time:0.1f}"), iteration
333 | )
334 | opt.writer.add_scalar(
335 | "valid/valid_loss", float(f"{valid_loss:0.5f}"), iteration
336 | )
337 | opt.writer.add_scalar(
338 | "valid/current_score", float(f"{current_score:0.2f}"), iteration
339 | )
340 | opt.writer.add_scalar(
341 | "valid/best_score", float(f"{best_score:0.2f}"), iteration
342 | )
343 |
344 | train_loss_avg.reset()
345 |
346 | """ Evaluation at the end of training """
347 | # print("Start evaluation on benchmark testset")
348 | """ keep evaluation model and result logs """
349 | os.makedirs(f"./result/{opt.exp_name}", exist_ok=True)
350 | os.makedirs(f"./evaluation_log", exist_ok=True)
351 | saved_best_model = f"./saved_models/{opt.exp_name}/best_score.pth"
352 | # os.system(f'cp {saved_best_model} ./result/{opt.exp_name}/')
353 | model.load_state_dict(torch.load(f"{saved_best_model}"))
354 |
355 | model.eval()
356 | with torch.no_grad():
357 | total_accuracy, eval_data_list, accuracy_list = benchmark_all_eval(
358 | model, criterion, converter, opt
359 | )
360 |
361 | opt.writer.add_scalar(
362 | "test/total_accuracy", float(f"{total_accuracy:0.2f}"), iteration
363 | )
364 | for eval_data, accuracy in zip(eval_data_list, accuracy_list):
365 | accuracy = float(accuracy)
366 | opt.writer.add_scalar(f"test/{eval_data}", float(f"{accuracy:0.2f}"), iteration)
367 |
368 | print(
369 | f'finished the experiment: {opt.exp_name}, "CUDA_VISIBLE_DEVICES" was {opt.CUDA_VISIBLE_DEVICES}'
370 | )
371 |
372 |
373 | if __name__ == "__main__":
374 | parser = argparse.ArgumentParser()
375 | parser.add_argument(
376 | "--train_data", default="data/training", help="path to training dataset",
377 | )
378 | parser.add_argument(
379 | "--valid_data", default="data/validation", help="path to validation dataset",
380 | )
381 | parser.add_argument(
382 | "--eval_data", default="data/evaluation/", help="path to evaluation dataset",
383 | )
384 | parser.add_argument(
385 | "--workers", type=int, default=4, help="number of data loading workers"
386 | )
387 | parser.add_argument("--batch_size", type=int, default=128, help="input batch size")
388 | parser.add_argument(
389 | "--num_iter", type=int, default=100000, help="number of iterations to train for"
390 | )
391 | parser.add_argument(
392 | "--val_interval",
393 | type=int,
394 | default=2000,
395 | help="Interval between each validation",
396 | )
397 | parser.add_argument(
398 | "--log_multiple_test", action="store_true", help="log_multiple_test"
399 | )
400 | parser.add_argument(
401 | "--FT", type=str, default="init", help="whether to do fine-tuning |init|freeze|"
402 | )
403 | parser.add_argument(
404 | "--grad_clip", type=float, default=5, help="gradient clipping value. default=5"
405 | )
406 | """ Optimizer """
407 | parser.add_argument(
408 | "--optimizer", type=str, default="adamw", help="optimizer |sgd|adadelta|adam|"
409 | )
410 | parser.add_argument(
411 | "--lr",
412 | type=float,
413 | # default=0.0005,
414 | default=0.001,
415 | help="learning rate, default=1.0 for Adadelta, 0.0005 for Adam",
416 | )
417 | parser.add_argument(
418 | "--sgd_momentum", default=0.9, type=float, help="momentum for SGD"
419 | )
420 | parser.add_argument(
421 | "--sgd_weight_decay", default=0.000001, type=float, help="weight decay for SGD"
422 | )
423 | parser.add_argument(
424 | "--rho",
425 | type=float,
426 | default=0.95,
427 | help="decay rate rho for Adadelta. default=0.95",
428 | )
429 | parser.add_argument(
430 | "--eps", type=float, default=1e-8, help="eps for Adadelta. default=1e-8"
431 | )
432 | parser.add_argument(
433 | "--schedule",
434 | default="super",
435 | nargs="*",
436 | help="(learning rate schedule. default is super for super convergence, 1 for None, [0.6, 0.8] for the same setting with ASTER",
437 | )
438 | parser.add_argument(
439 | "--lr_drop_rate",
440 | type=float,
441 | default=0.1,
442 | help="lr_drop_rate. default is the same setting with ASTER",
443 | )
444 | """ Model Architecture """
445 | parser.add_argument("--model_name", type=str, required=True, help="CRNN|SVTR")
446 | parser.add_argument(
447 | "--num_fiducial",
448 | type=int,
449 | default=20,
450 | help="number of fiducial points of TPS-STN",
451 | )
452 | parser.add_argument(
453 | "--input_channel",
454 | type=int,
455 | default=3,
456 | help="the number of input channel of Feature extractor",
457 | )
458 | parser.add_argument(
459 | "--output_channel",
460 | type=int,
461 | default=512,
462 | help="the number of output channel of Feature extractor",
463 | )
464 | parser.add_argument(
465 | "--hidden_size", type=int, default=256, help="the size of the LSTM hidden state"
466 | )
467 | """ Data processing """
468 | parser.add_argument(
469 | "--select_data", type=str, default="/", help="select training data",
470 | )
471 | parser.add_argument(
472 | "--batch_ratio",
473 | type=str,
474 | help="assign ratio for each selected data in the batch",
475 | )
476 | parser.add_argument(
477 | "--data_usage",
478 | type=str,
479 | default="all",
480 | help="assign number of data for each selected data",
481 | )
482 | parser.add_argument(
483 | "--batch_max_length", type=int, default=25, help="maximum-label-length"
484 | )
485 | parser.add_argument(
486 | "--imgH", type=int, default=32, help="the height of the input image"
487 | )
488 | parser.add_argument(
489 | "--imgW", type=int, default=100, help="the width of the input image"
490 | )
491 | parser.add_argument("--NED", default=True, help="For Normalized edit_distance")
492 | parser.add_argument(
493 | "--PAD",
494 | action="store_true",
495 | help="whether to keep ratio then pad for image resize",
496 | )
497 | parser.add_argument(
498 | "--Aug",
499 | type=str,
500 | default="None",
501 | help="whether to use augmentation |None|Crop|Rot|",
502 | )
503 | parser.add_argument(
504 | "--test", type=str, default="None", help="to verify temporal try"
505 | )
506 | """ exp_name and etc """
507 | parser.add_argument("--exp_name", help="Where to store logs and models")
508 | parser.add_argument(
509 | "--manual_seed", type=int, default=21, help="for random seed setting"
510 | )
511 | parser.add_argument(
512 | "--saved_model", default="", help="path to model to continue training"
513 | )
514 | parser.add_argument(
515 | "--dataset_name", default="MLT19", help="dataset name |MLT19|",
516 | )
517 | parser.add_argument(
518 | "--data_usage_LAT",
519 | type=str,
520 | default="all",
521 | help="assign number of data for each selected data",
522 | )
523 | parser.add_argument(
524 | "--data_usage_SynthMLT",
525 | type=str,
526 | default="all",
527 | help="assign number of data for each selected data",
528 | )
529 |
530 | opt = parser.parse_args()
531 |
532 | opt.select_data = opt.select_data.split("-")
533 | tmp_select_data = []
534 | for data_group in opt.select_data:
535 | if "lang_" in data_group:
536 | for lang in list(data_group.replace("lang_", "")):
537 | if lang == "A":
538 | tmp_select_data.append("ARA")
539 | elif lang == "L":
540 | tmp_select_data.append("LAT")
541 | elif lang == "C":
542 | tmp_select_data.append("CHI")
543 | elif lang == "J":
544 | tmp_select_data.append("JPN")
545 | elif lang == "K":
546 | tmp_select_data.append("KOR")
547 | elif lang == "B":
548 | tmp_select_data.append("BEN")
549 | elif lang == "H":
550 | tmp_select_data.append("HIN")
551 |
552 | elif "SynthMLT_" in data_group:
553 | for lang in list(data_group.replace("SynthMLT_", "")):
554 | if lang == "A":
555 | tmp_select_data.append("SynthMLT_ARA")
556 | elif lang == "L":
557 | tmp_select_data.append("SynthMLT_LAT")
558 | elif lang == "C":
559 | tmp_select_data.append("SynthMLT_CHI")
560 | elif lang == "J":
561 | tmp_select_data.append("SynthMLT_JPN")
562 | elif lang == "K":
563 | tmp_select_data.append("SynthMLT_KOR")
564 | elif lang == "B":
565 | tmp_select_data.append("SynthMLT_BEN")
566 | elif lang == "H":
567 | tmp_select_data.append("SynthMLT_HIN")
568 |
569 | else:
570 | tmp_select_data.append(data_group)
571 |
572 | opt.select_data = tmp_select_data
573 | print(opt.select_data)
574 |
575 | with open(f"charset/MLT19_charset.txt", "r", encoding="utf-8") as file:
576 | opt.character = file.read().splitlines()
577 |
578 | # for convenience
579 | if opt.model_name == "CRNN": # CRNN
580 | opt.Transformation = "None"
581 | opt.FeatureExtraction = "VGG"
582 | opt.SequenceModeling = "BiLSTM"
583 | opt.Prediction = "CTC"
584 |
585 | elif opt.model_name == "SVTR": # SVTR
586 | opt.Transformation = "None"
587 | opt.FeatureExtraction = "SVTR"
588 | opt.SequenceModeling = "None"
589 | opt.Prediction = "CTC"
590 |
591 | """ Seed and GPU setting """
592 | random.seed(opt.manual_seed)
593 | np.random.seed(opt.manual_seed)
594 | torch.manual_seed(opt.manual_seed)
595 | torch.cuda.manual_seed_all(opt.manual_seed) # if you are using multi-GPU.
596 | torch.cuda.manual_seed(opt.manual_seed)
597 |
598 | cudnn.benchmark = True # It fasten training.
599 | cudnn.deterministic = True
600 |
601 | opt.gpu_name = "_".join(torch.cuda.get_device_name().split())
602 | opt.CUDA_VISIBLE_DEVICES = os.environ["CUDA_VISIBLE_DEVICES"]
603 | opt.num_gpu = torch.cuda.device_count()
604 | if opt.num_gpu > 1:
605 | print(
606 | "We recommend to use 1 GPU, check your GPU number, you would miss CUDA_VISIBLE_DEVICES=0 or typo"
607 | )
608 | print("To use multi-gpu setting, remove or comment out these lines")
609 | sys.exit()
610 |
611 | """ directory and log setting """
612 | if not opt.exp_name:
613 | opt.exp_name = f"Seed{opt.manual_seed}-{opt.model_name}-{opt.gpu_name}"
614 |
615 | os.makedirs(f"./saved_models/{opt.exp_name}", exist_ok=True)
616 | log = open(f"./saved_models/{opt.exp_name}/log_train.txt", "a")
617 | command_line_input = " ".join(sys.argv)
618 | print(
619 | f"Command line input: CUDA_VISIBLE_DEVICES={opt.CUDA_VISIBLE_DEVICES} python {command_line_input}"
620 | )
621 | log.write(
622 | f"Command line input: CUDA_VISIBLE_DEVICES={opt.CUDA_VISIBLE_DEVICES} python {command_line_input}\n"
623 | )
624 | os.makedirs(f"./tensorboard", exist_ok=True)
625 | opt.writer = SummaryWriter(log_dir=f"./tensorboard/{opt.exp_name}")
626 |
627 | train(opt, log)
628 |
--------------------------------------------------------------------------------
/charset/MLT19_charset.txt:
--------------------------------------------------------------------------------
1 | ০
2 | ०
3 | ٠
4 | १
5 | ١
6 | ১
7 | २
8 | ২
9 | ٢
10 | ३
11 | ٣
12 | ৩
13 | ৪
14 | ٤
15 | ४
16 | ৫
17 | ٥
18 | ५
19 | ٦
20 | ৬
21 | ६
22 | ৭
23 | ٧
24 | ७
25 | ८
26 | ٨
27 | ৮
28 | ৯
29 | ٩
30 | ९
31 | A
32 | À
33 | Â
34 | Ä
35 | B
36 | C
37 | Ç
38 | D
39 | E
40 | È
41 | É
42 | Ê
43 | F
44 | G
45 | H
46 | I
47 | Ì
48 | J
49 | K
50 | L
51 | M
52 | N
53 | O
54 | Ò
55 | Ô
56 | Ö
57 | P
58 | Q
59 | R
60 | S
61 | T
62 | U
63 | Ù
64 | Ü
65 | V
66 | W
67 | X
68 | Y
69 | Z
70 | a
71 | à
72 | â
73 | ä
74 | b
75 | c
76 | ç
77 | d
78 | e
79 | è
80 | é
81 | ê
82 | f
83 | g
84 | h
85 | i
86 | ì
87 | î
88 | j
89 | k
90 | l
91 | m
92 | n
93 | o
94 | ò
95 | ó
96 | ô
97 | ö
98 | p
99 | q
100 | r
101 | s
102 | t
103 | u
104 | ù
105 | ú
106 | û
107 | ü
108 | v
109 | w
110 | x
111 | y
112 | z
113 | ß
114 | Œ
115 | œ
116 | ء
117 | ا
118 | آ
119 | أ
120 | إ
121 | ب
122 | ة
123 | ت
124 | ث
125 | ج
126 | ح
127 | خ
128 | د
129 | ذ
130 | ر
131 | ز
132 | س
133 | ش
134 | ص
135 | ض
136 | ط
137 | ظ
138 | ع
139 | غ
140 | ف
141 | ق
142 | ك
143 | ل
144 | م
145 | ن
146 | ه
147 | و
148 | ؤ
149 | ى
150 | ي
151 | ئ
152 | ٫
153 | ڥ
154 | ڨ
155 | अ
156 | अँ
157 | अं
158 | अॅ
159 | अॅँ
160 | आ
161 | आँ
162 | आं
163 | इ
164 | इं
165 | ई
166 | ईं
167 | उ
168 | ऊ
169 | ऊँ
170 | ऊं
171 | ऋ
172 | ए
173 | ऐं
174 | ऑ
175 | ओ
176 | ओं
177 | औ
178 | औं
179 | क
180 | कँ
181 | कं
182 | का
183 | काँ
184 | कां
185 | कि
186 | की
187 | कु
188 | कुं
189 | कू
190 | कृ
191 | कॅ
192 | के
193 | कें
194 | कै
195 | कॉ
196 | को
197 | क्ट
198 | क्टं
199 | क्ट्रि
200 | क्त
201 | क्ति
202 | क्त्दा
203 | क्मि
204 | क्य
205 | क्र
206 | क्रं
207 | क्रा
208 | क्रां
209 | क्रि
210 | क्रिं
211 | क्री
212 | क्रॉ
213 | क्लि
214 | क्ल्प
215 | क्वा
216 | क्श
217 | क्शं
218 | क्ष
219 | क्षा
220 | क्षि
221 | क्षे
222 | क्ष्मी
223 | क्ष्म्ण
224 | क्स
225 | क्स्
226 | ख
227 | खं
228 | खा
229 | खि
230 | खी
231 | खे
232 | ग
233 | गं
234 | गा
235 | गां
236 | गि
237 | गी
238 | गु
239 | गृ
240 | गे
241 | गो
242 | गौ
243 | ग्ण
244 | ग्णे
245 | ग्न
246 | ग्य
247 | ग्या
248 | ग्यु
249 | ग्र
250 | ग्रं
251 | ग्री
252 | ग्रु
253 | ग्रे
254 | ग्लि
255 | ग्लो
256 | घ
257 | घा
258 | घां
259 | घु
260 | घे
261 | घो
262 | घ्या
263 | घ्वा
264 | च
265 | चँ
266 | चं
267 | चा
268 | चि
269 | चिं
270 | ची
271 | चु
272 | चे
273 | चें
274 | चै
275 | चो
276 | चौ
277 | च्छ
278 | च्छ्ता
279 | च्या
280 | च्व्हा
281 | छ
282 | छा
283 | छो
284 | ज
285 | जं
286 | जा
287 | जि
288 | जिं
289 | जी
290 | जु
291 | जे
292 | जें
293 | जॉ
294 | जो
295 | ज्
296 | ज्ज
297 | ज्जा
298 | ज्जी
299 | ज्ञ
300 | ज्ञा
301 | ज्य
302 | ज्या
303 | ज्ये
304 | ज्वे
305 | झ
306 | झं
307 | झा
308 | झे
309 | ट
310 | टं
311 | टा
312 | टि
313 | टी
314 | टीं
315 | टे
316 | टो
317 | ट्ट
318 | ट्टी
319 | ट्टे
320 | ट्य
321 | ट्रा
322 | ट्राँ
323 | ट्री
324 | ट्रे
325 | ट्रो
326 | ट्स
327 | ठ
328 | ठा
329 | ठी
330 | ठे
331 | ठ्ठ
332 | ड
333 | डं
334 | ड़
335 | ड़ी
336 | डा
337 | डां
338 | डि
339 | डी
340 | डु
341 | डे
342 | डें
343 | डो
344 | ड्डा
345 | ड्र
346 | ड्रे
347 | ड्ला
348 | ढ
349 | ढ़ी
350 | ढा
351 | ढी
352 | ढे
353 | ढो
354 | ण
355 | णा
356 | णि
357 | णी
358 | णू
359 | णे
360 | ण्ड
361 | ण्डा
362 | ण्णा
363 | ण्य
364 | ण्या
365 | त
366 | तं
367 | ता
368 | तां
369 | ति
370 | ती
371 | तु
372 | तू
373 | तृ
374 | ते
375 | तो
376 | तों
377 | त्त
378 | त्ता
379 | त्ते
380 | त्न
381 | त्ना
382 | त्मा
383 | त्य
384 | त्या
385 | त्यि
386 | त्य्मे
387 | त्र
388 | त्रा
389 | त्रि
390 | त्री
391 | त्रे
392 | त्र्य
393 | त्र्या
394 | त्स
395 | त्सं
396 | त्सा
397 | थ
398 | था
399 | थां
400 | थि
401 | थी
402 | थु
403 | थे
404 | थो
405 | थ्वी
406 | द
407 | दा
408 | दां
409 | दि
410 | दी
411 | दु
412 | दुं
413 | दू
414 | दृ
415 | दे
416 | दो
417 | द्गु
418 | द्द
419 | द्दी
420 | द्द्
421 | द्ध
422 | द्धा
423 | द्धि
424 | द्धी
425 | द्ध्
426 | द्य
427 | द्या
428 | द्यो
429 | द्र
430 | द्रा
431 | द्री
432 | द्रे
433 | द्वा
434 | ध
435 | धं
436 | धा
437 | धि
438 | धी
439 | धु
440 | धे
441 | धो
442 | धों
443 | धौ
444 | ध्द
445 | ध्दी
446 | ध्य
447 | ध्या
448 | ध्व
449 | न
450 | नं
451 | ना
452 | नां
453 | नि
454 | निं
455 | नी
456 | नु
457 | नू
458 | नॅ
459 | ने
460 | नै
461 | नॉ
462 | नो
463 | नों
464 | न्ट
465 | न्ट्र
466 | न्ट्री
467 | न्ड
468 | न्ड्री
469 | न्त
470 | न्द्र
471 | न्द्रा
472 | न्न
473 | न्य
474 | न्यः
475 | न्या
476 | न्यु
477 | न्ये
478 | न्स
479 | न्सी
480 | न्स्
481 | न्स्ता
482 | न्ही
483 | न्हे
484 | प
485 | पँ
486 | पं
487 | पा
488 | पाँ
489 | पां
490 | पि
491 | पिं
492 | पी
493 | पीं
494 | पु
495 | पू
496 | पृ
497 | पॅ
498 | पे
499 | पै
500 | पॉ
501 | पो
502 | पौ
503 | प्ट
504 | प्त
505 | प्ती
506 | प्ना
507 | प्प
508 | प्पा
509 | प्यु
510 | प्र
511 | प्रा
512 | प्रि
513 | प्रिं
514 | प्रे
515 | प्रो
516 | प्रौ
517 | प्ला
518 | प्ले
519 | प्लॉ
520 | फ
521 | फँ
522 | फा
523 | फि
524 | फु
525 | फे
526 | फो
527 | फ्स्
528 | ब
529 | बँ
530 | बं
531 | बा
532 | बाँ
533 | बि
534 | बी
535 | बु
536 | बॅ
537 | बे
538 | बै
539 | बैँ
540 | बैं
541 | बो
542 | बौ
543 | ब्ध
544 | ब्धि
545 | ब्ब
546 | ब्या
547 | ब्यु
548 | ब्र
549 | ब्री
550 | ब्ला
551 | ब्ली
552 | ब्ल्यू
553 | भ
554 | भं
555 | भा
556 | भां
557 | भि
558 | भिं
559 | भी
560 | भु
561 | भू
562 | भे
563 | भै
564 | भो
565 | भों
566 | भ्यं
567 | म
568 | मं
569 | मः
570 | मा
571 | मां
572 | मि
573 | मी
574 | मु
575 | मू
576 | मृ
577 | मे
578 | में
579 | मै
580 | मो
581 | म्
582 | म्चं
583 | म्प
584 | म्पो
585 | म्प्ले
586 | म्ब
587 | म्म
588 | म्मा
589 | म्र
590 | म्रा
591 | म्स
592 | म्ह
593 | म्हा
594 | म्ही
595 | य
596 | यं
597 | या
598 | याँ
599 | यां
600 | यु
601 | यू
602 | ये
603 | यो
604 | य्य
605 | र
606 | रं
607 | रा
608 | रां
609 | रि
610 | री
611 | रु
612 | रू
613 | रूं
614 | रे
615 | रें
616 | रॉ
617 | रो
618 | र्क
619 | र्किं
620 | र्कु
621 | र्के
622 | र्ग
623 | र्गा
624 | र्गी
625 | र्च
626 | र्चि
627 | र्जी
628 | र्ट
629 | र्टी
630 | र्ट्मे
631 | र्ट्में
632 | र्ट्स्
633 | र्ड
634 | र्डा
635 | र्ण
636 | र्ती
637 | र्थ
638 | र्थी
639 | र्द
640 | र्दि
641 | र्ध
642 | र्धि
643 | र्नि
644 | र्फ
645 | र्म
646 | र्मा
647 | र्मि
648 | र्य
649 | र्या
650 | र्ल
651 | र्व
652 | र्वां
653 | र्वि
654 | र्वे
655 | र्श
656 | र्श्व
657 | र्ष
658 | र्षा
659 | र्षी
660 | र्स
661 | र्सिं
662 | ल
663 | लँ
664 | लं
665 | ला
666 | लां
667 | लि
668 | ली
669 | लीं
670 | लु
671 | लू
672 | ले
673 | लॉ
674 | लो
675 | लों
676 | ल्टी
677 | ल्ड
678 | ल्डीं
679 | ल्प
680 | ल्म
681 | ल्य
682 | ल्या
683 | ल्रा
684 | ल्ली
685 | ल्व्ह
686 | ल्स
687 | ल्स्
688 | ल्हा
689 | ळ
690 | ळं
691 | ळा
692 | ळी
693 | ळु
694 | ळू
695 | ळे
696 | ळो
697 | व
698 | वं
699 | वा
700 | वां
701 | वि
702 | विं
703 | वी
704 | वु
705 | वू
706 | वृ
707 | वे
708 | वें
709 | वै
710 | वैं
711 | व्दै
712 | व्य
713 | व्यं
714 | व्या
715 | व्ह
716 | व्हा
717 | व्हे
718 | व्हो
719 | श
720 | शं
721 | शा
722 | शां
723 | शि
724 | शिं
725 | शी
726 | शु
727 | शे
728 | शें
729 | शै
730 | शॉ
731 | शो
732 | शौ
733 | श्चं
734 | श्पां
735 | श्मा
736 | श्य
737 | श्या
738 | श्र
739 | श्रा
740 | श्रां
741 | श्री
742 | श्रृं
743 | श्रे
744 | श्व
745 | श्व्र
746 | श्ह
747 | ष
748 | षां
749 | षि
750 | षे
751 | ष्ट
752 | ष्टी
753 | ष्टृ
754 | ष्ट्र
755 | ष्ट्रा
756 | ष्ट्री
757 | ष्ठ
758 | ष्ठा
759 | ष्ण
760 | ष्णा
761 | ष्णु
762 | ष्णू
763 | ष्प
764 | ष्य
765 | स
766 | सं
767 | सा
768 | सां
769 | सि
770 | सिं
771 | सी
772 | सु
773 | सुं
774 | सू
775 | सृ
776 | से
777 | सें
778 | सै
779 | सो
780 | सौ
781 | स्
782 | स्क
783 | स्कू
784 | स्कृ
785 | स्जि
786 | स्ट
787 | स्टँ
788 | स्टा
789 | स्टि
790 | स्टी
791 | स्टु
792 | स्टे
793 | स्टॉ
794 | स्टो
795 | स्ट्री
796 | स्ठ
797 | स्त
798 | स्तं
799 | स्ता
800 | स्ति
801 | स्तु
802 | स्त्या
803 | स्त्र
804 | स्त्री
805 | स्थ
806 | स्था
807 | स्थे
808 | स्नँ
809 | स्नॅ
810 | स्पा
811 | स्पि
812 | स्पी
813 | स्पे
814 | स्म
815 | स्मा
816 | स्मृ
817 | स्मे
818 | स्य
819 | स्या
820 | स्री
821 | स्व
822 | स्वा
823 | स्वी
824 | ह
825 | हं
826 | हा
827 | हाँ
828 | हां
829 | हि
830 | हिं
831 | ही
832 | हु
833 | हू
834 | हे
835 | है
836 | हॉ
837 | हो
838 | हौ
839 | ह्वा
840 | ॐ
841 | ঃ
842 | অ
843 | অা
844 | অ্যা
845 | আ
846 | আা
847 | ই
848 | ইং
849 | ঈ
850 | উ
851 | ঋ
852 | এ
853 | এী
854 | এ্য
855 | এ্যা
856 | ও
857 | ওঁ
858 | ঔ
859 | ঔঁ
860 | ক
861 | কং
862 | কা
863 | কাঁ
864 | কি
865 | কিং
866 | কী
867 | কু
868 | কৃ
869 | কৃা
870 | কৄ
871 | কে
872 | কো
873 | কোং
874 | ক্ট
875 | ক্ত
876 | ক্তি
877 | ক্যা
878 | ক্র
879 | ক্রঃ
880 | ক্রা
881 | ক্রি
882 | ক্রী
883 | ক্রে
884 | ক্লা
885 | ক্লি
886 | ক্ল্ু
887 | ক্ষ
888 | ক্ষা
889 | ক্ষী
890 | ক্ষু
891 | ক্ষে
892 | ক্ষ্মী
893 | ক্স
894 | খ
895 | খা
896 | খাঁ
897 | খি
898 | খু
899 | খূ
900 | খে
901 | খো
902 | খ্রীঃ
903 | গ
904 | গা
905 | গাঁ
906 | গি
907 | গিং
908 | গী
909 | গু
910 | গুঁ
911 | গৃ
912 | গৄ
913 | গে
914 | গো
915 | গৌ
916 | গ্য
917 | গ্যা
918 | গ্র
919 | গ্রা
920 | গ্রী
921 | গ্রে
922 | গ্লা
923 | ঘ
924 | ঘা
925 | ঘো
926 | ঙি
927 | ঙ্ক
928 | ঙ্কে
929 | ঙ্খ
930 | ঙ্গ
931 | ঙ্গী
932 | চ
933 | চা
934 | চাঁ
935 | চি
936 | চিং
937 | চী
938 | চু
939 | চে
940 | চৌ
941 | চৈ
942 | চ্
943 | চ্চ
944 | চ্ছ
945 | চ্ছা
946 | চ্ছে
947 | ছ
948 | ছা
949 | ছি
950 | ছু
951 | ছে
952 | ছো
953 | জ
954 | জং
955 | জা
956 | জি
957 | জী
958 | জু
959 | জে
960 | জো
961 | জ্
962 | জ্ঞ
963 | জ্ঞা
964 | জ্য
965 | জ্যো
966 | ঝি
967 | ঞ্চ
968 | ঞ্চা
969 | ঞ্জ
970 | ঞ্জা
971 | ঞ্জি
972 | ঞ্জু
973 | ট
974 | টা
975 | টি
976 | টিং
977 | টী
978 | টু
979 | টে
980 | টো
981 | ট্
982 | ট্টা
983 | ট্যা
984 | ট্র
985 | ট্রা
986 | ট্রি
987 | ট্রে
988 | ঠ
989 | ঠা
990 | ঠি
991 | ড
992 | ডঃ
993 | ড়
994 | ড়
995 | ড়া
996 | ড়া
997 | ড়ি
998 | ড়ি
999 | ড়ী
1000 | ড়ী
1001 | ড়ীূূ
1002 | ড়ে
1003 | ড়ো
1004 | ডা
1005 | ডাঃ
1006 | ডি
1007 | ডিং
1008 | ডু
1009 | ডে
1010 | ডো
1011 | ড্ডা
1012 | ড্রা
1013 | ড্রে
1014 | ঢ
1015 | ঢা
1016 | ণ
1017 | ণা
1018 | ণি
1019 | ণী
1020 | ণে
1021 | ণ্জ
1022 | ণ্ড
1023 | ণ্ডা
1024 | ণ্ডি
1025 | ণ্ডিং
1026 | ণ্যা
1027 | ত
1028 | তা
1029 | তি
1030 | তী
1031 | তু
1032 | তৃ
1033 | তে
1034 | তো
1035 | তৈ
1036 | ত্ত
1037 | ত্ন
1038 | ত্বে
1039 | ত্য
1040 | ত্যা
1041 | ত্যে
1042 | ত্র
1043 | ত্রি
1044 | ত্রী
1045 | ত্ল
1046 | থ
1047 | থা
1048 | থি
1049 | থী
1050 | থু
1051 | থে
1052 | দ
1053 | দা
1054 | দাঁ
1055 | দি
1056 | দী
1057 | দু
1058 | দে
1059 | দো
1060 | দ্বা
1061 | দ্বি
1062 | দ্মা
1063 | দ্যা
1064 | দ্র
1065 | দ্রু
1066 | ধ
1067 | ধা
1068 | ধি
1069 | ধী
1070 | ধু
1071 | ধূ
1072 | ধে
1073 | ধো
1074 | ধ্য
1075 | ধ্যা
1076 | ন
1077 | নং
1078 | নঃ
1079 | না
1080 | নি
1081 | নিং
1082 | নী
1083 | নু
1084 | নে
1085 | নো
1086 | ন্
1087 | ন্ট
1088 | ন্টা
1089 | ন্টি
1090 | ন্টিং
1091 | ন্টী
1092 | ন্টে
1093 | ন্ট্রা
1094 | ন্ড
1095 | ন্ডা
1096 | ন্ডি
1097 | ন্ডু
1098 | ন্ডূ
1099 | ন্ত
1100 | ন্তা
1101 | ন্তি
1102 | ন্তু
1103 | ন্ত্রি
1104 | ন্ত্ু
1105 | ন্দ
1106 | ন্দা
1107 | ন্দি
1108 | ন্দী
1109 | ন্দু
1110 | ন্দ্য
1111 | ন্দ্যো
1112 | ন্দ্র
1113 | ন্দ্রে
1114 | ন্ধ
1115 | ন্ধা
1116 | ন্ধী
1117 | ন্ধু
1118 | ন্ধ্যা
1119 | ন্ন
1120 | ন্ব
1121 | ন্ম
1122 | ন্য
1123 | ন্যা
1124 | ন্স
1125 | ন্সা
1126 | ন্সি
1127 | প
1128 | পঃ
1129 | পা
1130 | পি
1131 | পী
1132 | পু
1133 | পূ
1134 | পে
1135 | পো
1136 | পৌ
1137 | প্
1138 | প্ত
1139 | প্যা
1140 | প্র
1141 | প্র়়
1142 | প্রা
1143 | প্রাঃ
1144 | প্রি
1145 | প্রী
1146 | প্রে
1147 | প্রো
1148 | প্রোঃ
1149 | প্ল
1150 | প্লা
1151 | প্লে
1152 | প্স
1153 | ফ
1154 | ফা
1155 | ফি
1156 | ফু
1157 | ফে
1158 | ফো
1159 | ফ্ট
1160 | ফ্যা
1161 | ফ্র
1162 | ফ্রা
1163 | ফ্রি
1164 | ফ্রী
1165 | ব
1166 | বঁা
1167 | বং
1168 | বা
1169 | বাঁ
1170 | বাং
1171 | বি
1172 | বী
1173 | বু
1174 | বৃ
1175 | বে
1176 | বো
1177 | বৈঃ
1178 | ব্দী
1179 | ব্ধ
1180 | ব্য
1181 | ব্যা
1182 | ব্যে
1183 | ব্রি
1184 | ব্রী
1185 | ব্রু
1186 | ব্ল
1187 | ব্লা
1188 | ব্লি
1189 | ব্লু
1190 | ভ
1191 | ভা
1192 | ভি
1193 | ভু
1194 | ভুঁ
1195 | ভূ
1196 | ভে
1197 | ভো
1198 | ভ্য
1199 | ভ্যা
1200 | ম
1201 | মা
1202 | মি
1203 | মী
1204 | মু
1205 | মূ
1206 | মে
1207 | মো
1208 | মোঃ
1209 | মৌ
1210 | মৈ
1211 | ম্
1212 | ম্প
1213 | ম্পা
1214 | ম্পি
1215 | ম্প্র
1216 | ম্প্রী
1217 | ম্প্রো
1218 | ম্ব
1219 | ম্বা
1220 | ম্বে
1221 | ম্ব্র
1222 | ম্ব্্র
1223 | ম্ভ
1224 | ম্ভা
1225 | ম্ম
1226 | ম্মে
1227 | ম্য
1228 | ম্যা
1229 | ম্স
1230 | য
1231 | য়
1232 | য়
1233 | য়া
1234 | য়া
1235 | য়ি
1236 | য়ী
1237 | য়ে
1238 | য়ে
1239 | য়ো
1240 | য়্যা
1241 | যা
1242 | যু
1243 | যে
1244 | যো
1245 | য্য
1246 | য্যা
1247 | র
1248 | রং
1249 | রা
1250 | রাঃ
1251 | রি
1252 | রিং
1253 | রী
1254 | রু
1255 | রূ
1256 | রে
1257 | রো
1258 | রৌ
1259 | র্ক
1260 | র্কা
1261 | র্কী
1262 | র্কে
1263 | র্গ
1264 | র্গা
1265 | র্গাো
1266 | র্গী
1267 | র্গো
1268 | র্চ
1269 | র্চি
1270 | র্চে
1271 | র্জ
1272 | র্জা
1273 | র্জি
1274 | র্জী
1275 | র্জু
1276 | র্জে
1277 | র্ট
1278 | র্ড
1279 | র্ডে
1280 | র্ণ
1281 | র্ণি
1282 | র্ত
1283 | র্তি
1284 | র্তী
1285 | র্ত্তী
1286 | র্থ
1287 | র্থি
1288 | র্থী
1289 | র্থে
1290 | র্দা
1291 | র্দ্য
1292 | র্ধ
1293 | র্ন
1294 | র্নি
1295 | র্পো
1296 | র্ব
1297 | র্বা
1298 | র্ব্ব
1299 | র্ভ
1300 | র্ভি
1301 | র্ভু
1302 | র্ম
1303 | র্মা
1304 | র্মি
1305 | র্মী
1306 | র্মে
1307 | র্ম্মা
1308 | র্য্য
1309 | র্য্যা
1310 | র্লা
1311 | র্শ
1312 | র্শি
1313 | র্ষ
1314 | র্ষি
1315 | র্স
1316 | র্সিং
1317 | ল
1318 | লং
1319 | লা
1320 | লি
1321 | লিং
1322 | লী
1323 | লু
1324 | লে
1325 | লো
1326 | ল্ক
1327 | ল্ট
1328 | ল্ড
1329 | ল্ডা
1330 | ল্প
1331 | ল্পী
1332 | ল্যা
1333 | ল্যে
1334 | ল্স
1335 | ল্স্
1336 | শ
1337 | শা
1338 | শি
1339 | শিং
1340 | শী
1341 | শু
1342 | শূ
1343 | শে
1344 | শো
1345 | শৌ
1346 | শৈ
1347 | শ্চি
1348 | শ্ব
1349 | শ্বা
1350 | শ্য
1351 | শ্যা
1352 | শ্র
1353 | শ্রা
1354 | শ্রী
1355 | শ্রে
1356 | শ্র্ু
1357 | ষ
1358 | ষি
1359 | ষু
1360 | ষে
1361 | ষো
1362 | ষ্ট
1363 | ষ্টা
1364 | ষ্টি
1365 | ষ্টী
1366 | ষ্টে
1367 | ষ্টো
1368 | ষ্ট্রা
1369 | ষ্ট্রে
1370 | ষ্ঠ
1371 | ষ্ঠা
1372 | ষ্ণ
1373 | ষ্ণু
1374 | ষ্যা
1375 | স
1376 | সং
1377 | স়
1378 | সা
1379 | সাং
1380 | সি
1381 | সিঁ
1382 | সিং
1383 | সী
1384 | সু
1385 | সূ
1386 | সে
1387 | সো
1388 | সৌ
1389 | স্
1390 | স্ক
1391 | স্কা
1392 | স্কু
1393 | স্ক্যা
1394 | স্ক্রী
1395 | স্ট
1396 | স্টা
1397 | স্টি
1398 | স্টী
1399 | স্টে
1400 | স্টো
1401 | স্ট্
1402 | স্ট্যা
1403 | স্ট্রি
1404 | স্ট্রী
1405 | স্ট্রে
1406 | স্ট্ু
1407 | স্ত
1408 | স্তু
1409 | স্ত্ত
1410 | স্ত্রা
1411 | স্ত্রী
1412 | স্ত্ু
1413 | স্থ
1414 | স্থা
1415 | স্থ্য
1416 | স্না
1417 | স্পা
1418 | স্পে
1419 | স্বা
1420 | স্বাঃ
1421 | স্বী
1422 | স্ব্্রা
1423 | স্ম
1424 | স্যা
1425 | হ
1426 | হং
1427 | হা
1428 | হি
1429 | হী
1430 | হু
1431 | হূ
1432 | হৃ
1433 | হে
1434 | হো
1435 | হ্নে
1436 | হ্যা
1437 | ৎ
1438 | ৷
1439 | 가
1440 | 각
1441 | 간
1442 | 갈
1443 | 감
1444 | 갑
1445 | 값
1446 | 강
1447 | 같
1448 | 갚
1449 | 개
1450 | 객
1451 | 거
1452 | 건
1453 | 걷
1454 | 걸
1455 | 검
1456 | 겁
1457 | 것
1458 | 겅
1459 | 게
1460 | 겠
1461 | 겡
1462 | 겨
1463 | 격
1464 | 견
1465 | 결
1466 | 겸
1467 | 겹
1468 | 경
1469 | 계
1470 | 고
1471 | 곡
1472 | 곤
1473 | 골
1474 | 곰
1475 | 곱
1476 | 곳
1477 | 공
1478 | 과
1479 | 곽
1480 | 관
1481 | 광
1482 | 괴
1483 | 굉
1484 | 교
1485 | 구
1486 | 국
1487 | 군
1488 | 굴
1489 | 굽
1490 | 궁
1491 | 궈
1492 | 권
1493 | 귀
1494 | 규
1495 | 균
1496 | 그
1497 | 극
1498 | 근
1499 | 글
1500 | 금
1501 | 급
1502 | 긋
1503 | 기
1504 | 긴
1505 | 길
1506 | 김
1507 | 깃
1508 | 깅
1509 | 깊
1510 | 까
1511 | 깐
1512 | 깔
1513 | 깡
1514 | 깨
1515 | 꺼
1516 | 껌
1517 | 께
1518 | 꼇
1519 | 꼬
1520 | 꼭
1521 | 꼼
1522 | 꽁
1523 | 꽃
1524 | 꽉
1525 | 꾜
1526 | 꾸
1527 | 꿈
1528 | 꿍
1529 | 끄
1530 | 끈
1531 | 끓
1532 | 끔
1533 | 끗
1534 | 끼
1535 | 낀
1536 | ᄂ드
1537 | 나
1538 | 낙
1539 | 난
1540 | 날
1541 | 남
1542 | 납
1543 | 낭
1544 | 내
1545 | 낸
1546 | 냉
1547 | 냐
1548 | 너
1549 | 널
1550 | 넓
1551 | 넘
1552 | 넛
1553 | 네
1554 | 넥
1555 | 넷
1556 | 넹
1557 | 년
1558 | 녋
1559 | 념
1560 | 노
1561 | 녹
1562 | 논
1563 | 농
1564 | 높
1565 | 놓
1566 | 뇌
1567 | 누
1568 | 눈
1569 | 눔
1570 | 눠
1571 | 뉴
1572 | 느
1573 | 는
1574 | 늘
1575 | 능
1576 | 니
1577 | 닉
1578 | 닌
1579 | 닐
1580 | 님
1581 | 닝
1582 | ᄃ장
1583 | 다
1584 | 닥
1585 | 단
1586 | 닫
1587 | 달
1588 | 닭
1589 | 닮
1590 | 담
1591 | 답
1592 | 당
1593 | 대
1594 | 더
1595 | 덕
1596 | 던
1597 | 덧
1598 | 덩
1599 | 덮
1600 | 데
1601 | 덱
1602 | 델
1603 | 뎀
1604 | 뎅
1605 | 도
1606 | 독
1607 | 돈
1608 | 돔
1609 | 동
1610 | 돟
1611 | 돱
1612 | 돼
1613 | 되
1614 | 된
1615 | 될
1616 | 됩
1617 | 두
1618 | 둠
1619 | 둥
1620 | 듀
1621 | 드
1622 | 득
1623 | 든
1624 | 들
1625 | 듬
1626 | 듭
1627 | 듯
1628 | 등
1629 | 디
1630 | 딜
1631 | 딥
1632 | 딨
1633 | 딩
1634 | 따
1635 | 딸
1636 | 때
1637 | 땐
1638 | 떡
1639 | 떳
1640 | 떻
1641 | 떼
1642 | 뗙
1643 | 뗜
1644 | 또
1645 | 뚜
1646 | 뛰
1647 | 뜨
1648 | 뜻
1649 | 띠
1650 | 띰
1651 | ᄅ핫
1652 | 라
1653 | 락
1654 | 란
1655 | 랄
1656 | 람
1657 | 랑
1658 | 래
1659 | 랙
1660 | 랜
1661 | 램
1662 | 랩
1663 | 랫
1664 | 량
1665 | 러
1666 | 런
1667 | 럼
1668 | 럽
1669 | 럿
1670 | 렇
1671 | 레
1672 | 렉
1673 | 렌
1674 | 렐
1675 | 려
1676 | 력
1677 | 련
1678 | 렬
1679 | 로
1680 | 록
1681 | 론
1682 | 롤
1683 | 롭
1684 | 롯
1685 | 롱
1686 | 뢰
1687 | 료
1688 | 루
1689 | 룩
1690 | 룬
1691 | 룸
1692 | 뤼
1693 | 류
1694 | 륜
1695 | 륨
1696 | 르
1697 | 륵
1698 | 륶
1699 | 른
1700 | 를
1701 | 름
1702 | 릅
1703 | 릇
1704 | 리
1705 | 릭
1706 | 린
1707 | 릴
1708 | 림
1709 | 립
1710 | 릿
1711 | 링
1712 | 마
1713 | 막
1714 | 만
1715 | 많
1716 | 말
1717 | 맛
1718 | 망
1719 | 맞
1720 | 맟
1721 | 매
1722 | 맥
1723 | 맨
1724 | 먜
1725 | 머
1726 | 먹
1727 | 먼
1728 | 멋
1729 | 멍
1730 | 메
1731 | 멕
1732 | 멘
1733 | 멜
1734 | 멤
1735 | 멩
1736 | 며
1737 | 면
1738 | 멸
1739 | 몁
1740 | 명
1741 | 몇
1742 | 모
1743 | 목
1744 | 몫
1745 | 몰
1746 | 몸
1747 | 못
1748 | 묘
1749 | 무
1750 | 묵
1751 | 묶
1752 | 문
1753 | 물
1754 | 뮈
1755 | 뮤
1756 | 믈
1757 | 믑
1758 | 미
1759 | 믹
1760 | 민
1761 | 믿
1762 | 밀
1763 | 밌
1764 | 밎
1765 | 및
1766 | 바
1767 | 바ᅵ
1768 | 박
1769 | 반
1770 | 받
1771 | 발
1772 | 밝
1773 | 밤
1774 | 밥
1775 | 밧
1776 | 방
1777 | 배
1778 | 백
1779 | 밴
1780 | 뱃
1781 | 뱅
1782 | 버
1783 | 번
1784 | 벌
1785 | 범
1786 | 법
1787 | 베
1788 | 벤
1789 | 벨
1790 | 벼
1791 | 벽
1792 | 변
1793 | 별
1794 | 병
1795 | 보
1796 | 복
1797 | 볶
1798 | 본
1799 | 볼
1800 | 봄
1801 | 봉
1802 | 봐
1803 | 부
1804 | 북
1805 | 붂
1806 | 분
1807 | 불
1808 | 붐
1809 | 붓
1810 | 붙
1811 | 뷔
1812 | 브
1813 | 브ᅡ
1814 | 븍
1815 | 블
1816 | 븥
1817 | 비
1818 | 빅
1819 | 빈
1820 | 빌
1821 | 빔
1822 | 빗
1823 | 빙
1824 | 빚
1825 | 빛
1826 | 빠
1827 | 빨
1828 | 빵
1829 | 뼈
1830 | 뽕
1831 | 뾰
1832 | 뿍
1833 | 뿐
1834 | 쁜
1835 | 사
1836 | 삭
1837 | 산
1838 | 살
1839 | 삼
1840 | 삿
1841 | 상
1842 | 새
1843 | 색
1844 | 샌
1845 | 샐
1846 | 샘
1847 | 생
1848 | 샤
1849 | 샴
1850 | 샵
1851 | 샾
1852 | 섕
1853 | 서
1854 | 석
1855 | 선
1856 | 설
1857 | 섭
1858 | 성
1859 | 세
1860 | 센
1861 | 셀
1862 | 셔
1863 | 셕
1864 | 션
1865 | 셰
1866 | 소
1867 | 속
1868 | 손
1869 | 솔
1870 | 솜
1871 | 송
1872 | 솥
1873 | 쇠
1874 | 쇼
1875 | 수
1876 | 숙
1877 | 순
1878 | 술
1879 | 숨
1880 | 숭
1881 | 숯
1882 | 쉐
1883 | 쉬
1884 | 쉴
1885 | 슈
1886 | 슐
1887 | 스
1888 | 슥
1889 | 슨
1890 | 슬
1891 | 슴
1892 | 습
1893 | 승
1894 | 슼
1895 | 싀
1896 | 시
1897 | 식
1898 | 신
1899 | 실
1900 | 싫
1901 | 심
1902 | 십
1903 | 싱
1904 | ᄊ랑
1905 | 싸
1906 | 싼
1907 | 쌀
1908 | 쌩
1909 | 써
1910 | 쏟
1911 | 쑤
1912 | 쓰
1913 | 씀
1914 | 씌
1915 | 씨
1916 | 아
1917 | 악
1918 | 안
1919 | 않
1920 | 알
1921 | 암
1922 | 압
1923 | 앗
1924 | 았
1925 | 앙
1926 | 앞
1927 | 애
1928 | 액
1929 | 앤
1930 | 앰
1931 | 야
1932 | 약
1933 | 양
1934 | 어
1935 | 억
1936 | 언
1937 | 얹
1938 | 얻
1939 | 얼
1940 | 엄
1941 | 업
1942 | 없
1943 | 었
1944 | 엉
1945 | 에
1946 | 엔
1947 | 엘
1948 | 엠
1949 | 엣
1950 | 여
1951 | 역
1952 | 연
1953 | 엳
1954 | 열
1955 | 염
1956 | 엽
1957 | 영
1958 | 옆
1959 | 예
1960 | 옛
1961 | 오
1962 | 옥
1963 | 온
1964 | 올
1965 | 옴
1966 | 옹
1967 | 와
1968 | 완
1969 | 왕
1970 | 외
1971 | 요
1972 | 요ᅵ
1973 | 욕
1974 | 욘
1975 | 용
1976 | 우
1977 | 욱
1978 | 운
1979 | 울
1980 | 움
1981 | 웃
1982 | 웅
1983 | 워
1984 | 웍
1985 | 원
1986 | 월
1987 | 웨
1988 | 웰
1989 | 웹
1990 | 위
1991 | 유
1992 | 육
1993 | 윤
1994 | 율
1995 | 융
1996 | 으
1997 | 은
1998 | 을
1999 | 음
2000 | 읍
2001 | 응
2002 | 의
2003 | 이
2004 | 익
2005 | 인
2006 | 일
2007 | 임
2008 | 입
2009 | 있
2010 | 잉
2011 | 잊
2012 | 자
2013 | 작
2014 | 잔
2015 | 잖
2016 | 잘
2017 | 잠
2018 | 잡
2019 | 장
2020 | 재
2021 | 잭
2022 | 쟁
2023 | 쟘
2024 | 쟝
2025 | 저
2026 | 적
2027 | 전
2028 | 절
2029 | 젊
2030 | 점
2031 | 접
2032 | 정
2033 | 제
2034 | 젠
2035 | 젤
2036 | 져
2037 | 조
2038 | 족
2039 | 존
2040 | 졸
2041 | 종
2042 | 좋
2043 | 좌
2044 | 죄
2045 | 주
2046 | 죽
2047 | 준
2048 | 줄
2049 | 중
2050 | 줘
2051 | 쥬
2052 | 즈
2053 | 즉
2054 | 즌
2055 | 즐
2056 | 증
2057 | 즤
2058 | 지
2059 | 직
2060 | 진
2061 | 질
2062 | 짐
2063 | 집
2064 | 징
2065 | 짜
2066 | 짝
2067 | 짬
2068 | 쩝
2069 | 쪼
2070 | 쪽
2071 | 쫄
2072 | 쭈
2073 | 찌
2074 | 찐
2075 | 찔
2076 | 차
2077 | 착
2078 | 찬
2079 | 찰
2080 | 참
2081 | 찻
2082 | 창
2083 | 찾
2084 | 채
2085 | 책
2086 | 챌
2087 | 챗
2088 | 챠
2089 | 처
2090 | 척
2091 | 천
2092 | 철
2093 | 첨
2094 | 첩
2095 | 첫
2096 | 청
2097 | 체
2098 | 첼
2099 | 쳤
2100 | 초
2101 | 촉
2102 | 총
2103 | 촬
2104 | 최
2105 | 추
2106 | 축
2107 | 춘
2108 | 출
2109 | 춤
2110 | 충
2111 | 취
2112 | 츠
2113 | 측
2114 | 층
2115 | 치
2116 | 칙
2117 | 친
2118 | 칠
2119 | 침
2120 | 칭
2121 | 카
2122 | 칸
2123 | 칼
2124 | 캅
2125 | 캐
2126 | 캔
2127 | 캠
2128 | 캡
2129 | 컘
2130 | 커
2131 | 컨
2132 | 컬
2133 | 컴
2134 | 컵
2135 | 케
2136 | 켈
2137 | 켓
2138 | 켜
2139 | 켭
2140 | 켸
2141 | 코
2142 | 콕
2143 | 콘
2144 | 콜
2145 | 콤
2146 | 콩
2147 | 쾌
2148 | 쿄
2149 | 쿠
2150 | 쿤
2151 | 쿨
2152 | 큐
2153 | 크
2154 | 큰
2155 | 클
2156 | 큼
2157 | 킈
2158 | 키
2159 | 킨
2160 | 킬
2161 | 킴
2162 | 킹
2163 | 타
2164 | 탁
2165 | 탄
2166 | 탈
2167 | 탐
2168 | 탑
2169 | 탕
2170 | 태
2171 | 택
2172 | 탠
2173 | 탱
2174 | 턔
2175 | 터
2176 | 턴
2177 | 털
2178 | 턺
2179 | 텀
2180 | 테
2181 | 텍
2182 | 텐
2183 | 텔
2184 | 템
2185 | 텨
2186 | 토
2187 | 톡
2188 | 톤
2189 | 톨
2190 | 통
2191 | 퇘
2192 | 투
2193 | 툰
2194 | 튀
2195 | 튬
2196 | 트
2197 | 특
2198 | 튼
2199 | 틀
2200 | 틈
2201 | 티
2202 | 틱
2203 | 틴
2204 | 틸
2205 | 팀
2206 | 팁
2207 | 팅
2208 | ᄑ젬
2209 | 파
2210 | 팍
2211 | 판
2212 | 팔
2213 | 팜
2214 | 팝
2215 | 팟
2216 | 팡
2217 | 팥
2218 | 패
2219 | 팩
2220 | 팬
2221 | 팻
2222 | 퍼
2223 | 펀
2224 | 펄
2225 | 페
2226 | 펴
2227 | 편
2228 | 평
2229 | 폐
2230 | 포
2231 | 폭
2232 | 폰
2233 | 폴
2234 | 폼
2235 | 표
2236 | 푸
2237 | 푼
2238 | 풀
2239 | 품
2240 | 풍
2241 | 퓨
2242 | 프
2243 | 픈
2244 | 플
2245 | 픔
2246 | 픗
2247 | 픚
2248 | 픠
2249 | 피
2250 | 핀
2251 | 핃
2252 | 필
2253 | 핌
2254 | 핏
2255 | 핑
2256 | 하
2257 | 학
2258 | 한
2259 | 할
2260 | 함
2261 | 합
2262 | 핫
2263 | 항
2264 | 해
2265 | 핸
2266 | 햄
2267 | 했
2268 | 행
2269 | 햐
2270 | 향
2271 | 허
2272 | 헌
2273 | 험
2274 | 헙
2275 | 헤
2276 | 헨
2277 | 혀
2278 | 혁
2279 | 현
2280 | 혈
2281 | 협
2282 | 형
2283 | 혜
2284 | 호
2285 | 혼
2286 | 홀
2287 | 홈
2288 | 홉
2289 | 홍
2290 | 화
2291 | 환
2292 | 활
2293 | 황
2294 | 홰
2295 | 회
2296 | 획
2297 | 효
2298 | 후
2299 | 훈
2300 | 훤
2301 | 훼
2302 | 휘
2303 | 휴
2304 | 흐
2305 | 흐ᅡ
2306 | 흑
2307 | 흔
2308 | 흡
2309 | 흥
2310 | 희
2311 | 흰
2312 | 히
2313 | 힉
2314 | 힌
2315 | 힐
2316 | 힘
2317 | ぁ
2318 | あ
2319 | ぃ
2320 | い
2321 | ぅ
2322 | う
2323 | ぇ
2324 | え
2325 | ぉ
2326 | お
2327 | か
2328 | が
2329 | き
2330 | ぎ
2331 | く
2332 | ぐ
2333 | け
2334 | げ
2335 | こ
2336 | ご
2337 | さ
2338 | ざ
2339 | し
2340 | じ
2341 | す
2342 | ず
2343 | せ
2344 | そ
2345 | ぞ
2346 | た
2347 | だ
2348 | ち
2349 | っ
2350 | つ
2351 | て
2352 | で
2353 | と
2354 | ど
2355 | な
2356 | に
2357 | ぬ
2358 | ね
2359 | の
2360 | は
2361 | ば
2362 | ぱ
2363 | ひ
2364 | び
2365 | ぴ
2366 | ふ
2367 | ぶ
2368 | ぷ
2369 | へ
2370 | べ
2371 | ぺ
2372 | ほ
2373 | ぼ
2374 | ぽ
2375 | ま
2376 | み
2377 | む
2378 | め
2379 | も
2380 | ゃ
2381 | や
2382 | ゆ
2383 | ょ
2384 | よ
2385 | ら
2386 | り
2387 | る
2388 | れ
2389 | ろ
2390 | ゎ
2391 | わ
2392 | を
2393 | ん
2394 | ァ
2395 | ア
2396 | ィ
2397 | イ
2398 | ウ
2399 | ヴ
2400 | ェ
2401 | エ
2402 | ォ
2403 | オ
2404 | カ
2405 | ガ
2406 | キ
2407 | ギ
2408 | ク
2409 | グ
2410 | ケ
2411 | ゲ
2412 | コ
2413 | ゴ
2414 | サ
2415 | ザ
2416 | シ
2417 | ジ
2418 | ス
2419 | ズ
2420 | セ
2421 | ゼ
2422 | ソ
2423 | ゾ
2424 | タ
2425 | ダ
2426 | チ
2427 | ッ
2428 | ツ
2429 | ヅ
2430 | テ
2431 | デ
2432 | ト
2433 | ド
2434 | ナ
2435 | ニ
2436 | ヌ
2437 | ネ
2438 | ノ
2439 | ハ
2440 | バ
2441 | パ
2442 | ヒ
2443 | ビ
2444 | ピ
2445 | フ
2446 | ブ
2447 | プ
2448 | ヘ
2449 | ベ
2450 | ペ
2451 | ホ
2452 | ボ
2453 | ポ
2454 | マ
2455 | ミ
2456 | ム
2457 | メ
2458 | モ
2459 | ャ
2460 | ヤ
2461 | ュ
2462 | ユ
2463 | ョ
2464 | ヨ
2465 | ラ
2466 | リ
2467 | ル
2468 | レ
2469 | ロ
2470 | ヮ
2471 | ワ
2472 | ヱ
2473 | ヲ
2474 | ン
2475 | ヵ
2476 | ヶ
2477 | ・
2478 | ー
2479 | 一
2480 | 丁
2481 | 七
2482 | 万
2483 | 三
2484 | 上
2485 | 下
2486 | 不
2487 | 与
2488 | 专
2489 | 且
2490 | 世
2491 | 丘
2492 | 业
2493 | 东
2494 | 丝
2495 | 両
2496 | 丢
2497 | 两
2498 | 严
2499 | 个
2500 | 丫
2501 | 中
2502 | 丰
2503 | 临
2504 | 丸
2505 | 丹
2506 | 为
2507 | 主
2508 | 丼
2509 | 丽
2510 | 举
2511 | 乃
2512 | 久
2513 | 么
2514 | 义
2515 | 之
2516 | 乌
2517 | 乎
2518 | 乐
2519 | 乗
2520 | 乘
2521 | 九
2522 | 也
2523 | 习
2524 | 乡
2525 | 书
2526 | 买
2527 | 乱
2528 | 亀
2529 | 了
2530 | 予
2531 | 争
2532 | 事
2533 | 二
2534 | 于
2535 | 云
2536 | 互
2537 | 五
2538 | 井
2539 | 亚
2540 | 交
2541 | 产
2542 | 亨
2543 | 享
2544 | 京
2545 | 亭
2546 | 亮
2547 | 人
2548 | 什
2549 | 仁
2550 | 今
2551 | 介
2552 | 仍
2553 | 从
2554 | 仑
2555 | 仓
2556 | 仕
2557 | 他
2558 | 付
2559 | 仙
2560 | 代
2561 | 令
2562 | 以
2563 | 仪
2564 | 们
2565 | 仲
2566 | 件
2567 | 价
2568 | 任
2569 | 份
2570 | 企
2571 | 伊
2572 | 伎
2573 | 伐
2574 | 休
2575 | 众
2576 | 优
2577 | 会
2578 | 伝
2579 | 伟
2580 | 传
2581 | 伤
2582 | 伸
2583 | 似
2584 | 但
2585 | 位
2586 | 低
2587 | 住
2588 | 佐
2589 | 佑
2590 | 体
2591 | 何
2592 | 佛
2593 | 作
2594 | 你
2595 | 佰
2596 | 佳
2597 | 佶
2598 | 使
2599 | 例
2600 | 供
2601 | 依
2602 | 侠
2603 | 価
2604 | 侧
2605 | 侨
2606 | 侯
2607 | 便
2608 | 係
2609 | 俄
2610 | 俊
2611 | 俏
2612 | 俗
2613 | 保
2614 | 信
2615 | 修
2616 | 俱
2617 | 倉
2618 | 個
2619 | 倍
2620 | 倒
2621 | 候
2622 | 倚
2623 | 倡
2624 | 债
2625 | 值
2626 | 假
2627 | 做
2628 | 停
2629 | 健
2630 | 側
2631 | 偷
2632 | 傅
2633 | 備
2634 | 催
2635 | 傷
2636 | 傾
2637 | 働
2638 | 像
2639 | 僕
2640 | 償
2641 | 優
2642 | 儿
2643 | 元
2644 | 兄
2645 | 先
2646 | 光
2647 | 克
2648 | 免
2649 | 児
2650 | 党
2651 | 入
2652 | 內
2653 | 全
2654 | 八
2655 | 公
2656 | 六
2657 | 兰
2658 | 共
2659 | 关
2660 | 兴
2661 | 兵
2662 | 其
2663 | 具
2664 | 典
2665 | 兹
2666 | 养
2667 | 冀
2668 | 内
2669 | 円
2670 | 冈
2671 | 再
2672 | 写
2673 | 军
2674 | 农
2675 | 冠
2676 | 冬
2677 | 冰
2678 | 冲
2679 | 冷
2680 | 净
2681 | 准
2682 | 凍
2683 | 凤
2684 | 処
2685 | 出
2686 | 击
2687 | 刀
2688 | 分
2689 | 切
2690 | 刊
2691 | 刑
2692 | 划
2693 | 列
2694 | 刘
2695 | 创
2696 | 初
2697 | 判
2698 | 別
2699 | 利
2700 | 别
2701 | 到
2702 | 制
2703 | 刷
2704 | 券
2705 | 刻
2706 | 剂
2707 | 削
2708 | 前
2709 | 剑
2710 | 剖
2711 | 剤
2712 | 剧
2713 | 剩
2714 | 剪
2715 | 割
2716 | 創
2717 | 劑
2718 | 力
2719 | 办
2720 | 功
2721 | 加
2722 | 务
2723 | 劣
2724 | 动
2725 | 助
2726 | 劫
2727 | 劳
2728 | 効
2729 | 勇
2730 | 勋
2731 | 勑
2732 | 勒
2733 | 動
2734 | 勘
2735 | 務
2736 | 勝
2737 | 募
2738 | 勢
2739 | 勤
2740 | 勿
2741 | 包
2742 | 化
2743 | 北
2744 | 匙
2745 | 匹
2746 | 区
2747 | 医
2748 | 十
2749 | 千
2750 | 升
2751 | 午
2752 | 半
2753 | 华
2754 | 协
2755 | 卒
2756 | 協
2757 | 单
2758 | 卖
2759 | 南
2760 | 単
2761 | 博
2762 | 卜
2763 | 占
2764 | 卡
2765 | 卧
2766 | 卫
2767 | 印
2768 | 危
2769 | 即
2770 | 却
2771 | 卷
2772 | 厂
2773 | 厅
2774 | 历
2775 | 压
2776 | 厕
2777 | 厚
2778 | 原
2779 | 厢
2780 | 厦
2781 | 厨
2782 | 厳
2783 | 去
2784 | 县
2785 | 参
2786 | 又
2787 | 及
2788 | 友
2789 | 双
2790 | 反
2791 | 収
2792 | 发
2793 | 取
2794 | 受
2795 | 变
2796 | 叠
2797 | 口
2798 | 古
2799 | 句
2800 | 只
2801 | 召
2802 | 叭
2803 | 可
2804 | 台
2805 | 史
2806 | 右
2807 | 叶
2808 | 号
2809 | 司
2810 | 吃
2811 | 各
2812 | 合
2813 | 吉
2814 | 吊
2815 | 同
2816 | 名
2817 | 后
2818 | 吐
2819 | 向
2820 | 君
2821 | 吞
2822 | 吧
2823 | 听
2824 | 启
2825 | 吴
2826 | 吸
2827 | 呂
2828 | 呉
2829 | 告
2830 | 员
2831 | 周
2832 | 味
2833 | 命
2834 | 和
2835 | 咨
2836 | 咽
2837 | 品
2838 | 哈
2839 | 哉
2840 | 响
2841 | 哗
2842 | 員
2843 | 哪
2844 | 唐
2845 | 售
2846 | 唱
2847 | 商
2848 | 問
2849 | 善
2850 | 喇
2851 | 喉
2852 | 喜
2853 | 喧
2854 | 喫
2855 | 喰
2856 | 営
2857 | 喷
2858 | 嗣
2859 | 嘉
2860 | 嘴
2861 | 噌
2862 | 器
2863 | 嚴
2864 | 囚
2865 | 四
2866 | 回
2867 | 因
2868 | 团
2869 | 団
2870 | 园
2871 | 困
2872 | 図
2873 | 围
2874 | 固
2875 | 国
2876 | 图
2877 | 圃
2878 | 圆
2879 | 圈
2880 | 國
2881 | 圏
2882 | 園
2883 | 圓
2884 | 圖
2885 | 團
2886 | 土
2887 | 圣
2888 | 圧
2889 | 在
2890 | 地
2891 | 圳
2892 | 场
2893 | 圾
2894 | 址
2895 | 均
2896 | 坊
2897 | 坎
2898 | 坏
2899 | 坐
2900 | 坑
2901 | 块
2902 | 坛
2903 | 坝
2904 | 坠
2905 | 坡
2906 | 坦
2907 | 垂
2908 | 垃
2909 | 型
2910 | 垚
2911 | 垣
2912 | 城
2913 | 域
2914 | 埠
2915 | 培
2916 | 基
2917 | 埼
2918 | 堂
2919 | 堅
2920 | 堆
2921 | 堤
2922 | 報
2923 | 場
2924 | 堵
2925 | 塑
2926 | 塔
2927 | 塗
2928 | 塚
2929 | 塞
2930 | 塩
2931 | 境
2932 | 墓
2933 | 増
2934 | 墙
2935 | 墨
2936 | 壁
2937 | 壇
2938 | 士
2939 | 壮
2940 | 声
2941 | 壱
2942 | 売
2943 | 壹
2944 | 处
2945 | 备
2946 | 変
2947 | 复
2948 | 夏
2949 | 夕
2950 | 外
2951 | 多
2952 | 夜
2953 | 够
2954 | 大
2955 | 天
2956 | 太
2957 | 夫
2958 | 央
2959 | 失
2960 | 头
2961 | 夹
2962 | 奇
2963 | 奈
2964 | 奉
2965 | 奋
2966 | 奎
2967 | 奖
2968 | 套
2969 | 奥
2970 | 女
2971 | 奶
2972 | 她
2973 | 好
2974 | 如
2975 | 妆
2976 | 妇
2977 | 妝
2978 | 妹
2979 | 姊
2980 | 始
2981 | 姐
2982 | 姓
2983 | 委
2984 | 姚
2985 | 威
2986 | 娘
2987 | 娱
2988 | 婚
2989 | 婦
2990 | 媒
2991 | 媳
2992 | 嫌
2993 | 子
2994 | 孔
2995 | 字
2996 | 存
2997 | 孝
2998 | 孟
2999 | 季
3000 | 学
3001 | 孩
3002 | 孵
3003 | 學
3004 | 宁
3005 | 宅
3006 | 宇
3007 | 守
3008 | 安
3009 | 完
3010 | 宏
3011 | 官
3012 | 定
3013 | 宜
3014 | 宝
3015 | 实
3016 | 実
3017 | 宠
3018 | 客
3019 | 宣
3020 | 室
3021 | 宫
3022 | 宮
3023 | 害
3024 | 宴
3025 | 家
3026 | 容
3027 | 宽
3028 | 宾
3029 | 宿
3030 | 寅
3031 | 密
3032 | 富
3033 | 寒
3034 | 寓
3035 | 察
3036 | 寧
3037 | 寨
3038 | 寶
3039 | 对
3040 | 寺
3041 | 寻
3042 | 导
3043 | 対
3044 | 寿
3045 | 封
3046 | 専
3047 | 将
3048 | 專
3049 | 小
3050 | 少
3051 | 尔
3052 | 尚
3053 | 就
3054 | 尻
3055 | 尼
3056 | 尽
3057 | 尾
3058 | 尿
3059 | 局
3060 | 层
3061 | 居
3062 | 届
3063 | 屋
3064 | 展
3065 | 属
3066 | 層
3067 | 屯
3068 | 山
3069 | 岁
3070 | 岔
3071 | 岗
3072 | 岛
3073 | 岡
3074 | 岩
3075 | 岳
3076 | 岸
3077 | 峡
3078 | 峭
3079 | 峰
3080 | 島
3081 | 崎
3082 | 崖
3083 | 嵩
3084 | 嶺
3085 | 嶽
3086 | 川
3087 | 州
3088 | 巡
3089 | 巢
3090 | 巣
3091 | 工
3092 | 左
3093 | 巧
3094 | 巨
3095 | 差
3096 | 己
3097 | 已
3098 | 巴
3099 | 巷
3100 | 巻
3101 | 巾
3102 | 市
3103 | 布
3104 | 帆
3105 | 师
3106 | 希
3107 | 帘
3108 | 帝
3109 | 带
3110 | 師
3111 | 席
3112 | 帮
3113 | 帯
3114 | 帰
3115 | 常
3116 | 帽
3117 | 幅
3118 | 幌
3119 | 幔
3120 | 幕
3121 | 幡
3122 | 干
3123 | 平
3124 | 年
3125 | 幸
3126 | 幹
3127 | 幻
3128 | 幼
3129 | 幾
3130 | 广
3131 | 庁
3132 | 広
3133 | 庄
3134 | 庆
3135 | 床
3136 | 库
3137 | 应
3138 | 底
3139 | 店
3140 | 庙
3141 | 府
3142 | 度
3143 | 座
3144 | 庫
3145 | 庭
3146 | 庵
3147 | 康
3148 | 廃
3149 | 廉
3150 | 廊
3151 | 廖
3152 | 延
3153 | 建
3154 | 开
3155 | 弁
3156 | 式
3157 | 引
3158 | 弘
3159 | 弟
3160 | 张
3161 | 弥
3162 | 弧
3163 | 弯
3164 | 張
3165 | 強
3166 | 弹
3167 | 强
3168 | 弾
3169 | 当
3170 | 录
3171 | 形
3172 | 彩
3173 | 影
3174 | 役
3175 | 往
3176 | 征
3177 | 径
3178 | 待
3179 | 很
3180 | 後
3181 | 徐
3182 | 得
3183 | 御
3184 | 復
3185 | 循
3186 | 徳
3187 | 徴
3188 | 德
3189 | 心
3190 | 必
3191 | 志
3192 | 忘
3193 | 応
3194 | 忠
3195 | 忧
3196 | 快
3197 | 念
3198 | 怀
3199 | 态
3200 | 思
3201 | 急
3202 | 性
3203 | 总
3204 | 恒
3205 | 恭
3206 | 息
3207 | 恵
3208 | 恶
3209 | 患
3210 | 悦
3211 | 您
3212 | 情
3213 | 惗
3214 | 惠
3215 | 想
3216 | 意
3217 | 愛
3218 | 感
3219 | 愿
3220 | 慈
3221 | 態
3222 | 慎
3223 | 慢
3224 | 慧
3225 | 慮
3226 | 慶
3227 | 戏
3228 | 成
3229 | 我
3230 | 戒
3231 | 或
3232 | 战
3233 | 戦
3234 | 戴
3235 | 戶
3236 | 户
3237 | 戸
3238 | 房
3239 | 所
3240 | 扁
3241 | 手
3242 | 才
3243 | 打
3244 | 払
3245 | 扣
3246 | 执
3247 | 扩
3248 | 扫
3249 | 扬
3250 | 扱
3251 | 扶
3252 | 批
3253 | 找
3254 | 承
3255 | 技
3256 | 抑
3257 | 抓
3258 | 投
3259 | 抗
3260 | 折
3261 | 抜
3262 | 択
3263 | 抢
3264 | 护
3265 | 报
3266 | 抱
3267 | 抵
3268 | 押
3269 | 拉
3270 | 拍
3271 | 拎
3272 | 拔
3273 | 拙
3274 | 招
3275 | 拜
3276 | 拥
3277 | 拳
3278 | 拾
3279 | 拿
3280 | 持
3281 | 挂
3282 | 指
3283 | 按
3284 | 挡
3285 | 挪
3286 | 振
3287 | 挿
3288 | 捐
3289 | 损
3290 | 换
3291 | 捨
3292 | 掌
3293 | 排
3294 | 探
3295 | 接
3296 | 推
3297 | 描
3298 | 提
3299 | 換
3300 | 握
3301 | 揮
3302 | 援
3303 | 損
3304 | 搬
3305 | 搭
3306 | 携
3307 | 摄
3308 | 摆
3309 | 摘
3310 | 摩
3311 | 播
3312 | 操
3313 | 擦
3314 | 攀
3315 | 支
3316 | 收
3317 | 改
3318 | 放
3319 | 政
3320 | 故
3321 | 效
3322 | 教
3323 | 散
3324 | 敬
3325 | 数
3326 | 整
3327 | 數
3328 | 文
3329 | 斋
3330 | 斗
3331 | 料
3332 | 斜
3333 | 斧
3334 | 断
3335 | 斯
3336 | 新
3337 | 方
3338 | 施
3339 | 旅
3340 | 族
3341 | 旗
3342 | 无
3343 | 日
3344 | 旦
3345 | 旧
3346 | 旨
3347 | 早
3348 | 旬
3349 | 旭
3350 | 时
3351 | 旺
3352 | 昆
3353 | 昇
3354 | 昊
3355 | 昌
3356 | 明
3357 | 易
3358 | 星
3359 | 映
3360 | 春
3361 | 昭
3362 | 是
3363 | 昼
3364 | 显
3365 | 時
3366 | 晟
3367 | 晩
3368 | 普
3369 | 景
3370 | 晴
3371 | 晶
3372 | 智
3373 | 暖
3374 | 暗
3375 | 暨
3376 | 曙
3377 | 曜
3378 | 曝
3379 | 曲
3380 | 更
3381 | 書
3382 | 曹
3383 | 替
3384 | 最
3385 | 會
3386 | 月
3387 | 有
3388 | 朋
3389 | 服
3390 | 朔
3391 | 朗
3392 | 望
3393 | 朝
3394 | 期
3395 | 朦
3396 | 木
3397 | 未
3398 | 末
3399 | 本
3400 | 札
3401 | 术
3402 | 机
3403 | 杀
3404 | 权
3405 | 杉
3406 | 李
3407 | 材
3408 | 村
3409 | 杜
3410 | 条
3411 | 来
3412 | 杨
3413 | 杭
3414 | 杰
3415 | 東
3416 | 松
3417 | 板
3418 | 极
3419 | 构
3420 | 枉
3421 | 析
3422 | 林
3423 | 枚
3424 | 果
3425 | 枝
3426 | 枫
3427 | 架
3428 | 柄
3429 | 柏
3430 | 染
3431 | 柔
3432 | 柜
3433 | 查
3434 | 柳
3435 | 査
3436 | 栄
3437 | 标
3438 | 栈
3439 | 栏
3440 | 树
3441 | 栓
3442 | 栗
3443 | 校
3444 | 株
3445 | 样
3446 | 核
3447 | 根
3448 | 格
3449 | 栽
3450 | 桂
3451 | 桃
3452 | 案
3453 | 桜
3454 | 档
3455 | 桥
3456 | 桶
3457 | 桷
3458 | 梅
3459 | 梓
3460 | 梦
3461 | 梯
3462 | 械
3463 | 检
3464 | 棄
3465 | 棋
3466 | 棍
3467 | 棚
3468 | 森
3469 | 棱
3470 | 棵
3471 | 椅
3472 | 植
3473 | 椎
3474 | 椒
3475 | 検
3476 | 楊
3477 | 楕
3478 | 楚
3479 | 業
3480 | 極
3481 | 楼
3482 | 楽
3483 | 概
3484 | 榆
3485 | 榛
3486 | 榜
3487 | 榧
3488 | 榭
3489 | 構
3490 | 槍
3491 | 様
3492 | 槟
3493 | 槳
3494 | 槽
3495 | 樂
3496 | 樓
3497 | 標
3498 | 樟
3499 | 模
3500 | 樣
3501 | 横
3502 | 樫
3503 | 樹
3504 | 樽
3505 | 橋
3506 | 橙
3507 | 機
3508 | 檔
3509 | 檬
3510 | 次
3511 | 欢
3512 | 欣
3513 | 欧
3514 | 歌
3515 | 歡
3516 | 止
3517 | 正
3518 | 此
3519 | 步
3520 | 武
3521 | 歩
3522 | 歯
3523 | 歳
3524 | 死
3525 | 残
3526 | 段
3527 | 殿
3528 | 毎
3529 | 每
3530 | 比
3531 | 毕
3532 | 毙
3533 | 毛
3534 | 毯
3535 | 民
3536 | 气
3537 | 気
3538 | 氟
3539 | 氣
3540 | 氩
3541 | 水
3542 | 氷
3543 | 永
3544 | 汀
3545 | 汁
3546 | 求
3547 | 汇
3548 | 汉
3549 | 汕
3550 | 汚
3551 | 江
3552 | 池
3553 | 污
3554 | 汤
3555 | 決
3556 | 汽
3557 | 沃
3558 | 沖
3559 | 沙
3560 | 沟
3561 | 没
3562 | 沢
3563 | 沪
3564 | 河
3565 | 油
3566 | 治
3567 | 沼
3568 | 沿
3569 | 泉
3570 | 泊
3571 | 法
3572 | 泛
3573 | 波
3574 | 注
3575 | 泰
3576 | 泳
3577 | 泵
3578 | 泽
3579 | 泾
3580 | 洁
3581 | 洋
3582 | 洗
3583 | 洛
3584 | 洞
3585 | 津
3586 | 洪
3587 | 洱
3588 | 洲
3589 | 活
3590 | 洼
3591 | 派
3592 | 流
3593 | 测
3594 | 济
3595 | 浓
3596 | 浙
3597 | 浜
3598 | 浦
3599 | 浪
3600 | 浮
3601 | 海
3602 | 涂
3603 | 消
3604 | 涧
3605 | 涪
3606 | 液
3607 | 涵
3608 | 淀
3609 | 淇
3610 | 淋
3611 | 淞
3612 | 深
3613 | 添
3614 | 清
3615 | 済
3616 | 渋
3617 | 渔
3618 | 減
3619 | 渡
3620 | 温
3621 | 測
3622 | 港
3623 | 游
3624 | 湖
3625 | 湘
3626 | 湾
3627 | 満
3628 | 溉
3629 | 源
3630 | 準
3631 | 溝
3632 | 溢
3633 | 溪
3634 | 滅
3635 | 滋
3636 | 滑
3637 | 滕
3638 | 滞
3639 | 满
3640 | 滦
3641 | 滨
3642 | 滴
3643 | 漆
3644 | 演
3645 | 漢
3646 | 漫
3647 | 潔
3648 | 潤
3649 | 潮
3650 | 澜
3651 | 澡
3652 | 激
3653 | 濟
3654 | 濡
3655 | 瀑
3656 | 瀬
3657 | 灌
3658 | 灏
3659 | 灣
3660 | 火
3661 | 灭
3662 | 灯
3663 | 灰
3664 | 灾
3665 | 炒
3666 | 炬
3667 | 炭
3668 | 炮
3669 | 炸
3670 | 点
3671 | 為
3672 | 烈
3673 | 烏
3674 | 烘
3675 | 烟
3676 | 烤
3677 | 烧
3678 | 烫
3679 | 热
3680 | 焊
3681 | 焖
3682 | 焙
3683 | 焚
3684 | 無
3685 | 然
3686 | 焼
3687 | 煌
3688 | 煎
3689 | 煙
3690 | 煤
3691 | 照
3692 | 煮
3693 | 熟
3694 | 熱
3695 | 燃
3696 | 營
3697 | 燥
3698 | 爆
3699 | 爨
3700 | 爬
3701 | 爱
3702 | 爷
3703 | 片
3704 | 版
3705 | 牌
3706 | 牙
3707 | 牛
3708 | 牢
3709 | 牧
3710 | 物
3711 | 特
3712 | 犬
3713 | 犯
3714 | 状
3715 | 独
3716 | 狮
3717 | 猫
3718 | 献
3719 | 獣
3720 | 玄
3721 | 率
3722 | 玉
3723 | 王
3724 | 玛
3725 | 玩
3726 | 环
3727 | 现
3728 | 玻
3729 | 珈
3730 | 珍
3731 | 珠
3732 | 班
3733 | 現
3734 | 球
3735 | 理
3736 | 琏
3737 | 琲
3738 | 琴
3739 | 瑞
3740 | 璃
3741 | 環
3742 | 瓜
3743 | 瓶
3744 | 甘
3745 | 生
3746 | 產
3747 | 産
3748 | 用
3749 | 甩
3750 | 甫
3751 | 田
3752 | 由
3753 | 甲
3754 | 申
3755 | 电
3756 | 男
3757 | 町
3758 | 画
3759 | 畅
3760 | 界
3761 | 畑
3762 | 留
3763 | 略
3764 | 番
3765 | 異
3766 | 疆
3767 | 疋
3768 | 疏
3769 | 疗
3770 | 疫
3771 | 疲
3772 | 疾
3773 | 病
3774 | 痛
3775 | 瘫
3776 | 療
3777 | 発
3778 | 登
3779 | 白
3780 | 百
3781 | 的
3782 | 皆
3783 | 皋
3784 | 皙
3785 | 皮
3786 | 皿
3787 | 盂
3788 | 益
3789 | 监
3790 | 盒
3791 | 盖
3792 | 盗
3793 | 盘
3794 | 盛
3795 | 盟
3796 | 盤
3797 | 目
3798 | 直
3799 | 相
3800 | 省
3801 | 眉
3802 | 看
3803 | 県
3804 | 真
3805 | 眠
3806 | 眺
3807 | 眼
3808 | 眾
3809 | 着
3810 | 睡
3811 | 督
3812 | 睿
3813 | 瞬
3814 | 矢
3815 | 知
3816 | 短
3817 | 矯
3818 | 石
3819 | 矿
3820 | 码
3821 | 砂
3822 | 研
3823 | 破
3824 | 硝
3825 | 硬
3826 | 碑
3827 | 碗
3828 | 碧
3829 | 碳
3830 | 確
3831 | 磁
3832 | 磨
3833 | 礎
3834 | 示
3835 | 礼
3836 | 社
3837 | 祉
3838 | 祖
3839 | 祜
3840 | 祝
3841 | 神
3842 | 祠
3843 | 祥
3844 | 票
3845 | 祭
3846 | 祸
3847 | 禁
3848 | 禄
3849 | 禅
3850 | 福
3851 | 禾
3852 | 秀
3853 | 私
3854 | 秋
3855 | 种
3856 | 科
3857 | 秘
3858 | 租
3859 | 移
3860 | 稅
3861 | 程
3862 | 税
3863 | 稚
3864 | 種
3865 | 稳
3866 | 稻
3867 | 稿
3868 | 穂
3869 | 穴
3870 | 究
3871 | 空
3872 | 突
3873 | 窃
3874 | 窄
3875 | 窓
3876 | 窗
3877 | 窝
3878 | 窺
3879 | 立
3880 | 站
3881 | 竞
3882 | 童
3883 | 端
3884 | 競
3885 | 竹
3886 | 笑
3887 | 笔
3888 | 符
3889 | 第
3890 | 笼
3891 | 等
3892 | 筋
3893 | 筑
3894 | 筒
3895 | 策
3896 | 筹
3897 | 签
3898 | 简
3899 | 算
3900 | 管
3901 | 箫
3902 | 箱
3903 | 節
3904 | 築
3905 | 簡
3906 | 籍
3907 | 米
3908 | 类
3909 | 粉
3910 | 粒
3911 | 粗
3912 | 粥
3913 | 粧
3914 | 粮
3915 | 粱
3916 | 精
3917 | 糀
3918 | 糖
3919 | 糯
3920 | 系
3921 | 約
3922 | 紅
3923 | 納
3924 | 級
3925 | 紛
3926 | 素
3927 | 紡
3928 | 索
3929 | 紧
3930 | 紫
3931 | 細
3932 | 紳
3933 | 紺
3934 | 終
3935 | 経
3936 | 結
3937 | 絡
3938 | 給
3939 | 絨
3940 | 統
3941 | 絵
3942 | 經
3943 | 続
3944 | 綦
3945 | 維
3946 | 綱
3947 | 緊
3948 | 総
3949 | 緑
3950 | 線
3951 | 締
3952 | 練
3953 | 縄
3954 | 總
3955 | 繁
3956 | 織
3957 | 繰
3958 | 红
3959 | 纤
3960 | 约
3961 | 级
3962 | 纪
3963 | 纯
3964 | 纱
3965 | 纳
3966 | 纹
3967 | 纽
3968 | 线
3969 | 绀
3970 | 组
3971 | 织
3972 | 终
3973 | 绍
3974 | 经
3975 | 绒
3976 | 结
3977 | 绕
3978 | 绘
3979 | 给
3980 | 络
3981 | 绝
3982 | 统
3983 | 继
3984 | 续
3985 | 维
3986 | 综
3987 | 绿
3988 | 缆
3989 | 编
3990 | 缘
3991 | 缩
3992 | 缶
3993 | 网
3994 | 罗
3995 | 罚
3996 | 罪
3997 | 置
3998 | 署
3999 | 美
4000 | 群
4001 | 義
4002 | 羽
4003 | 習
4004 | 翔
4005 | 翠
4006 | 翻
4007 | 耀
4008 | 老
4009 | 考
4010 | 者
4011 | 而
4012 | 耳
4013 | 耻
4014 | 职
4015 | 联
4016 | 聘
4017 | 聚
4018 | 聞
4019 | 聴
4020 | 肉
4021 | 肖
4022 | 股
4023 | 肥
4024 | 肩
4025 | 肪
4026 | 肯
4027 | 育
4028 | 肴
4029 | 背
4030 | 胜
4031 | 胡
4032 | 胧
4033 | 胶
4034 | 能
4035 | 脂
4036 | 脚
4037 | 脸
4038 | 腐
4039 | 腰
4040 | 腹
4041 | 腾
4042 | 膚
4043 | 膜
4044 | 臣
4045 | 臨
4046 | 自
4047 | 臭
4048 | 至
4049 | 致
4050 | 臻
4051 | 臼
4052 | 興
4053 | 舍
4054 | 舎
4055 | 舒
4056 | 舗
4057 | 舘
4058 | 舞
4059 | 舟
4060 | 航
4061 | 般
4062 | 舰
4063 | 舱
4064 | 船
4065 | 良
4066 | 色
4067 | 艺
4068 | 节
4069 | 芋
4070 | 芝
4071 | 芥
4072 | 芦
4073 | 花
4074 | 芳
4075 | 芸
4076 | 苍
4077 | 苏
4078 | 苑
4079 | 苗
4080 | 若
4081 | 英
4082 | 苹
4083 | 茂
4084 | 范
4085 | 茄
4086 | 茅
4087 | 茶
4088 | 茸
4089 | 荆
4090 | 草
4091 | 荔
4092 | 荣
4093 | 荥
4094 | 药
4095 | 荷
4096 | 荻
4097 | 莊
4098 | 莎
4099 | 莫
4100 | 莲
4101 | 获
4102 | 菀
4103 | 菇
4104 | 菌
4105 | 菓
4106 | 菜
4107 | 華
4108 | 菱
4109 | 菲
4110 | 营
4111 | 萬
4112 | 落
4113 | 葉
4114 | 葛
4115 | 董
4116 | 葵
4117 | 蒋
4118 | 蒲
4119 | 蒸
4120 | 蓄
4121 | 蓬
4122 | 蔦
4123 | 蔬
4124 | 蔵
4125 | 蔷
4126 | 蕨
4127 | 薄
4128 | 薇
4129 | 薙
4130 | 薬
4131 | 藍
4132 | 藏
4133 | 藤
4134 | 蘑
4135 | 蘭
4136 | 虎
4137 | 處
4138 | 虫
4139 | 虹
4140 | 蛟
4141 | 融
4142 | 螢
4143 | 螺
4144 | 血
4145 | 行
4146 | 術
4147 | 街
4148 | 衛
4149 | 衡
4150 | 衣
4151 | 表
4152 | 袋
4153 | 袍
4154 | 被
4155 | 裁
4156 | 装
4157 | 裕
4158 | 補
4159 | 裹
4160 | 製
4161 | 襄
4162 | 西
4163 | 要
4164 | 見
4165 | 規
4166 | 視
4167 | 親
4168 | 観
4169 | 觀
4170 | 见
4171 | 观
4172 | 规
4173 | 视
4174 | 览
4175 | 角
4176 | 解
4177 | 触
4178 | 計
4179 | 託
4180 | 記
4181 | 訪
4182 | 設
4183 | 許
4184 | 診
4185 | 証
4186 | 試
4187 | 詰
4188 | 話
4189 | 詳
4190 | 詹
4191 | 誉
4192 | 誌
4193 | 認
4194 | 誓
4195 | 誠
4196 | 説
4197 | 読
4198 | 課
4199 | 調
4200 | 談
4201 | 請
4202 | 警
4203 | 議
4204 | 计
4205 | 订
4206 | 认
4207 | 让
4208 | 训
4209 | 议
4210 | 讯
4211 | 记
4212 | 讲
4213 | 许
4214 | 论
4215 | 设
4216 | 访
4217 | 证
4218 | 评
4219 | 识
4220 | 试
4221 | 诗
4222 | 诚
4223 | 话
4224 | 诞
4225 | 询
4226 | 语
4227 | 误
4228 | 说
4229 | 请
4230 | 谁
4231 | 调
4232 | 谈
4233 | 谐
4234 | 谢
4235 | 谨
4236 | 谷
4237 | 豆
4238 | 豊
4239 | 象
4240 | 貝
4241 | 負
4242 | 貨
4243 | 販
4244 | 責
4245 | 貴
4246 | 買
4247 | 貸
4248 | 費
4249 | 賀
4250 | 賃
4251 | 資
4252 | 賞
4253 | 賢
4254 | 質
4255 | 贈
4256 | 财
4257 | 责
4258 | 货
4259 | 质
4260 | 购
4261 | 贵
4262 | 费
4263 | 贼
4264 | 赁
4265 | 资
4266 | 赛
4267 | 赤
4268 | 走
4269 | 赵
4270 | 起
4271 | 超
4272 | 越
4273 | 足
4274 | 跆
4275 | 跑
4276 | 跟
4277 | 跡
4278 | 跨
4279 | 路
4280 | 践
4281 | 踏
4282 | 踩
4283 | 踪
4284 | 身
4285 | 車
4286 | 転
4287 | 軽
4288 | 載
4289 | 輔
4290 | 輝
4291 | 輪
4292 | 輸
4293 | 车
4294 | 轩
4295 | 转
4296 | 轻
4297 | 辆
4298 | 辉
4299 | 输
4300 | 辛
4301 | 辣
4302 | 辰
4303 | 边
4304 | 辺
4305 | 込
4306 | 辽
4307 | 达
4308 | 迁
4309 | 过
4310 | 迈
4311 | 迎
4312 | 运
4313 | 近
4314 | 返
4315 | 还
4316 | 这
4317 | 进
4318 | 远
4319 | 违
4320 | 连
4321 | 迪
4322 | 迫
4323 | 迷
4324 | 迹
4325 | 追
4326 | 送
4327 | 适
4328 | 选
4329 | 透
4330 | 递
4331 | 途
4332 | 逗
4333 | 通
4334 | 速
4335 | 造
4336 | 連
4337 | 週
4338 | 進
4339 | 逸
4340 | 運
4341 | 過
4342 | 道
4343 | 達
4344 | 遗
4345 | 遠
4346 | 適
4347 | 遵
4348 | 選
4349 | 遺
4350 | 避
4351 | 邊
4352 | 那
4353 | 邦
4354 | 邮
4355 | 郁
4356 | 郊
4357 | 郎
4358 | 郑
4359 | 郝
4360 | 郡
4361 | 部
4362 | 郭
4363 | 郵
4364 | 郷
4365 | 都
4366 | 鄂
4367 | 酉
4368 | 酌
4369 | 配
4370 | 酒
4371 | 酵
4372 | 酷
4373 | 酸
4374 | 酿
4375 | 醉
4376 | 醒
4377 | 采
4378 | 里
4379 | 重
4380 | 野
4381 | 量
4382 | 金
4383 | 釜
4384 | 釣
4385 | 鉄
4386 | 銀
4387 | 銘
4388 | 銭
4389 | 鋪
4390 | 録
4391 | 鍋
4392 | 鎌
4393 | 鏡
4394 | 鑑
4395 | 鑫
4396 | 钓
4397 | 钛
4398 | 钟
4399 | 钢
4400 | 钮
4401 | 钱
4402 | 钻
4403 | 铁
4404 | 铝
4405 | 银
4406 | 铺
4407 | 链
4408 | 销
4409 | 锁
4410 | 锅
4411 | 锈
4412 | 锋
4413 | 锐
4414 | 锡
4415 | 锦
4416 | 镁
4417 | 镇
4418 | 镜
4419 | 長
4420 | 长
4421 | 門
4422 | 閉
4423 | 開
4424 | 間
4425 | 関
4426 | 閣
4427 | 閩
4428 | 门
4429 | 闭
4430 | 问
4431 | 闲
4432 | 间
4433 | 闸
4434 | 闻
4435 | 队
4436 | 防
4437 | 阳
4438 | 阻
4439 | 阿
4440 | 陀
4441 | 附
4442 | 际
4443 | 陈
4444 | 降
4445 | 限
4446 | 陕
4447 | 陡
4448 | 院
4449 | 除
4450 | 险
4451 | 陪
4452 | 陳
4453 | 陵
4454 | 陶
4455 | 陸
4456 | 険
4457 | 陽
4458 | 隅
4459 | 隆
4460 | 隊
4461 | 階
4462 | 随
4463 | 際
4464 | 障
4465 | 隣
4466 | 隧
4467 | 隨
4468 | 險
4469 | 难
4470 | 雄
4471 | 雅
4472 | 集
4473 | 雉
4474 | 雜
4475 | 離
4476 | 難
4477 | 雨
4478 | 雪
4479 | 雲
4480 | 零
4481 | 雷
4482 | 電
4483 | 需
4484 | 震
4485 | 霉
4486 | 霧
4487 | 露
4488 | 青
4489 | 靖
4490 | 静
4491 | 靜
4492 | 非
4493 | 靠
4494 | 靡
4495 | 面
4496 | 革
4497 | 鞋
4498 | 韩
4499 | 音
4500 | 韵
4501 | 頃
4502 | 頑
4503 | 領
4504 | 頭
4505 | 題
4506 | 額
4507 | 顔
4508 | 願
4509 | 類
4510 | 项
4511 | 顺
4512 | 须
4513 | 颁
4514 | 预
4515 | 领
4516 | 风
4517 | 飛
4518 | 飞
4519 | 食
4520 | 飬
4521 | 飯
4522 | 飲
4523 | 飴
4524 | 養
4525 | 餐
4526 | 館
4527 | 饭
4528 | 饮
4529 | 饰
4530 | 饺
4531 | 饼
4532 | 馆
4533 | 首
4534 | 香
4535 | 馨
4536 | 馬
4537 | 駅
4538 | 駐
4539 | 駿
4540 | 験
4541 | 马
4542 | 驰
4543 | 驶
4544 | 验
4545 | 骏
4546 | 骨
4547 | 高
4548 | 髙
4549 | 鬼
4550 | 魏
4551 | 魔
4552 | 魚
4553 | 鮨
4554 | 鮮
4555 | 鱼
4556 | 鲁
4557 | 鲍
4558 | 鳥
4559 | 鶏
4560 | 鷹
4561 | 鸟
4562 | 鸡
4563 | 鸭
4564 | 鸿
4565 | 鹏
4566 | 鹤
4567 | 鹿
4568 | 麒
4569 | 麓
4570 | 麗
4571 | 麟
4572 | 麦
4573 | 麺
4574 | 麻
4575 | 黄
4576 | 黒
4577 | 鼎
4578 | 鼻
4579 | 齐
4580 | 齿
4581 | 龋
4582 | 龍
4583 | 龙
4584 | 龜
4585 | ﭘ
4586 | ﭬ
4587 | ﭭ
4588 | ﯕ
4589 | ﷲ
4590 | ﺄ
4591 | ﺆ
4592 | ﺈ
4593 | ﺊ
4594 | ﺋ
4595 | ﺌ
4596 | ﺎ
4597 | ﺐ
4598 | ﺑ
4599 | ﺒ
4600 | ﺔ
4601 | ﺖ
4602 | ﺗ
4603 | ﺘ
4604 | ﺚ
4605 | ﺛ
4606 | ﺜ
4607 | ﺞ
4608 | ﺟ
4609 | ﺠ
4610 | ﺢ
4611 | ﺣ
4612 | ﺤ
4613 | ﺦ
4614 | ﺧ
4615 | ﺨ
4616 | ﺪ
4617 | ﺬ
4618 | ﺮ
4619 | ﺰ
4620 | ﺲ
4621 | ﺳ
4622 | ﺴ
4623 | ﺶ
4624 | ﺷ
4625 | ﺸ
4626 | ﺺ
4627 | ﺻ
4628 | ﺼ
4629 | ﺾ
4630 | ﺿ
4631 | ﻀ
4632 | ﻂ
4633 | ﻃ
4634 | ﻄ
4635 | ﻆ
4636 | ﻇ
4637 | ﻈ
4638 | ﻊ
4639 | ﻋ
4640 | ﻌ
4641 | ﻎ
4642 | ﻏ
4643 | ﻐ
4644 | ﻒ
4645 | ﻓ
4646 | ﻔ
4647 | ﻖ
4648 | ﻗ
4649 | ﻘ
4650 | ﻚ
4651 | ﻛ
4652 | ﻜ
4653 | ﻞ
4654 | ﻟ
4655 | ﻠ
4656 | ﻢ
4657 | ﻣ
4658 | ﻤ
4659 | ﻦ
4660 | ﻧ
4661 | ﻨ
4662 | ﻪ
4663 | ﻫ
4664 | ﻬ
4665 | ﻮ
4666 | ﻰ
4667 | ﻲ
4668 | ﻳ
4669 | ﻴ
4670 | ﻵ
4671 | ﻷ
4672 | ﻸ
4673 | ﻹ
4674 | ﻺ
4675 | ﻻ
4676 | ﻼ
4677 |
--------------------------------------------------------------------------------