├── LICENSE
├── NamesList.md
├── README.md
├── README.md.d
├── biangbiang.png
├── list-fonts-scale.png
├── list-fonts.png
└── screenshot.png
├── regression.sh
└── ugrep
/LICENSE:
--------------------------------------------------------------------------------
1 | GNU GENERAL PUBLIC LICENSE
2 | Version 3, 29 June 2007
3 |
4 | Copyright (C) 2007 Free Software Foundation, Inc.
5 | Everyone is permitted to copy and distribute verbatim copies
6 | of this license document, but changing it is not allowed.
7 |
8 | Preamble
9 |
10 | The GNU General Public License is a free, copyleft license for
11 | software and other kinds of works.
12 |
13 | The licenses for most software and other practical works are designed
14 | to take away your freedom to share and change the works. By contrast,
15 | the GNU General Public License is intended to guarantee your freedom to
16 | share and change all versions of a program--to make sure it remains free
17 | software for all its users. We, the Free Software Foundation, use the
18 | GNU General Public License for most of our software; it applies also to
19 | any other work released this way by its authors. You can apply it to
20 | your programs, too.
21 |
22 | When we speak of free software, we are referring to freedom, not
23 | price. Our General Public Licenses are designed to make sure that you
24 | have the freedom to distribute copies of free software (and charge for
25 | them if you wish), that you receive source code or can get it if you
26 | want it, that you can change the software or use pieces of it in new
27 | free programs, and that you know you can do these things.
28 |
29 | To protect your rights, we need to prevent others from denying you
30 | these rights or asking you to surrender the rights. Therefore, you have
31 | certain responsibilities if you distribute copies of the software, or if
32 | you modify it: responsibilities to respect the freedom of others.
33 |
34 | For example, if you distribute copies of such a program, whether
35 | gratis or for a fee, you must pass on to the recipients the same
36 | freedoms that you received. You must make sure that they, too, receive
37 | or can get the source code. And you must show them these terms so they
38 | know their rights.
39 |
40 | Developers that use the GNU GPL protect your rights with two steps:
41 | (1) assert copyright on the software, and (2) offer you this License
42 | giving you legal permission to copy, distribute and/or modify it.
43 |
44 | For the developers' and authors' protection, the GPL clearly explains
45 | that there is no warranty for this free software. For both users' and
46 | authors' sake, the GPL requires that modified versions be marked as
47 | changed, so that their problems will not be attributed erroneously to
48 | authors of previous versions.
49 |
50 | Some devices are designed to deny users access to install or run
51 | modified versions of the software inside them, although the manufacturer
52 | can do so. This is fundamentally incompatible with the aim of
53 | protecting users' freedom to change the software. The systematic
54 | pattern of such abuse occurs in the area of products for individuals to
55 | use, which is precisely where it is most unacceptable. Therefore, we
56 | have designed this version of the GPL to prohibit the practice for those
57 | products. If such problems arise substantially in other domains, we
58 | stand ready to extend this provision to those domains in future versions
59 | of the GPL, as needed to protect the freedom of users.
60 |
61 | Finally, every program is threatened constantly by software patents.
62 | States should not allow patents to restrict development and use of
63 | software on general-purpose computers, but in those that do, we wish to
64 | avoid the special danger that patents applied to a free program could
65 | make it effectively proprietary. To prevent this, the GPL assures that
66 | patents cannot be used to render the program non-free.
67 |
68 | The precise terms and conditions for copying, distribution and
69 | modification follow.
70 |
71 | TERMS AND CONDITIONS
72 |
73 | 0. Definitions.
74 |
75 | "This License" refers to version 3 of the GNU General Public License.
76 |
77 | "Copyright" also means copyright-like laws that apply to other kinds of
78 | works, such as semiconductor masks.
79 |
80 | "The Program" refers to any copyrightable work licensed under this
81 | License. Each licensee is addressed as "you". "Licensees" and
82 | "recipients" may be individuals or organizations.
83 |
84 | To "modify" a work means to copy from or adapt all or part of the work
85 | in a fashion requiring copyright permission, other than the making of an
86 | exact copy. The resulting work is called a "modified version" of the
87 | earlier work or a work "based on" the earlier work.
88 |
89 | A "covered work" means either the unmodified Program or a work based
90 | on the Program.
91 |
92 | To "propagate" a work means to do anything with it that, without
93 | permission, would make you directly or secondarily liable for
94 | infringement under applicable copyright law, except executing it on a
95 | computer or modifying a private copy. Propagation includes copying,
96 | distribution (with or without modification), making available to the
97 | public, and in some countries other activities as well.
98 |
99 | To "convey" a work means any kind of propagation that enables other
100 | parties to make or receive copies. Mere interaction with a user through
101 | a computer network, with no transfer of a copy, is not conveying.
102 |
103 | An interactive user interface displays "Appropriate Legal Notices"
104 | to the extent that it includes a convenient and prominently visible
105 | feature that (1) displays an appropriate copyright notice, and (2)
106 | tells the user that there is no warranty for the work (except to the
107 | extent that warranties are provided), that licensees may convey the
108 | work under this License, and how to view a copy of this License. If
109 | the interface presents a list of user commands or options, such as a
110 | menu, a prominent item in the list meets this criterion.
111 |
112 | 1. Source Code.
113 |
114 | The "source code" for a work means the preferred form of the work
115 | for making modifications to it. "Object code" means any non-source
116 | form of a work.
117 |
118 | A "Standard Interface" means an interface that either is an official
119 | standard defined by a recognized standards body, or, in the case of
120 | interfaces specified for a particular programming language, one that
121 | is widely used among developers working in that language.
122 |
123 | The "System Libraries" of an executable work include anything, other
124 | than the work as a whole, that (a) is included in the normal form of
125 | packaging a Major Component, but which is not part of that Major
126 | Component, and (b) serves only to enable use of the work with that
127 | Major Component, or to implement a Standard Interface for which an
128 | implementation is available to the public in source code form. A
129 | "Major Component", in this context, means a major essential component
130 | (kernel, window system, and so on) of the specific operating system
131 | (if any) on which the executable work runs, or a compiler used to
132 | produce the work, or an object code interpreter used to run it.
133 |
134 | The "Corresponding Source" for a work in object code form means all
135 | the source code needed to generate, install, and (for an executable
136 | work) run the object code and to modify the work, including scripts to
137 | control those activities. However, it does not include the work's
138 | System Libraries, or general-purpose tools or generally available free
139 | programs which are used unmodified in performing those activities but
140 | which are not part of the work. For example, Corresponding Source
141 | includes interface definition files associated with source files for
142 | the work, and the source code for shared libraries and dynamically
143 | linked subprograms that the work is specifically designed to require,
144 | such as by intimate data communication or control flow between those
145 | subprograms and other parts of the work.
146 |
147 | The Corresponding Source need not include anything that users
148 | can regenerate automatically from other parts of the Corresponding
149 | Source.
150 |
151 | The Corresponding Source for a work in source code form is that
152 | same work.
153 |
154 | 2. Basic Permissions.
155 |
156 | All rights granted under this License are granted for the term of
157 | copyright on the Program, and are irrevocable provided the stated
158 | conditions are met. This License explicitly affirms your unlimited
159 | permission to run the unmodified Program. The output from running a
160 | covered work is covered by this License only if the output, given its
161 | content, constitutes a covered work. This License acknowledges your
162 | rights of fair use or other equivalent, as provided by copyright law.
163 |
164 | You may make, run and propagate covered works that you do not
165 | convey, without conditions so long as your license otherwise remains
166 | in force. You may convey covered works to others for the sole purpose
167 | of having them make modifications exclusively for you, or provide you
168 | with facilities for running those works, provided that you comply with
169 | the terms of this License in conveying all material for which you do
170 | not control copyright. Those thus making or running the covered works
171 | for you must do so exclusively on your behalf, under your direction
172 | and control, on terms that prohibit them from making any copies of
173 | your copyrighted material outside their relationship with you.
174 |
175 | Conveying under any other circumstances is permitted solely under
176 | the conditions stated below. Sublicensing is not allowed; section 10
177 | makes it unnecessary.
178 |
179 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
180 |
181 | No covered work shall be deemed part of an effective technological
182 | measure under any applicable law fulfilling obligations under article
183 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or
184 | similar laws prohibiting or restricting circumvention of such
185 | measures.
186 |
187 | When you convey a covered work, you waive any legal power to forbid
188 | circumvention of technological measures to the extent such circumvention
189 | is effected by exercising rights under this License with respect to
190 | the covered work, and you disclaim any intention to limit operation or
191 | modification of the work as a means of enforcing, against the work's
192 | users, your or third parties' legal rights to forbid circumvention of
193 | technological measures.
194 |
195 | 4. Conveying Verbatim Copies.
196 |
197 | You may convey verbatim copies of the Program's source code as you
198 | receive it, in any medium, provided that you conspicuously and
199 | appropriately publish on each copy an appropriate copyright notice;
200 | keep intact all notices stating that this License and any
201 | non-permissive terms added in accord with section 7 apply to the code;
202 | keep intact all notices of the absence of any warranty; and give all
203 | recipients a copy of this License along with the Program.
204 |
205 | You may charge any price or no price for each copy that you convey,
206 | and you may offer support or warranty protection for a fee.
207 |
208 | 5. Conveying Modified Source Versions.
209 |
210 | You may convey a work based on the Program, or the modifications to
211 | produce it from the Program, in the form of source code under the
212 | terms of section 4, provided that you also meet all of these conditions:
213 |
214 | a) The work must carry prominent notices stating that you modified
215 | it, and giving a relevant date.
216 |
217 | b) The work must carry prominent notices stating that it is
218 | released under this License and any conditions added under section
219 | 7. This requirement modifies the requirement in section 4 to
220 | "keep intact all notices".
221 |
222 | c) You must license the entire work, as a whole, under this
223 | License to anyone who comes into possession of a copy. This
224 | License will therefore apply, along with any applicable section 7
225 | additional terms, to the whole of the work, and all its parts,
226 | regardless of how they are packaged. This License gives no
227 | permission to license the work in any other way, but it does not
228 | invalidate such permission if you have separately received it.
229 |
230 | d) If the work has interactive user interfaces, each must display
231 | Appropriate Legal Notices; however, if the Program has interactive
232 | interfaces that do not display Appropriate Legal Notices, your
233 | work need not make them do so.
234 |
235 | A compilation of a covered work with other separate and independent
236 | works, which are not by their nature extensions of the covered work,
237 | and which are not combined with it such as to form a larger program,
238 | in or on a volume of a storage or distribution medium, is called an
239 | "aggregate" if the compilation and its resulting copyright are not
240 | used to limit the access or legal rights of the compilation's users
241 | beyond what the individual works permit. Inclusion of a covered work
242 | in an aggregate does not cause this License to apply to the other
243 | parts of the aggregate.
244 |
245 | 6. Conveying Non-Source Forms.
246 |
247 | You may convey a covered work in object code form under the terms
248 | of sections 4 and 5, provided that you also convey the
249 | machine-readable Corresponding Source under the terms of this License,
250 | in one of these ways:
251 |
252 | a) Convey the object code in, or embodied in, a physical product
253 | (including a physical distribution medium), accompanied by the
254 | Corresponding Source fixed on a durable physical medium
255 | customarily used for software interchange.
256 |
257 | b) Convey the object code in, or embodied in, a physical product
258 | (including a physical distribution medium), accompanied by a
259 | written offer, valid for at least three years and valid for as
260 | long as you offer spare parts or customer support for that product
261 | model, to give anyone who possesses the object code either (1) a
262 | copy of the Corresponding Source for all the software in the
263 | product that is covered by this License, on a durable physical
264 | medium customarily used for software interchange, for a price no
265 | more than your reasonable cost of physically performing this
266 | conveying of source, or (2) access to copy the
267 | Corresponding Source from a network server at no charge.
268 |
269 | c) Convey individual copies of the object code with a copy of the
270 | written offer to provide the Corresponding Source. This
271 | alternative is allowed only occasionally and noncommercially, and
272 | only if you received the object code with such an offer, in accord
273 | with subsection 6b.
274 |
275 | d) Convey the object code by offering access from a designated
276 | place (gratis or for a charge), and offer equivalent access to the
277 | Corresponding Source in the same way through the same place at no
278 | further charge. You need not require recipients to copy the
279 | Corresponding Source along with the object code. If the place to
280 | copy the object code is a network server, the Corresponding Source
281 | may be on a different server (operated by you or a third party)
282 | that supports equivalent copying facilities, provided you maintain
283 | clear directions next to the object code saying where to find the
284 | Corresponding Source. Regardless of what server hosts the
285 | Corresponding Source, you remain obligated to ensure that it is
286 | available for as long as needed to satisfy these requirements.
287 |
288 | e) Convey the object code using peer-to-peer transmission, provided
289 | you inform other peers where the object code and Corresponding
290 | Source of the work are being offered to the general public at no
291 | charge under subsection 6d.
292 |
293 | A separable portion of the object code, whose source code is excluded
294 | from the Corresponding Source as a System Library, need not be
295 | included in conveying the object code work.
296 |
297 | A "User Product" is either (1) a "consumer product", which means any
298 | tangible personal property which is normally used for personal, family,
299 | or household purposes, or (2) anything designed or sold for incorporation
300 | into a dwelling. In determining whether a product is a consumer product,
301 | doubtful cases shall be resolved in favor of coverage. For a particular
302 | product received by a particular user, "normally used" refers to a
303 | typical or common use of that class of product, regardless of the status
304 | of the particular user or of the way in which the particular user
305 | actually uses, or expects or is expected to use, the product. A product
306 | is a consumer product regardless of whether the product has substantial
307 | commercial, industrial or non-consumer uses, unless such uses represent
308 | the only significant mode of use of the product.
309 |
310 | "Installation Information" for a User Product means any methods,
311 | procedures, authorization keys, or other information required to install
312 | and execute modified versions of a covered work in that User Product from
313 | a modified version of its Corresponding Source. The information must
314 | suffice to ensure that the continued functioning of the modified object
315 | code is in no case prevented or interfered with solely because
316 | modification has been made.
317 |
318 | If you convey an object code work under this section in, or with, or
319 | specifically for use in, a User Product, and the conveying occurs as
320 | part of a transaction in which the right of possession and use of the
321 | User Product is transferred to the recipient in perpetuity or for a
322 | fixed term (regardless of how the transaction is characterized), the
323 | Corresponding Source conveyed under this section must be accompanied
324 | by the Installation Information. But this requirement does not apply
325 | if neither you nor any third party retains the ability to install
326 | modified object code on the User Product (for example, the work has
327 | been installed in ROM).
328 |
329 | The requirement to provide Installation Information does not include a
330 | requirement to continue to provide support service, warranty, or updates
331 | for a work that has been modified or installed by the recipient, or for
332 | the User Product in which it has been modified or installed. Access to a
333 | network may be denied when the modification itself materially and
334 | adversely affects the operation of the network or violates the rules and
335 | protocols for communication across the network.
336 |
337 | Corresponding Source conveyed, and Installation Information provided,
338 | in accord with this section must be in a format that is publicly
339 | documented (and with an implementation available to the public in
340 | source code form), and must require no special password or key for
341 | unpacking, reading or copying.
342 |
343 | 7. Additional Terms.
344 |
345 | "Additional permissions" are terms that supplement the terms of this
346 | License by making exceptions from one or more of its conditions.
347 | Additional permissions that are applicable to the entire Program shall
348 | be treated as though they were included in this License, to the extent
349 | that they are valid under applicable law. If additional permissions
350 | apply only to part of the Program, that part may be used separately
351 | under those permissions, but the entire Program remains governed by
352 | this License without regard to the additional permissions.
353 |
354 | When you convey a copy of a covered work, you may at your option
355 | remove any additional permissions from that copy, or from any part of
356 | it. (Additional permissions may be written to require their own
357 | removal in certain cases when you modify the work.) You may place
358 | additional permissions on material, added by you to a covered work,
359 | for which you have or can give appropriate copyright permission.
360 |
361 | Notwithstanding any other provision of this License, for material you
362 | add to a covered work, you may (if authorized by the copyright holders of
363 | that material) supplement the terms of this License with terms:
364 |
365 | a) Disclaiming warranty or limiting liability differently from the
366 | terms of sections 15 and 16 of this License; or
367 |
368 | b) Requiring preservation of specified reasonable legal notices or
369 | author attributions in that material or in the Appropriate Legal
370 | Notices displayed by works containing it; or
371 |
372 | c) Prohibiting misrepresentation of the origin of that material, or
373 | requiring that modified versions of such material be marked in
374 | reasonable ways as different from the original version; or
375 |
376 | d) Limiting the use for publicity purposes of names of licensors or
377 | authors of the material; or
378 |
379 | e) Declining to grant rights under trademark law for use of some
380 | trade names, trademarks, or service marks; or
381 |
382 | f) Requiring indemnification of licensors and authors of that
383 | material by anyone who conveys the material (or modified versions of
384 | it) with contractual assumptions of liability to the recipient, for
385 | any liability that these contractual assumptions directly impose on
386 | those licensors and authors.
387 |
388 | All other non-permissive additional terms are considered "further
389 | restrictions" within the meaning of section 10. If the Program as you
390 | received it, or any part of it, contains a notice stating that it is
391 | governed by this License along with a term that is a further
392 | restriction, you may remove that term. If a license document contains
393 | a further restriction but permits relicensing or conveying under this
394 | License, you may add to a covered work material governed by the terms
395 | of that license document, provided that the further restriction does
396 | not survive such relicensing or conveying.
397 |
398 | If you add terms to a covered work in accord with this section, you
399 | must place, in the relevant source files, a statement of the
400 | additional terms that apply to those files, or a notice indicating
401 | where to find the applicable terms.
402 |
403 | Additional terms, permissive or non-permissive, may be stated in the
404 | form of a separately written license, or stated as exceptions;
405 | the above requirements apply either way.
406 |
407 | 8. Termination.
408 |
409 | You may not propagate or modify a covered work except as expressly
410 | provided under this License. Any attempt otherwise to propagate or
411 | modify it is void, and will automatically terminate your rights under
412 | this License (including any patent licenses granted under the third
413 | paragraph of section 11).
414 |
415 | However, if you cease all violation of this License, then your
416 | license from a particular copyright holder is reinstated (a)
417 | provisionally, unless and until the copyright holder explicitly and
418 | finally terminates your license, and (b) permanently, if the copyright
419 | holder fails to notify you of the violation by some reasonable means
420 | prior to 60 days after the cessation.
421 |
422 | Moreover, your license from a particular copyright holder is
423 | reinstated permanently if the copyright holder notifies you of the
424 | violation by some reasonable means, this is the first time you have
425 | received notice of violation of this License (for any work) from that
426 | copyright holder, and you cure the violation prior to 30 days after
427 | your receipt of the notice.
428 |
429 | Termination of your rights under this section does not terminate the
430 | licenses of parties who have received copies or rights from you under
431 | this License. If your rights have been terminated and not permanently
432 | reinstated, you do not qualify to receive new licenses for the same
433 | material under section 10.
434 |
435 | 9. Acceptance Not Required for Having Copies.
436 |
437 | You are not required to accept this License in order to receive or
438 | run a copy of the Program. Ancillary propagation of a covered work
439 | occurring solely as a consequence of using peer-to-peer transmission
440 | to receive a copy likewise does not require acceptance. However,
441 | nothing other than this License grants you permission to propagate or
442 | modify any covered work. These actions infringe copyright if you do
443 | not accept this License. Therefore, by modifying or propagating a
444 | covered work, you indicate your acceptance of this License to do so.
445 |
446 | 10. Automatic Licensing of Downstream Recipients.
447 |
448 | Each time you convey a covered work, the recipient automatically
449 | receives a license from the original licensors, to run, modify and
450 | propagate that work, subject to this License. You are not responsible
451 | for enforcing compliance by third parties with this License.
452 |
453 | An "entity transaction" is a transaction transferring control of an
454 | organization, or substantially all assets of one, or subdividing an
455 | organization, or merging organizations. If propagation of a covered
456 | work results from an entity transaction, each party to that
457 | transaction who receives a copy of the work also receives whatever
458 | licenses to the work the party's predecessor in interest had or could
459 | give under the previous paragraph, plus a right to possession of the
460 | Corresponding Source of the work from the predecessor in interest, if
461 | the predecessor has it or can get it with reasonable efforts.
462 |
463 | You may not impose any further restrictions on the exercise of the
464 | rights granted or affirmed under this License. For example, you may
465 | not impose a license fee, royalty, or other charge for exercise of
466 | rights granted under this License, and you may not initiate litigation
467 | (including a cross-claim or counterclaim in a lawsuit) alleging that
468 | any patent claim is infringed by making, using, selling, offering for
469 | sale, or importing the Program or any portion of it.
470 |
471 | 11. Patents.
472 |
473 | A "contributor" is a copyright holder who authorizes use under this
474 | License of the Program or a work on which the Program is based. The
475 | work thus licensed is called the contributor's "contributor version".
476 |
477 | A contributor's "essential patent claims" are all patent claims
478 | owned or controlled by the contributor, whether already acquired or
479 | hereafter acquired, that would be infringed by some manner, permitted
480 | by this License, of making, using, or selling its contributor version,
481 | but do not include claims that would be infringed only as a
482 | consequence of further modification of the contributor version. For
483 | purposes of this definition, "control" includes the right to grant
484 | patent sublicenses in a manner consistent with the requirements of
485 | this License.
486 |
487 | Each contributor grants you a non-exclusive, worldwide, royalty-free
488 | patent license under the contributor's essential patent claims, to
489 | make, use, sell, offer for sale, import and otherwise run, modify and
490 | propagate the contents of its contributor version.
491 |
492 | In the following three paragraphs, a "patent license" is any express
493 | agreement or commitment, however denominated, not to enforce a patent
494 | (such as an express permission to practice a patent or covenant not to
495 | sue for patent infringement). To "grant" such a patent license to a
496 | party means to make such an agreement or commitment not to enforce a
497 | patent against the party.
498 |
499 | If you convey a covered work, knowingly relying on a patent license,
500 | and the Corresponding Source of the work is not available for anyone
501 | to copy, free of charge and under the terms of this License, through a
502 | publicly available network server or other readily accessible means,
503 | then you must either (1) cause the Corresponding Source to be so
504 | available, or (2) arrange to deprive yourself of the benefit of the
505 | patent license for this particular work, or (3) arrange, in a manner
506 | consistent with the requirements of this License, to extend the patent
507 | license to downstream recipients. "Knowingly relying" means you have
508 | actual knowledge that, but for the patent license, your conveying the
509 | covered work in a country, or your recipient's use of the covered work
510 | in a country, would infringe one or more identifiable patents in that
511 | country that you have reason to believe are valid.
512 |
513 | If, pursuant to or in connection with a single transaction or
514 | arrangement, you convey, or propagate by procuring conveyance of, a
515 | covered work, and grant a patent license to some of the parties
516 | receiving the covered work authorizing them to use, propagate, modify
517 | or convey a specific copy of the covered work, then the patent license
518 | you grant is automatically extended to all recipients of the covered
519 | work and works based on it.
520 |
521 | A patent license is "discriminatory" if it does not include within
522 | the scope of its coverage, prohibits the exercise of, or is
523 | conditioned on the non-exercise of one or more of the rights that are
524 | specifically granted under this License. You may not convey a covered
525 | work if you are a party to an arrangement with a third party that is
526 | in the business of distributing software, under which you make payment
527 | to the third party based on the extent of your activity of conveying
528 | the work, and under which the third party grants, to any of the
529 | parties who would receive the covered work from you, a discriminatory
530 | patent license (a) in connection with copies of the covered work
531 | conveyed by you (or copies made from those copies), or (b) primarily
532 | for and in connection with specific products or compilations that
533 | contain the covered work, unless you entered into that arrangement,
534 | or that patent license was granted, prior to 28 March 2007.
535 |
536 | Nothing in this License shall be construed as excluding or limiting
537 | any implied license or other defenses to infringement that may
538 | otherwise be available to you under applicable patent law.
539 |
540 | 12. No Surrender of Others' Freedom.
541 |
542 | If conditions are imposed on you (whether by court order, agreement or
543 | otherwise) that contradict the conditions of this License, they do not
544 | excuse you from the conditions of this License. If you cannot convey a
545 | covered work so as to satisfy simultaneously your obligations under this
546 | License and any other pertinent obligations, then as a consequence you may
547 | not convey it at all. For example, if you agree to terms that obligate you
548 | to collect a royalty for further conveying from those to whom you convey
549 | the Program, the only way you could satisfy both those terms and this
550 | License would be to refrain entirely from conveying the Program.
551 |
552 | 13. Use with the GNU Affero General Public License.
553 |
554 | Notwithstanding any other provision of this License, you have
555 | permission to link or combine any covered work with a work licensed
556 | under version 3 of the GNU Affero General Public License into a single
557 | combined work, and to convey the resulting work. The terms of this
558 | License will continue to apply to the part which is the covered work,
559 | but the special requirements of the GNU Affero General Public License,
560 | section 13, concerning interaction through a network will apply to the
561 | combination as such.
562 |
563 | 14. Revised Versions of this License.
564 |
565 | The Free Software Foundation may publish revised and/or new versions of
566 | the GNU General Public License from time to time. Such new versions will
567 | be similar in spirit to the present version, but may differ in detail to
568 | address new problems or concerns.
569 |
570 | Each version is given a distinguishing version number. If the
571 | Program specifies that a certain numbered version of the GNU General
572 | Public License "or any later version" applies to it, you have the
573 | option of following the terms and conditions either of that numbered
574 | version or of any later version published by the Free Software
575 | Foundation. If the Program does not specify a version number of the
576 | GNU General Public License, you may choose any version ever published
577 | by the Free Software Foundation.
578 |
579 | If the Program specifies that a proxy can decide which future
580 | versions of the GNU General Public License can be used, that proxy's
581 | public statement of acceptance of a version permanently authorizes you
582 | to choose that version for the Program.
583 |
584 | Later license versions may give you additional or different
585 | permissions. However, no additional obligations are imposed on any
586 | author or copyright holder as a result of your choosing to follow a
587 | later version.
588 |
589 | 15. Disclaimer of Warranty.
590 |
591 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
592 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
593 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
594 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
595 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
596 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
597 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
598 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
599 |
600 | 16. Limitation of Liability.
601 |
602 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
603 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
604 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
605 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
606 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
607 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
608 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
609 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
610 | SUCH DAMAGES.
611 |
612 | 17. Interpretation of Sections 15 and 16.
613 |
614 | If the disclaimer of warranty and limitation of liability provided
615 | above cannot be given local legal effect according to their terms,
616 | reviewing courts shall apply local law that most closely approximates
617 | an absolute waiver of all civil liability in connection with the
618 | Program, unless a warranty or assumption of liability accompanies a
619 | copy of the Program in return for a fee.
620 |
621 | END OF TERMS AND CONDITIONS
622 |
623 | How to Apply These Terms to Your New Programs
624 |
625 | If you develop a new program, and you want it to be of the greatest
626 | possible use to the public, the best way to achieve this is to make it
627 | free software which everyone can redistribute and change under these terms.
628 |
629 | To do so, attach the following notices to the program. It is safest
630 | to attach them to the start of each source file to most effectively
631 | state the exclusion of warranty; and each file should have at least
632 | the "copyright" line and a pointer to where the full notice is found.
633 |
634 | {one line to give the program's name and a brief idea of what it does.}
635 | Copyright (C) {year} {name of author}
636 |
637 | This program is free software: you can redistribute it and/or modify
638 | it under the terms of the GNU General Public License as published by
639 | the Free Software Foundation, either version 3 of the License, or
640 | (at your option) any later version.
641 |
642 | This program is distributed in the hope that it will be useful,
643 | but WITHOUT ANY WARRANTY; without even the implied warranty of
644 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
645 | GNU General Public License for more details.
646 |
647 | You should have received a copy of the GNU General Public License
648 | along with this program. If not, see .
649 |
650 | Also add information on how to contact you by electronic and paper mail.
651 |
652 | If the program does terminal interaction, make it output a short
653 | notice like this when it starts in an interactive mode:
654 |
655 | {project} Copyright (C) {year} {fullname}
656 | This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
657 | This is free software, and you are welcome to redistribute it
658 | under certain conditions; type `show c' for details.
659 |
660 | The hypothetical commands `show w' and `show c' should show the appropriate
661 | parts of the General Public License. Of course, your program's commands
662 | might be different; for a GUI interface, you would use an "about box".
663 |
664 | You should also get your employer (if you work as a programmer) or school,
665 | if any, to sign a "copyright disclaimer" for the program, if necessary.
666 | For more information on this, and how to apply and follow the GNU GPL, see
667 | .
668 |
669 | The GNU General Public License does not permit incorporating your program
670 | into proprietary programs. If your program is a subroutine library, you
671 | may consider it more useful to permit linking proprietary applications with
672 | the library. If this is what you want to do, use the GNU Lesser General
673 | Public License instead of this License. But first, please read
674 | .
675 |
--------------------------------------------------------------------------------
/NamesList.md:
--------------------------------------------------------------------------------
1 | # About /usr/share/unicode/NamesList.txt
2 |
3 | ```
4 | @ Food symbols
5 | 1F374 FORK AND KNIFE
6 | = restaurant, meal
7 | * glyph may show a fork and spoon
8 | * glyph may show a crossed fork and knife
9 | x (fork and knife with plate - 1F37D)
10 | x (spoon - 1F944)
11 | ```
12 |
13 | A supplemental file for UnicodeData.txt. Contains synonyms, cross
14 | references, variations, and commentary. This is where one would find
15 | "EGGPLANT", for example, instead of the official name in
16 | UnicodeData.txt, "AUBERGINE".
17 |
18 | Each entry crosses multiple lines, which makes regex search a little
19 | trickier. (Especially since I can't seem to get Python's multiline
20 | support to work). However, there is *plenty* of information available
21 | on how it should be parsed, perhaps too much as the basics can be
22 | grasped just by looking at examples.
23 |
24 | * [Latest version of NamesList.txt](http://www.unicode.org/Public/UNIDATA/NamesList.txt).
25 | * [Full specification for the file format](http://www.unicode.org/Public/UCD/latest/ucd/NamesList.html)
26 | * [Simpler overview included in UniBook docs](http://unicode.org/unibook/help/nmlstfmt.htm)
27 |
28 | ### "NOT MACHINE READABLE"
29 |
30 | The NamesList.txt file begins with a header discouraging
31 | machine-reading.
32 |
33 |
34 | This file is semi-automatically derived from UnicodeData.txt and
35 | a set of manually created annotations using a script to select
36 | or suppress information from the data file. The rules used
37 | for this process are aimed at readability for the human reader,
38 | at the expense of some details; therefore, this file should not
39 | be parsed for machine-readable information.
40 |
41 |
42 | However, since this data is nowhere else in the Unicode standard, not
43 | only are we forced to parse it, so is the Unicode Consortium itself!
44 | The [Unibook](https://www.unicode.org/unibook/help/unibook.htm)
45 | software Unicode uses to create their character code charts relies on
46 | the machine readability of NamesList.txt.
47 |
48 | ## Examples
49 |
50 | In the following examples ⇥⃞ represents the TAB character and ␠ represents SPACE.
51 |
52 |
53 | | Character | Meaning |
54 | |-------------------|--------------|
55 | | @@⇥⃞START⇥⃞NAME⇥⃞END | Block Header |
56 |
57 |
58 | Note the START and END character positions surrounding the text of the
59 | block header.
60 |
61 |
62 | ```
63 | @@ 0000 C0 Controls and Basic Latin (Basic Latin) 007F
64 | @@+
65 | @ C0 controls
66 | @+ Alias names are those for ISO/IEC 6429:1992. Commonly used alternative aliases are also shown.
67 | 0000
68 | = NULL
69 | 0001
70 | = START OF HEADING
71 | 0002
72 | = START OF TEXT
73 | 0003
74 | = END OF TEXT
75 | 0004
76 | = END OF TRANSMISSION
77 | ```
78 |
79 | | Character | Meaning |
80 | |-----------|------------------------|
81 | | @⇥⃞⇥⃞LINE | Subheader |
82 | | @+⇥⃞⇥⃞LINE | Notice |
83 | | ⇥⃞⇥⃞x␠ | Xref in notice section |
84 |
85 | ```
86 | @ ASCII punctuation and symbols
87 | @+ Based on ISO/IEC 646.
88 | 0020 SPACE
89 | * sometimes considered a control code
90 | * other space characters: 2000-200A
91 | x (no-break space - 00A0)
92 | x (zero width space - 200B)
93 | x (narrow no-break space - 202F)
94 | x (word joiner - 2060)
95 | x (symbol for space - 2420)
96 | x (blank symbol - 2422)
97 | x (open box - 2423)
98 | x (ideographic space - 3000)
99 | x (zero width no-break space - FEFF)
100 | ```
101 |
102 |
103 |
104 | | Character | Meaning | Presentation |
105 | |--------------------|------------------------|---------------|
106 | | ⇥⃞%␠ | Formal alias | ※ |
107 | | ⇥⃞=␠ | Alias | = |
108 | | ⇥⃞*␠ | Commentary | • |
109 | | ⇥⃞x␠CHAR␠LCNAME | Cross reference | → CHAR LCNAME |
110 | | ⇥⃞x␠(LCNAME␠-␠CHAR) | Cross reference | → CHAR LCNAME |
111 | | ⇥⃞#␠ | Compatibility mapping | ≈ |
112 | | ⇥⃞:␠ | Decomposition | ≡ |
113 | | ⇥⃞~␠ | Variant glyph | |
114 |
115 |
116 | ```
117 | 1D0C5 BYZANTINE MUSICAL SYMBOL FHTORA SKLIRON CHROMA VASIS
118 | % BYZANTINE MUSICAL SYMBOL FTHORA SKLIRON CHROMA VASIS
119 | * misspelling of "FTHORA" in character name is a known defect
120 | ```
121 |
122 | ```
123 | 0023 NUMBER SIGN
124 | = pound sign (weight)
125 | = hashtag, hash
126 | = crosshatch, octothorpe
127 | * for denoting musical sharp 266F is preferred
128 | x (l b bar symbol - 2114)
129 | x (numero sign - 2116)
130 | x (viewdata square - 2317)
131 | x (music sharp sign - 266F)
132 | x (equals sign and slanted parallel - 29E3)
133 | ```
134 |
135 | ```
136 | 0030 DIGIT ZERO
137 | ~ 0030 FE00 short diagonal stroke form
138 | ```
139 |
140 | ```
141 | 2052 COMMERCIAL MINUS SIGN
142 | = abzüglich (German), med avdrag av (Swedish), piska (Swedish, "whip")
143 | * a common glyph variant and fallback representation looks like ./.
144 | * may also be used as a dingbat to indicate correctness
145 | * used in Finno-Ugric Phonetic Alphabet to indicate a related borrowed form with different sound
146 | x (percent sign - 0025)
147 | x (arabic percent sign - 066A)
148 | x (division sign - 00F7)
149 | ```
150 |
151 | ```
152 | 1FBB2 LEFT HALF RUNNING MAN
153 | * paired with 1FBB3, faces to the right
154 | * the Apple II documentation refers to these characters as "Running Man"
155 | x (runner - 1F3C3)
156 | ```
157 |
158 | ```
159 | 1D1BB MUSICAL SYMBOL MINIMA
160 | : 1D1B9 1D165
161 |
162 | ```
163 |
164 | | Character | Meaning | Presentation |
165 | |-----------|------------------------|--------------|
166 | | @+⇥⃞*␠ | Notice with bullet | • |
167 | | ⇥⃞⇥⃞x␠ | Xref in notice section
Applies to whole | → |
168 |
169 | ```
170 | 0269 LATIN SMALL LETTER IOTA
171 | * semi-high front unrounded vowel
172 | @+ * obsoleted by IPA in 1989
173 | * preferred use is 026A latin letter small capital i
174 | * uppercase is 0196
175 | x (greek small letter iota - 03B9)
176 | ```
177 |
178 | ```
179 | @ Accidentals
180 | @+ The most common accidentals are encoded in the Miscellaneous Symbols block.
181 | x (music flat sign - 266D)
182 | x (music natural sign - 266E)
183 | x (music sharp sign - 266F)
184 | 1D12A MUSICAL SYMBOL DOUBLE SHARP
185 | 1D12B MUSICAL SYMBOL DOUBLE FLAT
186 | ```
187 |
188 |
189 |
190 |
191 | ## NamesList.txt Syntax
192 |
193 | Here is a simplified understanding which works well enough for our purposes.
194 |
195 | Each character on its own line :
196 | hex⇥⃞charname | 1F346 AUBERGINE
197 | |
198 | Followed by zero or more lines: |
199 | Synonym: |
200 | ⇥⃞= text[, text...] | = eggplant
201 | Commentary on standards |
202 | @+⇥⃞* text | @+ * see ISO 69835 for details
203 | Commentary: |
204 | ⇥⃞* text | * most commonly depicts a penis
205 | Cross reference: |
206 | ⇥⃞x (text - hex) | x (hand with middle finger extended - 1F595)
207 | Variation: |
208 | ⇥⃞# hex... text | # 2642 male sign
209 | ⇥⃞# hex... | # 1F34C
210 | Canonical equivalent: |
211 | ⇥⃞: hex text | 00C0 LATIN CAPITAL LETTER A WITH GRAVE
212 | | : 0041 0300
213 | Alternate presentation form: |
214 | ⇥⃞~ hex... text | FF1F FULLWIDTH QUESTION MARK
215 | | ~ FF1F FE00 corner-justified form
216 | | ~ FF1F FE01 centered form
217 | | # 003F
218 | |
219 | Range: @@⇥⃞start⇥⃞description⇥⃞end | @@ 1F650 Ornamental Dingbats 1F67F
220 | Section dividers: @⇥⃞⇥⃞text | @ Fleurons
221 | Section comments: @+⇥⃞⇥⃞text | @+ Fleurons are leaf or floral-shaped ornaments used for text decoration.
222 | | 1F650 NORTH WEST POINTING LEAF
223 | | 1F651 SOUTH WEST POINTING LEAF
224 |
225 |
226 | Examples of regulard commentary and standards commentary:
227 | 1E37 LATIN SMALL LETTER L WITH DOT BELOW
228 | * Indic transliteration
229 | : 006C 0323
230 | @+ * see ISO 15919 on the use of dot below versus ring below in Indic transliteration
231 | x (combining ring below - 0325)
232 | 2301 ELECTRIC ARROW
233 | @+ * from ISO 2047
234 | * symbol for End of Transmission
235 |
236 | Example of commentary on sections (indented by two tabs):
237 | @ Ceilings and floors
238 | @+ These characters are tall and narrow mathematical delimiters, in contrast to the quine corners or half brackets. They are also distinct from CJK corner brackets, which are wide quotation marks.
239 |
240 |
241 | Variation Examples:
242 | 2007 FIGURE SPACE
243 | * space equal to tabular width of a font
244 | * this is equivalent to the digit width of fonts with fixed-width digits
245 | # 0020
246 | 200A HAIR SPACE
247 | * thinner than a thin space
248 | * in traditional typography, the thinnest space available
249 | # 0020 space
250 | 2026 HORIZONTAL ELLIPSIS
251 | = three dot leader
252 | x (vertical ellipsis - 22EE)
253 | x (presentation form for vertical horizontal ellipsis - FE19)
254 | # 002E 002E 002E
255 | 1FBF0 SEGMENTED DIGIT ZERO
256 | # 0030 digit zero
257 |
258 | Canonical equivalent example:
259 | 1D164 MUSICAL SYMBOL ONE HUNDRED TWENTY-EIGHTH NOTE
260 | = semihemidemisemiquaver, quasihemidemisemiquaver
261 | : 1D15F 1D172
262 |
263 | Alternate form example:
264 | 2205 EMPTY SET
265 | = null set
266 | * used in linguistics to indicate a null morpheme or phonological "zero"
267 | x (latin capital letter o with stroke - 00D8)
268 | x (diameter sign - 2300)
269 | ~ 2205 FE00 zero with long diagonal stroke overlay form
270 |
271 | Extended attribute lines attached to section comment:
272 | @ Accidentals
273 | @+ The most common accidentals are encoded in the Miscellaneous Symbols block.
274 | x (music flat sign - 266D)
275 | x (music natural sign - 266E)
276 | x (music sharp sign - 266F)
277 |
278 |
279 | What does @~ mean? It is always followed by either "⇥⃞!" or
280 | "⇥⃞Standardized Variation Sequences"
281 |
282 | 007F
283 | = DELETE
284 | @~ !
285 | @@ 0080 C1 Controls and Latin-1 Supplement (Latin-1 Supplement) 00FF
286 | @ C1 controls
287 | @+ Alias names are those for ISO/IEC 6429:1992.
288 | 0080
289 |
290 |
291 |
292 | All lines that start with @ have one of these prefixes:
293 | # @⇥⃞⇥⃞ |Section title
294 | # @+⇥⃞⇥⃞ | Section comment
295 | # @+⇥⃞* | Standards comment
296 | # @@⇥⃞ | Range Title
297 | # @@+ | [The space intentionally left blank]
298 | # @~⇥⃞ | "Standardized Variation Sequences"(?)
299 | # @@@ | Unicode Standard folderol
300 |
301 | Ignoring the first 12 lines which are just Unicode prologue,
302 | all lines that start with ⇥⃞ (TAB) have one of these prefixes
303 |
304 |
305 |
306 | ### Example of Unicode commentary:
307 | FEFF ZERO WIDTH NO-BREAK SPACE
308 | % BYTE ORDER MARK
309 | = BOM, ZWNBSP
310 | 0EA3 LAO LETTER LO LING
311 | % LAO LETTER RO
312 | = ro rot
313 | * name is a mistake, lo ling is the mnemonic for 0EA5
314 | 0EA5 LAO LETTER LO LOOT
315 | % LAO LETTER LO
316 | = lo ling
317 | * name is a mistake, lo loot is the mnemonic for 0EA3
318 |
319 | ### Three entries have abberant use of semicolons instead of commas:
320 | 1F70A ALCHEMICAL SYMBOL FOR VINEGAR
321 | = crucible; acid; distill; atrament; vitriol; red sulfur; borax; wine; alkali salt; mercurius vivus, quick silver
322 | x (cross of jerusalem - 2629)
323 | 0598 HEBREW ACCENT ZARQA
324 | = tsinorit, zinorit; tsinor, zinor
325 | * This character is to be used when Zarqa or Tsinor are placed above, and also for Tsinorit.
326 | x (hebrew accent zinor - 05AE)
327 | 05AE HEBREW ACCENT ZINOR
328 | = tsinor; zarqa
329 | * This character is to be used when Zarqa or Tsinor are placed above left.
330 | x (hebrew accent zarqa - 0598)
331 |
332 | ### Maybe should treat semicolons as commas, but that would break this one entry:
333 | 29DC INCOMPLETE INFINITY
334 | = ISOtech entity ⧜
335 | x (infinity - 221E)
336 |
337 |
338 |
339 |
340 |
341 | EXAMPLE ENTRIES
342 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 |
2 | 
3 |
4 |
5 | # ☙ ugrep ❧
6 |
7 | _Find unicode characters based on their names_
8 |
9 | ugrep is essentially [grep](https://www.gnu.org/software/grep/) for
10 | the Unicode table. It prints out the resulting unicode characters
11 | literally, so you can easily cut-and-paste. Ugrep is useful for
12 | looking up Emojis 😤, finding obscure symbols ⚸⅗ℏ℞☧☭, or beautiful
13 | glyphs to decorate your text. 🙶❡✯🟔❢🙷
14 |
15 | You can also use it for the reverse operation to lookup a single
16 | character (or a string of them) you've pasted into the terminal.
17 |
18 | As a bonus, it can list which fonts are installed that contain a
19 | particular unicode character and — through the magic of sixels — will
20 | show a rendering in each font.
21 |
22 | ## Installation
23 |
24 | It's just a Python 3 shell script. Download it to `/usr/local/bin` or `~/bin`
25 | and make it executable.
26 |
27 | cd /usr/local/bin
28 | wget https://github.com/hackerb9/ugrep/raw/master/ugrep
29 | chmod +x ugrep
30 |
31 | ## Usage
32 |
33 | * Search by name: **ugrep** [**-w**] _regex_
34 |
35 | Look up a character name where _regex_ is a regular
36 | expression. If you don't know [regular
37 | expressions](https://docs.python.org/3/howto/regex.html),
38 | don't worry. Just use plain strings and you'll rarely be
39 | wrong.
40 |
41 | ugrep runic
42 |
43 | If you find ugrep returning too many hits because the phrase you used
44 | is found in other terms, e.g., _thema_ found in _mathematical_, use
45 | the **-w** option to limit the search to complete words.
46 |
47 | * Search by number: **ugrep** _codepoint_**[..**_codepoint_**[..**_increment_**]]**
48 |
49 | Look up a character (or a range of them) using Unicode code points in
50 | hexadecimal. For example,
51 |
52 | ugrep 03c0
53 | ugrep 23b0..f
54 | ugrep 0..10ffff..1000
55 |
56 | * Search by character: **ugrep** [**-c**] _character string_
57 |
58 | Look up each character in a string. Note that if the string is a
59 | single character, e.g., `ugrep X`, then **-c** is implied and need not
60 | be specified.
61 |
62 | ugrep -c "(ﾟ∀ﾟ)"
63 |
64 | * List fonts for a character: **ugrep** [**-l**] _character_
65 |
66 | After showing the usual character information, list installed
67 | fonts that contain that character and show an example in each:
68 |
69 | ugrep -l mho
70 |
71 | ☝ *When `ssh`ed to another machine, `ugrep` shows the fonts
72 | installed on the remote machine.*
73 |
74 | * List fonts, scaled larger: **ugrep** [**-L** _scale_] _character_
75 |
76 | Same as `-l`, but scale up the example rendering in each font to
77 | be easier to read:
78 |
79 | ugrep -L2 -w om
80 |
81 | Useful scale values range from 2 to 8.
82 |
83 | ## Examples
84 |
85 | Note: output from all examples has been excerpted. (You'd be amazed
86 | how many heart emojis Unicode has. 😜)
87 |
88 |
89 |
90 | ### Fun things to try:
91 |
92 | To see some useful and lovely glyphs, try this:
93 |
94 | ugrep face
95 | ugrep alchemical
96 | ugrep ornament
97 | ugrep bullet
98 | ugrep '(vine|bud)'
99 | ugrep vai
100 | ugrep heavy
101 | ugrep drawing
102 | ugrep combining
103 |
104 | ### Plain text search is simple:
105 |
106 | $ ugrep heart
107 | ☙ U+2619 REVERSED ROTATED FLORAL HEART BULLET
108 | ❣ U+2763 HEAVY HEART EXCLAMATION MARK ORNAMENT
109 | ❤ U+2764 HEAVY BLACK HEART
110 | ⋮ [ ... truncated for brevity ... ]
111 | 💞 U+1F49E REVOLVING HEARTS
112 | 💟 U+1F49F HEART DECORATION
113 | 😍 U+1F60D SMILING FACE WITH HEART-SHAPED EYES
114 | 😻 U+1F63B SMILING CAT FACE WITH HEART-SHAPED EYES
115 |
116 | ### Paste in a single character to lookup its codepoint:
117 |
118 | $ ugrep ☺
119 | ☺ U+263A WHITE SMILING FACE
120 |
121 | ### Arguments on the command line have an implicit wildcard between them:
122 |
123 | $ ugrep right.*gle
124 | $ ugrep right gle # Equivalent
125 | » U+00BB RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK
126 | ’ U+2019 RIGHT SINGLE QUOTATION MARK
127 | ∟ U+221F RIGHT ANGLE
128 | ⊿ U+22BF RIGHT TRIANGLE
129 |
130 | ### You can use regular expressions for fancier searches:
131 |
132 | $ ugrep -w '(wo|hu)?m(a|e)ns?'
133 | ᛗ U+16D7 RUNIC LETTER MANNAZ MAN M
134 | ⛀ U+26C0 WHITE DRAUGHTS MAN
135 | ⛂ U+26C2 BLACK DRAUGHTS MAN
136 | ⼈ U+2F08 KANGXI RADICAL MAN
137 | ⼥ U+2F25 KANGXI RADICAL WOMAN
138 | 𝌂 U+1D302 DIGRAM FOR HUMAN EARTH
139 | 𝌄 U+1D304 DIGRAM FOR EARTHLY HUMAN
140 | 🕴 U+1F574 MAN IN BUSINESS SUIT LEVITATING
141 | 🕺 U+1F57A MAN DANCING
142 | 🚹 U+1F6B9 MENS SYMBOL
143 | 🚺 U+1F6BA WOMENS SYMBOL
144 | 🤰 U+1F930 PREGNANT WOMAN
145 | 🤵 U+1F935 MAN IN TUXEDO
146 |
147 | $ ugrep ^x # Regex anchors ^ and $ work
148 | ⊻ U+22BB XOR
149 | ⌧ U+2327 X IN A RECTANGLE BOX (clear key)
150 |
151 | ### Use the `-w` flag to search only for complete words:
152 |
153 | $ ugrep -w R # The letter R used as a word
154 | $ ugrep "\bR\b" # (regex equivalent)
155 | R U+0052 LATIN CAPITAL LETTER R
156 | Ŗ U+0156 LATIN CAPITAL LETTER R WITH CEDILLA
157 | ℛ U+211B SCRIPT CAPITAL R (Script r)
158 | ℜ U+211C BLACK-LETTER CAPITAL R (Black-letter r)
159 | ℝ U+211D DOUBLE-STRUCK CAPITAL R (Double-struck r)
160 |
161 | ### Use -c to display info for each character in a string.
162 |
163 | $ ugrep -c "ᕕ( ᐛ )ᕗ"
164 | ᕕ U+1555 CANADIAN SYLLABICS FI
165 | ( U+0028 LEFT PARENTHESIS (opening parenthesis)
166 | U+0020 SPACE
167 | ᐛ U+141B CANADIAN SYLLABICS NASKAPI WAA
168 | U+0020 SPACE
169 | ) U+0029 RIGHT PARENTHESIS (closing parenthesis)
170 | ᕗ U+1557 CANADIAN SYLLABICS FO
171 |
172 | ### Aliases (alternate names) are also searched:
173 |
174 | $ ugrep backslash
175 | \ U+005C REVERSE SOLIDUS (backslash)
176 |
177 | ### Use **..** to browse through a range of Unicode characters:
178 |
179 | $ ugrep 26b3..b
180 | ⚳ U+26B3 CERES
181 | ⚴ U+26B4 PALLAS
182 | ⚵ U+26B5 JUNO
183 | ⚶ U+26B6 VESTA
184 | ⚷ U+26B7 CHIRON
185 | ⚸ U+26B8 BLACK MOON LILITH
186 | ⚹ U+26B9 SEXTILE
187 | ⚺ U+26BA SEMISEXTILE
188 | ⚻ U+26BB QUINCUNX
189 |
190 | $ ugrep 1f470..ff | less
191 | 👰 U+1F470 BRIDE WITH VEIL
192 | 👱 U+1F471 PERSON WITH BLOND HAIR
193 | 👲 U+1F472 MAN WITH GUA PI MAO
194 | 👳 U+1F473 MAN WITH TURBAN
195 | 👴 U+1F474 OLDER MAN
196 | 👵 U+1F475 OLDER WOMAN
197 | 👶 U+1F476 BABY
198 | 👷 U+1F477 CONSTRUCTION WORKER
199 | 👸 U+1F478 PRINCESS
200 | 👹 U+1F479 JAPANESE OGRE
201 | 👺 U+1F47A JAPANESE GOBLIN
202 | 👻 U+1F47B GHOST
203 | 👼 U+1F47C BABY ANGEL
204 | 👽 U+1F47D EXTRATERRESTRIAL ALIEN
205 | ⋮ [ ... truncated for brevity ... ]
206 | 📼 U+1F4FC VIDEOCASSETTE
207 | 📽 U+1F4FD FILM PROJECTOR
208 | 📾 U+1F4FE PORTABLE STEREO
209 | 📿 U+1F4FF PRAYER BEADS
210 |
211 | Sometimes it's useful (or just fun) to page through the Unicode
212 | table and see what characters are defined in a region. (`ugrep
213 | 2700..ff`) Ranges are convenient, but very slow. Use regular
214 | expressions if you want speed. (`ugrep U+27..`)
215 |
216 | ### Ranges can have an optional increment:
217 |
218 | ```
219 | $ ugrep 0..ffff..1000
220 | � U+0000 (null)
221 | က U+1000 MYANMAR LETTER KA
222 | [ ] U+2000 EN QUAD
223 | [ ] U+3000 IDEOGRAPHIC SPACE
224 | 䀀 U+4000 cups; small cups ( M: fàn, C: fan3 fan4 fan6 )
225 | 倀 U+5000 bewildered; rash, wildly ( M: chāng, C: caang1 caang4 coeng1 zaang1, J: KURUU TAORERU, K: CHANG, V: trành )
226 | 怀 U+6000 bosom, breast; carry in bosom ( M: huái, C: waai4 )
227 | 瀀 U+7000 [CJK Unified Ideographs] ( M: yōu, J: ATSUI )
228 | 耀 U+8000 shine, sparkle, dazzle; glory ( M: yào, C: jiu6, J: KAGAYAKU, K: YO )
229 | 退 U+9000 step back, retreat, withdraw ( M: tuì, C: teoi3, J: SHIRIZOKU SHIRIZOKERU, K: THOY, V: thoái )
230 | ꀀ U+A000 YI SYLLABLE IT
231 | 뀀 U+B000 Block: [Hangul Syllables]
232 | 쀀 U+C000 Block: [Hangul Syllables]
233 | 퀀 U+D000 Block: [Hangul Syllables]
234 | � U+E000
235 | U+F000 Block: [Private Use Area]
236 | ```
237 |
238 | * Tip: pipe long output to `less` and search for a code point by
239 | pressing `/U\+A60F`.
240 |
241 | ### Use -l to list which installed fonts contain a certain glyph:
242 |
243 | 
244 |
245 |
246 | ugrep -l swash amp
247 |
248 | * Requires FontConfig. (Most GNU/Linux boxes should already be set).
249 |
250 | * The requested character may also be displayed in each of the
251 | listed typefaces, but only if your terminal supports sixel
252 | graphics (e.g., `xterm -ti vt340`) and you have ImageMagick
253 | installed.
254 |
255 | ### Use -L to scale up the font examples when listing fonts
256 |
257 |
258 | 
261 |
262 |
263 | ```
264 | ugrep -L4 fdfd
265 | ﷽ U+FDFD ARABIC LIGATURE BISMILLAH AR-RAHMAN AR-RAHEEM
266 | Aldhabi
267 | Trutypewriter PolyglOTT
268 | Unifont
269 | ```
270 |
271 | * Note that increasing the glyph size also increased the text size.
272 | Not all terminals are capable of "double height" text. If yours
273 | shows two lines of the same text in the usual size, try using
274 | `--never-double-text`.
275 |
276 | ### Copy whitespace from the terminal
277 |
278 | $ ugrep -w space
279 | [ ] U+0020 SPACE (SP)
280 | [ ] U+00A0 NO-BREAK SPACE (non-breaking space) (NBSP)
281 | [ ] U+1680 OGHAM SPACE MARK
282 | [ ] U+2002 EN SPACE
283 | [ ] U+2003 EM SPACE
284 | [ ] U+2004 THREE-PER-EM SPACE
285 | [ ] U+2005 FOUR-PER-EM SPACE
286 | [ ] U+2006 SIX-PER-EM SPACE
287 | [ ] U+2007 FIGURE SPACE
288 | [ ] U+2008 PUNCTUATION SPACE
289 | [ ] U+2009 THIN SPACE
290 | [ ] U+200A HAIR SPACE
291 |
292 | Whitespace characters are printed with square brackets around them
293 | to make it easy to highlight and copy them from the terminal. They
294 | will also be shown with a yellow background, if the terminal allows.
295 |
296 | ### Determine if an alias is actually a correction
297 |
298 | Ugrep shows the character name in all caps and aliases are usually
299 | lowercase in parentheses. Some aliases are treated differently.
300 | For aesthetic reasons, abbreviations are also shown in uppercase.
301 | For example:
302 |
303 | � U+FEFF ZERO WIDTH NO-BREAK SPACE (byte order mark) (BOM) (ZWNBSP)
304 |
305 | There are 31 characters in Unicode which have the wrong name in the
306 | UnicodeData.txt database. Unicode includes the correct name as an
307 | alias in NameAliases.txt. If that file exists on your system, then
308 | ugrep will show the correction in Title Case Letters and in red
309 | letters, if the terminal supports color text.
310 |
311 | ︘ U+FE18 PRESENTATION FORM FOR VERTICAL RIGHT WHITE LENTICULAR BRAKCET (Presentation Form For Vertical Right White Lenticular Bracket)
312 |
313 | ### View CJK (Chinese-Japanese-Korean) characters
314 |
315 | Unicode does not actually define most CJK characters, except
316 | indirectly via Unihan, which maps certain blocks of characters to
317 | other standards.
318 |
319 | * Ugrep allows one to specify the code point or paste in an example
320 | character to look up.
321 |
322 | $ ugrep 𰻞
323 | 𰻞 U+30EDE biangbiang noodles ( M: biáng )
324 |
325 | $ ugrep 8000
326 | 耀 U+8000 shine, sparkle, dazzle; glory ( M: yào, C: jiu6, J: KAGAYAKU, K: YO )
327 |
328 |
329 | 
332 |
333 |
334 | ### View _all_ characters defined by Unicode:
335 |
336 | $ ugrep .? | less
337 | ⋮ [ ... over 30,000 glyphs elided for brevity ... ]
338 |
339 | * Want just Unicode glyphs without the description? Please use
340 | [fonttable](https://github.com/hackerb9/fonttable). It shows all
341 | defined Unicode characters by default.
342 |
343 | ### Show all possible code points, even the ones _not_ defined in Unicode:
344 |
345 | $ ugrep 0..10FFFF | less
346 | ⋮ [ ... over a million lines elided for brevity ... ]
347 |
348 | ☝ This is currently very slow due to the way `ugrep` is implemented.
349 | You likely want to use
350 | [fonttable -u](https://github.com/hackerb9/fonttable) instead.
351 |
352 |
353 |
354 | ## Prerequisite: UnicodeData.txt
355 |
356 | Ugrep requires the Unicode data file
357 | [UnicodeData.txt](https://unicode.org/Public/UNIDATA/UnicodeData.txt)
358 | which can be installed on your system, in your home, or in the current
359 | directory.
360 |
361 | **Easiest**: On Ubuntu and Debian GNU/Linux, simply `apt install unicode-data`.
362 |
363 | **Still easy**: Or, you can download it by hand from
364 | [unicode.org](https://unicode.org/Public/UNIDATA/UnicodeData.txt)
365 | and place it in `~/.local/share/unicode/UnicodeData.txt`
366 |
367 | **Not hard**: Or, if you wish the file to be accessible to all users on
368 | your machine, place it in `/usr/local/share/unicode/UnicodeData.txt`.
369 |
370 | ## Unihan CJK Support
371 |
372 | If the file `Unihan_Readings.txt` exists, then ugrep will
373 | automatically use it to show an English gloss describing a character
374 | in the CJK (Chinese-Japanese-Korean) Ideographs region.
375 |
376 | Your OS may make it easy to install (e.g., `apt install unicode-data`).
377 | On other systems, you can do this
378 |
379 | mkdir -p ~/.local/share/unicode
380 | cd ~/.local/share/unicode
381 | wget ftp://ftp.unicode.org/Public/UNIDATA/Unihan.zip
382 | unzip Unihan.zip
383 |
384 | ### CJK example
385 |
386 |
387 |
388 | #### Example 1: Unicode code point
389 |
390 | ```
391 | $ ugrep 8000
392 | 耀 U+8000 shine, sparkle, dazzle; glory ( M: yào, C: jiu6, J: KAGAYAKU, K: YO )
393 | ```
394 |
395 | The parenthesized text at the end shows the romanized pronunciation of
396 | the character in **M**andarin (pinyin), **C**antonese (jyutping),
397 | **J**apanese (Hepburn), and **K**orean (Yale).
398 |
399 | #### Example 2: Using -c to see characters in a string
400 |
401 | ```
402 | $ ugrep -c 「⿺辶⿳穴⿰月⿰⿲⿱幺長⿱言馬⿱幺長刂心」
403 | 「 U+300C LEFT CORNER BRACKET (opening corner bracket)
404 | ⿺ U+2FFA IDEOGRAPHIC DESCRIPTION CHARACTER SURROUND FROM LOWER LEFT
405 | 辶 U+8FB6 walk; walking; KangXi radical 162 ( M: chuò, J: SHINNYOU )
406 | ⿳ U+2FF3 IDEOGRAPHIC DESCRIPTION CHARACTER ABOVE TO MIDDLE AND BELOW
407 | 穴 U+7A74 cave, den, hole; KangXi radical 116 ( M: xué, C: jyut6, J: ANA, K: HYEL, V: huyệt )
408 | ⿰ U+2FF0 IDEOGRAPHIC DESCRIPTION CHARACTER LEFT TO RIGHT
409 | 月 U+6708 moon; month; KangXi radical 74 ( M: yuè, C: jyut6, J: TSUKI, K: WEL, V: nguyệt )
410 | ⿰ U+2FF0 IDEOGRAPHIC DESCRIPTION CHARACTER LEFT TO RIGHT
411 | ⿲ U+2FF2 IDEOGRAPHIC DESCRIPTION CHARACTER LEFT TO MIDDLE AND RIGHT
412 | ⿱ U+2FF1 IDEOGRAPHIC DESCRIPTION CHARACTER ABOVE TO BELOW
413 | 幺 U+5E7A one; tiny, small ( M: yāo, C: jiu1, J: CHIISAI, K: YO )
414 | 長 U+9577 long; length; excel in; leader ( M: zhǎng, C: coeng4 zoeng2, J: NAGAI TAKERU OSA, K: CANG, V: trường )
415 | ⿱ U+2FF1 IDEOGRAPHIC DESCRIPTION CHARACTER ABOVE TO BELOW
416 | 言 U+8A00 words, speech; speak, say ( M: yán, C: jin4, J: KOTO IU KOTOBA, K: EN UN, V: ngôn )
417 | 馬 U+99AC horse; surname; KangXi radical 187 ( M: mǎ, C: maa5, J: UMA, K: MA, V: mã )
418 | ⿱ U+2FF1 IDEOGRAPHIC DESCRIPTION CHARACTER ABOVE TO BELOW
419 | 幺 U+5E7A one; tiny, small ( M: yāo, C: jiu1, J: CHIISAI, K: YO )
420 | 長 U+9577 long; length; excel in; leader ( M: zhǎng, C: coeng4 zoeng2, J: NAGAI TAKERU OSA, K: CANG, V: trường )
421 | 刂 U+5202 knife; radical number 18 ( M: dāo, C: dou1, J: RITSUTOU, K: TO )
422 | 心 U+5FC3 heart; mind, intelligence; soul ( M: xīn, C: sam1, J: KOKORO, K: SIM, V: tâm )
423 | 」 U+300D RIGHT CORNER BRACKET (closing corner bracket)
424 | ```
425 |
426 | ### Note 1: A "definition" is not a translation
427 | Unihan calls the English gloss the character's "definition", but that
428 | is meant in a very loose sense. CJK characters change meaning based
429 | upon the context they are used in. For example, most Chinese words are
430 | made of two characters, such as "蜂鳥", which means "hummingbird", but
431 | ugrep would shows it as:
432 |
433 | ```
434 | $ ugrep -c 蜂鳥
435 | 蜂 U+8702 bee, wasp, hornet ( M: fēng, C: fung1, J: HACHI, K: PONG, V: ong )
436 | 鳥 U+9CE5 bird; KangXi radical 196 ( M: niǎo, C: niu5, J: TORI, K: CO, V: điểu )
437 | ```
438 |
439 | ### Note 2: Not all characters have readings
440 |
441 | Unihan refers to this supplemental information — both the English
442 | gloss and the romanizations — as "readings". Readings are meant to be
443 | helpful, but are not normative and are only available for some
444 | characters.
445 |
446 | | | Count | Percent |
447 | |--------------------|-------:|--------:|
448 | | All CJK Characters | 93,858 | 100% |
449 | | Have any reading | 47,429 | 51% |
450 | | Mandarin Pinyin | 41,378 | 44% |
451 | | Cantonese Jyutping | 23,112 | 25% |
452 | | English definition | 21,076 | 23% |
453 | | Japanese Hepburn | 11,293 | 12% |
454 | | Korean Yale | 9,051 | 10% |
455 | | Vietnamese | 8,301 | 9% |
456 |
457 | #### Example of CJK with no Mandarin
458 |
459 | ```
460 | $ ugrep 2bac3
461 | 𫫃 U+2BAC3 (Cant.) sarcastic interrogative ( C: e1 )
462 | ```
463 | #### Example of CJK with no pronunciation
464 |
465 | ```
466 | $ ugrep 20015
467 | 𠀕 U+20015 Variant of U+4E99 亙
468 | ```
469 |
470 | #### Example of CJK with no English definition
471 |
472 | ```
473 | $ ugrep 20016
474 | 𠀖 U+20016 [CJK Unified Ideographs Extension B] ( V: khạng )
475 | ```
476 |
477 | #### Example of CJK with no readings whatsoever
478 |
479 | ```
480 | $ ugrep 2abcd
481 | 𪯍 U+2ABCD [CJK Unified Ideographs Extension C]
482 | ```
483 | Note that ugrep currently prints just the name of the block the
484 | character is in [within square brackets] if it has no better way to
485 | identify the character.
486 |
487 | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
488 |
489 |
490 |
491 | ## Boring Implementation notes
492 |
493 |
494 |
495 | This is a rewrite of b9's AWK ugrep into Python. While AWK makes more
496 | sense for what this program does (comparing fields based on regexps),
497 | a rewrite was necessary because GNU awk, while plenty powerful, uses
498 | `\y` for word edges instead of the standard `\b`. Gawk does this for
499 | backwards compatibility with historic AWK, but lacks a way to disable
500 | it for new scripts.
501 |
502 | Switching to Python did have the benefit of allowing more powerful
503 | Perlesque regexes (not that anyone has requested that).
504 |
505 | ### Why not use the unicodedata module?
506 |
507 | I do not use Python's `unicodedata` module because it is woefully
508 | insufficient. It allows one to search by character name only by
509 | specifying it fully and exactly: `unicodedata.lookup("ROTATED HEAVY
510 | BLACK HEART BULLET")`.
511 |
512 |
513 |
514 | ## Future Work
515 |
516 | ### Rename this project
517 |
518 | Although I believe this `ugrep` existed first, there is now another
519 | [ugrep](https://github.com/Genivia/ugrep) which is quite widely known
520 | — with good reason as it looks pretty nifty — which hasnothing to do
521 | with looking up Unicode characters. The 'U' appears to stand for
522 | _Ultra-fast_ as it is a very speedy `grep` with lots of bells and
523 | whistles.
524 |
525 | What shall this project's new name be? `ug` is also taken by the other
526 | ugrep. How about `ugre`? It's an ugly, ogreish name, but it's probably
527 | a safe bet nobody is going to use that name for something else.
528 |
529 | ### Maybe use Unihan_Readings.txt for grepping
530 |
531 | Currently if `Unihan_Readings.txt` is installed — which is the default if
532 | the user has done `apt install unicode-data`) — and the user requests a
533 | character that is not in UnicodeData.txt, then the Readings data is
534 | used to show information about the character. However, Unihan_Readings
535 | could be used in the future for searching for characters to show.
536 |
537 | Example data from Unihan_Readings for U+9B44 (魄):
538 |
539 | U+9B44 kCantonese bok3 paak3 tok3
540 | U+9B44 kDefinition vigor; body; dark part of moon
541 | U+9B44 kHangul 백:0N
542 | U+9B44 kHanyuPinlu pò(11)
543 | U+9B44 kHanyuPinyin 74431.090:pò,bó,tuò
544 | U+9B44 kJapaneseKun TAMASHII
545 | U+9B44 kJapaneseOn HAKU BAKU
546 | U+9B44 kKorean PAYK
547 | U+9B44 kMandarin pò
548 | U+9B44 kTGHZ2013 287.140:pò
549 | U+9B44 kTang *pæk
550 | U+9B44 kVietnamese phách
551 | U+9B44 kXHC1983 0084.110:bó 0887.020:pò 1175.020:tuò
552 |
553 | See [UAX #38: Unicode Han Database](https://www.unicode.org/reports/tr38/tr38-31.html).
554 |
555 | Two levels of Unihan support:
556 | 1. Show kDefinition if block name is *CJK Ideographs*
557 | 2. Search Unihan_Readings when searching for a word. Possible example:
558 | $ ugrep mononoke
559 | 魅 U+9B45 MONONOKE BAKEMONO SUDAMA (kind of forest demon, elf)
560 |
561 | Number 1 is finished and working, but number 2 may require a command
562 | line switch or some other way of enabling/disabling it as searching
563 | through the Readings file may be slow or cause other problems.
564 |
565 | ### Maybe use NamesList.txt
566 |
567 | It looks like
568 | [`NamesList.txt`](https://unicode.org/Public/UNIDATA/NamesList.txt)
569 | might be useful to also parse as it allows multiple aliases for a
570 | character. For example (from `grep -B1 [=%] NamesList.txt`):
571 |
572 | 0023 NUMBER SIGN
573 | = pound sign, hash, crosshatch, octothorpe
574 |
575 | 002E FULL STOP
576 | = period, dot, decimal point
577 | --
578 | 002F SOLIDUS
579 | = slash, virgule
580 |
581 | 1F70A ALCHEMICAL SYMBOL FOR VINEGAR
582 | = crucible; acid; distill; atrament; vitriol; red
583 | sulfur; borax; wine; alkali salt; mercurius vivus,
584 | quick silver
585 |
586 | I'm not sure how useful this will be (who is going to look up the
587 | number sign by searching on "octothorpe"), but it'd be nice to be able
588 | to at least show them as aliases.
589 |
590 | Also, NamesList.txt has a fascinating "cross reference" feature:
591 |
592 | 0021 EXCLAMATION MARK
593 | = factorial
594 | = bang
595 | x (inverted exclamation mark - 00A1)
596 | x (latin letter retroflex click - 01C3)
597 | x (double exclamation mark - 203C)
598 | x (interrobang - 203D)
599 | x (heavy exclamation mark ornament - 2762)
600 |
601 | How would one find the interrobang (‽) without such a cross reference?
602 |
603 | Note that the NamesList.txt file actually starts with a warning *not*
604 | to parse it as it says it is generated mechanically from
605 | UnicodeData.txt plus "manually created annotations". However, those
606 | annotations are what is interesting about the file (the aliases and
607 | cross references) and there appears to be no other official source of
608 | that data.
609 |
610 |
611 | ## Bugs, Misfeatures, and Workarounds
612 |
613 | * ugrep 3400 shows the text defined in UnicodeData.txt, which states
614 | that it is "". Now that ugrep can
615 | show ideograph definitions using Unihan_Readings.txt, we should
616 | (probably) replace any string in angle brackets with more useful info.
617 |
618 | * Brace expansion is confusing because of needing to be quoted from
619 | the shell. It is supported for ranges (not sequences), but is not
620 | currently documented because usage is tricky and the functionality
621 | is not actually that helpful. For example, the following works:
622 |
623 | ugrep {0..F}{0,4,8,C}00
624 |
625 | but is easier to understand using range expansion:
626 |
627 | ugrep 0..FFFF..400
628 |
629 | * Range expansion and a seemingly equivalent regular expression search
630 | will give different results.
631 |
632 | ugrep 0..FFFF..400 | wc -l
633 | 64
634 | ugrep U+[0-9A-F][048C]00 | wc -l
635 | 22
636 |
637 | This is because regexes currently only return valid code points from
638 | the UnicodeData.txt file, whereas range expansions can generate code
639 | points which are in regions not directly defined by Unicode. For
640 | example, the range from U+4E00 to U+9FEF is a block of CJK Ideographs.
641 | Both are useful: regexes are blazingly fast, while range expansions
642 | have more functionality.
643 |
644 | * [Note: The following is not a problem for people who are willing to
645 | use vector fonts (truetype, opentype, postscript) that may be
646 | antialiased. Xterm uses fontconfig just fine.]
647 |
648 |
649 |
650 | For bitmap fonts, Xterm (as of version 369) seems to be able to only
651 | use one font at a time, which means a single font must have all the
652 | glyphs you want shown. (Yes, you can have a second bitmap font for
653 | "wide" CJK, but that's still not enough.)
654 |
655 | The author (hackerb9) currently prefers using the Neep bitmap font
656 | like so in `~/.Xresources`:
657 |
658 | ! Neep looks nice, has good unicode coverage. Requires xfonts-jmk.
659 | xterm*vt100.font : *neep-medium-r-normal--20*10646*
660 | ! Neep lacks Asian characters
661 | xterm*vt100.wideFont : *fixed-medium-r-normal-ja-18*10646*
662 |
663 | Neep has two major downsides. 1. It is a bitmap font with only one
664 | size well implemented, so you can't zoom in or out. 2. It is limited
665 | to 65536 characters, which means it cannot show characters outside
666 | of Unicode's Basic Multilingual Plane, such as new emojis. Neep can
667 | be installed on Debian GNU/Linux systems with `apt install
668 | xfonts-jmk`.
669 |
670 |
671 | * Mlterm appears to have the same single font limitation as Xterm.
672 | Also, it right aligns text that has even a single character in a
673 | right-to-left alphabet, such as Arabic, so the output from ugrep
674 | will look a little funny.
675 |
676 |
677 |
678 | * Gnome-terminal uses `font-config`, so it has very nice Unicode
679 | support and can easily zoom in with Ctrl-+⃣ and Ctrl--⃣. Older
680 | versions had a bug where combining characters were combined with the
681 | following character instead of the previous, but this is now fixed.
682 |
683 | It does not support sixel graphics, so the -l option cannot show
684 | examples of the character in different fonts.
685 |
--------------------------------------------------------------------------------
/README.md.d/biangbiang.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/hackerb9/ugrep/67690b3be81a62cd872fec4f43d9b2f88a64488d/README.md.d/biangbiang.png
--------------------------------------------------------------------------------
/README.md.d/list-fonts-scale.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/hackerb9/ugrep/67690b3be81a62cd872fec4f43d9b2f88a64488d/README.md.d/list-fonts-scale.png
--------------------------------------------------------------------------------
/README.md.d/list-fonts.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/hackerb9/ugrep/67690b3be81a62cd872fec4f43d9b2f88a64488d/README.md.d/list-fonts.png
--------------------------------------------------------------------------------
/README.md.d/screenshot.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/hackerb9/ugrep/67690b3be81a62cd872fec4f43d9b2f88a64488d/README.md.d/screenshot.png
--------------------------------------------------------------------------------
/regression.sh:
--------------------------------------------------------------------------------
1 | #!/bin/bash
2 |
3 | # Some basic tests to make sure ugrep is working fine.
4 |
5 | errflag=""
6 | show-errs() {
7 | if [[ "$errflag" ]]; then
8 | echo
9 | echo "*** ERRORS WERE DETECTED ***"
10 | fi
11 | }
12 | trap show-errs EXIT
13 |
14 | runit() {
15 | # First argument is number of lines of output expected.
16 | # Remainder are the command to run and its arguments.
17 |
18 | expected="$1"
19 | shift
20 | command="$@"
21 | echo "Running '${command[@]}'"
22 | output=$(eval ${command[@]})
23 | lines=$(echo "$output" | wc -l)
24 | if [[ -z $output ]]; then lines=0; fi
25 | echo "$output" | expand | sed -e 's#^\(.\{,66\}\).*# \1#' | head -5
26 | if [[ $lines -gt 5 ]]; then echo " ..."; fi
27 | if [[ $lines -gt 10 ]]; then
28 | echo "$output" | expand | sed -e 's#^\(.\{,66\}\).*# \1#' | tail -5
29 | fi
30 | if [[ $lines == $expected ]]; then
31 | echo " OK ($lines)"
32 | return 0
33 | else
34 | s=$(plural $expected)
35 | echo -n $'\a'
36 | echo "*** ERROR: Expected $expected line$s of output, but got $lines"
37 | sleep 1
38 | errflag=yup
39 | return 1
40 | fi
41 | }
42 |
43 | plural() {
44 | [[ $1 != 1 ]] && echo -n "s"
45 | }
46 |
47 | # Literal character
48 | runit 1 ./ugrep $'\U273F'
49 |
50 | # Hex code point
51 | runit 1 ./ugrep 273a
52 |
53 | # Character name
54 | runit 81 ./ugrep TETRAGRAM
55 |
56 | # Multiple args are joined with .* between
57 | runit 1 ./ugrep math left tort
58 |
59 | # Ambiguous arg searches for both hex and charname
60 | runit 3 ./ugrep feed \| grep -i form
61 |
62 | # Unambiguous arg searches only hex
63 | runit 1 ./ugrep u+feed
64 |
65 | # Regex will not match codepoints not in UnicodeData.txt, such as CJK blocks.
66 | runit 6 ./ugrep U+'.000'
67 |
68 | # But codepoints specified explicitly are shown anyway
69 | runit 1 ./ugrep U+8000
70 |
71 | # Ranges are specified with two dots. Leading hexits are implied: f -> 240f.
72 | runit 16 ./ugrep 2400..f
73 |
74 | # "start..end..increment" (includes CJK blocks)
75 | runit 16 ./ugrep 0..FFFF..1000
76 |
77 | # Regex character ranges are faster, but no CJK.
78 | runit 256 ./ugrep U+'[0-9A-F]{2}'
79 | runit 0 ./ugrep U+'8[0-9A-F]{3}'
80 |
81 | # -w to match only whole words
82 | runit 4 ./ugrep -w greek.* pi
83 | # 'grep -ivw pi' strips out PI but would allow ethioPIc through
84 | runit 0 ./ugrep -w pi \| grep -ivw pi
85 |
86 | # -c to show each characters in a string
87 | runit 5 ./ugrep -c ASCII
88 | runit 4 ./ugrep -c $'x\U0300\U0301\U0302'
89 | runit 21 ./ugrep -c 「⿺辶⿳穴⿰月⿰⿲⿱幺長⿱言馬⿱幺長刂心」
90 |
91 | # Search aliases as well as character names
92 | runit 3 ./ugrep backslash \| grep \"REVERSE SOLIDUS\"
93 |
--------------------------------------------------------------------------------
/ugrep:
--------------------------------------------------------------------------------
1 | #!/usr/bin/python3
2 | # ugrep: find unicode characters based on their names or number.
3 | # Essentially grep for the Unicode table.
4 |
5 | # PREREQUISITE:
6 | #
7 | # * You'll need a copy of UnicodeData.txt installed.
8 | # On Debian GNU/Linux, this can be done by `apt install unicode-data`.
9 | # Or, you can download it by hand from the Unicode Consortium and place
10 | # it in `~/.local/share/unicode/UnicodeData.txt`.
11 |
12 | # ADDED FUNCTIONALITY:
13 | #
14 | # * If you have FontConfig installed, then you can use -l to see which
15 | # fonts contain a certain character.
16 | #
17 | # * If you have ImageMagick installed and your terminal can display
18 | # sixels, then -l will also show a rendering of the character in
19 | # each font.
20 | #
21 | # * If you have Unihan_Readings.txt then CJK ideographs will show definitions.
22 | #
23 |
24 |
25 | # Fun things to try:
26 |
27 | # ugrep alchemical
28 | # ugrep ornament
29 | # ugrep bullet
30 | # ugrep 'vine|bud'
31 | # ugrep vai
32 | # ugrep heavy
33 | # ugrep drawing
34 | # ugrep 2300..ff
35 |
36 | # GPL ≥3 (see LICENSE file)
37 | # B9 September 2018 – October 2021
38 |
39 | debug=True
40 | debug=False
41 |
42 | from re import *
43 | from sys import argv, stderr
44 | from os.path import expanduser, expandvars, basename
45 | from os import isatty
46 | from subprocess import getoutput, run
47 | import argparse
48 | from math import floor, copysign
49 | from collections import defaultdict
50 |
51 | from pprint import pprint as pp
52 |
53 | try:
54 | from posix import write
55 | import select
56 | import termios
57 | import tty
58 | except ImportError:
59 | print("Sorry, but your system isn't POSIX compatible", file=stderr)
60 | exit( 1 )
61 |
62 | # Allow fancier terminfo formatting, if python3-blessed is installed.
63 | try:
64 | import blessed
65 | curses = blessed.terminal.Terminal()
66 | except ImportError:
67 | pass
68 |
69 |
70 |
71 | def usage():
72 | print("""\
73 | ugrep: find unicode characters based on their names or codepoints
74 |
75 | Usage:
76 | ugrep [-wlx] [-L n] | |
77 | ugrep -c
78 |
79 | -w: Match only whole words: ugrep -w pi
80 | -c: Show each character in a string: ugrep -c "(ﾟ∀ﾟ)"
81 | -l: List installed fonts that include matching characters.
82 | -ll: Also includes style variants (bold, italic,...).
83 | -L scale: Scale font examples from 2 to 8x.
84 | -m: Only list monospace fonts.
85 | -x, -x2, -x3: Show usage examples.
86 |
87 | A single character; implies -c: ugrep ☙
88 |
89 | One or more hexadecimal numbers: ugrep U+1F639
90 | Ranges are allowed with two dots: ugrep 23b0..f
91 | Optional range increment: ugrep 0..ffff..1000
92 |
93 | A Unicode name (as a regex), e.g.: ugrep alchemical
94 |
95 | One or more characters. e.g.: ugrep -c "( ͡° ͜ʖ ͡°)"
96 | """, end='' )
97 |
98 | def examples():
99 | print("""\
100 | # You can search by character name.
101 | $ ugrep heart
102 | ☙ U+2619 REVERSED ROTATED FLORAL HEART BULLET
103 | ❣ U+2763 HEAVY HEART EXCLAMATION MARK ORNAMENT
104 | ❤ U+2764 HEAVY BLACK HEART
105 | [ ... examples truncated for brevity ... ]
106 | 😻 U+1F63B SMILING CAT FACE WITH HEART-SHAPED EYES
107 |
108 | # Or, you can search by pasting in a specific character.
109 | $ ugrep ✿
110 | ✿ U+273F BLACK FLORETTE
111 |
112 | # Or, you can search by code point.
113 | $ ugrep 273a
114 | ✺ U+273A SIXTEEN POINTED ASTERISK
115 |
116 | # By default, words match anywhere.
117 | $ ugrep clos brac # Equivalent to "clos.*brac"
118 | ] U+005D RIGHT SQUARE BRACKET (closing square bracket)
119 | } U+007D RIGHT CURLY BRACKET (closing curly bracket)
120 | 〉 U+3009 RIGHT ANGLE BRACKET (closing angle bracket)
121 | """
122 | )
123 |
124 | def examples2():
125 | print("""\
126 | # Use -w to match only whole words.
127 | $ ugrep -w "R" # Equivalent to "\\bR\\b"
128 | R U+0052 LATIN CAPITAL LETTER R
129 | ℛ U+211B SCRIPT CAPITAL R (Script r)
130 | ℜ U+211C BLACK-LETTER CAPITAL R (Black-letter r)
131 | ℝ U+211D DOUBLE-STRUCK CAPITAL R (Double-struck r)
132 |
133 | # Use -c to display info for each character in a string.
134 | $ ugrep -c "ᕕ( ᐛ )ᕗ"
135 | ᕕ U+1555 CANADIAN SYLLABICS FI
136 | ( U+0028 LEFT PARENTHESIS (opening parenthesis)
137 | ᐛ U+141B CANADIAN SYLLABICS NASKAPI WAA
138 | ) U+0029 RIGHT PARENTHESIS (closing parenthesis)
139 | ᕗ U+1557 CANADIAN SYLLABICS FO
140 |
141 | # Use 'U+' prefix to force a search for hexadecimal
142 | $ ugrep u+ee # "0xEE" also works
143 | î U+00EE LATIN SMALL LETTER I WITH CIRCUMFLEX
144 | """
145 | )
146 |
147 | def examples3():
148 | print("""\
149 | # Regex ^ and $ work, mostly
150 | $ ugrep ^x
151 | ⊻ U+22BB XOR
152 | ⌧ U+2327 X IN A RECTANGLE BOX (clear key)
153 |
154 | # ugrep finds aliases (and puts them in parens)
155 | $ ugrep backslash
156 | \\ U+005C REVERSE SOLIDUS (backslash)
157 |
158 | # Use -l to see which fonts are installed that can render a certain character
159 | $ ugrep -l math left tortoise
160 | ⟬ U+27EC MATHEMATICAL LEFT WHITE TORTOISE SHELL BRACKET
161 | Symbola
162 | Unifont
163 |
164 | # Use -L n to scale the font examples n-times larger; implies -l
165 | $ ugrep -L2 antimony
166 | [ Cannot show sixel examples here ]
167 |
168 | # Show every single Unicode character.
169 | $ ugrep '.?' | less
170 | """
171 | )
172 |
173 |
174 |
175 |
176 | # Main
177 | def main():
178 | global argv, args, debug
179 |
180 | # Load the Unicode Data into the global ucd variable.
181 | loaducd()
182 |
183 | if len(argv) == 1:
184 | usage() # ArgParse's usage is ugly, use our own.
185 | exit(1)
186 |
187 | if argv[1]=='-h' or argv[1]=='--help':
188 | usage()
189 | exit(0)
190 |
191 | if argv[1].startswith('-x') or argv[1].startswith('--example'):
192 | try: x=int(argv[1][-1])
193 | except ValueError: x=0
194 | if x == 0:
195 | examples()
196 | if isatty(1):
197 | print("Try 'ugrep --examples=2' to see more usage examples.")
198 | else:
199 | examples2()
200 | examples3()
201 | if x == 1:
202 | examples()
203 | print("Try 'ugrep --examples=2' to see more usage examples.")
204 | if x == 2:
205 | examples2()
206 | print("Try 'ugrep --examples=3' to see even more usage examples.")
207 | if x == 3:
208 | examples3()
209 | print("Try 'ugrep -x | less' to see all usage examples in one page.")
210 | exit(0)
211 |
212 | parser = argparse.ArgumentParser(
213 | description='find Unicode characters by name, number, or example.')
214 | parser.add_argument(
215 | 'characters', metavar='Character Name | Codepoint | Character',
216 | nargs='+', type=str,
217 | help='which character to show specified by name, codepoint or literal example')
218 | parser.add_argument('-w', '--word-regexp', action="store_true",
219 | help='matches only whole words, e.g., ugrep -w pi')
220 | parser.add_argument('-c', '--char-by-char', action="store_true",
221 | help='show each character in a string, e.g., ugrep -c "(ﾟ∀ﾟ)"')
222 | parser.add_argument('-l', '--list-fonts', action="store_true",
223 | help='list fonts that include a glyph for a given character')
224 | parser.add_argument('-m', '--monospace-fonts-only', action="store_true",
225 | help='when listing fonts, only show fixed-width fonts')
226 | parser.add_argument('-ll', '--list-fonts-long', action="store_true",
227 | help='list all fonts, including variants like italic, bold, etc.')
228 | # XXX Argh. argparse gives an error on 'ugrep -L foo' but works for 'ugrep foo -L'.
229 | parser.add_argument('-L', '--list-fonts-scale', metavar='scaling', dest='fonts_scale',
230 | nargs='?', const=2, default=None, type=float,
231 | help='list fonts, scaling the characters to be from 2 to 8x larger')
232 | parser.add_argument('--never-double-text', action="store_true",
233 | help="Don't use double-size terminal text even when font scale >= 2")
234 | parser.add_argument('--pipe-sixels', action="store_true",
235 | help="Allow sixel escape sequences when redirecting stdout")
236 | parser.add_argument('-x', '--examples', action="store_true",
237 | help='show example usage')
238 | parser.add_argument('--debug', action="store_true", help='internal debugging')
239 |
240 | args = parser.parse_args()
241 |
242 | # -m restricts listed fonts to monospace (implies -l)
243 | if args.monospace_fonts_only:
244 | args.list_fonts = True
245 |
246 | # -ll (list_fonts_long) same as -l but shows all font variants
247 | if args.list_fonts_long:
248 | args.list_fonts = True
249 |
250 | # -l (list fonts), defaults to a scaling of 1x
251 | if args.list_fonts and not args.fonts_scale:
252 | args.fonts_scale = 1
253 |
254 | # -L is like -l (list fonts), but with larger images.
255 | if args.fonts_scale:
256 | args.list_fonts = True;
257 |
258 | if args.debug: debug=True
259 |
260 | # If -w, then wrap each argument with \b (regexp word breaks)
261 | if args.word_regexp:
262 | args.characters = [ "\\b"+s+"\\b" for s in args.characters ]
263 |
264 | # Is it a single character? (ugrep 'A') Then only show that one character.
265 | if len(args.characters)==1 and len(args.characters[0])==1:
266 | args.char_by_char = True # Pretend -c flag was given.
267 |
268 | # Is it -c followed by a string of characters?
269 | if args.char_by_char:
270 | # For each string s, look up each character in the string.
271 | for s in args.characters:
272 | for i in range(len(s)):
273 | for j in range(0,len(s[i])):
274 | arg = hex(ord(s[i][j]))
275 | if not showcodepoint( arg ):
276 | # Not Unicode, but could be Chinese, Japanese, Korean
277 | if not showcjk( arg[2:] ):
278 | # Not CJK, so print block name
279 | showcharacterblock( arg )
280 |
281 | if s is not args.characters[-1]:
282 | print("") # Space between words, if multiple args
283 | exit(0)
284 |
285 | # Do range expansion on argv. (e.g., 2301..f)
286 | args.characters = rangeexpansion(args.characters)
287 |
288 | # First, try looking up each arg individually as hexadecimal codepoints.
289 | # Loop to allow input like "ugrep 23b{0..9}" (NB: slower than 23b[0-9])
290 | # Must be hexadecimal.
291 | for arg in args.characters:
292 | if arg.strip("Uu+0123456789ABCDEFXabcdefx[]-.*?^$(){,}"):
293 | debugprint("Stopped looking for hex/regex arguments at", arg)
294 | break
295 | # Show matches in UnicodeData.txt for each hex codepoint
296 | if not showcodepoint(arg):
297 | # Hmm... it wasn't in UnicodeData. Maybe CJK?
298 | if not showcjk(arg):
299 | # Nope, print the character anyway and its Unicode block.
300 | showcharacterblock(arg)
301 | # If -l, show installed fonts which contain the glyph.
302 | if args.list_fonts: show_fonts(arg)
303 |
304 | # Also look up the entire argv[] as a character name or alias.
305 | # Must NOT begin with U+ or 0x.
306 | if not match("(?i)^U+|^0x", args.characters[0]):
307 | s=makeregexcharname(args.characters)
308 | if printmatches(s):
309 | debugprint("argv[] array matched as a whole.")
310 |
311 | def showonecharacter(c):
312 | s=makeregexeither(c)
313 | if debug: compileit(s)
314 | return printmatches(s)
315 |
316 | def showcodepoint(c):
317 | s=makeregexcodepoint(c)
318 | if debug: compileit(s)
319 | return printmatches(s)
320 |
321 | def showcjk(x):
322 | """
323 | Prints a line for a codepoint x (hexadecimal string) using the data
324 | from Unihan_Readings.txt. If that file is not available or if the
325 | codepoint is not in a CJK region, then return False.
326 | """
327 |
328 | x=x.upper().lstrip("Uu+0Xx") # "u+face" -> "FACE"
329 | try:
330 | codepoint=cp2int(x)
331 | except ValueError:
332 | return False
333 |
334 | c=chr(codepoint)
335 | if not isunihan(codepoint):
336 | return False
337 |
338 | # Get the dictionary of "readings" for character x
339 | r = lookup_readings(x)
340 | try:
341 | definition = r["kDefinition"]
342 | except TypeError:
343 | return False
344 | except KeyError:
345 | definition = '[' + getblock(cp2int(x)) + ']'
346 |
347 | transliteration=[]
348 | shortlang = {
349 | "kMandarin": "M",
350 | "kCantonese": "C",
351 | "kJapaneseKun": "J",
352 | "kKorean": "K",
353 | "kVietnamese": "V"
354 | }
355 | for lang in (shortlang.keys()):
356 | try:
357 | transliteration.append(shortlang[lang] + ": " + r[lang] + ",")
358 | except KeyError:
359 | pass
360 |
361 | if transliteration:
362 | transliteration[-1] = transliteration[-1].rstrip(",")
363 |
364 | print(" ", end='') # Indent for MacOS
365 | print(c, end='\t')
366 | print ("U+" + x, end='\t')
367 | print(definition, end='')
368 | if transliteration:
369 | print(" (", *transliteration, ")", end='')
370 | print("", flush=True)
371 |
372 | return True
373 |
374 | def isunihan(x):
375 | """
376 | Given a codepoint x, e.g. "U+8000", return True if x is within
377 | a CJK Unified Ideographs block. Such characters are not defined in
378 | UnicodeData.txt but can be looked up in Unihan_Readings.txt.
379 | See lookup_readings().
380 | """
381 |
382 | try:
383 | codepoint=cp2int(x) # Convert hex str x to integer
384 | except ValueError:
385 | return False # Could not convert
386 |
387 | if getblock(codepoint).find('Ideographs') != -1:
388 | return True
389 | else:
390 | return False
391 |
392 |
393 | def lookup_readings(cp):
394 | """
395 | Given a hexadecimal string, cp ("2BAC3"), return a dictionary with
396 | an entry for each line in Unihan_Readings.txt. Example:
397 |
398 | For input "2BAC3", the result is:
399 | {'kDefinition': '(CANT.) SARCASTIC INTERROGATIVE',
400 | 'kCantonese': 'e1' }
401 |
402 | For input "9EBB", note that some of the results include embedded UTF-8:
403 | { 'kCantonese': 'maa4',
404 | 'kDefinition': 'hemp, jute, flax; sesame',
405 | 'kHangul': '\xeb\xa7\x88:0E',
406 | 'kHanyuPinlu': 'm\xc3\xa1(159) ma(11)',
407 | 'kHanyuPinyin': '74723.010:m\xc3\xa1,m\xc4\x81',
408 | 'kJapaneseKun': 'ASA',
409 | 'kJapaneseOn': 'MA BA',
410 | 'kKorean': 'MA',
411 | 'kMandarin': 'm\xc3\xa1',
412 | 'kTGHZ2013': '240.080:m\xc3\xa1',
413 | 'kTang': '*ma',
414 | 'kVietnamese': 'ma',
415 | 'kXHC1983': '0753.040:m\xc4\x81 0753.110:m\xc3\xa1'
416 | }
417 | """
418 |
419 | # As of 2021 Unihan includes these keys, sorted by frequency:
420 | # cat Unihan_Readings.txt | cut -f2 | sort | uniq -c | sort -n
421 | ############################################################
422 | # 3800 kHanyuPinlu Mandarin Pinyin from XDHYPLCD #
423 | # 3812 kTang Tang dynasty pronunciation #
424 | # 8106 kTGHZ2013 Mandarin Pinyin from TGHZ #
425 | # 8301 kVietnamese Vietnamese transliteration #
426 | # 8414 kHangul Korean pronunciation in Hangul #
427 | # 9051 kKorean Korean in Yale romanization #
428 | # 11,018 kXHC1983 Mandarin Pinyin from XHC #
429 | # 11,293 kJapaneseKun Japanese Hepburn romanization #
430 | # 13,176 kJapaneseOn Sino-Japanese pronunciation #
431 | # 21,076 kDefinition A short gloss in English #
432 | # 23,112 kCantonese Cantonese Jyutping transliteration #
433 | # 34,131 kHanyuPinyin Mandarin Pinyin from HDZ #
434 | # 41,378 kMandarin Mandarin Pinyin (most customary) #
435 | ############################################################
436 | # For more details, see Unicode Standard Annex #38
437 | # https://www.unicode.org/reports/tr38/tr38-31.html
438 |
439 | # Initialize 'readings' global variable
440 | if not loadreadings():
441 | return False
442 |
443 | cp = cp.lstrip("Uu+0Xx")
444 | results={}
445 | # Example entry from Unihan_Readings.txt:
446 | # U+2BAC3 kCantonese e1
447 | # U+2BAC3 kDefinition (Cant.) sarcastic interrogative
448 | # Regex to match a line in Unihan_Readings.txt
449 | r=r"\nU\+" + cp + "\t+" + "(?P[^\t]+)" + "\t+" + "(?P[^\t\n]+)"
450 | if type(readings) is bytes: r=bytes(r, 'utf-8')
451 | matches=findall(r, readings)
452 | if matches == None:
453 | debugprint("lookup_readings did not find", r)
454 | return None
455 | debugprint("lookup_readings found ", matches)
456 | for m in matches:
457 | if type(m[0]) is bytes: # bz lib's read() returns 'bytes'
458 | m0=str(m[0], 'UTF-8')
459 | m1=str(m[1], 'UTF-8')
460 | else:
461 | m0=m[0] # normal read() returns 'str'
462 | m1=m[1]
463 | results[m0] = m1 # Build dictionary.
464 | return results
465 |
466 | def lookup_namealias(cp):
467 | """
468 | Given a hexadecimal string, cp ("0004"), return a list with
469 | an element for each line in NameAliases.txt. Example:
470 | For input "0004", the result is: { 'end of transmission', 'EOT' }
471 |
472 | Returns {} if no alias was found or if NameAliases.txt does not exist.
473 | """
474 | # Initialize 'namealiases' global variable
475 | if not loadnamealiases():
476 | return {}
477 |
478 | cp = cp.lstrip("Uu+0Xx")
479 | results=[]
480 | # Example entry from NameAliases.txt:
481 | # 0004;END OF TRANSMISSION;control
482 | # 0004;EOT;abbreviation
483 | #
484 | # Regex to match a line in NameAliases.txt
485 | r=r"(?m)^%04X;(?P[^;\n]+);(?P[^;\n]+)$" % (cp2int(cp))
486 | matches=finditer(r, namealiases)
487 | if matches == None:
488 | debugprint("lookup_namealias did not find", r)
489 | return {}
490 |
491 | for m in matches:
492 | g=m.groupdict()
493 | debugprint("lookup_namealias", g)
494 | if g['type'] == 'abbreviation':
495 | # Allow abbreviations to also be uppercase
496 | results.append(g['alias']) # "EOT"
497 | elif g['type'] == 'correction':
498 | # Corrections to the official name are noteworthy.
499 | c=g['alias'].title() # "Weierstrass Elliptic Function"
500 | try: c=curses.red(c)
501 | except NameError: pass
502 | results.append(c)
503 | else:
504 | # For clarity, usually only names are in all caps.
505 | results.append(g['alias'].lower()) # "end of transmission"
506 |
507 |
508 | return results
509 |
510 | def showcharacterblock(codepoint):
511 | """
512 | Prints a line showing which character block a hex number belongs in.
513 | If showonecharacter() fails, this can be used to at least print something,
514 | """
515 |
516 | try:
517 | x=cp2int(codepoint) # Remove U+ or 0x prefix and return integer
518 | except ValueError:
519 | return
520 |
521 | block=getblock(x)
522 | if block:
523 | try: c=chr(x)
524 | except ValueError: return
525 | print(" ", end='') # Indent for MacOS
526 | print(c, end='')
527 | print('\t' + int2cp(x) + '\t', end='')
528 | print('Block: [' + block + ']')
529 |
530 |
531 | def cp2int(cp):
532 | """
533 | Given a Unicode codepoint as a string (e.g., U+1D0DE)
534 | convert the hexadecimal number to an integer and return it.
535 | Throws ValueError if cp is not convertible to an integer.
536 | """
537 |
538 | if type(cp) is int: return cp
539 | if cp is None: return 0
540 | cp = cp.lstrip("Uu+0Xx")
541 | if cp == '': return 0
542 | return( int(cp.lstrip("Uu+0Xx"), 16) )
543 |
544 | def int2cp(x):
545 | "Given an int, return it as a hexadecimal string in the format U+1D0DE)."
546 | return( "U+%04X" % (x) )
547 |
548 | def getblock(x):
549 | """
550 | Given an integer representing a Unicode code point, return its
551 | block. E.g, U+0041 -> Basic Latin, U+1F675 -> Ornamental Dingbats.
552 | This is useful to identify characters that are not officially part
553 | of Unicode, for example all the CJK Ideographs from 4E00 to 9FFF.
554 |
555 | An empty string is returned if the integer is not within any
556 | Unicode block, e.g., ugrep U+80000.
557 | """
558 |
559 | # Data from Unicode 2021 Blocks-14.0.0.txt
560 | blockstxt="""\
561 | 0000..007F; Basic Latin
562 | 0080..00FF; Latin-1 Supplement
563 | 0100..017F; Latin Extended-A
564 | 0180..024F; Latin Extended-B
565 | 0250..02AF; IPA Extensions
566 | 02B0..02FF; Spacing Modifier Letters
567 | 0300..036F; Combining Diacritical Marks
568 | 0370..03FF; Greek and Coptic
569 | 0400..04FF; Cyrillic
570 | 0500..052F; Cyrillic Supplement
571 | 0530..058F; Armenian
572 | 0590..05FF; Hebrew
573 | 0600..06FF; Arabic
574 | 0700..074F; Syriac
575 | 0750..077F; Arabic Supplement
576 | 0780..07BF; Thaana
577 | 07C0..07FF; NKo
578 | 0800..083F; Samaritan
579 | 0840..085F; Mandaic
580 | 0860..086F; Syriac Supplement
581 | 0870..089F; Arabic Extended-B
582 | 08A0..08FF; Arabic Extended-A
583 | 0900..097F; Devanagari
584 | 0980..09FF; Bengali
585 | 0A00..0A7F; Gurmukhi
586 | 0A80..0AFF; Gujarati
587 | 0B00..0B7F; Oriya
588 | 0B80..0BFF; Tamil
589 | 0C00..0C7F; Telugu
590 | 0C80..0CFF; Kannada
591 | 0D00..0D7F; Malayalam
592 | 0D80..0DFF; Sinhala
593 | 0E00..0E7F; Thai
594 | 0E80..0EFF; Lao
595 | 0F00..0FFF; Tibetan
596 | 1000..109F; Myanmar
597 | 10A0..10FF; Georgian
598 | 1100..11FF; Hangul Jamo
599 | 1200..137F; Ethiopic
600 | 1380..139F; Ethiopic Supplement
601 | 13A0..13FF; Cherokee
602 | 1400..167F; Unified Canadian Aboriginal Syllabics
603 | 1680..169F; Ogham
604 | 16A0..16FF; Runic
605 | 1700..171F; Tagalog
606 | 1720..173F; Hanunoo
607 | 1740..175F; Buhid
608 | 1760..177F; Tagbanwa
609 | 1780..17FF; Khmer
610 | 1800..18AF; Mongolian
611 | 18B0..18FF; Unified Canadian Aboriginal Syllabics Extended
612 | 1900..194F; Limbu
613 | 1950..197F; Tai Le
614 | 1980..19DF; New Tai Lue
615 | 19E0..19FF; Khmer Symbols
616 | 1A00..1A1F; Buginese
617 | 1A20..1AAF; Tai Tham
618 | 1AB0..1AFF; Combining Diacritical Marks Extended
619 | 1B00..1B7F; Balinese
620 | 1B80..1BBF; Sundanese
621 | 1BC0..1BFF; Batak
622 | 1C00..1C4F; Lepcha
623 | 1C50..1C7F; Ol Chiki
624 | 1C80..1C8F; Cyrillic Extended-C
625 | 1C90..1CBF; Georgian Extended
626 | 1CC0..1CCF; Sundanese Supplement
627 | 1CD0..1CFF; Vedic Extensions
628 | 1D00..1D7F; Phonetic Extensions
629 | 1D80..1DBF; Phonetic Extensions Supplement
630 | 1DC0..1DFF; Combining Diacritical Marks Supplement
631 | 1E00..1EFF; Latin Extended Additional
632 | 1F00..1FFF; Greek Extended
633 | 2000..206F; General Punctuation
634 | 2070..209F; Superscripts and Subscripts
635 | 20A0..20CF; Currency Symbols
636 | 20D0..20FF; Combining Diacritical Marks for Symbols
637 | 2100..214F; Letterlike Symbols
638 | 2150..218F; Number Forms
639 | 2190..21FF; Arrows
640 | 2200..22FF; Mathematical Operators
641 | 2300..23FF; Miscellaneous Technical
642 | 2400..243F; Control Pictures
643 | 2440..245F; Optical Character Recognition
644 | 2460..24FF; Enclosed Alphanumerics
645 | 2500..257F; Box Drawing
646 | 2580..259F; Block Elements
647 | 25A0..25FF; Geometric Shapes
648 | 2600..26FF; Miscellaneous Symbols
649 | 2700..27BF; Dingbats
650 | 27C0..27EF; Miscellaneous Mathematical Symbols-A
651 | 27F0..27FF; Supplemental Arrows-A
652 | 2800..28FF; Braille Patterns
653 | 2900..297F; Supplemental Arrows-B
654 | 2980..29FF; Miscellaneous Mathematical Symbols-B
655 | 2A00..2AFF; Supplemental Mathematical Operators
656 | 2B00..2BFF; Miscellaneous Symbols and Arrows
657 | 2C00..2C5F; Glagolitic
658 | 2C60..2C7F; Latin Extended-C
659 | 2C80..2CFF; Coptic
660 | 2D00..2D2F; Georgian Supplement
661 | 2D30..2D7F; Tifinagh
662 | 2D80..2DDF; Ethiopic Extended
663 | 2DE0..2DFF; Cyrillic Extended-A
664 | 2E00..2E7F; Supplemental Punctuation
665 | 2E80..2EFF; CJK Radicals Supplement
666 | 2F00..2FDF; Kangxi Radicals
667 | 2FF0..2FFF; Ideographic Description Characters
668 | 3000..303F; CJK Symbols and Punctuation
669 | 3040..309F; Hiragana
670 | 30A0..30FF; Katakana
671 | 3100..312F; Bopomofo
672 | 3130..318F; Hangul Compatibility Jamo
673 | 3190..319F; Kanbun
674 | 31A0..31BF; Bopomofo Extended
675 | 31C0..31EF; CJK Strokes
676 | 31F0..31FF; Katakana Phonetic Extensions
677 | 3200..32FF; Enclosed CJK Letters and Months
678 | 3300..33FF; CJK Compatibility
679 | 3400..4DBF; CJK Unified Ideographs Extension A
680 | 4DC0..4DFF; Yijing Hexagram Symbols
681 | 4E00..9FFF; CJK Unified Ideographs
682 | A000..A48F; Yi Syllables
683 | A490..A4CF; Yi Radicals
684 | A4D0..A4FF; Lisu
685 | A500..A63F; Vai
686 | A640..A69F; Cyrillic Extended-B
687 | A6A0..A6FF; Bamum
688 | A700..A71F; Modifier Tone Letters
689 | A720..A7FF; Latin Extended-D
690 | A800..A82F; Syloti Nagri
691 | A830..A83F; Common Indic Number Forms
692 | A840..A87F; Phags-pa
693 | A880..A8DF; Saurashtra
694 | A8E0..A8FF; Devanagari Extended
695 | A900..A92F; Kayah Li
696 | A930..A95F; Rejang
697 | A960..A97F; Hangul Jamo Extended-A
698 | A980..A9DF; Javanese
699 | A9E0..A9FF; Myanmar Extended-B
700 | AA00..AA5F; Cham
701 | AA60..AA7F; Myanmar Extended-A
702 | AA80..AADF; Tai Viet
703 | AAE0..AAFF; Meetei Mayek Extensions
704 | AB00..AB2F; Ethiopic Extended-A
705 | AB30..AB6F; Latin Extended-E
706 | AB70..ABBF; Cherokee Supplement
707 | ABC0..ABFF; Meetei Mayek
708 | AC00..D7AF; Hangul Syllables
709 | D7B0..D7FF; Hangul Jamo Extended-B
710 | D800..DB7F; High Surrogates
711 | DB80..DBFF; High Private Use Surrogates
712 | DC00..DFFF; Low Surrogates
713 | E000..F8FF; Private Use Area
714 | F900..FAFF; CJK Compatibility Ideographs
715 | FB00..FB4F; Alphabetic Presentation Forms
716 | FB50..FDFF; Arabic Presentation Forms-A
717 | FE00..FE0F; Variation Selectors
718 | FE10..FE1F; Vertical Forms
719 | FE20..FE2F; Combining Half Marks
720 | FE30..FE4F; CJK Compatibility Forms
721 | FE50..FE6F; Small Form Variants
722 | FE70..FEFF; Arabic Presentation Forms-B
723 | FF00..FFEF; Halfwidth and Fullwidth Forms
724 | FFF0..FFFF; Specials
725 | 10000..1007F; Linear B Syllabary
726 | 10080..100FF; Linear B Ideograms
727 | 10100..1013F; Aegean Numbers
728 | 10140..1018F; Ancient Greek Numbers
729 | 10190..101CF; Ancient Symbols
730 | 101D0..101FF; Phaistos Disc
731 | 10280..1029F; Lycian
732 | 102A0..102DF; Carian
733 | 102E0..102FF; Coptic Epact Numbers
734 | 10300..1032F; Old Italic
735 | 10330..1034F; Gothic
736 | 10350..1037F; Old Permic
737 | 10380..1039F; Ugaritic
738 | 103A0..103DF; Old Persian
739 | 10400..1044F; Deseret
740 | 10450..1047F; Shavian
741 | 10480..104AF; Osmanya
742 | 104B0..104FF; Osage
743 | 10500..1052F; Elbasan
744 | 10530..1056F; Caucasian Albanian
745 | 10570..105BF; Vithkuqi
746 | 10600..1077F; Linear A
747 | 10780..107BF; Latin Extended-F
748 | 10800..1083F; Cypriot Syllabary
749 | 10840..1085F; Imperial Aramaic
750 | 10860..1087F; Palmyrene
751 | 10880..108AF; Nabataean
752 | 108E0..108FF; Hatran
753 | 10900..1091F; Phoenician
754 | 10920..1093F; Lydian
755 | 10980..1099F; Meroitic Hieroglyphs
756 | 109A0..109FF; Meroitic Cursive
757 | 10A00..10A5F; Kharoshthi
758 | 10A60..10A7F; Old South Arabian
759 | 10A80..10A9F; Old North Arabian
760 | 10AC0..10AFF; Manichaean
761 | 10B00..10B3F; Avestan
762 | 10B40..10B5F; Inscriptional Parthian
763 | 10B60..10B7F; Inscriptional Pahlavi
764 | 10B80..10BAF; Psalter Pahlavi
765 | 10C00..10C4F; Old Turkic
766 | 10C80..10CFF; Old Hungarian
767 | 10D00..10D3F; Hanifi Rohingya
768 | 10E60..10E7F; Rumi Numeral Symbols
769 | 10E80..10EBF; Yezidi
770 | 10F00..10F2F; Old Sogdian
771 | 10F30..10F6F; Sogdian
772 | 10F70..10FAF; Old Uyghur
773 | 10FB0..10FDF; Chorasmian
774 | 10FE0..10FFF; Elymaic
775 | 11000..1107F; Brahmi
776 | 11080..110CF; Kaithi
777 | 110D0..110FF; Sora Sompeng
778 | 11100..1114F; Chakma
779 | 11150..1117F; Mahajani
780 | 11180..111DF; Sharada
781 | 111E0..111FF; Sinhala Archaic Numbers
782 | 11200..1124F; Khojki
783 | 11280..112AF; Multani
784 | 112B0..112FF; Khudawadi
785 | 11300..1137F; Grantha
786 | 11400..1147F; Newa
787 | 11480..114DF; Tirhuta
788 | 11580..115FF; Siddham
789 | 11600..1165F; Modi
790 | 11660..1167F; Mongolian Supplement
791 | 11680..116CF; Takri
792 | 11700..1174F; Ahom
793 | 11800..1184F; Dogra
794 | 118A0..118FF; Warang Citi
795 | 11900..1195F; Dives Akuru
796 | 119A0..119FF; Nandinagari
797 | 11A00..11A4F; Zanabazar Square
798 | 11A50..11AAF; Soyombo
799 | 11AB0..11ABF; Unified Canadian Aboriginal Syllabics Extended-A
800 | 11AC0..11AFF; Pau Cin Hau
801 | 11C00..11C6F; Bhaiksuki
802 | 11C70..11CBF; Marchen
803 | 11D00..11D5F; Masaram Gondi
804 | 11D60..11DAF; Gunjala Gondi
805 | 11EE0..11EFF; Makasar
806 | 11FB0..11FBF; Lisu Supplement
807 | 11FC0..11FFF; Tamil Supplement
808 | 12000..123FF; Cuneiform
809 | 12400..1247F; Cuneiform Numbers and Punctuation
810 | 12480..1254F; Early Dynastic Cuneiform
811 | 12F90..12FFF; Cypro-Minoan
812 | 13000..1342F; Egyptian Hieroglyphs
813 | 13430..1343F; Egyptian Hieroglyph Format Controls
814 | 14400..1467F; Anatolian Hieroglyphs
815 | 16800..16A3F; Bamum Supplement
816 | 16A40..16A6F; Mro
817 | 16A70..16ACF; Tangsa
818 | 16AD0..16AFF; Bassa Vah
819 | 16B00..16B8F; Pahawh Hmong
820 | 16E40..16E9F; Medefaidrin
821 | 16F00..16F9F; Miao
822 | 16FE0..16FFF; Ideographic Symbols and Punctuation
823 | 17000..187FF; Tangut
824 | 18800..18AFF; Tangut Components
825 | 18B00..18CFF; Khitan Small Script
826 | 18D00..18D7F; Tangut Supplement
827 | 1AFF0..1AFFF; Kana Extended-B
828 | 1B000..1B0FF; Kana Supplement
829 | 1B100..1B12F; Kana Extended-A
830 | 1B130..1B16F; Small Kana Extension
831 | 1B170..1B2FF; Nushu
832 | 1BC00..1BC9F; Duployan
833 | 1BCA0..1BCAF; Shorthand Format Controls
834 | 1CF00..1CFCF; Znamenny Musical Notation
835 | 1D000..1D0FF; Byzantine Musical Symbols
836 | 1D100..1D1FF; Musical Symbols
837 | 1D200..1D24F; Ancient Greek Musical Notation
838 | 1D2E0..1D2FF; Mayan Numerals
839 | 1D300..1D35F; Tai Xuan Jing Symbols
840 | 1D360..1D37F; Counting Rod Numerals
841 | 1D400..1D7FF; Mathematical Alphanumeric Symbols
842 | 1D800..1DAAF; Sutton SignWriting
843 | 1DF00..1DFFF; Latin Extended-G
844 | 1E000..1E02F; Glagolitic Supplement
845 | 1E100..1E14F; Nyiakeng Puachue Hmong
846 | 1E290..1E2BF; Toto
847 | 1E2C0..1E2FF; Wancho
848 | 1E7E0..1E7FF; Ethiopic Extended-B
849 | 1E800..1E8DF; Mende Kikakui
850 | 1E900..1E95F; Adlam
851 | 1EC70..1ECBF; Indic Siyaq Numbers
852 | 1ED00..1ED4F; Ottoman Siyaq Numbers
853 | 1EE00..1EEFF; Arabic Mathematical Alphabetic Symbols
854 | 1F000..1F02F; Mahjong Tiles
855 | 1F030..1F09F; Domino Tiles
856 | 1F0A0..1F0FF; Playing Cards
857 | 1F100..1F1FF; Enclosed Alphanumeric Supplement
858 | 1F200..1F2FF; Enclosed Ideographic Supplement
859 | 1F300..1F5FF; Miscellaneous Symbols and Pictographs
860 | 1F600..1F64F; Emoticons
861 | 1F650..1F67F; Ornamental Dingbats
862 | 1F680..1F6FF; Transport and Map Symbols
863 | 1F700..1F77F; Alchemical Symbols
864 | 1F780..1F7FF; Geometric Shapes Extended
865 | 1F800..1F8FF; Supplemental Arrows-C
866 | 1F900..1F9FF; Supplemental Symbols and Pictographs
867 | 1FA00..1FA6F; Chess Symbols
868 | 1FA70..1FAFF; Symbols and Pictographs Extended-A
869 | 1FB00..1FBFF; Symbols for Legacy Computing
870 | 20000..2A6DF; CJK Unified Ideographs Extension B
871 | 2A700..2B73F; CJK Unified Ideographs Extension C
872 | 2B740..2B81F; CJK Unified Ideographs Extension D
873 | 2B820..2CEAF; CJK Unified Ideographs Extension E
874 | 2CEB0..2EBEF; CJK Unified Ideographs Extension F
875 | 2F800..2FA1F; CJK Compatibility Ideographs Supplement
876 | 30000..3134F; CJK Unified Ideographs Extension G
877 | E0000..E007F; Tags
878 | E0100..E01EF; Variation Selectors Supplement
879 | F0000..FFFFF; Supplementary Private Use Area-A
880 | 100000..10FFFF; Supplementary Private Use Area-B
881 | """
882 | for line in blockstxt.splitlines():
883 | try:
884 | (r, block) = line.split('; ')
885 | (start, end) = r.split('..')
886 | start = int(start, 16)
887 | end = int(end, 16)
888 | if (start <= x) and (x <= end):
889 | return block
890 |
891 | except ValueError: continue
892 |
893 | # If we get here, then no block matched.
894 | return ""
895 |
896 | def compileit(s):
897 | """
898 | Compile the regular expresion in s, or die trying.
899 | This is for debugging of our regex generation, not for speed.
900 | """
901 |
902 | global ucd, args
903 | if debug:
904 | debugprint(s)
905 | m=search(s,ucd)
906 | if not m: debugprint("Uncompiled, definitely not in ucd")
907 | try:
908 | compile(s)
909 | except error as e:
910 | err("Error parsing regex: '%s'" % ".*".join(args.characters))
911 | err(e)
912 | exit(3)
913 |
914 | def isprint(c):
915 | # Given a category 'c', return True if it is printable.
916 | # We presume the only non-printable category is 'C' (control).
917 | # However, technically, much of category 'Z' (spaces) is non-printable.
918 |
919 | # Side note: Python is silly and regex clauses that match an empty
920 | # string are set to None instead of ''. That's why we doublecheck
921 | # that category is not None.
922 | return c and not c.startswith('C')
923 |
924 | def iscombining(c):
925 | # Given a category 'c', return True if it is a combining character.
926 | # We presume all combining characters are in category 'M' and vice-versa.
927 | if debug:
928 | debugprint ("Category is <" + str(c) + ">")
929 |
930 | return c and c.startswith('M')
931 |
932 | def printmatches(s):
933 | foundone=None
934 | for m in finditer(s, ucd):
935 | if printonematch(m):
936 | foundone=True
937 |
938 | # Return True if a match was found
939 | return foundone
940 |
941 | def printonematch(m):
942 | foundone=False
943 | if m:
944 | g=m.groupdict()
945 | debugprint(g)
946 |
947 | # Default to an empty string for missing regex keys.
948 | g=defaultdict(str, g)
949 |
950 | # What is the Pythonic way to do this?
951 | if not g["hex"]: g["hex"]=g["hextwo"]
952 | if not g["hex"]: g["hex"]=g["hexthree"]
953 | cphex=g["hex"]
954 | if not g["name"]: g["name"]=g["nametwo"]
955 | if not g["name"]: g["name"]=g["namethree"]
956 | name=g["name"]
957 | if not g["category"]: g["category"] = g["categorytwo"]
958 | if not g["category"]: g["category"] = g["categorythree"]
959 | category = g["category"]
960 | if not g["alias"]: g["alias"]=g["aliastwo"]
961 | if not g["alias"]: g["alias"]=g["aliasthree"]
962 | alias=g["alias"]
963 |
964 | # Do not use as the name if we've got an alias in UCD.
965 | if name == '' and alias:
966 | name = alias
967 | alias = ""
968 |
969 | try:
970 | codepoint=int(cphex, 16)
971 | except ValueError:
972 | err("BUG: Could not convert", cphex, "to integer.")
973 | return False
974 |
975 | c=chr(codepoint)
976 | foundone=True
977 | print(" ", end='') # Indent for MacOS
978 | maybe_open_box(codepoint, name, category)
979 | if not isprint(category): c="\uFFFD" # "Replacement Character"
980 | if iscombining(category): print('\u25CC', end='') # "Dotted circle"
981 | if should_box(codepoint, name, category):
982 | try: c=curses.black_on_yellow(c)
983 | except NameError: pass
984 | print(c, end='')
985 | maybe_close_box(codepoint, name, category)
986 | print(end='\t')
987 | print ("U+" + cphex, end='\t')
988 | print(name, end='')
989 | if (len(alias) and isdifferent(name, alias)):
990 | print (" (" + alias.lower() + ")", end='')
991 | for alias2 in lookup_namealias(cphex):
992 | if len(alias2):
993 | if not alias or isdifferent(alias, alias2):
994 | if not name or isdifferent(name, alias2):
995 | print (" (" + alias2 + ")", end='')
996 |
997 | print("", flush=True)
998 | # If -l, show list of installed fonts which contain the glyph.
999 | if args.list_fonts: show_fonts( cphex, category )
1000 |
1001 | return foundone
1002 |
1003 | def maybe_open_box(*args):
1004 | """
1005 | Display the next character differently if it is whitespace or has
1006 | some unusual meaning. This is similar to the Unicode charts which
1007 | put some characters in dashed boxes, but for ugrep we mostly only
1008 | care about marking the text cell should the user wish to copy some
1009 | invisible glyph from the terminal.
1010 | """
1011 |
1012 | if should_box(*args):
1013 | print("\b[", end='')
1014 |
1015 | def maybe_close_box(*args):
1016 | """
1017 | Display the previous character differently and disable any
1018 | formatting changes (such as color) set by maybe_open_box.
1019 | """
1020 |
1021 | if should_box(*args):
1022 | # print('\u20DE', end='') # "Enclosing box" # Unnecessary and ugly
1023 | print(']', end='')
1024 |
1025 | def should_box(x, name, category):
1026 | """
1027 | Input: codepoint x, character name, and category. return true if a
1028 | box of some sort should be drawn around the character to distinguish it.
1029 | This is mostly handy for invisible whitespace which is otherwise
1030 | impossible to cut and paste.
1031 |
1032 | Similar to the "Dashed Box Convention" in the Unicode Charts,
1033 | explained at https://www.unicode.org/versions/Unicode12.0.0/ch24.pdf#G8175.
1034 |
1035 | # As of version 12.1 it's used on
1036 | for y in ("0000-0020", "007F-00A0", "00AD", "034F", "0600-0605",
1037 | "061C", "06DD", "070F", "08E2", "0CF1-0CF2", "0D4E",
1038 | "0F0C", "1039", "115F-1160", "17B4-17B5", "17D2",
1039 | "180B-180F", "1A60", "1BAB", "1CF5-1CF6", "2000-200F",
1040 | "2011", "2028-202F", "205F-2064", "2066-206F", "2D7F",
1041 | "2E3A-2E3B", "3000", "303E", "3164", "AAF6",
1042 | "FE00-FE0F", "FEFF", "FFA0", "FFF9-FFFB", "10A3F",
1043 | "11003-11004", "1107F", "110BD", "110CD", "111C2-111C3",
1044 | "11A3A", "11A47", "11A84-11A89", "11A99", "11D45-11D46",
1045 | "11D97", "13430-13438", "16F8F-16F92", "1BC9D",
1046 | "1BCA0-1BCA3", "1D159", "1D173-1D17A", "1DA9B-1DA9F",
1047 | "1DAA1-1DAAF", "1F1E6-1F1FF", "E0001", "E0020-E007F",
1048 | "E0100-E01EF"):
1049 |
1050 | try:
1051 | (start, end) = y.split('-')
1052 | except ValueError:
1053 | start = y
1054 | end = start
1055 |
1056 | if x>=int(start, 16) and x<=int(end, 16): return True
1057 |
1058 | """
1059 |
1060 | # Do not box up the replacement character
1061 | if category and not isprint(category): return False
1062 |
1063 | # Box up non-zero width space.
1064 | if category and category.startswith('Z'):
1065 | return True
1066 |
1067 | return False
1068 |
1069 | def show_fonts(x, category=None):
1070 | """
1071 | Given a hexadecimal string representing a Unicode code point,
1072 | list every installed font that defines a glyph at that code point.
1073 | """
1074 |
1075 | # Normalize whatever they put in. 'U+d0E' to '0D0E'
1076 | x=x.lstrip("Uu+0Xx").upper().zfill(4)
1077 |
1078 | # try: fonts = getoutput("fc-list -f '%{family[0]}\t%{style[0]}\t%{file}\n' :charset=" + x + " | sort -u")
1079 |
1080 | # Find fonts that contain x, remove dupes, and put back in original order.
1081 | # (Dupes often caused by multiple point sizes of a bitmap font).
1082 | fclistcmd="fc-list :charset=" + x
1083 | fclistcmd+=" -f '%{family[0]}\t%{style[0]}\t%{file}\t%{spacing}\n'"
1084 | fclistcmd+=" | cat -n | sort -t'\t' -k2,3 -u | sort -n | cut -f2-"
1085 |
1086 | try: fonts = getoutput(fclistcmd)
1087 | except FileNotFoundError: print(""); return # No FontConfig
1088 | except KeyboardInterrupt: exit(1) # Exit on ^C
1089 |
1090 |
1091 | #debugprint(fonts) # List of every matching font.
1092 |
1093 | if debug: count=0
1094 | for f in fonts.splitlines():
1095 | try: (family, style, filename, spacing) = f.split('\t')
1096 | except: continue
1097 |
1098 | if debug:
1099 | count=count+1
1100 | debugprint(count, filename, family, style, spacing)
1101 |
1102 | # Don't print the style name if it is just "Regular"
1103 | if style == "Regular" or style == "Roman" or style == "Book":
1104 | style = ""
1105 |
1106 | # If we're not doing a long listing, then skip variant styles
1107 | if not args.list_fonts_long:
1108 | if style != "": continue
1109 |
1110 | # If user requested monospace fonts, then skip proportional fonts
1111 | if args.monospace_fonts_only:
1112 | if spacing == "": continue
1113 |
1114 | famstr="%(family)s %(style)s" % locals()
1115 |
1116 | # Text positioning is hairy due to variable text and font size
1117 | if args.fonts_scale <2: # The usual case:
1118 | double_text=False
1119 | else: # Alternately, with double size text:
1120 | double_text=True
1121 |
1122 | # Some terminals cannot handle double-size text
1123 | if args.never_double_text or not isatty(1):
1124 | double_text=False
1125 |
1126 | # Disable double-size text when debugging (at least for now)
1127 | if debug: double_text=False
1128 |
1129 | if not double_text:
1130 | famstr = '\t\t' + famstr # Two tabs, sixteen spaces
1131 | else:
1132 | famstr = '\t' + famstr # One double-size tab
1133 |
1134 | if isatty(1):
1135 | # Writing to a screen? Truncate text if it would wrap.
1136 | (rows, cols) = get_rows_cols()
1137 | if double_text: cols = int(cols/2)
1138 | if len(famstr.expandtabs()) >= cols:
1139 | famstr = famstr.expandtabs()[:cols-1]
1140 |
1141 | famstr = famstr + '\r'
1142 |
1143 | if is_sixel_capable():
1144 | # If terminal can show sixel graphics, do the following:
1145 | # 1. If fonts_scale >=2, double the size of the the fontname
1146 | # 2. Position the font name based on the fontscale
1147 | # 3. Render a sixel image of the glyph in the font
1148 |
1149 | esc=chr(0x1b)
1150 | try:
1151 | previous_line=curses.cuu(1)
1152 | except NameError:
1153 | previous_line = esc + '[A' # Just presume VT100
1154 |
1155 | # Escape sequence to print double size text (DECDHL).
1156 | # (previous_line is so we'll be positioned for sixels)
1157 | if double_text:
1158 | famstr = esc + '#3' + famstr + '\n' \
1159 | + esc + '#4' + famstr + previous_line
1160 |
1161 | # Align text lower for 4x, 8x sized font glyphs
1162 | if double_text and args.fonts_scale >= 4:
1163 | famstr = '\n' + famstr + previous_line
1164 | if double_text and args.fonts_scale >=8:
1165 | famstr = '\n' + famstr + previous_line
1166 |
1167 | print(famstr, end='')
1168 |
1169 | if not is_sixel_capable():
1170 | print("") # stdout is not a tty, so send a newline
1171 | else:
1172 | # stdout is a sixel terminal, show the glyph bitmaps
1173 | (foreground, background) = get_term_colors()
1174 | (width, height) = get_cell_size() # Character cell font size
1175 | height = args.fonts_scale * height # Increase size of font (maybe).
1176 | if not double_text:
1177 | width=(16-1)*width # Leave one space before fontname
1178 | else:
1179 | width=(16-2)*width # Leave one double-wide space
1180 | debugprint("Font scaling factor: %g" % (args.fonts_scale))
1181 | debugprint("Font canvas size: %g x %g" % (width, height))
1182 |
1183 | codepoint=chr(int(x,16))
1184 |
1185 | # If it's a combining character, add a dotted circle to combine with
1186 | if iscombining(category): codepoint='\u25CC'+codepoint
1187 |
1188 | command=[ "convert",
1189 | "-size",
1190 | "%(width)sx%(height)s" % locals(),
1191 | "xc:" + "%(background)s" % locals(),
1192 | "-fill", "%(foreground)s" % locals(),
1193 | "(",
1194 | "-background", "none",
1195 | "-font", "%(filename)s" % locals(),
1196 | "label:" + codepoint,
1197 | "-trim",
1198 | ")",
1199 | "-gravity", "east",
1200 | "-compose", "over",
1201 | "-composite",
1202 | "+dither",
1203 | "-colors", "16", # Faster sixels by reducing colors
1204 | "sixel:-" ]
1205 | if debug:
1206 | debugprint(*["'" + c + "'" for c in command], flush=True)
1207 |
1208 | try: output=run(command, capture_output=True)
1209 | except FileNotFoundError: print(""); continue # No ImageMagick
1210 | except KeyboardInterrupt: exit(1) # Exit on ^C
1211 | except (ValueError, TypeError):
1212 | # ValueError occurs on NULL,
1213 | # TypeError from Python <3.7 which did not have capture_output
1214 | if double_text: print("")
1215 | print("", flush=True)
1216 | continue
1217 |
1218 | if output.returncode != 0:
1219 | # Oops, font did not render.
1220 | if double_text: print("")
1221 | print("", flush=True)
1222 | continue
1223 |
1224 | # Workaround ImageMagick bug: graphics newline '-' for '\n'
1225 | output.stdout = output.stdout.replace(b'-\x1B\\', b'\x1B\\\n')
1226 |
1227 | print(flush=True, end='') # Send any pending output to stdout
1228 |
1229 | try: write(1, output.stdout) # Print raw bytes to stdout
1230 | except KeyboardInterrupt: # Catch ^C
1231 | exit(1) # Sixels stopped by cleanup()
1232 |
1233 | def is_sixel_capable():
1234 | "Return true if the terminal can show sixel graphics"
1235 |
1236 | try:
1237 | if is_sixel_capable.cached: return is_sixel_capable.cached
1238 | except AttributeError:
1239 | is_sixel_capable.cached=False
1240 | return False
1241 |
1242 | # Force sixels on if --pipe-sixels was given.
1243 | if args.pipe_sixels:
1244 | is_sixel_capable.cached=True
1245 | return True
1246 |
1247 | # If stdin or stdout is not a terminal, don't send sixels
1248 | if not isatty(0) or not isatty(1):
1249 | is_sixel_capable.cached=False
1250 | return False
1251 |
1252 | # check DA to see if sixels are supported
1253 | results=terminal_query_list('\x1B[c')
1254 | if "4" in results:
1255 | is_sixel_capable.cached=True
1256 | return True
1257 | else:
1258 | is_sixel_capable.cached=False
1259 | return False
1260 |
1261 | def get_rows_cols():
1262 | # Return number of (rows, columns) in current terminal window
1263 | # We use termios ioctl as it is dead simple.
1264 | #
1265 | # We cache rows/cols, but perhaps SIGWINCH ought to reset .cached = None.
1266 | try:
1267 | if get_rows_cols.cached: return get_rows_cols.cached
1268 | except AttributeError:
1269 | get_rows_cols.cached=None
1270 |
1271 | import array, fcntl, termios
1272 | buf = array.array('H', [0, 0, 0, 0]) # H == unsigned int
1273 | if isatty(0):
1274 | fcntl.ioctl(0, termios.TIOCGWINSZ, buf)
1275 | (rows, cols, x, y) = buf
1276 | if rows==0: rows=24
1277 | if cols==0: cols=80
1278 | get_rows_cols.cached = (rows,cols)
1279 | return (rows,cols)
1280 |
1281 | def get_cell_size():
1282 | """
1283 | Inquire from the terminal what its character cell size is.
1284 | Returns a tuple (width, height) in pixels.
1285 | """
1286 |
1287 | # XXX Shoud we bother to catch SIGWINCH when the font changes?
1288 | # (Probably not. User can just run ugrep again.)
1289 | try:
1290 | if get_cell_size.cached: return get_cell_size.cached
1291 | except AttributeError:
1292 | get_cell_size.cached=None
1293 |
1294 | # Ideally, we use an Esc seq to ask terminal what its char cell size is.
1295 | output = terminal_query_list('\x1B[16t')
1296 | if len(output) >= 3:
1297 | try:
1298 | get_cell_size.cached = ( int(output[2]), int(output[1]) )
1299 | return get_cell_size.cached
1300 | except:
1301 | # Response got mangled somehow. Ignore it.
1302 | pass
1303 |
1304 | # Didn't work, so try calculating cell size from geometry.
1305 | # We get geometry in three ways:
1306 |
1307 | # First, TIOCGWINSZ ioctl. It's simple but x,y are inaccurate.
1308 | import array, fcntl, termios
1309 | buf = array.array('H', [0, 0, 0, 0]) # H == unsigned int
1310 | if isatty(0):
1311 | fcntl.ioctl(0, termios.TIOCGWINSZ, buf)
1312 | # Beware of the stupid output order: y x x y. --Egmont Koblinger
1313 | (rows, cols, x, y) = buf;
1314 |
1315 | # Second, XTSMGRAPHICS escape sequence gives the best x,y accuracy.
1316 | output = terminal_query_list('\x1B[?2;1;0S')
1317 | if output:
1318 | try:
1319 | (x, y) = ( int(output[2]), int(output[3]) )
1320 | except ValueError:
1321 | # Escape sequence response got mangled somehow. Ignore it.
1322 | pass
1323 | else:
1324 | # Third, dtterm's x,y are (usually) better than TIOCGWINSZ.
1325 | output = terminal_query_list('\x1B[14t')
1326 | if output:
1327 | try:
1328 | (x, y) = ( int(output[2]), int(output[1]) )
1329 | except ValueError:
1330 | pass
1331 |
1332 | # VT340 on serial line would still have x and y set to zero.
1333 | if x==0: x=800
1334 | if y==0: y=480
1335 |
1336 | # TIOCGWINSZ is reliable for rows and cols, but just in case.
1337 | if rows==0: rows=24
1338 | if cols==0: cols=80
1339 |
1340 | get_cell_size.cached = ( floor(x/cols), floor(y/rows) )
1341 |
1342 | debugprint("rows: %(rows)s\tcols: %(cols)s\tx: %(x)s\ty: %(y)s" % locals())
1343 | debugprint("charcell w: %d\th: %d" % (floor(x/cols), floor(y/rows) ))
1344 | return get_cell_size.cached
1345 |
1346 | def get_term_colors():
1347 | """
1348 | Returns a terminals colors as a tuple (foreground, background)
1349 | Handles reversed video by checking mode 5.
1350 | """
1351 |
1352 | # In bash:
1353 | # read -r -p $'\e]10;?\007' -d$'\007' fgcolor
1354 | # read -r -p $'\e]11;?\007' -d$'\007' bgcolor
1355 |
1356 | # Example output from esc sequence:
1357 | # read -r -p $'\033]11;?\007' -d$'\007'
1358 | # ^[]11;rgb:1a1a/1a1a/1a1a^G
1359 |
1360 | try:
1361 | if get_term_colors.cached: return get_term_colors.cached
1362 | except AttributeError:
1363 | # Use VT340 colors by default if detection fails
1364 | get_term_colors.cached= ("gray48", "black")
1365 |
1366 | output = terminal_query_list('\x1B]10;?\x1B\\')
1367 | if output:
1368 | # output is of the form: ['10', 'rgb:1abc/2def/3456']
1369 | # but we want just "#1abc2def3456
1370 | fg=output[1].replace('rgb:', '#').replace('/', '')
1371 |
1372 | output = terminal_query_list('\x1B]11;?\x1B\\')
1373 | if output:
1374 | bg=output[1].replace('rgb:', '#').replace('/', '')
1375 |
1376 | # Check for reverse video (DECSCNM) i.e., $'\e[?5h' light background
1377 | output = terminal_request_mode(5)
1378 | if output == "set" or output == "permanently set":
1379 | get_term_colors.cached=(bg, fg) # Reversed video
1380 | else:
1381 | get_term_colors.cached=(fg, bg) # Normal (dark) background
1382 |
1383 | return get_term_colors.cached
1384 |
1385 | def isdifferent(a, b):
1386 | """Given two strings A & B, determine if B is different enough
1387 | from A that we don't need to print it as well. "Enough" means
1388 | "adds significant new information". For example, the following
1389 | would return False:
1390 |
1391 | "LATIN CAPITAL LETTER A WITH GRAVE", "LATIN CAPITAL LETTER A GRAVE"
1392 |
1393 | Note that order matters. For example,
1394 |
1395 | "BROKEN BAR", "BROKEN VERTICAL BAR" Should return True
1396 | "BROKEN VERTICAL BAR", "BROKEN BAR" Should return False
1397 |
1398 | """
1399 | a=a.upper()
1400 | b=b.upper()
1401 |
1402 | if a == b:
1403 | return False
1404 |
1405 | bb = b.replace("NON-SPACING", "COMBINING")
1406 | bb = bb.replace("SPACING ", "")
1407 | bb = bb.replace("OVERSCORE", "OVERLINE")
1408 | bb = bb.replace("UNDERSCORE", "LOW LINE")
1409 | aa = a.replace("PRESENTATION FORM", "GLYPH")
1410 |
1411 | if a == bb:
1412 | return False
1413 |
1414 | if aa.replace(" ACCENT", "") == bb:
1415 | return False
1416 | if aa == bb.replace(" DIGIT ", " "):
1417 | return False
1418 | if aa == bb.replace("FORMS ", "BOX DRAWINGS "):
1419 | return False
1420 | if aa == bb.replace("GRAPHIC ", "SYMBOL "):
1421 | return False
1422 | if aa == bb.replace("CENTER", "CENTRE"):
1423 | return False
1424 | if aa == bb.replace("CENTERED", "CENTRED"):
1425 | return False
1426 | if aa == bb.replace("SQUARED ", "SQUARE "):
1427 | return False
1428 | if aa == bb.replace("SPACING ", ""):
1429 | return False
1430 | if aa == bb.replace("BAR", "LINE"):
1431 | return False
1432 | if aa == bb.replace("BAR", "STROKE"):
1433 | return False
1434 | if aa == bb.replace("BAR", "WITH STROKE"):
1435 | return False
1436 | if aa == bb.replace("SLASH", "WITH STROKE"):
1437 | return False
1438 | if aa == bb.replace("OPENING", "LEFT"):
1439 | return False
1440 | if aa == bb.replace("CLOSING", "RIGHT"):
1441 | return False
1442 | if aa == bb.replace("EASTERN ARABIC", "EXTENDED ARABIC"):
1443 | return False
1444 |
1445 | if bb.find("HACEK") != -1: # Replaced with CARON
1446 | return False
1447 |
1448 | if bb.find("YOGH") != -1: # Replaced with EZH
1449 | return False
1450 |
1451 | if bb.find("GUILLEMET") != -1: # Replaced with ANGLE QUOTATION MARK
1452 | return False
1453 |
1454 | if bb.find("HAMZAH") != -1: # Replaced with HAMZA
1455 | return False
1456 |
1457 | if bb.find("ARABIC") != -1:
1458 | if aa.endswith("FORM"):
1459 | return False
1460 |
1461 | # If b is found within a, it is likely not different. (Return False)
1462 | # Word breaks, \b, are used to distinguish TAB from CHARACTER TABULATION.
1463 | r=r'\b'+b.replace(" ", ".*")+r'\b'
1464 | try:
1465 | m=search(r, a)
1466 | if m:
1467 | return False
1468 | else:
1469 | return True
1470 | except:
1471 | # regex should never fail, but if it does, we don't care.
1472 | return True
1473 |
1474 |
1475 | # if a.replace("WARDS ", " ") == b:
1476 | # return False
1477 | # if a.replace("S ", " ") == b:
1478 | # return False
1479 | # if a.rstrip("S") == b:
1480 | # return False
1481 |
1482 | return True
1483 |
1484 | def loaducd():
1485 | "Find the UnicodeData.txt file and load it up into ucd global variable."
1486 |
1487 | global ucd
1488 | ucd=None
1489 | ucdplaces=( "UnicodeData.txt",
1490 | "~/.local/share/unicode/UnicodeData.txt",
1491 | "/usr/local/share/unicode/UnicodeData.txt",
1492 | "/usr/share/unicode/UnicodeData.txt"
1493 | )
1494 | for f in ucdplaces:
1495 | f=expanduser(expandvars(f)) # Python's open() is silly
1496 | try:
1497 | ucd=open(f).read()
1498 | debugprint("Found data file at " + f)
1499 | except:
1500 | continue
1501 |
1502 | # Sanity check: did we find the UnicodeData.txt file?
1503 | if ucd == None:
1504 | err("""\
1505 | Could not find UnicodeData.txt in:
1506 |
1507 | %s
1508 | On most GNU/Linux systems your package manager can install it easily.
1509 | For example: apt install unicode-data.
1510 |
1511 | Alternately, you can grab it via wget like so:
1512 |
1513 | mkdir -p ~/.local/share/unicode
1514 | cd ~/.local/share/unicode
1515 | wget ftp://ftp.unicode.org/Public/UNIDATA/UnicodeData.txt
1516 | """ %
1517 | ("".join([" " + s + "\n" for s in ucdplaces])))
1518 | exit(2)
1519 |
1520 | if debug:
1521 | debugprint("Number of Unicode chars: %d" % len(ucd.splitlines()))
1522 |
1523 | return ucd
1524 |
1525 |
1526 | readings = None
1527 | def loadreadings():
1528 | """
1529 | Find the Unihan_Readings.txt file and load it up into the
1530 | 'readings' global variable. Note that the file may be compressed
1531 | with bzip2.
1532 |
1533 | Unicode distributes it within a .zip file, but we do not yet
1534 | handle reading it directly from that file.
1535 | """
1536 |
1537 | global readings
1538 | if readings != None: return readings # Only search once.
1539 |
1540 | ucdplaces=( "",
1541 | "~/.local/share/unicode/",
1542 | "/usr/local/share/unicode/",
1543 | "/usr/share/unicode/"
1544 | )
1545 | for f in ucdplaces:
1546 | if readings: break
1547 | f=f+"Unihan_Readings.txt" # Plain text file, not compressed.
1548 | f=expanduser(expandvars(f)) # Python's open() is silly
1549 | try:
1550 | readings=open(f).read()
1551 | debugprint("Found data file at " + f)
1552 | except FileNotFoundError:
1553 | debugprint("File not found at " + f)
1554 | continue
1555 |
1556 | if readings == None:
1557 | # Plain text not found. Maybe it is compressed?
1558 | try:
1559 | import bz2
1560 | for f in ucdplaces:
1561 | if readings: break
1562 | f=f+"Unihan_Readings.txt.bz2" # bz2 compressed
1563 | f=expanduser(expandvars(f)) # Python's open() is silly
1564 | try:
1565 | readings=bz2.open(f).read()
1566 | debugprint("Found data file at " + f)
1567 | except FileNotFoundError:
1568 | debugprint("File not found at " + f)
1569 | continue
1570 | except ImportError:
1571 | debugprint("Could not import bz2. Skipped looking for Unihan_Readings.txt.bz2.")
1572 | pass
1573 |
1574 | # Sanity check: did we find the Unihan_Readings.txt file?
1575 | if readings == None:
1576 | readings = False # Don't check again for the file
1577 | debugprint("""\
1578 | Could not find Unihan_Readings.txt in:
1579 |
1580 | %s
1581 | On most GNU/Linux systems your package manager can install it easily.
1582 | For example: apt install unicode-data.
1583 |
1584 | Alternately, you can grab it via wget like so:
1585 |
1586 | mkdir -p ~/.local/share/unicode
1587 | cd ~/.local/share/unicode
1588 | wget ftp://ftp.unicode.org/Public/UNIDATA/Unihan.zip
1589 | unzip Unihan.zip
1590 | """ %
1591 | ("".join([" " + s + "Unihan_Readings.txt" + "\n" for s in ucdplaces])))
1592 |
1593 | return readings
1594 |
1595 |
1596 | namealiases = None
1597 | def loadnamealiases():
1598 | """Find the NameAliases.txt file and load it up into the 'namealiases'
1599 | global variable. This is useful for a handful of aliases which
1600 | should have been in UnicodeData.txt but are not.
1601 |
1602 | Unfortunately, much of the file is redundant or unhelpful. For
1603 | example, it has abbreviations of nearly everything, whether it is
1604 | useful (ZWJ) or not (VS256). Most of the aliases which might be
1605 | helpful also exist in the alias field in UnicodeData.txt, and thus
1606 | already used by ugrep.
1607 |
1608 | In order to not print redundant information, it would be best to
1609 | double check that the namealias is different enough from the
1610 | standard alias. (See the isdifferent() function).
1611 | """
1612 |
1613 | global namealiases
1614 | if namealiases != None: return namealiases # Only search once.
1615 |
1616 | ucdplaces=( "",
1617 | "~/.local/share/unicode/",
1618 | "/usr/local/share/unicode/",
1619 | "/usr/share/unicode/"
1620 | )
1621 | for f in ucdplaces:
1622 | f=f+"NameAliases.txt" # Plain text file
1623 | f=expanduser(expandvars(f)) # Python's open() is silly
1624 | try:
1625 | namealiases=open(f).read()
1626 | debugprint("Found data file at " + f)
1627 | break
1628 | except FileNotFoundError:
1629 | debugprint("File not found at " + f)
1630 | continue
1631 |
1632 | # Sanity check: did we find the NameAliases.txt file?
1633 | if namealiases == None:
1634 | namealiases = False # Don't check again for the file
1635 | debugprint("""\
1636 | Could not find NameAliases.txt in:
1637 |
1638 | %s
1639 | On most GNU/Linux systems your package manager can install it easily.
1640 | For example: apt install unicode-data.
1641 |
1642 | Alternately, you can grab it via wget like so:
1643 |
1644 | mkdir -p ~/.local/share/unicode
1645 | cd ~/.local/share/unicode
1646 | wget ftp://ftp.unicode.org/Public/UNIDATA/NameAliases.txt
1647 | """ %
1648 | ("".join([" " + s + "NameAliases.txt" + "\n" for s in ucdplaces])))
1649 |
1650 | return namealiases
1651 |
1652 |
1653 | def makeregexcharname(words):
1654 | "Create a regular expression to search for a character name"
1655 |
1656 | # Format of ucd: each character is on a separate line.
1657 | # Each line is fifteen columns separated by semicolons.
1658 | # We only care about columns 1, 2, 3 and 11
1659 | # 1: Code value
1660 | # 2: Character name
1661 | # 3: General category
1662 | # 11: ISO 10646 comment field (usually an alias)
1663 | #
1664 | # For example:
1665 | # 002F;SOLIDUS;Po;0;CS;;;;;N;SLASH;;;;
1666 | #
1667 | # See Tech Report 44 for more details.
1668 | #
1669 | # XXX Todo: maybe look up in table of confusable entities.
1670 |
1671 | # combine and quote the command line arguments so we can use them in
1672 | # verbose regex. Also, allow the user to use ^ and $, just like in awk
1673 | # to refer to the beginning and end of the field instead of line.
1674 | if type(words) is list:
1675 | words=["("+x+")" for x in words] # wrap in parens to fix alternation
1676 | arghs=".*".join(words) # search terms can have junk between
1677 | else:
1678 | arghs="(" + "".join(words) + ")"
1679 | debugprint("arghs is", arghs)
1680 |
1681 | arghs=sub(r"(\s)", r"\\\1", arghs) # quote whitespace for verbose regex
1682 | arghs=sub(r"^\(\^", r"(?<=;)(", arghs) # ^ matches semicolon before field
1683 | arghs=sub(r"\$\)$", r"(?=;))", arghs) # $ matches semicolon after field
1684 |
1685 | # s is a search regex for field 2 (name) based on the command line arguments
1686 | s=r"""
1687 | ^(?P[^;\n]*) # first field is hexadecimal codepoint
1688 | ;(?P[^;\n]*
1689 | """ + arghs + """ # field 2 (charactername) matches?
1690 | [^;\n]*)
1691 | ;(?P[^;\n]*) # third field is general category
1692 | (;[^;\n]*){7} # skip next 7 fields
1693 | ;(?P[^;\n]*) # field 11 is comment/alias
1694 | (;[^;\n]*){4}$ # line ends with four more fields
1695 | """
1696 |
1697 | # # r is a search regex for field 11 (alias)
1698 | r=r"""
1699 | ^(?P[^;\n]*) # first field is hexadecimal codepoint
1700 | ;(?P[^;\n]*) # second field is charactername
1701 | ;(?P[^;\n]*) # third field is general category
1702 | (;[^;\n]*){7} # skip next 7 fields
1703 | ;(?P[^;\n]*(
1704 | """ + arghs + r""" # field 11 (alias) matches?
1705 | )[^;\n]*)
1706 | (;[^;\n]*){4}$ # line ends with four more fields
1707 | """
1708 |
1709 | s=s+"|"+r # Search for either name or alias
1710 | s="(?mix)"+s # multiline, case insensitive, verbose
1711 | return s
1712 |
1713 | def makeregexcodepoint(cp):
1714 | "Make a regular expression to search for a hexadecimal codepoint"
1715 |
1716 | # User specified a codepoint, e.g., "U+23fb", "0x23FB", "[0-9A-F]{2}"
1717 | # Normalize whatever they put in into "23FB"
1718 | debugprint("given codepoint was", cp)
1719 | cp=cp.upper()
1720 | if cp.find('0X') == 0: cp=cp.replace('0X', '')
1721 | cp='0*' + cp.lstrip("U+").zfill(4)
1722 | debugprint("codepoint we'll search for is", cp)
1723 |
1724 | # s is a search regex for field 1 (hexadecimal)
1725 | s=r"""
1726 | ^(?P"""+cp+""") # first field is hex codepoint
1727 | ;(?P[^;\n]*) # second field is charactername
1728 | ;(?P[^;\n]*) # third field is category
1729 | (;[^;\n]*){7} # skip next 7 fields
1730 | ;(?P[^;\n]*) # field 11 is comment/alias
1731 | (;[^;\n]*){4}$ # line ends with four more fields
1732 | """
1733 | s="(?mix)"+s # multiline, case insensitive, verbose
1734 | return(s)
1735 |
1736 |
1737 | def makeregexeither(wordsorcp):
1738 | """Make a regular expression to search for a hexadecimal codepoint
1739 | or for a character name / alias."""
1740 |
1741 | r=makeregexcharname(wordsorcp)
1742 |
1743 | if (type(wordsorcp) is list) and (len(wordsorcp) == 1):
1744 | wordsorcp=wordsorcp[0]
1745 |
1746 | if (type(wordsorcp) is list):
1747 | debugprint("Multiple args, so complete search regex is", r)
1748 | return r
1749 |
1750 | s=makeregexcodepoint(wordsorcp)
1751 |
1752 | r = r.lstrip('(?mix)')
1753 |
1754 | debugprint("Complete search regex is " + s + "|" + r)
1755 |
1756 | return(s + "|" + r)
1757 |
1758 |
1759 |
1760 | # XXXX TODO: This is slow for 0..10FFFF
1761 | def rangeexpansion(oldargv):
1762 | r"""
1763 | Look for ranges "a..z" or "a..z..i" in oldargv[] and expand by
1764 | inserting new elements. The new argv is returned to the caller.
1765 |
1766 | 0..7 --> 0 1 2 3 4 5 6 7
1767 | abc0..7 --> abc0 abc1 abc2 abc3 abc4 abc5 abc6 abc7
1768 | 23b8..ff --> 23b8 23b9 23ba 23bb ... 23fd 23fe 23ff
1769 | 7fff..8 --> 7fff 7ffe 7ffd 7ffc 7ffb 7ffa 7ff9 7ff8
1770 | 0..FFFF..1000 --> 0000 1000 2000 3000 ... D000 E000 F000
1771 |
1772 | Optional U+ or 0x is allowed: U+0230..0xF.
1773 | Increment i defaults to +1 or -1 as appropriate.
1774 | Range is inclusive of both ends.
1775 | Output prepends 0x at beginning to force hexadecimal lookup.
1776 | (fdb..ff should not match OGHAM FEATHER MARK)
1777 |
1778 | DESIGN NOTES ON CURLY BRACE EXPANSION:
1779 |
1780 | * TL;DR: Avoid curly braces due to quoting confusion with shell.
1781 | * Ranges within curly braces '{' and '}' are minimally supported.
1782 | * Format: preamble{start..end..increment}postscript.
1783 | * Multiple ranges and nested braces are not supported by ugrep.
1784 | * The shell's brace expansion is inadequate because it is decimal only.
1785 | * ugrep {0..9}000: works because shell expands it.
1786 | * ugrep {0..F}000: works because we expand it.
1787 | * ugrep {0..10}000: fails because the shell interprets it as decimal.
1788 | * ugrep \{0..10}000: works because it is quoted so ugrep interprets it.
1789 | * The shell handles comma sequences, but ugrep does not.
1790 | ugrep {0..F}{0,4,8,C}00: works due to both shell and ugrep, but
1791 | ugrep 0..FFFF..400: works better as it is sorted correctly.
1792 | * Bash interprets an end number with fewer digits differently than ugrep!
1793 | ugrep \{8000..8}: ugrep counts up from 8000 to 8008
1794 | ugrep {8000..8}: Bash counts DOWN (!) from 8000 to 0008 (in DECIMAL).
1795 | * ugrep \{8000..0008}: ugrep counts down from 8000 to 0008 (in hex).
1796 |
1797 | A NOTE ABOUT SLOWNESS.
1798 |
1799 | Ranges are slow in the current implementation because ugrep was
1800 | originally written to create a single regexp and then use the
1801 | language's optimized routines to find it within a file.
1802 | Ranges currently make a new regexp over and over, which is silly.
1803 | """
1804 |
1805 |
1806 |
1807 | s=r"""(?mix)
1808 | (U+?|0?x)?
1809 | ((?P[0-9A-F]*)(?P{))?
1810 | (?P[0-9A-F]+)
1811 | \.\. # Literal two periods. MANDATORY.
1812 | (U+?|0?x)?
1813 | (?P[0-9A-F]+)
1814 | (\.\.)? # Literal two periods. OPTIONAL.
1815 | (0?x)?
1816 | (?P[-0-9A-F]+)?
1817 | (}(?P[0-9A-F]*))?
1818 | """
1819 | newargv=[]
1820 |
1821 | for arg in oldargv:
1822 | m=search(s, arg)
1823 | if not m:
1824 | newargv.append(arg)
1825 | else:
1826 | debugprint("Doing range expansion on " + arg)
1827 | g=m.groupdict()
1828 | preamble=g["preamble"]
1829 | start=g["start"]
1830 | end=g["end"]
1831 | increment=g["increment"]
1832 | postscript=g["postscript"]
1833 |
1834 | if not preamble: preamble=""
1835 | if not postscript: postscript=""
1836 | if not increment: increment="1"
1837 |
1838 | # 21b0..f is the same as 21b0..21bf
1839 | end=start[:-len(end)]+end
1840 | z=max(len(start), len(end))
1841 |
1842 | try:
1843 | start = int(start, 16)
1844 | end = int(end, 16)
1845 | increment = int(increment, 16)
1846 | increment = int(copysign(increment, end-start))
1847 | # Range is inclusive
1848 | end = end + int(copysign(1, increment))
1849 |
1850 | if debug:
1851 | if preamble or postscript:
1852 | debugprint("Preamble: '%s', Postscript: '%s'" %
1853 | (preamble, postscript))
1854 | debugprint("Found hex range %x to %x, increment %x" %
1855 | (start, end, increment))
1856 |
1857 | for i in range(start, end, increment):
1858 | hexits='0x' + preamble + hex(i)[2:].zfill(z) + postscript
1859 | newargv.append( hexits )
1860 | debugprint(" added: " + hexits)
1861 | except ValueError:
1862 | newargv.append(arg)
1863 |
1864 | return(newargv)
1865 |
1866 |
1867 | def eprint(*args, **kwargs):
1868 | "Print to stderr"
1869 | print(*args, file=stderr, **kwargs)
1870 |
1871 | def err(*args, **kwargs):
1872 | "Print to stderr with program name prefixed"
1873 | eprint(basename(argv[0]) + ": ", end='')
1874 | eprint(*args, **kwargs)
1875 |
1876 | def debugprint(*args, **kwargs):
1877 | "If debug var is True, print to stderr with progname prefix"
1878 | if (debug):
1879 | err(*args, **kwargs)
1880 |
1881 | def cleanup():
1882 | "Clean up mess before exiting due to an error"
1883 | if isatty(2):
1884 | termios.tcdrain(2) # Wait for stderr to be printed
1885 | if isatty(1):
1886 | termios.tcflush(1, termios.TCOFLUSH) # Discard stdout
1887 |
1888 | esc=chr(0x1b)
1889 | eprint(esc+"\\", end='') # String Terminator for sixels (ST)
1890 | eprint(esc+"#5", end='') # Single width line (DECSWL)
1891 | eprint("\r" + esc + "[J", end='') # Erase to end of display (ED)
1892 |
1893 |
1894 | ## See man termios(3) for details on tcgetattr
1895 | from termios import *
1896 | def terminal_query(seq, delimiter=None, timeout=0.2):
1897 | """
1898 | Given an escape SEQuence, and optionally a DELIMITER and a TIMEOUT,
1899 | print SEQ to stderr and read a response from stdin until the
1900 | character DELIMITER is read or TIMEOUT seconds is reached.
1901 | Input that is read is returned to the calling function.
1902 |
1903 | If TIMEOUT is not specified, it defaults to 0.2s. (Minimum is 0.1).
1904 | If DELIMITER is not specified, it defaults to the last character of SEQ.
1905 |
1906 | If neither DELIMITER nor SEQ are specified, then "" is returned.
1907 |
1908 | """
1909 | import sys, copy, posix
1910 |
1911 | if not delimiter and not seq:
1912 | return ""
1913 |
1914 | # Make sure we can query the terminal via stdin and stderr.
1915 | if not isatty(0) or not isatty(2) :
1916 | return ""
1917 |
1918 | if not seq: seq="" # Allow simply reading from terminal.
1919 |
1920 | # Responses usually end with the same character as the request.
1921 | if not delimiter and len(seq)>0:
1922 | delimiter=seq[-1]
1923 |
1924 | oldmode = tcgetattr(sys.stdin.fileno())
1925 |
1926 | # tcgetattr returns [iflag, oflag, cflag, lflag, ispeed, ospeed, cc]
1927 | [ iflag, oflag, cflag, lflag, ispeed, ospeed, cc ] = oldmode
1928 |
1929 | ## CBREAK MODE: Read byte by byte.
1930 | ## Cbreak is like Raw but allow ^C interrupt signal and does
1931 | ## not clear OPOST (print newlines as carriage return + newline)
1932 |
1933 | # Do not transmogrify input in any way...
1934 | iflag = iflag & ~(INPCK | ISTRIP | IXON | ICRNL | INLCR | IGNCR)
1935 | # ... except do allow BREAK (^C) to flush queues and send SIGINT
1936 | iflag = (iflag & ~IGNBRK) | BRKINT
1937 |
1938 | lflag = lflag & ~ICANON # Noncanonical: read by bytes, not lines
1939 | lflag = lflag & ~ECHO # Do not echo characters received
1940 |
1941 | # Clear character size and disable parity checking
1942 | cflag = cflag & ~(CSIZE | PARENB)
1943 | cflag = cflag | CS8 # Set 8-bit character size
1944 |
1945 | ## Polling read (MIN == 0, TIME == 0) See termios(3).
1946 | cc[VMIN] = 0
1947 | cc[VTIME] = 0
1948 | pollingread = copy.deepcopy([ iflag, oflag, cflag, lflag, ispeed, ospeed, cc ])
1949 |
1950 | ## Read with timeout (MIN == 0, TIME > 0) See termios(3).
1951 | cc[VMIN] = 0
1952 | cc[VTIME] = max(int(timeout*1+0.5), 1) # VTIME is in tenths of a second
1953 | readwithtimeout = copy.deepcopy([ iflag, oflag, cflag, lflag, ispeed, ospeed, cc ])
1954 |
1955 | # Drain stdin in case there's junk from a prev req in there already.
1956 | tcsetattr(sys.stdin.fileno(), TCSANOW, pollingread)
1957 | while posix.read(sys.stdin.fileno(), 1024):
1958 | debugprint(".", end='')
1959 |
1960 | output="" # String of output so far.
1961 | c = None # Currently read character.
1962 | try:
1963 | # Next read() should timeout if no byte becomes available.
1964 | tcsetattr(sys.stdin.fileno(), TCSANOW, readwithtimeout)
1965 |
1966 | print(seq, file=stderr, end='', flush=True) # Send Esc seq to terminal
1967 |
1968 | # Accumulate response in 'output' until delimiter or timeout
1969 | while c != delimiter:
1970 | c = posix.read(sys.stdin.fileno(), 1).decode()
1971 | if c:
1972 | output=output+c
1973 | # debugprint("got", repr(c), "waiting for", repr(delimiter))
1974 | else:
1975 | debugprint("read() returned 0 characters after", repr(seq))
1976 | # Timeout
1977 | break
1978 | finally:
1979 | tcsetattr(sys.stdin.fileno(), TCSANOW, oldmode)
1980 |
1981 | if (debug):
1982 | if output:
1983 | debugprint("Terminal query received: ", repr(output))
1984 |
1985 | if (c == delimiter):
1986 | debugprint("Exited on delimiter", repr(c))
1987 | else:
1988 | debugprint("Exited after timeout of", cc[VTIME]/10, "seconds")
1989 |
1990 | return(output)
1991 |
1992 | def terminal_query_list(seq, delimiter=None, timeout=0.2):
1993 | r"""
1994 | Same as terminal_query() above, but returns the results as a list.
1995 | Also removes known extraneous parts, such as Esc [ at start or Esc \ at end.
1996 | """
1997 |
1998 | output=terminal_query(seq, delimiter, timeout)
1999 |
2000 | if not output: return output
2001 |
2002 | # Example: read -r -p $'\033[c' -dc -t .2
2003 | # ^[[?63;1;2;4;6;9;15;16;22;28c
2004 |
2005 | if not delimiter and seq and len(seq)>0:
2006 | delimiter=seq[-1]
2007 |
2008 | # BUG: Not only ugly but wrong. Python strips by char, not string.
2009 | output = output.lstrip('\x1B[').lstrip('\x1B]').lstrip('\x1B').lstrip(';')
2010 | output = output.rstrip(delimiter).rstrip('\x1B\\')
2011 | output = output.split(';')
2012 |
2013 | return output
2014 |
2015 | def terminal_request_mode(n):
2016 | """
2017 | DECRQM: Request info on a mode by number.
2018 |
2019 | Mode 3 is DECCOLM (132 column mode)
2020 | Mode 5 is DECSCNM (reverse video mode)
2021 | Mode 7 is DECAWM (autowrap mode)
2022 | Mode 80 is DECSDM (sixel display mode)
2023 | """
2024 |
2025 | status=("not recognized" "set" "reset" "permanently set" "permanently reset")
2026 | output = terminal_query_list('\x1B[?'+str(n)+'$p', delimiter='y')
2027 | if not output: return output
2028 |
2029 | output=output[1].strip('$')
2030 | return status[int(output)]
2031 |
2032 | ######################################################################
2033 |
2034 | import atexit
2035 | atexit.register(cleanup) # When exiting, cleanup sixels
2036 |
2037 |
2038 | # VT340 is Latin-1 ISO8859-1 encoding.
2039 | # Python3 defaults to dying horribly on simple things like print('\u2020')
2040 | import sys
2041 | if sys.stdout:
2042 | try:
2043 | sys.stdout.reconfigure(errors='replace') # Print a ? instead of dying.
2044 | except AttributeError:
2045 | pass # Python<3.7 cannot reconfigure.
2046 |
2047 | ### Run the main routine
2048 | try:
2049 | main()
2050 | except BrokenPipeError:
2051 | # Ignore non-error errors. For example: 'ugrep -w pi | head'
2052 | sys.stdout = None;
2053 | cleanup()
2054 | except KeyboardInterrupt: # Catch ^C
2055 | cleanup()
2056 |
2057 | atexit.unregister(cleanup) # Normal exit, no need to cleanup.
2058 |
2059 | debugprint("Normal exit")
2060 |
2061 | # Implementation notes:
2062 |
2063 | # This is a rewrite of b9's AWK ugrep in Python. While AWK makes a lot
2064 | # more sense for what this program does (comparing fields based on
2065 | # regexps), a rewrite was necessary because GNU awk, while plenty
2066 | # powerful, uses \y for word edges instead of \b and that was bugging
2067 | # me. Gawk does this for backwards compatibility with historic AWK,
2068 | # which is all well and good, but gawk has no way to disable it for
2069 | # new scripts.
2070 | #
2071 | # Switching to Python did have the benefit of allowing more powerful
2072 | # Perl-like regexes (not that anyone has requested that).
2073 | #
2074 | # One downside is that I needed a huge hairy regex to simply search
2075 | # only in a certain field of each line. Maybe there's some Pythonic
2076 | # way to do it, but it's not obvious. Perhaps a 2D array?
2077 | #
2078 | # Also, I took for granted that awk let me use ^ and $ to search for
2079 | # the beginning and ending of fields instead of lines. I tried to
2080 | # reimplement that in Python, but it's not quite right as it only
2081 | # checks the first and last character. For example, ugrep "^x" works,
2082 | # but ugrep "(^x)" does not.
2083 |
2084 |
2085 | # Note: I do not use Python's `unicodedata` module because it is
2086 | # insufficient. It allows one to search only by precise character
2087 | # name: `unicodedata.lookup("ROTATED HEAVY BLACK HEART BULLET")`.
2088 |
2089 |
2090 | ################################################################################
2091 | # General_Category Values from Unicode TR 44, Table 10. #
2092 | ################################################################################
2093 | # Abbr Long Description #
2094 | # Lu Uppercase_Letter an uppercase letter #
2095 | # Ll Lowercase_Letter a lowercase letter #
2096 | # Lt Titlecase_Letter a digraphic character, with first part uppercase #
2097 | # Lm Modifier_Letter a modifier letter #
2098 | # Lo Other_Letter other letters, including syllables and ideographs #
2099 | # Mn Nonspacing_Mark a nonspacing combining mark (zero advance width) #
2100 | # Mc Spacing_Mark a spacing combining mark (positive advance width) #
2101 | # Me Enclosing_Mark an enclosing combining mark #
2102 | # Nd Decimal_Number a decimal digit #
2103 | # Nl Letter_Number a letterlike numeric character #
2104 | # No Other_Number a numeric character of other type #
2105 | # Pc Connector_Punctuation a connecting punctuation mark, like a tie #
2106 | # Pd Dash_Punctuation a dash or hyphen punctuation mark #
2107 | # Ps Open_Punctuation an opening punctuation mark (of a pair) #
2108 | # Pe Close_Punctuation a closing punctuation mark (of a pair) #
2109 | # Pi Initial_Punctuation an initial quotation mark #
2110 | # Pf Final_Punctuation a final quotation mark #
2111 | # Po Other_Punctuation a punctuation mark of other type #
2112 | # Sm Math_Symbol a symbol of primarily mathematical use #
2113 | # Sc Currency_Symbol a currency sign #
2114 | # Sk Modifier_Symbol a non-letterlike modifier symbol #
2115 | # So Other_Symbol a symbol of other type #
2116 | # Zs Space_Separator a space character (of various non-zero widths) #
2117 | # Zl Line_Separator U+2028 LINE SEPARATOR only #
2118 | # Zp Paragraph_Separator U+2029 PARAGRAPH SEPARATOR only #
2119 | # Cc Control a C0 or C1 control code #
2120 | # Cf Format a format control character #
2121 | # Cs Surrogate a surrogate code point #
2122 | # Co Private_Use a private-use character #
2123 | # Cn Unassigned a reserved unassigned code point or noncharacter #
2124 | ################################################################################
2125 |
2126 |
--------------------------------------------------------------------------------
3 |
4 |
5 | # ☙ ugrep ❧
6 |
7 | _Find unicode characters based on their names_
8 |
9 | ugrep is essentially [grep](https://www.gnu.org/software/grep/) for
10 | the Unicode table. It prints out the resulting unicode characters
11 | literally, so you can easily cut-and-paste. Ugrep is useful for
12 | looking up Emojis 😤, finding obscure symbols ⚸⅗ℏ℞☧☭, or beautiful
13 | glyphs to decorate your text. 🙶❡✯🟔❢🙷
14 |
15 | You can also use it for the reverse operation to lookup a single
16 | character (or a string of them) you've pasted into the terminal.
17 |
18 | As a bonus, it can list which fonts are installed that contain a
19 | particular unicode character and — through the magic of sixels — will
20 | show a rendering in each font.
21 |
22 | ## Installation
23 |
24 | It's just a Python 3 shell script. Download it to `/usr/local/bin` or `~/bin`
25 | and make it executable.
26 |
27 | cd /usr/local/bin
28 | wget https://github.com/hackerb9/ugrep/raw/master/ugrep
29 | chmod +x ugrep
30 |
31 | ## Usage
32 |
33 | * Search by name: **ugrep** [**-w**] _regex_
34 |
35 | Look up a character name where _regex_ is a regular
36 | expression. If you don't know [regular
37 | expressions](https://docs.python.org/3/howto/regex.html),
38 | don't worry. Just use plain strings and you'll rarely be
39 | wrong.
40 |
41 | ugrep runic
42 |
43 | If you find ugrep returning too many hits because the phrase you used
44 | is found in other terms, e.g., _thema_ found in _mathematical_, use
45 | the **-w** option to limit the search to complete words.
46 |
47 | * Search by number: **ugrep** _codepoint_**[..**_codepoint_**[..**_increment_**]]**
48 |
49 | Look up a character (or a range of them) using Unicode code points in
50 | hexadecimal. For example,
51 |
52 | ugrep 03c0
53 | ugrep 23b0..f
54 | ugrep 0..10ffff..1000
55 |
56 | * Search by character: **ugrep** [**-c**] _character string_
57 |
58 | Look up each character in a string. Note that if the string is a
59 | single character, e.g., `ugrep X`, then **-c** is implied and need not
60 | be specified.
61 |
62 | ugrep -c "(ﾟ∀ﾟ)"
63 |
64 | * List fonts for a character: **ugrep** [**-l**] _character_
65 |
66 | After showing the usual character information, list installed
67 | fonts that contain that character and show an example in each:
68 |
69 | ugrep -l mho
70 |
71 | ☝ *When `ssh`ed to another machine, `ugrep` shows the fonts
72 | installed on the remote machine.*
73 |
74 | * List fonts, scaled larger: **ugrep** [**-L** _scale_] _character_
75 |
76 | Same as `-l`, but scale up the example rendering in each font to
77 | be easier to read:
78 |
79 | ugrep -L2 -w om
80 |
81 | Useful scale values range from 2 to 8.
82 |
83 | ## Examples
84 |
85 | Note: output from all examples has been excerpted. (You'd be amazed
86 | how many heart emojis Unicode has. 😜)
87 |
88 | 
244 |
245 |
246 | ugrep -l swash amp
247 |
248 | * Requires FontConfig. (Most GNU/Linux boxes should already be set).
249 |
250 | * The requested character may also be displayed in each of the
251 | listed typefaces, but only if your terminal supports sixel
252 | graphics (e.g., `xterm -ti vt340`) and you have ImageMagick
253 | installed.
254 |
255 | ### Use -L to scale up the font examples when listing fonts
256 |
257 |
258 | 
261 |
262 |
263 | ```
264 | ugrep -L4 fdfd
265 | ﷽ U+FDFD ARABIC LIGATURE BISMILLAH AR-RAHMAN AR-RAHEEM
266 | Aldhabi
267 | Trutypewriter PolyglOTT
268 | Unifont
269 | ```
270 |
271 | * Note that increasing the glyph size also increased the text size.
272 | Not all terminals are capable of "double height" text. If yours
273 | shows two lines of the same text in the usual size, try using
274 | `--never-double-text`.
275 |
276 | ### Copy whitespace from the terminal
277 |
278 | $ ugrep -w space
279 | [ ] U+0020 SPACE (SP)
280 | [ ] U+00A0 NO-BREAK SPACE (non-breaking space) (NBSP)
281 | [ ] U+1680 OGHAM SPACE MARK
282 | [ ] U+2002 EN SPACE
283 | [ ] U+2003 EM SPACE
284 | [ ] U+2004 THREE-PER-EM SPACE
285 | [ ] U+2005 FOUR-PER-EM SPACE
286 | [ ] U+2006 SIX-PER-EM SPACE
287 | [ ] U+2007 FIGURE SPACE
288 | [ ] U+2008 PUNCTUATION SPACE
289 | [ ] U+2009 THIN SPACE
290 | [ ] U+200A HAIR SPACE
291 |
292 | Whitespace characters are printed with square brackets around them
293 | to make it easy to highlight and copy them from the terminal. They
294 | will also be shown with a yellow background, if the terminal allows.
295 |
296 | ### Determine if an alias is actually a correction
297 |
298 | Ugrep shows the character name in all caps and aliases are usually
299 | lowercase in parentheses. Some aliases are treated differently.
300 | For aesthetic reasons, abbreviations are also shown in uppercase.
301 | For example:
302 |
303 | � U+FEFF ZERO WIDTH NO-BREAK SPACE (byte order mark) (BOM) (ZWNBSP)
304 |
305 | There are 31 characters in Unicode which have the wrong name in the
306 | UnicodeData.txt database. Unicode includes the correct name as an
307 | alias in NameAliases.txt. If that file exists on your system, then
308 | ugrep will show the correction in Title Case Letters and in red
309 | letters, if the terminal supports color text.
310 |
311 | ︘ U+FE18 PRESENTATION FORM FOR VERTICAL RIGHT WHITE LENTICULAR BRAKCET (Presentation Form For Vertical Right White Lenticular Bracket)
312 |
313 | ### View CJK (Chinese-Japanese-Korean) characters
314 |
315 | Unicode does not actually define most CJK characters, except
316 | indirectly via Unihan, which maps certain blocks of characters to
317 | other standards.
318 |
319 | * Ugrep allows one to specify the code point or paste in an example
320 | character to look up.
321 |
322 | $ ugrep 𰻞
323 | 𰻞 U+30EDE biangbiang noodles ( M: biáng )
324 |
325 | $ ugrep 8000
326 | 耀 U+8000 shine, sparkle, dazzle; glory ( M: yào, C: jiu6, J: KAGAYAKU, K: YO )
327 |
328 |
329 |