├── .flake8 ├── .gitignore ├── LICENSE ├── Makefile ├── README.md ├── pyproject.toml ├── retronews.py ├── screenshot.png ├── tests.py └── tests ├── test_hn_01.html ├── test_hn_01.out ├── test_hn_02.html ├── test_hn_02.out ├── test_hn_03.html ├── test_hn_03.out ├── test_hn_04.html ├── test_hn_04.out ├── test_hn_05.html ├── test_hn_05.out ├── test_hn_06.html ├── test_hn_06.out ├── test_lb_01.html ├── test_lb_01.out ├── test_lb_02.html ├── test_lb_02.out ├── test_lb_03.html ├── test_lb_03.out ├── test_lb_04.html ├── test_lb_04.out ├── test_lb_05.html ├── test_lb_05.out ├── test_lb_06.html ├── test_lb_06.out ├── test_lb_07.html ├── test_lb_07.out ├── test_lb_08.html ├── test_lb_08.out ├── test_lb_09.html └── test_lb_09.out /.flake8: -------------------------------------------------------------------------------- 1 | [flake8] 2 | max-line-length = 120 3 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | /.mypy_cache 2 | /.venv 3 | /__pycache__ 4 | /cache/* 5 | /tmp/* 6 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 2, June 1991 3 | 4 | Copyright (C) 1989, 1991 Free Software Foundation, Inc., 5 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA 6 | Everyone is permitted to copy and distribute verbatim copies 7 | of this license document, but changing it is not allowed. 8 | 9 | Preamble 10 | 11 | The licenses for most software are designed to take away your 12 | freedom to share and change it. By contrast, the GNU General Public 13 | License is intended to guarantee your freedom to share and change free 14 | software--to make sure the software is free for all its users. This 15 | General Public License applies to most of the Free Software 16 | Foundation's software and to any other program whose authors commit to 17 | using it. (Some other Free Software Foundation software is covered by 18 | the GNU Lesser General Public License instead.) You can apply it to 19 | your programs, too. 20 | 21 | When we speak of free software, we are referring to freedom, not 22 | price. Our General Public Licenses are designed to make sure that you 23 | have the freedom to distribute copies of free software (and charge for 24 | this service if you wish), that you receive source code or can get it 25 | if you want it, that you can change the software or use pieces of it 26 | in new free programs; and that you know you can do these things. 27 | 28 | To protect your rights, we need to make restrictions that forbid 29 | anyone to deny you these rights or to ask you to surrender the rights. 30 | These restrictions translate to certain responsibilities for you if you 31 | distribute copies of the software, or if you modify it. 32 | 33 | For example, if you distribute copies of such a program, whether 34 | gratis or for a fee, you must give the recipients all the rights that 35 | you have. You must make sure that they, too, receive or can get the 36 | source code. And you must show them these terms so they know their 37 | rights. 38 | 39 | We protect your rights with two steps: (1) copyright the software, and 40 | (2) offer you this license which gives you legal permission to copy, 41 | distribute and/or modify the software. 42 | 43 | Also, for each author's protection and ours, we want to make certain 44 | that everyone understands that there is no warranty for this free 45 | software. If the software is modified by someone else and passed on, we 46 | want its recipients to know that what they have is not the original, so 47 | that any problems introduced by others will not reflect on the original 48 | authors' reputations. 49 | 50 | Finally, any free program is threatened constantly by software 51 | patents. We wish to avoid the danger that redistributors of a free 52 | program will individually obtain patent licenses, in effect making the 53 | program proprietary. To prevent this, we have made it clear that any 54 | patent must be licensed for everyone's free use or not licensed at all. 55 | 56 | The precise terms and conditions for copying, distribution and 57 | modification follow. 58 | 59 | GNU GENERAL PUBLIC LICENSE 60 | TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 61 | 62 | 0. This License applies to any program or other work which contains 63 | a notice placed by the copyright holder saying it may be distributed 64 | under the terms of this General Public License. The "Program", below, 65 | refers to any such program or work, and a "work based on the Program" 66 | means either the Program or any derivative work under copyright law: 67 | that is to say, a work containing the Program or a portion of it, 68 | either verbatim or with modifications and/or translated into another 69 | language. (Hereinafter, translation is included without limitation in 70 | the term "modification".) Each licensee is addressed as "you". 71 | 72 | Activities other than copying, distribution and modification are not 73 | covered by this License; they are outside its scope. The act of 74 | running the Program is not restricted, and the output from the Program 75 | is covered only if its contents constitute a work based on the 76 | Program (independent of having been made by running the Program). 77 | Whether that is true depends on what the Program does. 78 | 79 | 1. You may copy and distribute verbatim copies of the Program's 80 | source code as you receive it, in any medium, provided that you 81 | conspicuously and appropriately publish on each copy an appropriate 82 | copyright notice and disclaimer of warranty; keep intact all the 83 | notices that refer to this License and to the absence of any warranty; 84 | and give any other recipients of the Program a copy of this License 85 | along with the Program. 86 | 87 | You may charge a fee for the physical act of transferring a copy, and 88 | you may at your option offer warranty protection in exchange for a fee. 89 | 90 | 2. You may modify your copy or copies of the Program or any portion 91 | of it, thus forming a work based on the Program, and copy and 92 | distribute such modifications or work under the terms of Section 1 93 | above, provided that you also meet all of these conditions: 94 | 95 | a) You must cause the modified files to carry prominent notices 96 | stating that you changed the files and the date of any change. 97 | 98 | b) You must cause any work that you distribute or publish, that in 99 | whole or in part contains or is derived from the Program or any 100 | part thereof, to be licensed as a whole at no charge to all third 101 | parties under the terms of this License. 102 | 103 | c) If the modified program normally reads commands interactively 104 | when run, you must cause it, when started running for such 105 | interactive use in the most ordinary way, to print or display an 106 | announcement including an appropriate copyright notice and a 107 | notice that there is no warranty (or else, saying that you provide 108 | a warranty) and that users may redistribute the program under 109 | these conditions, and telling the user how to view a copy of this 110 | License. (Exception: if the Program itself is interactive but 111 | does not normally print such an announcement, your work based on 112 | the Program is not required to print an announcement.) 113 | 114 | These requirements apply to the modified work as a whole. If 115 | identifiable sections of that work are not derived from the Program, 116 | and can be reasonably considered independent and separate works in 117 | themselves, then this License, and its terms, do not apply to those 118 | sections when you distribute them as separate works. But when you 119 | distribute the same sections as part of a whole which is a work based 120 | on the Program, the distribution of the whole must be on the terms of 121 | this License, whose permissions for other licensees extend to the 122 | entire whole, and thus to each and every part regardless of who wrote it. 123 | 124 | Thus, it is not the intent of this section to claim rights or contest 125 | your rights to work written entirely by you; rather, the intent is to 126 | exercise the right to control the distribution of derivative or 127 | collective works based on the Program. 128 | 129 | In addition, mere aggregation of another work not based on the Program 130 | with the Program (or with a work based on the Program) on a volume of 131 | a storage or distribution medium does not bring the other work under 132 | the scope of this License. 133 | 134 | 3. You may copy and distribute the Program (or a work based on it, 135 | under Section 2) in object code or executable form under the terms of 136 | Sections 1 and 2 above provided that you also do one of the following: 137 | 138 | a) Accompany it with the complete corresponding machine-readable 139 | source code, which must be distributed under the terms of Sections 140 | 1 and 2 above on a medium customarily used for software interchange; or, 141 | 142 | b) Accompany it with a written offer, valid for at least three 143 | years, to give any third party, for a charge no more than your 144 | cost of physically performing source distribution, a complete 145 | machine-readable copy of the corresponding source code, to be 146 | distributed under the terms of Sections 1 and 2 above on a medium 147 | customarily used for software interchange; or, 148 | 149 | c) Accompany it with the information you received as to the offer 150 | to distribute corresponding source code. (This alternative is 151 | allowed only for noncommercial distribution and only if you 152 | received the program in object code or executable form with such 153 | an offer, in accord with Subsection b above.) 154 | 155 | The source code for a work means the preferred form of the work for 156 | making modifications to it. For an executable work, complete source 157 | code means all the source code for all modules it contains, plus any 158 | associated interface definition files, plus the scripts used to 159 | control compilation and installation of the executable. However, as a 160 | special exception, the source code distributed need not include 161 | anything that is normally distributed (in either source or binary 162 | form) with the major components (compiler, kernel, and so on) of the 163 | operating system on which the executable runs, unless that component 164 | itself accompanies the executable. 165 | 166 | If distribution of executable or object code is made by offering 167 | access to copy from a designated place, then offering equivalent 168 | access to copy the source code from the same place counts as 169 | distribution of the source code, even though third parties are not 170 | compelled to copy the source along with the object code. 171 | 172 | 4. You may not copy, modify, sublicense, or distribute the Program 173 | except as expressly provided under this License. Any attempt 174 | otherwise to copy, modify, sublicense or distribute the Program is 175 | void, and will automatically terminate your rights under this License. 176 | However, parties who have received copies, or rights, from you under 177 | this License will not have their licenses terminated so long as such 178 | parties remain in full compliance. 179 | 180 | 5. You are not required to accept this License, since you have not 181 | signed it. However, nothing else grants you permission to modify or 182 | distribute the Program or its derivative works. These actions are 183 | prohibited by law if you do not accept this License. Therefore, by 184 | modifying or distributing the Program (or any work based on the 185 | Program), you indicate your acceptance of this License to do so, and 186 | all its terms and conditions for copying, distributing or modifying 187 | the Program or works based on it. 188 | 189 | 6. Each time you redistribute the Program (or any work based on the 190 | Program), the recipient automatically receives a license from the 191 | original licensor to copy, distribute or modify the Program subject to 192 | these terms and conditions. You may not impose any further 193 | restrictions on the recipients' exercise of the rights granted herein. 194 | You are not responsible for enforcing compliance by third parties to 195 | this License. 196 | 197 | 7. If, as a consequence of a court judgment or allegation of patent 198 | infringement or for any other reason (not limited to patent issues), 199 | conditions are imposed on you (whether by court order, agreement or 200 | otherwise) that contradict the conditions of this License, they do not 201 | excuse you from the conditions of this License. If you cannot 202 | distribute so as to satisfy simultaneously your obligations under this 203 | License and any other pertinent obligations, then as a consequence you 204 | may not distribute the Program at all. For example, if a patent 205 | license would not permit royalty-free redistribution of the Program by 206 | all those who receive copies directly or indirectly through you, then 207 | the only way you could satisfy both it and this License would be to 208 | refrain entirely from distribution of the Program. 209 | 210 | If any portion of this section is held invalid or unenforceable under 211 | any particular circumstance, the balance of the section is intended to 212 | apply and the section as a whole is intended to apply in other 213 | circumstances. 214 | 215 | It is not the purpose of this section to induce you to infringe any 216 | patents or other property right claims or to contest validity of any 217 | such claims; this section has the sole purpose of protecting the 218 | integrity of the free software distribution system, which is 219 | implemented by public license practices. Many people have made 220 | generous contributions to the wide range of software distributed 221 | through that system in reliance on consistent application of that 222 | system; it is up to the author/donor to decide if he or she is willing 223 | to distribute software through any other system and a licensee cannot 224 | impose that choice. 225 | 226 | This section is intended to make thoroughly clear what is believed to 227 | be a consequence of the rest of this License. 228 | 229 | 8. If the distribution and/or use of the Program is restricted in 230 | certain countries either by patents or by copyrighted interfaces, the 231 | original copyright holder who places the Program under this License 232 | may add an explicit geographical distribution limitation excluding 233 | those countries, so that distribution is permitted only in or among 234 | countries not thus excluded. In such case, this License incorporates 235 | the limitation as if written in the body of this License. 236 | 237 | 9. The Free Software Foundation may publish revised and/or new versions 238 | of the General Public License from time to time. Such new versions will 239 | be similar in spirit to the present version, but may differ in detail to 240 | address new problems or concerns. 241 | 242 | Each version is given a distinguishing version number. If the Program 243 | specifies a version number of this License which applies to it and "any 244 | later version", you have the option of following the terms and conditions 245 | either of that version or of any later version published by the Free 246 | Software Foundation. If the Program does not specify a version number of 247 | this License, you may choose any version ever published by the Free Software 248 | Foundation. 249 | 250 | 10. If you wish to incorporate parts of the Program into other free 251 | programs whose distribution conditions are different, write to the author 252 | to ask for permission. For software which is copyrighted by the Free 253 | Software Foundation, write to the Free Software Foundation; we sometimes 254 | make exceptions for this. Our decision will be guided by the two goals 255 | of preserving the free status of all derivatives of our free software and 256 | of promoting the sharing and reuse of software generally. 257 | 258 | NO WARRANTY 259 | 260 | 11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY 261 | FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN 262 | OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES 263 | PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED 264 | OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 265 | MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS 266 | TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE 267 | PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, 268 | REPAIR OR CORRECTION. 269 | 270 | 12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 271 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR 272 | REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, 273 | INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING 274 | OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED 275 | TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY 276 | YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER 277 | PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE 278 | POSSIBILITY OF SUCH DAMAGES. 279 | 280 | END OF TERMS AND CONDITIONS 281 | 282 | How to Apply These Terms to Your New Programs 283 | 284 | If you develop a new program, and you want it to be of the greatest 285 | possible use to the public, the best way to achieve this is to make it 286 | free software which everyone can redistribute and change under these terms. 287 | 288 | To do so, attach the following notices to the program. It is safest 289 | to attach them to the start of each source file to most effectively 290 | convey the exclusion of warranty; and each file should have at least 291 | the "copyright" line and a pointer to where the full notice is found. 292 | 293 | 294 | Copyright (C) 295 | 296 | This program is free software; you can redistribute it and/or modify 297 | it under the terms of the GNU General Public License as published by 298 | the Free Software Foundation; either version 2 of the License, or 299 | (at your option) any later version. 300 | 301 | This program is distributed in the hope that it will be useful, 302 | but WITHOUT ANY WARRANTY; without even the implied warranty of 303 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 304 | GNU General Public License for more details. 305 | 306 | You should have received a copy of the GNU General Public License along 307 | with this program; if not, write to the Free Software Foundation, Inc., 308 | 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. 309 | 310 | Also add information on how to contact you by electronic and paper mail. 311 | 312 | If the program is interactive, make it output a short notice like this 313 | when it starts in an interactive mode: 314 | 315 | Gnomovision version 69, Copyright (C) year name of author 316 | Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 317 | This is free software, and you are welcome to redistribute it 318 | under certain conditions; type `show c' for details. 319 | 320 | The hypothetical commands `show w' and `show c' should show the appropriate 321 | parts of the General Public License. Of course, the commands you use may 322 | be called something other than `show w' and `show c'; they could even be 323 | mouse-clicks or menu items--whatever suits your program. 324 | 325 | You should also get your employer (if you work as a programmer) or your 326 | school, if any, to sign a "copyright disclaimer" for the program, if 327 | necessary. Here is a sample; alter the names: 328 | 329 | Yoyodyne, Inc., hereby disclaims all copyright interest in the program 330 | `Gnomovision' (which makes passes at compilers) written by James Hacker. 331 | 332 | , 1 April 1989 333 | Ty Coon, President of Vice 334 | 335 | This General Public License does not permit incorporating your program into 336 | proprietary programs. If your program is a subroutine library, you may 337 | consider it more useful to permit linking proprietary applications with the 338 | library. If this is what you want to do, use the GNU Lesser General 339 | Public License instead of this License. 340 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | SCRIPTS = retronews.py tests.py 2 | 3 | .PHONY: venv-check 4 | venv-check: 5 | ifeq ("$(VIRTUAL_ENV)","") 6 | @echo "Venv is not active, run make venv-build and \`make venv-activate\`" 7 | @echo 8 | exit 1 9 | endif 10 | 11 | venv-build: 12 | python3 -m venv .venv 13 | 14 | venv-activate: 15 | @echo "source .venv/bin/activate" 16 | 17 | install: venv-check 18 | pip install black isort flake8 mypy 19 | 20 | lint: venv-check 21 | black $(SCRIPTS) 22 | isort $(SCRIPTS) 23 | flake8 $(SCRIPTS) 24 | mypy $(SCRIPTS) 25 | 26 | test: 27 | python3 tests.py 28 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # retronews 2 | 3 | A Python script for browsing [Hacker News](https://news.ycombinator.com/) 4 | and [Lobsters](https://lobste.rs/) discussions with a nostalgic interface emulating 5 | classical usenet and mail readers, like slrn and mutt. 6 | 7 | It was primarily written so I could highlight interesting threads and keep 8 | track of read / unread messages (see [blog post](https://luke8086.dev/retronews.html)). 9 | The UI showing one message at a time also encourages slower, more focused reading. 10 | 11 | On Unix-like systems it only depends on Python 3.9. On Windows you may also need 12 | to install [windows-curses](https://pypi.org/project/windows-curses/). 13 | 14 | It doesn't require installation. You can run it simply with: 15 | 16 | ```bash 17 | $ curl -LO https://raw.githubusercontent.com/luke8086/retronews/main/retronews.py 18 | $ python3 ./retronews.py 19 | ``` 20 | 21 | Press `?` to see available keybindings. 22 | 23 | 24 | 25 | ## Customization 26 | 27 | To customize retronews without directly editing the script, you can put any valid 28 | Python code in `~/.retronewsrc.py` (or other location specified with `--rcfile`) to 29 | be executed on startup. For example: 30 | 31 | ```python 32 | # Ignore type warnings 33 | from typing import Any 34 | retronews: Any 35 | 36 | # Example: Custom key bindings 37 | retronews.KEY_BINDINGS[ord('a')] = lambda app: retronews.cmd_prev(app) 38 | retronews.KEY_BINDINGS[ord('z')] = lambda app: retronews.cmd_next(app) 39 | 40 | # Example: Custom colors 41 | retronews.COLORS['author'] = (retronews.curses.COLOR_RED, -1) 42 | ``` 43 | 44 | ## Known issues and limitations 45 | 46 | - The reader is read-only, there are no plans to support voting and posting 47 | - Message formatting is not perfect, but works well enough most of the time 48 | - Detecting if threads contain unread responses works by only checking their 49 | count, it's not reliable if any responses were deleted 50 | 51 | ## Why not an NNTP gateway? 52 | 53 | NNTP doesn't support browsing threads by title (let alone paginated) and requesting 54 | their messages on demand. Clients need to fetch metadata of all available messages 55 | in all available threads in advance. Given the volume of messages on HN, synchronizing 56 | them to the gateway is not practical. Even when attempted, some clients struggle 57 | with the sheer number of messages in a single group. 58 | 59 | ## Related projects 60 | 61 | - [HN Search @ Algolia](https://hn.algolia.com/about) - The underlying API used to retrieve messages 62 | - [nntpit](https://github.com/taviso/nntpit) - An NNTP gateway to reddit.com 63 | - [circumflex](https://github.com/bensadeh/circumflex) - Another, more advanced TUI for HN 64 | -------------------------------------------------------------------------------- /pyproject.toml: -------------------------------------------------------------------------------- 1 | [tool.isort] 2 | profile = "black" 3 | 4 | [tool.black] 5 | line-length = 120 6 | -------------------------------------------------------------------------------- /retronews.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | # 3 | # Copyright (c) luke8086 4 | # 5 | # This program is free software: you can redistribute it and/or modify 6 | # it under the terms of the GNU General Public License version 2 as published by 7 | # the Free Software Foundation. 8 | # 9 | 10 | import sys 11 | 12 | if sys.version_info < (3, 9): 13 | sys.stderr.write("Python 3.9 or newer is required.\n") 14 | sys.exit(1) 15 | 16 | import argparse 17 | import curses 18 | import curses.textpad 19 | import dataclasses 20 | import html.parser 21 | import json 22 | import logging 23 | import os 24 | import re 25 | import sqlite3 26 | import traceback 27 | import unicodedata 28 | import urllib.request 29 | import webbrowser 30 | from collections import defaultdict 31 | from datetime import datetime 32 | from functools import partial, reduce 33 | from textwrap import wrap 34 | from typing import ( 35 | TYPE_CHECKING, 36 | Any, 37 | Callable, 38 | Generator, 39 | Literal, 40 | NewType, 41 | Optional, 42 | TypedDict, 43 | TypeVar, 44 | Union, 45 | cast, 46 | overload, 47 | ) 48 | 49 | KEY_BINDINGS: dict[int, Callable[["AppState"], None]] = { 50 | ord("q"): lambda app: cmd_quit(app), 51 | ord("?"): lambda app: cmd_help(app), 52 | ord("\n"): lambda app: cmd_open(app), 53 | ord(" "): lambda app: cmd_open(app), 54 | ord("o"): lambda app: cmd_show_links(app), 55 | ord("x"): lambda app: cmd_close(app), 56 | ord("s"): lambda app: cmd_star(app), 57 | ord("S"): lambda app: cmd_star_thread(app), 58 | ord("u"): lambda app: cmd_set_unread(app), 59 | ord("D"): lambda app: cmd_dump(app), 60 | ord("r"): lambda app: cmd_toggle_raw_mode(app), 61 | ord("k"): lambda app: cmd_up(app), 62 | ord("j"): lambda app: cmd_down(app), 63 | ord("p"): lambda app: cmd_prev(app), 64 | ord("n"): lambda app: cmd_next(app), 65 | ord("N"): lambda app: cmd_next_unread(app), 66 | ord("P"): lambda app: cmd_parent(app), 67 | ord(";"): lambda app: cmd_mark_set(app), 68 | ord(","): lambda app: cmd_mark_jump(app), 69 | ord("R"): lambda app: cmd_reload_page(app), 70 | ord("<"): lambda app: cmd_load_prev_page(app), 71 | ord(">"): lambda app: cmd_load_next_page(app), 72 | ord("g"): lambda app: cmd_load_page(app), 73 | curses.KEY_UP: lambda app: cmd_prev(app), 74 | curses.KEY_DOWN: lambda app: cmd_next(app), 75 | curses.KEY_PPAGE: lambda app: cmd_page_up(app), 76 | curses.KEY_NPAGE: lambda app: cmd_page_down(app), 77 | curses.KEY_RESIZE: lambda app: cmd_resize(app), 78 | } | {ord(str(i)): lambda app, i=i: cmd_load_tab(app, i) for i in range(1, 10)} 79 | 80 | HELP_MENU = "q:Quit ?:Help p:Prev n:Next N:Next-Unread j:Down k:Up x:Close s:Star" 81 | 82 | HELP_SCREEN = """\ 83 | q Quit retronews 84 | UP, DOWN Go up / down by one message / pager line 85 | PG UP, PG DOWN Gp up / down by one page of messages / pager lines 86 | p, n Go to previous / next message 87 | N Go to next unread message 88 | P Go to parent message 89 | ; , Set mark, jump to mark & swap (valid within thread) 90 | RETURN, SPACE Open selected message 91 | x Close current message / thread 92 | o Select link and open in browser 93 | 1 - 9 Change group 94 | R Refresh current page 95 | < > Go to previous / next page 96 | g Go to specific page 97 | k j Scroll pager up / down by one line 98 | s Star / unstar current message 99 | S Star / unstar current thread 100 | u Mark current message as unread 101 | r Toggle raw HTML mode 102 | 103 | See https://github.com/luke8086/retronews for more information.""" 104 | 105 | Color = Literal[ 106 | "author", 107 | "code", 108 | "cursor", 109 | "date", 110 | "default", 111 | "empty_pager_line", 112 | "deleted_message_pager_line", 113 | "menu", 114 | "menu_active", 115 | "nested_quote", 116 | "quote", 117 | "starred_subject", 118 | "header_subject", 119 | "tree", 120 | "unread_comments", 121 | "url", 122 | ] 123 | 124 | COLORS: dict[Color, tuple[int, int]] = { 125 | "author": (curses.COLOR_YELLOW, -1), 126 | "code": (curses.COLOR_GREEN, -1), 127 | "cursor": (curses.COLOR_BLACK, curses.COLOR_CYAN), 128 | "date": (curses.COLOR_CYAN, -1), 129 | "default": (curses.COLOR_WHITE, -1), 130 | "empty_pager_line": (curses.COLOR_GREEN, -1), 131 | "deleted_message_pager_line": (curses.COLOR_RED, -1), 132 | "menu": (curses.COLOR_GREEN, curses.COLOR_BLUE), 133 | "menu_active": (curses.COLOR_YELLOW, curses.COLOR_BLUE), 134 | "nested_quote": (curses.COLOR_CYAN, -1), 135 | "quote": (curses.COLOR_YELLOW, -1), 136 | "starred_subject": (curses.COLOR_CYAN, -1), 137 | "header_subject": (curses.COLOR_GREEN, -1), 138 | "tree": (curses.COLOR_RED, -1), 139 | "unread_comments": (curses.COLOR_GREEN, -1), 140 | "url": (curses.COLOR_MAGENTA, -1), 141 | } 142 | 143 | REQUEST_TIMEOUT = 10 144 | 145 | # Recognize ">text", "> text", ">>text", ">> text", etc. 146 | QUOTE_REX = re.compile(r"^(> ?)+") 147 | 148 | # Recognize "[n] link", "[n]: link", "[n] - link", etc. 149 | REFERENCE_REX = re.compile(r"^\[\d+\][ :-]*https?://[^ ]*$") 150 | 151 | # Recognize http/https URLs 152 | URL_REX = re.compile(r"(https?://[^\s\)\"<,]+[^\s\)\"<,\.])") 153 | 154 | # Recognize HN message URLs 155 | HN_URL_REX = re.compile(r"^https://news\.ycombinator\.com/item\?id=(\d+)$") 156 | 157 | # FIXME: Use TypeAlias after migrating to Python 3.10 158 | DB = NewType("DB", "sqlite3.Connection") 159 | 160 | if TYPE_CHECKING: 161 | from _curses import _CursesWindow 162 | 163 | Window = _CursesWindow 164 | else: 165 | Window = Any 166 | 167 | T = TypeVar("T") 168 | 169 | 170 | class ExitException(Exception): 171 | code: int 172 | message: str 173 | 174 | def __init__(self, code: int = 0, message: str = ""): 175 | self.code = code 176 | self.message = message 177 | 178 | super().__init__(message) 179 | 180 | 181 | @dataclasses.dataclass(frozen=True) 182 | class Provider: 183 | fetch_thread: Callable[[str], "Message"] 184 | fetch_threads_by_id: Callable[[list[str]], list["Message"]] 185 | 186 | 187 | PROVIDERS: dict[str, Provider] = { 188 | "hn": Provider( 189 | fetch_thread=lambda msg_id: hn_fetch_thread(msg_id), 190 | fetch_threads_by_id=lambda msg_ids: hn_fetch_threads_by_id(msg_ids), 191 | ), 192 | "lb": Provider( 193 | fetch_thread=lambda msg_id: lb_fetch_thread(msg_id), 194 | fetch_threads_by_id=lambda msg_ids: [lb_fetch_thread(x) for x in msg_ids], 195 | ), 196 | } 197 | 198 | 199 | @dataclasses.dataclass(frozen=True) 200 | class Group: 201 | label: str 202 | fetch: Callable[[DB, int], list["Message"]] 203 | page: int = 1 204 | 205 | 206 | GROUP_TABS: list[Group] = [ 207 | Group(label="Front HN", fetch=lambda db, page: hn_fetch_threads("news", page)), 208 | Group(label="New HN", fetch=lambda db, page: hn_fetch_new_threads(page)), 209 | Group(label="Ask HN", fetch=lambda db, page: hn_fetch_threads("ask", page)), 210 | Group(label="Show HN", fetch=lambda db, page: hn_fetch_threads("show", page)), 211 | Group(label="Front LB", fetch=lambda db, page: lb_fetch_threads("", page)), 212 | Group(label="New LB", fetch=lambda db, page: lb_fetch_threads("newest", page)), 213 | Group(label="Starred", fetch=lambda db, page: group_fetch_starred_threads(db, page)), 214 | ] 215 | 216 | 217 | @dataclasses.dataclass 218 | class MessageFlags: 219 | read: bool = False 220 | starred: bool = False 221 | 222 | 223 | @dataclasses.dataclass 224 | class Message: 225 | msg_id: str 226 | thread_id: str 227 | content_location: str 228 | date: datetime 229 | author: Optional[str] 230 | title: str 231 | body: Optional[str] = None 232 | lines: list[str] = dataclasses.field(default_factory=list) 233 | parent: Optional["Message"] = None 234 | children: Optional[list["Message"]] = None 235 | flags: MessageFlags = dataclasses.field(default_factory=MessageFlags) 236 | read_comments: int = 0 237 | total_comments: int = 0 238 | index_position: int = 0 239 | index_tree: str = "" 240 | 241 | @property 242 | def is_read(self) -> bool: 243 | return self.flags.read 244 | 245 | @property 246 | def is_shown_as_read(self) -> bool: 247 | # If the message is an unloaded thread, check if all comments are read 248 | return self.read_comments >= self.total_comments if self.is_thread and self.children is None else self.is_read 249 | 250 | @property 251 | def is_thread(self) -> bool: 252 | return self.msg_id == self.thread_id 253 | 254 | @property 255 | def is_deleted(self) -> bool: 256 | return self.author is None 257 | 258 | 259 | @dataclasses.dataclass 260 | class Layout: 261 | lines: int = 0 262 | cols: int = 0 263 | top_menu_row: int = 0 264 | index_start: int = 1 265 | index_height: int = 0 266 | middle_menu_row: Optional[int] = None 267 | pager_start: Optional[int] = None 268 | pager_height: Optional[int] = None 269 | bottom_menu_row: int = 0 270 | flash_menu_row: int = 0 271 | 272 | 273 | @dataclasses.dataclass 274 | class AppState: 275 | screen: Window 276 | db: DB 277 | group: Group 278 | ascii: bool = False 279 | monochrome: bool = False 280 | colors: dict[Color, int] = dataclasses.field(default_factory=dict) 281 | messages: list[Message] = dataclasses.field(default_factory=list) 282 | messages_by_id: dict[str, Message] = dataclasses.field(default_factory=dict) 283 | selected_message: Optional[Message] = None 284 | marked_message_id: Optional[str] = None 285 | layout: Layout = dataclasses.field(default_factory=Layout) 286 | pager_visible: bool = False 287 | pager_offset: int = 0 288 | raw_mode: bool = False 289 | flash: Optional[str] = None 290 | 291 | 292 | class HNSearchHit(TypedDict): 293 | objectID: int 294 | author: Optional[str] 295 | title: str 296 | created_at_i: int 297 | story_text: Optional[str] 298 | url: Optional[str] 299 | num_comments: int 300 | 301 | 302 | class HNEntry(TypedDict): 303 | author: Optional[str] 304 | # FIXME: Recursive declarations are not yet supported in TypedDicts 305 | children: list[Any] 306 | created_at_i: int 307 | id: int 308 | parent_id: Optional[int] 309 | text: Optional[str] 310 | title: Optional[str] 311 | url: Optional[str] 312 | 313 | 314 | class LBThread(TypedDict): 315 | short_id: str 316 | short_id_url: str 317 | created_at: str 318 | title: str 319 | url: str 320 | description: str 321 | submitter_user: str 322 | comment_count: int 323 | comments: Optional[list[Any]] 324 | 325 | 326 | class LBComment(TypedDict): 327 | short_id: str 328 | created_at: str 329 | url: str 330 | commenting_user: str 331 | parent_comment: Optional[str] 332 | 333 | 334 | HTML_BLOCK_TAGS = set(("root", "p", "pre", "blockquote", "ul", "ol", "li", "hr")) 335 | HTML_INLINE_TAGS = set(("code", "a", "em", "strong", "b", "br")) 336 | HTML_KNOWN_TAGS = HTML_BLOCK_TAGS.union(HTML_INLINE_TAGS) 337 | HTML_AUTOCLOSE_TAGS = set(("hr", "br")) 338 | 339 | 340 | @dataclasses.dataclass 341 | class HTMLNode: 342 | tag: str 343 | 344 | parent: Optional["HTMLNode"] = None 345 | 346 | prev_sibling: Optional["HTMLNode"] = None 347 | next_sibling: Optional["HTMLNode"] = None 348 | 349 | first_child: Optional["HTMLNode"] = None 350 | last_child: Optional["HTMLNode"] = None 351 | 352 | attrs: dict[str, Optional[str]] = dataclasses.field(default_factory=dict) 353 | text: str = "" 354 | pre: bool = False 355 | 356 | 357 | class HTMLParser(html.parser.HTMLParser): 358 | root_node: HTMLNode 359 | current_node: HTMLNode 360 | pre_level = 0 361 | 362 | def __init__(self): 363 | super().__init__() 364 | 365 | self.root_node = self.current_node = HTMLNode(tag="root") 366 | 367 | def handle_data(self, data: str) -> None: 368 | if self.current_node.tag in HTML_AUTOCLOSE_TAGS: 369 | self.handle_endtag(self.current_node.tag) 370 | 371 | node = HTMLNode(tag="text", text=data, pre=self.pre_level > 0) 372 | html_node_append(self.current_node, node) 373 | 374 | def handle_starttag(self, tag: str, attrs: list[tuple[str, Optional[str]]]) -> None: 375 | if self.current_node.tag in HTML_AUTOCLOSE_TAGS: 376 | self.handle_endtag(self.current_node.tag) 377 | 378 | if tag not in HTML_KNOWN_TAGS: 379 | return 380 | 381 | if tag == "pre": 382 | self.pre_level += 1 383 | 384 | node = HTMLNode(tag=tag, attrs=dict(attrs), pre=self.pre_level > 0) 385 | html_node_append(self.current_node, node) 386 | self.current_node = node 387 | 388 | def handle_endtag(self, tag: str) -> None: 389 | if tag not in HTML_KNOWN_TAGS: 390 | return 391 | 392 | if tag == "pre": 393 | self.pre_level = max(0, self.pre_level - 1) 394 | 395 | while True: 396 | node = self.current_node 397 | 398 | if node.parent is None: 399 | break 400 | 401 | self.current_node = node.parent 402 | 403 | if node.tag == tag: 404 | break 405 | 406 | 407 | def text_wrap(text: str, width=70) -> str: 408 | if len(text) == 0: 409 | # Preserve empty lines 410 | return "" 411 | 412 | if text.startswith(" "): 413 | # Preserve code indentation 414 | return text 415 | 416 | if REFERENCE_REX.match(text): 417 | # Keep reference numbers with long links in the same line 418 | return text 419 | 420 | indent = "" 421 | 422 | if (match := QUOTE_REX.match(text)) is not None: 423 | # Preserve quotation symbols in subsequent lines 424 | indent = match[0] 425 | 426 | lines = wrap(text, width, subsequent_indent=indent, break_on_hyphens=False, break_long_words=False) 427 | lines = [line.rstrip() for line in lines] 428 | 429 | return "\n".join(lines) 430 | 431 | 432 | def text_clean(text: str, ascii: bool = False) -> str: 433 | """Cleanup text for rendering, currently only removes non-ascii characters in ascii mode""" 434 | 435 | if ascii: 436 | text = text.encode("ascii", "replace").decode("ascii") 437 | 438 | return text 439 | 440 | 441 | def text_unindent(text: str) -> str: 442 | lines = text.split("\n") 443 | 444 | while all(line.startswith(" ") or line == "" for line in lines): 445 | lines = [line[1:] for line in lines] 446 | 447 | return "\n".join(lines) 448 | 449 | 450 | def text_sanitize(text: Optional[str]) -> str: 451 | # For safety, remove any control characters except for \n and \t 452 | # At least on HN some messages contain \x00 characters 453 | 454 | text = text or "" 455 | allowed_cc = set(("\n", "\t")) 456 | chars = (c for c in text if c in allowed_cc or unicodedata.category(c) != "Cc") 457 | text = "".join(chars) 458 | 459 | return text 460 | 461 | 462 | def text_split_urls(text: str) -> list[str]: 463 | return [p for p in URL_REX.split(text) if p != ""] 464 | 465 | 466 | def html_node_children(parent: HTMLNode) -> list[HTMLNode]: 467 | ret: list[HTMLNode] = [] 468 | node = parent.first_child 469 | 470 | while node is not None: 471 | ret.append(node) 472 | node = node.next_sibling 473 | 474 | return ret 475 | 476 | 477 | def html_node_append(parent: HTMLNode, child: HTMLNode) -> None: 478 | if parent.first_child is None: 479 | parent.first_child = child 480 | 481 | if parent.last_child is not None: 482 | parent.last_child.next_sibling = child 483 | child.prev_sibling = parent.last_child 484 | 485 | parent.last_child = child 486 | child.parent = parent 487 | 488 | 489 | def html_node_unlink(node: HTMLNode) -> None: 490 | if node.prev_sibling: 491 | node.prev_sibling.next_sibling = node.next_sibling 492 | 493 | if node.next_sibling: 494 | node.next_sibling.prev_sibling = node.prev_sibling 495 | 496 | if node.parent and node.parent.first_child is node: 497 | node.parent.first_child = node.next_sibling 498 | 499 | if node.parent and node.parent.last_child is node: 500 | node.parent.last_child = node.prev_sibling 501 | 502 | node.parent = node.prev_sibling = node.next_sibling = None 503 | 504 | 505 | def html_node_dump(node: HTMLNode) -> str: 506 | lines = [] 507 | lines.append(f"{node.tag} {repr(node.attrs)}") 508 | 509 | for child in html_node_children(node): 510 | if child.tag == "text": 511 | lines.append(" text " + repr(child.text)) 512 | else: 513 | lines += [" " + line for line in html_node_dump(child).split("\n")] 514 | 515 | return "\n".join(lines) 516 | 517 | 518 | def html_node_trim_whitespace(node: HTMLNode) -> None: 519 | if node.pre: 520 | node.text = node.text.rstrip("\r\n\t ") 521 | return 522 | 523 | text = node.text.strip("\r\n\t ") 524 | text = re.sub(r"[\r\n\t ]+", " ", text) 525 | text = text.replace("\x00", "\n") 526 | text = re.sub(r" *\n *", "\n", text) 527 | 528 | node.text = text 529 | 530 | 531 | def html_node_process_inline(node: HTMLNode, inline=False) -> str: 532 | """Traverse tree flattening all inline nodes into text nodes""" 533 | 534 | if node.tag not in HTML_BLOCK_TAGS: 535 | # Never disable inline if already enabled 536 | inline = True 537 | 538 | text = "".join(html_node_process_inline(c, inline) for c in html_node_children(node)) 539 | 540 | if not inline: 541 | return "" 542 | 543 | if node.tag == "text": 544 | text = node.text 545 | elif node.tag == "br": 546 | text = "\x00" 547 | elif node.tag == "em" or node.tag == "i": 548 | text = f"/{text}/" 549 | elif node.tag == "strong" or node.tag == "b": 550 | text = f"*{text}*" 551 | elif node.tag == "code" and not node.pre: 552 | text = f"`{text}`" 553 | elif node.tag == "a": 554 | href = node.attrs.get("href", "") or "" 555 | if text.endswith("...") and href.startswith(text[:-3]): 556 | # Workaround for link formatting on HN 557 | text = href 558 | elif text != href: 559 | text = f"{text} {href}" 560 | 561 | node.tag = "text" 562 | node.text = text 563 | node.first_child = node.last_child = None 564 | 565 | return text 566 | 567 | 568 | def html_node_process_text(node: HTMLNode): 569 | """Traverse tree merging, trimming and pruning text nodes""" 570 | 571 | for child in html_node_children(node): 572 | html_node_process_text(child) 573 | 574 | # Merge adjacent text nodes 575 | for child in html_node_children(node): 576 | prev = child.prev_sibling 577 | if child.tag == "text" and prev is not None and prev.tag == "text" and child.pre == prev.pre: 578 | child.text = prev.text + child.text 579 | html_node_unlink(prev) 580 | 581 | # Trim whitespace from text nodes 582 | for child in html_node_children(node): 583 | if child.tag == "text": 584 | html_node_trim_whitespace(child) 585 | 586 | # Remove empty text nodes 587 | for child in html_node_children(node): 588 | if child.tag == "text" and child.text == "": 589 | html_node_unlink(child) 590 | 591 | 592 | def html_node_render_block(node: HTMLNode, width=70) -> str: 593 | if node.tag == "blockquote" or node.tag == "li" or node.tag == "pre": 594 | width -= 2 595 | 596 | parts = [] 597 | 598 | if node.tag == "hr": 599 | parts.append("-" * width) 600 | 601 | for child in html_node_children(node): 602 | if child.tag == "text" and child.pre: 603 | parts.append(child.text) 604 | 605 | elif child.tag == "text" and not child.pre: 606 | subparts = [text_wrap(p, width) for p in child.text.split("\n")] 607 | parts.append("\n".join(subparts)) 608 | 609 | else: 610 | parts.append(html_node_render_block(child, width)) 611 | 612 | if child.next_sibling is not None and child.tag != "li": 613 | parts.append("") 614 | 615 | text = "\n".join(parts) 616 | 617 | if node.tag == "blockquote": 618 | text = "\n".join((">" if line.startswith("> ") else "> ") + line for line in text.split("\n")) 619 | elif node.tag == "pre": 620 | text = text_unindent(text) 621 | text = "\n".join("| " + line for line in text.split("\n")) 622 | elif node.tag == "li": 623 | lines = text.split("\n") 624 | lines = [("- " + lines[0]).rstrip()] + [(" " + line).rstrip() for line in lines[1:]] 625 | text = "\n".join(lines) 626 | 627 | return text 628 | 629 | 630 | def html_render(html: str) -> str: 631 | # This renderer works well for HN messages because their markup is simple, and it can do 632 | # some custom optimizations, like expanding ellipsis-shortened links, preserving quote 633 | # symbols in wrapped lines, and preventing references with long urls from being broken 634 | # into separate lines. For other backends it may make more sense to use an external app 635 | # (links, w3m, etc) 636 | 637 | html = text_sanitize(html) 638 | 639 | parser = HTMLParser() 640 | parser.feed(html) 641 | parser.close() 642 | 643 | node = parser.root_node 644 | 645 | log_sep = "\n" + "-" * 80 + "\n" 646 | logging.debug(f"Initial HTML tree{log_sep}{html_node_dump(node)}{log_sep}") 647 | 648 | html_node_process_inline(node) 649 | html_node_process_text(node) 650 | 651 | logging.debug(f"Processed HTML tree{log_sep}{html_node_dump(node)}{log_sep}") 652 | 653 | return html_node_render_block(node) 654 | 655 | 656 | def fetch(url: str) -> str: 657 | logging.debug(f"Fetching '{url}'...") 658 | 659 | headers = {"User-Agent": "retronews"} 660 | req = urllib.request.Request(url, headers=headers) 661 | resp = urllib.request.urlopen(req, timeout=REQUEST_TIMEOUT).read().decode() 662 | 663 | return resp 664 | 665 | 666 | @overload 667 | def list_get(lst: list[T], index: int, default: T) -> T: ... 668 | 669 | 670 | @overload 671 | def list_get(lst: list[T], index: int, default: Optional[T] = None) -> Optional[T]: ... 672 | 673 | 674 | def list_get(lst, index, default=None): 675 | return lst[index] if 0 <= index < len(lst) else default 676 | 677 | 678 | def list_chunk(lst: list[T], size: int) -> list[list[T]]: 679 | # Flake8 conflicts with Black here - https://github.com/PyCQA/pycodestyle/issues/373 680 | return [lst[i : i + size] for i in range(0, len(lst), size)] # noqa: E203 681 | 682 | 683 | def cmd_quit(_: AppState): 684 | raise ExitException() 685 | 686 | 687 | def cmd_help(app: AppState): 688 | app_show_help_screen(app) 689 | 690 | 691 | def cmd_show_links(app: AppState): 692 | app_show_links_screen(app) 693 | 694 | 695 | def cmd_up(app: AppState) -> None: 696 | cmd_pager_up(app) if app.pager_visible else cmd_prev(app) 697 | 698 | 699 | def cmd_down(app: AppState) -> None: 700 | cmd_pager_down(app) if app.pager_visible else cmd_next(app) 701 | 702 | 703 | def cmd_prev(app: AppState) -> None: 704 | pos = app.selected_message.index_position - 1 if app.selected_message else 0 705 | app_select_message(app, list_get(app.messages, pos, app.selected_message)) 706 | 707 | 708 | def cmd_next(app: AppState) -> None: 709 | pos = app.selected_message.index_position + 1 if app.selected_message else 0 710 | app_select_message(app, list_get(app.messages, pos, app.selected_message)) 711 | 712 | 713 | def cmd_next_unread(app: AppState) -> None: 714 | pos = app.selected_message.index_position + 1 if app.selected_message else 0 715 | message = next((msg for msg in app.messages[pos:] if not msg.is_shown_as_read), None) 716 | if message is not None: 717 | app_select_message(app, message) 718 | 719 | 720 | def cmd_next_sibling(app: AppState) -> None: 721 | if (msg := app.selected_message) and (parent_msg := msg.parent) and parent_msg.children: 722 | # find the next sibling 723 | try: 724 | idx = parent_msg.children.index(msg) 725 | if idx < len(parent_msg.children): 726 | app_select_message(app, parent_msg.children[idx + 1]) 727 | except IndexError: 728 | pass 729 | 730 | 731 | def cmd_prev_sibling(app: AppState) -> None: 732 | if (msg := app.selected_message) and (parent_msg := msg.parent) and parent_msg.children: 733 | # find the previous sibling 734 | try: 735 | idx = parent_msg.children.index(msg) 736 | if idx > 0: 737 | app_select_message(app, parent_msg.children[idx - 1]) 738 | except IndexError: 739 | pass 740 | 741 | 742 | def cmd_mark_thread_as_read(app: AppState) -> None: 743 | # recursively mark us and all children as read 744 | # then jump to the next sibling 745 | def iterate(message: Message): 746 | if message.children: 747 | for child in message.children: 748 | child.flags.read = True 749 | db_save_message(app.db, child) 750 | iterate(child) 751 | 752 | if (msg := app.selected_message) is not None: 753 | iterate(msg) 754 | # jump to the next sibling 755 | cmd_next_sibling(app) 756 | 757 | 758 | def cmd_parent(app: AppState) -> None: 759 | if (msg := app.selected_message) is not None and (parent_msg := msg.parent) is not None: 760 | app_select_message(app, parent_msg) 761 | 762 | 763 | def cmd_mark_set(app: AppState) -> None: 764 | if (msg := app.selected_message) is not None: 765 | app.marked_message_id = msg.msg_id 766 | app_show_flash(app, "Mark set") 767 | 768 | 769 | def cmd_mark_jump(app: AppState) -> None: 770 | marked_msg = app.messages_by_id.get(app.marked_message_id) if app.marked_message_id else None 771 | cmd_mark_set(app) 772 | if marked_msg is not None: 773 | app_select_message(app, marked_msg) 774 | app_show_flash(app, "Mark swapped") 775 | 776 | 777 | def cmd_pager_up(app: AppState) -> None: 778 | app.pager_offset = max(0, app.pager_offset - 1) 779 | 780 | 781 | def cmd_pager_down(app: AppState) -> None: 782 | if app.selected_message is not None and app.layout.pager_height is not None: 783 | app.pager_offset = min(app.pager_offset + 1, max(0, len(app.selected_message.lines) - app.layout.pager_height)) 784 | 785 | 786 | def cmd_page_up(app: AppState) -> None: 787 | cmd_pager_page_up(app) if app.pager_visible else cmd_index_page_up(app) 788 | 789 | 790 | def cmd_page_down(app: AppState) -> None: 791 | cmd_pager_page_down(app) if app.pager_visible else cmd_index_page_down(app) 792 | 793 | 794 | def cmd_index_page_up(app: AppState) -> None: 795 | pos = app.selected_message.index_position - app.layout.index_height if app.selected_message else 0 796 | pos = max(pos, 0) 797 | app_select_message(app, list_get(app.messages, pos, app.selected_message)) 798 | 799 | 800 | def cmd_index_page_down(app: AppState) -> None: 801 | pos = app.selected_message.index_position + app.layout.index_height if app.selected_message else 0 802 | pos = min(pos, len(app.messages) - 1) 803 | app_select_message(app, list_get(app.messages, pos, app.selected_message)) 804 | 805 | 806 | def cmd_pager_page_up(app: AppState) -> None: 807 | if app.layout.pager_height is not None: 808 | app.pager_offset = max(0, app.pager_offset - app.layout.pager_height) 809 | 810 | 811 | def cmd_pager_page_down(app: AppState) -> None: 812 | if (message := app.selected_message) is not None and (pager_height := app.layout.pager_height) is not None: 813 | app.pager_offset = min(app.pager_offset + pager_height, max(0, len(message.lines) - pager_height)) 814 | 815 | 816 | def cmd_load_tab(app: AppState, tab: int) -> None: 817 | if group := list_get(GROUP_TABS, tab - 1): 818 | app_load_group(app, group) 819 | 820 | 821 | def cmd_reload_page(app: AppState) -> None: 822 | app_load_group(app, app.group) 823 | 824 | 825 | def cmd_load_prev_page(app: AppState) -> None: 826 | app_load_group(app, group_advance_page(app.group, -1)) 827 | 828 | 829 | def cmd_load_next_page(app: AppState) -> None: 830 | app_load_group(app, group_advance_page(app.group, 1)) 831 | 832 | 833 | def cmd_load_page(app: AppState) -> None: 834 | user_input = app_prompt(app, "Go to page (empty to cancel): ") 835 | 836 | if user_input.isnumeric() and (page := int(user_input)) >= 1: 837 | app_load_group(app, group_set_page(app.group, page)) 838 | elif len(user_input) > 0: 839 | app_show_flash(app, "Invalid page number") 840 | 841 | 842 | def cmd_open(app: AppState) -> None: 843 | if (msg := app.selected_message) is None: 844 | return 845 | 846 | if msg.is_thread: 847 | app_open_thread(app, msg) 848 | else: 849 | app_select_message(app, msg, show_pager=True) 850 | 851 | 852 | def cmd_close(app: AppState) -> None: 853 | if app.pager_visible: 854 | app.pager_visible = False 855 | else: 856 | app_close_thread(app) 857 | 858 | 859 | def cmd_star(app: AppState) -> None: 860 | if (msg := app.selected_message) is not None: 861 | msg.flags.starred = not msg.flags.starred 862 | db_save_message(app.db, msg) 863 | cmd_next(app) 864 | 865 | 866 | def cmd_star_thread(app: AppState) -> None: 867 | if (msg := app.selected_message) is None: 868 | return 869 | 870 | if (thread_msg := app.messages_by_id.get(msg.thread_id)) is None: 871 | return 872 | 873 | thread_msg.flags.starred = not thread_msg.flags.starred 874 | db_save_message(app.db, thread_msg) 875 | cmd_next(app) 876 | 877 | 878 | def cmd_set_unread(app: AppState) -> None: 879 | if (msg := app.selected_message) is not None: 880 | msg.flags.read = False 881 | db_save_message(app.db, msg) 882 | cmd_next(app) 883 | 884 | 885 | def cmd_dump(app: AppState) -> None: 886 | if (msg := app.selected_message) is None: 887 | return 888 | 889 | filename = f"{msg.msg_id}.html" 890 | 891 | with open(filename, "w") as fp: 892 | fp.write(msg.body or "") 893 | 894 | app_show_flash(app, f"Message body dumped to {filename}") 895 | 896 | 897 | def cmd_toggle_raw_mode(app: AppState) -> None: 898 | app.raw_mode = not app.raw_mode 899 | app_select_message(app, app.selected_message) 900 | 901 | 902 | def cmd_resize(app: AppState) -> None: 903 | app_refresh_message(app) 904 | 905 | 906 | def cmd_unknown(app: AppState) -> None: 907 | app.flash = "Unknown key" 908 | 909 | 910 | def db_init(path: str) -> DB: 911 | path = os.path.expanduser(path) 912 | create_table_sql = """ 913 | CREATE TABLE IF NOT EXISTS messages ( 914 | msg_id TEXT NOT NULL PRIMARY KEY, 915 | thread_id TEXT NOT NULL, 916 | date INTEGER NOT NULL, 917 | flags JSON NOT NULL 918 | ); 919 | 920 | CREATE INDEX IF NOT EXISTS messages_starred_date ON messages ( 921 | JSON_EXTRACT(flags, '$.starred'), 922 | date 923 | ); 924 | """ 925 | 926 | db = sqlite3.connect(path) 927 | db.row_factory = sqlite3.Row 928 | db.executescript(create_table_sql) 929 | db.commit() 930 | 931 | return cast(DB, db) 932 | 933 | 934 | def db_save_message(db: DB, message: Message) -> None: 935 | sql = """INSERT OR REPLACE INTO messages (msg_id, thread_id, date, flags) VALUES (?, ?, ?, ?)""" 936 | date = int(message.date.timestamp()) 937 | flags_json = json.dumps(dataclasses.asdict(message.flags)) 938 | db.execute(sql, (message.msg_id, message.thread_id, date, flags_json)) 939 | db.commit() 940 | 941 | 942 | def db_load_message_flags(db: DB, messages_by_id: dict[str, Message]) -> None: 943 | message_ids = list(messages_by_id.keys()) 944 | sql = f"SELECT * FROM messages WHERE msg_id IN ({','.join('?' for _ in message_ids)})" 945 | 946 | for row in db.execute(sql, message_ids): 947 | flags = json.loads(row["flags"]) 948 | messages_by_id[row["msg_id"]].flags = MessageFlags(**flags) 949 | 950 | 951 | def db_load_read_comments(db: DB, messages_by_id: dict[str, Message]) -> None: 952 | threads_by_id = {msg.msg_id: msg for msg in messages_by_id.values() if msg.is_thread} 953 | thread_ids = list(threads_by_id.keys()) 954 | 955 | sql = f""" 956 | SELECT thread_id, COUNT(*) AS count 957 | FROM messages 958 | WHERE thread_id IN ({','.join('?' for _ in thread_ids)}) AND JSON_EXTRACT(flags, '$.read') 959 | GROUP BY thread_id 960 | """ 961 | 962 | for row in db.execute(sql, thread_ids): 963 | threads_by_id[row["thread_id"]].read_comments = row["count"] 964 | 965 | 966 | def db_load_starred_thread_ids(db: DB, page: int = 1) -> list[str]: 967 | page_size = 30 968 | offset = (page - 1) * page_size 969 | sql = """ 970 | SELECT thread_id 971 | FROM messages 972 | WHERE JSON_EXTRACT(flags, '$.starred') 973 | GROUP BY thread_id 974 | ORDER BY date DESC 975 | LIMIT ? 976 | OFFSET ? 977 | """ 978 | 979 | return [row["thread_id"] for row in db.execute(sql, (page_size, offset))] 980 | 981 | 982 | def msg_flatten_thread( 983 | msg: Message, prefix: str = "", is_last_child: bool = False, ascii: bool = False 984 | ) -> Generator[Message, None, None]: 985 | blcorner = "'-" if ascii else "└─" 986 | ltee = "|-" if ascii else "├─" 987 | vline = "| " if ascii else "│ " 988 | 989 | msg.index_tree = "" if msg.is_thread else f"{prefix}{blcorner if is_last_child else ltee}> " 990 | yield msg 991 | 992 | children = msg.children or [] 993 | 994 | child_count = len(children) 995 | child_prefix = "" if msg.is_thread else f"{prefix}{' ' if is_last_child else vline}" 996 | 997 | for i, child_node in enumerate(children): 998 | child_is_last = i == child_count - 1 999 | for child in msg_flatten_thread(child_node, prefix=child_prefix, is_last_child=child_is_last, ascii=ascii): 1000 | yield child 1001 | 1002 | 1003 | def msg_build_raw_lines(msg: Message) -> list[str]: 1004 | text = text_sanitize(msg.body) 1005 | 1006 | # Unescape selected entities for better readability 1007 | repl = {"/": "/", "'": "'", """: '"'} 1008 | for k, v in repl.items(): 1009 | text = text.replace(k, v) 1010 | 1011 | return reduce(lambda acc, line: acc + wrap(line, width=120, replace_whitespace=False), text.split("\n"), []) 1012 | 1013 | 1014 | def msg_build_lines(msg: Message) -> list[str]: 1015 | lines = [ 1016 | f"Content-Location: {msg.content_location}", 1017 | f"Date: {msg.date.strftime('%Y-%m-%d %H:%M')}", 1018 | f"From: {msg.author or ''}", 1019 | f"Subject: {msg.title}", 1020 | "", 1021 | ] 1022 | 1023 | lines += html_render(msg.body or "").split("\n") if not msg.is_deleted else [""] 1024 | 1025 | return lines 1026 | 1027 | 1028 | def msg_unload(msg: Message) -> Message: 1029 | msg.children = None 1030 | msg.body = None 1031 | return msg 1032 | 1033 | 1034 | def hn_parse_search_hit(hit: HNSearchHit) -> Message: 1035 | return Message( 1036 | msg_id=f"{hit['objectID']}@hn", 1037 | thread_id=f"{hit['objectID']}@hn", 1038 | content_location=f"https://news.ycombinator.com/item?id={hit['objectID']}", 1039 | date=datetime.fromtimestamp(hit["created_at_i"]), 1040 | author=hit["author"], 1041 | title=html.unescape(hit["title"]), 1042 | total_comments=(hit["num_comments"] or 0) + 1, 1043 | ) 1044 | 1045 | 1046 | def hn_parse_entry(entry: HNEntry, thread_id: str = "", parent: Optional[Message] = None) -> Message: 1047 | thread_id = thread_id or str(entry["id"]) 1048 | 1049 | my_title = html.unescape(entry["title"]) if entry["title"] else None 1050 | 1051 | parent_title = parent.title if parent else "" 1052 | parent_title = parent_title if parent_title.startswith("Re: ") else f"Re: {parent_title}" 1053 | 1054 | body = f"

{entry['url']}

" if entry["url"] else "" 1055 | body = f"{body}{entry['text']}" if entry["text"] else body 1056 | 1057 | msg = Message( 1058 | msg_id=f"{entry['id']}@hn", 1059 | thread_id=f"{thread_id}@hn", 1060 | content_location=f"https://news.ycombinator.com/item?id={entry['id']}", 1061 | date=datetime.fromtimestamp(entry["created_at_i"]), 1062 | author=entry["author"], 1063 | title=my_title or parent_title, 1064 | body=body, 1065 | parent=parent, 1066 | ) 1067 | 1068 | msg.children = [hn_parse_entry(child, thread_id, msg) for child in entry["children"]] 1069 | 1070 | return msg 1071 | 1072 | 1073 | def hn_fetch_threads_by_id(thread_ids: list[str]) -> list[Message]: 1074 | story_tags = ",".join(f"story_{x}" for x in thread_ids) 1075 | url = f"https://hn.algolia.com/api/v1/search_by_date?hitsPerPage={len(thread_ids)}&tags=story,({story_tags})" 1076 | hits = json.loads(fetch(url))["hits"] 1077 | hits_by_id = {hit["objectID"]: hit for hit in hits} 1078 | threads = [hn_parse_search_hit(hits_by_id[tid]) for tid in thread_ids if tid in hits_by_id] 1079 | 1080 | return threads 1081 | 1082 | 1083 | def hn_fetch_threads(group: str = "news", page: int = 1) -> list[Message]: 1084 | rex = re.compile(r'href="item\?id=(\d+)"') 1085 | 1086 | html = fetch(f"https://news.ycombinator.com/{group}?p={page}") 1087 | thread_ids = list(dict.fromkeys(match.group(1) for match in rex.finditer(html))) 1088 | 1089 | return hn_fetch_threads_by_id(thread_ids) 1090 | 1091 | 1092 | def hn_fetch_new_threads(page: int = 1) -> list[Message]: 1093 | url = f"https://hn.algolia.com/api/v1/search_by_date?tags=story&hitsPerPage=30&page={page}" 1094 | hits = json.loads(fetch(url))["hits"] 1095 | 1096 | return [hn_parse_search_hit(hit) for hit in hits] 1097 | 1098 | 1099 | def hn_fetch_thread(entry_id: Union[str, int]) -> Message: 1100 | resp = fetch(f"http://hn.algolia.com/api/v1/items/{entry_id}") 1101 | entry: HNEntry = json.loads(resp) 1102 | return hn_parse_entry(entry) 1103 | 1104 | 1105 | def lb_parse_thread(thread: LBThread) -> Message: 1106 | comments = {} 1107 | 1108 | thread_body = f"{thread['description']}" 1109 | thread_body = f"

{thread['url']}

{thread_body}" if thread["url"] else thread_body 1110 | 1111 | ret = Message( 1112 | msg_id=f"{thread['short_id']}@lb", 1113 | thread_id=f"{thread['short_id']}@lb", 1114 | content_location=thread["short_id_url"], 1115 | date=datetime.fromtimestamp(datetime.fromisoformat(thread["created_at"]).timestamp()), 1116 | author=thread["submitter_user"], 1117 | title=thread["title"], 1118 | body=thread_body, 1119 | children=None if thread.get("comments") is None else [], 1120 | total_comments=thread["comment_count"] + 1, 1121 | ) 1122 | 1123 | for comment in thread.get("comments", []) or []: 1124 | comments[comment["short_id"]] = Message( 1125 | msg_id=f"{comment['short_id']}@lb", 1126 | thread_id=f"{thread['short_id']}@lb", 1127 | content_location=comment["url"], 1128 | date=datetime.fromtimestamp(datetime.fromisoformat(comment["created_at"]).timestamp()), 1129 | author=comment["commenting_user"], 1130 | title=f"Re: {thread['title']}", 1131 | body=comment["comment"], 1132 | children=[], 1133 | ) 1134 | 1135 | for comment in thread.get("comments", []) or []: 1136 | msg = comments[comment["short_id"]] 1137 | parent_msg = comments[comment["parent_comment"]] if comment["parent_comment"] else ret 1138 | msg.parent = parent_msg 1139 | if parent_msg.children is not None: 1140 | parent_msg.children.append(msg) 1141 | 1142 | return ret 1143 | 1144 | 1145 | def lb_fetch_threads(group: str = "", page: int = 1) -> list[Message]: 1146 | group_path = f"{group}/" if group else "" 1147 | resp = fetch(f"https://lobste.rs/{group_path}page/{page}.json") 1148 | threads: list[LBThread] = json.loads(resp) 1149 | 1150 | return [lb_parse_thread(thread) for thread in threads] 1151 | 1152 | 1153 | def lb_fetch_thread(entry_id: str) -> Message: 1154 | resp = fetch(f"https://lobste.rs/s/{entry_id}.json") 1155 | thread: LBThread = json.loads(resp) 1156 | 1157 | return lb_parse_thread(thread) 1158 | 1159 | 1160 | def group_set_page(group: Group, page: int) -> Group: 1161 | return dataclasses.replace(group, page=page) 1162 | 1163 | 1164 | def group_advance_page(group: Group, offset: int = 1) -> Group: 1165 | return group_set_page(group, page=max(1, group.page + offset)) 1166 | 1167 | 1168 | def group_fetch_starred_threads(db: DB, page: int = 1) -> list[Message]: 1169 | thread_ids = db_load_starred_thread_ids(db, page) 1170 | threads_by_provider_id: dict[str, list[str]] = {} 1171 | threads = [] 1172 | 1173 | for source_id, provider_id in (t.split("@") for t in thread_ids): 1174 | threads_by_provider_id.setdefault(provider_id, list()).append(source_id) 1175 | 1176 | for provider_id, thread_ids in threads_by_provider_id.items(): 1177 | provider = PROVIDERS[provider_id] 1178 | threads += provider.fetch_threads_by_id(thread_ids) 1179 | 1180 | threads.sort(key=lambda x: x.date, reverse=True) 1181 | 1182 | return threads 1183 | 1184 | 1185 | def group_fetch_thread(thread_id: str) -> Message: 1186 | (msg_id, provider_id) = thread_id.split("@") 1187 | provider = PROVIDERS[provider_id] 1188 | 1189 | return provider.fetch_thread(msg_id) 1190 | 1191 | 1192 | def group_for_msg_url(url: str) -> Group: 1193 | if (match := HN_URL_REX.match(url)) is not None: 1194 | msg_id = match[1] 1195 | return Group(label="", fetch=lambda *x: hn_fetch_threads_by_id([msg_id])) 1196 | 1197 | msg = "Unknown URL, available patterns: \n" + "\n".join(f"- {r.pattern}" for r in [HN_URL_REX]) 1198 | raise ExitException(1, msg) 1199 | 1200 | 1201 | def app_safe_run(app: AppState, fn: Callable[[], T], flash: Optional[str]) -> Optional[T]: 1202 | if flash is not None: 1203 | app_show_flash(app, flash) 1204 | 1205 | ret = None 1206 | 1207 | try: 1208 | ret = fn() 1209 | except Exception as e: 1210 | logging.debug("\n".join(traceback.format_exception(e))) 1211 | app_show_flash(app, f"Error: {e}") 1212 | else: 1213 | if flash is not None: 1214 | app_show_flash(app, None) 1215 | 1216 | return ret 1217 | 1218 | 1219 | def app_refresh_message(app: AppState) -> None: 1220 | app.pager_offset = 0 1221 | 1222 | # Converting html to lines lazily on render for easier debugging 1223 | if (msg := app.selected_message) is not None: 1224 | msg.lines = msg_build_raw_lines(msg) if app.raw_mode else msg_build_lines(msg) 1225 | 1226 | 1227 | def app_select_message(app: AppState, message: Optional[Message], show_pager: bool = False) -> None: 1228 | app.selected_message = message 1229 | 1230 | app_refresh_message(app) 1231 | 1232 | if message is None or message.body is None: 1233 | app.pager_visible = False 1234 | return 1235 | 1236 | if show_pager: 1237 | app.pager_visible = True 1238 | 1239 | if app.pager_visible: 1240 | message.flags.read = True 1241 | db_save_message(app.db, message) 1242 | db_load_read_comments(app.db, {message.thread_id: app.messages_by_id[message.thread_id]}) 1243 | 1244 | 1245 | def app_load_messages( 1246 | app: AppState, messages: list[Message], selected_message_id: Optional[str] = None, show_pager: bool = False 1247 | ) -> None: 1248 | if selected_message_id is None and app.selected_message is not None: 1249 | selected_message_id = app.selected_message.msg_id 1250 | 1251 | selected_message = None 1252 | 1253 | for i, message in enumerate(messages): 1254 | message.index_position = i 1255 | 1256 | if message.msg_id == selected_message_id: 1257 | selected_message = message 1258 | 1259 | if selected_message is None and len(messages) > 0: 1260 | selected_message = messages[0] 1261 | 1262 | app.messages = messages 1263 | app.messages_by_id = {msg.msg_id: msg for msg in messages} 1264 | 1265 | db_load_message_flags(app.db, app.messages_by_id) 1266 | db_load_read_comments(app.db, app.messages_by_id) 1267 | 1268 | app_select_message(app, selected_message, show_pager) 1269 | 1270 | 1271 | def app_load_group(app: AppState, group: Group) -> None: 1272 | fn = partial(group.fetch, app.db, group.page) 1273 | flash = f"Fetching stories from '{group.label}' (page {group.page})..." 1274 | 1275 | if (messages := app_safe_run(app, fn, flash=flash)) is None: 1276 | return 1277 | 1278 | app_load_messages(app, messages) 1279 | app.group = group 1280 | 1281 | 1282 | def app_close_thread(app: AppState) -> None: 1283 | selected_thread_id = app.selected_message.thread_id if app.selected_message else None 1284 | filtered_messages = [msg_unload(msg) for msg in app.messages if msg.is_thread] 1285 | 1286 | app_load_messages(app, filtered_messages, selected_message_id=selected_thread_id) 1287 | 1288 | 1289 | def app_open_thread(app: AppState, thread_message: Message) -> None: 1290 | fn = partial(group_fetch_thread, thread_message.thread_id) 1291 | flash = f"Fetching thread '{thread_message.thread_id}'..." 1292 | 1293 | if (new_thread_message := app_safe_run(app, fn, flash=flash)) is None: 1294 | return 1295 | 1296 | app_close_thread(app) 1297 | 1298 | index_pos = thread_message.index_position 1299 | thread_messages = list(msg_flatten_thread(new_thread_message, ascii=app.ascii)) 1300 | new_thread_message.total_comments = len(thread_messages) 1301 | messages = app.messages[:index_pos] + thread_messages + app.messages[index_pos + 1 :] # noqa: E203 1302 | 1303 | app_load_messages(app, messages, selected_message_id=thread_message.msg_id, show_pager=True) 1304 | 1305 | 1306 | def app_update_layout(app: AppState) -> None: 1307 | lt = app.layout 1308 | 1309 | (lt.lines, lt.cols) = app.screen.getmaxyx() 1310 | 1311 | if lt.lines < 12 or lt.cols < 80: 1312 | raise ExitException(1, "At least 80x12 terminal is required") 1313 | 1314 | max_index_height = lt.lines - 3 1315 | lt.index_height = (max_index_height // 3) if app.pager_visible else max_index_height 1316 | 1317 | lt.middle_menu_row = lt.index_start + lt.index_height if app.pager_visible else None 1318 | lt.pager_start = lt.index_start + lt.index_height + 1 if app.pager_visible else None 1319 | lt.pager_height = lt.lines - lt.pager_start - 2 if lt.pager_start is not None else None 1320 | 1321 | lt.bottom_menu_row = lt.lines - 2 1322 | lt.flash_menu_row = lt.lines - 1 1323 | 1324 | 1325 | def app_show_help_screen(app: AppState) -> None: 1326 | max_lines = app.layout.lines - 4 1327 | 1328 | help_lines = HELP_SCREEN.split("\n") 1329 | help_pages = ["\n".join(lines) for lines in list_chunk(help_lines, max_lines)] 1330 | 1331 | for page in help_pages: 1332 | app.screen.erase() 1333 | app.screen.addstr(0, 0, "Available commands:\n\n") 1334 | app.screen.addstr(page) 1335 | app.screen.addstr("\n\nPress any key to continue...") 1336 | app.screen.refresh() 1337 | app.screen.getch() 1338 | 1339 | 1340 | def app_show_links_screen(app: AppState) -> None: 1341 | lines = app.selected_message.lines if app.selected_message is not None else [] 1342 | 1343 | # Max amount of keys is 21 to fit on 25-line terminals 1344 | keys = "1234567890abcdefghijk" 1345 | urls = URL_REX.findall(" ".join(lines)) 1346 | items = dict(zip((ord(k) for k in keys), urls)) 1347 | 1348 | if len(items) == 0: 1349 | return app_show_flash(app, "No links available for opening") 1350 | 1351 | app.screen.erase() 1352 | app.screen.addstr(0, 0, "Select link to open:") 1353 | 1354 | for i, (key, url) in enumerate(items.items()): 1355 | app.screen.addstr(i + 2, 0, f"{chr(key)} - {url}") 1356 | 1357 | app.screen.addstr(i + 4, 0, "To change browser run: BROWSER='firefox %s' ./retronews.py") 1358 | app.screen.refresh() 1359 | 1360 | key = app.screen.getch() 1361 | 1362 | if key not in items.keys(): 1363 | return app_show_flash(app, "Unknown key") 1364 | 1365 | url = items[key] 1366 | 1367 | app_show_flash(app, f"Opening {url}") 1368 | webbrowser.open(url) 1369 | 1370 | # Refresh window in case a terminal browser was used 1371 | app.screen.clearok(True) 1372 | 1373 | 1374 | def app_show_flash(app: AppState, flash: Optional[str]) -> None: 1375 | app.flash = flash 1376 | app_render(app) 1377 | 1378 | 1379 | def app_prompt(app: AppState, prompt: str) -> str: 1380 | lt = app.layout 1381 | 1382 | app.screen.insstr(lt.flash_menu_row, 0, prompt.ljust(lt.cols)) 1383 | app.screen.refresh() 1384 | 1385 | curses.curs_set(1) 1386 | win = curses.newwin(1, lt.cols - len(prompt), lt.flash_menu_row, len(prompt)) 1387 | 1388 | textbox = curses.textpad.Textbox(win) 1389 | textbox.stripspaces = True 1390 | ret = textbox.edit().strip() 1391 | 1392 | del win 1393 | curses.curs_set(0) 1394 | 1395 | return ret 1396 | 1397 | 1398 | def app_render_index_row(app: AppState, row: int, message: Message) -> None: 1399 | cols = app.layout.cols 1400 | date = message.date.strftime("%Y-%m-%d %H:%M") 1401 | author = (message.author or "")[:10].ljust(10) 1402 | 1403 | is_response = message.title.startswith("Re:") and not message.is_thread 1404 | is_selected = message == app.selected_message 1405 | hide_title = is_response and row > app.layout.index_start and not message.flags.starred and not is_selected 1406 | title = "" if hide_title else text_clean(message.title, ascii=app.ascii) 1407 | 1408 | unread = ( 1409 | str(max(min(message.total_comments - message.read_comments, 9999), 0)).rjust(4) if message.is_thread else " " 1410 | ) 1411 | 1412 | app.screen.insstr(row, 0, f"[{date}] [{author}] [{unread}] {message.index_tree}{title}") 1413 | 1414 | if is_selected: 1415 | cursor_attr = curses.A_REVERSE if app.monochrome else 0 1416 | app.screen.chgat(row, 0, cols, app.colors["cursor"] | cursor_attr) 1417 | else: 1418 | read_attr = 0 if message.is_shown_as_read else curses.A_BOLD 1419 | subject_attr = app.colors["starred_subject"] if message.flags.starred else app.colors["default"] 1420 | subject_attr = subject_attr | read_attr 1421 | 1422 | app.screen.chgat(row, 1, 16, app.colors["date"] | read_attr) 1423 | app.screen.chgat(row, 21, 10, app.colors["author"] | read_attr) 1424 | app.screen.chgat(row, 35, 4, app.colors["unread_comments"] | read_attr) 1425 | app.screen.chgat(row, 42, len(message.index_tree), app.colors["tree"]) 1426 | app.screen.chgat(row, 42 + len(message.index_tree), cols - 42 - len(message.index_tree), subject_attr) 1427 | 1428 | 1429 | def app_render_index(app: AppState) -> None: 1430 | height = app.layout.index_height 1431 | 1432 | offset = app.selected_message.index_position - height // 2 if app.selected_message else 0 1433 | offset = min(offset, len(app.messages) - height) 1434 | offset = max(offset, 0) 1435 | 1436 | rows_to_render = min(height, len(app.messages) - offset) 1437 | 1438 | for i in range(rows_to_render): 1439 | app_render_index_row(app, app.layout.index_start + i, app.messages[i + offset]) 1440 | 1441 | 1442 | def app_get_pager_line_attr(app: AppState, line: str) -> int: 1443 | if line.startswith("Content-Location: "): 1444 | return app.colors["tree"] 1445 | elif line.startswith("Date: "): 1446 | return app.colors["date"] 1447 | elif line.startswith("From: "): 1448 | return app.colors["author"] 1449 | elif line.startswith("Subject: "): 1450 | return app.colors["header_subject"] 1451 | elif line.startswith(">>") or line.startswith("> >"): 1452 | return app.colors["nested_quote"] 1453 | elif line.startswith(">"): 1454 | return app.colors["quote"] 1455 | elif line.startswith("| "): 1456 | return app.colors["code"] 1457 | elif line == "~": 1458 | return app.colors["empty_pager_line"] 1459 | elif line == "" or line == "[dead]": 1460 | return app.colors["deleted_message_pager_line"] 1461 | else: 1462 | return 0 1463 | 1464 | 1465 | def app_render_pager_line(app: AppState, row: int, line: str) -> None: 1466 | hl_lines = not app.raw_mode 1467 | line_attr = app_get_pager_line_attr(app, line) if hl_lines else 0 1468 | hl_urls = line_attr == 0 and hl_lines 1469 | 1470 | line = text_clean(line, ascii=app.ascii) 1471 | 1472 | app.screen.move(row, 0) 1473 | app.screen.clrtoeol() 1474 | app.screen.move(row, 0) 1475 | 1476 | for part in text_split_urls(line): 1477 | is_url = URL_REX.fullmatch(part) 1478 | part_attr = app.colors["url"] if is_url and hl_urls else line_attr 1479 | app.screen.addstr(part, part_attr) 1480 | 1481 | 1482 | def app_render_pager(app: AppState) -> None: 1483 | message = app.selected_message 1484 | start = app.layout.pager_start 1485 | height = app.layout.pager_height 1486 | 1487 | if message is None or start is None or height is None: 1488 | return 1489 | 1490 | for i in range(height): 1491 | line = list_get(message.lines, i + app.pager_offset, "~") 1492 | app_render_pager_line(app, i + start, line) 1493 | 1494 | 1495 | def app_render_top_menu(app: AppState) -> None: 1496 | lt = app.layout 1497 | cols = lt.cols 1498 | base_attr = curses.A_REVERSE if app.monochrome else curses.A_BOLD 1499 | app.screen.insstr(lt.top_menu_row, 0, HELP_MENU[:cols].ljust(cols), app.colors["menu"] | base_attr) 1500 | 1501 | 1502 | def app_render_middle_menu(app: AppState) -> None: 1503 | if (row := app.layout.middle_menu_row) is None: 1504 | return 1505 | 1506 | if (message := app.selected_message) is None: 1507 | return 1508 | 1509 | if (thread_message := app.messages_by_id.get(message.thread_id)) is None: 1510 | return 1511 | 1512 | cols = app.layout.cols 1513 | total = thread_message.total_comments 1514 | unread = total - thread_message.read_comments 1515 | 1516 | text = f"--({unread}/{total} unread)" 1517 | if thread_message.flags.starred: 1518 | text += "--(starred thread)" 1519 | if app.raw_mode: 1520 | text += "--(raw mode on)" 1521 | text = text[:cols].ljust(cols, "-") 1522 | 1523 | base_attr = curses.A_REVERSE if app.monochrome else curses.A_BOLD 1524 | app.screen.insstr(row, 0, text, app.colors["menu"] | base_attr) 1525 | 1526 | 1527 | def app_render_bottom_menu(app: AppState) -> None: 1528 | lt = app.layout 1529 | base_attr = curses.A_REVERSE if app.monochrome else curses.A_BOLD 1530 | 1531 | app.screen.chgat(lt.bottom_menu_row, 0, lt.cols, app.colors["menu"] | base_attr) 1532 | app.screen.move(lt.bottom_menu_row, 0) 1533 | 1534 | for i, group in enumerate(GROUP_TABS): 1535 | is_active = group.label == app.group.label 1536 | item_attr = app.colors["menu_active"] | curses.A_BOLD if is_active else app.colors["menu"] 1537 | item_attr = item_attr | base_attr 1538 | app.screen.addstr(f"{i+1}:{group.label}", item_attr) 1539 | app.screen.addstr(" ", app.colors["menu"] | base_attr) 1540 | 1541 | page_text = f"page: {app.group.page}" 1542 | app.screen.insstr(lt.bottom_menu_row, lt.cols - len(page_text), page_text, app.colors["menu"] | base_attr) 1543 | 1544 | 1545 | def app_render(app: AppState) -> None: 1546 | app_update_layout(app) 1547 | app.screen.erase() 1548 | app_render_index(app) 1549 | app_render_pager(app) 1550 | app_render_top_menu(app) 1551 | app_render_middle_menu(app) 1552 | app_render_bottom_menu(app) 1553 | app.screen.insstr(app.layout.flash_menu_row, 0, app.flash or "") 1554 | app.screen.refresh() 1555 | 1556 | 1557 | def app_init_colors(app: AppState) -> None: 1558 | if app.monochrome: 1559 | app.colors = defaultdict(lambda: 0) 1560 | return 1561 | 1562 | try: 1563 | curses.use_default_colors() 1564 | for i, (name, (fg, bg)) in enumerate(COLORS.items()): 1565 | curses.init_pair(i + 1, fg, bg) 1566 | app.colors[name] = curses.color_pair(i + 1) 1567 | except curses.error: 1568 | app.colors = defaultdict(lambda: 0) 1569 | app.monochrome = True 1570 | 1571 | 1572 | def app_main(screen: Window, db: DB, group: Group, ascii: bool, monochrome: bool) -> int: 1573 | curses.curs_set(0) 1574 | 1575 | app = AppState(screen=screen, db=db, group=group, ascii=ascii, monochrome=monochrome) 1576 | 1577 | app_init_colors(app) 1578 | app_load_group(app, app.group) 1579 | 1580 | while True: 1581 | app_render(app) 1582 | app.flash = "" 1583 | c = app.screen.getch() 1584 | KEY_BINDINGS.get(c, cmd_unknown)(app) 1585 | 1586 | 1587 | def setup_logging(path: Optional[str]) -> None: 1588 | if path is None: 1589 | return logging.disable() 1590 | 1591 | format = "%(asctime)s %(levelname)s: %(message)s" 1592 | stream = sys.stderr if path == "-" else open(path, "a") 1593 | logging.basicConfig(format=format, level="DEBUG", stream=stream) 1594 | logging.debug("Session started") 1595 | 1596 | 1597 | def run_rcfile(path: str) -> None: 1598 | path = os.path.expanduser(path) 1599 | 1600 | if not os.path.isfile(path): 1601 | return 1602 | 1603 | code = compile(open(path).read(), path, "exec") 1604 | exec(code, {"retronews": sys.modules[__name__]}) 1605 | 1606 | 1607 | if __name__ == "__main__": 1608 | tab_choices = range(1, len(GROUP_TABS) + 1) 1609 | 1610 | ap = argparse.ArgumentParser( 1611 | formatter_class=lambda prog: argparse.ArgumentDefaultsHelpFormatter(prog, max_help_position=32) 1612 | ) 1613 | ap.add_argument("--ascii", action="store_true", help="show only ascii characters") 1614 | ap.add_argument("--monochrome", action="store_true", help="disable colors") 1615 | ap.add_argument("-c", "--rcfile", metavar="PATH", default="~/.retronewsrc.py", help="optional startup code path") 1616 | ap.add_argument("-d", "--db", metavar="PATH", default="~/.retronews.db", help="database path") 1617 | ap.add_argument("-l", "--logfile", metavar="PATH", default=None, help="debug logfile path") 1618 | ap.add_argument("-t", "--tab", metavar="TAB", type=int, default=1, choices=tab_choices, help="initial tab") 1619 | ap.add_argument("-r", "--render", metavar="PATH", default=None, help="render raw html message and quit") 1620 | ap.add_argument("-m", "--msg", metavar="URL", default=None, help="render message from URL") 1621 | args = ap.parse_args() 1622 | 1623 | setup_logging(args.logfile) 1624 | run_rcfile(args.rcfile) 1625 | 1626 | if (path := args.render) is not None: 1627 | with open(path) as fp: 1628 | print(html_render(fp.read())) 1629 | sys.exit(0) 1630 | 1631 | try: 1632 | db = db_init(args.db) 1633 | 1634 | if args.msg: 1635 | group = group_for_msg_url(args.msg) 1636 | else: 1637 | group = GROUP_TABS[args.tab - 1] 1638 | 1639 | ascii = args.ascii 1640 | monochrome = args.monochrome or "NO_COLOR" in os.environ 1641 | 1642 | ret = curses.wrapper(app_main, db=db, group=group, ascii=ascii, monochrome=monochrome) 1643 | except ExitException as e: 1644 | if e.message: 1645 | sys.stderr.write(e.message + "\n") 1646 | ret = e.code 1647 | except BaseException as e: 1648 | sys.stderr.write("\n".join(traceback.format_exception(e))) 1649 | ret = 1 1650 | finally: 1651 | db.close() 1652 | sys.exit(ret) 1653 | -------------------------------------------------------------------------------- /screenshot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/luke8086/retronews/b3fd323e9053755288884be017ecc4f40395a6d1/screenshot.png -------------------------------------------------------------------------------- /tests.py: -------------------------------------------------------------------------------- 1 | import os 2 | import subprocess 3 | import unittest 4 | 5 | import retronews 6 | 7 | TC_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "tests") 8 | 9 | 10 | class TestHtmlRender(unittest.TestCase): 11 | maxDiff = None 12 | 13 | def checkRendering(self, name: str): 14 | html_path = os.path.join(TC_DIR, f"{name}.html") 15 | out_path = os.path.join(TC_DIR, f"{name}.out") 16 | 17 | with open(html_path) as fp: 18 | html = fp.read() 19 | 20 | actual = retronews.html_render(html) 21 | 22 | if not os.path.exists(out_path): 23 | with open(out_path, "w") as fp: 24 | fp.write(actual) 25 | return 26 | 27 | cmd = ["diff", "-Nru", "--color=always", out_path, "-"] 28 | proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True) 29 | stdout, stderr = proc.communicate(input=actual) 30 | 31 | if proc.returncode != 0: 32 | self.fail(f"Unexpected rendering output\n{stdout}") 33 | 34 | 35 | def setup_test_cases(): 36 | tcs = [x.split(".")[0] for x in sorted(os.listdir(TC_DIR)) if x.endswith(".html")] 37 | 38 | for tc in tcs: 39 | setattr(TestHtmlRender, tc, lambda self, tc=tc: self.checkRendering(tc)) 40 | 41 | 42 | if __name__ == "__main__": 43 | setup_test_cases() 44 | unittest.main() 45 | -------------------------------------------------------------------------------- /tests/test_hn_01.html: -------------------------------------------------------------------------------- 1 |

Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.

Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.

-------------------------------------------------------------------------------- /tests/test_hn_01.out: -------------------------------------------------------------------------------- 1 | Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus 2 | te. Noster nominati recteque no has. 3 | 4 | Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus 5 | te. Noster nominati recteque no has. -------------------------------------------------------------------------------- /tests/test_hn_02.html: -------------------------------------------------------------------------------- 1 |

>>Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.

> Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.

Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.

2 | -------------------------------------------------------------------------------- /tests/test_hn_02.out: -------------------------------------------------------------------------------- 1 | >>Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus 2 | >>te. Noster nominati recteque no has. 3 | 4 | > Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus 5 | > te. Noster nominati recteque no has. 6 | 7 | Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus 8 | te. Noster nominati recteque no has. -------------------------------------------------------------------------------- /tests/test_hn_03.html: -------------------------------------------------------------------------------- 1 | https://example.com/foo... 2 | -------------------------------------------------------------------------------- /tests/test_hn_03.out: -------------------------------------------------------------------------------- 1 | https://example.com/foo/bar -------------------------------------------------------------------------------- /tests/test_hn_04.html: -------------------------------------------------------------------------------- 1 |

Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus

[0]: https://long.long.long.long.long.long.long.long.long.long.long.example.com

[1] - https://long.long.long.long.long.long.long.long.long.long.long.example.com

[2] https://long.long.long.long.long.long.long.long.long.long.long.example.com

2 | -------------------------------------------------------------------------------- /tests/test_hn_04.out: -------------------------------------------------------------------------------- 1 | Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus 2 | 3 | [0]: https://long.long.long.long.long.long.long.long.long.long.long.example.com 4 | 5 | [1] - https://long.long.long.long.long.long.long.long.long.long.long.example.com 6 | 7 | [2] https://long.long.long.long.long.long.long.long.long.long.long.example.com -------------------------------------------------------------------------------- /tests/test_hn_05.html: -------------------------------------------------------------------------------- 1 |

Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.

    def hello():
 2 |         print('Hello World')
 3 |         return None
 4 |
5 | Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.

    def hello():
 6 |         print('Hello World')
 7 |         return None
 8 |
9 | Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus te. Noster nominati recteque no has.

10 | -------------------------------------------------------------------------------- /tests/test_hn_05.out: -------------------------------------------------------------------------------- 1 | Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus 2 | te. Noster nominati recteque no has. 3 | 4 | | def hello(): 5 | | print('Hello World') 6 | | return None 7 | 8 | Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus 9 | te. Noster nominati recteque no has. 10 | 11 | | def hello(): 12 | | print('Hello World') 13 | | return None 14 | 15 | Lorem ipsum dolor sit amet, pro eu soleat civibus. Mel quas sensibus 16 | te. Noster nominati recteque no has. -------------------------------------------------------------------------------- /tests/test_hn_06.html: -------------------------------------------------------------------------------- 1 |

Lorem ipsum dolor sit amet, pro eu soleat civibus.

    lambda L: [] if L==[] else qsort([x for x in L[1:] if x< L[0]]) + L[0:1] + qsort([x for x in L[1:] if x>=L[0]])
2 |
Mel quas sensibus te. Noster nominati recteque no has.

3 | -------------------------------------------------------------------------------- /tests/test_hn_06.out: -------------------------------------------------------------------------------- 1 | Lorem ipsum dolor sit amet, pro eu soleat civibus. 2 | 3 | | lambda L: [] if L==[] else qsort([x for x in L[1:] if x< L[0]]) + L[0:1] + qsort([x for x in L[1:] if x>=L[0]]) 4 | 5 | Mel quas sensibus te. Noster nominati recteque no has. -------------------------------------------------------------------------------- /tests/test_lb_01.html: -------------------------------------------------------------------------------- 1 |
2 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

3 |
4 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

5 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

6 |
7 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

8 |
9 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

10 | -------------------------------------------------------------------------------- /tests/test_lb_01.out: -------------------------------------------------------------------------------- 1 | > Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 2 | > eiusmod tempor 3 | 4 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 5 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 6 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 7 | adipiscing elit, sed do eiusmod tempor 8 | 9 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 10 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 11 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 12 | adipiscing elit, sed do eiusmod tempor 13 | 14 | > Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 15 | > eiusmod tempor 16 | 17 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 18 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 19 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 20 | adipiscing elit, sed do eiusmod tempor -------------------------------------------------------------------------------- /tests/test_lb_02.html: -------------------------------------------------------------------------------- 1 |
2 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

3 |
4 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

5 | 
def hello_world():
 6 |   print('hello')
 7 |   if True:
 8 |     print('world')
 9 |   print('!!!')
10 | 
11 | hello_world()
12 |
13 | -------------------------------------------------------------------------------- /tests/test_lb_02.out: -------------------------------------------------------------------------------- 1 | > Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 2 | > eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 3 | > elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 4 | > adipiscing elit, sed do eiusmod tempor 5 | 6 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 7 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 8 | elit, sed do eiusmod tempor. Lorem `ipsum` dolor sit amet, 9 | `consectetur adipiscing` elit, sed do eiusmod tempor 10 | 11 | | def hello_world(): 12 | | print('hello') 13 | | if True: 14 | | print('world') 15 | | print('!!!') 16 | | 17 | | hello_world() -------------------------------------------------------------------------------- /tests/test_lb_03.html: -------------------------------------------------------------------------------- 1 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

2 |
    3 |
  • Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor.
    • 4 |
  • Lorem: 5 |
      6 |
    • Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor.
      • 7 |
    • Lorem ipsum dolor sit amet, consectetur.
      • 8 |
    • Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor.
      • 9 |
    10 |
    • 11 |
12 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

13 | -------------------------------------------------------------------------------- /tests/test_lb_03.out: -------------------------------------------------------------------------------- 1 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 2 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 3 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 4 | adipiscing elit, sed do eiusmod tempor 5 | 6 | - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 7 | eiusmod tempor. 8 | - Lorem: 9 | 10 | - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 11 | eiusmod tempor. 12 | - Lorem ipsum dolor sit amet, consectetur. 13 | - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 14 | eiusmod tempor. 15 | 16 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 17 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 18 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 19 | adipiscing elit, sed do eiusmod tempor -------------------------------------------------------------------------------- /tests/test_lb_04.html: -------------------------------------------------------------------------------- 1 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit

2 |
3 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, example sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

4 |
5 |

6 | Lorem ipsum dolor sit
dolor amet, consectetur 7 | 8 | adipiscing 9 | 10 | 11 | elit 12 |

13 |
14 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

15 |
16 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

17 |
18 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit

-------------------------------------------------------------------------------- /tests/test_lb_04.out: -------------------------------------------------------------------------------- 1 | Lorem ipsum dolor sit amet, consectetur adipiscing elit 2 | 3 | > Lorem ipsum dolor sit amet, consectetur adipiscing elit, example 4 | > https://example.com/ sed do eiusmod tempor. Lorem ipsum dolor sit 5 | > amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem 6 | > ipsum dolor sit amet, consectetur `adipiscing elit`, sed do eiusmod 7 | > tempor 8 | > 9 | >> Lorem ipsum /dolor/ *sit 10 | >> dolor* amet, consectetur adipiscing elit 11 | >> 12 | >> ------------------------------------------------------------------ 13 | >> 14 | >> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 15 | >> eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 16 | >> elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, 17 | >> consectetur adipiscing elit, sed do eiusmod tempor 18 | > 19 | > Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 20 | > eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 21 | > elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 22 | > adipiscing elit, sed do eiusmod tempor 23 | 24 | Lorem ipsum dolor sit amet, consectetur adipiscing elit -------------------------------------------------------------------------------- /tests/test_lb_05.html: -------------------------------------------------------------------------------- 1 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

2 |
3 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

4 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

5 |
6 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

7 | -------------------------------------------------------------------------------- /tests/test_lb_05.out: -------------------------------------------------------------------------------- 1 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 2 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 3 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 4 | adipiscing elit, sed do eiusmod tempor 5 | 6 | ---------------------------------------------------------------------- 7 | 8 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 9 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 10 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 11 | adipiscing elit, sed do eiusmod tempor 12 | 13 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 14 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 15 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 16 | adipiscing elit, sed do eiusmod tempor 17 | 18 | ---------------------------------------------------------------------- 19 | 20 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 21 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 22 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 23 | adipiscing elit, sed do eiusmod tempor -------------------------------------------------------------------------------- /tests/test_lb_06.html: -------------------------------------------------------------------------------- 1 |
2 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit

3 |
4 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

5 |
6 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

7 |
8 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit

9 |
10 |
    11 |
  • Lorem ipsum dolor sit amet, consectetur adipiscing elit
    • 12 |
  • Lorem ipsum dolor sit amet, consectetur adipiscing elit
    • 13 |
  • Lorem ipsum dolor sit amet, consectetur adipiscing elit
    • 14 |
15 |
16 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

17 | -------------------------------------------------------------------------------- /tests/test_lb_06.out: -------------------------------------------------------------------------------- 1 | > Lorem ipsum dolor sit amet, consectetur adipiscing elit 2 | 3 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 4 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 5 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 6 | adipiscing elit, sed do eiusmod tempor 7 | 8 | > Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 9 | > eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 10 | > elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 11 | > adipiscing elit, sed do eiusmod tempor 12 | 13 | Lorem ipsum dolor sit amet, consectetur adipiscing elit 14 | 15 | > - Lorem ipsum dolor sit amet, consectetur adipiscing elit 16 | > - Lorem ipsum dolor sit amet, consectetur adipiscing elit 17 | > - Lorem ipsum dolor sit amet, consectetur adipiscing elit 18 | 19 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 20 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 21 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 22 | adipiscing elit, sed do eiusmod tempor -------------------------------------------------------------------------------- /tests/test_lb_07.html: -------------------------------------------------------------------------------- 1 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit

2 |
    3 |
  1. 4 |

    Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

    5 |
    2. 6 |
  2. 7 |

    Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

    8 |
    3. 9 |
  3. 10 |

    Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

    11 |
    4. 12 |
13 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

14 | -------------------------------------------------------------------------------- /tests/test_lb_07.out: -------------------------------------------------------------------------------- 1 | Lorem ipsum dolor sit amet, consectetur adipiscing elit 2 | 3 | - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 4 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 5 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 6 | adipiscing elit, sed do eiusmod tempor 7 | - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 8 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 9 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 10 | adipiscing elit, sed do eiusmod tempor 11 | - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 12 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 13 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 14 | adipiscing elit, sed do eiusmod tempor 15 | 16 | Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 17 | eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 18 | elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 19 | adipiscing elit, sed do eiusmod tempor -------------------------------------------------------------------------------- /tests/test_lb_08.html: -------------------------------------------------------------------------------- 1 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit

2 |
3 |
4 |
    5 |
  1. Lorem ipsum dolor sit amet, consectetur adipiscing elit
    2. 6 |
  2. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
    3. 7 |
  3. Lorem ipsum dolor sit amet, consectetur adipiscing elit
    4. 8 |
9 |
10 |
11 |
12 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit

13 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor

14 |
15 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit

16 | -------------------------------------------------------------------------------- /tests/test_lb_08.out: -------------------------------------------------------------------------------- 1 | Lorem ipsum dolor sit amet, consectetur adipiscing elit 2 | 3 | >> - Lorem ipsum dolor sit amet, consectetur adipiscing elit 4 | >> - Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 5 | >> eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 6 | >> adipiscing elit, sed do eiusmod tempor. Lorem ipsum dolor sit 7 | >> amet, consectetur adipiscing elit, sed do eiusmod tempor 8 | >> - Lorem ipsum dolor sit amet, consectetur adipiscing elit 9 | 10 | > Lorem ipsum dolor sit amet, consectetur adipiscing elit 11 | > 12 | > Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do 13 | > eiusmod tempor. Lorem ipsum dolor sit amet, consectetur adipiscing 14 | > elit, sed do eiusmod tempor. Lorem ipsum dolor sit amet, consectetur 15 | > adipiscing elit, sed do eiusmod tempor 16 | 17 | Lorem ipsum dolor sit amet, consectetur adipiscing elit -------------------------------------------------------------------------------- /tests/test_lb_09.html: -------------------------------------------------------------------------------- 1 |
2 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit

3 |
4 |
5 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit

6 | 
def hello_world():
 7 |   print('hello')
 8 |   if True:
 9 |     print('world')
10 |   print('!!!')
11 | 
12 | hello_world()
13 |
14 |
15 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit

16 | 
def hello_world():
17 |   print('hello')
18 |   if True:
19 |     print('world')
20 |   print('!!!')
21 | 
22 | hello_world()
23 |
24 |

Lorem ipsum dolor sit amet, consectetur adipiscing elit

25 | -------------------------------------------------------------------------------- /tests/test_lb_09.out: -------------------------------------------------------------------------------- 1 | > Lorem ipsum dolor sit amet, consectetur adipiscing elit 2 | 3 | > Lorem ipsum dolor sit amet, consectetur adipiscing elit 4 | > 5 | > | def hello_world(): 6 | > | print('hello') 7 | > | if True: 8 | > | print('world') 9 | > | print('!!!') 10 | > | 11 | > | hello_world() 12 | 13 | Lorem ipsum dolor sit amet, consectetur adipiscing elit 14 | 15 | | def hello_world(): 16 | | print('hello') 17 | | if True: 18 | | print('world') 19 | | print('!!!') 20 | | 21 | | hello_world() 22 | 23 | Lorem ipsum dolor sit amet, consectetur adipiscing elit --------------------------------------------------------------------------------