├── README.md ├── build.py ├── init-tiny.el ├── obs_postproc.py ├── publish.el └── screenshots ├── apple-watch-sleep-sleep-feldman-books.png ├── obs_backlinks.jpg ├── obs_code.jpg ├── obs_image.jpg └── obs_search.jpg /README.md: -------------------------------------------------------------------------------- 1 | # braindump4000: heavily modified jethrokuan braindump 2 | 3 | Convert your nested org mode database with broken links into a Hugo 4 | website. 5 | 6 | In addition to nesting and broken links, braindump4000 also deals with 7 | it if you are already using Hugo to publish parts of your org mode 8 | database. 9 | 10 | Find the original braindump at 11 | 12 | You could also use org-publish, but on my 1000+ file database, that 13 | takes forever, even when most of the files don't have to be republished. 14 | 15 | P.S. This tool is now also able to munge that Hugo-destined output for 16 | Obsidian, so that you are able to access your org-mode database using 17 | the Obsidian mobile app. 18 | 19 | ## Quickstart 20 | 21 | Let's say the root directory containing your nested org-mode database is 22 | `~/notes/pkb4000` and you want a Hugo version in `~/notes/web-pkb4000` 23 | then you could do the following: 24 | 25 | ### Create the hugo website 26 | 27 | ``` shell 28 | cd ~/notes 29 | hugo new site web-pkb4000 30 | ``` 31 | 32 | Install jethrokuan's [cortex](https://github.com/jethrokuan/cortex) 33 | theme: 34 | 35 | ``` shell 36 | cd ~/notes/web-pkb4000/themes 37 | git clone https://github.com/jethrokuan/cortex.git 38 | ``` 39 | 40 | Install his 41 | [config.toml](https://github.com/jethrokuan/braindump/blob/master/config.toml) 42 | at the top-level. 43 | 44 | Add the root level of the config, add: 45 | 46 | ``` yaml 47 | refLinksErrorLevel = "WARNING" 48 | ``` 49 | 50 | ### Add two important pages 51 | 52 | Create `web-pkb4000/content/_index.md` for the front page: 53 | 54 | ``` markdown 55 | Well hello! 56 | 57 | See the [index](posts). 58 | ``` 59 | 60 | Very importantly, create an empty `web-pkb4000/search/_index.md` just so 61 | that your search works. 62 | 63 | ### Build your site with braindump4000 64 | 65 | ``` shell 66 | # I usually clone braindump4000 inside my notes database pkb4000 67 | cd notes/pkb4000 68 | git clone https://github.com/cpbotha/braindump4000.git 69 | cd notes/pkb4000/braindump4000 70 | # note that the destination directory is the desired Hugo section within "content" 71 | # for the cortex theme, this must be "posts" 72 | python3 build.py ~/notes/pkb4000 ~/notes/web-pkb4000/content/posts 73 | ``` 74 | 75 | The instructions above are just following my example. 76 | 77 | However, as long as you specify your input and output directories 78 | correctly, everything should work for different setups. 79 | 80 | ## Obsidian support 81 | 82 | braindump4000 is also able to transform your org-mode database into an 83 | Obsidian vault. 84 | 85 | This is a bit of a hack, but it does yield an Obsidian vault which is 86 | quite usable, including image attachments, backlinks and so on. 87 | 88 | This howto is simpler, because we don't need the whole Hugo site, only 89 | something that looks like an Obsidian vault: 90 | 91 | ``` shell 92 | # I usually clone braindump4000 inside my notes database pkb4000 93 | cd notes/pkb4000 94 | git clone https://github.com/cpbotha/braindump4000.git 95 | cd notes/pkb4000/braindump4000 96 | python3 build.py --obsidian ~/notes/pkb4000 ~/notes/obs-pkb4000/content/posts 97 | ``` 98 | 99 | After this, you can open `~/notes/obs-pkb4000/` as an Obsidian vault. 100 | 101 | The markdown in `content/posts/` has been massaged (see 102 | [`obs_postproc.py`](./obs_postproc.py) for details) to support Obsidian's 103 | particular expectations, and images generally end up in 104 | `static/ox-hugo`. 105 | 106 | Personally, I use [syncthing](https://syncthing.net/) and [Möbius 107 | Sync](https://www.mobiussync.com/) to push the converted vault into 108 | Obsidian's sandbox on my iPhone in order to get mobile access to my Org 109 | mode database. 110 | 111 | ### Obsidian app configuration 112 | 113 | Activate `Settings - Editor - Display - Srict line breaks` as the ox-hugo / 114 | markdown generally does add single line breaks inside paragraphs. 115 | 116 | These scripts add the org-mode note title as the first h1 / `#` heading to the 117 | file, whilst the Obsidian app wants to display the filename as the title. 118 | 119 | In order to hide the built-in Obsidian filename-title display, create the file 120 | `vault/.obsidian/snippets/hide-title.css` with the following contents: 121 | 122 | ```css 123 | div.inline-title { 124 | display: none; 125 | } 126 | ``` 127 | 128 | Then, in `settings - appearance - css snippets`, refresh and activate 129 | `hide-title`. 130 | 131 | ## Obligatory screenshot(s) 132 | 133 | In the screenshot below, I started with the Apple WatchOS 9 sleep 134 | tracking video note, from there the Sleep backlinks page, then the Seven 135 | and a Half Lessons about the Brain book notes and finally my Books 136 | backlinks. 137 | 138 | [![](screenshots/apple-watch-sleep-sleep-feldman-books.png)](file:screenshots/apple-watch-sleep-sleep-feldman-books.png) 139 | 140 | My braindump site is being served using 141 | [goStatic,](https://github.com/PierreZ/goStatic) tightly bound to a 142 | private [tailnet](https://tailscale.com/) IP. 143 | 144 | ### Obsidian mobile app 145 | 146 | [](screenshots/obs_search.jpg) 147 | [](screenshots/obs_code.jpg) 148 | [](screenshots/obs_image.jpg) 149 | [](screenshots/obs_backlinks.jpg) 150 | 151 | 152 | ## FAQ 153 | 154 | ### Why is this README markdown and not org? 155 | 156 | It started as an org-file, but then I ran into github's org mode support not 157 | including a way to specify image display size. See e.g. 158 | https://stackoverflow.com/questions/54926052/github-org-mode-html-export-image-resizing 159 | -------------------------------------------------------------------------------- /build.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import argparse 4 | import os 5 | from pathlib import Path 6 | 7 | parser = argparse.ArgumentParser( 8 | description="Convert Emacs Org mode database to Hugo / Obsidian" 9 | ) 10 | parser.add_argument( 11 | "org_dir", help="Directory containing your nested org mode database" 12 | ) 13 | parser.add_argument( 14 | "out_dir", 15 | help='Output directory. This must take the form hugo-site/content/something, where "something" is usually posts', 16 | ) 17 | parser.add_argument( 18 | "--obsidian", 19 | action="store_true", 20 | help="Perform Obsidian-targeted markdown transformations. Default: Hugo only.", 21 | ) 22 | parser.add_argument( 23 | "-j", type=int, help="Number of ninja threads. Leave out for default #CPUs." 24 | ) 25 | 26 | args = parser.parse_args() 27 | 28 | # you can reconfigure the below two paths for your setup. 29 | # In my case: 30 | # 1. this resolves to pkb4000: the top-level of my org-mode database 31 | org_dir = Path(args.org_dir).resolve() 32 | # 2. this is the destination hugo site SECTION, e.g. hugo-site/content/posts/ OR the obsidian vault 33 | out_dir = Path(args.out_dir).resolve() 34 | 35 | 36 | # we need to determine the hugo site dir based on the specified section dir 37 | # ox-hugo has some logic that relies on the "content/" convention 38 | try: 39 | # find last occurrence of "content" -- everything before that is hugo-site dir 40 | ci = next( 41 | ( 42 | i 43 | for i in range(len(out_dir.parts) - 1, -1, -1) 44 | if out_dir.parts[i] == "content" 45 | ) 46 | ) 47 | except StopIteration: 48 | print( 49 | "Your out_dir argument should contain the 'content' path component, e.g. hugo_site/content/posts/" 50 | ) 51 | exit() 52 | 53 | hugo_dir = Path(*out_dir.parts[0:ci]) 54 | 55 | 56 | # get full names of the el scripts ninja will require 57 | this_dir = Path(__file__).parent.resolve() 58 | init_tiny_el = this_dir / "init-tiny.el" 59 | publish_el = this_dir / "publish.el" 60 | obs_postproc_py = this_dir / "obs_postproc.py" 61 | 62 | # ... as we will be creating build.ninja and everything else in the output dir 63 | out_dir.mkdir(parents=True, exist_ok=True) 64 | os.chdir(out_dir) 65 | 66 | # the main goal is to convert all of these nested org files into md files in the output 67 | org_files_input = org_dir.rglob("*.org") 68 | # ... but we also want to copy across any source .md files 69 | md_files_input = org_dir.rglob("*.md") 70 | 71 | # optionally add script that does extra post-proc for obsidian 72 | post_proc = f" && {obs_postproc_py} $out" if args.obsidian else "" 73 | 74 | # for ninja, we have to escape space with $, i.e. " " -> "$ " 75 | def _ninja_escape(path: Path) -> str: 76 | """Escape space characters with `$` as required by ninja""" 77 | return str(path).replace(" ", "$ ") 78 | 79 | def _make_in_out(f): 80 | """Calculate and escape full input and output filenames""" 81 | rf = f.relative_to(org_dir) 82 | output_file = _ninja_escape(out_dir / rf.with_suffix(".md")) 83 | input_file = _ninja_escape(org_dir / rf) 84 | return input_file, output_file 85 | 86 | 87 | # - we create build.ninja in the output dir 88 | # - experiment: -nw added because it looked like the many emacs instances were messing with my wslg 89 | with Path("build.ninja").open("w") as ninja_file: 90 | ninja_file.write( 91 | f""" 92 | rule org2md 93 | command = emacs -nw --batch -l {init_tiny_el} -l {publish_el} --eval \"(cpb/publish \\"{org_dir}\\" \\"$in_\\" \\"{hugo_dir}\\" \\"$out_\\" )\"{post_proc} 94 | description = org2md $in 95 | 96 | rule COPY 97 | command = cp $in $out 98 | """ 99 | ) 100 | 101 | out_files_main = [] 102 | for f in org_files_input: 103 | input_file, output_file = _make_in_out(f) 104 | out_files_main.append(output_file) 105 | # note: we have to pass through our own $in_ and $out_ to the rule, 106 | # because if we use built-in $in and $out, ninja will single quote filenames 107 | # with spaces in them, and Emacs then reads those as literal single quotes 108 | ninja_file.write( 109 | f""" 110 | build {output_file}: org2md {input_file} 111 | in_ = {input_file} 112 | out_ = {output_file} 113 | """ 114 | ) 115 | 116 | # in the final stage, we copy across any .md files that already live in the 117 | # org hierarchy. This is because I sometimes have useful .md snippets between 118 | # my org-files. We only do this if it won't overwrite a same-named .md file 119 | # that was converted from .org into the output directory 120 | for f in md_files_input: 121 | input_file, output_file = _make_in_out(f) 122 | if output_file not in out_files_main: 123 | ninja_file.write( 124 | f""" 125 | build {output_file}: COPY {input_file} 126 | """ 127 | ) 128 | 129 | 130 | import subprocess 131 | 132 | cmd = ["ninja"] 133 | if args.j: 134 | cmd.extend(["-j", str(args.j)]) 135 | 136 | subprocess.call(cmd) 137 | -------------------------------------------------------------------------------- /init-tiny.el: -------------------------------------------------------------------------------- 1 | ;; - If you see this: 2 | ;; cpbotha@meepzen3:/tmp$ convert -trim -antialias orgtexFrCVbZ.pdf -quality 100 bleh.png 3 | ;; convert-im6.q16: attempt to perform an operation not allowed by the security policy `PDF' @ error/constitute.c/IsCoderAuthorized/408. 4 | ;; OR: "org-compile-file: File "/tmp/orgtexjdH822.png" wasn’t produced. Please adjust ‘imagemagick’ part of ‘org-preview-latex-process-alist’." 5 | ;; delete the 6 "disable ghostscript format types" lines in /etc/ImageMagick-6/policy.xml 6 | ;; https://stackoverflow.com/questions/52998331/imagemagick-security-policy-pdf-blocking-conversion 7 | 8 | ;; try to redirect AND disable recentf so it doesn't mess with ~/.emacs.d/recentf 9 | ;; (the problem is that when you disable recentf-mode, it tries to save the list first) 10 | (setq recentf-save-file (expand-file-name "recentf" "/tmp")) 11 | (recentf-mode -1) 12 | 13 | ;; we add melpa so we can install ox-hugo 14 | (setq package-archives 15 | '( 16 | ("elpa" . "https://elpa.gnu.org/packages/") 17 | ("nongnu" . "https://elpa.nongnu.org/nongnu/") 18 | ("melpa" . "https://melpa.org/packages/") 19 | ;; fallback for when the official ones act up 20 | ;;("melpa" . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/melpa/") 21 | ;;("gnu" . "https://raw.githubusercontent.com/d12frosted/elpa-mirror/master/gnu/") 22 | ) 23 | ) 24 | 25 | 26 | (package-initialize) 27 | 28 | ;; Bootstrap `use-package' 29 | ;; http://www.lunaryorn.com/2015/01/06/my-emacs-configuration-with-use-package.html 30 | ;; use-package autoloads will make sure it get pulled in at the right time 31 | ;; read "package autoloads": http://www.lunaryorn.com/2014/07/02/autoloads-in-emacs-lisp.html 32 | (unless (package-installed-p 'use-package) 33 | (package-refresh-contents) 34 | (package-install 'use-package)) 35 | 36 | (use-package ox-hugo 37 | :ensure t 38 | :config 39 | (setq org-export-with-broken-links 'mark) 40 | (setq org-hugo-external-file-extensions-allowed-for-copying 41 | '("jpg" "jpeg" "js" "json" "tiff" "png" "svg" "gif" "mp4" "pdf" "odt" "doc" "ppt" "xls" "docx" "pptx" "xlsx")) 42 | 43 | (plist-put (cdr (assoc 'imagemagick org-preview-latex-process-alist)) :latex-compiler '("pdflatex -shell-escape -interaction nonstopmode -output-directory %o %f") ) 44 | 45 | 46 | ) 47 | 48 | -------------------------------------------------------------------------------- /obs_postproc.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | #%% 4 | import argparse 5 | from pathlib import Path 6 | import re 7 | 8 | # this is for testing 9 | s = """ 10 | 11 | +++ 12 | title = "shelly-homekit firmware" 13 | author = ["meepzen3"] 14 | date = 2022-03-05T16:36:00+02:00 15 | draft = false 16 | +++ 17 | 18 | Hello 19 | 20 | [Note-taking]({{< relref "note_taking.md" >}}), [Personal Knowledge Management]({{< relref "personal_knowledge_management.md" >}}) 21 | 22 | some more links 23 | 24 | [Note-taking]({{< relref "note_taking.md#my-anchor" >}}), [Personal Knowledge Management]({{< relref "personal_knowledge_management.md" >}}) 25 | 26 | some more bleh bleh 27 | 28 | {{< figure src="/ox-hugo/2022-12-31_16-17-19_screenshot.png" caption="Figure 2: Random drop-outs in graph below. At least finished strong in last week of year." >}} 29 | 30 | ## what about pandoc {#what-about-pandoc} 31 | 32 | Some text here about why not pandoc. 33 | 34 | 35 | ### Some title {#some-title} 36 | 37 | Here we go again. 38 | 39 | 40 | Another para. 41 | 42 | ## test section {#test-section} 43 | 44 | Skip to [what about pandoc](#what-about-pandoc) 45 | 46 | It should not rewrite links like this [google search](https://google.com#something) 47 | 48 | """ 49 | 50 | # https://regex101.com/ is your friend! 51 | def transform(text): 52 | # replace whole hugo header with "# the title" 53 | # note the two strategically placed non-greedy *? operators else this will eat up additional example +++ blocks 54 | t2 = re.sub(r'^\s*\+\+\+.*?title = "([^"]+)".*?\+\+\+', "# \\1", text, flags=re.DOTALL) 55 | 56 | # 1. replace all [Name]({{< relref "localfile.md" >}}) links with "normal" [title]() links 57 | # - note we only search and replace the part within the latter [...] part 58 | # - we surround the linkdest inside [..] with <> as per CommonMark https://spec.commonmark.org/0.30/#link-destination 59 | # so that filenames with spaces and other ASCII control characters also work 60 | # 2. In addition, work around Obsidian's silly lack of user-defined named anchor support: 61 | # rewrite all links with anchors to `#^anchor` obs blockrefs; here we do only the relref (otherfile) ones 62 | # See t35 next, and t5 below for the rest of this hack 63 | t3 = re.sub(r'{{< relref "([^"#]+)(#([^"]+))?" >}}', lambda m: "<" + m.group(1) + (f"#^{m.group(3)}" if m.group(3) else '') + ">", t2) 64 | 65 | # here we rewrite all local [label](#anchor) to [label](#^anchor) 66 | t35 = re.sub(r'(\[[^]]+\])\(#([^)]+)\)', '\\1(#^\\2)', t3) 67 | 68 | # replace 69 | # {{< figure src="/ox-hugo/2022-12-31_16-17-19_screenshot.png" caption="Figure 2: Random drop-outs in graph below. At least finished strong in last week of year." >}} 70 | # with 71 | # ![](static/ox-hugo/...) 72 | # BTW obsidian finds the figure even without any prepended path components, as long as it's in the vault 73 | # until we figure out a better scheme, we rewrite the path to be relative to the site / vault top-level 74 | t4 = re.sub('{{< figure src="/ox-hugo/([^"]+)".* >}}', '![](static/ox-hugo/\\1)', t35) 75 | 76 | # here we rewrite e.g. "## Some heading {#some-heading}" into "## Some heading\n^some-heading" 77 | # in other words, to work around Obsidian's disappointing lack of named anchor support we 78 | # use a ^block-marker, which means that we also have to rewrite all links 79 | # (For obsidian, we would suggest either 80 | # 1. ``, 81 | # 2. a setting to follow github convention in transforming "My Heading" into anchor name "my-heading" or 82 | # 3. supporting extended "# My heading {#my-heading}" markdown anchors.) 83 | t5 = re.sub(r'^( *#+ +[^{]+)\s+{#([^}]+)}', '\\1\n^\\2', t4, flags=re.MULTILINE) 84 | 85 | return t5 86 | 87 | # uncomment when testing with cell-based execution 88 | #print(transform(s)) 89 | 90 | #%% 91 | def main(): 92 | parser = argparse.ArgumentParser(description='perform obsidian post-processing on ox-hugo export') 93 | parser.add_argument('md_file', 94 | help='Markdown file that will be transformed in place') 95 | 96 | args = parser.parse_args() 97 | 98 | p = Path(args.md_file) 99 | 100 | text = p.read_text() 101 | t_text = transform(text) 102 | p.write_text(t_text) 103 | 104 | 105 | if __name__ == "__main__": 106 | main() 107 | 108 | 109 | -------------------------------------------------------------------------------- /publish.el: -------------------------------------------------------------------------------- 1 | ;; cpbotha: if org-hugo--search-and-get-anchor (called by org-hugo-link) yields error, 2 | ;; export stops, so here we advise it to just report and return the "normal" anochor-not-found "" 3 | ;; (my database is old, sometimes there are links to files outside of ith that don't exist anymore) 4 | (defun oh--saga (orig-org-hugo--search-and-get-anchor &rest args) 5 | (condition-case err 6 | (apply orig-org-hugo--search-and-get-anchor args) 7 | (error (progn 8 | (message "=====> org-link-search IGNORED ERROR: %s" err) 9 | "")))) 10 | 11 | ;; if you DO WANT ox-hugo to error out on each missing link so you can fix it, just comment out the line below 12 | (advice-add #'org-hugo--search-and-get-anchor :around #'oh--saga) 13 | 14 | ;; this is the main publish org -> md function 15 | ;; it improves over jethrokuan's original in the following ways: 16 | ;; - supports nested org-files (it will maintain the directory structure on the hugo side) 17 | ;; - overrides any HUGO directives you might already have in your org mode files 18 | (defun cpb/publish (org-dir in_file hugo-base-dir out_file) 19 | (with-current-buffer (find-file-noselect in_file) 20 | ;; unfortunately, these are overridden by e.g. #+HUGO_BASE_DIR in the file if they are present 21 | (let* ((org-hugo-base-dir hugo-base-dir) 22 | ;; directory-files-recursively is 10x faster than find-lisp-find-files 23 | (org-id-extra-files (directory-files-recursively org-dir "\.org$")) 24 | (org-hugo-content (file-name-concat org-hugo-base-dir "content")) 25 | ;; the section is the directory relative to "content" containing the output md 26 | (org-hugo-section (file-name-directory (file-relative-name out_file org-hugo-content))) 27 | (cpb-oh-pub-dir (file-name-concat org-hugo-content org-hugo-section))) 28 | 29 | ;; org-hugo--get-pub-dir is called by org-hugo-export-to-md right after reading all hugo vars 30 | ;; we override it in order to 31 | ;; temporarily override org-hugo--get-pub-dir else #+HUGO_* file properties override ours! 32 | ;; we restore _our_ base/section into info, and we also return the full pub-dir to where we want it 33 | (cl-letf (((symbol-function 'org-hugo--get-pub-dir) #'(lambda (info) 34 | (plist-put info :hugo-base-dir org-hugo-base-dir) 35 | (plist-put info :hugo-section org-hugo-section) 36 | (plist-put info :hugo-bundle nil) 37 | cpb-oh-pub-dir))) 38 | 39 | (org-hugo-export-to-md) 40 | )))) 41 | -------------------------------------------------------------------------------- /screenshots/apple-watch-sleep-sleep-feldman-books.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpbotha/braindump4000/5faba281343eac1663e7860d7e854405f4068d93/screenshots/apple-watch-sleep-sleep-feldman-books.png -------------------------------------------------------------------------------- /screenshots/obs_backlinks.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpbotha/braindump4000/5faba281343eac1663e7860d7e854405f4068d93/screenshots/obs_backlinks.jpg -------------------------------------------------------------------------------- /screenshots/obs_code.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpbotha/braindump4000/5faba281343eac1663e7860d7e854405f4068d93/screenshots/obs_code.jpg -------------------------------------------------------------------------------- /screenshots/obs_image.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpbotha/braindump4000/5faba281343eac1663e7860d7e854405f4068d93/screenshots/obs_image.jpg -------------------------------------------------------------------------------- /screenshots/obs_search.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/cpbotha/braindump4000/5faba281343eac1663e7860d7e854405f4068d93/screenshots/obs_search.jpg --------------------------------------------------------------------------------