├── README.md ├── bin ├── zk ├── zk-assets-localize ├── zk-backlinks ├── zk-dups ├── zk-fix-links ├── zk-fts-search ├── zk-related-tags ├── zkn ├── zkrt ├── zks ├── zksim ├── zkt └── zkt-raw ├── lib └── note.rb ├── samples ├── 18113220711 C.md ├── 201811220711 D.md ├── 202001111211 A.md └── 202002020636 B.md └── test └── note_test.rb /README.md: -------------------------------------------------------------------------------- 1 | # zk 2 | 3 | Stupid-fast plain-text Zettelkasten (`zk`) built for (terminal) nerds. Heavily 4 | featuring `fzf`, `ripgrep`, `bat`, and `sqlite`. 5 | 6 | The goal of the `zk` repository is to collect scripts and configurations for 7 | other plain-text Zettelkasten users. 8 | 9 | **Note:** This is an on-going extraction from my local environment, but has the 10 | utilities I use by far the most often. I can almost guarantee you you're going 11 | to run into a stacktrace somewhere because of some utility that isn't installed, 12 | etc. Please open PRs/issues if it's not working, or you have questions, 13 | concerns, comments. I'd also love contributions of scripts to `bin/`, such as 14 | showing related notes, polish to search, Vim configs, etc. 15 | 16 | If you are looking for something slightly more complete that retains the 17 | Markdown-nature, you should take a look at [Obsidian][2]. After switching 18 | note-taking system every year for years, I am no longer interested in custom 19 | software and will stick to time-tested utilities. 20 | 21 | ![](https://pbs.twimg.com/media/EQGYhAJUYAEPC4j?format=jpg&name=4096x4096) 22 | 23 | In this `screenshot`, we have `zks` running in the top-right, `zkt` in the 24 | bottom-right, and `vim` in the left pane. `zk` can be used without `tmux`, but 25 | it's recommended to use `tmux`. 26 | 27 | ## Usage 28 | 29 | The scripts assumes `$ZK_PATH` is set to your Zettelkasten directory. Your 30 | Zettelkasten are markdown files in this directory. Nesting is presently not 31 | supported. Each note must have a 12-number prefix (date note was created), e.g. 32 | `202005050837 Monkey Ladder.md`. It's recommended to do some kind of backup, 33 | e.g. Dropbox/iCloud/... 34 | 35 | Your `zk` Zettelkasten is designed to be edited with your favourite editor. 36 | Currently `zk` only supports Vim natively. `zk` augments your editor with 37 | various scripts to help extract further value. 38 | 39 | `zk`. Open `vim` in the left pane, and `zks` in the right pane. Your 40 | launch-point! 41 | 42 | `zks`. `fzf`-enabled full-text search (top-right pane in screenshot above) over 43 | all your notes, using `sqlite`. The index updates automatically based on file 44 | modification. See the `FZF_DEFAULT_OPTS` below for various key-bindings you can 45 | use to open splits in Vim, copy to clipboard, etc. directly from here. `Alt-S` 46 | will find similar notes with `zksim`. 47 | 48 | `zksim`. Finds similar notes to the note passed as an argument. See [#1][1] for 49 | more. 50 | 51 | `zkt`. `fzf`-enabled tag browser. Pressing enter on a tag will show you notes 52 | with that tag. notes, using `sqlite`. See the `FZF_DEFAULT_OPTS` below for 53 | various key-bindings you can use to open splits, copy to clipboard, etc. 54 | directly from here. 55 | 56 | `zkt-raw`. Raw list of tags sorted by totals. Useful for other analysis. Used by 57 | `zkt`. 58 | 59 | `zkn`. Create a new note, with an appropriate prefix. 60 | 61 | `zk-assets-localize`. Given a file, downloads/copies the markdown images to `media/`. 62 | 63 | `zk-backlinks`. Adds back-links to each note. I.e., if A links to B, but B 64 | doesn't link to A, then it'll append `Backlink: [[A]]` to B. 65 | 66 | `zkrt`/`zk-related-tags`. Finds tags related to the ones in the passed file. You can 67 | pass `-t` to see a tree of tags. 68 | 69 | ## Installation 70 | 71 | Clone `zk` and add `bin/` to your `$PATH`: 72 | 73 | ``` 74 | $ git clone https://github.com/sirupsen/zk.git ~/zk 75 | $ echo 'export PATH=$PATH:$HOME/zk/bin' >> ~/.bashrc 76 | $ echo 'export ZK_PATH="$HOME/Zettelkasten"' >> ~/.bashrc 77 | ``` 78 | 79 | Install the dependencies with your package manager. 80 | 81 | MacOS: 82 | ```bash 83 | # brew install ripgrep fzf sqlite3 bat 84 | # gem install sqlite3 85 | ``` 86 | 87 | Linux: 88 | 89 | `build-essential`,`libsqlite3-dev` and `ruby-dev` are needed to install the sqlite3 gem. For example—on Debian/Ubuntu, run: 90 | ```bash 91 | # apt install ripgrep fzf sqlite3 bat build-essential libsqlite3-dev ruby ruby-dev 92 | # gem install sqlite3 93 | ``` 94 | 95 | For **Vim**, browse through my Vim config 96 | [this](https://github.com/sirupsen/dotfiles/blob/master/home/.vimrc) to add a 97 | `:Note`, shortcut for tags, auto-completing other notes after typing `[[`, etc. 98 | Another function to look at is `:GPT`, which will take your range and send it to 99 | GPT3 for completion. Very cool for a perspective! One day this'll be a plugin 100 | that's easier to install. 101 | 102 | If you're using `fzf` with `vim`, it's recommended to add this to your `bash` 103 | configuration. It adds super useful key-bindings to open files in splits 104 | (`Ctrl-X`/`Ctrl-V`) from `zkt` and `zks` directly. It also adds `Ctrl-O` to 105 | insert the file-name of whatever you're hovering into Vim, which is handy for 106 | links!: 107 | 108 | ```bash 109 | export FZF_DEFAULT_OPTS="--height=40% --multi --tiebreak=begin \ 110 | --bind 'ctrl-y:execute-silent(echo {} | pbcopy)' \ 111 | --bind 'alt-down:preview-down,alt-up:preview-up' \ 112 | --bind \"ctrl-v:execute-silent[ \ 113 | tmux send-keys -t \{left\} Escape :vs Space && \ 114 | tmux send-keys -t \{left\} -l {} && \ 115 | tmux send-keys -t \{left\} Enter \ 116 | ]\" 117 | --bind \"ctrl-x:execute-silent[ \ 118 | tmux send-keys -t \{left\} Escape :sp Space && \ 119 | tmux send-keys -t \{left\} -l {} && \ 120 | tmux send-keys -t \{left\} Enter \ 121 | ]\" 122 | --bind \"ctrl-o:execute-silent[ \ 123 | tmux send-keys -t \{left\} Escape :read Space ! Space echo Space && \ 124 | tmux send-keys -t \{left\} -l \\\"{}\\\" && \ 125 | tmux send-keys -t \{left\} Enter \ 126 | ]\"" 127 | ``` 128 | 129 | ## Syncing 130 | 131 | I recommend storing the notes in iCloud/Google Drive/Dropbox or whatever you use 132 | to normally sync files. Nice and simple. Some people store them in Git. 133 | 134 | [1]: https://github.com/sirupsen/zk/pull/1 135 | [2]: https://obsidian.md/ 136 | -------------------------------------------------------------------------------- /bin/zk: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | tmux new-window -c "${ZK_PATH}" "bash -l -c -i vim" \; split-window -h "bash -i -l "zks"" 4 | -------------------------------------------------------------------------------- /bin/zk-assets-localize: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | require 'pathname' 4 | 5 | file = ARGV[0] 6 | media_path = ENV["ZK_PATH"] + "/media" 7 | file_path = ENV["ZK_PATH"] + "/" + file 8 | body = File.read(file_path) 9 | id = file[/\A\d+/] 10 | images = body.scan(/!\[.*\]\((.+)\)/) 11 | media_identifier = Dir[ENV['ZK_PATH'] + "/media/#{id}-*"].map { |name| 12 | name.match(/#{id}-(\d+)/)[1].to_i 13 | }.max || 0 14 | 15 | images.each do |(img)| 16 | unless img.start_with?("media/") || img.start_with?(media_path) 17 | media_identifier += 1 18 | img_path = Pathname.new(img) 19 | # regex below to not deal with ?omg=hi&lol=what from urls 20 | new_path = "media/#{id}-#{media_identifier}#{img_path.extname[/\A\.\w+/i]}" 21 | full_new_path = "#{ENV["ZK_PATH"]}/#{new_path}" 22 | 23 | if img.start_with?("http") 24 | system("curl '#{img}' -o '#{full_new_path}'") 25 | else 26 | system("cp '#{img}' '#{full_new_path}'") 27 | end 28 | 29 | body.sub!(img, new_path) 30 | end 31 | end 32 | 33 | File.open(file_path, "w+") do |f| 34 | f.write(body) 35 | end 36 | -------------------------------------------------------------------------------- /bin/zk-backlinks: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | require_relative '../lib/note' 4 | 5 | confirm = !(ARGV.include?('--no-confirm') || ENV['SKIP_CONFIRM'] == 'true' || ARGV.include?('-f') || ARGV.include?('--force')) 6 | 7 | i = 0 8 | Note.all.each do |note| 9 | i += 1 10 | note.append_backlinks(confirm: confirm) 11 | end 12 | -------------------------------------------------------------------------------- /bin/zk-dups: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | require 'time' 3 | 4 | names_to_paths = {} 5 | id_to_paths = {} 6 | path_to_contents = {} 7 | path_to_id = {} 8 | 9 | Dir["*.md"].each do |path| 10 | if match = path.match(/(?\d+) (?.+?)\.md/) 11 | names_to_paths[match['name']] ||= [] 12 | names_to_paths[match['name']] << path 13 | id_to_paths[match['id']] ||= [] 14 | id_to_paths[match['id']] << path 15 | 16 | path_to_contents[path] = File.read(path) 17 | path_to_id[path] = match['id'] 18 | end 19 | end 20 | 21 | names_to_paths.select { |name, paths| paths.size > 1 }.each do |name, paths| 22 | paths.sort_by { |path| Time.parse(path_to_id[path]) } 23 | contents = paths.map { |path| path_to_contents[path] } 24 | all_equal = (contents.uniq.size == 1) 25 | 26 | puts '=========' 27 | puts "all_equal: #{all_equal}" 28 | puts paths 29 | 30 | if all_equal 31 | # save earliest time 32 | paths[1..-1].each do |path| 33 | File.delete(path) 34 | end 35 | end 36 | end 37 | -------------------------------------------------------------------------------- /bin/zk-fix-links: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | files = Dir[ENV['ZK_PATH'] + '/*.md'] 3 | basenames = files.map { |e| File.basename(e) } 4 | 5 | files.zip(basenames).each do |path, basename| 6 | body = File.read(path) 7 | new_body = body 8 | 9 | body.scan(/\[\[(.+?)\]\]/).each do |match| 10 | match = match.first 11 | case match 12 | # Convert raw numbers to full reference 13 | # [20200101 => 20200101 Hello World.md] 14 | when /\A\d+\z/ 15 | reference = basenames.find { |f| f =~ /\A#{Regexp.escape(match)}/ } 16 | reference ||= basenames.find { |f| f =~ /\A20#{Regexp.escape(match)}/ } 17 | raise "Match for #{match} not found for #{basename}" unless reference 18 | 19 | new_body = new_body.sub("[[#{match}]]", "[[#{reference}]]") 20 | puts "Fixing #{match}" 21 | # Convert files without .md extension to have it 22 | # This allows `gf` in Vim to work out. 23 | when /^[\w\s]+(?!.md)$/ 24 | new_body = new_body.sub("[[#{match}]]", "[[#{match}.md]]") 25 | else 26 | # We check whether the link is still valid and try to fix it.. 27 | # Usually due to a rename. 28 | unless basenames.find { |f| f == match } 29 | puts "#{basename}: Unable to find exact match for #{match}, looking by id.." 30 | id = match[/\A\d+/] 31 | next unless id 32 | 33 | right_note = basenames.find { |f| f.start_with?(id) } 34 | next unless right_note 35 | 36 | new_body = new_body.sub("[[#{match}]]", "[[#{right_note}]]") 37 | end 38 | end 39 | end 40 | 41 | if new_body != body 42 | puts "Fixing up #{basename}.." 43 | File.open(path, 'w') { |f| f.write(new_body) } 44 | end 45 | end 46 | -------------------------------------------------------------------------------- /bin/zk-fts-search: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | require 'set' 3 | require 'yaml' 4 | require 'English' 5 | 6 | # avoid rubygems 7 | # libx = File.expand_path("../sqlite3-ruby/lib", __FILE__) 8 | # $LOAD_PATH.unshift(libx) unless $LOAD_PATH.include?(libx) 9 | require 'sqlite3' 10 | 11 | if ARGV.delete('--backup') 12 | backup_file = "backup/#{Time.now.strftime('%Y-%m-%d')}-index.db" 13 | unless File.exist?(backup_file) 14 | system("cp index.db #{backup_file}") 15 | raise "Didn't back up properly" unless $CHILD_STATUS.success? 16 | end 17 | end 18 | 19 | # require 'byebug' 20 | # Will be rebuilt at any time. Nice and incremental. 21 | db = SQLite3::Database.new 'index.db' 22 | 23 | # Keep prefix indexes for "mos*" searches. 24 | # 25 | # TODO: It doesn't seem like SQLITE FTS5 supports synonyms well. That's ok, but 26 | # we're going to want that. We can download this database from Princeton, write 27 | # a parser for it (or use grind(1)). This should allow us to do potentially do 28 | # `OR` queries. Alternatively, and probably better, woluld be to see if Lucene 29 | # supports this. 30 | # 31 | # https://wordnet.princeton.edu/download/current-version 32 | db.execute <<-SQL 33 | CREATE VIRTUAL TABLE IF NOT EXISTS zettelkasten 34 | USING fts5(title, body, tags, mtime UNINDEXED, prefix = 3, tokenize = "porter unicode61"); 35 | SQL 36 | 37 | # Weigh tags higher, and title a bit higher. 38 | db.execute <<-SQL 39 | INSERT INTO zettelkasten (zettelkasten, rank) VALUES('rank', 'bm25(2.0, 1.0, 5.0, 0.0)'); 40 | SQL 41 | 42 | existing = {} 43 | raw_existing = db.execute('SELECT title, mtime FROM zettelkasten') 44 | raw_existing.each { |(title, mtime)| existing[title] = mtime.to_i } 45 | 46 | directories = Dir['*.md'] + Dir['highlights/*.md'] 47 | directories.each do |path| 48 | mtime = File.stat(path).mtime.to_i 49 | 50 | # Any file that's been modified since its entry in the full-text search index 51 | # will get updated (or if it doesn't exist, of course). 52 | if !existing[path] 53 | contents = File.read(path) 54 | tags = contents.scan(/#[\w-]+/).join(' ') 55 | db.execute(<<-SQL, [path, contents, tags, mtime]) 56 | INSERT INTO zettelkasten (title, body, tags, mtime) VALUES (?, ?, ?, ?); 57 | SQL 58 | elsif mtime > existing[path] # to_i because the stat may have more precision 59 | contents = File.read(path) 60 | tags = contents.scan(/#[\w-]+/).join(' ') 61 | db.execute(<<-SQL, [contents, tags, mtime.to_s, path]) 62 | UPDATE zettelkasten SET body = ?, tags = ?, mtime = ? WHERE title = ? 63 | SQL 64 | end 65 | 66 | existing[path] = 'VISITED' 67 | end 68 | 69 | # Delete any entries in the full text index that don't have files! 70 | existing.each do |(path, present)| 71 | puts db.execute('DELETE FROM zettelkasten WHERE title = ?;', [path]) unless present == 'VISITED' 72 | end 73 | 74 | file_cat = ARGV.delete('-f') 75 | 76 | # For preview 77 | if file_cat 78 | if !ARGV[1].empty? 79 | results = db.execute(<<-SQL, ARGV[0], ARGV[1]) 80 | SELECT rank, highlight(zettelkasten, 1, '\x1b[0;41m', '\x1b[0m') 81 | FROM zettelkasten WHERE title = ? AND zettelkasten MATCH ? ORDER BY rank; 82 | SQL 83 | # This is when it starts and there's no query input... 84 | else 85 | puts ARGV[0] 86 | results = db.execute(<<-SQL, ARGV[0]) 87 | SELECT rank, body FROM zettelkasten WHERE title = ?; 88 | SQL 89 | end 90 | elsif ARGV[0] 91 | results = db.execute(<<-SQL, ARGV.join(' ').gsub(/-_/, ' ')) 92 | SELECT rank, highlight(zettelkasten, 0, '\x1b[0;41m', '\x1b[0m'), tags 93 | FROM zettelkasten WHERE zettelkasten MATCH ? ORDER BY rank; 94 | SQL 95 | else 96 | results = db.execute('SELECT title FROM zettelkasten;') 97 | end 98 | 99 | results.each do |(_score, content, _tags)| 100 | # puts score 101 | # puts "\"#{content}\"\t\x1b[33m\"#{tags}\"\x1b[0m" 102 | if content.start_with?('highlights/') 103 | puts "\x1b[0;90m#{content}\x1b[0m" 104 | else 105 | puts content 106 | end 107 | end 108 | -------------------------------------------------------------------------------- /bin/zk-related-tags: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | # Crate a hash with all tags, their counts, and tags they relate to 4 | all_tags = {} 5 | all_md_files = Dir["*.md"] 6 | all_md_files.each do |file| 7 | content = File.read(file) 8 | file_tags = content.scan(/#[\w-]+/) 9 | file_tags.each do |file_tag| 10 | all_tags[file_tag] ||= [] 11 | all_tags[file_tag] += file_tags 12 | end 13 | end 14 | all_tags = Hash[all_tags.map { |(tag, related)| 15 | related = Hash[related.group_by { |e| e }.map { |(k, tags)| 16 | [k, tags.size] 17 | }.sort_by { |(_k, count)| -count }] 18 | 19 | related.delete(tag) 20 | related.delete("#notes-permanent") 21 | related.delete("#import-dynalist") 22 | related.delete("#zettelkasten") 23 | related.delete("#zettelkasten-import") 24 | related.reject! { |e| e.size <= 3 } 25 | 26 | [tag, related] 27 | }] 28 | 29 | tree = ARGV.delete("-t") 30 | # Get the list of files and tags, then merge together all the counts 31 | tags_and_files = ARGV.dup 32 | raise "pass one or more tag or file" if tags_and_files.empty? 33 | 34 | if tree 35 | def tree(file, depth = 0, visited = {}) 36 | visited[file] = file 37 | return unless File.exists?(file) 38 | content = File.read(file) 39 | tags = content.scan(/#[\w-]+/) 40 | links = content.scan(/\[\[(.+?)\]\]+/) 41 | 42 | padding = " " * depth 43 | puts padding + "\x1b[31m" + file 44 | 45 | tags.each do |tag| 46 | puts padding + " \x1b[34m" + tag 47 | end 48 | puts 49 | 50 | links.each do |(link)| 51 | tree(link, depth + 1, visited) unless visited[link] 52 | end 53 | end 54 | 55 | tree(tags_and_files[0]) 56 | else 57 | tags = tags_and_files.flat_map do |tag_or_file| 58 | if tag_or_file.end_with?(".md") 59 | content = File.read(tag_or_file) 60 | content.scan(/#[\w-]+/) 61 | elsif tag_or_file =~ /\A(\d{12})/ 62 | id = $1 63 | file = all_md_files.find { |file| file[0..11] == id } 64 | content = File.read(file) 65 | content.scan(/#[\w-]+/) 66 | else 67 | tag_or_file.start_with?("#") ? tag_or_file : "##{tag_or_file}" 68 | end 69 | end 70 | 71 | composite = {} 72 | related_files = tags_and_files.flat_map do |tag_or_file| 73 | if tag_or_file.end_with?(".md") 74 | file = tag_or_file 75 | content = File.read(tag_or_file) 76 | linked_files = content.scan(/\[\[(.+?)\]\]+/) 77 | 78 | composite[file] = 1 79 | 80 | # Only depth = 1 for now, allow deeper! 81 | linked_files.each do |(linked_file)| 82 | linked_file += ".md" unless linked_file.end_with?(".md") 83 | linked_file_content = File.read(linked_file) 84 | linked_file_tags = linked_file_content.scan(/#[\w-]+/) 85 | linked_file_links = linked_file_content.scan(/\[\[(.+?)\]\]+/) + [linked_file] 86 | 87 | linked_file_links.each do |(link)| 88 | # tag += linked_file_tags 89 | if composite[link] 90 | composite[link] += 1 91 | else 92 | composite[link] = 1 93 | end 94 | end 95 | end 96 | end 97 | end 98 | 99 | tags.each do |tag| 100 | if all_tags[tag] 101 | all_tags[tag].each do |(k, v)| 102 | if composite[k] 103 | composite[k] += v 104 | else 105 | composite[k] = v 106 | end 107 | end 108 | end 109 | end 110 | 111 | composite = composite.sort_by { |(_k, v)| -v } 112 | composite.each do |(tag, n)| 113 | if tags.include?(tag) 114 | puts "\x1b[90m#{n}\t#{tag}\x1b[0m" 115 | else 116 | puts "#{n}\t#{tag}" 117 | end 118 | end 119 | end 120 | -------------------------------------------------------------------------------- /bin/zkn: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | if [[ -z $1 ]]; then 3 | zks 4 | else 5 | args="$@" 6 | nvim -c ":set autochdir" "$ZK_PATH/$(date +"%Y%m%d%H%M") $args.md" 7 | fi 8 | -------------------------------------------------------------------------------- /bin/zkrt: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | zk-related 3 | -------------------------------------------------------------------------------- /bin/zks: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | cd "$ZK_PATH" 3 | 4 | fzf --ansi --height 100% --preview 'zk-fts-search --backup -f {} {q} | bat --language md --style=plain --color always' \ 5 | --bind "ctrl-o:execute-silent@tmux send-keys -t \{left\} Escape :read Space ! Space echo Space && \ 6 | tmux send-keys -t \{left\} -l '\"'\[\[{}]]'\"' && \ 7 | tmux send-keys -t \{left\} Enter@" \ 8 | --bind "enter:execute-silent[ \ 9 | tmux send-keys -t \{left\} :e Space && \ 10 | tmux send-keys -t \{left\} -l {} && \ 11 | tmux send-keys -t \{left\} Enter \ 12 | ]" \ 13 | --bind "change:reload:zk-fts-search --backup '{q}'" \ 14 | --phony --preview-window=top:65% --no-info --no-multi \ 15 | --bind "alt-s:execute:zksim {}" --query="${@}" 16 | -------------------------------------------------------------------------------- /bin/zksim: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | import sys 3 | import argparse 4 | import os 5 | import sqlite3 6 | 7 | try: 8 | from sklearn.metrics.pairwise import linear_kernel 9 | from sklearn.feature_extraction.text import TfidfVectorizer 10 | import pandas as pd 11 | except ImportError as e: 12 | print(f"Missing {e.name}! Please run pip3 install scikit-learn pandas") 13 | exit() 14 | 15 | 16 | """ 17 | This script takes a zettelkasten note filename and sorts the available notes by their similarity 18 | using Term Frequency-Inverse Document Frequency (TF-IDF). The idea is to use this to help one make 19 | connections in their zettelkasten they may have not thought of initially. 20 | """ 21 | 22 | 23 | def vectorize_text(series): 24 | """ 25 | Uses sklearn text vectorizer. Will do the necessary preprocessing like removing stop words, 26 | transform to lowercase etc. 27 | 28 | :param series: Pandas Series object 29 | :return: matrix of tf-idf features 30 | """ 31 | return TfidfVectorizer().fit_transform(series.values) 32 | 33 | 34 | def index_from_title(series, title): 35 | index = series[series == title].index 36 | if len(index) == 0: 37 | raise ValueError("Invalid note title! This note title does not exist in your zk.") 38 | return index 39 | 40 | 41 | def similarity_index(search_index, vectors): 42 | """ 43 | Uses cosine similarity to find which documents are the most similar to the file index passed 44 | in search_index 45 | :param search_index: Index of file you want similar files to 46 | :param vectors: Vector of TF-IDF vector features for each document 47 | :return: Returns the index numbers of the decuments in order of similarity 48 | """ 49 | cosine_similarities = linear_kernel(vectors[search_index], vectors).flatten() 50 | return (-cosine_similarities).argsort() 51 | 52 | 53 | def relevant_titles(df, title, title_col, text_col): 54 | """ 55 | Uses indexes from similarity_index to sort the DataFrame of notes by similarity 56 | :param df: DataFame of notes (from zettelkasten database) 57 | :param title: Title to search for similar files 58 | :param title_col: Name of column in DataFrame that has titles of each note 59 | :param text_col: Name of column in DataFrame that has the body of each note 60 | :return: DataFrame sorted by similarity to the note title passed 61 | """ 62 | vectors = vectorize_text(df[text_col]) 63 | searching_index = index_from_title(df[title_col], title) 64 | sim_index = similarity_index(searching_index, vectors) 65 | return df.iloc[sim_index][title_col].values 66 | 67 | 68 | class MyParser(argparse.ArgumentParser): 69 | def error(self, message): 70 | sys.stderr.write('error: %s\n' % message) 71 | self.print_help() 72 | sys.exit(2) 73 | 74 | 75 | class CustomAction(argparse.Action): 76 | def __call__(self, parser, namespace, values, option_string=None): 77 | setattr(namespace, self.dest, " ".join(values)) 78 | 79 | 80 | class TfidfSearch: 81 | 82 | def __init__(self): 83 | 84 | if 'ZK_PATH' in os.environ: 85 | self.zk_path = os.environ['ZK_PATH'] 86 | else: 87 | raise KeyError("ZK_PATH variable not defined! Run $ echo 'export ZK_PATH=$HOME/Zettelkasten' >> ~/.bashrc") 88 | 89 | self.conn = sqlite3.connect(os.path.join(self.zk_path, "index.db")) 90 | self.cursor = self.conn.cursor() 91 | self.num_files_to_show = 20 92 | 93 | def application_logic(self, filename): 94 | df = pd.read_sql("SELECT * FROM zettelkasten WHERE title NOT LIKE 'highlights/%'", con=self.conn) 95 | for file in relevant_titles(df, filename, title_col="title", text_col="body")[:self.num_files_to_show]: 96 | print(file) 97 | 98 | def run(self): 99 | parser = argparse.ArgumentParser(description='Perform document similarity search based on TF-IDF') 100 | parser.add_argument('filename', metavar='filename', type=str, nargs='+', action=CustomAction, 101 | help='filename to search for similarity') 102 | 103 | if len(sys.argv) == 1: 104 | parser.print_help(sys.stderr) 105 | sys.exit(1) 106 | 107 | args = parser.parse_args() 108 | self.application_logic(args.filename) 109 | 110 | 111 | if __name__ == "__main__": 112 | TfidfSearch().run() 113 | -------------------------------------------------------------------------------- /bin/zkt: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | zkt-raw | fzf --height 100% --no-info --no-multi \ 3 | --bind "ctrl-o:execute-silent[tmux send-keys -t \{left\} Escape :read Space ! Space echo Space && \ 4 | tmux send-keys -t \{left\} -l '\"\\'{2}'\"' && \ 5 | tmux send-keys -t \{left\} Enter]" \ 6 | --bind "ctrl-y:execute-silent(echo {2} | pbcopy),enter:execute[ \ 7 | rg -F --color=always -i {2} *.md -l | \ 8 | fzf --ansi --height 100% --preview-window=top:65% \ 9 | --bind 'enter:execute-silent$ \ 10 | tmux send-keys -t \{left\} Escape :e Space && \ 11 | tmux send-keys -t \{left\} -l \{} && \ 12 | tmux send-keys -t \{left\} Enter \ 13 | $' \ 14 | --bind \"ctrl-o:execute-silent[ \ 15 | tmux send-keys -t \{left\} Escape :read Space ! Space echo Space && \ 16 | tmux send-keys -t \{left\} -l '\\' \[ '\\' \[ \{} ]] && \ 17 | tmux send-keys -t \{left\} Enter \ 18 | ]\" \ 19 | --preview 'bat --color always --language md --style plain \{}' \ 20 | ]" 21 | -------------------------------------------------------------------------------- /bin/zkt-raw: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # useful for editors, used in zkt 3 | rg -oP "(?<=^|\s)#[\w\-_]{3,}" -t md -N --no-filename "$ZK_PATH" --glob '!scripts' | 4 | rg -v "^#(notes-|import-)" | \ 5 | awk ' { tot[$0]++ } END { for (i in tot) print tot[i], "\t", i } ' | \ 6 | gsort -r --numeric-sort 7 | -------------------------------------------------------------------------------- /lib/note.rb: -------------------------------------------------------------------------------- 1 | require 'time' 2 | require 'byebug' 3 | 4 | class Note 5 | attr_accessor :absolute_file_path 6 | 7 | @id_to_note = {} 8 | @backlinks = {} 9 | 10 | def initialize(absolute_file_path) 11 | @absolute_file_path = absolute_file_path 12 | end 13 | 14 | def ==(other) 15 | id == other.id 16 | end 17 | 18 | def eql?(other) 19 | id == other.id 20 | end 21 | 22 | def <=>(other) 23 | id == other.id 24 | end 25 | 26 | def self.from_absolute_path(absolute_path) 27 | Note.new(absolute_path) 28 | end 29 | 30 | def self.from_name(name) 31 | return from_id(name) if name =~ /\A\d+\z/ 32 | 33 | Note.new(File.join(ENV['ZK_PATH'], "#{name.chomp('.md')}.md")) 34 | end 35 | 36 | def self.from_id(id) 37 | all if @id_to_note.empty? 38 | @id_to_note[id.to_s] 39 | end 40 | 41 | def id 42 | @id ||= absolute_file_path.match(%r{#{Regexp.escape(ENV["ZK_PATH"])}/(\d+)})[1] 43 | end 44 | 45 | def body 46 | File.read(absolute_file_path) 47 | end 48 | 49 | def body=(new_body) 50 | File.open(absolute_file_path, 'w+') { |file| file.write(new_body) } 51 | end 52 | 53 | def backlinks 54 | self.class.backlinks[id] || [] 55 | end 56 | 57 | def append(text) 58 | body_before = body 59 | body_after = body_before.clone 60 | body_after << text << "\n" 61 | self.body = body_after 62 | end 63 | 64 | def append_backlinks(confirm: true) 65 | backlinks_text = (backlinks - links).map { |note| note.to_backlink }.join("\n") 66 | return if backlinks_text.empty? 67 | 68 | confirm(body, body + backlinks_text) if confirm 69 | append backlinks_text 70 | end 71 | 72 | def confirm(before, after) 73 | puts <<~EOS 74 | About to commit changes to #{name_with_ext} 75 | 76 | Before 77 | #{before} 78 | 79 | After 80 | #{after}\n 81 | EOS 82 | 83 | print 'Confirm? [Y/n] ' 84 | raise ArgumentError, 'Bailing!' unless $stdin.gets.strip.downcase == 'y' 85 | end 86 | 87 | def name_without_id 88 | name_without_ext.match(/\d+ (.+)/)[1] 89 | end 90 | 91 | def created_at 92 | Time.strptime(id, '%Y%m%d%H%M') 93 | end 94 | 95 | def name_without_ext 96 | name_with_ext.chomp('.md') 97 | end 98 | 99 | def name_with_ext 100 | File.basename(@absolute_file_path) 101 | end 102 | 103 | def to_link 104 | "[[#{name_with_ext}]]" 105 | end 106 | 107 | def to_backlink 108 | "Backlink: #{to_link}" 109 | end 110 | 111 | def self.backlinks 112 | return @backlinks unless @backlinks.empty? 113 | 114 | all.each do |note| 115 | note.links.each do |link| 116 | @backlinks[link.id] ||= [] 117 | @backlinks[link.id] << note 118 | @backlinks[link.id].uniq! 119 | end 120 | end 121 | 122 | @backlinks 123 | end 124 | 125 | def valid? 126 | return false unless File.exist?(@absolute_file_path) 127 | return false unless @absolute_file_path =~ %r{#{Regexp.escape(ENV["ZK_PATH"])}/\d+ } 128 | 129 | true 130 | end 131 | 132 | def links 133 | body.scan(/\[\[(.*)\]\]/).map { |(link)| Note.from_name(link) }.select { |note| note.valid? } 134 | end 135 | 136 | def self.all 137 | notes = Dir[File.join(ENV['ZK_PATH'], '/*.md')].select do |file| 138 | file =~ %r{#{Regexp.escape(ENV["ZK_PATH"])}/\d+ } 139 | end.map { |file| Note.from_absolute_path(file) } 140 | 141 | # TODO: raise on conflicts 142 | notes.each { |note| @id_to_note[note.id] = note } 143 | 144 | notes 145 | end 146 | end 147 | -------------------------------------------------------------------------------- /samples/18113220711 C.md: -------------------------------------------------------------------------------- 1 | [[202001111211]] 2 | -------------------------------------------------------------------------------- /samples/201811220711 D.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/sirupsen/zk/3883bac66289890031f28f12bfc2c5dedfa8b9c1/samples/201811220711 D.md -------------------------------------------------------------------------------- /samples/202001111211 A.md: -------------------------------------------------------------------------------- 1 | Hello World 2 | 3 | [[202002020636 B]] 4 | [[18113220711]] 5 | [[201811220711 D.md]] 6 | -------------------------------------------------------------------------------- /samples/202002020636 B.md: -------------------------------------------------------------------------------- 1 | [[invalid link]] 2 | -------------------------------------------------------------------------------- /test/note_test.rb: -------------------------------------------------------------------------------- 1 | require "minitest/autorun" 2 | require_relative "../lib/note" 3 | ENV["ZK_PATH"] = File.expand_path(File.join(File.dirname(__FILE__), "../samples")) 4 | 5 | class TestNote < Minitest::Test 6 | def setup 7 | @id = "202001111211" 8 | @name_without_ext = "#{@id} A" 9 | @name_with_ext = "#{@name_without_ext}.md" 10 | @note_abs_path = File.join(ENV["ZK_PATH"], @name_with_ext) 11 | end 12 | 13 | def test_from_absolute_path 14 | note = Note.from_absolute_path(@note_abs_path) 15 | assert_equal @note_abs_path, note.absolute_file_path 16 | assert_equal @id, note.id 17 | assert note.body 18 | end 19 | 20 | def test_from_name_without_extension 21 | note = Note.from_name(@name_without_ext) 22 | assert_equal @note_abs_path, note.absolute_file_path 23 | assert_equal @id, note.id 24 | assert note.body 25 | end 26 | 27 | def test_from_name_with_extension 28 | note = Note.from_name(@name_with_ext) 29 | assert_equal @note_abs_path, note.absolute_file_path 30 | assert_equal @id, note.id 31 | assert note.body 32 | end 33 | 34 | def test_from_id 35 | note = Note.from_id(@id) 36 | assert_equal @note_abs_path, note.absolute_file_path 37 | assert_equal @id, note.id 38 | assert note.body 39 | end 40 | 41 | def test_links 42 | note = Note.from_absolute_path(@note_abs_path) 43 | note.links.map { |link| link.body } # will raise if file doesn't exist 44 | end 45 | 46 | def test_backlinks 47 | note = Note.from_id("202002020636") 48 | assert_equal ["202001111211"], note.backlinks.map(&:id) 49 | end 50 | 51 | def test_name_with_extension 52 | note = Note.from_id(@id) 53 | assert_equal @name_with_ext, note.name_with_ext 54 | end 55 | 56 | def test_name_without_extension 57 | note = Note.from_id(@id) 58 | assert_equal @name_without_ext, note.name_without_ext 59 | end 60 | 61 | def test_name_without_id 62 | note = Note.from_id(@id) 63 | assert_equal "A", note.name_without_id 64 | end 65 | 66 | def test_created_at 67 | note = Note.from_id(@id) 68 | assert_equal Time.parse("2020-01-11 12:11:00"), note.created_at 69 | end 70 | 71 | def test_to_link 72 | note = Note.from_id(@id) 73 | assert_equal "[[202001111211 A]]", note.to_link 74 | end 75 | 76 | def test_append 77 | note = Note.from_id(@id) 78 | body_before = note.body 79 | note.append("more content") 80 | assert_equal "more content", note.body.split("\n").last 81 | ensure 82 | note.body = body_before 83 | end 84 | 85 | def test_append_backlinks 86 | note = Note.from_id("202002020636") 87 | body_before = note.body 88 | note.append_backlinks(confirm: false) 89 | assert_equal "Backlink: [[202001111211 A]]", note.body.split("\n").last 90 | ensure 91 | note.body = body_before 92 | end 93 | 94 | def test_append_no_backlinks_when_already_linked 95 | note = Note.from_id("18113220711") 96 | body_before = note.body 97 | note.append_backlinks(confirm: false) 98 | assert_equal body_before, note.body 99 | ensure 100 | note.body = body_before 101 | end 102 | end 103 | --------------------------------------------------------------------------------