.
675 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 |
2 |
3 | 
4 |
5 | 
7 |
8 | 
10 |
11 | 
13 |
14 | 
16 |
17 |
18 | ANNOUNCEMENT: This plugin has been rewritten with new name Juggl. It no longer requires Neo4j and Python, and has a lot more features than Neo4j graph view.
19 | You can install this new plugin from the Obsidian community plugins settings!
20 | Note that the Neo4j Graph View plugin will be removed from the community plugins soon.
21 |
22 | ## Neo4j Graph View
23 | 
24 |
25 | Documentation at https://juggl.io/Neo4j+Graph+View/Neo4j+Graph+View+Plugin.
26 |
27 | Join the new Discord server to discuss the plugin: https://discord.gg/sAmSGpaPgM
28 |
29 | Adds a new and much more functional graph view to Obsidian. It does so by connecting
30 | to a [Neo4j](https://neo4j.com/) database. Features:
31 | - Selectively style nodes and edges by tags, folders and link types
32 | - Selective expansion and hiding of nodes
33 | - View images within the graph
34 | - [Cypher](https://neo4j.com/developer/cypher/) querying
35 | - Typed links using `- linkType [[note 1]], [[note 2|alias]]`
36 | - Hierarchical layout
37 |
38 | Next up:
39 | - [x] Remove the need to install Neo4j and Python
40 | - [ ] Different and more stable front end
41 | - [x] Standardize style sheet using CSS instead of JSON
42 |
43 | A [Roadmap](https://juggl.io/Roadmap) with planned features is also available.
44 |
45 | 
46 |
47 | ### Installation
48 | Detailed installation instructions is at https://juggl.io/Neo4j+Graph+View/Installation+of+Neo4j+Graph+View+Plugin
49 | 1. Make sure you have [Python 3.6+](https://www.python.org/downloads/) installed. It needs the system-installed Python. Make sure to add Python to PATH!
50 | 2. Make sure you have [Neo4j desktop](https://neo4j.com/download/) installed
51 | 4. Create a new database in Neo4j desktop and start it. Record the password you use!
52 | 5. In the settings of the plugin, enter the password. Then run the restart command.
53 |
54 | If installing Python seems daunting, you can wait a couple of weeks. The goal is to port that code to Javascript.
55 |
56 | ### Use
57 | Detailed getting started guide is at https://juggl.io/Neo4j+Graph+View/Using+the+Neo4j+Graph+View
58 |
59 | On an open note, use the command "Neo4j Graph View: Open local graph of note". You can run commands using ctrl/cmd+p. Alternatively, you can bind this command to a hotkey in the settings.
60 |
61 | The settings contains several options, such as coloring based on folders and a hierarchical layout.
62 |
63 | #### Cypher Querying
64 | Create code blocks with language `cypher`. In this code block, create your Cypher query. Then, when the cursor is on this
65 | code block, use the Obsidian command 'Neo4j Graph View: Execute Cypher query'. Example:
66 |
67 | 
68 |
69 |
70 | ### Possible problems
71 | All changes made in obsidian should be automatically reflected in Neo4j, but this is still very buggy.
72 |
73 | If you are running into issues, see https://juggl.io/Neo4j+Graph+View/Installation+of+Neo4j+Graph+View+Plugin#troubleshooting
74 | ### Semantics
75 | The plugin collects all notes with extension .md in the input directory (default: `markdown/`). Each note is interpreted as follows:
76 | - Interprets tags as entity types
77 | - Interprets YAML frontmatter as entity properties
78 | - Interprets wikilinks as links with type `inline`, and adds content
79 | - Lines of the format `"- linkType [[note 1]], [[note 2|alias]]"` creates links with type `linkType` from the current note to `note 1` and `note 2`.
80 | - The name of the note is stored in the property `name`
81 | - The content of the note (everything except YAML frontmatter and typed links) is stored in the property `content`
82 | - Links to notes that do not exist yet are created without any types.
83 |
84 |
85 | ## Other visualization and querying options
86 | Another use case for this plugin is to use your Obsidian vault in one of the many apps in the Neo4j desktop
87 | Graph Apps Store. Using with this plugin active will automatically connect it to your vault. Here are some suggestions:
88 | ### Neo4j Bloom
89 | [Neo4j bloom](https://neo4j.com/product/bloom/) is very powerful graph visualization software. Compared to the embedded
90 | graph view in Obsidian, it offers much more freedom in customization.
91 |
92 | 
93 |
94 |
95 | ### GraphXR
96 | [GraphXR](https://www.kineviz.com/) is a 3D graph view, which looks quite gorgeous!
97 |
98 | 
99 |
100 |
101 | ### Neo4j Browser
102 | A query browser that uses the Cypher language to query your vault. Can be used for advanced queries or data anlysis of
103 | your vault.
104 |
105 | 
106 |
107 |
108 | ## Python code: Semantic Markdown to Neo4j
109 | This Obsidian plugin uses the Python package `semantic-markdown-converter`, which is also in this repo.
110 | It creates an active data stream from a folder of Markdown notes to a Neo4j database.
111 | For documentation, see https://juggl.io/Neo4j+Graph+View/Semantic+Markdown+Converter
112 |
--------------------------------------------------------------------------------
/manifest.json:
--------------------------------------------------------------------------------
1 | {
2 | "id": "neo4j-graph-view",
3 | "name": "Neo4j Graph View",
4 | "version": "0.2.6",
5 | "minAppVersion": "0.9.16",
6 | "description": "An Obsidian plugin for advanced graph visualization and querying using Neo4j.",
7 | "author": "Emile",
8 | "authorUrl": "https://twitter.com/emilevankrieken",
9 | "isDesktopOnly": true
10 | }
11 |
--------------------------------------------------------------------------------
/neo4j-graph-view/.gitignore:
--------------------------------------------------------------------------------
1 | # Intellij
2 | *.iml
3 | .idea
4 |
5 | # npm
6 | node_modules
7 | package-lock.json
8 |
9 | # build
10 | main.js
11 | *.js.map
--------------------------------------------------------------------------------
/neo4j-graph-view/README.md:
--------------------------------------------------------------------------------
1 | ## Neo4j Graph View
2 | 
3 |
4 | Adds a new and much more functional graph view to Obsidian. It does so by connecting
5 | to a [Neo4j](https://neo4j.com/) database. Features:
6 | - Color nodes by tags
7 | - Selective expansion and hiding of nodes
8 | - Typed links using `- linkType [[note 1]], [[note 2|alias]]`
9 | - Hierarchical layout
10 |
11 | ### Installation
12 | 1. Make sure you have python 3.6+ installed
13 | 2. Make sure you have [Neo4j desktop](https://neo4j.com/download/) installed
14 | 4. Create a new database in Neo4j desktop and start it. Record the password you use!
15 | 5. In the settings of the plugin, enter the password. Then run the restart command.
16 |
17 | ### Use
18 | On an open node, use the command "Neo4j Graph View: Open local graph of note".
19 | - Click on a node to open it in the Markdown view
20 | - Double-click on a node to expand its neighbors
21 | - Shift-drag in the graph view to select nodes
22 | - Use E to expand the neighbors of all selected nodes
23 | - Use H or Backspace to hide all selected nodes from the view
24 | - Use I (invert) to select all nodes that are not currently selected
25 | - Use A to select all nodes
26 | - All notes visited are added to the graph
27 |
28 |
29 | ### Possible problems
30 | All changes made in obsidian should be automatically reflected in Neo4j, but this is still very buggy. There also seem
31 | to be problems with duplicate nodes in the graph.
32 |
33 | ### Semantics
34 | This collects all notes with extension .md in the input directory (default: `markdown/`). Each note is interpreted as follows:
35 | - Interprets tags as entity types
36 | - Interprets YAML frontmatter as entity properties
37 | - Interprets wikilinks as links with type `inline`, and adds content
38 | - Lines of the format `"- linkType [[note 1]], [[note 2|alias]]"` creates links with type `linkType` from the current note to `note 1` and `note 2`.
39 | - The name of the note is stored in the property `name`
40 | - The content of the note (everything except YAML frontmatter and typed links) is stored in the property `content`
41 | - Links to notes that do not exist yet are created without any types.
42 |
43 | This uses a very simple syntax for typed links. There is no agreed-upon Markdown syntax for this as of yet.
44 | If you are interested in using a different syntax than the list format `"- linkType [[note 1]], [[note 2|alias]]"`,
45 | please submit an issue.
46 |
--------------------------------------------------------------------------------
/neo4j-graph-view/main.ts:
--------------------------------------------------------------------------------
1 | import {
2 | FileSystemAdapter,
3 | MarkdownView, MenuItem, normalizePath,
4 | Notice,
5 | Plugin, Scope, TAbstractFile, TFile,
6 | WorkspaceLeaf
7 | } from 'obsidian';
8 | import {INeo4jViewSettings, Neo4jViewSettingTab, DefaultNeo4jViewSettings} from "./settings";
9 | import {exec, ChildProcess, spawn} from 'child_process';
10 | import {promisify} from "util";
11 | import {PythonShell} from "python-shell";
12 | import {NV_VIEW_TYPE, NeoVisView, MD_VIEW_TYPE, PROP_VAULT} from "./visualization";
13 | // import 'express';
14 | import {IncomingMessage, Server, ServerResponse} from "http";
15 | import {Editor} from "codemirror";
16 | import {start} from "repl";
17 | import {Neo4jError} from "neo4j-driver";
18 | import {IdType} from "vis-network";
19 |
20 | // I got this from https://github.com/SilentVoid13/Templater/blob/master/src/fuzzy_suggester.ts
21 | const exec_promise = promisify(exec);
22 |
23 | const STATUS_OFFLINE = "Neo4j stream offline";
24 |
25 | const DEVELOP_MODE = false;
26 |
27 | export default class Neo4jViewPlugin extends Plugin {
28 | settings: INeo4jViewSettings;
29 | stream_process: PythonShell;
30 | path: string;
31 | statusBar: HTMLElement;
32 | neovisView: NeoVisView;
33 | imgServer: Server;
34 |
35 | async onload() {
36 | let noticeText = "WARNING: Neo4j Graph View is deprecated and replaced by the new Obsidian plugin Juggl."
37 | new Notice(noticeText);
38 | console.log(noticeText);
39 | if (this.app.vault.adapter instanceof FileSystemAdapter) {
40 | this.path = this.app.vault.adapter.getBasePath();
41 | }
42 |
43 | this.settings = Object.assign(DefaultNeo4jViewSettings, await this.loadData());//(await this.loadData()) || DefaultNeo4jViewSettings;
44 | this.statusBar = this.addStatusBarItem();
45 | this.statusBar.setText(STATUS_OFFLINE);
46 |
47 | // this.registerView(NV_VIEW_TYPE, (leaf: WorkspaceLeaf) => this.neovisView=new NeoVisView(leaf, this.app.workspace.activeLeaf?.getDisplayText(), this))
48 |
49 | this.addCommand({
50 | id: 'restart-stream',
51 | name: 'Restart Neo4j stream',
52 | callback: () => {
53 | console.log('Restarting stream');
54 | this.restart();
55 | },
56 | });
57 |
58 | this.addCommand({
59 | id: 'stop-stream',
60 | name: 'Stop Neo4j stream',
61 | callback: () => {
62 | this.shutdown();
63 | },
64 | });
65 |
66 | // this.addCommand({
67 | // id: 'open-bloom-link',
68 | // name: 'Open note in Neo4j Bloom',
69 | // callback: () => {
70 | // if (!this.stream_process) {
71 | // new Notice("Cannot open in Neo4j Bloom as neo4j stream is not active.")
72 | // }
73 | // let active_view = this.app.workspace.getActiveViewOfType(MarkdownView);
74 | // if (active_view == null) {
75 | // return;
76 | // }
77 | // let name = active_view.getDisplayText();
78 | // // active_view.getState().
79 | //
80 | // console.log(encodeURI("neo4j://graphapps/neo4j-bloom?search=SMD_no_tags with name " + name));
81 | // open(encodeURI("neo4j://graphapps/neo4j-bloom?search=SMD_no_tags with name " + name));
82 | // // require("electron").shell.openExternal("www.google.com");
83 | // },
84 | // });
85 |
86 | this.addCommand({
87 | id: 'open-vis',
88 | name: 'Open local graph of note',
89 | callback: () => {
90 | let active_view = this.app.workspace.getActiveViewOfType(MarkdownView);
91 | if (active_view == null) {
92 | return;
93 | }
94 | let name = active_view.getDisplayText();
95 | this.openLocalGraph(name);
96 | },
97 | });
98 |
99 | this.addCommand({
100 | id: 'execute-query',
101 | name: 'Execute Cypher query',
102 | callback: () => {
103 | if (!this.stream_process) {
104 | new Notice("Cannot open local graph as neo4j stream is not active.")
105 | return;
106 | }
107 | this.executeQuery();
108 | },
109 | });
110 |
111 | this.addSettingTab(new Neo4jViewSettingTab(this.app, this));
112 |
113 | this.app.workspace.on("file-menu", ((menu, file: TFile) => {
114 | menu.addItem((item) =>{
115 | item.setTitle("Open Neo4j Graph View").setIcon("dot-network")
116 | .onClick(evt => {
117 | if (file.extension === "md") {
118 | this.openLocalGraph(file.basename);
119 | }
120 | else {
121 | this.openLocalGraph(file.name);
122 | }
123 | });
124 | })
125 | }));
126 |
127 |
128 | await this.initialize();
129 |
130 |
131 | }
132 |
133 | public getFileFromAbsolutePath(abs_path: string): TAbstractFile {
134 | const path = require('path');
135 | const relPath = path.relative(this.path, abs_path);
136 | return this.app.vault.getAbstractFileByPath(relPath);
137 | }
138 |
139 | public async openFile(file: TFile) {
140 | const md_leaves = this.app.workspace.getLeavesOfType(MD_VIEW_TYPE).concat(this.app.workspace.getLeavesOfType('image'));
141 | // this.app.workspace.iterateAllLeaves(leaf => console.log(leaf.view.getViewType()));
142 | if (md_leaves.length > 0) {
143 | await md_leaves[0].openFile(file);
144 | }
145 | else {
146 | await this.app.workspace.getLeaf(true).openFile(file);
147 | }
148 | }
149 |
150 | public async restart() {
151 | new Notice("Restarting Neo4j stream.");
152 | await this.shutdown();
153 | await this.initialize();
154 | }
155 |
156 | public async initialize() {
157 | console.log('Initializing Neo4j stream');
158 | try {
159 | let out = await exec_promise("pip3 install --upgrade pip " +
160 | "--user ", {timeout: 10000000});
161 |
162 | if (this.settings.debug) {
163 | console.log(out.stdout);
164 | }
165 | console.log(out.stderr);
166 | let {stdout, stderr} = await exec_promise("pip3 install --upgrade semantic-markdown-converter " +
167 | "--no-warn-script-location " +
168 | (DEVELOP_MODE ? "--index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple " : "") +
169 | "--user ", {timeout: 10000000});
170 | if (this.settings.debug) {
171 | console.log(stdout);
172 | }
173 | console.log(stderr);
174 | }
175 | catch (e) {
176 | console.log("Error during updating semantic markdown: \n", e);
177 | new Notice("Error during updating semantic markdown. Check the console for crash report.");
178 | }
179 | let options = {
180 | args: ['--input', this.path,
181 | '--password', this.settings.password,
182 | '--typed_links_prefix', this.settings.typed_link_prefix,
183 | '--community', this.settings.community]
184 | .concat(this.settings.debug ? ["--debug"] : [])
185 | .concat(this.settings.convert_markdown ? ["--convert_markdown"] : [])
186 | };
187 | try {
188 | // @ts-ignore
189 | this.stream_process = PythonShell.runString("from smdc.stream import main;" +
190 | "main();", options, function(err, results) {
191 | if (err) throw err;
192 | console.log('Neo4j stream killed');
193 | });
194 | let plugin = this;
195 | process.on("exit", function() {
196 | plugin.shutdown();
197 | })
198 | let statusbar = this.statusBar;
199 | let settings = this.settings;
200 | this.stream_process.on('message', function (message) {
201 | // received a message sent from the Python script (a simple "print" statement)
202 | if (message === 'Stream is active!') {
203 | console.log(message);
204 | new Notice("Neo4j stream online!");
205 | statusbar.setText("Neo4j stream online");
206 | }
207 | else if (message === 'invalid user credentials') {
208 | console.log(message);
209 | new Notice('Please provide a password in the Neo4j Graph View settings');
210 | statusbar.setText(STATUS_OFFLINE);
211 | }
212 | else if (message === 'no connection to db') {
213 | console.log(message);
214 | new Notice("No connection to Neo4j database. Please start Neo4j Database in Neo4j Desktop");
215 | statusbar.setText(STATUS_OFFLINE);
216 | }
217 | else if (/^onSMD/.test(message)) {
218 | if (settings.debug) {console.log(message)}
219 | console.log("handling event");
220 | const parts = message.split("/");
221 | const leaves = plugin.app.workspace.getLeavesOfType(NV_VIEW_TYPE);
222 | const name = parts[1];
223 | leaves.forEach((leaf) =>{
224 | let view = leaf.view as NeoVisView;
225 | if (parts[0] === "onSMDModifyEvent") {
226 | if (view.expandedNodes.includes(name)) {
227 | view.updateWithCypher(plugin.localNeighborhoodCypher(name));
228 | }
229 | else {
230 | view.updateWithCypher(plugin.nodeCypher(name));
231 | }
232 | }
233 | else if (parts[0] === "onSMDMovedEvent") {
234 | let new_name = parts[2];
235 | if (view.expandedNodes.includes(name)) {
236 | view.updateWithCypher(plugin.localNeighborhoodCypher(new_name));
237 | view.expandedNodes.remove(name);
238 | view.expandedNodes.push(new_name);
239 | }
240 | else {
241 | view.updateWithCypher(plugin.nodeCypher(new_name));
242 | }
243 | }
244 | else if (parts[0] === "onSMDDeletedEvent") {
245 | // TODO: Maybe automatically update to dangling link by running an update query.
246 | view.deleteNode(parts[1]);
247 | // view.updateStyle();
248 | }
249 | else if (parts[0] === "onSMDRelDeletedEvent") {
250 | parts.slice(1).forEach((id: IdType) => {
251 | view.deleteEdge(id);
252 | })
253 | }
254 | });
255 | }
256 | else if (settings.debug) {
257 | console.log(message);
258 | }
259 | });
260 |
261 | new Notice("Initializing Neo4j stream.");
262 | this.statusBar.setText('Initializing Neo4j stream');
263 | }
264 | catch(error) {
265 | console.log("Error during initialization of semantic markdown: \n", error);
266 | new Notice("Error during initialization of the Neo4j stream. Check the console for crash report.");
267 | }
268 | this.httpServer();
269 | }
270 |
271 | async httpServer() {
272 | let path = require('path');
273 | let http = require('http');
274 | let fs = require('fs');
275 |
276 | let dir = path.join(this.path);
277 |
278 | let mime = {
279 | gif: 'image/gif',
280 | jpg: 'image/jpeg',
281 | png: 'image/png',
282 | svg: 'image/svg+xml',
283 | };
284 | let settings = this.settings;
285 | this.imgServer = http.createServer(function (req: IncomingMessage, res: ServerResponse) {
286 |
287 | let reqpath = req.url.toString().split('?')[0];
288 | if (req.method !== 'GET') {
289 | res.statusCode = 501;
290 | res.setHeader('Content-Type', 'text/plain');
291 | return res.end('Method not implemented');
292 | }
293 | let file = path.join(dir, decodeURI(reqpath.replace(/\/$/, '/index.html')));
294 | if (settings.debug) {
295 | console.log("entering query");
296 | console.log(req);
297 | console.log(file);
298 | }
299 | if (file.indexOf(dir + path.sep) !== 0) {
300 | res.statusCode = 403;
301 | res.setHeader('Content-Type', 'text/plain');
302 | return res.end('Forbidden');
303 | }
304 | // @ts-ignore
305 | let type = mime[path.extname(file).slice(1)];
306 | let s = fs.createReadStream(file);
307 | s.on('open', function () {
308 | res.setHeader('Content-Type', type);
309 | s.pipe(res);
310 | });
311 | s.on('error', function () {
312 | res.setHeader('Content-Type', 'text/plain');
313 | res.statusCode = 404;
314 | res.end('Not found');
315 | });
316 | });
317 | try {
318 | let port = this.settings.imgServerPort;
319 | this.imgServer.listen(port, function () {
320 | console.log('Image server listening on http://localhost:' + port + '/');
321 | });
322 | }
323 | catch (e){
324 | console.log(e);
325 | new Notice("Neo4j: Couldn't start image server, see console");
326 | }
327 | }
328 |
329 | openLocalGraph(name: string) {
330 | if (!this.stream_process) {
331 | new Notice("Cannot open local graph as neo4j stream is not active.")
332 | return;
333 | }
334 |
335 | const leaf = this.app.workspace.splitActiveLeaf(this.settings.splitDirection);
336 | const query = this.localNeighborhoodCypher(name);
337 | const neovisView = new NeoVisView(leaf, query, this);
338 | leaf.open(neovisView);
339 | neovisView.expandedNodes.push(name);
340 | }
341 |
342 | getLinesOffsetToGoal(start: number, goal: string, step = 1, cm: Editor): number {
343 | // Code taken from https://github.com/mrjackphil/obsidian-text-expand/blob/0.6.4/main.ts
344 | const lineCount = cm.lineCount();
345 | let offset = 0;
346 |
347 | while (!isNaN(start + offset) && start + offset < lineCount && start + offset >= 0) {
348 | const result = goal === cm.getLine(start + offset);
349 | if (result) {
350 | return offset;
351 | }
352 | offset += step;
353 | }
354 |
355 | return start;
356 | }
357 |
358 | getContentBetweenLines(fromLineNum: number, startLine: string, endLine: string, cm: Editor) {
359 | // Code taken from https://github.com/mrjackphil/obsidian-text-expand/blob/0.6.4/main.ts
360 | const topOffset = this.getLinesOffsetToGoal(fromLineNum, startLine, -1, cm);
361 | const botOffset = this.getLinesOffsetToGoal(fromLineNum, endLine, 1, cm);
362 |
363 | const topLine = fromLineNum + topOffset + 1;
364 | const botLine = fromLineNum + botOffset - 1;
365 |
366 | if (!(cm.getLine(topLine - 1) === startLine && cm.getLine(botLine + 1) === endLine)) {
367 | return "";
368 | }
369 |
370 | return cm.getRange({line: topLine || fromLineNum, ch: 0},
371 | {line: botLine || fromLineNum, ch: cm.getLine(botLine)?.length });
372 | }
373 |
374 | nodeCypher(label: string): string {
375 | return "MATCH (n) WHERE n.name=\"" + label +
376 | "\" AND n." + PROP_VAULT + "=\"" + this.app.vault.getName() +
377 | "\" RETURN n"
378 | }
379 |
380 | localNeighborhoodCypher(label:string): string {
381 | return "MATCH (n {name: \"" + label +
382 | "\", " + PROP_VAULT + ":\"" + this.app.vault.getName() +
383 | "\"}) OPTIONAL MATCH (n)-[r]-(m) RETURN n,r,m"
384 | }
385 |
386 | executeQuery() {
387 | // Code taken from https://github.com/mrjackphil/obsidian-text-expand/blob/0.6.4/main.ts
388 | const currentView = this.app.workspace.activeLeaf.view;
389 |
390 | if (!(currentView instanceof MarkdownView)) {
391 | return;
392 | }
393 |
394 | const cmDoc = currentView.sourceMode.cmEditor;
395 | const curNum = cmDoc.getCursor().line;
396 | const query = this.getContentBetweenLines(curNum, '```cypher', '```', cmDoc);
397 | if (query.length > 0) {
398 | const leaf = this.app.workspace.splitActiveLeaf(this.settings.splitDirection);
399 | try {
400 | const neovisView = new NeoVisView(leaf, query, this);
401 | leaf.open(neovisView);
402 | }
403 | catch(e) {
404 | if (e instanceof Neo4jError) {
405 | new Notice("Invalid cypher query. Check console for more info.");
406 | }
407 | else {
408 | throw e;
409 | }
410 | }
411 | }
412 | }
413 |
414 | public async shutdown() {
415 | if(this.stream_process) {
416 | new Notice("Stopping Neo4j stream");
417 | this.stream_process.kill();
418 | this.statusBar.setText("Neo4j stream offline");
419 | this.stream_process = null;
420 | this.imgServer.close();
421 | this.imgServer = null;
422 | }
423 | }
424 |
425 | async onunload() {
426 | console.log('Unloading Neo4j Graph View plugin');
427 | await this.shutdown();
428 | }
429 |
430 | }
431 |
432 |
--------------------------------------------------------------------------------
/neo4j-graph-view/package.json:
--------------------------------------------------------------------------------
1 | {
2 | "name": "neovis-graph-visualization",
3 | "version": "0.9.12",
4 | "description": "An Obsidian plugin for advanced graph visualization and querying using Neovis.js.",
5 | "main": "main.js",
6 | "repository": {
7 | "type": "git",
8 | "url": "https://github.com/HEmile/semantic-markdown-converter.git"
9 | },
10 | "scripts": {
11 | "dev": "rollup --config rollup.config.js -w",
12 | "build": "rollup --config rollup.config.js"
13 | },
14 | "keywords": [],
15 | "author": "",
16 | "license": "MIT",
17 | "devDependencies": {
18 | "@egjs/hammerjs": "^2.0.17",
19 | "@rollup/plugin-commonjs": "^15.1.0",
20 | "@rollup/plugin-node-resolve": "^9.0.0",
21 | "@rollup/plugin-typescript": "^6.0.0",
22 | "@types/node": "^14.14.2",
23 | "component-emitter": "^1.3.0",
24 | "hammerjs": "^2.0.8",
25 | "keycharm": "^0.2.0",
26 | "moment": "^2.29.1",
27 | "obsidian": "https://github.com/obsidianmd/obsidian-api/tarball/master",
28 | "rollup": "^2.32.1",
29 | "timsort": "^0.3.0",
30 | "tslib": "^2.0.3",
31 | "typescript": "^4.0.3",
32 | "uuid": "^8.3.2",
33 | "vis-data": "^6.6.1",
34 | "vis-util": "^4.3.4"
35 | },
36 | "dependencies": {
37 | "@rollup/plugin-json": "^4.1.0",
38 | "child_process": "^1.0.2",
39 | "neovis.js": "^1.6.0",
40 | "open": "^7.3.0",
41 | "python-shell": "^2.0.3"
42 | }
43 | }
44 |
--------------------------------------------------------------------------------
/neo4j-graph-view/resources/bloom_screenshot.jpg:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/HEmile/obsidian-neo4j-graph-view/77afd66a1167ca2ed825ae000b463084745033d2/neo4j-graph-view/resources/bloom_screenshot.jpg
--------------------------------------------------------------------------------
/neo4j-graph-view/resources/browser_screenshot.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/HEmile/obsidian-neo4j-graph-view/77afd66a1167ca2ed825ae000b463084745033d2/neo4j-graph-view/resources/browser_screenshot.png
--------------------------------------------------------------------------------
/neo4j-graph-view/resources/cypher_querying.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/HEmile/obsidian-neo4j-graph-view/77afd66a1167ca2ed825ae000b463084745033d2/neo4j-graph-view/resources/cypher_querying.png
--------------------------------------------------------------------------------
/neo4j-graph-view/resources/graphxr.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/HEmile/obsidian-neo4j-graph-view/77afd66a1167ca2ed825ae000b463084745033d2/neo4j-graph-view/resources/graphxr.gif
--------------------------------------------------------------------------------
/neo4j-graph-view/resources/obsidian neo4j plugin.gif:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/HEmile/obsidian-neo4j-graph-view/77afd66a1167ca2ed825ae000b463084745033d2/neo4j-graph-view/resources/obsidian neo4j plugin.gif
--------------------------------------------------------------------------------
/neo4j-graph-view/resources/styled_screenshot.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/HEmile/obsidian-neo4j-graph-view/77afd66a1167ca2ed825ae000b463084745033d2/neo4j-graph-view/resources/styled_screenshot.png
--------------------------------------------------------------------------------
/neo4j-graph-view/rollup.config.js:
--------------------------------------------------------------------------------
1 | import typescript from '@rollup/plugin-typescript';
2 | import {nodeResolve} from '@rollup/plugin-node-resolve';
3 | import commonjs from '@rollup/plugin-commonjs';
4 | // import json from '@rollup/plugin-json';
5 |
6 | export default {
7 | input: 'main.ts',
8 | output: {
9 | dir: '.',
10 | sourcemap: 'inline',
11 | format: 'cjs',
12 | exports: 'default'
13 | },
14 | external: ['obsidian'],
15 | plugins: [
16 | typescript(),
17 | nodeResolve({browser: true}),
18 | commonjs(),
19 | // json(),
20 | ]
21 | };
--------------------------------------------------------------------------------
/neo4j-graph-view/settings.ts:
--------------------------------------------------------------------------------
1 | import {App, Notice, PluginSettingTab, Setting, SplitDirection} from "obsidian";
2 |
3 | import Neo4jViewPlugin from './main';
4 | import {EdgeOptions, NodeOptions} from "vis-network";
5 | import {NeoVisView, NV_VIEW_TYPE} from "./visualization";
6 |
7 | export interface INeo4jViewSettings {
8 | index_content: boolean;
9 | auto_expand: boolean;
10 | auto_add_nodes: boolean;
11 | community: string;
12 | hierarchical: boolean;
13 | convert_markdown: boolean;
14 | show_arrows: boolean;
15 | inlineContext: boolean;
16 | password: string;
17 | typed_link_prefix: string;
18 | splitDirection: SplitDirection; // 'horizontal';
19 | imgServerPort: number;
20 | debug: boolean;
21 | nodeSettings: string;
22 | edgeSettings: string;
23 | }
24 |
25 | export const DefaultNodeSettings: NodeOptions = {
26 | size: 9,
27 | font: {
28 | size: 12,
29 | strokeWidth: 1
30 | },
31 | borderWidth: 0,
32 | widthConstraint: {maximum: 200},
33 | }
34 |
35 | export const DefaultEdgeSettings: EdgeOptions = {
36 | font: {
37 | size: 12,
38 | strokeWidth: 2
39 | },
40 | width: 0.5,
41 | }
42 |
43 | export const DefaultNeo4jViewSettings: INeo4jViewSettings = {
44 | auto_add_nodes: true,
45 | auto_expand: false,
46 | hierarchical: false,
47 | index_content: false,
48 | convert_markdown: true,
49 | community: "tags",
50 | password: "",
51 | show_arrows: true,
52 | inlineContext: false,
53 | splitDirection: 'horizontal',
54 | typed_link_prefix: '-',
55 | imgServerPort: 3837,
56 | debug: false,
57 | nodeSettings: JSON.stringify({
58 | "defaultStyle": DefaultNodeSettings,
59 | "exampleTag": {
60 | size: 20,
61 | color: "yellow"
62 | },
63 | "image": {
64 | size: 40,
65 | font: {
66 | size: 0
67 | }
68 | },
69 | }),
70 | edgeSettings: JSON.stringify({
71 | "defaultStyle": DefaultEdgeSettings,
72 | })
73 | }
74 |
75 |
76 |
77 | export class Neo4jViewSettingTab extends PluginSettingTab {
78 | plugin: Neo4jViewPlugin;
79 | constructor(app: App, plugin: Neo4jViewPlugin) {
80 | super(app, plugin);
81 | this.plugin = plugin;
82 | }
83 |
84 | display(): void {
85 | let {containerEl} = this;
86 | containerEl.empty();
87 |
88 | containerEl.createEl('h3');
89 | containerEl.createEl('h3', {text: 'Neo4j Graph View'});
90 |
91 | let doc_link = document.createElement("a");
92 | doc_link.href = "https://juggl.io/Neo4j+Graph+View/Neo4j+Graph+View+Plugin";
93 | doc_link.target = '_blank';
94 | doc_link.innerHTML = 'the documentation';
95 |
96 | let discord_link = document.createElement("a");
97 | discord_link.href = "https://discord.gg/sAmSGpaPgM";
98 | discord_link.target = '_blank';
99 | discord_link.innerHTML = 'the Discord server';
100 |
101 | let juggl_link = document.createElement("a");
102 | juggl_link.href = "https://juggl.io/";
103 | juggl_link.target = '_blank';
104 | juggl_link.innerHTML = 'Juggl';
105 |
106 | let introPar = document.createElement("p");
107 | introPar.innerHTML = "WARNING: Neo4j Graph View is deprecated and will not receive any more updates. " +
108 | "It will be removed from the community plugins soon. It is replaced by " + juggl_link.outerHTML + ".
" +
109 | "Check out " + doc_link.outerHTML + " for installation help and a getting started guide.
" +
110 | "Join " + discord_link.outerHTML + " for nice discussion and additional help."
111 |
112 | containerEl.appendChild(introPar);
113 |
114 | new Setting(containerEl)
115 | .setName("Neo4j database password")
116 | .setDesc("The password of your neo4j graph database. WARNING: This is stored in plaintext in your vault. " +
117 | "Don't use sensitive passwords here!")
118 | .addText(text => {
119 | text.setPlaceholder("")
120 | .setValue(this.plugin.settings.password)
121 | .onChange((new_folder) => {
122 | this.plugin.settings.password = new_folder;
123 | this.plugin.saveData(this.plugin.settings);
124 | }).inputEl.setAttribute("type", "password")
125 | });
126 |
127 | containerEl.createEl('h3');
128 | containerEl.createEl('h3', {text: 'Appearance'});
129 |
130 | new Setting(containerEl)
131 | .setName("Color-coding")
132 | .setDesc("What property to choose for coloring the nodes in the graph. Requires a server restart.")
133 | .addDropdown(dropdown => dropdown
134 | .addOption('tags','Tags')
135 | .addOption('folders','Folders')
136 | .addOption('none','No color-coding')
137 | .setValue(this.plugin.settings.community)
138 | .onChange((value) => {
139 | this.plugin.settings.community = value;
140 | this.plugin.saveData(this.plugin.settings);
141 | }));
142 |
143 |
144 | new Setting(containerEl)
145 | .setName("Hierarchical layout")
146 | .setDesc("Use the hierarchical graph layout instead of the normal one.")
147 | .addToggle(toggle => {
148 | toggle.setValue(this.plugin.settings.hierarchical)
149 | .onChange((new_value) => {
150 | this.plugin.settings.hierarchical = new_value;
151 | this.plugin.saveData(this.plugin.settings);
152 | })
153 | });
154 |
155 | new Setting(containerEl)
156 | .setName("Show arrows")
157 | .setDesc("Show arrows on edges.")
158 | .addToggle(toggle => {
159 | toggle.setValue(this.plugin.settings.show_arrows)
160 | .onChange((new_value) => {
161 | this.plugin.settings.show_arrows = new_value;
162 | this.plugin.saveData(this.plugin.settings);
163 | })
164 | });
165 | new Setting(containerEl)
166 | .setName("Show context on inline links")
167 | .setDesc("Shows the paragraph where an inline link is in on the edge.")
168 | .addToggle(toggle => {
169 | toggle.setValue(this.plugin.settings.inlineContext)
170 | .onChange((new_value) => {
171 | this.plugin.settings.inlineContext = new_value;
172 | this.plugin.saveData(this.plugin.settings);
173 | })
174 | });
175 | containerEl.createEl('h4');
176 | containerEl.createEl('h4', {text: 'Node Styling'});
177 |
178 | const div = document.createElement("div");
179 | div.className = "neovis_setting";
180 | this.containerEl.children[this.containerEl.children.length - 1].appendChild(div);
181 | div.setAttr("style", "height: 100%; width:100%");
182 |
183 | let input = div.createEl("textarea");
184 | input.placeholder = JSON.stringify(DefaultNodeSettings);
185 | input.value = this.plugin.settings.nodeSettings;
186 | input.onchange = (ev) => {
187 | this.plugin.settings.nodeSettings = input.value;
188 | this.plugin.saveData(this.plugin.settings);
189 | let leaves = this.plugin.app.workspace.getLeavesOfType(NV_VIEW_TYPE);
190 | leaves.forEach((leaf) =>{
191 | (leaf.view as NeoVisView).updateStyle();
192 | });
193 | };
194 | input.setAttr("style", "height: 300px; width: 100%; " +
195 | "-webkit-box-sizing: border-box; -moz-box-sizing: border-box; box-sizing: border-box;");
196 |
197 | let temp_link = document.createElement("a");
198 | temp_link.href = "https://publish.obsidian.md/semantic-obsidian/Node+styling";
199 | temp_link.target = '_blank';
200 | temp_link.innerHTML ='this link';
201 |
202 | let par = document.createElement("p");
203 | par.innerHTML = "Styling of nodes in .json format.
" +
204 | "Use {\"defaultStyle\": {}} for the default styling of nodes. " +
205 | "Use {\"image\": {}} to style images. Use {\"SMD_dangling\": {}} to style dangling notes.
" +
206 | "When color-coding is set to Folders, use the path to the folder for this key. " +
207 | "Use {\"/\" for the root folder.
" +
208 | "See " + temp_link.outerHTML + " for help with styling nodes. "
209 |
210 | containerEl.appendChild(par);
211 |
212 | containerEl.createEl('h4');
213 | containerEl.createEl('h4', {text: 'Edge Styling'});
214 |
215 | const div2 = document.createElement("div");
216 | div2.className = "neovis_setting2";
217 | this.containerEl.children[this.containerEl.children.length - 1].appendChild(div2);
218 | div2.setAttr("style", "height: 100%; width:100%");
219 |
220 | let input2 = div2.createEl("textarea");
221 | input2.placeholder = JSON.stringify(DefaultEdgeSettings);
222 | input2.value = this.plugin.settings.edgeSettings;
223 | input2.onchange = (ev) => {
224 | this.plugin.settings.edgeSettings = input2.value;
225 | this.plugin.saveData(this.plugin.settings);
226 | let leaves = this.plugin.app.workspace.getLeavesOfType(NV_VIEW_TYPE);
227 | leaves.forEach((leaf) =>{
228 | (leaf.view as NeoVisView).updateStyle();
229 | });
230 | };
231 | input2.setAttr("style", "height: 300px; width: 100%; " +
232 | "-webkit-box-sizing: border-box; -moz-box-sizing: border-box; box-sizing: border-box;");
233 |
234 | let temp_link2 = document.createElement("a");
235 | temp_link2.href = "https://publish.obsidian.md/semantic-obsidian/Edge+styling";
236 | temp_link2.target = '_blank';
237 | temp_link2.innerHTML = 'this link';
238 |
239 | let par2 = document.createElement("p");
240 | par2.innerHTML = "Styling of edges is done in .json format.
" +
241 | "The first key determines what types of links to apply this style to. " +
242 | "Use {\"defaultStyle\": {}} for the default styling of edges, and {\"inline\":{} } for the styling of untyped links. " +
243 | "See " + temp_link2.outerHTML + " for help with styling edges."
244 |
245 | containerEl.appendChild(par2);
246 |
247 |
248 | containerEl.createEl('h3');
249 | containerEl.createEl('h3', {text: 'Advanced'});
250 |
251 | new Setting(containerEl)
252 | .setName("Automatic expand")
253 | .setDesc("This will automatically expand the neighbourhood around any nodes clicked on or added to the graph. " +
254 | "This normally only happens when pressing E or when double-clicking.")
255 | .addToggle(toggle => {
256 | toggle.setValue(this.plugin.settings.auto_expand)
257 | .onChange((new_value) => {
258 | this.plugin.settings.auto_expand = new_value;
259 | this.plugin.saveData(this.plugin.settings);
260 | })
261 | });
262 | new Setting(containerEl)
263 | .setName("Automatically add nodes")
264 | .setDesc("This will automatically add nodes to the graph whenever a note is opened.")
265 | .addToggle(toggle => {
266 | toggle.setValue(this.plugin.settings.auto_add_nodes)
267 | .onChange((new_value) => {
268 | this.plugin.settings.auto_add_nodes = new_value;
269 | this.plugin.saveData(this.plugin.settings);
270 | })
271 | });
272 |
273 | new Setting(containerEl)
274 | .setName("Convert Markdown")
275 | .setDesc("If true, the server will convert the content of notes to HTML. This can slow the server. " +
276 | "Turn it off to increase server performance at the cost of not having proper previews on hovering in the graph. ")
277 | .addToggle(toggle => {
278 | toggle.setValue(this.plugin.settings.convert_markdown)
279 | .onChange((new_value) => {
280 | this.plugin.settings.convert_markdown = new_value;
281 | this.plugin.saveData(this.plugin.settings);
282 | })
283 | });
284 |
285 | new Setting(containerEl)
286 | .setName("Index note content")
287 | .setDesc("This will full-text index the content of notes. " +
288 | "This allows searching within notes using the Neo4j Bloom search bar. However, it could decrease performance.")
289 | .addToggle(toggle => {
290 | toggle.setValue(this.plugin.settings.index_content)
291 | .onChange((new_value) => {
292 | this.plugin.settings.index_content = new_value;
293 | this.plugin.saveData(this.plugin.settings);
294 | })
295 | });
296 |
297 | new Setting(containerEl)
298 | .setName("Typed links prefix")
299 | .setDesc("Prefix to use for typed links. Default is '-'. Requires a server restart.")
300 | .addText(text => {
301 | text.setPlaceholder("")
302 | .setValue(this.plugin.settings.typed_link_prefix)
303 | .onChange((new_folder) => {
304 | this.plugin.settings.typed_link_prefix = new_folder;
305 | this.plugin.saveData(this.plugin.settings);
306 | })
307 | });
308 |
309 | new Setting(containerEl)
310 | .setName("Image server port")
311 | .setDesc("Set the port of the image server. If you use multiple vaults, these need to be set differently. Default 3000.")
312 | .addText(text => {
313 | text.setValue(this.plugin.settings.imgServerPort + '')
314 | .setPlaceholder('3000')
315 | .onChange((new_value) => {
316 | this.plugin.settings.imgServerPort = parseInt(new_value.trim());
317 | this.plugin.saveData(this.plugin.settings);
318 | })
319 | });
320 |
321 | new Setting(containerEl)
322 | .setName("Debug")
323 | .setDesc("Enable debug mode. Prints a lot of stuff in the developers console. Requires a server restart.")
324 | .addToggle(toggle => {
325 | toggle.setValue(this.plugin.settings.debug)
326 | .onChange((new_value) => {
327 | this.plugin.settings.debug = new_value;
328 | this.plugin.saveData(this.plugin.settings);
329 | })
330 | });
331 |
332 |
333 | }
334 | }
--------------------------------------------------------------------------------
/neo4j-graph-view/styles.css:
--------------------------------------------------------------------------------
1 | div.vis-tooltip {
2 | white-space: pre-line;
3 | font-family: inherit;
4 | font-size: inherit;
5 | width: fit-content;
6 | max-width: 500px;
7 | border: 0;
8 | padding: 15px;
9 | background-color: white;
10 | }
11 |
12 | div.neovis_setting {
13 | width: content-box;
14 | }
15 |
--------------------------------------------------------------------------------
/neo4j-graph-view/tsconfig.json:
--------------------------------------------------------------------------------
1 | {
2 | "compilerOptions": {
3 | "baseUrl": ".",
4 | "inlineSourceMap": true,
5 | "inlineSources": true,
6 | "module": "ESNext",
7 | "target": "es5",
8 | "allowJs": true,
9 | "noImplicitAny": true,
10 | "moduleResolution": "node",
11 | "importHelpers": true,
12 | "lib": [
13 | "dom",
14 | "es5",
15 | "scripthost",
16 | "es2015"
17 | ]
18 | },
19 | "include": [
20 | "**/*.ts"
21 | ]
22 | }
23 |
--------------------------------------------------------------------------------
/neo4j-graph-view/versions.json:
--------------------------------------------------------------------------------
1 | {
2 | "1.0.1": "0.9.12",
3 | "1.0.0": "0.9.7"
4 | }
5 |
--------------------------------------------------------------------------------
/neo4j-graph-view/visualization.ts:
--------------------------------------------------------------------------------
1 | import {IEdge, INode, IRelationshipConfig, NEOVIS_DEFAULT_CONFIG} from "neovis.js";
2 | import NeoVis from 'neovis.js';
3 | import {INeo4jViewSettings} from "./settings";
4 | import {EventRef, ItemView, MarkdownView, Menu, normalizePath, TFile, Vault, Workspace, WorkspaceLeaf} from "obsidian";
5 | import Neo4jViewPlugin from "./main";
6 | import {Relationship, Node} from "neo4j-driver";
7 | import {Data, IdType, Network, NodeOptions} from "vis-network";
8 |
9 | export const NV_VIEW_TYPE = "neovis";
10 | export const MD_VIEW_TYPE = 'markdown';
11 |
12 | export const PROP_VAULT = "SMD_vault"
13 | export const PROP_PATH = "SMD_path"
14 | export const PROP_COMMUNITY = "SMD_community"
15 |
16 | let VIEW_COUNTER = 0;
17 |
18 | export class NeoVisView extends ItemView{
19 |
20 | workspace: Workspace;
21 | listeners: EventRef[];
22 | settings: INeo4jViewSettings;
23 | initial_query: string;
24 | vault: Vault;
25 | plugin: Neo4jViewPlugin;
26 | viz: NeoVis;
27 | network: Network;
28 | hasClickListener = false;
29 | rebuildRelations = true;
30 | selectName: string = undefined;
31 | expandedNodes: string[] = [];
32 | nodes: Record;
33 | edges: Record;
34 |
35 | constructor(leaf: WorkspaceLeaf, initial_query: string, plugin: Neo4jViewPlugin) {
36 | super(leaf);
37 | this.settings = plugin.settings;
38 | this.workspace = this.app.workspace;
39 | this.initial_query = initial_query;
40 | this.vault = this.app.vault;
41 | this.plugin = plugin;
42 | }
43 |
44 | async onOpen() {
45 | const div = document.createElement("div");
46 | div.id = "neovis_id" + VIEW_COUNTER;
47 | VIEW_COUNTER += 1;
48 | this.containerEl.children[1].appendChild(div);
49 | div.setAttr("style", "height: 100%; width:100%");
50 | // console.log(this.containerEl);
51 | const config = {
52 | container_id: div.id,
53 | server_url: "bolt://localhost:7687",
54 | server_user: "neo4j",
55 | server_password: this.settings.password,
56 | arrows: this.settings.show_arrows,
57 | hierarchical: this.settings.hierarchical,
58 | labels: {
59 | [NEOVIS_DEFAULT_CONFIG]: {
60 | "caption": "name",
61 | //"size": this.settings.node_size,
62 | "community": PROP_COMMUNITY,
63 | "title_properties": [
64 | "aliases",
65 | "content"
66 | ],
67 | }
68 | },
69 | relationships: {
70 | "inline": {
71 | "thickness": "weight",
72 | "caption": this.settings.inlineContext ? "context": false,
73 | "title_properties": [
74 | "parsedContext"
75 | ]
76 | },
77 | [NEOVIS_DEFAULT_CONFIG]: {
78 | "thickness": "defaultThicknessProperty",
79 | "caption": true
80 | }
81 | },
82 | initial_cypher: this.initial_query
83 | };
84 | this.viz = new NeoVis(config);
85 | this.viz.registerOnEvent("completed", (e)=>{
86 | if (!this.hasClickListener) {
87 | // @ts-ignore
88 | this.network = this.viz["_network"] as Network;
89 | // @ts-ignore
90 | this.nodes = this.viz._nodes;
91 | // @ts-ignore
92 | this.edges = this.viz._edges;
93 | // Register on click event
94 | this.network.on("click", (event) => {
95 | if (event.nodes.length > 0) {
96 | this.onClickNode(this.findNode(event.nodes[0]));
97 | }
98 | else if (event.edges.length == 1) {
99 | this.onClickEdge(this.findEdge(event.edges[0]));
100 | }
101 | });
102 | this.network.on("doubleClick", (event) => {
103 | if (event.nodes.length > 0) {
104 | this.onDoubleClickNode(this.findNodeRaw(event.nodes[0]));
105 | }
106 | });
107 | this.network.on("oncontext", (event) => {
108 | // Thanks Liam for sharing how to do context menus
109 | const fileMenu = new Menu(this.plugin.app); // Creates empty file menu
110 | let nodeId = this.network.getNodeAt(event.pointer.DOM);
111 |
112 | if (!(nodeId === undefined)) {
113 | let node = this.findNode(nodeId);
114 | let file = this.getFileFromNode(node);
115 | if (!(file === undefined)) {
116 | // hook for plugins to populate menu with "file-aware" menu items
117 | this.app.workspace.trigger("file-menu", fileMenu, file, "my-context-menu", null);
118 | }
119 | }
120 | fileMenu.addItem((item) =>{
121 | item.setTitle("Expand selection (E)").setIcon("dot-network")
122 | .onClick(evt => {
123 | this.expandSelection();
124 | });
125 | });
126 | fileMenu.addItem((item) =>{
127 | item.setTitle("Hide selection (H)").setIcon("dot-network")
128 | .onClick(evt => {
129 | this.hideSelection();
130 | });
131 | });
132 | fileMenu.addItem((item) =>{
133 | item.setTitle("Invert selection (I)").setIcon("dot-network")
134 | .onClick(evt => {
135 | this.invertSelection();
136 | });
137 | });
138 | fileMenu.addItem((item) =>{
139 | item.setTitle("Select all (A)").setIcon("dot-network")
140 | .onClick(evt => {
141 | this.hideSelection();
142 | });
143 | });
144 | let domRect = this.containerEl.getBoundingClientRect();
145 | // console.log("DOM", event.pointer.DOM);
146 | // console.log("Canvas", event.pointer.canvas);
147 | // console.log("offset", domRect.left, domRect.top)
148 | // console.log("DOM offset", { x: event.pointer.DOM.x + domRect.left, y: event.pointer.DOM.y + domRect.top });
149 | // console.log("Canvas offset", { x: event.pointer.canvas.x + domRect.left, y: event.pointer.canvas.y + domRect.top });
150 | // Actually open the menu
151 | fileMenu.showAtPosition({ x: event.pointer.DOM.x + domRect.left, y: event.pointer.DOM.y + domRect.top });
152 | })
153 | this.hasClickListener = true;
154 | }
155 | if (this.rebuildRelations) {
156 | let inQuery = this.getInQuery(this.viz.nodes.getIds());
157 | let query = "MATCH (n)-[r]-(m) WHERE n." + PROP_VAULT + "= \"" + this.vault.getName() + "\" AND n.name " + inQuery
158 | + " AND m." + PROP_VAULT + "= \"" + this.vault.getName() + "\" AND m.name " + inQuery +
159 | " RETURN r";
160 | this.viz.updateWithCypher(query);
161 | this.rebuildRelations = false;
162 | }
163 | this.updateStyle();
164 | if (!(this.selectName=== undefined)) {
165 | this.viz.nodes.forEach(node => {
166 | if (node.label === this.selectName) {
167 | this.network.setSelection({nodes: [node.id], edges: []});
168 | this.selectName = undefined;
169 | }
170 | })
171 | }
172 | if (this.settings.debug) {
173 | // @ts-ignore
174 | console.log(this.nodes);
175 | // @ts-ignore
176 | console.log(this.edges);
177 | }
178 | });
179 | this.load();
180 | this.viz.render();
181 |
182 | // Register on file open event
183 | this.workspace.on("file-open", (file) => {
184 | if (file && this.settings.auto_add_nodes) {
185 | const name = file.basename;
186 | //todo: Select node
187 | if (this.settings.auto_expand) {
188 | this.updateWithCypher(this.plugin.localNeighborhoodCypher(name));
189 | }
190 | else {
191 | this.updateWithCypher(this.plugin.nodeCypher(name));
192 | }
193 | this.selectName = name;
194 | }
195 | });
196 |
197 | // Register keypress event
198 | this.containerEl.addEventListener("keydown", (evt) => {
199 | if (evt.key === "e"){
200 | this.expandSelection();
201 | }
202 | else if (evt.key === "h" || evt.key === "Backspace"){
203 | this.hideSelection();
204 | }
205 | else if (evt.key === "i") {
206 | this.invertSelection();
207 | }
208 | else if (evt.key === "a") {
209 | this.selectAll();
210 | }
211 | });
212 | }
213 |
214 | findNodeRaw(id: IdType): Node {
215 | // @ts-ignore
216 | return this.viz.nodes.get(id)?.raw as Node;
217 | }
218 |
219 | findNode(id: IdType): INode {
220 | return this.viz.nodes.get(id) as INode;
221 | }
222 |
223 | findEdge(id: IdType): Relationship {
224 | // @ts-ignore
225 | return this.viz.edges.get(id)?.raw as Relationship;
226 | }
227 |
228 | updateWithCypher(cypher: string) {
229 | if (this.settings.debug) {
230 | console.log(cypher);
231 | }
232 | this.viz.updateWithCypher(cypher);
233 | this.rebuildRelations = true;
234 | }
235 |
236 | getFileFromNode(node: INode) {
237 | // @ts-ignore
238 | let label = node.raw.properties["name"];
239 | return this.app.metadataCache.getFirstLinkpathDest(label, '');
240 | }
241 |
242 | updateStyle() {
243 | let nodeOptions = JSON.parse(this.settings.nodeSettings);
244 | this.viz.nodes.forEach((node) => {
245 | let nodeId = this.network.findNode(node.id);
246 |
247 | let specificOptions: NodeOptions[] = [];
248 | let file = this.getFileFromNode(node);
249 | if (this.settings.community === "tags") {
250 | node.raw.labels.forEach((label) => {
251 | if (label in nodeOptions) {
252 | specificOptions.push(nodeOptions[label]);
253 | }
254 | });
255 | }
256 | else if (this.settings.community === "folders" && !(file === undefined)) {
257 | // @ts-ignore
258 | const path = file.parent.path;
259 | if (path in nodeOptions) {
260 | specificOptions.push(nodeOptions[path]);
261 | }
262 | }
263 | // Style images
264 | if (/(\.png|\.jpg|\.jpeg|\.gif|\.svg)$/.test(node.label) && !(file === undefined)) {
265 | specificOptions.push({shape: "image", image: "http://localhost:" +
266 | this.settings.imgServerPort + "/"
267 | + encodeURI(file.path)});
268 | if ("image" in nodeOptions) {
269 | specificOptions.push(nodeOptions["image"]);
270 | }
271 | }
272 | // @ts-ignore
273 | let node_sth = this.network.body.nodes[nodeId];
274 | if (!(node_sth === undefined)) {
275 | node_sth.setOptions(Object.assign({}, nodeOptions["defaultStyle"], ...specificOptions));
276 | } else if(this.settings.debug) {
277 | console.log(node);
278 | }
279 | });
280 | let edgeOptions = JSON.parse(this.settings.edgeSettings);
281 | this.viz.edges.forEach((edge) => {
282 | // @ts-ignore
283 | let edge_sth = this.network.body.edges[edge.id];
284 | let type = edge.raw.type;
285 | let specificOptions = type in edgeOptions ? [edgeOptions[type]] : [];
286 | if (!(edge_sth === undefined)) {
287 | edge_sth.setOptions(Object.assign({}, edgeOptions["defaultStyle"], ...specificOptions));
288 | } else if (this.settings.debug) {
289 | console.log(edge);
290 | }
291 | });
292 | }
293 |
294 | async onClickNode(node: INode) {
295 | const file = this.getFileFromNode(node);
296 | // @ts-ignore
297 | let label = node.raw.properties["name"];
298 | if (file) {
299 | await this.plugin.openFile(file);
300 | }
301 | else {
302 | // Create dangling file
303 | // TODO: Add default folder
304 | // @ts-ignore
305 | const filename = label + ".md";
306 | const createdFile = await this.vault.create(filename, '');
307 | await this.plugin.openFile(createdFile);
308 | }
309 | if (this.settings.auto_expand) {
310 | await this.updateWithCypher(this.plugin.localNeighborhoodCypher(label));
311 | }
312 | }
313 |
314 | async onDoubleClickNode(node: Node) {
315 | // @ts-ignore
316 | const label = node.properties["name"];
317 | this.expandedNodes.push(label);
318 | await this.updateWithCypher(this.plugin.localNeighborhoodCypher(label));
319 | }
320 |
321 | async onClickEdge(edge: Object) {
322 | // @ts-ignore
323 | // if (!edge.raw) {
324 | // return;
325 | // }
326 | // // @ts-ignore
327 | // const rel = edge.raw as Relationship;
328 | // console.log(edge);
329 | // // @ts-ignore
330 | // const file = rel.properties["context"];
331 | // const node = this.viz.nodes.get(rel.start.high);
332 | // const label = node.label;
333 |
334 | // TODO: Figure out how to open a node at the context point
335 | // this.workspace.openLinkText()
336 |
337 | }
338 |
339 | getInQuery(nodes: IdType[]): string {
340 | let query = "IN ["
341 | let first = true;
342 | for (let id of nodes) {
343 | // @ts-ignore
344 | const title = this.findNodeRaw(id).properties["name"] as string;
345 | if (!first) {
346 | query += ", ";
347 | }
348 | query += "\"" + title + "\"";
349 | first = false;
350 | }
351 | query += "]"
352 | return query;
353 | }
354 |
355 | async expandSelection() {
356 | let selected_nodes = this.network.getSelectedNodes();
357 | if (selected_nodes.length === 0) {
358 | return;
359 | }
360 | let query = "MATCH (n)-[r]-(m) WHERE n." + PROP_VAULT + "= \"" + this.vault.getName() + "\" AND n.name ";
361 | query += this.getInQuery(selected_nodes);
362 | query += " RETURN r,m";
363 | let expandedNodes = this.expandedNodes;
364 | selected_nodes.forEach(id => {
365 | // @ts-ignore
366 | const title = this.findNodeRaw(id).properties["name"] as string;
367 | if (!expandedNodes.includes(title)) {
368 | expandedNodes.push(title);
369 | }
370 | });
371 | this.updateWithCypher(query);
372 | }
373 |
374 | deleteNode(id: IdType) {
375 | // console.log(this.viz.nodes);
376 | // @ts-ignore
377 |
378 | let node = this.findNode(id) || this.nodes[id];
379 | if (node === undefined) {
380 | return;
381 | }
382 | // @ts-ignore
383 | const title = node.raw.properties["name"] as string;
384 | if (this.expandedNodes.includes(title)) {
385 | this.expandedNodes.remove(title);
386 | }
387 | let expandedNodes = this.expandedNodes;
388 | this.network.getConnectedNodes(id).forEach((value: any) => {
389 | this.findNodeRaw(value);
390 | // @ts-ignore
391 | const n_title = this.findNodeRaw(value).properties["name"] as string;
392 | if (expandedNodes.includes(n_title)) {
393 | expandedNodes.remove(n_title);
394 | }
395 | });
396 |
397 | let edges_to_remove: IEdge[] = [];
398 | this.viz.edges.forEach((edge) => {
399 | if (edge.from === id || edge.to === id) {
400 | edges_to_remove.push(edge);
401 | }
402 | });
403 | edges_to_remove.forEach(edge => {
404 | this.viz.edges.remove(edge);
405 | });
406 |
407 | this.viz.nodes.remove(id);
408 |
409 | let keys_to_remove = [];
410 | for (let key in this.edges) {
411 | let edge = this.edges[key];
412 | if (edge.to === id || edge.from === id) {
413 | keys_to_remove.push(key);
414 | }
415 | }
416 | keys_to_remove.forEach((key) => {
417 | // @ts-ignore
418 | delete this.edges[key];
419 | });
420 |
421 | delete this.nodes[id as number];
422 | }
423 |
424 | deleteEdge(id: IdType) {
425 | let edge = this.edges[id];
426 | if (edge === undefined) {
427 | return;
428 | }
429 |
430 | let nodes = [edge.from, edge.to];
431 |
432 | this.viz.edges.remove(edge);
433 |
434 | delete this.edges[id];
435 |
436 | // TODO: Check if the node deletion is using the right rule
437 | // Current rule: The connected nodes are not expanded, and also have no other edges.
438 | nodes.forEach(node_id => {
439 | let node = this.findNodeRaw(node_id);
440 | // @ts-ignore
441 | if (!this.expandedNodes.contains(node.properties["name"])
442 | && this.network.getConnectedEdges(node_id).length === 0) {
443 | this.deleteNode(node_id);
444 | }
445 | });
446 | }
447 |
448 | async hideSelection() {
449 | if (this.network.getSelectedNodes().length === 0) {
450 | return;
451 | }
452 | // Update expanded nodes. Make sure to not automatically expand nodes of which a neighbor was hidden.
453 | // Otherwise, one would have to keep hiding nodes.
454 | this.network.getSelectedNodes().forEach(id => {
455 | this.deleteNode(id);
456 | });
457 | // this.network.deleteSelected();
458 |
459 | // This super hacky code is used because neovis.js doesn't like me removing nodes from the graph.
460 | // Essentially, whenever it'd execute a new query, it'd re-add all hidden nodes!
461 | // This resets the state of NeoVis so that it only acts as an interface with neo4j instead of also keeping
462 | // track of the data.
463 | // @ts-ignore
464 | // let data = {nodes: this.viz.nodes, edges: this.viz.edges} as Data;
465 | // this.viz.clearNetwork();
466 | // this.network.setData(data);
467 | this.updateStyle();
468 | }
469 |
470 | invertSelection() {
471 | let selectedNodes = this.network.getSelectedNodes();
472 | let network = this.network;
473 | let inversion = this.viz.nodes.get({filter: function(item){
474 | return !selectedNodes.contains(item.id) && network.findNode(item.id).length > 0;
475 | }}).map((item) => item.id);
476 | this.network.setSelection({nodes: inversion, edges: []})
477 | }
478 |
479 |
480 | selectAll() {
481 | this.network.unselectAll();
482 | this.invertSelection();
483 | }
484 |
485 | async checkAndUpdate() {
486 | try {
487 | if(await this.checkActiveLeaf()) {
488 | await this.update();
489 | }
490 | } catch (error) {
491 | console.error(error)
492 | }
493 | }
494 |
495 | async update(){
496 | this.load();
497 | }
498 |
499 | async checkActiveLeaf() {
500 | return false;
501 | }
502 |
503 | getDisplayText(): string {
504 | return "Neo4j Graph";
505 | }
506 |
507 | getViewType(): string {
508 | return NV_VIEW_TYPE;
509 | }
510 |
511 |
512 | }
--------------------------------------------------------------------------------
/package-lock.json:
--------------------------------------------------------------------------------
1 | {
2 | "requires": true,
3 | "lockfileVersion": 1,
4 | "dependencies": {
5 | "is-docker": {
6 | "version": "2.1.1",
7 | "resolved": "https://registry.npmjs.org/is-docker/-/is-docker-2.1.1.tgz",
8 | "integrity": "sha512-ZOoqiXfEwtGknTiuDEy8pN2CfE3TxMHprvNer1mXiqwkOT77Rw3YVrUQ52EqAOU3QAWDQ+bQdx7HJzrv7LS2Hw=="
9 | },
10 | "is-wsl": {
11 | "version": "2.2.0",
12 | "resolved": "https://registry.npmjs.org/is-wsl/-/is-wsl-2.2.0.tgz",
13 | "integrity": "sha512-fKzAra0rGJUUBwGBgNkHZuToZcn+TtXHpeCgmkMJMMYx1sQDYaCSyjJBSCa2nH1DGm7s3n1oBnohoVTBaN7Lww==",
14 | "requires": {
15 | "is-docker": "^2.0.0"
16 | }
17 | },
18 | "open": {
19 | "version": "7.3.0",
20 | "resolved": "https://registry.npmjs.org/open/-/open-7.3.0.tgz",
21 | "integrity": "sha512-mgLwQIx2F/ye9SmbrUkurZCnkoXyXyu9EbHtJZrICjVAJfyMArdHp3KkixGdZx1ZHFPNIwl0DDM1dFFqXbTLZw==",
22 | "requires": {
23 | "is-docker": "^2.0.0",
24 | "is-wsl": "^2.1.1"
25 | }
26 | }
27 | }
28 | }
29 |
--------------------------------------------------------------------------------
/setup.py:
--------------------------------------------------------------------------------
1 | from setuptools import setup, find_packages
2 |
3 | with open("README.md", "r", encoding="utf-8") as fh:
4 | long_description = fh.read()
5 |
6 |
7 | setup(name='semantic-markdown-converter',
8 | version='0.5.9',
9 | description='Converts different typed link formats in Markdown into each other and to external formats. Supports Obsidian Neo4j plugin.',
10 | long_description=long_description,
11 | long_description_content_type="text/markdown",
12 | url='https://github.com/HEmile/semantic-markdown-converter',
13 | packages=find_packages(),
14 | install_requires=['pyyaml', 'tqdm', 'py2neo', 'watchdog>=1.0.2',
15 | 'markdown', 'mdx-wikilink-plus'],
16 | entry_points={
17 | 'console_scripts': ['smdc=smdc.convert:main', 'smds=smdc.stream:main']
18 | },
19 | classifiers=[
20 | "Programming Language :: Python :: 3",
21 | "License :: OSI Approved :: MIT License",
22 | "Operating System :: OS Independent",
23 | ],
24 | author='Emile van Krieken',
25 | author_email='emilevankrieken@live.nl',
26 | license='MIT',
27 | zip_safe=False,
28 | python_requires='>=3.6',)
--------------------------------------------------------------------------------
/smdc/__init__.py:
--------------------------------------------------------------------------------
1 | DEBUG = False
2 | from .note import Note, Relationship
3 | from .args import convert_args, server_args
4 | from smdc.format import FORMAT_TYPES
5 | from .convert import convert
6 | from .parse import parse_folder, parse_note, note_name, obsidian_url
7 | from .stream import stream, main
8 |
9 |
--------------------------------------------------------------------------------
/smdc/args.py:
--------------------------------------------------------------------------------
1 | import argparse
2 | from pathlib import Path
3 |
4 | import smdc
5 |
6 | def _mutual_args(parser):
7 | parser.add_argument('--input', metavar='i', type=str, default=".",
8 | help='directory with markdown files')
9 | parser.add_argument('--input_format', metavar='f', type=str, default="typed_list",
10 | help='format of inputs')
11 | parser.add_argument('--extension', metavar='e', type=str, default=".md",
12 | help='extension of markdown files')
13 | parser.add_argument('--retaindb', action="store_true", default=False)
14 | parser.add_argument('--password', metavar='p', type=str, default="")
15 | parser.add_argument("--vault_name", metavar='n', type=str, default=None, help="Defaults to input directory name")
16 | parser.add_argument("--batch_size", metavar='b', type=int, default=75, help="Batch size for sending to neo4j")
17 | parser.add_argument("--debug", action="store_true", help="Debug mode")
18 | parser.add_argument("--index_content", action="store_true", help="Use to index the content in neo4j. Can highly impact performance")
19 | parser.add_argument("--typed_links_prefix", metavar='t', type=str, default="-")
20 | parser.add_argument('--community', metavar='c', type=str, default="tags", help="Options: {tags, folders, none}. Used for color coding in Obsidian plugin.")
21 | parser.add_argument('--convert_markdown', action="store_true", help="Converts content property to HTML. Takes quite a bit longer on startup. ")
22 | parser.add_argument('-r', action="store_false", default=True)
23 |
24 | def server_args():
25 | parser = argparse.ArgumentParser(
26 | description='Stream changes in Obsidian vaults to active neo4j server.')
27 | _mutual_args(parser)
28 | args = parser.parse_args()
29 | if not args.vault_name:
30 | args.vault_name = Path(args.input).name
31 | if args.debug:
32 | smdc.DEBUG = True
33 | print(args)
34 | return args
35 |
36 | def convert_args():
37 | parser = argparse.ArgumentParser(
38 | description='Convert different typed links representations of Markdown into other formats.')
39 | _mutual_args(parser)
40 | parser.add_argument('--output_format', metavar='r', type=str, default="neo4j",
41 | help='format of inputs')
42 | parser.add_argument('--output', metavar='o', type=str, default="out",
43 | help='directory for output files')
44 | args = parser.parse_args()
45 | if not args.vault_name:
46 | args.vault_name = Path(args.input).name
47 | if args.debug:
48 | smdc.DEBUG = True
49 | print(args)
50 | return args
51 |
--------------------------------------------------------------------------------
/smdc/convert.py:
--------------------------------------------------------------------------------
1 | import smdc.parse as parse
2 | from smdc import convert_args
3 | from smdc.format import FORMAT_TYPES
4 |
5 |
6 | def convert(args):
7 | notes = parse.parse_folder(FORMAT_TYPES[args.input_format], args)
8 | return FORMAT_TYPES[args.output_format].write(notes, args)
9 |
10 |
11 | def main():
12 | args = convert_args()
13 | convert(args)
14 |
15 |
16 | if __name__ == "__main__":
17 | main()
18 |
--------------------------------------------------------------------------------
/smdc/format/__init__.py:
--------------------------------------------------------------------------------
1 | from .format import Format
2 | from .typed_list import TypedList
3 | from .cypher import Cypher
4 | from .neo4j import Neo4j
5 | FORMAT_TYPES = {"typed_list": TypedList(), "cypher": Cypher(), "neo4j": Neo4j()}
--------------------------------------------------------------------------------
/smdc/format/csv.py:
--------------------------------------------------------------------------------
1 | from smdc.format import Format
2 | from smdc.note import Note, Relationship
3 | import io
4 | import os
5 | from smdc.format.util import escape_quotes
6 |
7 |
8 | class CSV(Format):
9 |
10 | def parse(self, file: io.TextIOWrapper, name, parsed_notes: [Note]) -> Note:
11 | raise NotImplementedError
12 |
13 | def write(self, file, parsed_notes: [Note]):
14 | # First create all nodes in the graph before doing the relationships, so they all exist.
15 | with open(file + '.csv', 'w') as f:
16 |
17 | for name, note in parsed_notes.items():
18 |
19 | line = "CREATE ("
20 | if note.tags:
21 | line += ":" + ":".join(note.tags)
22 | line += " { name: '" + escape_quotes(name) + "', content: '" + escape_quotes(note.content) + "'"
23 | for property, value in note.properties.items():
24 | line += ", " + property + ": '" + escape_quotes(str(value)) + "'"
25 | line += "});" + os.linesep
26 | f.write(line)
27 |
28 | for name, note in parsed_notes.items():
29 | f.write("MATCH (a)" + os.linesep + "WHERE a.name == '" + escape_quotes(name) + "'" + os.linesep)
30 | i = 0
31 | for trgt, rels in note.out_rels.items():
32 | f.write(
33 | "MATCH (b" + str(i) + ")" + os.linesep + "WHERE b" + str(i) + ".name == '" +
34 | escape_quotes(trgt) + "'" + os.linesep)
35 | for rel in rels:
36 | f.write("CREATE (a)-[r:" + rel.type + "]->(b" + str(i) + ")" + os.linesep)
37 | i += 1
38 | f.write(";" + os.linesep)
39 |
40 |
--------------------------------------------------------------------------------
/smdc/format/cypher.py:
--------------------------------------------------------------------------------
1 | from smdc.format import Format
2 | from smdc.note import Note, Relationship
3 | import io
4 | import os
5 | from smdc.format.util import escape_quotes
6 |
7 |
8 | def escape_cypher(string):
9 | # r = escape_quotes(string)
10 | # Note: CYPHER doesn't allow putting semicolons in text, for some reason. This is lossy!
11 | # r = r.replace(";", ",")
12 | r = string.replace("\\u", "\\\\u")
13 | if r and r[-1] == '\\':
14 | r += ' '
15 | return r
16 |
17 | def to_cyper(parsed_notes: [Note]):
18 | lines = []
19 | # First create all nodes in the graph before doing the relationships, so they all exist.
20 | for name, note in parsed_notes.items():
21 | line = "CREATE ("
22 | if note.tags:
23 | line += ":" + ":".join(note.tags) + " {"
24 | properties = []
25 | for property, value in note.properties.items():
26 | properties.append(property + ": '" + escape_cypher(str(value)) + "'")
27 | line += ", ".join(properties)
28 | line += "});"
29 | lines.append(line)
30 |
31 | notes_in_cypher = set(parsed_notes.keys())
32 |
33 | for name, note in parsed_notes.items():
34 | if not note.out_rels.keys():
35 | continue
36 | match_a = "MATCH (a)" + os.linesep + "WHERE a.name = '" + escape_cypher(name) + "'" + os.linesep
37 | for trgt, rels in note.out_rels.items():
38 | if trgt not in notes_in_cypher:
39 | lines.append("CREATE ({name:'" + escape_cypher(trgt) + "'});")
40 | notes_in_cypher.add(trgt)
41 | match_b = "MATCH (b)" + os.linesep + "WHERE b.name = '" + \
42 | escape_cypher(trgt) + "'" + os.linesep
43 | for rel in rels:
44 | line = "CREATE (a)-[:" + rel.type + " {"
45 | properties = []
46 | for property, value in rel.properties.items():
47 | properties.append(property + ": '" + escape_cypher(str(value)) + "'")
48 | line += ", ".join(properties)
49 | line += "}]->(b);"
50 | lines.append(match_a + match_b + line)
51 | return lines
52 |
53 | class Cypher(Format):
54 |
55 | def parse(self, file: io.TextIOWrapper, name, parsed_notes: [Note]) -> Note:
56 | raise NotImplementedError
57 |
58 | def write(self, parsed_notes: [Note], args):
59 | lines = to_cyper(parsed_notes)
60 | with open(args.output + '.cypher', 'w', encoding='utf-8') as f:
61 | f.writelines(os.linesep.join(lines))
62 |
63 |
64 |
--------------------------------------------------------------------------------
/smdc/format/format.py:
--------------------------------------------------------------------------------
1 | import abc
2 | from smdc.note import Note
3 | import io
4 |
5 | class Format(abc.ABC):
6 |
7 | @abc.abstractmethod
8 | def parse(self, file: io.TextIOWrapper, name: str, parsed_notes: [Note], args) -> Note:
9 | ...
10 |
11 | @abc.abstractmethod
12 | def write(self, parsed_notes: [Note], args):
13 | ...
--------------------------------------------------------------------------------
/smdc/format/neo4j.py:
--------------------------------------------------------------------------------
1 | from py2neo.client import ConnectionUnavailable
2 |
3 | from smdc.format import Format
4 | from smdc.note import Note
5 | import io
6 | from smdc.format.cypher import escape_cypher
7 | from smdc.parse import obsidian_url, PROP_VAULT, PROP_PATH
8 | from py2neo import Graph, Node, Relationship, Subgraph
9 | from py2neo.database.work import ClientError
10 | from pathlib import Path
11 | import tqdm
12 |
13 | CAT_DANGLING = "SMD_dangling"
14 | CAT_NO_TAGS = "SMD_no_tags"
15 |
16 | PROP_COMMUNITY = "SMD_community"
17 | INDEX_PROPS = ['name', 'aliases']
18 |
19 | def get_community(note: Note, communities: [str], community_type: str):
20 | if community_type == "tags":
21 | if note.tags:
22 | community = escape_cypher(note.tags[0])
23 | else:
24 | community = CAT_NO_TAGS
25 | elif community_type == "folders":
26 | community = str(Path(note.properties[PROP_PATH]).parent)
27 | if community not in communities:
28 | communities.append(community)
29 | return communities.index(community)
30 |
31 | def node_from_note(note: Note, all_tags: [str], all_communities: [str], community_type: str) -> Node:
32 | tags = [CAT_NO_TAGS]
33 | if note.tags:
34 | tags = list(map(escape_cypher, note.tags))
35 | for tag in tags:
36 | if tag not in all_tags:
37 | all_tags.append(tag)
38 | properties = {}
39 | for property, value in note.properties.items():
40 | properties[property] = escape_cypher(str(value))
41 | properties[PROP_COMMUNITY] = get_community(note, all_communities, community_type)
42 | return Node(*tags, **properties)
43 |
44 | def add_rels_between_nodes(rels, src_node, trgt_node, subgraph: [Relationship]):
45 | # Adds all relations between src node and trgt node as described in rels to subgraph
46 | for rel in rels:
47 | properties = {}
48 | for property, value in rel.properties.items():
49 | properties[property] = escape_cypher(str(value))
50 | subgraph.append(Relationship(src_node, escape_cypher(rel.type), trgt_node, **properties))
51 |
52 |
53 | def create_index(graph, tag):
54 | try:
55 | for prop in INDEX_PROPS:
56 | graph.run(f"CREATE INDEX index_{prop}_{tag} IF NOT EXISTS FOR (n:{tag}) ON (n.{prop})")
57 | graph.run(f"CREATE INDEX index_name_vault IF NOT EXISTS for (n:{tag}) ON (n.{prop})")
58 | except ClientError as e:
59 | print(e)
60 | print(f"Warning: Could not create index for {tag}", flush=True)
61 |
62 | def create_dangling(name:str, vault_name:str, all_communities: [str]) -> Node:
63 | n = Node(CAT_DANGLING, name=escape_cypher(name), community=all_communities.index(CAT_DANGLING),
64 | obsidian_url=escape_cypher(obsidian_url(name, vault_name)))
65 | n[PROP_VAULT] = vault_name
66 | return n
67 |
68 | class Neo4j(Format):
69 |
70 | def parse(self, file: io.TextIOWrapper, name, parsed_notes: [Note], args) -> Note:
71 | raise NotImplementedError
72 |
73 | def write(self, parsed_notes: [Note], args):
74 | try:
75 | g = Graph(password=args.password)
76 | except ClientError as e:
77 | print("invalid user credentials", flush=True)
78 | raise e
79 | except ConnectionUnavailable as e:
80 | print("no connection to db", flush=True)
81 | raise e
82 | tx = g.begin()
83 | if not args.retaindb:
84 | print("Clearing neo4j database")
85 | tx.run(f"MATCH (n) WHERE n.{PROP_VAULT}='{args.vault_name}' DETACH DELETE n")
86 |
87 | nodes = {}
88 | print("Converting nodes", flush=True)
89 | all_tags = [CAT_DANGLING, CAT_NO_TAGS]
90 | all_communities = all_tags if args.community == "tags" else [CAT_DANGLING]
91 | # First create all nodes in the graph before doing the relationships, so they all exist.
92 | for name, note in tqdm.tqdm(parsed_notes.items()):
93 | node = node_from_note(note, all_tags, all_communities, args.community)
94 | nodes[name] = node
95 |
96 | if nodes:
97 | print("Transferring nodes to graph", flush=True)
98 | tx.create(Subgraph(nodes=nodes.values()))
99 |
100 | rels_to_create = []
101 | nodes_to_create = []
102 | print("Creating relationships", flush=True)
103 | i = 1
104 | for name, note in tqdm.tqdm(parsed_notes.items()):
105 | if not note.out_rels.keys():
106 | continue
107 | src_node = nodes[name]
108 | for trgt, rels in note.out_rels.items():
109 | if trgt not in nodes:
110 | nodes[trgt] = create_dangling(trgt, args.vault_name, all_communities)
111 | nodes_to_create.append(nodes[trgt])
112 | trgt_node = nodes[trgt]
113 | add_rels_between_nodes(rels, src_node, trgt_node, rels_to_create)
114 | # Send batches to server. Greatly speeds up conversion.
115 | if i % args.batch_size == 0:
116 | tx.create(Subgraph(nodes=nodes_to_create, relationships=rels_to_create))
117 | rels_to_create = []
118 | nodes_to_create = []
119 | i += 1
120 | if rels_to_create or nodes_to_create:
121 | tx.create(Subgraph(nodes=nodes_to_create, relationships=rels_to_create))
122 | print("Committing data", flush=True)
123 | tx.commit()
124 |
125 | print("Creating index", flush=True)
126 | # TODO: Schema inference for auto-indexing?
127 | for tag in tqdm.tqdm(all_tags):
128 | create_index(g, tag)
129 | try:
130 | g.run("CALL db.index.fulltext.drop(\"SMDnameAlias\")")
131 | except ClientError:
132 | pass
133 | try:
134 | g.run("CALL db.index.fulltext.drop(\"SMDcontent\")")
135 | except ClientError:
136 | pass
137 | if all_tags:
138 | g.run("CALL db.index.fulltext.createNodeIndex(\"SMDnameAlias\", [\"" + "\", \"".join(all_tags) + "\"], [\"name\", \"aliases\"])")
139 | if args.index_content:
140 | g.run("CALL db.index.fulltext.createNodeIndex(\"SMDcontent\", [\"" + "\", \"".join(all_tags) + "\"], [\"content\"])")
141 | return g, all_tags, all_communities
142 |
143 |
144 |
--------------------------------------------------------------------------------
/smdc/format/typed_list.py:
--------------------------------------------------------------------------------
1 | from smdc.format import Format
2 | from smdc.note import Note, Relationship
3 | import io
4 | import os
5 | from smdc.format.util import parse_yaml_header, get_tags_from_line, get_wikilinks_from_line, parse_wikilink, \
6 | PUNCTUATION, markdownToHtml
7 |
8 |
9 | class TypedList(Format):
10 |
11 | def parse_word(self, line, index, breaks=[' ', os.linesep, ',']):
12 | if index >= len(line):
13 | return index
14 | for j in range(index, len(line)):
15 | if line[j] in breaks:
16 | return j
17 |
18 | return j + 1
19 |
20 | def move_index(self, line, index):
21 | if index >= len(line):
22 | return index
23 | for j in range(index, len(line)):
24 | if line[j] != ' ':
25 | return j
26 |
27 | return j
28 |
29 | def parse(self, file: io.TextIOWrapper, name, parsed_notes: [Note], args) -> Note:
30 | line = file.readline()
31 | parsed_yaml = None
32 | # Find YAML header, or continue
33 | while line:
34 | if line.strip():
35 | if line == '---' + os.linesep:
36 | try:
37 | parsed_yaml = parse_yaml_header(file)
38 | except Exception as e:
39 | print(e)
40 | break
41 | line = file.readline()
42 |
43 | content = []
44 | relations = {}
45 | tags = []
46 | while line:
47 | if line.startswith(f"{args.typed_links_prefix} ") and len(line) > 2:
48 | is_rel = True
49 | index = self.parse_word(line, 2, breaks=PUNCTUATION)
50 | type = line[2:index]
51 | if len(type) != 0:
52 | index = self.move_index(line, index + 1)
53 | words = []
54 | while index < len(line) - 2:
55 | new_index = self.parse_word(line, index)
56 | words.append(line[index:new_index])
57 | index = self.move_index(line, new_index + 1)
58 | year = None
59 | trgts = []
60 | active_trgt = None
61 | for i, word in enumerate(words):
62 | if word[:2] == "[[":
63 | if active_trgt:
64 | is_rel = False
65 | break
66 | if word[-2:] == "]]":
67 | trgts.append(parse_wikilink(word[2:-2], name))
68 | else:
69 | active_trgt = word[2:]
70 | continue
71 | elif word[-2:] == "]]":
72 | if not active_trgt:
73 | is_rel = False
74 | break
75 | trgts.append(parse_wikilink(active_trgt + " " + word[:-2], name))
76 | active_trgt = None
77 | continue
78 | if i == 0 and type in ['publishedIn', 'at']:
79 | year = word
80 | elif active_trgt:
81 | active_trgt += " " + word
82 | else:
83 | is_rel = False
84 | break
85 | if is_rel:
86 | for trgt in trgts:
87 | properties = {}
88 | if year:
89 | properties["year"] = year
90 | rel = Relationship(type, properties)
91 | if trgt in relations:
92 | relations[trgt].append(rel)
93 | else:
94 | relations[trgt] = [rel]
95 | line = file.readline()
96 | continue
97 | content.append(line)
98 | for tag in get_tags_from_line(line):
99 | if tag not in tags:
100 | tags.append(tag)
101 | # TODO: Save aliases as Relation property
102 | for wikilink in get_wikilinks_from_line(line, name):
103 | rel = Relationship("inline",
104 | properties={"context": line,
105 | "parsedContext": markdownToHtml(line) if args.convert_markdown else ""})
106 | if wikilink in relations:
107 | relations[wikilink].append(rel)
108 | else:
109 | relations[wikilink] = [rel]
110 | line = file.readline()
111 | raw_content = markdownToHtml("".join(content)) if args.convert_markdown else "".join(content)
112 | return Note(name, tags, raw_content, out_rels=relations, properties=parsed_yaml if parsed_yaml else {})
113 |
114 | def write(self, file, parsed_notes: [Note]):
115 | raise NotImplementedError
--------------------------------------------------------------------------------
/smdc/format/util.py:
--------------------------------------------------------------------------------
1 | import yaml
2 | import os
3 | import markdown
4 | from markdown.extensions import Extension
5 | from markdown.preprocessors import Preprocessor
6 | import re
7 |
8 | def parse_yaml_header(file):
9 | lines = []
10 | line = file.readline()
11 | while line != "---" + os.linesep and line:
12 | lines.append(line)
13 | line = file.readline()
14 |
15 | return yaml.safe_load("".join(lines))
16 |
17 |
18 | class HashtagExtension(Extension):
19 | # Code based on https://github.com/Kongaloosh/python-markdown-hashtag-extension/blob/master/markdown_hashtags/markdown_hashtag_extension.py
20 | # Used to extract tags from markdown
21 | def extendMarkdown(self, md):
22 | """ Add FencedBlockPreprocessor to the Markdown instance. """
23 | md.registerExtension(self)
24 | md.preprocessors.register(HashtagPreprocessor(md), 'hashtag', 10) # After HTML Pre Processor
25 |
26 |
27 | class HashtagPreprocessor(Preprocessor):
28 | ALBUM_GROUP_RE = re.compile(
29 | r"""(?:(?<=\s)|^)#(\w*[A-Za-z_]+\w*)"""
30 | )
31 |
32 | def __init__(self, md):
33 | super(HashtagPreprocessor, self).__init__(md)
34 |
35 | def run(self, lines):
36 | """ Match and store Fenced Code Blocks in the HtmlStash. """
37 | HASHTAG_WRAP = ''' #{0}'''
38 | text = "\n".join(lines)
39 | while True:
40 | hashtag = ''
41 | m = self.ALBUM_GROUP_RE.search(text)
42 | if m:
43 | hashtag += HASHTAG_WRAP.format(m.group()[1:])
44 | placeholder = self.markdown.htmlStash.store(hashtag)
45 | text = '%s %s %s' % (text[:m.start()], placeholder, text[m.end():])
46 | else:
47 | break
48 | return text.split('\n')
49 |
50 |
51 | # def makeExtension(*args, **kwargs):
52 | # return HashtagExtension(*args, **kwargs)
53 |
54 | def markdownToHtml(md_text):
55 | return markdown.markdown(md_text, extensions=['mdx_wikilink_plus','fenced_code',
56 | 'footnotes', 'tables', HashtagExtension()],
57 | extension_configs={
58 | 'mdx_wikilink_plus': {
59 | 'html_class': 'internal-link',
60 | 'url_whitespace': ' '
61 | }
62 | })
63 |
64 | PUNCTUATION = ["#", "$", "!", ".", ",", "?", "/", ":", ";", "`", " ", "-", "+", "=", "|", os.linesep] + [str(i) for i in range(0, 10)]
65 |
66 |
67 | def get_tags_from_line(line) -> [str]:
68 | pos_tags = [i for i, char in enumerate(line) if char == '#']
69 | tags = []
70 | for i in pos_tags:
71 | if i == 0 or line[i - 1] == ' ':
72 | index = next((index for index, c in enumerate(line[i+1:]) if c in PUNCTUATION), -1)
73 | if index == -1:
74 | tags.append(line[i+1:])
75 | else:
76 | tag = line[i + 1:index + i + 1]
77 | if len(tag) > 0:
78 | tags.append(tag)
79 | return tags
80 |
81 | def parse_wikilink(between_brackets:str, note_title: str) -> str:
82 | first_arg = between_brackets.split("|")[0]
83 | if len(first_arg) != 0:
84 | title = first_arg.split("#")[0]
85 | if len(title) == 0:
86 | # Wikilinks like [[#header]] refer to itself
87 | return note_title
88 | else:
89 | # Wikilinks like [[title#header]]
90 | return title
91 | return ""
92 |
93 | def get_wikilinks_from_line(line, note_title) -> [str]:
94 | result = re.findall('\[\[(.*?)\]\]', line)
95 | if result:
96 | r = []
97 | for wikilink in result:
98 | title = parse_wikilink(wikilink, note_title)
99 | if title:
100 | r.append(title)
101 | return r
102 | return []
103 |
104 | def escape_quotes(string) -> str:
105 | r1 = string.replace("\\\'", "\\\\\'")
106 | r1 = r1.replace("'", "\\\'")
107 | return r1.replace('\"', "\\\"")
108 |
--------------------------------------------------------------------------------
/smdc/note.py:
--------------------------------------------------------------------------------
1 | import os
2 |
3 | class Relationship:
4 | def __init__(self, type: str, properties={}):
5 | self.type = type
6 | self.properties = properties
7 |
8 | def __str__(self):
9 | return self.type + self.properties.__str__()
10 |
11 |
12 | class Note:
13 |
14 | def __init__(self, name:str, tags: [str], content:str, properties={}, out_rels={}, in_rels={}):
15 | self.tags = tags
16 | self.out_rels = out_rels
17 | self.in_rels = in_rels
18 | self.properties = properties
19 | self.properties['name'] = name
20 | self.properties['content'] = content
21 |
22 |
23 | def add_out_rel(self, to:str, rel:Relationship):
24 | if to in self.out_rels:
25 | self.out_rels[to].append(rel)
26 | else:
27 | self.out_rels[to] = [rel]
28 |
29 | def add_in_rel(self, src: str, rel: Relationship):
30 | if src in self.in_rels:
31 | self.in_rels[src].append(rel)
32 | else:
33 | self.in_rels[src] = [rel]
34 |
35 | @property
36 | def name(self):
37 | return self.properties['name']
38 |
39 | @property
40 | def content(self):
41 | return self.properties['content']
42 |
43 | def __str__(self):
44 | return self.name + os.linesep + self.tags.__str__() + os.linesep + self.content + os.linesep + self.out_rels.__str__()
45 |
--------------------------------------------------------------------------------
/smdc/parse.py:
--------------------------------------------------------------------------------
1 | from pathlib import Path
2 | from smdc.format import Format
3 | from tqdm import tqdm
4 | import os
5 | from urllib.parse import quote
6 |
7 | PROP_OBSIDIAN_URL = "obsidian_url"
8 | PROP_PATH = "SMD_path"
9 | PROP_VAULT = "SMD_vault"
10 |
11 | def note_name(path, extension=".md"):
12 | return os.path.basename(path)[:-len(extension)]
13 |
14 | def obsidian_url(name:str, vault:str) -> str:
15 | return "obsidian://open?vault=" + quote(vault) + "&file=" + quote(name) + ".md"
16 |
17 | def parse_note(format: Format, note_path, args):
18 | name = note_name(note_path)
19 | with open(Path(note_path), 'r', encoding='utf-8') as f:
20 | # TODO: This isn't passing parsed notes right now. But this isn't currently used.
21 | note = format.parse(f, name, [], args)
22 | # Assign automatic properties for handling data and plugins
23 | note.properties[PROP_OBSIDIAN_URL] = obsidian_url(name, args.vault_name)
24 | note.properties[PROP_PATH] = note_path
25 | note.properties[PROP_VAULT] = args.vault_name
26 | return note
27 |
28 | def parse_folder(format: Format, args):
29 | notes_path = args.input
30 | note_extension = args.extension
31 | vault_name = args.vault_name
32 | if args.r:
33 | iterate = Path(notes_path).rglob("*" + note_extension)
34 | else:
35 | iterate = Path(notes_path).glob("*" + note_extension)
36 | all_files = list(iterate)
37 | parsed_notes = {}
38 | print("Parsing notes", flush=True)
39 | for path in tqdm(all_files):
40 | with open(path, mode='r', encoding='utf-8') as f:
41 | name = note_name(path, note_extension)
42 | try:
43 | note = format.parse(f, name, parsed_notes, args)
44 | note.properties[PROP_OBSIDIAN_URL] = obsidian_url(name, vault_name)
45 | note.properties[PROP_PATH] = path
46 | note.properties[PROP_VAULT] = vault_name
47 | parsed_notes[name] = note
48 | except Exception as e:
49 | print(e)
50 | print("Exception raised during parsing " + str(path) + ". Skipping this note! Please report this.", flush=True)
51 | print("Finished parsing notes", flush=True)
52 | return parsed_notes
53 |
54 |
--------------------------------------------------------------------------------
/smdc/stream.py:
--------------------------------------------------------------------------------
1 | from smdc import server_args, convert, parse_note, FORMAT_TYPES, Note, note_name, obsidian_url
2 | from watchdog.events import PatternMatchingEventHandler
3 | from watchdog.observers import Observer
4 | import time
5 | from py2neo import Node, Subgraph, Relationship, Graph
6 | from py2neo.data import walk
7 | from smdc.format.neo4j import node_from_note, add_rels_between_nodes, CAT_DANGLING, CAT_NO_TAGS, create_index, \
8 | create_dangling, PROP_COMMUNITY, get_community
9 | from smdc.format.cypher import escape_cypher
10 | from pathlib import Path
11 | import smdc
12 | from smdc.parse import PROP_VAULT, PROP_PATH
13 |
14 |
15 | def wrapper(fn):
16 | def _return(event):
17 | try:
18 | fn(event)
19 | except BaseException as e:
20 | print(e)
21 | return _return
22 |
23 |
24 |
25 | class SMDSEventHandler():
26 | def __init__(self, graph: Graph, tags: [str], communities: [str], args):
27 | self.graph = graph
28 | self.nodes = graph.nodes
29 | self.relationships = graph.relationships
30 | self.args = args
31 | self.input_format = FORMAT_TYPES[args.input_format]
32 | self.vault_name = args.vault_name
33 | self.index_content = args.index_content
34 | self.tags = tags
35 | self.communities = communities
36 |
37 | def _clear_outgoing(self, node: Node):
38 | rels = self.relationships.match([node, None])
39 | if len(rels) > 0:
40 | self.graph.separate(Subgraph(relationships=rels))
41 |
42 | def _process_node_on_graph(self, note: Note):
43 | if smdc.DEBUG:
44 | print(note, flush=True)
45 | in_graph = self.nodes.match(**{'name': note.name, PROP_VAULT: self.vault_name})
46 | if len(in_graph) == 0:
47 | # Create new node
48 | node = node_from_note(note, self.tags, self.communities, self.args.community)
49 | if smdc.DEBUG:
50 | print("creating")
51 | print(node, flush=True)
52 | self.graph.create(node)
53 | return
54 | # Update
55 | node = in_graph.first()
56 | if smdc.DEBUG:
57 | print("updating")
58 | print(node, flush=True)
59 | # Update labels
60 | node.clear_labels()
61 | note_tags = [CAT_NO_TAGS]
62 | if note.tags:
63 | note_tags = list(map(escape_cypher, note.tags))
64 | node.update_labels(note_tags)
65 | for tag in note_tags:
66 | if tag not in self.tags:
67 | create_index(self.graph, tag)
68 | self.tags.append(tag)
69 | # Update properties
70 | node.clear()
71 | escaped_properties = {}
72 | for key, value in note.properties.items():
73 | escaped_properties[key] = escape_cypher(str(value))
74 | escaped_properties[PROP_COMMUNITY] = get_community(note, self.communities, self.args.community)
75 | node.update(escaped_properties)
76 | self.graph.push(node)
77 |
78 | # # Delete active relations
79 | # self._clear_outgoing(node)
80 |
81 | # Insert up-to-date relations
82 | rels_to_create = []
83 | nodes_to_create = []
84 | not_matched_active_rels = list(map(lambda r: r.identity, self.relationships.match([node, None])))
85 | for trgt, rels in note.out_rels.items():
86 | trgt_node = self.nodes.match(**{'name': trgt, PROP_VAULT: self.vault_name})
87 | if len(trgt_node) == 0:
88 | trgt_node = create_dangling(trgt, self.vault_name, self.tags)
89 | nodes_to_create.append(trgt_node)
90 | else:
91 | trgt_node = trgt_node.first()
92 | # Possibly refactor this with
93 | for i, rel in enumerate(rels):
94 | properties = {}
95 | for property, value in rel.properties.items():
96 | properties[property] = escape_cypher(str(value))
97 | rel_type = escape_cypher(rel.type)
98 | found_rel = False
99 | active_rels = list(filter(lambda r: r.identity in not_matched_active_rels,
100 | self.relationships.match([node, None])))
101 | # Update instead of removing makes sure the relationship has a persistent id
102 | for active_rel in active_rels:
103 | walks = list(walk(active_rel))
104 | if type(active_rel).__name__ == rel_type and walks[2] == trgt_node:
105 | # Maybe this can leave dangling properties? But that'' an edge case. Not sure how to clear properties.
106 | active_rel.clear()
107 | active_rel.update(properties)
108 | self.graph.push(active_rel)
109 | found_rel = True
110 | not_matched_active_rels.remove(active_rel.identity)
111 | break
112 | if not found_rel:
113 | rels_to_create.append(Relationship(node, rel_type, trgt_node, **properties))
114 |
115 | if rels_to_create or nodes_to_create:
116 | self.graph.create(Subgraph(nodes=nodes_to_create, relationships=rels_to_create))
117 | if len(not_matched_active_rels) > 0:
118 | rels = list(filter(lambda r: r.identity in not_matched_active_rels,
119 | self.relationships.match([node, None])))
120 | if len(rels) > 0:
121 | self.graph.separate(Subgraph(relationships=rels))
122 | print("onSMDRelDeletedEvent/" + "/".join(map(str, not_matched_active_rels)))
123 |
124 | def _print_debug_rel(self, node, relationship):
125 | print(len(list(self.relationships.match([node, None]))))
126 | l = list(walk(relationship))
127 | print(l[0].identity, l[0]["name"])
128 | print(l[1])
129 | print(l[2].identity, l[2]["name"])
130 |
131 | def on_created(self):
132 | def _on_created(event):
133 | if smdc.DEBUG:
134 | print("On created", event.src_path, flush=True)
135 | # TODO: What if this name already exists in the vault? Does it make sense to override old data?
136 | note = parse_note(self.input_format, event.src_path, self.args)
137 | self._process_node_on_graph(note)
138 | return wrapper(_on_created)
139 |
140 | def on_deleted(self):
141 | def _on_deleted(event):
142 | if smdc.DEBUG:
143 | print("On deleted", event.src_path, flush=True)
144 | name = note_name(event.src_path)
145 | node = self.nodes.match(name=name).first()
146 | node_id = node.identity
147 | in_rels = self.relationships.match([None, node])
148 | if len(in_rels) > 0:
149 | # If there are still active incoming links, keep the node as a reference
150 | node.clear()
151 | node.clear_labels()
152 | node.add_label(CAT_DANGLING)
153 | node.name = escape_cypher(name)
154 | node.obsidian_url = escape_cypher(obsidian_url(name, self.vault_name))
155 | self._clear_outgoing(node)
156 | else:
157 | self.graph.delete(node)
158 | print(f"onSMDDeletedEvent/{node_id}", flush=True)
159 | return wrapper(_on_deleted)
160 |
161 | def on_modified(self):
162 | def _on_modified(event):
163 | if smdc.DEBUG:
164 | print("On modified", event.src_path, flush=True)
165 | note = parse_note(self.input_format, event.src_path, self.args)
166 | self._process_node_on_graph(note)
167 | print(f"onSMDModifyEvent/{note.name}", flush=True)
168 | return wrapper(_on_modified)
169 |
170 | def on_moved(self):
171 | def _on_moved(event):
172 | if smdc.DEBUG:
173 | print("On moved", event.src_path, event.dest_path, flush=True)
174 | old_name = note_name(event.src_path)
175 | node = self.nodes.match(name=old_name).first()
176 | new_name = note_name(event.dest_path)
177 | # TODO: What if this name already exists in the vault?
178 | node['name'] = new_name
179 | node['obsidian_url'] = obsidian_url(new_name, self.vault_name)
180 | node[PROP_PATH] = event.dest_path
181 | self.graph.push(node)
182 | print(f"onSMDMovedEvent/{old_name}/{new_name}", flush=True)
183 | return wrapper(_on_moved)
184 |
185 | def stream(graph, tags, communities, args):
186 | # Code credit: http://thepythoncorner.com/dev/how-to-create-a-watchdog-in-python-to-look-for-filesystem-changes/
187 | event_handler = PatternMatchingEventHandler(patterns=["*.md"], case_sensitive=True)
188 |
189 | smds_event_handler = SMDSEventHandler(graph, tags, communities, args)
190 | event_handler.on_created = smds_event_handler.on_created()
191 | event_handler.on_deleted = smds_event_handler.on_deleted()
192 | event_handler.on_modified = smds_event_handler.on_modified()
193 | event_handler.on_moved = smds_event_handler.on_moved()
194 |
195 | observer = Observer()
196 | path = Path(args.input)
197 | if smdc.DEBUG:
198 | print(path.absolute(), flush=True)
199 | observer.schedule(event_handler, path=Path(args.input), recursive=args.r)
200 |
201 | observer.start()
202 |
203 | try:
204 | print("Stream is active!", flush=True)
205 | import sys
206 | sys.stdout.flush()
207 | while True:
208 | # TODO: Catch when connection to neo4j server is down.
209 | time.sleep(1)
210 | except KeyboardInterrupt:
211 | observer.stop()
212 | observer.join()
213 |
214 | def main():
215 | args = server_args()
216 | args.output_format = 'neo4j'
217 | # Initialize the database
218 | graph, tags, communities = convert(args)
219 | # return
220 | # Start the server
221 | stream(graph, tags, communities, args)
222 |
223 |
224 | if __name__ == "__main__":
225 | main()
226 |
--------------------------------------------------------------------------------