├── .gitattributes ├── .gitignore ├── README.md ├── cozodb.asd ├── cozodb.lisp ├── library ├── libcozo_c.a ├── libcozo_c.d └── libcozo_c.so ├── package.lisp ├── system-index.txt ├── table.lisp └── test.lisp /.gitattributes: -------------------------------------------------------------------------------- 1 | library/libcozo_c.a filter=lfs diff=lfs merge=lfs -text 2 | library/libcozo_c.d filter=lfs diff=lfs merge=lfs -text 3 | library/libcozo_c.so filter=lfs diff=lfs merge=lfs -text 4 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *~ 2 | *.fasl 3 | \#*\# 4 | .\#* 5 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Lisp Cozodb 2 | 3 | ## About Cozodb 4 | 5 | This is a Lisp wrapper for embedding [Cozodb](https://www.cozodb.org). 6 | 7 | Cozodb is a really nice database with some interesting features: 8 | 9 | - Graph traversals 10 | - Vector search 11 | - Whole graph algorithms 12 | - Time travelling 13 | - Deduplication 14 | - Full-text search 15 | - Hi performance (up to 100K QPS) 16 | - Transactions 17 | 18 | Cozodb is based upon Datalog and is really easy to do pretty advanced logic with some easy queries. Syntax is really nice, readable and more powerfull than SQL - as far as I can see :) 19 | 20 | Documentation on usage is [here](https://docs.cozodb.org/en/latest/). 21 | 22 | ## Lisp api 23 | 24 | ### Installation 25 | 26 | (ql:quickload "cozodb") 27 | 28 | First you need the binary file. You can get it two ways: 29 | 30 | - Download Cozodb and follow instructions for [building](https://github.com/cozodb/cozo/tree/main/cozo-lib-c) 31 | - Or you can use the Linux version which is in the library folder in this project. If you have a Mac/Windows feel free to compile and make a pull request. 32 | 33 | Make sure the .so file is in your path. Easy way on Linux is to do a export LD_LIBRARY_PATH=/path_to_dir_with_so_file in the same whell you start lisp job from (or put it in your .bashrc for example) 34 | 35 | ### Usage 36 | 37 | First move to the package 38 | 39 | ``` 40 | (in-package cozodb) 41 | ``` 42 | 43 | Then open a database with: 44 | 45 | ``` 46 | (open-db "/tmp/mydir") 47 | ``` 48 | 49 | This returns a integer which will be needed when doing a query. Query can be done two ways: 50 | 51 | 1) Raw query. Following line creates a table and load it with two rows 52 | 53 | ``` 54 | (run-query db-id "?[l1, l2] <- [['a', 1], ['b',2]] :create stored {l1, l2}") 55 | ``` 56 | 57 | 2) Interpolated query. The same query, but with interpolation. The percentage sign is the interpolation char. 58 | 59 | ``` 60 | (query db-id "?[l1, l2] <- % :create stored {l1, l2}" '(("a" 1) ("b" 2)) 61 | ``` 62 | 63 | To query the data in the newly created table we can du this: 64 | ``` 65 | (run-query db-id "?[a, b] := *stored[a, b]") 66 | ``` 67 | 68 | The results of the query is in json format. 69 | 70 | To close database: 71 | ``` 72 | (close-db db-id) 73 | ``` 74 | 75 | There are two optional params when doing a query. They are extra-params and immutable. Have a look in the Cozodb for an exlpanation of these if you need them. 76 | 77 | Running examples are in the test.lisp file 78 | 79 | ## Backup and restore 80 | 81 | ``` 82 | (backup db-id "filename") 83 | (restore db-id "filename") 84 | ``` 85 | 86 | See in the test-file for example on how to backup/restore. 87 | 88 | ## Viewing queries in a tabular format 89 | 90 | The json can be tricky to read if you get more than a couple of rows. 91 | 92 | The function 93 | 94 | ``` 95 | (show-as-table (query...)) 96 | ``` 97 | 98 | can be used to show results in a tabular, nice format. It will also show time taken to run the query. 99 | 100 | ## Tests 101 | 102 | Test can be found in the test.lisp file 103 | 104 | They can be run with: 105 | ``` 106 | (in-package #:cozodb-test) 107 | (run! 'test-cozodb) 108 | ``` 109 | 110 | 111 | ### License 112 | 113 | Lisp part is BSD or Apache, your choice. 114 | 115 | -------------------------------------------------------------------------------- /cozodb.asd: -------------------------------------------------------------------------------- 1 | ;;;; cozodb.asd 2 | 3 | (asdf:defsystem #:cozodb 4 | :description "Binding to Cozodb" 5 | :author "Petter Egesund " 6 | :license "BSD-license" 7 | :version "0.0.1" 8 | :serial t 9 | :depends-on (#:cl-json #:cffi #:alexandria #:fiveam #:cl-fad) 10 | :components ((:file "package") 11 | (:file "cozodb") 12 | (:file "table") 13 | (:file "test") 14 | )) 15 | -------------------------------------------------------------------------------- /cozodb.lisp: -------------------------------------------------------------------------------- 1 | (in-package #:cozodb) 2 | 3 | (load-foreign-library '(:default "libcozo_c")) 4 | 5 | (defcfun "cozo_open_db" :string (engine :string) (path :string) (options :string) (db_id :pointer)) 6 | (defcfun "cozo_close_db" :boolean (id :int)) 7 | (defcfun "cozo_run_query" :pointer (db_id :int) (script_raw :string) (params_raw :string) (immutable_query :boolean)) 8 | (defcfun "cozo_free_str" :void (s :pointer)) 9 | (defcfun "cozo_export_relations" :pointer (db_id :int) (json_payload :string)) 10 | (defcfun "cozo_import_relations" :pointer (db_id :int) (json_payload :string)) 11 | (defcfun "cozo_backup" :pointer (db_id :int) (out_path :string)) 12 | (defcfun "cozo_restore" :pointer (db_id :int) (in_path :string)) 13 | 14 | 15 | ; use like this: (export-relations 0 "{\"relations\":[\"rel\"]}") 16 | ; returns a json to be written 17 | 18 | (defun export-relations(db-id json) 19 | (let* ((s (cozo-export-relations db-id json )) 20 | (ret (cffi:foreign-string-to-lisp s))) 21 | (cozo-free-str s) 22 | (with-input-from-string (s ret) 23 | (json:decode-json s)))) 24 | 25 | (defun import-relations(db-id json) 26 | (let* ((s (cozo-import-relations db-id json )) 27 | (ret (cffi:foreign-string-to-lisp s))) 28 | (cozo-free-str s) 29 | (with-input-from-string (s ret) 30 | (json:decode-json s)))) 31 | 32 | (defun backup(db-id out-path) 33 | (let* ((s (cozo-backup db-id out-path )) 34 | (ret (cffi:foreign-string-to-lisp s))) 35 | (cozo-free-str s) 36 | (with-input-from-string (s ret) 37 | (json:decode-json s)))) 38 | 39 | (defun restore(db-id in-path) 40 | (let* ((s (cozo-restore db-id in-path )) 41 | (ret (cffi:foreign-string-to-lisp s))) 42 | (cozo-free-str s) 43 | (with-input-from-string (s ret) 44 | (json:decode-json s)))) 45 | 46 | (defun open-db(db-dir) 47 | (let ((db-id-ptr (foreign-alloc :int :initial-element 0))) 48 | (cozo-open-db "rocksdb" db-dir "" db-id-ptr) 49 | (let ((db-id (mem-ref db-id-ptr :int))) 50 | (foreign-free db-id-ptr) 51 | db-id))) 52 | 53 | (defun close-db(id) 54 | (cozo-close-db id)) 55 | 56 | 57 | (defun val-to-chars(v) 58 | (cond ((typep v 'number) (coerce (write-to-string v) 'list)) 59 | ((typep v 'string) (append (cons #\' (coerce v 'list)) (list #\'))) 60 | ((typep v 'list) (concatenate 'list '(#\[) 61 | (mapcar #'(lambda (x) 62 | (append (val-to-chars x) (list #\,))) 63 | (butlast v)) 64 | (val-to-chars (car (last v))) 65 | '(#\]))) 66 | )) 67 | 68 | (defun run-query(id sr &optional (pr "") (iq nil)) 69 | (let* ((s (cozo-run-query id sr pr iq)) 70 | (ret (cffi:foreign-string-to-lisp s))) 71 | (cozo-free-str s) 72 | (with-input-from-string (s ret) 73 | (json:decode-json s)))) 74 | 75 | (defun query-interpolate(raw-string in-quote interpolated-string &rest params) 76 | (if (null raw-string) 77 | (coerce (alexandria:flatten interpolated-string) 'string) 78 | (let ((c (car raw-string))) 79 | (cond ((equal c #\') (query-interpolate (cdr raw-string) (not in-quote) (cons c interpolated-string) (car params))) 80 | (in-quote (query-interpolate (cdr raw-string) T (cons c interpolated-string) (car params))) 81 | ((equal c #\%) (query-interpolate (cdr raw-string) nil (append (val-to-chars (caar params)) interpolated-string) (cdar params))) 82 | (T (query-interpolate (cdr raw-string) nil (cons c interpolated-string) (car params))))))) 83 | 84 | (defun query-interpolate-top(raw-string &rest params) 85 | (query-interpolate (reverse (coerce raw-string 'list)) nil '() (reverse params))) 86 | 87 | (defun query(db-id query-string params &optional (extra-params "") (immutable nil)) 88 | (let* ((qs (query-interpolate-top query-string params)) 89 | (res (run-query db-id qs extra-params immutable)) 90 | (rows (assoc :ROWS res))) 91 | (if rows 92 | (cdr rows) 93 | res))) 94 | -------------------------------------------------------------------------------- /library/libcozo_c.a: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:c3d57e3a01301731b54520c412e6d119fcedbccb0f512046904a42584998f7b3 3 | size 128547936 4 | -------------------------------------------------------------------------------- /library/libcozo_c.d: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:23ac08f5efc9160dea46076c4b6765a14f7bc1f29a584c854e8baaef8fb72e1f 3 | size 6842 4 | -------------------------------------------------------------------------------- /library/libcozo_c.so: -------------------------------------------------------------------------------- 1 | version https://git-lfs.github.com/spec/v1 2 | oid sha256:3ba8235fb118f078fb3a249b1b9989ae1882798de77623cb14ee6217b7dc3814 3 | size 40643312 4 | -------------------------------------------------------------------------------- /package.lisp: -------------------------------------------------------------------------------- 1 | ;;;; package.lisp 2 | 3 | (defpackage #:cozodb 4 | (:use #:cl #:cffi) 5 | (:export #:open-db 6 | #:close-db 7 | #:run-query 8 | #:query 9 | #:backup 10 | #:restore 11 | #:show-as-table 12 | ) 13 | ) 14 | 15 | (defpackage #:cozodb-test 16 | (:use #:cl #:cozodb #:fiveam)) 17 | -------------------------------------------------------------------------------- /system-index.txt: -------------------------------------------------------------------------------- 1 | cozodb.asd 2 | -------------------------------------------------------------------------------- /table.lisp: -------------------------------------------------------------------------------- 1 | (in-package #:cozodb) 2 | 3 | ;code for formating output is borrowed from this url: https://gist.github.com/WetHat/a49e6f2140b401a190d45d31e052af8f 4 | 5 | (defvar CELL-FORMATS '(:left "~vA" 6 | :center "~v:@<~A~>" 7 | :right "~v@A")) 8 | 9 | (defun format-table (stream data &key (column-label (loop for i from 1 to (length (car data)) 10 | collect (format nil "COL~D" i))) 11 | (column-align (loop for i from 1 to (length (car data)) 12 | collect :left))) 13 | (let* ((col-count (length column-label)) 14 | (strtable (cons column-label ; table header 15 | (loop for row in data ; table body with all cells as strings 16 | collect (loop for cell in row 17 | collect (if (stringp cell) 18 | cell 19 | ;else 20 | (format nil "~A" cell)))))) 21 | (col-widths (loop with widths = (make-array col-count :initial-element 0) 22 | for row in strtable 23 | do (loop for cell in row 24 | for i from 0 25 | do (setf (aref widths i) 26 | (max (aref widths i) (length cell)))) 27 | finally (return widths)))) 28 | ;------------------------------------------------------------------------------------ 29 | ; splice in the header separator 30 | (setq strtable 31 | (nconc (list (car strtable) ; table header 32 | (loop for align in column-align ; generate separator 33 | for width across col-widths 34 | collect (case align 35 | (:left (format nil ":~v@{~A~:*~}" 36 | width "-")) 37 | (:right (format nil "~v@{~A~:*~}" 38 | width "-")) 39 | (:center (format nil ":~v@{~A~:*~}" 40 | width "-"))))) 41 | (cdr strtable))) ; table body 42 | ;------------------------------------------------------------------------------------ 43 | ; Generate the formatted table 44 | (let ((row-fmt (format nil "| ~{~A~^ | ~} |~~%" ; compile the row format 45 | (loop for align in column-align 46 | collect (getf CELL-FORMATS align)))) 47 | (widths (loop for w across col-widths collect w))) 48 | ; write each line to the given stream 49 | (dolist (row strtable) 50 | (apply #'format stream row-fmt (mapcan #'list widths row))))) 51 | ) 52 | 53 | 54 | (defun show-as-table(content) 55 | (if (assoc :CAUSES content) 56 | (format t "~a" content) 57 | (let* ((cl (cdr (assoc :HEADERS content))) 58 | (ca (make-list (length cl) :initial-element :right)) 59 | (rows (cdr (assoc :ROWS content))) 60 | (time-taken (cdr (assoc :TOOK content)))) 61 | (format-table t rows :column-label cl :column-align ca) 62 | (format t "~% Time taken in seconds: ~5$ " time-taken) 63 | ))) 64 | 65 | 66 | ; end def format table 67 | -------------------------------------------------------------------------------- /test.lisp: -------------------------------------------------------------------------------- 1 | (in-package #:cozodb-test) 2 | 3 | (def-suite test-cozodb 4 | :description "Test the read-file-as-string function. Run all test with: (run! 'test-cozodb) ") 5 | (in-suite test-cozodb) 6 | 7 | 8 | 9 | (test world-is-sane 10 | (is (= 1 1))) 11 | 12 | (test create-database-and-close 13 | (fad:delete-directory-and-files "/tmp/cozo_test1" :if-does-not-exist :ignore) 14 | (let ((db-id (open-db "/tmp/cozo/cozo_test2"))) 15 | (is (numberp db-id)) 16 | (close-db db-id)) 17 | (fad:delete-directory-and-files "/tmp/cozo_test1" :if-does-not-exist :ignore) 18 | ) 19 | 20 | 21 | ; Create database, insert some raw data, quere these and verify these are the same, close and clean 22 | (test raw-query 23 | (fad:delete-directory-and-files "/tmp/cozo_test2" :if-does-not-exist :ignore) 24 | (let* ((db-id (open-db "/tmp/cozo/cozo_test2")) 25 | (res (progn 26 | (run-query db-id"?[l1, l2] <- [['a', 1], ['b',2]] :create stored {l1, l2}") 27 | (run-query db-id "?[a, b] := *stored[a, b]") 28 | )) 29 | ) 30 | (is (equal (cdr (assoc :ROWS res)) '(("a" 1) ("b" 2)))) 31 | (close-db db-id)) 32 | (fad:delete-directory-and-files "/tmp/cozo_test2" :if-does-not-exist :ignore) 33 | ) 34 | 35 | ; Create database, insert some raw data, quere these and verify these are the same, close and clean 36 | (test interpolated-query 37 | (fad:delete-directory-and-files "/tmp/cozo_test3" :if-does-not-exist :ignore) 38 | (let* ((db-id (open-db "/tmp/cozo/cozo_test3")) 39 | (res (progn 40 | (query db-id "?[l1, l2] <- % :create stored {l1, l2}" '(("a" 1) ("b" 2))) 41 | (run-query db-id "?[a, b] := *stored[a, b]") 42 | )) 43 | ) 44 | (is (equal (cdr (assoc :ROWS res)) '(("a" 1) ("b" 2)))) 45 | (close-db db-id)) 46 | (fad:delete-directory-and-files "/tmp/cozo_test3" :if-does-not-exist :ignore) 47 | ) 48 | 49 | 50 | ; create a db and insert some values, backup and restore to another db 51 | ; make sure that vals are in the restored db 52 | (test backup 53 | (fad:delete-directory-and-files "/tmp/cozo_test4" :if-does-not-exist :ignore) 54 | (fad:delete-directory-and-files "/tmp/cozo_test5" :if-does-not-exist :ignore) 55 | (when (probe-file "/tmp/backup") (delete-file "/tmp/backup")) 56 | (let ((db-id1 (open-db "/tmp/cozo_test4")) 57 | (db-id2 (open-db "/tmp/cozo_test5"))) 58 | (query db-id1 "?[l1, l2] <- % :create stored {l1, l2}" '(("a" 1) ("b" 2))) 59 | (backup db-id1 "/tmp/cozo_backup") 60 | (restore db-id2 "/tmp/cozo_backup") 61 | (let ((res (run-query db-id2 "?[a, b] := *stored[a, b]"))) 62 | (print res) 63 | (is (equal (cdr (assoc :ROWS res)) '(("a" 1) ("b" 2))))) 64 | (close-db db-id1) 65 | (close-db db-id2)) 66 | (fad:delete-directory-and-files "/tmp/cozo_test4" :if-does-not-exist :ignore) 67 | (fad:delete-directory-and-files "/tmp/cozo_test5" :if-does-not-exist :ignore) 68 | (when (probe-file "/tmp/backup") (delete-file "/tmp/backup"))) 69 | 70 | 71 | 72 | 73 | 74 | --------------------------------------------------------------------------------