├── LICENSE ├── README.md ├── SophiaDatabase.lua ├── examples ├── batch.lua ├── crud.lua └── cursor.lua ├── extras.lua ├── libc.lua ├── sophia.lua ├── sophia_ffi.lua └── test ├── README.md ├── test_common.lua ├── test_cursor.lua ├── test_delete.lua ├── test_get.lua ├── test_sophia.lua └── test_sophiadb.lua /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Microsoft Public License (MS-PL) 3 | 4 | [OSI Approved License] 5 | 6 | 7 | This license governs use of the accompanying software. If you use the software, you 8 | accept this license. If you do not accept the license, do not use the software. 9 | 10 | 1. Definitions 11 | The terms "reproduce," "reproduction," "derivative works," and "distribution" have the 12 | same meaning here as under U.S. copyright law. 13 | A "contribution" is the original software, or any additions or changes to the software. 14 | A "contributor" is any person that distributes its contribution under this license. 15 | "Licensed patents" are a contributor's patent claims that read directly on its contribution. 16 | 17 | 2. Grant of Rights 18 | (A) Copyright Grant- Subject to the terms of this license, including the license conditions and limitations in section 3, each contributor grants you a non-exclusive, worldwide, royalty-free copyright license to reproduce its contribution, prepare derivative works of its contribution, and distribute its contribution or any derivative works that you create. 19 | (B) Patent Grant- Subject to the terms of this license, including the license conditions and limitations in section 3, each contributor grants you a non-exclusive, worldwide, royalty-free license under its licensed patents to make, have made, use, sell, offer for sale, import, and/or otherwise dispose of its contribution in the software or derivative works of the contribution in the software. 20 | 21 | 3. Conditions and Limitations 22 | (A) No Trademark License- This license does not grant you rights to use any contributors' name, logo, or trademarks. 23 | (B) If you bring a patent claim against any contributor over patents that you claim are infringed by the software, your patent license from such contributor to the software ends automatically. 24 | (C) If you distribute any portion of the software, you must retain all copyright, patent, trademark, and attribution notices that are present in the software. 25 | (D) If you distribute any portion of the software in source code form, you may do so only under this license by including a complete copy of this license with your distribution. If you distribute any portion of the software in compiled or object code form, you may only do so under a license that complies with this license. 26 | (E) The software is licensed "as-is." You bear the risk of using it. The contributors give no express warranties, guarantees or conditions. You may have additional consumer rights under your local laws which this license cannot change. To the extent permitted under your local laws, the contributors exclude the implied warranties of merchantability, fitness for a particular purpose and non-infringement. 27 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | LJIT2Sophia 2 | =========== 3 | 4 | Overview 5 | -------- 6 | LuaJIT FFI binding to the Sophia (1.1) database: https://github.com/pmwkaa/sophia 7 | 8 | 9 | The sophia database is a relatively new (September, 2013) key value store 10 | which is small and embeddable. This LuaJIT binding seems appropriate. 11 | 12 | The Sophia API has changed dramatically since the 1.1 version. The 1.1 13 | compatible version of this LuaJIT binding will be saved off as a 'release' 14 | and no further work will be done on it. 15 | 16 | A 1.2 version of the binding is in the works, so the files currently checked 17 | in represent that work in progress. 18 | 19 | All the documentation below corresponds to the 1.1 version of the API. Some 20 | or all of it might change for the 1.2 binding. 21 | 22 | API Access 23 | ---------- 24 | The binding gives you access to the component at two distinct levels. 25 | If you want to use sophia in a style similar to what you would do 26 | using the 'C' language, then you can simply do this: 27 | 28 | ``` 29 | sophia_ffi = require("sophia_ffi"); 30 | sophia_ffi.sp_env(); 31 | sophia_ffi.sp_db(...); 32 | ``` 33 | 34 | All of the functions are accessible through a simple table interface. If you would like to promote them to the global namespace, then call: sophia_ffi.promoteToGlobal(). If you do this, you can write code that looks almost identical to 'C' code. 35 | 36 | Object Oriented Access 37 | ---------------------- 38 | This is Lua, and the better way to access sophia is using more lua like semantics and constructs. An object model is presented, which makes it relatively easy to manipulate sophia databases. 39 | 40 | To create and/or open a database, simply call the constructor on the SophiaDatabase object. This will open up an existing database, or create it anew if it does not exist. 41 | 42 | ``` 43 | local sophia = require("sophia") 44 | local db, err = sophia.SophiaDatabase("./db"); 45 | ``` 46 | inserting a value 47 | ----------------- 48 | 49 | Once you have a database object, there are two ways to get values into it. The 'set()' method mimics the standard 'C' interface function, so you must pass all 4 parameters. 50 | 51 | ``` 52 | local success, err = db:set(keybuff, ffi.sizeof(keybuff), value, #str); 53 | ``` 54 | 55 | The 'upsert()' method allows you to set a value using a simple key/value pair, which are assumed to be lua string objects. 56 | 57 | ``` 58 | local success, err = db:upsert(key, value) 59 | ``` 60 | 61 | For this upsert operation, if the value does not exist in the database, it will be added. If the key already exists, then the value for that key will be changed to the value being presented. 62 | 63 | retrieving a value 64 | ------------------ 65 | 66 | There are two ways to retrieve a value from the database. The first mimics the standard 'C' interface function, and requires 4 parameters. 67 | 68 | ``` 69 | local success, err = db:get(keybuff, ffi.sizeof(keybuff), value, valuesize); 70 | ``` 71 | 72 | The second mechnism allows you to specify a single parameter, which is the key associated with the value you want to retrieve. A second 'keysize' parameter is allowed, but if it is not supplied, the #key length will be used. 73 | 74 | 75 | ``` 76 | local value, err = db:retrieve(key, keysize) 77 | ``` 78 | 79 | Upon completion, the 'value' will contain the value specified by the key. Otherwise, it will return false, and the associated error. 80 | 81 | 82 | 83 | Using a cursor to iterate over the entire database 84 | -------------------------------------------------- 85 | 86 | In addition to retrieving single values at a time, you can iterate over a range of values in the database. 87 | 88 | ``` 89 | for key, keysize, value, valuesize in db:iterate() do 90 | 91 | print(ffi.cast("int *",key)[0]); 92 | 93 | print(ffi.string(value, valuesize)); 94 | 95 | end 96 | ``` 97 | 98 | Rather than follow the approach of simply creating objects to wrap 99 | groupings of APIs, the binding presents what seems to make the most 100 | sense from a usability standpoint. 101 | 102 | The SophiaDatabase object is the primary interaction point. The 103 | SophiaEnvironment is a secondary player, which is used by the 104 | SophiaDatabase internally, but for the most part is hidden. 105 | 106 | The Database:iterate() function has the following signature: 107 | 108 | ``` 109 | SophiaDatabase.iterate = function(self, key, keysize, sporder) 110 | keysize = keysize or 0; 111 | sporder = sporder or ffi.C.SPGTE; 112 | ``` 113 | 114 | In this way, by default, the iterator will iterate over the entire 115 | database, from the beginning. The next logical thing the user 116 | is likely to want is to iterate from a specific point, so the key 117 | value is the first parameter. Lastly, they might want to specify 118 | an order. The thinking is, if they're specifying an order, then they 119 | MUST also specify a key, so the odering can come last. 120 | 121 | Using an iterator is natural for a lua program, so that's the choice. 122 | Of course, you can use the lower level sophia_ffi calls and create 123 | whatever specialized interface you prefer, but this is the default 124 | set of convenience provided. 125 | 126 | Going forward, the set/get/iterate methods could do a little bit of 127 | magic and assume things like lengths of lua strings can be inferred, 128 | and if the key is a 'number', it can be converted to a 4 or 8 byte 129 | value. this will make it a little bit convenient to work with these 130 | values without having to specify lengths explicitly. 131 | 132 | LICENSE 133 | ------- 134 | This Software is licensed under the Microsoft Public License, which is a fairly straight forward open source license. 135 | 136 | http://opensource.org/licenses/ms-pl 137 | 138 | AUTHOR 139 | ------ 140 | William A Adams 141 | http://williamaadams.wordpress.com 142 | 143 | -------------------------------------------------------------------------------- /SophiaDatabase.lua: -------------------------------------------------------------------------------- 1 | 2 | local sophia = require("sophia") 3 | 4 | ffi.cdef[[ 5 | typedef struct { 6 | void * Handle; 7 | } SophiaHandle; 8 | ]] 9 | 10 | local SophiaHandle = ffi.typeof("SophiaHandle"); 11 | local SophiaHandle_mt = { 12 | __new = function(ct, rawHandle) 13 | return ffi.new(ct, rawHandle); 14 | end, 15 | 16 | __gc = function(self) 17 | --print("GC SophiaHandle: ", self.Handle) 18 | if self.Handle == nil then 19 | return ; 20 | end 21 | 22 | sophia_ffi.sp_destroy(self.Handle); 23 | 24 | self.Handle = nil; 25 | end, 26 | 27 | __index = { 28 | free = function(self) 29 | sophia_ffi.sp_destroy(self.Handle); 30 | self.Handle = nil; 31 | end, 32 | }, 33 | } 34 | ffi.metatype(SophiaHandle, SophiaHandle_mt); 35 | 36 | 37 | 38 | local SophiaEnvironment = {} 39 | setmetatable(SophiaEnvironment, { 40 | __call = function(self, ...) 41 | return self:create(...); 42 | end; 43 | }); 44 | 45 | SophiaEnvironment_mt = { 46 | __index = SophiaEnvironment; 47 | } 48 | 49 | SophiaEnvironment.init = function(self, safeHandle) 50 | 51 | local obj = { 52 | Handle = safeHandle; 53 | } 54 | 55 | setmetatable(obj, SophiaEnvironment_mt); 56 | return obj; 57 | end 58 | 59 | SophiaEnvironment.create = function(self, directory) 60 | 61 | directory = directory or "./db" 62 | 63 | local rawHandle = sophia_ffi.sp_env(); 64 | if rawHandle == nil then 65 | return nil; 66 | end 67 | 68 | local envHandle = SophiaHandle(rawHandle); 69 | if not envHandle then 70 | return nil; 71 | end 72 | 73 | local rc = sophia_ffi.sp_ctl(envHandle.Handle, ffi.C.SPDIR, 74 | ffi.cast("int32_t",bor(ffi.C.SPO_CREAT,ffi.C.SPO_RDWR)), 75 | directory); 76 | 77 | if (rc == -1) then 78 | return nil, sophia_ffi.sp_error(envHandle.Handle); 79 | end 80 | 81 | return self:init(envHandle); 82 | end 83 | 84 | 85 | SophiaEnvironment.getNativeHandle = function(self) 86 | return self.Handle.Handle; 87 | end 88 | 89 | SophiaEnvironment.open = function(self) 90 | local dbhandle = sophia_ffi.sp_open(self:getNativeHandle()); 91 | 92 | 93 | if (dbhandle == nil) then 94 | return nil, sophia_ffi.sp_error(self:getNativeHandle()); 95 | end 96 | 97 | return SophiaHandle(dbhandle); 98 | end 99 | 100 | 101 | 102 | 103 | 104 | local SophiaDatabase = {} 105 | setmetatable(SophiaDatabase, { 106 | __call = function(self, ...) 107 | return self:create(...); 108 | end, 109 | }); 110 | 111 | local SophiaDatabase_mt = { 112 | __index = SophiaDatabase; 113 | } 114 | 115 | SophiaDatabase.init = function(self, env, dbhandle) 116 | local obj = { 117 | Environment = env; 118 | DBHandle = dbhandle; 119 | } 120 | 121 | setmetatable(obj, SophiaDatabase_mt); 122 | 123 | return obj 124 | end 125 | 126 | SophiaDatabase.create = function(self, dbname) 127 | -- create the environment 128 | local env, err = SophiaEnvironment(dbname); 129 | 130 | if not env then 131 | return nil, err; 132 | end 133 | 134 | -- open the database 135 | local dbhandle, err = env:open(); 136 | 137 | if not dbhandle then 138 | return nil, err 139 | end 140 | 141 | -- initialize the object 142 | return self:init(env, dbhandle); 143 | end 144 | 145 | SophiaDatabase.getNativeHandle = function(self) 146 | return self.DBHandle.Handle; 147 | end 148 | 149 | -- Database operations 150 | SophiaDatabase.set = function(self, key, keysize, value, valuesize) 151 | local rc = sophia_ffi.sp_set(self:getNativeHandle(), 152 | key, keysize, 153 | value, valuesize); 154 | 155 | if rc == -1 then 156 | return false, sophia_ffi.sp_error(self:getNativeHandle()); 157 | end 158 | 159 | return true 160 | end 161 | 162 | SophiaDatabase.upsert = function(self, ...) 163 | local nargs = select('#',...) 164 | -- if it's 4 values, then they MUST be 165 | -- key, keysize, value, valuesize 166 | -- if it's two values, then they MUST be 167 | -- key, value 168 | 169 | local key, keysize, value, valuesize 170 | 171 | if nargs == 4 then 172 | key = select(1, ...) 173 | keysize = select(2, ...) 174 | value = select(3, ...) 175 | valuesize = select(4, ...) 176 | elseif nargs == 2 then 177 | key = select(1, ...) 178 | keysize = #key 179 | value = select(2, ...) 180 | valuesize = #value 181 | else 182 | return false, "invalid number of arguments" 183 | end 184 | 185 | return self:set(key, keysize, value, valuesize); 186 | end 187 | 188 | 189 | SophiaDatabase.get = function(self, key, keysize, value, valuesize) 190 | local rc = sophia_ffi.sp_get(self:getNativeHandle(), key, keysize, value, valuesize); 191 | 192 | if rc == -1 then 193 | return false, sophia_ffi.sp_error(self:getNativeHandle()); 194 | end 195 | 196 | return true 197 | end 198 | 199 | SophiaDatabase.retrieve = function(self, ...) 200 | local nargs = select('#',...) 201 | 202 | -- if it's 4 values, then they MUST be 203 | -- key, keysize, value, valuesize 204 | -- if it's two values, then they MUST be 205 | -- key, value 206 | local key, keysize, value, valuesize 207 | if nargs == 4 then 208 | key = select(1, ...) 209 | keysize = select(2, ...) 210 | value = select(3, ...) 211 | valuesize = select(4, ...) 212 | elseif nargs == 1 then 213 | key = select(1, ...) 214 | keysize = #key 215 | value = ffi.new("void *[1]"); 216 | valuesize = ffi.new("size_t[1]"); 217 | else 218 | return false, "invalid number of arguments" 219 | end 220 | 221 | local success, err = self:get(key, keysize, value, valuesize); 222 | if not success then 223 | return false, err 224 | end 225 | 226 | local str = ffi.string(value[0], valuesize[0]) 227 | ffi.C.free(value[0]); 228 | 229 | return str 230 | end 231 | 232 | SophiaDatabase.delete = function(self, key, keysize) 233 | keysize = keysize or #key 234 | 235 | local rc = sophia_ffi.sp_delete(self:getNativeHandle(), key, keysize); 236 | if rc == -1 then 237 | return false, sophia_ffi.sp_error(self:getNativeHandle()); 238 | end 239 | 240 | return true 241 | end 242 | 243 | -- Transactions 244 | SophiaDatabase.begin = function(self) 245 | local rc = sophia_ffi.sp_begin(self:getNativeHandle()); 246 | 247 | if rc == -1 then 248 | return false, sophia_ffi.sp_error(self:getNativeHandle()); 249 | end 250 | 251 | return true 252 | end 253 | 254 | SophiaDatabase.commit = function(self) 255 | local rc = sophia_ffi.commit(self:getNativeHandle()); 256 | 257 | if rc == -1 then 258 | return false, sophia_ffi.sp_error(self:getNativeHandle()); 259 | end 260 | 261 | return true 262 | end 263 | 264 | SophiaDatabase.rollback = function(self) 265 | local rc = sophia_ffi.rollback(self:getNativeHandle()); 266 | 267 | if rc == -1 then 268 | return false, sophia_ffi.sp_error(self:getNativeHandle()); 269 | end 270 | 271 | return true 272 | end 273 | 274 | -- Cursors 275 | SophiaDatabase.iteration = function(self, key, keysize, sporder) 276 | keysize = keysize or 0; 277 | sporder = sporder or ffi.C.SPGTE; 278 | 279 | local cHandle = SophiaHandle(sophia_ffi.sp_cursor(self:getNativeHandle(), sporder, key, keysize)); 280 | 281 | print("cursor HANDLE: ", cHandle.Handle); 282 | 283 | local function closure() 284 | if cHandle.Handle == nil then 285 | return nil; 286 | end 287 | 288 | local rc = sophia_ffi.sp_fetch(cHandle.Handle) 289 | -- a value of '0' indicates no more records 290 | if rc == 0 then 291 | cHandle:free(); 292 | --collectgarbage(); 293 | return nil; 294 | end 295 | 296 | return sophia_ffi.sp_key(cHandle.Handle), sophia_ffi.sp_keysize(cHandle.Handle), 297 | sophia_ffi.sp_value(cHandle.Handle), sophia_ffi.sp_valuesize(cHandle.Handle) 298 | end 299 | 300 | return closure; 301 | end 302 | 303 | 304 | 305 | 306 | return { 307 | config = config; 308 | 309 | SophiaDatabase = SophiaDatabase; 310 | } -------------------------------------------------------------------------------- /examples/batch.lua: -------------------------------------------------------------------------------- 1 | package.path = package.path..";../?.lua;../lib/?.lua" 2 | 3 | local ffi = require("ffi") 4 | local sophia = require("sophia")() 5 | local libc = require("libc")() 6 | 7 | -- Do fast atomic write of 100 inserts. 8 | 9 | local function main() 10 | -- open or create environment and database 11 | local env = sp_env(); 12 | sp_setstring(env, config.sophia.path, "_test", 0); 13 | sp_setstring(env, "db", "test", 0); 14 | local db = sp_getobject(env, "db.test"); 15 | local rc = sp_open(env); 16 | assert(rc ~= -1, sp_strerror(env)) 17 | 18 | 19 | -- create batch object 20 | local batch = sp_batch(db); 21 | 22 | -- insert 100 keys 23 | local key = ffi.new("uint32_t[1]", 0); 24 | while (key[0] < 100) do 25 | local o = sp_object(db); 26 | sp_setstring(o, "key", key, ffi.sizeof("uint32_t")); 27 | rc = sp_set(batch, o); 28 | assert(rc ~= -1, sp_strerror(env)) 29 | 30 | key[0] = key[0] + 1; 31 | end 32 | 33 | -- write batch 34 | rc = sp_commit(batch); 35 | assert(rc ~= -1, sp_strerror(env)) 36 | 37 | -- finish work 38 | sp_destroy(env); 39 | 40 | return true; 41 | end 42 | 43 | main() 44 | 45 | -------------------------------------------------------------------------------- /examples/crud.lua: -------------------------------------------------------------------------------- 1 | package.path = package.path..";../?.lua" 2 | 3 | local ffi = require("ffi") 4 | local sophia = require("sophia")() 5 | local libc = require("libc")() 6 | 7 | 8 | 9 | local function main(argc, argv) 10 | 11 | -- Do set, get, delete operations. (see transaction.c) 12 | 13 | -- open or create environment and database 14 | local env = sp_env(); 15 | sp_setstring(env, "sophia.path", "_test", 0); 16 | sp_setstring(env, "db", "test", 0); 17 | local db = sp_getobject(env, "db.test"); 18 | local rc = sp_open(env); 19 | if (rc == -1) then 20 | error(rc); 21 | end 22 | 23 | -- set 24 | local key = ffi.new("uint32_t[1]", 1); 25 | local o = sp_object(db); 26 | sp_setstring(o, "key", key, ffi.sizeof("uint32_t")); 27 | sp_setstring(o, "value", key, ffi.sizeof("uint32_t")); 28 | rc = sp_set(db, o); 29 | if (rc == -1) then 30 | error(rc); 31 | end 32 | 33 | -- get 34 | o = sp_object(db); 35 | sp_setstring(o, "key", key, ffi.sizeof(key)); 36 | o = sp_get(db, o); 37 | if (o ~= nil) then 38 | -- ensure key and value are correct 39 | local size = ffi.new("int[1]"); 40 | local ptr = sp_getstring(o, "key", size); 41 | assert(size[0] == ffi.sizeof("uint32_t")); 42 | assert(ffi.cast("uint32_t*",ptr)[0] == key[0]); 43 | 44 | ptr = sp_getstring(o, "value", size); 45 | assert(size[0] == ffi.sizeof("uint32_t")); 46 | assert(ffi.cast("uint32_t*",ptr)[0] == key[0]); 47 | 48 | sp_destroy(o); 49 | end 50 | 51 | -- delete 52 | o = sp_object(db); 53 | sp_setstring(o, "key", key, ffi.sizeof(key)); 54 | rc = sp_delete(db, o); 55 | if (rc == -1) then 56 | error(rc); 57 | end 58 | 59 | -- finish work 60 | sp_destroy(env); 61 | 62 | return true; 63 | 64 | --[[ 65 | error:; 66 | int size; 67 | char *error = sp_getstring(env, "sophia.error", &size); 68 | printf("error: %s\n", error); 69 | free(error); 70 | sp_destroy(env); 71 | return 1; 72 | --]] 73 | end 74 | 75 | main(#arg, arg) 76 | -------------------------------------------------------------------------------- /examples/cursor.lua: -------------------------------------------------------------------------------- 1 | package.path = package.path..";../?.lua;../lib/?.lua" 2 | 3 | local ffi = require("ffi") 4 | local sophia = require("sophia")() 5 | local config = sophia.config; 6 | local libc = require("libc")() 7 | 8 | local function cursorForward(env, db) 9 | -- create cursor and do forward iteration 10 | local cursor = sp_cursor(env); 11 | local o = sp_object(db); 12 | 13 | repeat 14 | o = sp_get(cursor, o) 15 | if (o ~= nil) then 16 | local str = sp_getstring(o, "key", nil) 17 | local value = ffi.cast("uint32_t *", str)[0]; 18 | print("STR, VALUE: ", str, value) 19 | --print(string.format("%d", value)) 20 | end 21 | until o == nil 22 | sp_destroy(cursor); 23 | end 24 | 25 | local function cursorBack(env, db) 26 | -- create cursor and do backward iteration 27 | local cursor = sp_cursor(env); 28 | local o = sp_object(db); 29 | sp_setstring(o, "order", "<", 0); 30 | 31 | repeat 32 | o = sp_get(cursor, o) 33 | print("O: ", o) 34 | if (o ~= nil) then 35 | local value = sp_getstring(o, "key", nil) 36 | print("Value: ", value) 37 | --printf("%"PRIu32"\n", *(uint32_t*)sp_getstring(o, "key", NULL)); 38 | end 39 | until o == nil 40 | 41 | sp_destroy(cursor); 42 | end 43 | 44 | local function main() 45 | -- Do cursor iteration. 46 | -- open or create environment and database 47 | local env = sp_env(); 48 | sp_setstring(env, config.sophia.path, "_test", 0); 49 | sp_setstring(env, "db", "test", 0); 50 | local db = sp_getobject(env, "db.test"); 51 | local rc = sp_open(env); 52 | assert(rc ~= -1, sp_strerror(env)) 53 | 54 | 55 | -- insert 10 keys 56 | local key = ffi.new("uint32_t[1]",0); 57 | while (key[0] < 10) do 58 | print("KEY: ", key[0]) 59 | local o = sp_object(db); 60 | sp_setstring(o, "key", key, ffi.sizeof("uint32_t")); 61 | rc = sp_set(db, o); 62 | assert(rc ~= -1, sp_strerror(env)) 63 | 64 | key[0] = key[0] + 1; 65 | end 66 | 67 | cursorForward(env, db); 68 | --cursorBack(env, db); 69 | 70 | -- finish work 71 | sp_destroy(env); 72 | 73 | return true; 74 | 75 | 76 | end 77 | 78 | main() 79 | 80 | -------------------------------------------------------------------------------- /extras.lua: -------------------------------------------------------------------------------- 1 | 2 | ffi.cdef[[ 3 | typedef void *(*spallocf)(void *ptr, size_t size, void *arg); 4 | typedef int (*spcmpf)(char *a, size_t asz, char *b, size_t bsz, void *arg); 5 | 6 | typedef enum { 7 | /* env related */ 8 | SPDIR, /* uint32_t, char* */ 9 | SPALLOC, /* spallocf, void* */ 10 | SPCMP, /* spcmpf, void* */ 11 | SPPAGE, /* uint32_t */ 12 | SPGC, /* int */ 13 | SPGCF, /* double */ 14 | SPGROW, /* uint32_t, double */ 15 | SPMERGE, /* int */ 16 | SPMERGEWM, /* uint32_t */ 17 | /* db related */ 18 | SPMERGEFORCE, 19 | /* unrelated */ 20 | SPVERSION /* uint32_t*, uint32_t* */ 21 | } spopt; 22 | 23 | typedef enum { 24 | SPO_RDONLY = 1, 25 | SPO_RDWR = 2, 26 | SPO_CREAT = 4, 27 | SPO_SYNC = 8 28 | } spflags; 29 | 30 | typedef enum { 31 | SPGT, 32 | SPGTE, 33 | SPLT, 34 | SPLTE 35 | } sporder; 36 | 37 | typedef struct { 38 | uint32_t epoch; 39 | uint64_t psn; 40 | uint32_t repn; 41 | uint32_t repndb; 42 | uint32_t repnxfer; 43 | uint32_t catn; 44 | uint32_t indexn; 45 | uint32_t indexpages; 46 | } spstat; 47 | ]] 48 | -------------------------------------------------------------------------------- /libc.lua: -------------------------------------------------------------------------------- 1 | --libc.lua 2 | local ffi = require("ffi") 3 | 4 | ffi.cdef[[ 5 | // string.h 6 | void *memset (void *, int, size_t); 7 | 8 | 9 | // Memory 10 | void free(void *); 11 | void * malloc(const size_t size); 12 | 13 | 14 | // Files 15 | typedef struct _IO_FILE FILE; 16 | 17 | FILE *fopen(const char *__restrict, const char *__restrict); 18 | int fclose(FILE *); 19 | 20 | int fprintf(FILE *__restrict, const char *__restrict, ...); 21 | size_t fread(void *__restrict, size_t, size_t, FILE *__restrict); 22 | size_t fwrite(const void *__restrict, size_t, size_t, FILE *__restrict); 23 | 24 | ]] 25 | 26 | local exports = { 27 | 28 | -- Memory Management 29 | free = ffi.C.free; 30 | malloc = ffi.C.malloc; 31 | memset = ffi.C.memset; 32 | 33 | -- File manipulation 34 | fopen = ffi.C.fopen; 35 | fclose = ffi.C.fclose; 36 | fprintf = ffi.C.fprintf; 37 | fwrite = ffi.C.fwrite; 38 | } 39 | 40 | setmetatable(exports, { 41 | __call = function(self, tbl) 42 | tbl = tbl or _G; 43 | for k,v in pairs(self) do 44 | tbl[k] = v; 45 | end; 46 | return self; 47 | end, 48 | }) 49 | 50 | return exports 51 | -------------------------------------------------------------------------------- /sophia.lua: -------------------------------------------------------------------------------- 1 | local ffi = require("ffi"); 2 | local bit = require("bit"); 3 | local bor = bit.bor; 4 | 5 | local Lib_sophia = require("sophia_ffi") 6 | local libc = require("libc") 7 | 8 | 9 | 10 | -- Configuration string values 11 | -- These are used in conjunction with the sp_get() function 12 | -- depending on what type of object you're operating against, 13 | -- the strings have their intended values. 14 | local config = { 15 | sophia = { 16 | version = "sophia.version"; -- string, read-only 17 | build = "sophia.build"; -- string, read-only 18 | ["error"] = "sophia.error"; -- string 19 | path = "sophia.path"; -- string, mandatory 20 | path_create = "sophia.path_create"; -- u32 21 | }; 22 | 23 | memory = { 24 | limit = "memory.limit"; -- u64 25 | used = "memory.used"; -- u64, read-only 26 | pager_pool_size = "memory.pager_pool_size"; -- u32, read-only 27 | pager_page_size = "memory.pager_page_size"; -- u32, read-only 28 | pager_pools = "memory.pager_pools"; -- u32, read-only 29 | }; 30 | 31 | -- TODO, need to flesh this out because of 32 | -- different redzones (0, 80) 33 | compaction = { 34 | node_size = "compaction.node_size"; 35 | page_size = "compaction.page_size"; 36 | redzone = { 37 | mode = "compaction.redzone.mode"; 38 | compact_wm = "compaction.0.compact_wm"; 39 | }; 40 | }; 41 | 42 | scheduler = { 43 | threads = "scheduler.threads"; -- u32 44 | -- .trace = "scheduler.%d.trace"; 45 | zone = "scheduler.zone"; -- u32, read-only 46 | checkpoint_active = "scheduler.checkpoint_active"; 47 | checkpoint_lsn = "scheduler.checkpoint_lsn"; 48 | checkpoint_lsn_last = "scheduler.checkpoint_lsn_last"; 49 | checkpoint_on_complete = "scheduler.checkpoint_on_complete"; -- function 50 | checkpoint = "scheduler.checkpoint"; -- function 51 | gc_active = "scheduler.gc_active"; -- u32, read-only 52 | gc = "scheduler.gc"; -- function 53 | run = "scheduler.run"; -- function 54 | }; 55 | 56 | metric = { 57 | dsn = "metric.dsn"; 58 | nsn = "metric.nsn"; 59 | bsn = "metric.bsn"; 60 | lsn = "metric.lsn"; 61 | lfsn = "metric.lfsn"; 62 | tsn = "metric.tsn"; 63 | }; 64 | 65 | log = { 66 | enable = "log.enable"; 67 | path = "log.path"; 68 | sync = "log.sync"; 69 | rotate_wm = "log.rotate_wm"; 70 | rotate_sync = "log.rotate_sync"; 71 | rotate = "log.rotate"; 72 | gc = "log.gc"; 73 | files = "log.files"; 74 | two_phase_recovery = "log.two_phase_recovery"; 75 | commit_lsn = "log.commit_lsn"; 76 | }; 77 | 78 | snapshot = { 79 | -- .lsn = "snapshot.%s.lsn"; 80 | }; 81 | 82 | backup = { 83 | path = "backkup.path"; -- string 84 | run = "backup.run"; -- function 85 | active = "backup.active"; -- u32, read-only 86 | last = "backup.last"; -- u32, read-only 87 | last_complete = "backup.last_complete"; -- u32, readonly 88 | on_complete = "backup.on_complete"; -- function 89 | }; 90 | 91 | -- db configuration needs to be tied to a specific name 92 | db = { 93 | name = { 94 | name = "db.%s.name"; 95 | id = "db.%s.id"; 96 | status = "db.%s.status"; 97 | 98 | }; 99 | }; 100 | } 101 | 102 | -- Error handling and meta information 103 | 104 | local function sp_strerror(env) 105 | local size = ffi.new("int[1]") 106 | local errstr = Lib_sophia.sp_getstring(env, config.sophia.error, size) 107 | if errstr == nil then 108 | return "UNKNOWN ERROR"; 109 | end 110 | 111 | errstr = ffi.string(errstr, size[0]) 112 | ffi.C.free(errstr); 113 | 114 | return errstr; 115 | end 116 | 117 | 118 | local exports = { 119 | -- reference to lib so it doesn't get 120 | -- garbage collected 121 | Lib_sophia = Lib_sophia; 122 | 123 | sp_env = Lib_sophia.sp_env; 124 | sp_object = Lib_sophia.sp_object; 125 | sp_open = Lib_sophia.sp_open; 126 | sp_drop = Lib_sophia.sp_drop; 127 | sp_destroy = Lib_sophia.sp_destroy; 128 | sp_error = Lib_sophia.sp_error; 129 | sp_asynchronous = Lib_sophia.sp_asynchronous; 130 | sp_poll = Lib_sophia.sp_poll; 131 | sp_setobject = Lib_sophia.sp_setobject; 132 | sp_setstring = Lib_sophia.sp_setstring; 133 | sp_setint = Lib_sophia.sp_setint; 134 | sp_getobject = Lib_sophia.sp_getobject; 135 | sp_getstring = Lib_sophia.sp_getstring; 136 | sp_getint = Lib_sophia.sp_getint; 137 | sp_set = Lib_sophia.sp_set; 138 | sp_update = Lib_sophia.sp_update; 139 | sp_delete = Lib_sophia.sp_delete; 140 | sp_get = Lib_sophia.sp_get; 141 | sp_cursor = Lib_sophia.sp_cursor; 142 | sp_batch = Lib_sophia.sp_batch; 143 | 144 | -- Transactions 145 | sp_begin = Lib_sophia.sp_begin; 146 | sp_prepare = Lib_sophia.sp_prepare; 147 | sp_commit = Lib_sophia.sp_commit; 148 | 149 | -- local functions 150 | sp_strerror = sp_strerror; 151 | 152 | -- Configuration strings 153 | config = config; 154 | 155 | } 156 | 157 | 158 | --[[ 159 | Make functions accessible through global namespace 160 | --]] 161 | setmetatable(exports, { 162 | __call = function(self, tbl) 163 | tbl = tbl or _G; 164 | for k,v in pairs(self) do 165 | tbl[k] = v; 166 | end 167 | 168 | return self; 169 | end, 170 | }) 171 | 172 | 173 | return exports 174 | -------------------------------------------------------------------------------- /sophia_ffi.lua: -------------------------------------------------------------------------------- 1 | local ffi = require("ffi") 2 | 3 | local Lib_sophia = ffi.load("sophia") 4 | local Lib_pthread = ffi.load("pthread", true); 5 | 6 | 7 | ffi.cdef[[ 8 | void *sp_env(void); 9 | void *sp_object(void*); 10 | int sp_open(void*); 11 | int sp_drop(void*); 12 | int sp_destroy(void*); 13 | int sp_error(void*); 14 | void *sp_asynchronous(void*); 15 | void *sp_poll(void*); 16 | int sp_setobject(void*, const char*, const void*); 17 | int sp_setstring(void*, const char*, const void*, int); 18 | int sp_setint(void*, const char*, int64_t); 19 | void *sp_getobject(void*, const char*); 20 | void *sp_getstring(void*, const char*, int*); 21 | int64_t sp_getint(void*, const char*); 22 | int sp_set(void*, void*); 23 | int sp_update(void*, void*); 24 | int sp_delete(void*, void*); 25 | void *sp_get(void*, void*); 26 | void *sp_cursor(void*); 27 | void *sp_batch(void*); 28 | void *sp_begin(void*); 29 | int sp_prepare(void*); 30 | int sp_commit(void*); 31 | ]] 32 | 33 | return Lib_sophia; 34 | 35 | -------------------------------------------------------------------------------- /test/README.md: -------------------------------------------------------------------------------- 1 | Tests 2 | ----- 3 | 4 | In order to run these tests, libsophia.so must exist somewhere on the path 5 | where luajit can find it. Placing a copy within this directory is the easiest 6 | way to do it, although if libsophia.so is installed in a place like /usr/local/lib 7 | that will work as well. 8 | 9 | 10 | -------------------------------------------------------------------------------- /test/test_common.lua: -------------------------------------------------------------------------------- 1 | package.path = package.path..";../?.lua" 2 | 3 | local ffi = require("ffi"); 4 | 5 | local sophia = require("sophia")() 6 | local libc = require("libc")() 7 | 8 | 9 | function strdup(str) 10 | local newstr = ffi.new("char[?]", #str+1); 11 | ffi.copy(newstr, ffi.cast("const char *",str), #str); 12 | newstr[#str] = 0; 13 | return newstr; 14 | end 15 | 16 | return sophia 17 | 18 | -------------------------------------------------------------------------------- /test/test_cursor.lua: -------------------------------------------------------------------------------- 1 | local ffi = require("ffi"); 2 | 3 | local sophia_ffi = require("sophia_ffi") 4 | sophia_ffi.promoteToGlobal(); 5 | local config = sophia_ffi.config; 6 | 7 | require("test_common"); 8 | 9 | local function main() 10 | local env = sp_env(); 11 | 12 | -- open or create environment and database 13 | local env = sp_env(); 14 | sp_setstring(env, config.sophia.path, "_test", 0); 15 | sp_setstring(env, "db", "test", 0); 16 | local db = sp_getobject(env, "db.test"); 17 | local rc = sp_open(env); 18 | assert(rc ~= -1, "error on sp_open"); 19 | 20 | -- insert some keys 21 | local key = ffi.new("uint32_t[1]",0); 22 | while (key[0] < 10) do 23 | local o = sp_object(db); 24 | rc = sp_set(o, "key", key, 4); 25 | 26 | print("SET KEY: ", key[0], rc); 27 | 28 | rc = sp_set(db, o); 29 | 30 | assert( rc ~= -1, "sp_set() error"); 31 | 32 | key[0] = key[0] + 1; 33 | end 34 | 35 | -- create cursor to iterate in >= order (default) 36 | local obj = sp_object(db); 37 | sp_set(obj, "order", ">="); -- >, >=, <, <= 38 | local cursor = sp_cursor(db, obj); 39 | 40 | assert(cursor ~= nil, "cursor == nil"); 41 | 42 | local o = nil; 43 | 44 | repeat 45 | o = sp_get(cursor); 46 | print("sp_get(cursor)", o); 47 | if (o ~= nil) then 48 | print(sp_get(o, "key", nil)); 49 | end 50 | until o == nil; 51 | 52 | sp_destroy(cursor); 53 | 54 | sp_destroy(env); 55 | 56 | -------------------------------------------------------------------------------- /test/test_delete.lua: -------------------------------------------------------------------------------- 1 | -- 2 | -- test_delete.lua 3 | -- 4 | local ffi = require("ffi"); 5 | 6 | local sophia_ffi = require("test_common"); 7 | local config = sophia_ffi.config; 8 | 9 | 10 | local env = sp_env(); 11 | local ctl = sp_ctl(env); 12 | 13 | sp_set(ctl, config.sophia.path, "./storage"); 14 | sp_set(ctl, "db", "test"); 15 | 16 | local rc = sp_open(env); 17 | assert(rc ~= -1, "error on sp_open"); 18 | 19 | local db = sp_get(ctl, "db.test"); 20 | assert(db ~= nil, "failed to get database test from control object"); 21 | 22 | -- insert some keys 23 | local key = "hello"; 24 | local value = "world"; 25 | 26 | -- insert 27 | -- 28 | local o = sp_object(db); 29 | sp_set(o, "key", key, ffi.cast("uint32_t", #key)); 30 | sp_set(o, "value", value, ffi.cast("uint32_t", #value)); 31 | rc = sp_set(db, o); 32 | 33 | assert(rc~= -1, "error with setting key/value on db object"); 34 | 35 | -- 36 | -- delete 37 | -- 38 | od = sp_object(db); 39 | sp_set(od, "key", key, #key); 40 | rc = sp_delete(db, od); 41 | 42 | assert(rc ~= -1, "error deleting key"); 43 | 44 | -- finish 45 | rc = sp_destroy(env); 46 | 47 | -------------------------------------------------------------------------------- /test/test_get.lua: -------------------------------------------------------------------------------- 1 | local ffi = require("ffi"); 2 | 3 | local sophia_ffi = require("test_common") 4 | local sophia = require("sophia") 5 | local config = sophia.config; 6 | 7 | local env = sp_env(); 8 | 9 | sp_setstring(env, config.sophia.path, "./storage") 10 | sp_setstring(env, "db", "test") 11 | 12 | local db = sp_getobject(env, "db.test") 13 | sp_open(db) 14 | 15 | local key = "hello"; 16 | local value = "world"; 17 | 18 | local function setValues() 19 | 20 | local o = sp_object(db); 21 | sp_setstring(o, "key", ffi.cast("char *",key), #key); 22 | sp_setstring(o, "value", ffi.cast("char *", value), #value); 23 | local rc = sp_set(db, o); 24 | 25 | assert(rc ~= -1, "sp_set() failure"); 26 | end 27 | 28 | local function getValues() 29 | -- get 30 | local o = sp_object(db); 31 | sp_set(o, "key", key, #key); 32 | local result = sp_get(db, o); 33 | 34 | if result ~= nil then 35 | local valuesize = ffi.new("uint32_t[1]"); 36 | local value = sp_get(result, "value", valuesize); 37 | print(string.format("%s", value)); 38 | sp_destroy(result); 39 | end 40 | end 41 | 42 | -- finish 43 | --rc = sp_destroy(env); 44 | 45 | 46 | setValues(); 47 | getValues(); 48 | -------------------------------------------------------------------------------- /test/test_sophia.lua: -------------------------------------------------------------------------------- 1 | package.path = package.path..";../?.lua" 2 | 3 | local sophia = require("sophia") 4 | 5 | -------------------------------------------------------------------------------- /test/test_sophiadb.lua: -------------------------------------------------------------------------------- 1 | package.path = package.path..";../?.lua" 2 | 3 | local sophia = require("sophia") 4 | 5 | local ffi = require("ffi"); 6 | 7 | ffi.cdef[[ 8 | void free(void *); 9 | ]] 10 | 11 | local function free(value) 12 | return ffi.C.free(value); 13 | end 14 | 15 | local function strdup(str) 16 | local newstr = ffi.new("char[?]", #str+1); 17 | ffi.copy(newstr, ffi.cast("char *",str), #str); 18 | newstr[#str] = 0; 19 | return newstr; 20 | end 21 | 22 | local db, err = sophia.SophiaDatabase("./db"); 23 | 24 | print("DB: ", db, err); 25 | 26 | if not db then 27 | print("Database Creation ERROR: ", err); 28 | return false, err 29 | end 30 | 31 | -- Insert some values into the database 32 | local str = "hello world" 33 | local value = strdup(str); 34 | local key = 0; 35 | 36 | while key < 10 do 37 | local success, err = db:upsert(tostring(key), str); 38 | 39 | if not success then 40 | print("db:set(), ERROR: ", key, err); 41 | end 42 | print("SET: ", key); 43 | key = key + 1; 44 | end 45 | 46 | -- Retrieve values from database using raw calls 47 | print("== GET ==") 48 | key = 0; 49 | while key < 10 do 50 | local keystr = tostring(key); 51 | local value = ffi.new("void *[1]"); 52 | local valuesize = ffi.new("size_t[1]"); 53 | local success, err = db:get(keystr, #keystr, value, valuesize); 54 | if not success then 55 | print("sp_get ERROR: ", err); 56 | break; 57 | end 58 | 59 | print(string.format("key: %d, value: %s", key, ffi.string(value[0], valuesize[0]))); 60 | ffi.C.free(value[0]); 61 | key= key + 1; 62 | end 63 | 64 | -- Retrieve values from database using cooked calls 65 | print("== RETRIEVE ==") 66 | key = 0; 67 | while key < 10 do 68 | local value, err = db:retrieve(tostring(key)) 69 | if not value then 70 | print("sp_get ERROR: ", err); 71 | break; 72 | end 73 | 74 | print(string.format("key: %d, value: %s", key, value)); 75 | key= key + 1; 76 | end 77 | 78 | 79 | -- Retrieve values using cursor 80 | print("== ITERATE ==") 81 | local ikey = tostring(5); 82 | for key, keysize, value, valuesize in db:iteration(ikey, #ikey, ffi.C.SPGT) do 83 | print(ffi.cast("int *",key)[0], ffi.string(value, valuesize)); 84 | end 85 | 86 | 87 | --------------------------------------------------------------------------------