├── .gitignore ├── LICENSE ├── README.md ├── Setup.hs ├── app └── Main.hs ├── cbits └── appendvfs.c ├── haskell-static-data-sqlite.cabal ├── shell.nix └── stack.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | dist 2 | dist-* 3 | cabal-dev 4 | *.o 5 | *.hi 6 | *.chi 7 | *.chs.h 8 | *.dyn_o 9 | *.dyn_hi 10 | .hpc 11 | .hsenv 12 | .cabal-sandbox/ 13 | cabal.sandbox.config 14 | *.prof 15 | *.aux 16 | *.hp 17 | *.eventlog 18 | .stack-work/ 19 | cabal.project.local 20 | cabal.project.local~ 21 | .HTF/ 22 | .ghc.environment.* 23 | 24 | ghcid.txt 25 | *.sublime-workspace 26 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2020 Niklas Hambüchen 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy 4 | of this software and associated documentation files (the "Software"), to deal 5 | in the Software without restriction, including without limitation the rights 6 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 7 | copies of the Software, and to permit persons to whom the Software is 8 | furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in all 11 | copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 19 | SOFTWARE. 20 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # How to append static data to an executable using sqlite 2 | 3 | Sometimes you want to ship some static data with a Haskell executable, such as: 4 | 5 | * numerical lookup tables 6 | * whitelists 7 | * example data 8 | 9 | This is especially relevant if you are building [static executables](https://github.com/nh2/static-haskell-nix) for as-self-contained-as-possible deployments, such as for Amazon Lambda, or to make life for your users easy. 10 | 11 | This example code shows how you can use SQLite's [append VFS](https://sqlite.org/src/file/ext/misc/appendvfs.c), which 12 | 13 | > allows an SQLite database to be appended onto the end of some other file, such as an executable 14 | 15 | An SQLite database can be conveniently added to your Haskell executable using the `sqlite3` command line tool with the `--append` flag. 16 | You can then open it from Haskell in read-only mode. 17 | 18 | After opening it as shown in `app/Main.hs`, you can use it like any other SQLite DB, such as with the [`sqlite-simple`](https://hackage.haskell.org/package/sqlite-simple) package. 19 | 20 | 21 | ## Usage 22 | 23 | Build the example `exe` using `stack build`. 24 | 25 | Use `sqlite3 --append` to append a DB with some sample static data to it: 26 | 27 | ```bash 28 | sqlite3 --append $(stack path --dist-dir)/build/exe/exe \ 29 | "CREATE TABLE testtable (field1 TEXT); 30 | INSERT INTO testtable (field1) VALUES ('hello'), ('world');" 31 | ``` 32 | 33 | Run `exe` to see that it successfully reads the data: 34 | 35 | ```bash 36 | $ $(stack path --dist-dir)/build/exe/exe 37 | Database contents: 38 | "hello" 39 | "world" 40 | ``` 41 | 42 | 43 | ## How the `sqlite3` executable uses it 44 | 45 | * [`#include appendvfs.c`](https://github.com/sqlite/sqlite/blob/14c98a4f4016bb60679535e3d2d9fe6c49bfe04a/src/shell.c.in#L994) 46 | * Calling [`sqlite3_appendvfs_init(0,0,0)`](https://github.com/sqlite/sqlite/blob/14c98a4f4016bb60679535e3d2d9fe6c49bfe04a/src/shell.c.in#L10542) 47 | * Documenting the [`--append FILE`](https://github.com/sqlite/sqlite/blob/14c98a4f4016bb60679535e3d2d9fe6c49bfe04a/src/shell.c.in#L3530) switch 48 | * Passing the [`apndvfs`](https://github.com/sqlite/sqlite/blob/14c98a4f4016bb60679535e3d2d9fe6c49bfe04a/src/shell.c.in#L4200-L4202) open flag to `sqlite3_open_v2()` 49 | * You can also use the older `sqlite3_open()` with `?vfs=apndvfs` if `SQLITE_USE_URI` is enabled (it is enabled by default; [docs](https://www.sqlite.org/uri.html)) 50 | 51 | 52 | ## Alternative approaches and comparison 53 | 54 | There are other ways how you can include static data into your executables: 55 | 56 | * TemplateHaskell like [`file-embed`](https://hackage.haskell.org/package/file-embed) 57 | * splices large `ByteString` `"literals"` into source code 58 | * can be slow at compile-time because the compiler has to parse it 59 | * can be not-so-fast at run-time because data structures (such as `Map`s) have to be re-parsed/deserialised from the ByteStrings (at startup) 60 | * changing the data requires recompilation 61 | * Use of TemplateHaskell can trigger [The `TH` recompilation problem](https://gist.github.com/nh2/14e653bcbdc7f40042da3755539e554a) 62 | * alternatively, you can avoid recompilation if the amont of Bytes to embed is constant, using [`dummySpace` and `injectFile`](https://hackage.haskell.org/package/file-embed-0.0.11.2/docs/Data-FileEmbed.html#g:3) 63 | * At link time using a custom assembly script 64 | * Shown in Sylvain Henry's blog post ["Fast file embedding with GHC!"](https://hsyl20.fr/home/posts/2019-01-15-fast-file-embedding-with-ghc.html) 65 | * fast at compile-time 66 | * can be not-so-fast at run-time because data structures (such as `Map`s) have to be re-parsed/deserialised from the ByteStrings (at startup) 67 | * changing the data requires re-linking 68 | * Storing serialised _compact regions_ using the above methods 69 | * Using `Data.Compact.Serialize` from [`compact`](https://hackage.haskell.org/package/compact-0.1.0.1) 70 | * fast at run-time because data structures (such as `Map`s) do not have to be re-parsed/deserialised 71 | * cannot store some data (e.g. crashes on `ByteStrings` because they are pinned memory) 72 | * very new and untested, recent bugs have been found (in 2019/2020) 73 | 74 | The approach shown here: 75 | 76 | * SQLite's `vfs=apndvfs` 77 | * fast at compile-time 78 | * fast at run-time because data structures (such as `Map`s) do not have to be re-parsed/deserialised 79 | * instead, SQL queries can be directly made 80 | * performance is that of SQLite 81 | * `sqlite3` can be used to inspect/manipulate the data (works on all platforms) 82 | * changing the data requires no recompilation or re-linking 83 | * requires linking in [`appendvfs.c`](https://sqlite.org/src/file/ext/misc/appendvfs.c) (see `.cabal` file) 84 | -------------------------------------------------------------------------------- /Setup.hs: -------------------------------------------------------------------------------- 1 | import Distribution.Simple 2 | main = defaultMain 3 | -------------------------------------------------------------------------------- /app/Main.hs: -------------------------------------------------------------------------------- 1 | {-# LANGUAGE OverloadedStrings #-} 2 | {-# LANGUAGE ForeignFunctionInterface #-} 3 | 4 | import Control.Monad (when) 5 | import Data.Foldable (for_) 6 | import Database.SQLite.Simple 7 | import Database.SQLite3.Bindings.Types (CError(..)) 8 | import qualified Data.Text as T 9 | import Foreign.C.Types (CInt(..)) 10 | import Foreign.Ptr (Ptr, nullPtr) 11 | import System.Environment (getExecutablePath) 12 | 13 | 14 | -- | No need for exact type signature, we call it as 15 | -- > sqlite3_appendvfs_init(0,0,0); 16 | -- only, just like the sqlite3 binary (`shell.c.in`) does 17 | -- (in C you can pass `0` instead of NULL for pointers). 18 | foreign import ccall "sqlite3_appendvfs_init" 19 | sqlite3_appendvfs_init :: Ptr () -> Ptr () -> Ptr () -> IO CError 20 | 21 | 22 | main :: IO () 23 | main = do 24 | -- Register the "appendvfs" extension. See: 25 | -- https://www.sqlite.org/loadext.html#persistent_loadable_extensions 26 | -- We do NOT check the returned `CError` code with `fromFFI` here because 27 | -- `decodeError` in `direct-sqlite`'s `Database.SQLite3.Bindings.Types` 28 | -- is partial and does not currently handle extended result values like 29 | -- `SQLITE_OK_LOAD_PERMANENTLY`; it will `error` with `decodeError 256`. 30 | -- https://www.sqlite.org/c3ref/extended_result_codes.html documents that 31 | -- extended result codes are off by default, but extensions like 32 | -- `appendvfs.c` do not currently follow that rule. 33 | CError result <- sqlite3_appendvfs_init nullPtr nullPtr nullPtr 34 | when (result /= 256) $ do -- 256 == SQLITE_OK_LOAD_PERMANENTLY 35 | error $ "sqlite3_appendvfs_init failed with code " ++ show result 36 | 37 | exePath <- getExecutablePath 38 | -- Open the DB appended to the executable. 39 | -- Requires that you appended a DB first using e.g.: 40 | -- sqlite3 --append path/to/exe \ 41 | -- "CREATE TABLE testtable (field1 TEXT); 42 | -- INSERT INTO testtable (field1) VALUES ('hello'), ('world');" 43 | -- Open options: 44 | -- * `apndvfs` - needs sqlite's `ext/misc/appendvfs.c` linked in 45 | -- * `ro` - read-only-mode is important because you usually cannot modify 46 | -- the contents of a running executable (certainly not on Linux). 47 | conn <- open ("file:" ++ exePath ++ "?vfs=apndvfs&mode=ro") 48 | 49 | r <- query_ conn "SELECT * from testtable" :: IO [Only T.Text] 50 | putStrLn "Database contents:" 51 | for_ r $ \(Only text) -> print text 52 | 53 | close conn 54 | -------------------------------------------------------------------------------- /cbits/appendvfs.c: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | /* 4 | ** 2017-10-20 5 | ** 6 | ** The author disclaims copyright to this source code. In place of 7 | ** a legal notice, here is a blessing: 8 | ** 9 | ** May you do good and not evil. 10 | ** May you find forgiveness for yourself and forgive others. 11 | ** May you share freely, never taking more than you give. 12 | ** 13 | ****************************************************************************** 14 | ** 15 | ** This file implements a VFS shim that allows an SQLite database to be 16 | ** appended onto the end of some other file, such as an executable. 17 | ** 18 | ** A special record must appear at the end of the file that identifies the 19 | ** file as an appended database and provides an offset to page 1. For 20 | ** best performance page 1 should be located at a disk page boundary, though 21 | ** that is not required. 22 | ** 23 | ** When opening a database using this VFS, the connection might treat 24 | ** the file as an ordinary SQLite database, or it might treat is as a 25 | ** database appended onto some other file. Here are the rules: 26 | ** 27 | ** (1) When opening a new empty file, that file is treated as an ordinary 28 | ** database. 29 | ** 30 | ** (2) When opening a file that begins with the standard SQLite prefix 31 | ** string "SQLite format 3", that file is treated as an ordinary 32 | ** database. 33 | ** 34 | ** (3) When opening a file that ends with the appendvfs trailer string 35 | ** "Start-Of-SQLite3-NNNNNNNN" that file is treated as an appended 36 | ** database. 37 | ** 38 | ** (4) If none of the above apply and the SQLITE_OPEN_CREATE flag is 39 | ** set, then a new database is appended to the already existing file. 40 | ** 41 | ** (5) Otherwise, SQLITE_CANTOPEN is returned. 42 | ** 43 | ** To avoid unnecessary complications with the PENDING_BYTE, the size of 44 | ** the file containing the database is limited to 1GB. This VFS will refuse 45 | ** to read or write past the 1GB mark. This restriction might be lifted in 46 | ** future versions. For now, if you need a large database, then keep the 47 | ** database in a separate file. 48 | ** 49 | ** If the file being opened is not an appended database, then this shim is 50 | ** a pass-through into the default underlying VFS. 51 | **/ 52 | #include "sqlite3ext.h" 53 | SQLITE_EXTENSION_INIT1 54 | #include 55 | #include 56 | 57 | /* The append mark at the end of the database is: 58 | ** 59 | ** Start-Of-SQLite3-NNNNNNNN 60 | ** 123456789 123456789 12345 61 | ** 62 | ** The NNNNNNNN represents a 64-bit big-endian unsigned integer which is 63 | ** the offset to page 1. 64 | */ 65 | #define APND_MARK_PREFIX "Start-Of-SQLite3-" 66 | #define APND_MARK_PREFIX_SZ 17 67 | #define APND_MARK_SIZE 25 68 | 69 | /* 70 | ** Maximum size of the combined prefix + database + append-mark. This 71 | ** must be less than 0x40000000 to avoid locking issues on Windows. 72 | */ 73 | #define APND_MAX_SIZE (65536*15259) 74 | 75 | /* 76 | ** Forward declaration of objects used by this utility 77 | */ 78 | typedef struct sqlite3_vfs ApndVfs; 79 | typedef struct ApndFile ApndFile; 80 | 81 | /* Access to a lower-level VFS that (might) implement dynamic loading, 82 | ** access to randomness, etc. 83 | */ 84 | #define ORIGVFS(p) ((sqlite3_vfs*)((p)->pAppData)) 85 | #define ORIGFILE(p) ((sqlite3_file*)(((ApndFile*)(p))+1)) 86 | 87 | /* An open file */ 88 | struct ApndFile { 89 | sqlite3_file base; /* IO methods */ 90 | sqlite3_int64 iPgOne; /* File offset to page 1 */ 91 | sqlite3_int64 iMark; /* Start of the append-mark */ 92 | }; 93 | 94 | /* 95 | ** Methods for ApndFile 96 | */ 97 | static int apndClose(sqlite3_file*); 98 | static int apndRead(sqlite3_file*, void*, int iAmt, sqlite3_int64 iOfst); 99 | static int apndWrite(sqlite3_file*,const void*,int iAmt, sqlite3_int64 iOfst); 100 | static int apndTruncate(sqlite3_file*, sqlite3_int64 size); 101 | static int apndSync(sqlite3_file*, int flags); 102 | static int apndFileSize(sqlite3_file*, sqlite3_int64 *pSize); 103 | static int apndLock(sqlite3_file*, int); 104 | static int apndUnlock(sqlite3_file*, int); 105 | static int apndCheckReservedLock(sqlite3_file*, int *pResOut); 106 | static int apndFileControl(sqlite3_file*, int op, void *pArg); 107 | static int apndSectorSize(sqlite3_file*); 108 | static int apndDeviceCharacteristics(sqlite3_file*); 109 | static int apndShmMap(sqlite3_file*, int iPg, int pgsz, int, void volatile**); 110 | static int apndShmLock(sqlite3_file*, int offset, int n, int flags); 111 | static void apndShmBarrier(sqlite3_file*); 112 | static int apndShmUnmap(sqlite3_file*, int deleteFlag); 113 | static int apndFetch(sqlite3_file*, sqlite3_int64 iOfst, int iAmt, void **pp); 114 | static int apndUnfetch(sqlite3_file*, sqlite3_int64 iOfst, void *p); 115 | 116 | /* 117 | ** Methods for ApndVfs 118 | */ 119 | static int apndOpen(sqlite3_vfs*, const char *, sqlite3_file*, int , int *); 120 | static int apndDelete(sqlite3_vfs*, const char *zName, int syncDir); 121 | static int apndAccess(sqlite3_vfs*, const char *zName, int flags, int *); 122 | static int apndFullPathname(sqlite3_vfs*, const char *zName, int, char *zOut); 123 | static void *apndDlOpen(sqlite3_vfs*, const char *zFilename); 124 | static void apndDlError(sqlite3_vfs*, int nByte, char *zErrMsg); 125 | static void (*apndDlSym(sqlite3_vfs *pVfs, void *p, const char*zSym))(void); 126 | static void apndDlClose(sqlite3_vfs*, void*); 127 | static int apndRandomness(sqlite3_vfs*, int nByte, char *zOut); 128 | static int apndSleep(sqlite3_vfs*, int microseconds); 129 | static int apndCurrentTime(sqlite3_vfs*, double*); 130 | static int apndGetLastError(sqlite3_vfs*, int, char *); 131 | static int apndCurrentTimeInt64(sqlite3_vfs*, sqlite3_int64*); 132 | static int apndSetSystemCall(sqlite3_vfs*, const char*,sqlite3_syscall_ptr); 133 | static sqlite3_syscall_ptr apndGetSystemCall(sqlite3_vfs*, const char *z); 134 | static const char *apndNextSystemCall(sqlite3_vfs*, const char *zName); 135 | 136 | static sqlite3_vfs apnd_vfs = { 137 | 3, /* iVersion (set when registered) */ 138 | 0, /* szOsFile (set when registered) */ 139 | 1024, /* mxPathname */ 140 | 0, /* pNext */ 141 | "apndvfs", /* zName */ 142 | 0, /* pAppData (set when registered) */ 143 | apndOpen, /* xOpen */ 144 | apndDelete, /* xDelete */ 145 | apndAccess, /* xAccess */ 146 | apndFullPathname, /* xFullPathname */ 147 | apndDlOpen, /* xDlOpen */ 148 | apndDlError, /* xDlError */ 149 | apndDlSym, /* xDlSym */ 150 | apndDlClose, /* xDlClose */ 151 | apndRandomness, /* xRandomness */ 152 | apndSleep, /* xSleep */ 153 | apndCurrentTime, /* xCurrentTime */ 154 | apndGetLastError, /* xGetLastError */ 155 | apndCurrentTimeInt64, /* xCurrentTimeInt64 */ 156 | apndSetSystemCall, /* xSetSystemCall */ 157 | apndGetSystemCall, /* xGetSystemCall */ 158 | apndNextSystemCall /* xNextSystemCall */ 159 | }; 160 | 161 | static const sqlite3_io_methods apnd_io_methods = { 162 | 3, /* iVersion */ 163 | apndClose, /* xClose */ 164 | apndRead, /* xRead */ 165 | apndWrite, /* xWrite */ 166 | apndTruncate, /* xTruncate */ 167 | apndSync, /* xSync */ 168 | apndFileSize, /* xFileSize */ 169 | apndLock, /* xLock */ 170 | apndUnlock, /* xUnlock */ 171 | apndCheckReservedLock, /* xCheckReservedLock */ 172 | apndFileControl, /* xFileControl */ 173 | apndSectorSize, /* xSectorSize */ 174 | apndDeviceCharacteristics, /* xDeviceCharacteristics */ 175 | apndShmMap, /* xShmMap */ 176 | apndShmLock, /* xShmLock */ 177 | apndShmBarrier, /* xShmBarrier */ 178 | apndShmUnmap, /* xShmUnmap */ 179 | apndFetch, /* xFetch */ 180 | apndUnfetch /* xUnfetch */ 181 | }; 182 | 183 | 184 | 185 | /* 186 | ** Close an apnd-file. 187 | */ 188 | static int apndClose(sqlite3_file *pFile){ 189 | pFile = ORIGFILE(pFile); 190 | return pFile->pMethods->xClose(pFile); 191 | } 192 | 193 | /* 194 | ** Read data from an apnd-file. 195 | */ 196 | static int apndRead( 197 | sqlite3_file *pFile, 198 | void *zBuf, 199 | int iAmt, 200 | sqlite_int64 iOfst 201 | ){ 202 | ApndFile *p = (ApndFile *)pFile; 203 | pFile = ORIGFILE(pFile); 204 | return pFile->pMethods->xRead(pFile, zBuf, iAmt, iOfst+p->iPgOne); 205 | } 206 | 207 | /* 208 | ** Add the append-mark onto the end of the file. 209 | */ 210 | static int apndWriteMark(ApndFile *p, sqlite3_file *pFile){ 211 | int i; 212 | unsigned char a[APND_MARK_SIZE]; 213 | memcpy(a, APND_MARK_PREFIX, APND_MARK_PREFIX_SZ); 214 | for(i=0; i<8; i++){ 215 | a[APND_MARK_PREFIX_SZ+i] = (p->iPgOne >> (56 - i*8)) & 0xff; 216 | } 217 | return pFile->pMethods->xWrite(pFile, a, APND_MARK_SIZE, p->iMark); 218 | } 219 | 220 | /* 221 | ** Write data to an apnd-file. 222 | */ 223 | static int apndWrite( 224 | sqlite3_file *pFile, 225 | const void *zBuf, 226 | int iAmt, 227 | sqlite_int64 iOfst 228 | ){ 229 | int rc; 230 | ApndFile *p = (ApndFile *)pFile; 231 | pFile = ORIGFILE(pFile); 232 | if( iOfst+iAmt>=APND_MAX_SIZE ) return SQLITE_FULL; 233 | rc = pFile->pMethods->xWrite(pFile, zBuf, iAmt, iOfst+p->iPgOne); 234 | if( rc==SQLITE_OK && iOfst + iAmt + p->iPgOne > p->iMark ){ 235 | sqlite3_int64 sz = 0; 236 | rc = pFile->pMethods->xFileSize(pFile, &sz); 237 | if( rc==SQLITE_OK ){ 238 | p->iMark = sz - APND_MARK_SIZE; 239 | if( iOfst + iAmt + p->iPgOne > p->iMark ){ 240 | p->iMark = p->iPgOne + iOfst + iAmt; 241 | rc = apndWriteMark(p, pFile); 242 | } 243 | } 244 | } 245 | return rc; 246 | } 247 | 248 | /* 249 | ** Truncate an apnd-file. 250 | */ 251 | static int apndTruncate(sqlite3_file *pFile, sqlite_int64 size){ 252 | int rc; 253 | ApndFile *p = (ApndFile *)pFile; 254 | pFile = ORIGFILE(pFile); 255 | rc = pFile->pMethods->xTruncate(pFile, size+p->iPgOne+APND_MARK_SIZE); 256 | if( rc==SQLITE_OK ){ 257 | p->iMark = p->iPgOne+size; 258 | rc = apndWriteMark(p, pFile); 259 | } 260 | return rc; 261 | } 262 | 263 | /* 264 | ** Sync an apnd-file. 265 | */ 266 | static int apndSync(sqlite3_file *pFile, int flags){ 267 | pFile = ORIGFILE(pFile); 268 | return pFile->pMethods->xSync(pFile, flags); 269 | } 270 | 271 | /* 272 | ** Return the current file-size of an apnd-file. 273 | */ 274 | static int apndFileSize(sqlite3_file *pFile, sqlite_int64 *pSize){ 275 | ApndFile *p = (ApndFile *)pFile; 276 | int rc; 277 | pFile = ORIGFILE(p); 278 | rc = pFile->pMethods->xFileSize(pFile, pSize); 279 | if( rc==SQLITE_OK && p->iPgOne ){ 280 | *pSize -= p->iPgOne + APND_MARK_SIZE; 281 | } 282 | return rc; 283 | } 284 | 285 | /* 286 | ** Lock an apnd-file. 287 | */ 288 | static int apndLock(sqlite3_file *pFile, int eLock){ 289 | pFile = ORIGFILE(pFile); 290 | return pFile->pMethods->xLock(pFile, eLock); 291 | } 292 | 293 | /* 294 | ** Unlock an apnd-file. 295 | */ 296 | static int apndUnlock(sqlite3_file *pFile, int eLock){ 297 | pFile = ORIGFILE(pFile); 298 | return pFile->pMethods->xUnlock(pFile, eLock); 299 | } 300 | 301 | /* 302 | ** Check if another file-handle holds a RESERVED lock on an apnd-file. 303 | */ 304 | static int apndCheckReservedLock(sqlite3_file *pFile, int *pResOut){ 305 | pFile = ORIGFILE(pFile); 306 | return pFile->pMethods->xCheckReservedLock(pFile, pResOut); 307 | } 308 | 309 | /* 310 | ** File control method. For custom operations on an apnd-file. 311 | */ 312 | static int apndFileControl(sqlite3_file *pFile, int op, void *pArg){ 313 | ApndFile *p = (ApndFile *)pFile; 314 | int rc; 315 | pFile = ORIGFILE(pFile); 316 | rc = pFile->pMethods->xFileControl(pFile, op, pArg); 317 | if( rc==SQLITE_OK && op==SQLITE_FCNTL_VFSNAME ){ 318 | *(char**)pArg = sqlite3_mprintf("apnd(%lld)/%z", p->iPgOne, *(char**)pArg); 319 | } 320 | return rc; 321 | } 322 | 323 | /* 324 | ** Return the sector-size in bytes for an apnd-file. 325 | */ 326 | static int apndSectorSize(sqlite3_file *pFile){ 327 | pFile = ORIGFILE(pFile); 328 | return pFile->pMethods->xSectorSize(pFile); 329 | } 330 | 331 | /* 332 | ** Return the device characteristic flags supported by an apnd-file. 333 | */ 334 | static int apndDeviceCharacteristics(sqlite3_file *pFile){ 335 | pFile = ORIGFILE(pFile); 336 | return pFile->pMethods->xDeviceCharacteristics(pFile); 337 | } 338 | 339 | /* Create a shared memory file mapping */ 340 | static int apndShmMap( 341 | sqlite3_file *pFile, 342 | int iPg, 343 | int pgsz, 344 | int bExtend, 345 | void volatile **pp 346 | ){ 347 | pFile = ORIGFILE(pFile); 348 | return pFile->pMethods->xShmMap(pFile,iPg,pgsz,bExtend,pp); 349 | } 350 | 351 | /* Perform locking on a shared-memory segment */ 352 | static int apndShmLock(sqlite3_file *pFile, int offset, int n, int flags){ 353 | pFile = ORIGFILE(pFile); 354 | return pFile->pMethods->xShmLock(pFile,offset,n,flags); 355 | } 356 | 357 | /* Memory barrier operation on shared memory */ 358 | static void apndShmBarrier(sqlite3_file *pFile){ 359 | pFile = ORIGFILE(pFile); 360 | pFile->pMethods->xShmBarrier(pFile); 361 | } 362 | 363 | /* Unmap a shared memory segment */ 364 | static int apndShmUnmap(sqlite3_file *pFile, int deleteFlag){ 365 | pFile = ORIGFILE(pFile); 366 | return pFile->pMethods->xShmUnmap(pFile,deleteFlag); 367 | } 368 | 369 | /* Fetch a page of a memory-mapped file */ 370 | static int apndFetch( 371 | sqlite3_file *pFile, 372 | sqlite3_int64 iOfst, 373 | int iAmt, 374 | void **pp 375 | ){ 376 | ApndFile *p = (ApndFile *)pFile; 377 | pFile = ORIGFILE(pFile); 378 | return pFile->pMethods->xFetch(pFile, iOfst+p->iPgOne, iAmt, pp); 379 | } 380 | 381 | /* Release a memory-mapped page */ 382 | static int apndUnfetch(sqlite3_file *pFile, sqlite3_int64 iOfst, void *pPage){ 383 | ApndFile *p = (ApndFile *)pFile; 384 | pFile = ORIGFILE(pFile); 385 | return pFile->pMethods->xUnfetch(pFile, iOfst+p->iPgOne, pPage); 386 | } 387 | 388 | /* 389 | ** Check to see if the file is an ordinary SQLite database file. 390 | */ 391 | static int apndIsOrdinaryDatabaseFile(sqlite3_int64 sz, sqlite3_file *pFile){ 392 | int rc; 393 | char zHdr[16]; 394 | static const char aSqliteHdr[] = "SQLite format 3"; 395 | if( sz<512 ) return 0; 396 | rc = pFile->pMethods->xRead(pFile, zHdr, sizeof(zHdr), 0); 397 | if( rc ) return 0; 398 | return memcmp(zHdr, aSqliteHdr, sizeof(zHdr))==0; 399 | } 400 | 401 | /* 402 | ** Try to read the append-mark off the end of a file. Return the 403 | ** start of the appended database if the append-mark is present. If 404 | ** there is no append-mark, return -1; 405 | */ 406 | static sqlite3_int64 apndReadMark(sqlite3_int64 sz, sqlite3_file *pFile){ 407 | int rc, i; 408 | sqlite3_int64 iMark; 409 | unsigned char a[APND_MARK_SIZE]; 410 | 411 | if( sz<=APND_MARK_SIZE ) return -1; 412 | rc = pFile->pMethods->xRead(pFile, a, APND_MARK_SIZE, sz-APND_MARK_SIZE); 413 | if( rc ) return -1; 414 | if( memcmp(a, APND_MARK_PREFIX, APND_MARK_PREFIX_SZ)!=0 ) return -1; 415 | iMark = ((sqlite3_int64)(a[APND_MARK_PREFIX_SZ]&0x7f))<<56; 416 | for(i=1; i<8; i++){ 417 | iMark += (sqlite3_int64)a[APND_MARK_PREFIX_SZ+i]<<(56-8*i); 418 | } 419 | return iMark; 420 | } 421 | 422 | /* 423 | ** Open an apnd file handle. 424 | */ 425 | static int apndOpen( 426 | sqlite3_vfs *pVfs, 427 | const char *zName, 428 | sqlite3_file *pFile, 429 | int flags, 430 | int *pOutFlags 431 | ){ 432 | ApndFile *p; 433 | sqlite3_file *pSubFile; 434 | sqlite3_vfs *pSubVfs; 435 | int rc; 436 | sqlite3_int64 sz; 437 | pSubVfs = ORIGVFS(pVfs); 438 | if( (flags & SQLITE_OPEN_MAIN_DB)==0 ){ 439 | return pSubVfs->xOpen(pSubVfs, zName, pFile, flags, pOutFlags); 440 | } 441 | p = (ApndFile*)pFile; 442 | memset(p, 0, sizeof(*p)); 443 | pSubFile = ORIGFILE(pFile); 444 | p->base.pMethods = &apnd_io_methods; 445 | rc = pSubVfs->xOpen(pSubVfs, zName, pSubFile, flags, pOutFlags); 446 | if( rc ) goto apnd_open_done; 447 | rc = pSubFile->pMethods->xFileSize(pSubFile, &sz); 448 | if( rc ){ 449 | pSubFile->pMethods->xClose(pSubFile); 450 | goto apnd_open_done; 451 | } 452 | if( apndIsOrdinaryDatabaseFile(sz, pSubFile) ){ 453 | memmove(pFile, pSubFile, pSubVfs->szOsFile); 454 | return SQLITE_OK; 455 | } 456 | p->iMark = 0; 457 | p->iPgOne = apndReadMark(sz, pFile); 458 | if( p->iPgOne>0 ){ 459 | return SQLITE_OK; 460 | } 461 | if( (flags & SQLITE_OPEN_CREATE)==0 ){ 462 | pSubFile->pMethods->xClose(pSubFile); 463 | rc = SQLITE_CANTOPEN; 464 | } 465 | p->iPgOne = (sz+0xfff) & ~(sqlite3_int64)0xfff; 466 | apnd_open_done: 467 | if( rc ) pFile->pMethods = 0; 468 | return rc; 469 | } 470 | 471 | /* 472 | ** All other VFS methods are pass-thrus. 473 | */ 474 | static int apndDelete(sqlite3_vfs *pVfs, const char *zPath, int dirSync){ 475 | return ORIGVFS(pVfs)->xDelete(ORIGVFS(pVfs), zPath, dirSync); 476 | } 477 | static int apndAccess( 478 | sqlite3_vfs *pVfs, 479 | const char *zPath, 480 | int flags, 481 | int *pResOut 482 | ){ 483 | return ORIGVFS(pVfs)->xAccess(ORIGVFS(pVfs), zPath, flags, pResOut); 484 | } 485 | static int apndFullPathname( 486 | sqlite3_vfs *pVfs, 487 | const char *zPath, 488 | int nOut, 489 | char *zOut 490 | ){ 491 | return ORIGVFS(pVfs)->xFullPathname(ORIGVFS(pVfs),zPath,nOut,zOut); 492 | } 493 | static void *apndDlOpen(sqlite3_vfs *pVfs, const char *zPath){ 494 | return ORIGVFS(pVfs)->xDlOpen(ORIGVFS(pVfs), zPath); 495 | } 496 | static void apndDlError(sqlite3_vfs *pVfs, int nByte, char *zErrMsg){ 497 | ORIGVFS(pVfs)->xDlError(ORIGVFS(pVfs), nByte, zErrMsg); 498 | } 499 | static void (*apndDlSym(sqlite3_vfs *pVfs, void *p, const char *zSym))(void){ 500 | return ORIGVFS(pVfs)->xDlSym(ORIGVFS(pVfs), p, zSym); 501 | } 502 | static void apndDlClose(sqlite3_vfs *pVfs, void *pHandle){ 503 | ORIGVFS(pVfs)->xDlClose(ORIGVFS(pVfs), pHandle); 504 | } 505 | static int apndRandomness(sqlite3_vfs *pVfs, int nByte, char *zBufOut){ 506 | return ORIGVFS(pVfs)->xRandomness(ORIGVFS(pVfs), nByte, zBufOut); 507 | } 508 | static int apndSleep(sqlite3_vfs *pVfs, int nMicro){ 509 | return ORIGVFS(pVfs)->xSleep(ORIGVFS(pVfs), nMicro); 510 | } 511 | static int apndCurrentTime(sqlite3_vfs *pVfs, double *pTimeOut){ 512 | return ORIGVFS(pVfs)->xCurrentTime(ORIGVFS(pVfs), pTimeOut); 513 | } 514 | static int apndGetLastError(sqlite3_vfs *pVfs, int a, char *b){ 515 | return ORIGVFS(pVfs)->xGetLastError(ORIGVFS(pVfs), a, b); 516 | } 517 | static int apndCurrentTimeInt64(sqlite3_vfs *pVfs, sqlite3_int64 *p){ 518 | return ORIGVFS(pVfs)->xCurrentTimeInt64(ORIGVFS(pVfs), p); 519 | } 520 | static int apndSetSystemCall( 521 | sqlite3_vfs *pVfs, 522 | const char *zName, 523 | sqlite3_syscall_ptr pCall 524 | ){ 525 | return ORIGVFS(pVfs)->xSetSystemCall(ORIGVFS(pVfs),zName,pCall); 526 | } 527 | static sqlite3_syscall_ptr apndGetSystemCall( 528 | sqlite3_vfs *pVfs, 529 | const char *zName 530 | ){ 531 | return ORIGVFS(pVfs)->xGetSystemCall(ORIGVFS(pVfs),zName); 532 | } 533 | static const char *apndNextSystemCall(sqlite3_vfs *pVfs, const char *zName){ 534 | return ORIGVFS(pVfs)->xNextSystemCall(ORIGVFS(pVfs), zName); 535 | } 536 | 537 | 538 | #ifdef _WIN32 539 | __declspec(dllexport) 540 | #endif 541 | /* 542 | ** This routine is called when the extension is loaded. 543 | ** Register the new VFS. 544 | */ 545 | int sqlite3_appendvfs_init( 546 | sqlite3 *db, 547 | char **pzErrMsg, 548 | const sqlite3_api_routines *pApi 549 | ){ 550 | int rc = SQLITE_OK; 551 | sqlite3_vfs *pOrig; 552 | SQLITE_EXTENSION_INIT2(pApi); 553 | (void)pzErrMsg; 554 | (void)db; 555 | pOrig = sqlite3_vfs_find(0); 556 | apnd_vfs.iVersion = pOrig->iVersion; 557 | apnd_vfs.pAppData = pOrig; 558 | apnd_vfs.szOsFile = pOrig->szOsFile + sizeof(ApndFile); 559 | rc = sqlite3_vfs_register(&apnd_vfs, 0); 560 | #ifdef APPENDVFS_TEST 561 | if( rc==SQLITE_OK ){ 562 | rc = sqlite3_auto_extension((void(*)(void))apndvfsRegister); 563 | } 564 | #endif 565 | if( rc==SQLITE_OK ) rc = SQLITE_OK_LOAD_PERMANENTLY; 566 | return rc; 567 | } 568 | -------------------------------------------------------------------------------- /haskell-static-data-sqlite.cabal: -------------------------------------------------------------------------------- 1 | name: haskell-static-data-sqlite 2 | version: 0.1.0.0 3 | synopsis: Example how to append data to an executable using sqlite 4 | description: Example how to append data to an executable using sqlite 5 | homepage: https://github.com/nh2/haskell-static-data-sqlite#readme 6 | license: MIT 7 | license-file: LICENSE 8 | author: Niklas Hambüchen 9 | maintainer: Niklas Hambüchen 10 | copyright: 2020 Niklas Hambüchen 11 | category: System 12 | build-type: Simple 13 | cabal-version: >=1.10 14 | 15 | extra-source-files: 16 | README.md 17 | stack.yaml 18 | 19 | executable exe 20 | default-language: Haskell2010 21 | hs-source-dirs: app 22 | main-is: Main.hs 23 | c-sources: 24 | -- Copied from https://sqlite.org/src/file/ext/misc/appendvfs.c 25 | cbits/appendvfs.c 26 | cc-options: 27 | -- We need to set either `-DSQLITE_OMIT_LOAD_EXTENSION` or `-DSQLITE_CORE` 28 | -- (see `sqlite3ext.h`; it is not clear yet which one is more correct 29 | -- or whether it is intended by sqlite authors that these are needed) 30 | -- when compiling `appendvfs.c` separate from the `sqlite.c` amalgamation 31 | -- to ensure `sqlite3_vfs_find()` does not segfault at first call. 32 | -DSQLITE_OMIT_LOAD_EXTENSION 33 | -- For better debugging with gdb 34 | -g 35 | build-depends: 36 | base 37 | , direct-sqlite >= 2.3.24 38 | , sqlite-simple >= 0.4.16.0 39 | , text 40 | ghc-options: 41 | -Wall -threaded -rtsopts -with-rtsopts=-N 42 | 43 | source-repository head 44 | type: git 45 | location: https://github.com/nh2/haskell-static-data-sqlite 46 | -------------------------------------------------------------------------------- /shell.nix: -------------------------------------------------------------------------------- 1 | with (import {}); 2 | let 3 | # Needs NUR from https://github.com/nix-community/NUR 4 | ghc = nur.repos.mpickering.ghc.ghc865; # Keep in sync with the GHC version defined by stack.yaml! 5 | in 6 | haskell.lib.buildStackProject { 7 | inherit ghc; 8 | name = "myEnv"; 9 | # System dependencies used at build-time go in here. 10 | nativeBuildInputs = [ 11 | ]; 12 | # System dependencies used at run-time go in here. 13 | buildInputs = [ 14 | ]; 15 | } 16 | -------------------------------------------------------------------------------- /stack.yaml: -------------------------------------------------------------------------------- 1 | # When changing the resolver, also update shell.nix accordingly! 2 | resolver: lts-14.27 3 | packages: 4 | - . 5 | extra-deps: [] 6 | 7 | nix: 8 | shell-file: shell.nix 9 | --------------------------------------------------------------------------------