├── .gitignore ├── LICENSE ├── README.md ├── cevfs.xcworkspace └── contents.xcworkspacedata ├── cevfs ├── cevfs.c ├── cevfs.h ├── cevfs.xcodeproj │ └── project.pbxproj └── sqlite │ └── readme.md ├── cevfs_build ├── cevfs_build.c ├── cevfs_build.xcodeproj │ └── project.pbxproj ├── cevfs_mod.c └── xMethods.c └── cevfs_example ├── cevfs_example.xcodeproj └── project.pbxproj └── main.c /.gitignore: -------------------------------------------------------------------------------- 1 | # SQLite source files 2 | /cevfs/sqlite/* 3 | !readme.md 4 | 5 | # OSX 6 | # 7 | .DS_Store 8 | 9 | # Xcode 10 | # 11 | build/ 12 | *.pbxuser 13 | !default.pbxuser 14 | *.mode1v3 15 | !default.mode1v3 16 | *.mode2v3 17 | !default.mode2v3 18 | *.perspectivev3 19 | !default.perspectivev3 20 | xcuserdata 21 | *.xccheckout 22 | *.moved-aside 23 | DerivedData 24 | *.hmap 25 | *.ipa 26 | *.xcuserstate 27 | project.xcworkspace 28 | 29 | # Android/IJ 30 | # 31 | .idea 32 | .gradle 33 | local.properties 34 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 Ryan Homer, Murage Inc. 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # CEVFS: Compression & Encryption VFS for SQLite 3 2 | CEVFS is a SQLite 3 Virtual File System for compressing and encrypting data at the pager level. Once set up, you use SQLite as you normally would and the compression and encryption is transparently handled during database read/write operations via the SQLite pager. 3 | 4 | ## Introduction 5 | CEVFS is an open source SQLite extension that combines some of the functionality of [CEROD](http://www.sqlite.org/cerod/doc/trunk/www/index.wiki) and [ZIPVFS](http://www.sqlite.org/zipvfs/doc/trunk/www/index.wiki) into one package. Even though no testing has been done yet beyond macOS and iOS, the goal is to make it as portable as SQLite itself -- with the help of the community. 6 | 7 | CEVFS gives you convenient hooks into SQLite that allow you to easily implement your own compression and encryption functions. As different operating systems come preloaded with different compression and encryption libraries, no default compression or encryption functions are supplied with CEVFS. The **cevfs_example** project uses Zlib and CCCryptor (3cc), both of which are included in macOS and iOS. 8 | 9 | ## How It Works 10 | CEVFS is a [SQLite Virtual File System](http://www.sqlite.org/vfs.html) which uses its own pager and is inserted between the pager used by the b-tree (herein referred to as the upper pager) and the OS interface. This allows it to intercept the read/write operations to the database and seamlessly compress/decompress and encrypt/decrypt the data. See "[How ZIPVFS Works](http://www.sqlite.org/zipvfs/doc/trunk/www/howitworks.wiki)" for more details. Unlike ZIPVFS, the page size of the pager used by CEVFS (herein referred to as the lower pager) is determined when the CEVFS database is created and is a persistent property of the database. WAL mode is not yet supported. 11 | 12 | ## How To Build 13 | 14 | #### Get the SQLite Source Code 15 | 1. Go to the [Downloads page](http://www.sqlite.org/download.html). 16 | 1. Select one of the geographically located sites ([Dallas TX](http://www.sqlite.org/cgi/src), [Newark NJ](http://www2.sqlite.org/cgi/src), [Fremont CA](http://www3.sqlite.org/cgi/src)) from the Source Code Repositories section at the bottom of the page. 17 | 1. Select the **Tags** tab at the top of the page. 18 | 1. Select the SQLite version you're interested in. 19 | 1. Select the Fossil SHA1 (10-char hex string) commit link. 20 | 1. Finally, download the tarball or ZIP archive. 21 | 22 | _For easy reference, it will be assumed that the SQLite source code is in the `sqlite/` directory._ 23 | 24 | #### Create the Amalgamation file. 25 | 1. Decompress the archive and `cd` to the root of the unarchived directory. 26 | 1. `./configure` 27 | 1. `make sqlite3.c` 28 | 29 | #### Build a Static Library 30 | 1. Create a temporary `build` directory and `cd` to it. 31 | 1. Copy `sqlite3.c`, `cevfs.c` and `cevfs.h` to the `build` directory. 32 | 1. Combine the files: `cat sqlite3.c cevfs.c > cevfs-all.c` 33 | 1. Compile: `clang -c cevfs-all.c -o sqlite3.o -Os` 34 | 1. Create static lib: `libtool -static sqlite3.o -o sqlite3.a` 35 | 36 | ### Creating a Command-Line Build Tool 37 | If you are using macOS, you can use the `cevfs_build` example which implements compression using Zlib and encryption using 3cc. Otherwise, modify the _xFunctions_ first to accommodate your operating system. 38 | 39 | Copy the following files to your temporary build directory: 40 | - sqlite/sqlite3.c 41 | - cevfs/cevfs.c 42 | - cevfs\_build/cevfs\_build.c 43 | - cevfs_build/xMethods.c 44 | 45 | Build: 46 | ``` 47 | build> $ cat sqlite3.c cevfs.c > cevfs-all.c 48 | build> $ clang cevfs-all.c cevfs_build.c -O2 -o cevfs_build -lz 49 | ``` 50 | 51 | Then to create a CEVFS database: 52 | ``` 53 | ./cevfs_build UNCOMPRESSED COMPRESSED VFS_NAME KEY 54 | ``` 55 | 56 | parameters: 57 | - **UNCOMPRESSED**: path to uncompressed database 58 | - **COMPRESSED**: path to new compressed database 59 | - **VFS_NAME**: name to embed in header (10 chars. max.) 60 | - **KEY**: encryption key 61 | 62 | E.g.: 63 | 64 | ``` 65 | ./cevfs_build myDatabase.db myNewDatabase.db default "x'2F3A995FCE317EA22F3A995FCE317EA22F3A995FCE317EA22F3A995FCE317EA2'" 66 | ``` 67 | 68 | (hex key is 32 pairs of 2-digit hex values) 69 | 70 | You can also try different block sizes and compare the sizes of the new databases to see which one uses less space. To specify the block size, specify the destination path using a URI and append `?block_size=`: 71 | 72 | ``` 73 | ./cevfs_build myDatabase.db "file:///absolute/path/to/myNewDatabase.db?block_size=4096" default "x'2F3A995FCE317EA2...'" 74 | ``` 75 | 76 | ### Creating a Custom Version of SQLite 77 | It is helpful to have a custom command-line version of `sqlite3` on your development workstation for opening/testing your newly created databases. 78 | 79 | Copy the following files to your temporary `build` directory. 80 | - sqlite3.c (from SQLite source) 81 | - shell.c (from SQLite source) 82 | - cevfs/cevfs.c 83 | - cevfs_build/cevfs_mod.c 84 | - cevfs_build/xMethods.c 85 | 86 | Again, modify your _xFunctions_ to accommodate your operating system. You may also need to install the Readline lib. 87 | 88 | Build: 89 | ``` 90 | build> $ cat sqlite3.c cevfs.c cevfs_mod.c > cevfs-all.c 91 | build> $ clang cevfs-all.c shell.c -DSQLITE_ENABLE_CEROD=1 -DHAVE_READLINE=1 -O2 -o sqlite3 -lz -lreadline 92 | ``` 93 | 94 | _If you get errors related to implicit declaration of functions under C99, you can add `-Wno-implicit-function-declaration` to disable them._ 95 | 96 | Then, to open a CEVFS database: 97 | 98 | ``` 99 | $ ./sqlite3 100 | sqlite> PRAGMA activate_extensions("cerod-x''"); 101 | sqlite> .open path/to/your/cevfs/db 102 | ``` 103 | 104 | By specifying `SQLITE_ENABLE_CEROD` we can make use of an API hook that's built into SQLite for the CEROD extension that will allow you to conveniently activate CEVFS. It has the following signature: 105 | 106 | ``` 107 | SQLITE_API void SQLITE_STDCALL sqlite3_activate_cerod(const char *zPassPhrase); 108 | ``` 109 | 110 | If you are using CEVFS, chances are that you are _not_ currently making use of this API hook. You can use the `const char *` param to pass something other than the intended activation key, such as the encryption key. This `sqlite3_activate_cerod` function has been implemented in `cevfs_build/cevfs_mod.c` as an example. Alternatively, you can roll out your own [Run-Time Loadable Extension](http://www.sqlite.org/loadext.html) for use with a standard SQLite 3 build. 111 | 112 | ## Limitations 113 | 114 | - WAL mode is not (yet) supported. 115 | - Free nodes are not managed which could result in wasted space. Not an issue if the database you create with `cevfs_build()` is intended to be used as a read-only database. 116 | - VACUUM not yet implemented to recover lost space. 117 | 118 | ## SQLite3 Compatible Versions 119 | 120 | |Version|Combatibility| 121 | |-|-| 122 | |3.10.2|Most development was originally with this version.| 123 | |3.11.x|Testing OK. No changes were required.| 124 | |3.12.0 - 3.15.2|Default pager page size changed. CEVFS updated for these versions.| 125 | |3.16.0 - 3.34.0|`sqlite3PagerClose` API signature changed: CEVFS updated for these versions.| 126 | 127 | ## Contributing to the project 128 | If you would like to contribute back to the project, please fork the repo and submit pull requests. For more information, please read this [wiki page](https://github.com/ryanhomer/sqlite3-compression-encryption-vfs/wiki/Developing-in-Xcode) 129 | 130 | Here are some things that need to be implemented: 131 | - Proper mutex & multi-threading support 132 | - TCL unit tests 133 | - WAL support 134 | - Full text indexing support 135 | - Keep track of free space lost when data is moved to a new page and reuse free space during subsequent write operations 136 | - Implement database compacting (using 'vacuum' keyword if possible) 137 | -------------------------------------------------------------------------------- /cevfs.xcworkspace/contents.xcworkspacedata: -------------------------------------------------------------------------------- 1 | 2 | 4 | 6 | 7 | 9 | 10 | 12 | 13 | 14 | -------------------------------------------------------------------------------- /cevfs/cevfs.c: -------------------------------------------------------------------------------- 1 | /** 2 | CEVFS 3 | Compression & Encryption VFS 4 | 5 | Copyright (c) 2016 Ryan Homer, Murage Inc. 6 | 7 | MIT License 8 | 9 | Permission is hereby granted, free of charge, to any person obtaining a copy 10 | of this software and associated documentation files (the "Software"), to deal 11 | in the Software without restriction, including without limitation the rights 12 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 13 | copies of the Software, and to permit persons to whom the Software is 14 | furnished to do so, subject to the following conditions: 15 | 16 | The above copyright notice and this permission notice shall be included in all 17 | copies or substantial portions of the Software. 18 | 19 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 20 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 21 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 22 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 23 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 24 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 25 | SOFTWARE. 26 | */ 27 | 28 | #ifndef SQLITE_AMALGAMATION 29 | #include 30 | #include 31 | #include "sqlite3.h" 32 | #include "sqliteInt.h" 33 | #include "pager.h" 34 | #include "btreeInt.h" 35 | #endif 36 | #include 37 | #include "cevfs.h" 38 | 39 | // Standard Sqlite3 pager header 40 | #define CEVFS_DB_HEADER1_OFST 000 41 | #define CEVFS_DB_HEADER1_SZ 100 42 | 43 | // cevfs-specific pager header 44 | #define CEVFS_DB_HEADER2_OFST CEVFS_DB_HEADER1_OFST+CEVFS_DB_HEADER1_SZ 45 | #define CEVFS_DB_HEADER2_SZ 100 46 | 47 | // Total header size 48 | #define CEVFS_DB_HEADER_SIZE (CEVFS_DB_HEADER1_SZ + CEVFS_DB_HEADER2_SZ) 49 | 50 | // Offset to master map table, starts just after header 51 | #define CEVFS_DB_MMTBL_OFST CEVFS_DB_HEADER2_OFST+CEVFS_DB_HEADER2_SZ 52 | 53 | #define CEVFS_FILE_SCHEMA_NO 1 54 | #define CEVFS_FIRST_MAPPED_PAGE 3 55 | 56 | #ifdef SQLITE_DEBUG 57 | #define CEVFS_PRINTF(a,b,...) cevfs_printf(a,b,##__VA_ARGS__) 58 | #else 59 | #define CEVFS_PRINTF(a,b,...) 60 | #endif 61 | 62 | SQLITE_PRIVATE int sqlite3PagerCloseShim(Pager *pPager, sqlite3* db){ 63 | #if SQLITE_VERSION_NUMBER < 3016000 64 | return sqlite3PagerClose(pPager); 65 | #else 66 | return sqlite3PagerClose(pPager, db); 67 | #endif 68 | } 69 | 70 | #ifndef EXTRA_SIZE 71 | #define EXTRA_SIZE sizeof(MemPage) 72 | #endif 73 | 74 | // Compression size and offset types 75 | typedef u16 CevfsCmpSize; 76 | typedef u16 CevfsCmpOfst; 77 | 78 | /* 79 | ** The header string that appears at the beginning of every 80 | ** SQLite database. 81 | */ 82 | #ifndef SQLITE_AMALGAMATION 83 | static const char zMagicHeader[] = SQLITE_FILE_HEADER; 84 | #endif 85 | 86 | /* 87 | ** Keeps track of data we need to persist for the pager. 88 | ** This will be stored uncompressed at offset 100-199. 89 | */ 90 | typedef struct cevfs_header cevfs_header; 91 | struct cevfs_header { 92 | u8 schema; // 01 file schema version number 93 | Pgno currPgno; // 04 curr lower pager pgno being filled 94 | CevfsCmpOfst currPageOfst; // 02 curr offset for next compressed page 95 | u16 pgMapCnt; // 02 num elements of last page map 96 | u32 uppPgSz; // 04 Upper pager page size. Could be different from lower pager's 97 | Pgno uppPageFile; // 04 max pgno in upper pager, used to report filesize 98 | u16 mmTblMaxCnt; // 02 max entries avail for master map table, computed when table is loaded 99 | u16 mmTblCurrCnt; // 02 curr total elements used in master map table 100 | unsigned char reserved[79]; // 79 pad structure to 100 bytes 101 | }; 102 | 103 | /* 104 | ** Page 1, bytes from offset CEVFS_DB_HEADER2_OFST to end of page, will have a master map for coordinating 105 | ** all the other mapping tables. If table becomes full, perhaps a larger pagesize will help. 106 | ** This can be set using PRAGMA page_size (e.g.: PRAGMA page_size = 2048) 107 | */ 108 | typedef struct CevfsMMTblEntry CevfsMMTblEntry; 109 | struct __attribute__ ((__packed__)) CevfsMMTblEntry { 110 | Pgno lwrPgno; // 04 lower pager pgno where actual page map data is stored 111 | }; 112 | 113 | /* 114 | ** Each time we read a page, it'll be associated with a CevfsMemPage 115 | ** to store temporary in-memory data that belongs to this page. 116 | */ 117 | typedef struct CevfsMemPage CevfsMemPage; 118 | struct CevfsMemPage { 119 | DbPage *pDbPage; // Pager page handle 120 | Pgno pgno; // The pgno to which this belongs 121 | u16 dbHdrOffset; // Offset to the beginning of the header 122 | u16 pgHdrOffset; // Offset to the beginning of the data 123 | u8 *aData; // Pointer to disk image of the page data 124 | }; 125 | 126 | /* 127 | ** Mapping table for uncompressed to compressed content. 128 | ** The table is stored on page 2 at offset 0. 129 | ** The maximum size of table depends on the pager page size. 130 | ** If that is not enough, multiple tables will be used. 131 | ** As each new table is created, it is stored on the next available page. 132 | */ 133 | typedef struct cevfs_map_entry cevfs_map_entry; 134 | struct __attribute__ ((__packed__)) cevfs_map_entry { 135 | Pgno lwrPgno; // 04 mapped lower pager pgno 136 | CevfsCmpSize cmprSz; // 02 size of compressed page 137 | CevfsCmpOfst cmprOfst; // 02 lower page offset for compressed page 138 | }; 139 | 140 | /* 141 | ** An instance of this structure is attached to each cevfs VFS to 142 | ** provide auxiliary non-persisted information. 143 | */ 144 | typedef struct cevfs_file cevfs_file; 145 | typedef struct cevfs_info cevfs_info; 146 | struct cevfs_info { 147 | sqlite3_vfs *pRootVfs; // The underlying real VFS 148 | const char *zVfsName; // Name of this VFS 149 | sqlite3_vfs *pCevfsVfs; // Pointer back to the cevfs VFS 150 | cevfs_file *pFile; // Pointer back to the cevfs_file representing the dest. db. 151 | sqlite3 *pDb; // Pointer to sqlite3 instance 152 | int cerod_activated; // if extension is enabled, make sure read only 153 | 154 | // Pointers to custom compress/encryption functions implemented by the user 155 | void *pCtx; 156 | t_xAutoDetect xAutoDetect; 157 | 158 | u32 upperPgSize; // Temp storage for upperPgSize 159 | }; 160 | 161 | /* 162 | ** The sqlite3_file object for the shim. 163 | */ 164 | struct cevfs_file { 165 | sqlite3_file base; // Base class. Must be first 166 | sqlite3_file *pReal; // The real underlying file 167 | cevfs_info *pInfo; // Custom info for this file 168 | const char *zFName; // Base name of the file 169 | char *zUppJournalPath; // Path to redirect upper journal 170 | unsigned char zDbHeader[100]; // Sqlite3 DB header 171 | cevfs_header cevfsHeader; // Cevfs header with page mapping data 172 | CevfsMethods vfsMethods; // Custom methods for compression/enctyption 173 | size_t nEncIvSz; // IV blob size in bytes for encryption/decryption routines 174 | 175 | // map 176 | CevfsMMTblEntry *mmTbl; // The master mapping table 177 | u16 mmTblCurrIx; // Index of the current page map in mmTbl 178 | cevfs_map_entry *pPgMap; // The current page map 179 | cevfs_map_entry *pBigEndianPgMap; // Used for converting integers to big-endian when saving 180 | u16 pgMapMaxCnt; // Max entries for a page map, based on page size 181 | u16 pgMapSz; // Size in bytes for the page map allocation 182 | u32 nBytesPerPgMap; // Performance optimization premultiplication store 183 | 184 | // pager 185 | CevfsMemPage *pPage1; // Page 1 of the pager 186 | Pager *pPager; // Pager for I/O with compressed/encrypted file 187 | Pgno lwrPageFile; // max pgno in lower pager, used to update pager header 188 | u32 pageSize; // Page size of the lower pager 189 | u32 usableSize; // Number of usable bytes on each page 190 | u8 nTransactions; // Number of open transactions on the pager 191 | 192 | // bools 193 | u8 bPgMapDirty:1; // Curr page map needs to be persisted 194 | u8 bReadOnly:1; // True when db was open for read-only 195 | u8 bCompressionEnabled:1; 196 | u8 bEncryptionEnabled:1; 197 | 198 | }; 199 | 200 | /* 201 | ** Method declarations for cevfs_file. 202 | */ 203 | static int cevfsClose(sqlite3_file*); 204 | static int cevfsRead(sqlite3_file*, void*, int iAmt, sqlite3_int64 iOfst); 205 | static int cevfsWrite(sqlite3_file*,const void*,int iAmt, sqlite3_int64); 206 | static int cevfsTruncate(sqlite3_file*, sqlite3_int64 size); 207 | static int cevfsSync(sqlite3_file*, int flags); 208 | static int cevfsFileSize(sqlite3_file*, sqlite3_int64 *pSize); 209 | static int cevfsLock(sqlite3_file*, int); 210 | static int cevfsUnlock(sqlite3_file*, int); 211 | static int cevfsCheckReservedLock(sqlite3_file*, int *); 212 | static int cevfsFileControl(sqlite3_file*, int op, void *pArg); 213 | static int cevfsSectorSize(sqlite3_file*); 214 | static int cevfsDeviceCharacteristics(sqlite3_file*); 215 | static int cevfsShmLock(sqlite3_file*,int,int,int); 216 | static int cevfsShmMap(sqlite3_file*,int,int,int, void volatile **); 217 | static void cevfsShmBarrier(sqlite3_file*); 218 | static int cevfsShmUnmap(sqlite3_file*,int); 219 | 220 | /* 221 | ** Method declarations for cevfs_vfs. 222 | */ 223 | static int cevfsOpen(sqlite3_vfs*, const char *, sqlite3_file*, int , int *); 224 | static int cevfsDelete(sqlite3_vfs*, const char *zName, int syncDir); 225 | static int cevfsAccess(sqlite3_vfs*, const char *zName, int flags, int *); 226 | static int cevfsFullPathname(sqlite3_vfs*, const char *zName, int, char *); 227 | static void *cevfsDlOpen(sqlite3_vfs*, const char *zFilename); 228 | static void cevfsDlError(sqlite3_vfs*, int nByte, char *zErrMsg); 229 | static void (*cevfsDlSym(sqlite3_vfs*,void*, const char *zSymbol))(void); 230 | static void cevfsDlClose(sqlite3_vfs*, void*); 231 | static int cevfsRandomness(sqlite3_vfs*, int nByte, char *zOut); 232 | static int cevfsSleep(sqlite3_vfs*, int microseconds); 233 | static int cevfsCurrentTime(sqlite3_vfs*, double*); 234 | static int cevfsGetLastError(sqlite3_vfs*, int, char*); 235 | static int cevfsCurrentTimeInt64(sqlite3_vfs*, sqlite3_int64*); 236 | 237 | /* 238 | ** Forward declarations 239 | */ 240 | static CevfsMemPage *memPageFromDbPage(DbPage *pDbPage, Pgno mappedPgno); 241 | static int cevfsNewDatabase(cevfs_file *pFile); 242 | static int cevfsWriteUncompressed(cevfs_file *, Pgno, CevfsCmpOfst, const void *zBuf, int iAmt); 243 | static int cevfsReadUncompressed(cevfs_file *, Pgno, CevfsCmpOfst, void *zBuf, int iAmt); 244 | static int cevfsSaveHeader(cevfs_file *p); 245 | static int cevfsLoadHeader(cevfs_file *p); 246 | 247 | /* 248 | ** Return a pointer to the tail of the pathname. Examples: 249 | ** 250 | ** /home/drh/xyzzy.txt -> xyzzy.txt 251 | ** xyzzy.txt -> xyzzy.txt 252 | */ 253 | const char *fileTail(const char *z){ 254 | int i; 255 | if( z==0 ) return 0; 256 | i = (int)strlen(z)-1; 257 | while( i>0 && z[i-1]!='/' ){ i--; } 258 | return &z[i]; 259 | } 260 | 261 | /* 262 | ** Map the upper pager's journal file onto a different name. 263 | ** findCreateFileMode() in os_unix.c requires journal file to be in same directory 264 | ** and not have additional '-' in name. We'll just append "btree" to distinguish it from ours. 265 | ** Note: everything after the '-' must be alphanumeric only. No punctuation allowed 266 | ** or an assertion will be triggered in debug mode. 267 | ** 268 | ** If we always had a pointer to the associated cevfs_file when processing a journal filename then 269 | ** we could associate the renamed journal filename with the corresponding database filename and 270 | ** encapsulate the memory management. Since we cannot, we use bMustRelease to assist with memory management. 271 | ** 272 | ** pFile or bMustRelease must be NULL. Both cannot be NULL. 273 | ** If bMustRelease is true upon return, the newly created string must be freed by the caller. 274 | ** Use sqlite3_free to free memory. 275 | */ 276 | static char * cevfsMapPath(cevfs_file *pFile, const char *zName, bool *bMustRelease){ 277 | // Only one of pFile or bMustRelease MUST be NULL. 278 | assert( !(pFile && bMustRelease) && (pFile || bMustRelease) ); 279 | 280 | static const char *zTail = "btree"; 281 | if (bMustRelease) *bMustRelease = false; 282 | if( strstr(zName, "-journal")==0 ){ 283 | return (char *)zName; 284 | } 285 | char *zUppJournalPath = pFile ? pFile->zUppJournalPath : NULL; 286 | if( zUppJournalPath == NULL ){ 287 | zUppJournalPath = sqlite3_malloc((int)(strlen(zName)+strlen(zTail))+1); 288 | *zUppJournalPath = '\0'; 289 | strcat(zUppJournalPath, zName); 290 | strcat(zUppJournalPath, zTail); 291 | if(pFile) 292 | pFile->zUppJournalPath = zUppJournalPath; 293 | else 294 | if (bMustRelease) *bMustRelease = true; 295 | } 296 | return zUppJournalPath; 297 | } 298 | 299 | /* 300 | ** Send trace output defined by zFormat and subsequent arguments. 301 | */ 302 | static void cevfs_printf( 303 | cevfs_info *pInfo, 304 | const char *zFormat, 305 | ... 306 | ){ 307 | va_list ap; 308 | char *zMsg; 309 | va_start(ap, zFormat); 310 | zMsg = sqlite3_vmprintf(zFormat, ap); 311 | va_end(ap); 312 | fputs(zMsg, stdout); 313 | sqlite3_free(zMsg); 314 | } 315 | 316 | /* 317 | ** Convert value rc into a string and print it using zFormat. zFormat 318 | ** should have exactly one %s 319 | */ 320 | static void cevfs_print_errcode( 321 | cevfs_info *pInfo, 322 | const char *zFormat, 323 | int rc 324 | ){ 325 | char zBuf[50]; 326 | char *zVal; 327 | switch( rc ){ 328 | case SQLITE_OK: zVal = "SQLITE_OK"; break; 329 | case SQLITE_ERROR: zVal = "SQLITE_ERROR"; break; 330 | case SQLITE_PERM: zVal = "SQLITE_PERM"; break; 331 | case SQLITE_ABORT: zVal = "SQLITE_ABORT"; break; 332 | case SQLITE_BUSY: zVal = "SQLITE_BUSY"; break; 333 | case SQLITE_NOMEM: zVal = "SQLITE_NOMEM"; break; 334 | case SQLITE_READONLY: zVal = "SQLITE_READONLY"; break; 335 | case SQLITE_INTERRUPT: zVal = "SQLITE_INTERRUPT"; break; 336 | case SQLITE_IOERR: zVal = "SQLITE_IOERR"; break; 337 | case SQLITE_CORRUPT: zVal = "SQLITE_CORRUPT"; break; 338 | case SQLITE_FULL: zVal = "SQLITE_FULL"; break; 339 | case SQLITE_CANTOPEN: zVal = "SQLITE_CANTOPEN"; break; 340 | case SQLITE_PROTOCOL: zVal = "SQLITE_PROTOCOL"; break; 341 | case SQLITE_EMPTY: zVal = "SQLITE_EMPTY"; break; 342 | case SQLITE_SCHEMA: zVal = "SQLITE_SCHEMA"; break; 343 | case SQLITE_CONSTRAINT: zVal = "SQLITE_CONSTRAINT"; break; 344 | case SQLITE_MISMATCH: zVal = "SQLITE_MISMATCH"; break; 345 | case SQLITE_MISUSE: zVal = "SQLITE_MISUSE"; break; 346 | case SQLITE_NOLFS: zVal = "SQLITE_NOLFS"; break; 347 | case SQLITE_IOERR_READ: zVal = "SQLITE_IOERR_READ"; break; 348 | case SQLITE_IOERR_SHORT_READ: zVal = "SQLITE_IOERR_SHORT_READ"; break; 349 | case SQLITE_IOERR_WRITE: zVal = "SQLITE_IOERR_WRITE"; break; 350 | case SQLITE_IOERR_FSYNC: zVal = "SQLITE_IOERR_FSYNC"; break; 351 | case SQLITE_IOERR_DIR_FSYNC: zVal = "SQLITE_IOERR_DIR_FSYNC"; break; 352 | case SQLITE_IOERR_TRUNCATE: zVal = "SQLITE_IOERR_TRUNCATE"; break; 353 | case SQLITE_IOERR_FSTAT: zVal = "SQLITE_IOERR_FSTAT"; break; 354 | case SQLITE_IOERR_UNLOCK: zVal = "SQLITE_IOERR_UNLOCK"; break; 355 | case SQLITE_IOERR_RDLOCK: zVal = "SQLITE_IOERR_RDLOCK"; break; 356 | case SQLITE_IOERR_DELETE: zVal = "SQLITE_IOERR_DELETE"; break; 357 | case SQLITE_IOERR_BLOCKED: zVal = "SQLITE_IOERR_BLOCKED"; break; 358 | case SQLITE_IOERR_NOMEM: zVal = "SQLITE_IOERR_NOMEM"; break; 359 | case SQLITE_IOERR_ACCESS: zVal = "SQLITE_IOERR_ACCESS"; break; 360 | case SQLITE_IOERR_CHECKRESERVEDLOCK: 361 | zVal = "SQLITE_IOERR_CHECKRESERVEDLOCK"; break; 362 | case SQLITE_IOERR_LOCK: zVal = "SQLITE_IOERR_LOCK"; break; 363 | case SQLITE_IOERR_CLOSE: zVal = "SQLITE_IOERR_CLOSE"; break; 364 | case SQLITE_IOERR_DIR_CLOSE: zVal = "SQLITE_IOERR_DIR_CLOSE"; break; 365 | case SQLITE_IOERR_SHMOPEN: zVal = "SQLITE_IOERR_SHMOPEN"; break; 366 | case SQLITE_IOERR_SHMSIZE: zVal = "SQLITE_IOERR_SHMSIZE"; break; 367 | case SQLITE_IOERR_SHMLOCK: zVal = "SQLITE_IOERR_SHMLOCK"; break; 368 | case SQLITE_IOERR_SHMMAP: zVal = "SQLITE_IOERR_SHMMAP"; break; 369 | case SQLITE_IOERR_SEEK: zVal = "SQLITE_IOERR_SEEK"; break; 370 | case SQLITE_IOERR_GETTEMPPATH: zVal = "SQLITE_IOERR_GETTEMPPATH"; break; 371 | case SQLITE_IOERR_CONVPATH: zVal = "SQLITE_IOERR_CONVPATH"; break; 372 | case SQLITE_READONLY_DBMOVED: zVal = "SQLITE_READONLY_DBMOVED"; break; 373 | case SQLITE_LOCKED_SHAREDCACHE: zVal = "SQLITE_LOCKED_SHAREDCACHE"; break; 374 | case SQLITE_BUSY_RECOVERY: zVal = "SQLITE_BUSY_RECOVERY"; break; 375 | case SQLITE_CANTOPEN_NOTEMPDIR: zVal = "SQLITE_CANTOPEN_NOTEMPDIR"; break; 376 | default: { 377 | sqlite3_snprintf(sizeof(zBuf), zBuf, "%d", rc); 378 | zVal = zBuf; 379 | break; 380 | } 381 | } 382 | CEVFS_PRINTF(pInfo, zFormat, zVal); 383 | } 384 | 385 | /* 386 | ** Append to a buffer. 387 | */ 388 | static void strappend(char *z, int *pI, const char *zAppend){ 389 | int i = *pI; 390 | while( zAppend[0] ){ z[i++] = *(zAppend++); } 391 | z[i] = 0; 392 | *pI = i; 393 | } 394 | 395 | static int cevfsReadUncompressed( 396 | cevfs_file *p, 397 | Pgno pgno, 398 | CevfsCmpOfst offset, 399 | void *zBuf, 400 | int iAmt 401 | ){ 402 | int rc; 403 | DbPage *pPage; 404 | if( (rc = sqlite3PagerGet(p->pPager, pgno, &pPage, 0)) == SQLITE_OK ){ 405 | void *data = sqlite3PagerGetData(pPage); 406 | memcpy(zBuf, data+offset, iAmt); 407 | sqlite3PagerUnref(pPage); 408 | } 409 | return rc; 410 | } 411 | 412 | static void cevfsReleasePage1(cevfs_file *p){ 413 | if( p->pPage1 ){ 414 | sqlite3PagerUnref(p->pPage1->pDbPage); 415 | p->pPage1 = NULL; 416 | } 417 | } 418 | 419 | /* 420 | ** Create master mapping table. 421 | ** This table will point to the actual upper-to-lower page maps. 422 | ** Its size depends on the page size of the lower pager. 423 | */ 424 | static int cevfsCreateMMTbl(cevfs_file *p, int *memSzOut){ 425 | u16 maxSz = p->pageSize - CEVFS_DB_HEADER_SIZE; 426 | u16 maxEntries = maxSz / sizeof(CevfsMMTblEntry); 427 | 428 | // At this point, header may already be loaded from persistent storage 429 | // so be careful modifying header values that could be needed elsewhere. 430 | p->cevfsHeader.mmTblMaxCnt = maxEntries; 431 | p->mmTblCurrIx = -1; // u16, so results in some large number to mean "not defined" 432 | 433 | // allocate 434 | int memSz = maxEntries*sizeof(CevfsMMTblEntry); 435 | if( !(p->mmTbl = sqlite3_malloc(memSz)) ) return SQLITE_NOMEM; 436 | 437 | // out param 438 | if( memSzOut ) *memSzOut = memSz; 439 | return SQLITE_OK; 440 | } 441 | 442 | static int cevfsSavePagemapData(cevfs_file *p){ 443 | int rc = SQLITE_OK; 444 | if( p->bPgMapDirty ){ 445 | Pgno pgno = p->mmTbl[p->mmTblCurrIx].lwrPgno; 446 | rc = cevfsWriteUncompressed(p, pgno, 0, p->pBigEndianPgMap, p->pgMapSz); 447 | if( rc==SQLITE_OK ) p->bPgMapDirty = 0; 448 | } 449 | return rc; 450 | } 451 | 452 | static int cevfsSaveMMTbl(cevfs_file *p){ 453 | int rc; 454 | assert( p->bReadOnly==0 ); 455 | cevfs_header *header = &p->cevfsHeader; 456 | int memSz = header->mmTblMaxCnt*sizeof(CevfsMMTblEntry); 457 | CevfsMMTblEntry *buf = sqlite3_malloc(memSz); 458 | if( buf ){ 459 | for(u16 i=0; immTblCurrCnt; i++){ 460 | put2byte((u8 *)&buf[i].lwrPgno, p->mmTbl[i].lwrPgno); 461 | } 462 | if( (rc = cevfsWriteUncompressed(p, 1, CEVFS_DB_MMTBL_OFST, buf, memSz))==SQLITE_OK){ 463 | sqlite3_free(buf); 464 | rc = cevfsSavePagemapData(p); 465 | } 466 | }else rc = SQLITE_NOMEM; 467 | return rc; 468 | } 469 | 470 | static int cevfsLoadPagemapData(cevfs_file *p, u16 ix){ 471 | int rc; 472 | cevfs_header *header = &p->cevfsHeader; 473 | assert( p->bPgMapDirty==0 ); 474 | assert( ix != p->mmTblCurrIx ); // mmTblCurrIx initially large number to mean no entries yet 475 | Pgno pgno = p->mmTbl[ix].lwrPgno; 476 | rc = cevfsReadUncompressed(p, pgno, 0, p->pBigEndianPgMap, p->pgMapSz); 477 | if( rc==SQLITE_OK ){ 478 | u16 maxCnt = ix==header->mmTblCurrCnt-1 ? header->pgMapCnt : p->pgMapMaxCnt; 479 | for(u16 i = 0; ipPgMap[i].lwrPgno = get4byte((u8 *)&p->pBigEndianPgMap[i].lwrPgno); 481 | p->pPgMap[i].cmprSz = get2byte((u8 *)&p->pBigEndianPgMap[i].cmprSz); 482 | p->pPgMap[i].cmprOfst = get2byte((u8 *)&p->pBigEndianPgMap[i].cmprOfst); 483 | } 484 | p->mmTblCurrIx = ix; 485 | } 486 | return rc; 487 | } 488 | 489 | static int cevfsLoadMMTbl(cevfs_file *p){ 490 | int rc; 491 | int memSz; 492 | cevfs_header *header = &p->cevfsHeader; 493 | assert( p->mmTbl==NULL ); 494 | 495 | // Header must have already been loaded 496 | assert( header->mmTblCurrCnt>0 ); 497 | 498 | if( (rc = cevfsCreateMMTbl(p, &memSz))==SQLITE_OK ){ 499 | CevfsMMTblEntry *buf = sqlite3_malloc(memSz); 500 | if( buf ){ 501 | if( (rc = cevfsReadUncompressed(p, 1, CEVFS_DB_MMTBL_OFST, buf, memSz))==SQLITE_OK){ 502 | for(u16 i=0; immTblCurrCnt; i++){ 503 | p->mmTbl[i].lwrPgno = get2byte((u8 *)&buf[i].lwrPgno); 504 | } 505 | sqlite3_free(buf); 506 | } 507 | } 508 | }else rc = SQLITE_NOMEM; 509 | return rc; 510 | } 511 | 512 | static int cevfsPagerLock(cevfs_file *p){ 513 | int rc; 514 | assert( p->pPage1==0 ); 515 | if( (rc = sqlite3PagerSharedLock(p->pPager))==SQLITE_OK ){ 516 | DbPage *pDbPage1; 517 | if( (rc = sqlite3PagerGet(p->pPager, 1, &pDbPage1, 0))==SQLITE_OK ){ 518 | p->pPage1 = memPageFromDbPage(pDbPage1, 1); 519 | int nPageFile = 0; 520 | sqlite3PagerPagecount(p->pPager, &nPageFile); 521 | 522 | // calc max entries for each page map based on page size 523 | p->pgMapMaxCnt = p->pageSize / sizeof(cevfs_map_entry); 524 | p->pgMapSz = p->pgMapMaxCnt * sizeof(cevfs_map_entry); 525 | 526 | // Optimization: Do this multiplication and store it for later use. 527 | p->nBytesPerPgMap = p->pgMapMaxCnt * p->cevfsHeader.uppPgSz; 528 | 529 | /* Allocate space for a single page map. 530 | Only one page map will be in memory at a time. */ 531 | p->pPgMap = sqlite3_malloc(p->pgMapSz); 532 | p->pBigEndianPgMap = sqlite3_malloc(p->pgMapSz); 533 | if( p->pPgMap && p->pBigEndianPgMap ){ 534 | memset((void *)p->pPgMap, 0, p->pgMapSz); 535 | memset((void *)p->pBigEndianPgMap, 0, p->pgMapSz); 536 | if( nPageFile==0 ){ 537 | /* We will be creating a new database so set up some data that is 538 | needed right away that would be too late to do in cevfsNewDatabase(). */ 539 | if( (rc = cevfsCreateMMTbl(p, NULL))==SQLITE_OK ){ 540 | p->mmTbl[0].lwrPgno = 2; 541 | p->cevfsHeader.mmTblCurrCnt = 1; 542 | } 543 | }else{ 544 | // restore some data 545 | rc = cevfsLoadHeader(p); 546 | if( rc==SQLITE_OK && p->cevfsHeader.schema > CEVFS_FILE_SCHEMA_NO ){ 547 | // The file schema# is larger than this version can handle. 548 | // A newer version is needed to read this file. 549 | rc = CEVFS_ERROR_EXT_VERSION_TOO_OLD; 550 | } 551 | if( rc==SQLITE_OK ) rc = cevfsLoadMMTbl(p); 552 | if( rc==SQLITE_OK ) rc = cevfsLoadPagemapData(p, 0); 553 | } 554 | /* reminder: do not call sqlite3PagerUnref(pDbPage1) here as this will 555 | cause pager state to reset to PAGER_OPEN which is not desirable for writing to pager. */ 556 | }else rc = SQLITE_NOMEM; 557 | } 558 | } 559 | return rc; 560 | } 561 | 562 | static int cevfsPagerWrite(cevfs_file *p, PgHdr *pPg){ 563 | int rc = SQLITE_OK; 564 | if( p->nTransactions == 0 ){ 565 | if( (rc = sqlite3PagerBegin(p->pPager, 0, 1))==SQLITE_OK ){ 566 | p->nTransactions++; 567 | if( p->lwrPageFile==0 ){ 568 | rc = cevfsNewDatabase(p); 569 | } 570 | } 571 | } 572 | if( rc==SQLITE_OK ) return sqlite3PagerWrite(pPg); 573 | return rc; 574 | } 575 | 576 | static int cevfsWriteUncompressed( 577 | cevfs_file *pFile, 578 | Pgno pgno, 579 | CevfsCmpOfst offset, 580 | const void *zBuf, 581 | int iAmt 582 | ){ 583 | int rc; 584 | DbPage *pPage = NULL; 585 | if( (rc = sqlite3PagerGet(pFile->pPager, pgno, &pPage, 0)) == SQLITE_OK ){ 586 | void *data = sqlite3PagerGetData(pPage); 587 | if( (rc = cevfsPagerWrite(pFile, pPage)) == SQLITE_OK ){ 588 | memcpy(data+offset, zBuf, iAmt); 589 | } 590 | sqlite3PagerUnref(pPage); 591 | } 592 | return rc; 593 | } 594 | 595 | static int cevfsSaveHeader(cevfs_file *p){ 596 | assert( p->bReadOnly==0 ); 597 | cevfs_header *header = &p->cevfsHeader; 598 | u8 buf[CEVFS_DB_HEADER2_SZ]; 599 | memcpy(buf, &header->schema, 1); 600 | put4byte(buf+1, header->currPgno); 601 | put2byte(buf+5, header->currPageOfst); 602 | put2byte(buf+7, header->pgMapCnt); 603 | put4byte(buf+9, header->uppPgSz); 604 | put4byte(buf+13, header->uppPageFile); 605 | put2byte(buf+17, header->mmTblMaxCnt); 606 | put2byte(buf+19, header->mmTblCurrCnt); 607 | memset(buf+21, 0, 79); 608 | return cevfsWriteUncompressed(p, 1, CEVFS_DB_HEADER2_OFST, buf, CEVFS_DB_HEADER2_SZ); 609 | } 610 | 611 | static int cevfsLoadHeader(cevfs_file *p){ 612 | cevfs_header *header = &p->cevfsHeader; 613 | u8 buf[CEVFS_DB_HEADER2_SZ]; 614 | int rc; 615 | if( (rc = cevfsReadUncompressed(p, 1, CEVFS_DB_HEADER2_OFST, buf, CEVFS_DB_HEADER2_SZ))==SQLITE_OK ){ 616 | header->schema = buf[0]; 617 | header->currPgno = get4byte(buf+1); 618 | header->currPageOfst = get2byte(buf+5); 619 | header->pgMapCnt = get2byte(buf+7); 620 | header->uppPgSz = get4byte(buf+9); 621 | header->uppPageFile = get4byte(buf+13); 622 | header->mmTblMaxCnt = get2byte(buf+17); 623 | header->mmTblCurrCnt = get2byte(buf+19); 624 | } 625 | return rc; 626 | } 627 | 628 | static CevfsMemPage *memPageFromDbPage(DbPage *pDbPage, Pgno mappedPgno){ 629 | CevfsMemPage* pPg = (CevfsMemPage *)sqlite3PagerGetExtra(pDbPage); 630 | if( mappedPgno != pPg->pgno ){ 631 | pPg->pgno = mappedPgno; 632 | pPg->pDbPage = pDbPage; 633 | pPg->dbHdrOffset = mappedPgno==1 ? CEVFS_DB_HEADER_SIZE : 0; 634 | pPg->pgHdrOffset = 0; // Not used anymore 635 | pPg->pDbPage->pgno = mappedPgno; // pager uses this to determine pager size 636 | pPg->aData = sqlite3PagerGetData(pDbPage); 637 | } 638 | return pPg; 639 | } 640 | 641 | static int cevfsNewDatabase(cevfs_file *pFile){ 642 | CevfsMemPage *pP1 = pFile->pPage1; 643 | unsigned char *data = pP1->aData; 644 | cevfs_info *pInfo = (cevfs_info *)pFile->pInfo; 645 | int rc; 646 | 647 | if( (rc = cevfsPagerWrite(pFile, pP1->pDbPage))==SQLITE_OK ){ 648 | // since we are using a secondary pager, set up a proper pager header (see btree.c:1898) 649 | 650 | // Set first 16 characters (including NULL terminator) 651 | // to the name of the VFS prefixed with CEVFS- 652 | // This leaves the user with 10 characters to identify the VFS. 653 | const char *prefix = "CEVFS-"; 654 | const size_t iLen1 = strlen(prefix); 655 | const size_t iLen2 = 15-iLen1; 656 | const size_t iNameLen = strlen(pInfo->zVfsName); 657 | memset(data, 0, 16); 658 | memcpy(data, prefix, iLen1); 659 | memcpy(data+iLen1, pInfo->zVfsName, iNameLen > iLen2 ? iLen2 : iNameLen); 660 | assert( strlen((const char *)data)<16 ); 661 | 662 | data[16] = (u8)((pFile->pageSize>>8)&0xff); 663 | data[17] = (u8)((pFile->pageSize>>16)&0xff); 664 | data[18] = 1; 665 | data[19] = 1; 666 | assert( pFile->usableSize<=pFile->pageSize && pFile->usableSize+255>=pFile->pageSize); 667 | data[20] = (u8)(pFile->pageSize - pFile->usableSize); 668 | data[21] = 64; 669 | data[22] = 32; 670 | data[23] = 32; 671 | memset(&data[24], 0, 100-24); 672 | data[31] = 1; // 28-31 size of the database file in pages 673 | } 674 | return rc; 675 | } 676 | 677 | /* 678 | ** Switch to a specific page map based on pager offset, 679 | ** saving the current page map if needed. 680 | ** @returns index# of page map switched to. 681 | */ 682 | static u16 cevfsSwitchPageMap(cevfs_file *p, sqlite_int64 iUppOfst){ 683 | int rc = SQLITE_ERROR; 684 | cevfs_info *pInfo = p->pInfo; 685 | cevfs_header *header = &p->cevfsHeader; 686 | u16 ix = 0; 687 | 688 | /* 689 | Calculate map index based on upper pager offset. 690 | Check last entry first as an optimization in case we are writing. 691 | Perhaps we can do this check only when we are writing, skip for read-only. 692 | */ 693 | ix = iUppOfst >= (p->nBytesPerPgMap * header->mmTblCurrCnt) 694 | ? header->mmTblCurrCnt-1 695 | : (u16)(iUppOfst / p->nBytesPerPgMap); 696 | if( ixmmTblCurrCnt ) rc=SQLITE_OK; 697 | 698 | // switch 699 | if( rc==SQLITE_OK && ix != p->mmTblCurrIx ){ 700 | CEVFS_PRINTF(pInfo, "Switching to map #%u for offset %lld\n", (unsigned)ix, iUppOfst); 701 | // save 702 | if( (rc = cevfsSavePagemapData(p))==SQLITE_OK ){ 703 | // reset 704 | memset(p->pPgMap, 0, p->pgMapSz); 705 | //load 706 | rc = cevfsLoadPagemapData(p, ix); 707 | if( rc==SQLITE_OK ) p->mmTblCurrIx = ix; 708 | } 709 | } 710 | return ix; 711 | } 712 | 713 | static int cevfsPageMapGet( 714 | cevfs_file *pFile, 715 | sqlite_uint64 uSrcOfst, 716 | Pgno *outUppPgno, 717 | Pgno *outLwrPgno, 718 | CevfsCmpOfst *outCmpOfst, 719 | CevfsCmpSize *outCmpSz, 720 | u16 *outIx 721 | ){ 722 | cevfs_header *header = &pFile->cevfsHeader; 723 | if( outUppPgno ) *outUppPgno = (Pgno)(uSrcOfst/header->uppPgSz+1); 724 | int currPgMapNo = cevfsSwitchPageMap(pFile, uSrcOfst); 725 | if( pFile->pPgMap ){ 726 | // determine max elements based on if last page map is currently in memory 727 | u16 maxCnt = pFile->mmTblCurrIx==header->mmTblCurrCnt-1 ? header->pgMapCnt : pFile->pgMapMaxCnt; 728 | // determine which page map 729 | u16 pgMapIx = (u16)(uSrcOfst/pFile->nBytesPerPgMap); 730 | // determine index of entry on page map 731 | int ix = uSrcOfst % pFile->nBytesPerPgMap / header->uppPgSz; 732 | if( 733 | ixpPgMap[ix].lwrPgno; 737 | if( outCmpSz ) *outCmpSz = pFile->pPgMap[ix].cmprSz; 738 | if( outCmpOfst ) *outCmpOfst = pFile->pPgMap[ix].cmprOfst; 739 | if( outIx ) *outIx = ix; 740 | return SQLITE_OK; 741 | } 742 | } 743 | return SQLITE_ERROR; 744 | } 745 | 746 | /* 747 | ** Allocate space to store a compressed page. 748 | ** 749 | ** "Allocate" here simply means to determine a page and offset 750 | ** within the lower pager where the data will be stored. 751 | */ 752 | void cevfsAllocCmpPageSpace( 753 | cevfs_file *pFile, 754 | CevfsCmpSize cmpSz, // Current compressed size of data for allocation 755 | u16 pgMapIx // Index of map entry to record allocation data 756 | ){ 757 | cevfs_header *header = &pFile->cevfsHeader; 758 | CevfsCmpOfst ofst = header->currPageOfst; 759 | cevfs_map_entry *pMapEntry = &pFile->pPgMap[pgMapIx]; 760 | cevfs_map_entry *pBigEndianPgMapEntry = &pFile->pBigEndianPgMap[pgMapIx]; 761 | // Since we no longer write compressed pages to page 1, we can optimize this 762 | //u32 realPageSize = pFile->pageSize - (header->currPgno == 1 ? CEVFS_DB_HEADER_SIZE : 0); 763 | header->currPageOfst += cmpSz; 764 | if( header->currPageOfst > /*realPageSize*/ pFile->pageSize ){ 765 | // current page can't hold any more, start new page. 766 | ofst = 0; 767 | header->currPageOfst = cmpSz; 768 | // Make sure to not use a pgno that we allocated to a pagemap page. 769 | Pgno lstAllocatedPgMapPgno = pFile->mmTbl[header->mmTblCurrCnt-1].lwrPgno; 770 | if( header->currPgno <= lstAllocatedPgMapPgno ) 771 | header->currPgno = lstAllocatedPgMapPgno + 1; 772 | else 773 | header->currPgno++; 774 | } 775 | // Set data in map and in Big Endian version of map for fast save to persistent storage 776 | pMapEntry->lwrPgno = header->currPgno; 777 | pMapEntry->cmprOfst = ofst; 778 | pMapEntry->cmprSz = cmpSz; 779 | put4byte((u8 *)&pBigEndianPgMapEntry->lwrPgno, pMapEntry->lwrPgno); 780 | put2byte((u8 *)&pBigEndianPgMapEntry->cmprSz, pMapEntry->cmprSz); 781 | put2byte((u8 *)&pBigEndianPgMapEntry->cmprOfst, pMapEntry->cmprOfst); 782 | pFile->bPgMapDirty = 1; 783 | } 784 | 785 | int cevfsAddPageEntry( 786 | cevfs_file *pFile, 787 | sqlite3_int64 uppOfst, 788 | CevfsCmpSize cmpSz, 789 | CevfsCmpOfst *outCmpOfst, 790 | Pgno *outLwrPgno 791 | ){ 792 | assert( (!outCmpOfst && !outLwrPgno) || (outCmpOfst && outLwrPgno) ); 793 | cevfs_header *header = &pFile->cevfsHeader; 794 | 795 | // if no more room, start a new pagemap 796 | if( header->pgMapCnt == pFile->pgMapMaxCnt ){ 797 | if( pFile->mmTblCurrIx == header->mmTblMaxCnt ){ 798 | // We've run out of room in the master map table. 799 | // User will need to increase pager size. 800 | return CEVFS_ERROR_PAGE_SIZE_TOO_SMALL; 801 | } 802 | CevfsMMTblEntry *entry = &pFile->mmTbl[header->mmTblCurrCnt]; 803 | entry->lwrPgno = header->currPgno+1; // use next pgno but don't incr. counter! 804 | header->mmTblCurrCnt++; 805 | header->pgMapCnt = 0; 806 | // reminder: can't change pInfo->mmTblCurrIx until after cevfsSwitchPageMap 807 | cevfsSwitchPageMap(pFile, uppOfst); 808 | } 809 | 810 | // add new page map entry 811 | u16 ix = header->pgMapCnt++; 812 | cevfs_map_entry *pPgMapEntry = &pFile->pPgMap[ix]; 813 | 814 | // assign space to store compressed page 815 | cevfsAllocCmpPageSpace(pFile, cmpSz, ix); 816 | 817 | // for placeholder entries, set some data to zero 818 | if( !outLwrPgno ) pPgMapEntry->lwrPgno = 0; 819 | if( !outCmpOfst ) pPgMapEntry->cmprOfst = 0; 820 | 821 | // output params 822 | if( outLwrPgno ) *outLwrPgno = header->currPgno; 823 | if( outCmpOfst ) *outCmpOfst = pPgMapEntry->cmprOfst; 824 | 825 | return SQLITE_OK; 826 | } 827 | 828 | /* 829 | ** Add pager map entry before writing to lower pager 830 | ** to get pgno & offset for pager write operation. 831 | ** 832 | ** uppOfst - upper pager offset 833 | ** cmpSz - compressed size to save 834 | ** outLwrPgno - mapped pgno to write to 835 | ** outCmpOfst - offset to write compressed data to 836 | **/ 837 | static int cevfsPageMapSet( 838 | cevfs_file *pFile, 839 | sqlite_int64 uppOfst, 840 | CevfsCmpSize cmpSz, 841 | Pgno *outUppPgno, 842 | Pgno *outLwrPgno, 843 | CevfsCmpOfst *outCmpOfst 844 | ){ 845 | cevfs_info *pInfo = pFile->pInfo; 846 | cevfs_header *header = &pFile->cevfsHeader; 847 | CevfsCmpSize oldCmpSz; 848 | int rc = SQLITE_OK; 849 | u16 ix; 850 | 851 | assert( outUppPgno ); 852 | assert( outLwrPgno ); 853 | assert( outCmpOfst ); 854 | 855 | if( (rc = cevfsPageMapGet(pFile, uppOfst, outUppPgno, outLwrPgno, outCmpOfst, &oldCmpSz, &ix))==SQLITE_OK ){ 856 | /* 857 | ** We found a map entry. It's either a placeholder entry that needs valid data, 858 | ** an outdated entry that needs updating, or a valid up-to-date entry. 859 | ** If the entry needs updating, we will reuse the space used to hold the previously compressed 860 | ** data if the compressed data now takes up less space or allocate a new space at the end of 861 | ** the db if it now needs more space. 862 | ** Any previously used and now abandoned space will need to be recovered through a vacuum process. 863 | */ 864 | if( oldCmpSz==0 || cmpSz>oldCmpSz ){ 865 | // entry found was either a placeholder or we now need more room, so allocate new space. 866 | cevfs_map_entry *pMapEntry = &pFile->pPgMap[ix]; 867 | cevfsAllocCmpPageSpace(pFile, cmpSz, ix); 868 | 869 | *outLwrPgno = pMapEntry->lwrPgno; 870 | *outCmpOfst = pMapEntry->cmprOfst; 871 | CEVFS_PRINTF(pInfo, "Updated entry (uppOfst=%lld, lwrPgno=%lu,cmpOfst=%lu,cmpSz=%lu)\n", 872 | (long long)uppOfst, (unsigned long)pMapEntry->lwrPgno, (unsigned long)pMapEntry->cmprOfst, (unsigned long)pMapEntry->cmprSz); 873 | return SQLITE_OK; 874 | }else if( cmpSzpPgMap[ix].cmprSz = cmpSz; 877 | put2byte((u8 *)&pFile->pBigEndianPgMap[ix].cmprSz, cmpSz); 878 | pFile->bPgMapDirty = 1; 879 | } 880 | return rc; 881 | }else{ 882 | sqlite3_int64 nextOfst = ((header->mmTblCurrCnt-1) * header->uppPgSz * pFile->pgMapMaxCnt) + (header->pgMapCnt * header->uppPgSz); 883 | while( uppOfst>nextOfst ){ 884 | cevfsAddPageEntry(pFile, nextOfst, 0, NULL, NULL); 885 | CEVFS_PRINTF(pInfo, "Added intermin entry (uppOfst=%lld, lwrPgno=0,cmpOfst=0,cmpSz=0)\n", (long long)nextOfst); 886 | nextOfst += header->uppPgSz; 887 | } 888 | assert( uppOfst==nextOfst ); 889 | cevfsAddPageEntry(pFile, uppOfst, cmpSz, outCmpOfst, outLwrPgno); 890 | } 891 | return SQLITE_OK; 892 | } 893 | 894 | /* 895 | ** Close a cevfs-file. 896 | */ 897 | static int cevfsClose(sqlite3_file *pFile){ 898 | cevfs_file *p = (cevfs_file *)pFile; 899 | cevfs_info *pInfo = p->pInfo; 900 | int rc = SQLITE_OK; 901 | CEVFS_PRINTF(pInfo, "%s.xClose(%s)", pInfo->zVfsName, p->zFName); 902 | 903 | if( p->pPager ){ 904 | if( !p->bReadOnly ){ 905 | int nPageFile = 0; /* Number of pages in the database file */ 906 | sqlite3PagerPagecount(p->pPager, &nPageFile); 907 | assert( p->lwrPageFile==nPageFile ); 908 | 909 | u8 buf[4]; 910 | sqlite3Put4byte(buf, p->lwrPageFile); 911 | rc = cevfsWriteUncompressed(p, 1, 28, buf, 4); 912 | rc = cevfsSaveHeader(p); 913 | 914 | if( (rc = cevfsSaveMMTbl(p))==SQLITE_OK ){ 915 | for(int i=0; inTransactions; i++){ 916 | if( (rc = sqlite3PagerCommitPhaseOne(p->pPager, NULL, 0))==SQLITE_OK ){ 917 | sqlite3PagerCommitPhaseTwo(p->pPager); 918 | } 919 | } 920 | p->nTransactions = 0; 921 | } 922 | } 923 | 924 | if( rc==SQLITE_OK ){ 925 | cevfsReleasePage1(p); 926 | if( (rc = sqlite3PagerCloseShim(p->pPager, pInfo->pDb))==SQLITE_OK ){ 927 | p->pPager = NULL; 928 | if( p->zUppJournalPath ){ 929 | sqlite3_free(p->zUppJournalPath); 930 | p->zUppJournalPath = NULL; 931 | } 932 | if( p->mmTbl ){ 933 | sqlite3_free(p->mmTbl); 934 | p->mmTbl = NULL; 935 | } 936 | if( p->pPgMap ){ 937 | sqlite3_free(p->pPgMap); 938 | p->pPgMap = NULL; 939 | } 940 | if( p->pBigEndianPgMap ){ 941 | sqlite3_free(p->pBigEndianPgMap); 942 | p->pBigEndianPgMap = NULL; 943 | } 944 | } 945 | } 946 | } 947 | 948 | if( (rc == SQLITE_OK) && ((rc = p->pReal->pMethods->xClose(p->pReal)) == SQLITE_OK) ){ 949 | sqlite3_free((void*)p->base.pMethods); 950 | p->base.pMethods = NULL; 951 | } 952 | 953 | cevfs_print_errcode(pInfo, " -> %s\n", rc); 954 | return rc; 955 | } 956 | 957 | /* 958 | ** Read data from a cevfs-file. 959 | */ 960 | static int cevfsRead( 961 | sqlite3_file *pFile, 962 | void *zBuf, 963 | int iAmt, 964 | sqlite_int64 iOfst 965 | ){ 966 | cevfs_file *p = (cevfs_file *)pFile; 967 | cevfs_info *pInfo = p->pInfo; 968 | u32 uppPgSz = p->cevfsHeader.uppPgSz; 969 | int rc; 970 | 971 | if( p->pPager && p->pPage1 ){ 972 | DbPage *pPage; 973 | Pgno uppPgno, mappedPgno; 974 | CevfsCmpOfst cmprPgOfst; 975 | CevfsCmpSize uCmpPgSz; 976 | 977 | if( (rc = cevfsPageMapGet(p, iOfst, &uppPgno, &mappedPgno, &cmprPgOfst, &uCmpPgSz, NULL)) == SQLITE_OK ){ 978 | if( rc==SQLITE_OK && 979 | (rc = sqlite3PagerGet(p->pPager, mappedPgno, &pPage, 0))==SQLITE_OK 980 | ){ 981 | void *pDecBuf = NULL; 982 | void *pUncBuf = NULL; 983 | void *pDstData = NULL; 984 | 985 | CevfsMemPage *pMemPage = memPageFromDbPage(pPage, mappedPgno); 986 | CEVFS_PRINTF( 987 | pInfo, "%s.xRead(%s,pgno=%u->%u,ofst=%08lld->%u,amt=%d->%u)", 988 | pInfo->zVfsName, p->zFName, uppPgno, mappedPgno, iOfst, cmprPgOfst, iAmt, uCmpPgSz 989 | ); 990 | assert( uCmpPgSz > 0 ); 991 | 992 | size_t iDstAmt = uppPgSz; 993 | int bSuccess = 1; 994 | 995 | void *pSrcData = 996 | (char *)pMemPage->aData 997 | +pMemPage->dbHdrOffset 998 | +pMemPage->pgHdrOffset 999 | +cmprPgOfst; 1000 | 1001 | // src = dst, assuming no encryption or compression 1002 | pDstData = pSrcData; 1003 | 1004 | if( p->bEncryptionEnabled ){ 1005 | // The IV is stored first followed by the enctypted data 1006 | void *iv = pSrcData; 1007 | 1008 | pDecBuf = sqlite3_malloc(uCmpPgSz); 1009 | if( pDecBuf ){ 1010 | void *srcData = iv+p->nEncIvSz; 1011 | size_t nDataInSize = uCmpPgSz-p->nEncIvSz; 1012 | size_t nFinalSz; 1013 | 1014 | bSuccess = p->vfsMethods.xDecrypt( 1015 | pInfo->pCtx, 1016 | srcData, // dataIn 1017 | nDataInSize, // data-in length 1018 | iv, // IvIn 1019 | pDecBuf, // dataOut; result is written here. 1020 | uCmpPgSz, // The size of the dataOut buffer in bytes 1021 | &nFinalSz // On successful return, the number of bytes written to dataOut. 1022 | ); 1023 | 1024 | if( bSuccess ){ 1025 | uCmpPgSz = nFinalSz; 1026 | pSrcData = pDstData = pDecBuf; 1027 | }else rc=CEVFS_ERROR_DECRYPTION_FAILED; 1028 | }else rc=SQLITE_NOMEM; 1029 | } // encryption 1030 | 1031 | if( p->bCompressionEnabled && bSuccess && bSuccess && rc==SQLITE_OK ){ 1032 | pUncBuf = sqlite3_malloc((int)iDstAmt); 1033 | if( pUncBuf ){ 1034 | bSuccess = p->vfsMethods.xUncompress(pInfo->pCtx, pUncBuf, &iDstAmt, pSrcData, (int)uCmpPgSz); 1035 | if( bSuccess ){ 1036 | assert( iDstAmt==uppPgSz ); 1037 | pDstData = pUncBuf; 1038 | }else rc=CEVFS_ERROR_DECOMPRESSION_FAILED; 1039 | }else rc=SQLITE_NOMEM; 1040 | } 1041 | 1042 | if( bSuccess && rc==SQLITE_OK ){ 1043 | u16 uBufOfst = iOfst % uppPgSz; 1044 | memcpy(zBuf, pDstData+uBufOfst, iAmt); 1045 | } 1046 | 1047 | if( pDecBuf ) sqlite3_free( pDecBuf ); 1048 | if( pUncBuf ) sqlite3_free( pUncBuf ); 1049 | sqlite3PagerUnref(pPage); 1050 | } 1051 | }else{ 1052 | CEVFS_PRINTF(pInfo, "%s.xRead(%s,ofst=%08lld,amt=%d)", pInfo->zVfsName, p->zFName, iOfst, iAmt); 1053 | memset(zBuf, 0, iAmt); 1054 | rc = SQLITE_OK; 1055 | } 1056 | }else{ 1057 | CEVFS_PRINTF(pInfo, "%s.xRead(%s,ofst=%08lld,amt=%d)", pInfo->zVfsName, p->zFName, iOfst, iAmt); 1058 | rc = p->pReal->pMethods->xRead(p->pReal, zBuf, iAmt, iOfst); 1059 | } 1060 | cevfs_print_errcode(pInfo, " -> %s\n", rc); 1061 | return rc; 1062 | } 1063 | 1064 | /* 1065 | ** Write data to a cevfs-file. 1066 | */ 1067 | static int cevfsWrite( 1068 | sqlite3_file *pFile, 1069 | const void *zBuf, 1070 | int iAmt, 1071 | sqlite_int64 iOfst 1072 | ){ 1073 | cevfs_file *p = (cevfs_file *)pFile; 1074 | cevfs_info *pInfo = p->pInfo; 1075 | int rc = SQLITE_OK; 1076 | 1077 | if( p->pPager ){ 1078 | if( p->bReadOnly ) rc = SQLITE_READONLY; 1079 | else{ 1080 | void *pCmpBuf = NULL; 1081 | void *pEncBuf = NULL; 1082 | void *pSrcData = (void *)zBuf; 1083 | size_t nSrcAmt = iAmt; 1084 | int bSuccess = 1; 1085 | 1086 | if( p->bCompressionEnabled ){ 1087 | size_t nDest = p->vfsMethods.xCompressBound(pInfo->pCtx, nSrcAmt); 1088 | pCmpBuf = sqlite3_malloc((int)nDest); 1089 | if( pCmpBuf ){ 1090 | bSuccess = p->vfsMethods.xCompress(pInfo->pCtx, pCmpBuf, &nDest, pSrcData, nSrcAmt); 1091 | if( bSuccess ){ 1092 | pSrcData = pCmpBuf; 1093 | nSrcAmt = nDest; 1094 | } 1095 | }else rc=SQLITE_NOMEM; 1096 | } 1097 | 1098 | if( p->bEncryptionEnabled && bSuccess ){ 1099 | size_t tmp_csz = 0; 1100 | void *iv = sqlite3_malloc((int)p->nEncIvSz); 1101 | if( iv ){ 1102 | bSuccess = p->vfsMethods.xEncrypt( 1103 | pInfo->pCtx, 1104 | pSrcData, // dataIn 1105 | nSrcAmt, // data-in length 1106 | iv, // IV out 1107 | &pEncBuf, // dataOut; result is written here. 1108 | &tmp_csz, // On successful return, the number of bytes written to dataOut. 1109 | sqlite3_malloc 1110 | ); 1111 | if( bSuccess && pEncBuf ){ 1112 | // Join IV and pEncBuf. If IV is greater than pInfo->nEncIvSz, it will be truncated. 1113 | void *pIvEncBuf = NULL; 1114 | CevfsCmpSize uIvEncSz = p->nEncIvSz+tmp_csz; 1115 | pIvEncBuf = sqlite3_realloc(iv, (int)(uIvEncSz)); 1116 | memcpy(pIvEncBuf+p->nEncIvSz, pEncBuf, tmp_csz); 1117 | sqlite3_free(pEncBuf); 1118 | pSrcData = pEncBuf = pIvEncBuf; 1119 | nSrcAmt = uIvEncSz; 1120 | }else rc=CEVFS_ERROR_ENCRYPTION_FAILED; 1121 | }else rc=SQLITE_NOMEM; 1122 | } 1123 | 1124 | // Make sure dest/lwr page size is large enough for incoming page of data 1125 | assert( nSrcAmt <= p->pageSize ); 1126 | if( rc==SQLITE_OK ){ 1127 | if( nSrcAmt <= p->pageSize ){ 1128 | DbPage *pPage; 1129 | Pgno uppPgno, mappedPgno; 1130 | CevfsCmpOfst cmprPgOfst; 1131 | 1132 | cevfsPageMapSet(p, iOfst, nSrcAmt, &uppPgno, &mappedPgno, &cmprPgOfst); 1133 | 1134 | // write 1135 | if( rc==SQLITE_OK && (rc = sqlite3PagerGet(p->pPager, mappedPgno, &pPage, 0))==SQLITE_OK ){ 1136 | CevfsMemPage *pMemPage = memPageFromDbPage(pPage, mappedPgno); 1137 | if( rc==SQLITE_OK && (rc = cevfsPagerWrite(p, pPage))==SQLITE_OK ){ 1138 | CEVFS_PRINTF( 1139 | pInfo, 1140 | "%s.xWrite(%s, pgno=%u->%u, offset=%08lld->%06lu, amt=%06d->%06d)", 1141 | pInfo->zVfsName, p->zFName, 1142 | uppPgno, mappedPgno, 1143 | iOfst, (unsigned long)(pMemPage->dbHdrOffset+pMemPage->pgHdrOffset+cmprPgOfst), 1144 | iAmt, nSrcAmt 1145 | ); 1146 | memcpy( 1147 | pMemPage->aData 1148 | +pMemPage->dbHdrOffset 1149 | +pMemPage->pgHdrOffset 1150 | +cmprPgOfst, 1151 | pSrcData, 1152 | nSrcAmt 1153 | ); 1154 | 1155 | // Keep track of sizes of upper and lower pagers 1156 | if( p->cevfsHeader.uppPageFilecevfsHeader.uppPageFile = uppPgno; 1157 | if( p->lwrPageFilelwrPageFile = mappedPgno; 1158 | } 1159 | sqlite3PagerUnref(pPage); 1160 | } 1161 | }else rc=CEVFS_ERROR_PAGE_SIZE_TOO_SMALL; 1162 | } 1163 | 1164 | if( pEncBuf ) sqlite3_free( pEncBuf ); 1165 | if( pCmpBuf ) sqlite3_free( pCmpBuf ); 1166 | } 1167 | }else{ 1168 | CEVFS_PRINTF(pInfo, "%s.xWrite(%s, offset=%08lld, amt=%06d)", pInfo->zVfsName, p->zFName, iOfst, iAmt); 1169 | rc = p->pReal->pMethods->xWrite(p->pReal, zBuf, iAmt, iOfst); 1170 | } 1171 | cevfs_print_errcode(pInfo, " -> %s\n", rc); 1172 | return rc; 1173 | } 1174 | 1175 | /* 1176 | ** Truncate a cevfs-file. 1177 | */ 1178 | static int cevfsTruncate(sqlite3_file *pFile, sqlite_int64 size){ 1179 | cevfs_file *p = (cevfs_file *)pFile; 1180 | cevfs_info *pInfo = p->pInfo; 1181 | int rc; 1182 | CEVFS_PRINTF(pInfo, "%s.xTruncate(%s,%lld)", pInfo->zVfsName, p->zFName, size); 1183 | rc = p->pReal->pMethods->xTruncate(p->pReal, size); 1184 | CEVFS_PRINTF(pInfo, " -> %d\n", rc); 1185 | return rc; 1186 | } 1187 | 1188 | /* 1189 | ** Sync a cevfs-file. 1190 | */ 1191 | static int cevfsSync(sqlite3_file *pFile, int flags){ 1192 | cevfs_file *p = (cevfs_file *)pFile; 1193 | cevfs_info *pInfo = p->pInfo; 1194 | int rc; 1195 | int i; 1196 | char zBuf[100]; 1197 | memcpy(zBuf, "|0", 3); 1198 | i = 0; 1199 | if( flags & SQLITE_SYNC_FULL ) strappend(zBuf, &i, "|FULL"); 1200 | else if( flags & SQLITE_SYNC_NORMAL ) strappend(zBuf, &i, "|NORMAL"); 1201 | if( flags & SQLITE_SYNC_DATAONLY ) strappend(zBuf, &i, "|DATAONLY"); 1202 | if( flags & ~(SQLITE_SYNC_FULL|SQLITE_SYNC_DATAONLY) ){ 1203 | sqlite3_snprintf(sizeof(zBuf)-i, &zBuf[i], "|0x%x", flags); 1204 | } 1205 | CEVFS_PRINTF(pInfo, "%s.xSync(%s,%s)", pInfo->zVfsName, p->zFName, &zBuf[1]); 1206 | rc = p->pReal->pMethods->xSync(p->pReal, flags); 1207 | CEVFS_PRINTF(pInfo, " -> %d\n", rc); 1208 | return rc; 1209 | } 1210 | 1211 | /* 1212 | ** Return ficticious uncompressed file size based on number of pages from source pager 1213 | ** otherwise internal checks in pager.c will fail. 1214 | */ 1215 | static int cevfsFileSize(sqlite3_file *pFile, sqlite_int64 *pSize){ 1216 | cevfs_file *p = (cevfs_file *)pFile; 1217 | cevfs_info *pInfo = p->pInfo; 1218 | cevfs_header *header = &p->cevfsHeader; 1219 | int rc; 1220 | CEVFS_PRINTF(pInfo, "%s.xFileSize(%s)", pInfo->zVfsName, p->zFName); 1221 | if( p->pPager ){ 1222 | *pSize = header->uppPageFile * header->uppPgSz; 1223 | rc = SQLITE_OK; 1224 | }else{ 1225 | rc = p->pReal->pMethods->xFileSize(p->pReal, pSize); 1226 | } 1227 | cevfs_print_errcode(pInfo, " -> %s,", rc); 1228 | CEVFS_PRINTF(pInfo, " size=%lld\n", *pSize); 1229 | return rc; 1230 | } 1231 | 1232 | /* 1233 | ** Return the name of a lock. 1234 | */ 1235 | static const char *lockName(int eLock){ 1236 | const char *azLockNames[] = { 1237 | "NONE", "SHARED", "RESERVED", "PENDING", "EXCLUSIVE" 1238 | }; 1239 | if( eLock<0 || eLock>=sizeof(azLockNames)/sizeof(azLockNames[0]) ){ 1240 | return "???"; 1241 | }else{ 1242 | return azLockNames[eLock]; 1243 | } 1244 | } 1245 | 1246 | /* 1247 | ** Lock a cevfs-file. 1248 | ** Never lock database file for upper pager as it doesn't directly control database file anymore. 1249 | */ 1250 | static int cevfsLock(sqlite3_file *pFile, int eLock){ 1251 | cevfs_file *p = (cevfs_file *)pFile; 1252 | cevfs_info *pInfo = p->pInfo; 1253 | int rc = SQLITE_OK; 1254 | CEVFS_PRINTF(pInfo, "%s.xLock(%s,%s) BYPASS", pInfo->zVfsName, p->zFName, lockName(eLock)); 1255 | cevfs_print_errcode(pInfo, " -> %s\n", rc); 1256 | return rc; 1257 | } 1258 | 1259 | /* 1260 | ** Unlock a cevfs-file. 1261 | ** Never unlock database file for upper pager as it doesn't directly control database file anymore. 1262 | */ 1263 | static int cevfsUnlock(sqlite3_file *pFile, int eLock){ 1264 | cevfs_file *p = (cevfs_file *)pFile; 1265 | cevfs_info *pInfo = p->pInfo; 1266 | int rc = SQLITE_OK; 1267 | CEVFS_PRINTF(pInfo, "%s.xUnlock(%s,%s) BYPASS", pInfo->zVfsName, p->zFName, lockName(eLock)); 1268 | cevfs_print_errcode(pInfo, " -> %s\n", rc); 1269 | return rc; 1270 | } 1271 | 1272 | /* 1273 | ** Check if another file-handle holds a RESERVED lock on a cevfs-file. 1274 | ** Bypass checks here since upper pager doesn't directly control database file anymore. 1275 | */ 1276 | static int cevfsCheckReservedLock(sqlite3_file *pFile, int *pResOut){ 1277 | cevfs_file *p = (cevfs_file *)pFile; 1278 | cevfs_info *pInfo = p->pInfo; 1279 | int rc = SQLITE_OK; 1280 | *pResOut = 0; // not locked 1281 | CEVFS_PRINTF(pInfo, "%s.xCheckReservedLock(%s,%d) BYPASS", pInfo->zVfsName, p->zFName); 1282 | cevfs_print_errcode(pInfo, " -> %s", rc); 1283 | CEVFS_PRINTF(pInfo, ", out=%d\n", *pResOut); 1284 | CEVFS_PRINTF(pInfo, "\n"); 1285 | return rc; 1286 | } 1287 | 1288 | static int cevfsPragma(sqlite3_file *pFile, const char *op, const char *arg){ 1289 | cevfs_file *p = (cevfs_file *)pFile; 1290 | int rc = SQLITE_OK; 1291 | if( strcmp(op, "page_size")==0 ){ 1292 | p->cevfsHeader.uppPgSz = (u32)sqlite3Atoi(arg); 1293 | } 1294 | return rc; 1295 | } 1296 | 1297 | /* 1298 | ** File control method. For custom operations on a cevfs-file. 1299 | */ 1300 | static int cevfsFileControl(sqlite3_file *pFile, int op, void *pArg){ 1301 | cevfs_file *p = (cevfs_file *)pFile; 1302 | cevfs_info *pInfo = p->pInfo; 1303 | int rc; 1304 | char zBuf[100]; 1305 | char *zOp; 1306 | switch( op ){ 1307 | #ifdef SQLITE_DEBUG 1308 | case SQLITE_FCNTL_LOCKSTATE: zOp = "LOCKSTATE"; break; 1309 | case SQLITE_GET_LOCKPROXYFILE: zOp = "GET_LOCKPROXYFILE"; break; 1310 | case SQLITE_SET_LOCKPROXYFILE: zOp = "SET_LOCKPROXYFILE"; break; 1311 | case SQLITE_LAST_ERRNO: zOp = "LAST_ERRNO"; break; 1312 | case SQLITE_FCNTL_SIZE_HINT: { 1313 | sqlite3_snprintf(sizeof(zBuf), zBuf, "SIZE_HINT,%lld", *(sqlite3_int64*)pArg); 1314 | zOp = zBuf; 1315 | break; 1316 | } 1317 | case SQLITE_FCNTL_CHUNK_SIZE: { 1318 | sqlite3_snprintf(sizeof(zBuf), zBuf, "CHUNK_SIZE,%d", *(int*)pArg); 1319 | zOp = zBuf; 1320 | break; 1321 | } 1322 | case SQLITE_FCNTL_FILE_POINTER: zOp = "FILE_POINTER"; break; 1323 | case SQLITE_FCNTL_SYNC_OMITTED: zOp = "SYNC_OMITTED"; break; 1324 | case SQLITE_FCNTL_WIN32_AV_RETRY: zOp = "WIN32_AV_RETRY"; break; 1325 | case SQLITE_FCNTL_PERSIST_WAL: zOp = "PERSIST_WAL"; break; 1326 | case SQLITE_FCNTL_OVERWRITE: zOp = "OVERWRITE"; break; 1327 | case SQLITE_FCNTL_VFSNAME: zOp = "VFSNAME"; break; 1328 | case SQLITE_FCNTL_TEMPFILENAME: zOp = "TEMPFILENAME"; break; 1329 | case SQLITE_FCNTL_DB_UNCHANGED: zOp = "DB_UNCHANGED"; break; 1330 | case SQLITE_FCNTL_MMAP_SIZE: { 1331 | sqlite3_snprintf(sizeof(zBuf), zBuf, "SQLITE_FCNTL_MMAP_SIZE,%d", *(int*)pArg); 1332 | zOp = zBuf; 1333 | break; 1334 | } 1335 | #endif 1336 | case SQLITE_FCNTL_PRAGMA: { 1337 | const char *const* a = (const char*const*)pArg; 1338 | sqlite3_snprintf(sizeof(zBuf), zBuf, "PRAGMA,[%s,%s]",a[1],a[2]); 1339 | zOp = zBuf; 1340 | cevfsPragma(pFile, a[1], a[2]); 1341 | break; 1342 | } 1343 | default: { 1344 | #ifdef SQLITE_DEBUG 1345 | sqlite3_snprintf(sizeof(zBuf), zBuf, "%d", op); 1346 | zOp = zBuf; 1347 | #endif 1348 | break; 1349 | } 1350 | } 1351 | CEVFS_PRINTF(pInfo, "%s.xFileControl(%s,%s)", pInfo->zVfsName, p->zFName, zOp); 1352 | rc = p->pReal->pMethods->xFileControl(p->pReal, op, pArg); 1353 | cevfs_print_errcode(pInfo, " -> %s\n", rc); 1354 | 1355 | if( op==SQLITE_FCNTL_VFSNAME && rc==SQLITE_OK ){ 1356 | *(char**)pArg = sqlite3_mprintf("cevfs.%s/%z", pInfo->zVfsName, *(char**)pArg); 1357 | } 1358 | 1359 | if( (op==SQLITE_FCNTL_PRAGMA || op==SQLITE_FCNTL_TEMPFILENAME) 1360 | && rc==SQLITE_OK && *(char**)pArg ){ 1361 | CEVFS_PRINTF(pInfo, "%s.xFileControl(%s,%s) returns %s", pInfo->zVfsName, p->zFName, zOp, *(char**)pArg); 1362 | } 1363 | return rc; 1364 | } 1365 | 1366 | /* 1367 | ** Return the sector-size in bytes for a cevfs-file. 1368 | */ 1369 | static int cevfsSectorSize(sqlite3_file *pFile){ 1370 | cevfs_file *p = (cevfs_file *)pFile; 1371 | cevfs_info *pInfo = p->pInfo; 1372 | int rc; 1373 | CEVFS_PRINTF(pInfo, "%s.xSectorSize(%s)", pInfo->zVfsName, p->zFName); 1374 | rc = p->pReal->pMethods->xSectorSize(p->pReal); 1375 | CEVFS_PRINTF(pInfo, " -> %d\n", rc); 1376 | return rc; 1377 | } 1378 | 1379 | /* 1380 | ** Return the device characteristic flags supported by a cevfs-file. 1381 | */ 1382 | static int cevfsDeviceCharacteristics(sqlite3_file *pFile){ 1383 | cevfs_file *p = (cevfs_file *)pFile; 1384 | cevfs_info *pInfo = p->pInfo; 1385 | int rc; 1386 | CEVFS_PRINTF(pInfo, "%s.xDeviceCharacteristics(%s)", pInfo->zVfsName, p->zFName); 1387 | rc = p->pReal->pMethods->xDeviceCharacteristics(p->pReal); 1388 | CEVFS_PRINTF(pInfo, " -> 0x%08x\n", rc); 1389 | return rc; 1390 | } 1391 | 1392 | /* 1393 | ** Shared-memory operations. 1394 | */ 1395 | static int cevfsShmLock(sqlite3_file *pFile, int ofst, int n, int flags){ 1396 | cevfs_file *p = (cevfs_file *)pFile; 1397 | cevfs_info *pInfo = p->pInfo; 1398 | int rc; 1399 | char zLck[100]; 1400 | int i = 0; 1401 | memcpy(zLck, "|0", 3); 1402 | if( flags & SQLITE_SHM_UNLOCK ) strappend(zLck, &i, "|UNLOCK"); 1403 | if( flags & SQLITE_SHM_LOCK ) strappend(zLck, &i, "|LOCK"); 1404 | if( flags & SQLITE_SHM_SHARED ) strappend(zLck, &i, "|SHARED"); 1405 | if( flags & SQLITE_SHM_EXCLUSIVE ) strappend(zLck, &i, "|EXCLUSIVE"); 1406 | if( flags & ~(0xf) ){ 1407 | sqlite3_snprintf(sizeof(zLck)-i, &zLck[i], "|0x%x", flags); 1408 | } 1409 | CEVFS_PRINTF(pInfo, "%s.xShmLock(%s,ofst=%d,n=%d,%s)", pInfo->zVfsName, p->zFName, ofst, n, &zLck[1]); 1410 | rc = p->pReal->pMethods->xShmLock(p->pReal, ofst, n, flags); 1411 | cevfs_print_errcode(pInfo, " -> %s\n", rc); 1412 | return rc; 1413 | } 1414 | static int cevfsShmMap( 1415 | sqlite3_file *pFile, 1416 | int iRegion, 1417 | int szRegion, 1418 | int isWrite, 1419 | void volatile **pp 1420 | ){ 1421 | cevfs_file *p = (cevfs_file *)pFile; 1422 | cevfs_info *pInfo = p->pInfo; 1423 | int rc; 1424 | CEVFS_PRINTF(pInfo, "%s.xShmMap(%s,iRegion=%d,szRegion=%d,isWrite=%d,*)", pInfo->zVfsName, p->zFName, iRegion, szRegion, isWrite); 1425 | rc = p->pReal->pMethods->xShmMap(p->pReal, iRegion, szRegion, isWrite, pp); 1426 | cevfs_print_errcode(pInfo, " -> %s\n", rc); 1427 | return rc; 1428 | } 1429 | static void cevfsShmBarrier(sqlite3_file *pFile){ 1430 | cevfs_file *p = (cevfs_file *)pFile; 1431 | cevfs_info *pInfo = p->pInfo; 1432 | CEVFS_PRINTF(pInfo, "%s.xShmBarrier(%s)\n", pInfo->zVfsName, p->zFName); 1433 | p->pReal->pMethods->xShmBarrier(p->pReal); 1434 | } 1435 | static int cevfsShmUnmap(sqlite3_file *pFile, int delFlag){ 1436 | cevfs_file *p = (cevfs_file *)pFile; 1437 | cevfs_info *pInfo = p->pInfo; 1438 | int rc; 1439 | CEVFS_PRINTF(pInfo, "%s.xShmUnmap(%s,delFlag=%d)", pInfo->zVfsName, p->zFName, delFlag); 1440 | rc = p->pReal->pMethods->xShmUnmap(p->pReal, delFlag); 1441 | cevfs_print_errcode(pInfo, " -> %s\n", rc); 1442 | return rc; 1443 | } 1444 | 1445 | 1446 | static void cevfsPageReinit(DbPage *pData){ 1447 | 1448 | } 1449 | 1450 | /* 1451 | ** Open a cevfs file handle. 1452 | */ 1453 | static int cevfsOpen( 1454 | sqlite3_vfs *pVfs, 1455 | const char *_zName, 1456 | sqlite3_file *pFile, 1457 | int flags, 1458 | int *pOutFlags 1459 | ){ 1460 | // TODO: check to make sure db is not already open. 1461 | 1462 | u8 nReserve; 1463 | int rc; 1464 | 1465 | cevfs_file *p = (cevfs_file *)pFile; 1466 | cevfs_info *pInfo = (cevfs_info*)pVfs->pAppData; 1467 | sqlite3_vfs *pRoot = pInfo->pRootVfs; 1468 | u32 nParamBlockSz = 0; 1469 | 1470 | // Zero-initialize 1471 | int offset = sizeof(p->base)+sizeof(*(p->pReal)); 1472 | int size = sizeof(*p)-offset; 1473 | memset((void*)p + offset, 0, size); 1474 | 1475 | // Initialize 1476 | p->pInfo = pInfo; 1477 | const char *zName = cevfsMapPath(p, _zName, NULL); 1478 | p->zFName = zName ? fileTail(zName) : ""; 1479 | p->pReal = (sqlite3_file *)&p[1]; 1480 | p->cevfsHeader.schema = CEVFS_FILE_SCHEMA_NO; 1481 | p->cevfsHeader.currPgno = CEVFS_FIRST_MAPPED_PAGE; 1482 | 1483 | // Set upper page size from temp storage else use default 1484 | p->cevfsHeader.uppPgSz = pInfo->upperPgSize ? pInfo->upperPgSize : SQLITE_DEFAULT_PAGE_SIZE; 1485 | 1486 | // We need this for import 1487 | pInfo->pFile = p; 1488 | 1489 | // Set readonly flag 1490 | if( pInfo->cerod_activated && strcmp(pInfo->zVfsName, "cevfs-cerod")==0 ){ 1491 | p->bReadOnly = 1; 1492 | }else if( flags & SQLITE_OPEN_READONLY ){ 1493 | p->bReadOnly = 1; 1494 | } 1495 | 1496 | // Process URI parameters 1497 | if( flags & SQLITE_OPEN_URI ){ 1498 | // block_size 1499 | const char *zParamBlockSize = sqlite3_uri_parameter(_zName, "block_size"); 1500 | if( zParamBlockSize ) nParamBlockSz = (u32)sqlite3Atoi(zParamBlockSize); 1501 | } 1502 | 1503 | // open file 1504 | rc = pRoot->xOpen(pRoot, zName, p->pReal, flags, pOutFlags); 1505 | CEVFS_PRINTF(pInfo, "%s.xOpen(%s,flags=0x%x)",pInfo->zVfsName, p->zFName, flags); 1506 | 1507 | if( rc==SQLITE_OK ){ 1508 | // hook up I/O methods 1509 | if( p->pReal->pMethods ){ 1510 | sqlite3_io_methods *pNew = sqlite3_malloc( sizeof(*pNew) ); 1511 | const sqlite3_io_methods *pSub = p->pReal->pMethods; 1512 | memset(pNew, 0, sizeof(*pNew)); 1513 | pNew->iVersion = pSub->iVersion; 1514 | pNew->xClose = cevfsClose; 1515 | pNew->xRead = cevfsRead; 1516 | pNew->xWrite = cevfsWrite; 1517 | pNew->xTruncate = cevfsTruncate; 1518 | pNew->xSync = cevfsSync; 1519 | pNew->xFileSize = cevfsFileSize; 1520 | pNew->xLock = cevfsLock; 1521 | pNew->xUnlock = cevfsUnlock; 1522 | pNew->xCheckReservedLock = cevfsCheckReservedLock; 1523 | pNew->xFileControl = cevfsFileControl; 1524 | pNew->xSectorSize = cevfsSectorSize; 1525 | pNew->xDeviceCharacteristics = cevfsDeviceCharacteristics; 1526 | if( pNew->iVersion>=2 ){ 1527 | pNew->xShmMap = pSub->xShmMap ? cevfsShmMap : 0; 1528 | pNew->xShmLock = pSub->xShmLock ? cevfsShmLock : 0; 1529 | pNew->xShmBarrier = pSub->xShmBarrier ? cevfsShmBarrier : 0; 1530 | pNew->xShmUnmap = pSub->xShmUnmap ? cevfsShmUnmap : 0; 1531 | } 1532 | pFile->pMethods = pNew; 1533 | } 1534 | 1535 | // create pager to handle I/O to compressed/encrypted underlying db 1536 | if( flags & (SQLITE_OPEN_MAIN_DB | SQLITE_OPEN_TEMP_DB | SQLITE_OPEN_TRANSIENT_DB) ){ 1537 | if( (rc = sqlite3PagerOpen(pInfo->pRootVfs, &p->pPager, zName, EXTRA_SIZE, 0, flags, cevfsPageReinit))==SQLITE_OK){ 1538 | if( rc==SQLITE_OK ){ 1539 | sqlite3PagerSetJournalMode(p->pPager, PAGER_JOURNALMODE_DELETE); 1540 | // sqlite3PagerJournalSizeLimit(p->pPager, -1); 1541 | // rc = sqlite3PagerLockingMode(p->pPager, PAGER_LOCKINGMODE_NORMAL); 1542 | // sqlite3PagerSetMmapLimit(pBt->pPager, db->szMmap); /* advisory, except if 0 */ 1543 | if( (rc = sqlite3PagerReadFileheader(p->pPager,sizeof(p->zDbHeader),p->zDbHeader)) == SQLITE_OK ){ 1544 | p->pageSize = (p->zDbHeader[16]<<8) | (p->zDbHeader[17]<<16); 1545 | if( p->pageSize<512 || p->pageSize>SQLITE_MAX_PAGE_SIZE 1546 | || ((p->pageSize-1)&p->pageSize)!=0 ){ 1547 | p->pageSize = nParamBlockSz; // if 0, sqlite3PagerSetPagesize will set page size 1548 | nReserve = 0; 1549 | }else{ 1550 | nReserve = p->zDbHeader[20]; 1551 | p->lwrPageFile = sqlite3Get4byte(p->zDbHeader+28); 1552 | } 1553 | sqlite3PagerSetMmapLimit(p->pPager, 0); 1554 | if( (rc = sqlite3PagerSetPagesize(p->pPager, &p->pageSize, nReserve)) == SQLITE_OK ){ 1555 | p->usableSize = p->pageSize - nReserve; 1556 | sqlite3PagerSetCachesize(p->pPager, SQLITE_DEFAULT_CACHE_SIZE); 1557 | rc = cevfsPagerLock(p); 1558 | } 1559 | 1560 | // Call user xAutoDetect to set up VFS methods 1561 | if (pInfo->xAutoDetect) { 1562 | pInfo->xAutoDetect(pInfo->pCtx, _zName, (const char *)p->zDbHeader+6, &p->nEncIvSz, &p->vfsMethods); 1563 | if (p->vfsMethods.xCompressBound && p->vfsMethods.xCompress && p->vfsMethods.xUncompress) 1564 | p->bCompressionEnabled = true; 1565 | if (p->vfsMethods.xEncrypt && p->vfsMethods.xDecrypt) 1566 | p->bEncryptionEnabled = true; 1567 | } 1568 | } 1569 | }else{ 1570 | cevfsClose(pFile); 1571 | } 1572 | } 1573 | } 1574 | } 1575 | 1576 | cevfs_print_errcode(pInfo, " -> %s", rc); 1577 | if( pOutFlags ){ 1578 | CEVFS_PRINTF(pInfo, ", outFlags=0x%x\n", *pOutFlags); 1579 | }else{ 1580 | CEVFS_PRINTF(pInfo, "\n"); 1581 | } 1582 | return rc; 1583 | } 1584 | 1585 | /* 1586 | ** Delete the file located at zPath. If the dirSync argument is true, 1587 | ** ensure the file-system modifications are synced to disk before 1588 | ** returning. 1589 | */ 1590 | static int cevfsDelete(sqlite3_vfs *pVfs, const char *_zPath, int dirSync){ 1591 | cevfs_info *pInfo = (cevfs_info*)pVfs->pAppData; 1592 | sqlite3_vfs *pRoot = pInfo->pRootVfs; 1593 | int rc; 1594 | bool bMustRelease; 1595 | 1596 | char *zPath = cevfsMapPath(NULL, _zPath, &bMustRelease); 1597 | CEVFS_PRINTF(pInfo, "%s.xDelete(\"%s\",%d)", pInfo->zVfsName, zPath, dirSync); 1598 | rc = pRoot->xDelete(pRoot, zPath, dirSync); 1599 | cevfs_print_errcode(pInfo, " -> %s\n", rc); 1600 | if (bMustRelease)sqlite3_free(zPath); 1601 | return rc; 1602 | } 1603 | 1604 | /* 1605 | ** Test for access permissions. 1606 | ** Return true via *pResOut if the requested permission 1607 | ** is available, or false otherwise. 1608 | */ 1609 | static int cevfsAccess( 1610 | sqlite3_vfs *pVfs, 1611 | const char *_zPath, 1612 | int flags, 1613 | int *pResOut 1614 | ){ 1615 | cevfs_info *pInfo = (cevfs_info*)pVfs->pAppData; 1616 | sqlite3_vfs *pRoot = pInfo->pRootVfs; 1617 | bool bMustRelease; 1618 | char *zPath = cevfsMapPath(NULL, _zPath, &bMustRelease); 1619 | int rc = SQLITE_OK; 1620 | 1621 | CEVFS_PRINTF(pInfo, "%s.xAccess(\"%s\",%d)", pInfo->zVfsName, zPath, flags); 1622 | rc = pRoot->xAccess(pRoot, zPath, flags, pResOut); 1623 | cevfs_print_errcode(pInfo, " -> %s", rc); 1624 | CEVFS_PRINTF(pInfo, ", out=%d\n", *pResOut); 1625 | if (bMustRelease) sqlite3_free(zPath); 1626 | return rc; 1627 | } 1628 | 1629 | /* 1630 | ** Populate buffer zOut with the full canonical pathname corresponding 1631 | ** to the pathname in zPath. zOut is guaranteed to point to a buffer 1632 | ** of at least (DEVSYM_MAX_PATHNAME+1) bytes. 1633 | */ 1634 | static int cevfsFullPathname( 1635 | sqlite3_vfs *pVfs, 1636 | const char *zPath, 1637 | int nOut, 1638 | char *zOut 1639 | ){ 1640 | cevfs_info *pInfo = (cevfs_info*)pVfs->pAppData; 1641 | sqlite3_vfs *pRoot = pInfo->pRootVfs; 1642 | int rc; 1643 | CEVFS_PRINTF(pInfo, "%s.xFullPathname(\"%s\")", pInfo->zVfsName, zPath); 1644 | rc = pRoot->xFullPathname(pRoot, zPath, nOut, zOut); 1645 | cevfs_print_errcode(pInfo, " -> %s", rc); 1646 | CEVFS_PRINTF(pInfo, ", out=\"%.*s\"\n", nOut, zOut); 1647 | return rc; 1648 | } 1649 | 1650 | /* 1651 | ** Open the dynamic library located at zPath and return a handle. 1652 | */ 1653 | static void *cevfsDlOpen(sqlite3_vfs *pVfs, const char *zPath){ 1654 | cevfs_info *pInfo = (cevfs_info*)pVfs->pAppData; 1655 | sqlite3_vfs *pRoot = pInfo->pRootVfs; 1656 | CEVFS_PRINTF(pInfo, "%s.xDlOpen(\"%s\")\n", pInfo->zVfsName, zPath); 1657 | return pRoot->xDlOpen(pRoot, zPath); 1658 | } 1659 | 1660 | /* 1661 | ** Populate the buffer zErrMsg (size nByte bytes) with a human readable 1662 | ** utf-8 string describing the most recent error encountered associated 1663 | ** with dynamic libraries. 1664 | */ 1665 | static void cevfsDlError(sqlite3_vfs *pVfs, int nByte, char *zErrMsg){ 1666 | cevfs_info *pInfo = (cevfs_info*)pVfs->pAppData; 1667 | sqlite3_vfs *pRoot = pInfo->pRootVfs; 1668 | CEVFS_PRINTF(pInfo, "%s.xDlError(%d)", pInfo->zVfsName, nByte); 1669 | pRoot->xDlError(pRoot, nByte, zErrMsg); 1670 | CEVFS_PRINTF(pInfo, " -> \"%s\"", zErrMsg); 1671 | } 1672 | 1673 | /* 1674 | ** Return a pointer to the symbol zSymbol in the dynamic library pHandle. 1675 | */ 1676 | static void (*cevfsDlSym(sqlite3_vfs *pVfs,void *p,const char *zSym))(void){ 1677 | cevfs_info *pInfo = (cevfs_info*)pVfs->pAppData; 1678 | sqlite3_vfs *pRoot = pInfo->pRootVfs; 1679 | CEVFS_PRINTF(pInfo, "%s.xDlSym(\"%s\")\n", pInfo->zVfsName, zSym); 1680 | return pRoot->xDlSym(pRoot, p, zSym); 1681 | } 1682 | 1683 | /* 1684 | ** Close the dynamic library handle pHandle. 1685 | */ 1686 | static void cevfsDlClose(sqlite3_vfs *pVfs, void *pHandle){ 1687 | cevfs_info *pInfo = (cevfs_info*)pVfs->pAppData; 1688 | sqlite3_vfs *pRoot = pInfo->pRootVfs; 1689 | CEVFS_PRINTF(pInfo, "%s.xDlOpen()\n", pInfo->zVfsName); 1690 | pRoot->xDlClose(pRoot, pHandle); 1691 | } 1692 | 1693 | /* 1694 | ** Populate the buffer pointed to by zBufOut with nByte bytes of 1695 | ** random data. 1696 | */ 1697 | static int cevfsRandomness(sqlite3_vfs *pVfs, int nByte, char *zBufOut){ 1698 | cevfs_info *pInfo = (cevfs_info*)pVfs->pAppData; 1699 | sqlite3_vfs *pRoot = pInfo->pRootVfs; 1700 | CEVFS_PRINTF(pInfo, "%s.xRandomness(%d)\n", pInfo->zVfsName, nByte); 1701 | return pRoot->xRandomness(pRoot, nByte, zBufOut); 1702 | } 1703 | 1704 | /* 1705 | ** Sleep for nMicro microseconds. Return the number of microseconds 1706 | ** actually slept. 1707 | */ 1708 | static int cevfsSleep(sqlite3_vfs *pVfs, int nMicro){ 1709 | cevfs_info *pInfo = (cevfs_info*)pVfs->pAppData; 1710 | sqlite3_vfs *pRoot = pInfo->pRootVfs; 1711 | return pRoot->xSleep(pRoot, nMicro); 1712 | } 1713 | 1714 | /* 1715 | ** Return the current time as a Julian Day number in *pTimeOut. 1716 | */ 1717 | static int cevfsCurrentTime(sqlite3_vfs *pVfs, double *pTimeOut){ 1718 | cevfs_info *pInfo = (cevfs_info*)pVfs->pAppData; 1719 | sqlite3_vfs *pRoot = pInfo->pRootVfs; 1720 | return pRoot->xCurrentTime(pRoot, pTimeOut); 1721 | } 1722 | static int cevfsCurrentTimeInt64(sqlite3_vfs *pVfs, sqlite3_int64 *pTimeOut){ 1723 | cevfs_info *pInfo = (cevfs_info*)pVfs->pAppData; 1724 | sqlite3_vfs *pRoot = pInfo->pRootVfs; 1725 | return pRoot->xCurrentTimeInt64(pRoot, pTimeOut); 1726 | } 1727 | 1728 | /* 1729 | ** Return the emost recent error code and message 1730 | */ 1731 | static int cevfsGetLastError(sqlite3_vfs *pVfs, int iErr, char *zErr){ 1732 | cevfs_info *pInfo = (cevfs_info*)pVfs->pAppData; 1733 | sqlite3_vfs *pRoot = pInfo->pRootVfs; 1734 | return pRoot->xGetLastError(pRoot, iErr, zErr); 1735 | } 1736 | 1737 | /* 1738 | ** Clients invoke cevfs_create_vfs to construct a new cevfs. 1739 | ** 1740 | ** Return SQLITE_OK on success. 1741 | ** 1742 | ** SQLITE_NOMEM is returned in the case of a memory allocation error. 1743 | ** SQLITE_NOTFOUND is returned if zOldVfsName does not exist. 1744 | */ 1745 | int cevfs_create_vfs( 1746 | char const *zName, // Name of the newly constructed VFS. 1747 | char const *zParent, // Name of the underlying VFS. NULL to use default. 1748 | void *pCtx, // Context pointer to be passed to CEVFS methods. 1749 | t_xAutoDetect xAutoDetect, // Pointer to xAutoDetect custom supplied function. 1750 | int makeDefault 1751 | ){ 1752 | sqlite3_vfs *pNew; 1753 | sqlite3_vfs *pRoot; 1754 | cevfs_info *pInfo; 1755 | int nName; 1756 | int nByte; 1757 | 1758 | // Allow parameters to be passed with database filename in URI form. 1759 | sqlite3_config(SQLITE_CONFIG_URI, 1); 1760 | 1761 | // Don't register VFS with same name more than once 1762 | if( sqlite3_vfs_find(zName) ) 1763 | return CEVFS_ERROR_VFS_ALREADY_EXISTS; 1764 | 1765 | pRoot = sqlite3_vfs_find(zParent); 1766 | if( pRoot==0 ) return SQLITE_NOTFOUND; 1767 | nName = (int)strlen(zName); 1768 | 1769 | // Allocate memory for a new sqlite3_vfs, cevfs_info and the name of the new VFS. 1770 | nByte = sizeof(*pNew) + sizeof(*pInfo) + nName + 1; 1771 | pNew = sqlite3_malloc( nByte ); 1772 | if( pNew==0 ) return SQLITE_NOMEM; 1773 | memset(pNew, 0, nByte); 1774 | 1775 | // Hook up the rest of the allocated memory 1776 | pInfo = (cevfs_info*)&pNew[1]; 1777 | pNew->zName = (char*)&pInfo[1]; 1778 | 1779 | // Intialize data 1780 | memcpy((char*)&pInfo[1], zName, nName+1); 1781 | pNew->iVersion = pRoot->iVersion; 1782 | pNew->szOsFile = pRoot->szOsFile + sizeof(cevfs_file); 1783 | pNew->mxPathname = pRoot->mxPathname; 1784 | pNew->pAppData = pInfo; 1785 | pNew->xOpen = cevfsOpen; 1786 | pNew->xDelete = cevfsDelete; 1787 | pNew->xAccess = cevfsAccess; 1788 | pNew->xFullPathname = cevfsFullPathname; 1789 | pNew->xDlOpen = pRoot->xDlOpen==0 ? 0 : cevfsDlOpen; 1790 | pNew->xDlError = pRoot->xDlError==0 ? 0 : cevfsDlError; 1791 | pNew->xDlSym = pRoot->xDlSym==0 ? 0 : cevfsDlSym; 1792 | pNew->xDlClose = pRoot->xDlClose==0 ? 0 : cevfsDlClose; 1793 | pNew->xRandomness = cevfsRandomness; 1794 | pNew->xSleep = cevfsSleep; 1795 | pNew->xCurrentTime = cevfsCurrentTime; 1796 | pNew->xGetLastError = pRoot->xGetLastError==0 ? 0 : cevfsGetLastError; 1797 | if( pNew->iVersion>=2 ){ 1798 | pNew->xCurrentTimeInt64 = pRoot->xCurrentTimeInt64==0 ? 0 : cevfsCurrentTimeInt64; 1799 | if( pNew->iVersion>=3 ){ 1800 | pNew->xSetSystemCall = 0; 1801 | pNew->xGetSystemCall = 0; 1802 | pNew->xNextSystemCall = 0; 1803 | } 1804 | } 1805 | pInfo->pRootVfs = pRoot; 1806 | pInfo->zVfsName = pNew->zName; 1807 | pInfo->pCevfsVfs = pNew; 1808 | pInfo->pCtx = pCtx; 1809 | pInfo->xAutoDetect = xAutoDetect; 1810 | 1811 | CEVFS_PRINTF(pInfo, "%s.enabled_for(\"%s\")\n", pInfo->zVfsName, pRoot->zName); 1812 | return sqlite3_vfs_register(pNew, makeDefault); 1813 | } 1814 | 1815 | static int _cevfs_destroy_vfs(sqlite3_vfs *pVfs) { 1816 | sqlite3_free(pVfs); 1817 | return SQLITE_OK; 1818 | } 1819 | 1820 | int cevfs_destroy_vfs(const char *zName){ 1821 | sqlite3_vfs *pVfs = sqlite3_vfs_find(zName); 1822 | if( pVfs ){ 1823 | //cevfs_info *pInfo = (cevfs_info *)pVfs->pAppData; 1824 | return _cevfs_destroy_vfs(pVfs); 1825 | } 1826 | return CEVFS_ERROR_VFS_DOES_NOT_EXIST; 1827 | } 1828 | 1829 | int cevfs_build( 1830 | const char *zSrcFilename, 1831 | const char *zDestFilename, 1832 | const char *vfsName, 1833 | void *pCtx, 1834 | t_xAutoDetect xAutoDetect 1835 | ){ 1836 | int rc = SQLITE_OK; 1837 | unsigned char zDbHeader[100]; 1838 | sqlite3_vfs *pDestVfs = NULL; 1839 | 1840 | // cevfs_create_vfs must be done early enough to avoid SQLITE_MISUSE error 1841 | rc = cevfs_create_vfs(vfsName, NULL, pCtx, xAutoDetect, 0); 1842 | if( rc==SQLITE_OK || rc==CEVFS_ERROR_VFS_ALREADY_EXISTS ){ 1843 | pDestVfs = sqlite3_vfs_find(vfsName); 1844 | } 1845 | 1846 | if( pDestVfs ){ 1847 | sqlite3_vfs *pSrcVfs = sqlite3_vfs_find(NULL); 1848 | if( pSrcVfs ){ 1849 | Pager *pPager; 1850 | int vfsFlags = SQLITE_OPEN_READONLY | SQLITE_OPEN_MAIN_DB | SQLITE_OPEN_URI; 1851 | if( (rc = sqlite3PagerOpen(pSrcVfs, &pPager, zSrcFilename, EXTRA_SIZE, 0, vfsFlags, cevfsPageReinit))==SQLITE_OK ){ 1852 | sqlite3PagerSetJournalMode(pPager, PAGER_JOURNALMODE_OFF); 1853 | if( (rc = sqlite3PagerReadFileheader(pPager,sizeof(zDbHeader),zDbHeader)) == SQLITE_OK ){ 1854 | u32 pageSize = (zDbHeader[16]<<8) | (zDbHeader[17]<<16); 1855 | if( pageSize>=512 && pageSize<=SQLITE_MAX_PAGE_SIZE // validate range 1856 | && ((pageSize-1)&pageSize)==0 ){ // validate page size is a power of 2 1857 | u8 nReserve = zDbHeader[20]; 1858 | if( (rc = sqlite3PagerSetPagesize(pPager, &pageSize, nReserve)) == SQLITE_OK ){ 1859 | u32 fileChangeCounter = sqlite3Get4byte(zDbHeader+24); 1860 | u32 pageCount = sqlite3Get4byte(zDbHeader+28); 1861 | u32 versionValidForNumber = sqlite3Get4byte(zDbHeader+92); 1862 | 1863 | // If we didn't get page count, figure it out from the file size 1864 | if( !(pageCount>0 && fileChangeCounter==versionValidForNumber) ){ 1865 | struct stat st; 1866 | if( stat(zSrcFilename, &st)==0 ){ 1867 | off_t size = st.st_size; 1868 | pageCount = (u32)(size/pageSize); 1869 | } 1870 | } 1871 | 1872 | // lock pager, prepare to read 1873 | if( rc==SQLITE_OK && (rc = sqlite3PagerSharedLock(pPager))==SQLITE_OK ){ 1874 | // get destination ready to receive data 1875 | sqlite3 *pDb; 1876 | 1877 | // Must set upper page size before sqlite3_open_v2 1878 | // as cevfsOpen will be invoked and expecting this value. 1879 | cevfs_info *pInfo = (cevfs_info *)pDestVfs->pAppData; 1880 | pInfo->upperPgSize = pageSize; 1881 | 1882 | if( (rc = sqlite3_open_v2(zDestFilename, &pDb, SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE, vfsName))==SQLITE_OK ){ 1883 | // Needed for sqlite3 API shims 1884 | pInfo->pDb = pDb; 1885 | 1886 | DbPage *pPage1 = NULL; 1887 | // import all pages 1888 | for(Pgno pgno=0; pgnopFile, pData, pageSize, pageSize*pgno); 1897 | if( pgno==0 ){ 1898 | // To be deallocated later 1899 | pPage1 = pPage; 1900 | }else{ 1901 | sqlite3PagerUnref(pPage); 1902 | } 1903 | if( rc != SQLITE_OK ) break; 1904 | }else{ 1905 | break; 1906 | } 1907 | } 1908 | if (pPage1) sqlite3PagerUnref(pPage1); 1909 | sqlite3PagerCloseShim(pPager, pDb); 1910 | rc = sqlite3_close(pDb); 1911 | } 1912 | } 1913 | } 1914 | } else rc = SQLITE_CORRUPT; 1915 | } 1916 | } 1917 | } else rc = SQLITE_INTERNAL; 1918 | _cevfs_destroy_vfs(pDestVfs); 1919 | } 1920 | return rc; 1921 | } 1922 | -------------------------------------------------------------------------------- /cevfs/cevfs.h: -------------------------------------------------------------------------------- 1 | /** 2 | CEVFS 3 | Compression & Encryption VFS 4 | 5 | Copyright (c) 2016 Ryan Homer, Murage Inc. 6 | 7 | MIT License 8 | 9 | Permission is hereby granted, free of charge, to any person obtaining a copy 10 | of this software and associated documentation files (the "Software"), to deal 11 | in the Software without restriction, including without limitation the rights 12 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 13 | copies of the Software, and to permit persons to whom the Software is 14 | furnished to do so, subject to the following conditions: 15 | 16 | The above copyright notice and this permission notice shall be included in all 17 | copies or substantial portions of the Software. 18 | 19 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 20 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 21 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 22 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 23 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 24 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 25 | SOFTWARE. 26 | */ 27 | 28 | #ifndef __CEVFS_H__ 29 | #define __CEVFS_H__ 30 | 31 | #define CEVFS_OK 0 32 | 33 | #define CEVFS_ERROR 111 34 | #define CEVFS_ERROR_PAGE_SIZE_TOO_SMALL (CEVFS_ERROR | (1<<8)) 35 | #define CEVFS_ERROR_MALFORMED_KEY (CEVFS_ERROR | (2<<8)) 36 | #define CEVFS_ERROR_EXT_VERSION_TOO_OLD (CEVFS_ERROR | (3<<8)) 37 | #define CEVFS_ERROR_VFS_ALREADY_EXISTS (CEVFS_ERROR | (4<<8)) 38 | #define CEVFS_ERROR_VFS_DOES_NOT_EXIST (CEVFS_ERROR | (5<<8)) 39 | #define CEVFS_ERROR_COMPRESSION_FAILED (CEVFS_ERROR | (6<<8)) 40 | #define CEVFS_ERROR_DECOMPRESSION_FAILED (CEVFS_ERROR | (7<<8)) 41 | #define CEVFS_ERROR_ENCRYPTION_FAILED (CEVFS_ERROR | (8<<8)) 42 | #define CEVFS_ERROR_DECRYPTION_FAILED (CEVFS_ERROR | (9<<8)) 43 | 44 | struct CevfsMethods { 45 | void *pCtx; 46 | 47 | int (*xCompressBound)(void *pCtx, size_t nDataInSize); 48 | int (*xCompress) (void *pCtx, char *aDest, size_t *pnDataOutSize, char *aSrc, size_t nDataInSize); 49 | int (*xUncompress)(void *pCtx, char *aDest, size_t *pnDataOutSize, char *aSrc, size_t nDataInSize); 50 | 51 | int (*xEncrypt)( 52 | void *pCtx, // in: the context 53 | const void *pDataIn, // in: the unencrypted data 54 | size_t nDataInSize, // in: the size of the unencrypted data 55 | void *pIvOut, // out: the randomly created IV 56 | void **pDataOut, // out: the encrypted data 57 | size_t *nDataSizeOut, // out: size of encrypted data 58 | void *sqlite3_malloc(int n) // in: pointer to the sqlite3_malloc function 59 | ); 60 | 61 | int (*xDecrypt)( 62 | void *pCtx, 63 | const void *pDataIn, 64 | size_t nDataInSize, 65 | const void *pIvIn, 66 | void *pDataOut, 67 | size_t nDataBufferSizeOut, 68 | size_t *nDataSizeOut 69 | ); 70 | }; 71 | typedef struct CevfsMethods CevfsMethods; 72 | 73 | typedef int (*t_xAutoDetect)( 74 | void *pCtx, // Pointer to context passed in via 3rd param of cevfs_create_vfs. 75 | const char *zFile, // Pointer to buffer containing the database filename. 76 | const char *zHdr, // NULL if new database, otherwise database header after CEVFS- prefix. 77 | size_t *pEncIvSz, // Pointer to encryption initialization vector (IV) size. 78 | CevfsMethods*); // Pointer to compression/encryption methods. 79 | 80 | int cevfs_create_vfs( 81 | char const *zName, // Name of the newly constructed VFS. 82 | char const *zParent, // Name of the underlying VFS. NULL to use default. 83 | void *pCtx, // Context pointer to be passed to CEVFS methods. 84 | t_xAutoDetect, // xAutoDetect method to set up xMethods. 85 | int makeDefault // BOOL: Make this the default VFS? Typically false. 86 | ); 87 | 88 | int cevfs_destroy_vfs(const char *zName); 89 | 90 | /*! 91 | \brief Create new compresses/encrypted database from existing database. 92 | \param zSrcFilename Standard path to existing uncompressed, unencrypted SQLite database file. 93 | \param zDestFilename URI of non-existing destination database file with optional parameters (block_size, key) 94 | */ 95 | int cevfs_build( 96 | const char *zSrcFilename, // Source SQLite DB filename, including path. Can be a URI. 97 | const char *zDestFilename, // Destination SQLite DB filename, including path. Can be a URI. 98 | const char *vfsName, // This will be embedded into the header of the database file. 99 | void *pCtx, // Context pointer to be passed to CEVFS xMethods. 100 | t_xAutoDetect // xAutoDetect method to set up xMethods. 101 | ); 102 | 103 | #endif /* __CEVFS_H__ */ 104 | -------------------------------------------------------------------------------- /cevfs/cevfs.xcodeproj/project.pbxproj: -------------------------------------------------------------------------------- 1 | // !$*UTF8*$! 2 | { 3 | archiveVersion = 1; 4 | classes = { 5 | }; 6 | objectVersion = 46; 7 | objects = { 8 | 9 | /* Begin PBXBuildFile section */ 10 | 9442235E1D2027C60016E082 /* cevfs-all.c in Sources */ = {isa = PBXBuildFile; fileRef = 9442235D1D2027C60016E082 /* cevfs-all.c */; }; 11 | 944223611D20445D0016E082 /* cevfs.h in Headers */ = {isa = PBXBuildFile; fileRef = 9479F8D71C64EB4100901192 /* cevfs.h */; settings = {ATTRIBUTES = (Public, ); }; }; 12 | /* End PBXBuildFile section */ 13 | 14 | /* Begin PBXFileReference section */ 15 | 944223451D1F5EF20016E082 /* sqlite3-1.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = "sqlite3-1.c"; sourceTree = ""; }; 16 | 944223461D1F5EF20016E082 /* sqlite3-2.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = "sqlite3-2.c"; sourceTree = ""; }; 17 | 944223471D1F5EF20016E082 /* sqlite3-3.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = "sqlite3-3.c"; sourceTree = ""; }; 18 | 944223481D1F5EF20016E082 /* sqlite3-4.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = "sqlite3-4.c"; sourceTree = ""; }; 19 | 944223491D1F5EF20016E082 /* sqlite3-5.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = "sqlite3-5.c"; sourceTree = ""; }; 20 | 9442234A1D1F5EF20016E082 /* sqlite3-6.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = "sqlite3-6.c"; sourceTree = ""; }; 21 | 9442234B1D1F5EF20016E082 /* sqlite3-7.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = "sqlite3-7.c"; sourceTree = ""; }; 22 | 9442234C1D1F5EF20016E082 /* sqlite3-all.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = "sqlite3-all.c"; sourceTree = ""; }; 23 | 9442235D1D2027C60016E082 /* cevfs-all.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = "cevfs-all.c"; sourceTree = ""; }; 24 | 9479F8D71C64EB4100901192 /* cevfs.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = cevfs.h; sourceTree = ""; }; 25 | 9497BD111C53D9D600ADD79F /* libsqlite.a */ = {isa = PBXFileReference; explicitFileType = archive.ar; includeInIndex = 0; path = libsqlite.a; sourceTree = BUILT_PRODUCTS_DIR; }; 26 | 94D80F9C1C5716580091C9D9 /* cevfs.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = cevfs.c; sourceTree = ""; }; 27 | /* End PBXFileReference section */ 28 | 29 | /* Begin PBXFrameworksBuildPhase section */ 30 | 9497BD0E1C53D9D600ADD79F /* Frameworks */ = { 31 | isa = PBXFrameworksBuildPhase; 32 | buildActionMask = 2147483647; 33 | files = ( 34 | ); 35 | runOnlyForDeploymentPostprocessing = 0; 36 | }; 37 | /* End PBXFrameworksBuildPhase section */ 38 | 39 | /* Begin PBXGroup section */ 40 | 944223441D1F5EF20016E082 /* sqlite */ = { 41 | isa = PBXGroup; 42 | children = ( 43 | 944223451D1F5EF20016E082 /* sqlite3-1.c */, 44 | 944223461D1F5EF20016E082 /* sqlite3-2.c */, 45 | 944223471D1F5EF20016E082 /* sqlite3-3.c */, 46 | 944223481D1F5EF20016E082 /* sqlite3-4.c */, 47 | 944223491D1F5EF20016E082 /* sqlite3-5.c */, 48 | 9442234A1D1F5EF20016E082 /* sqlite3-6.c */, 49 | 9442234B1D1F5EF20016E082 /* sqlite3-7.c */, 50 | 9442234C1D1F5EF20016E082 /* sqlite3-all.c */, 51 | ); 52 | path = sqlite; 53 | sourceTree = ""; 54 | }; 55 | 9497BD081C53D9D600ADD79F = { 56 | isa = PBXGroup; 57 | children = ( 58 | 9479F8D71C64EB4100901192 /* cevfs.h */, 59 | 94D80F9C1C5716580091C9D9 /* cevfs.c */, 60 | 9442235D1D2027C60016E082 /* cevfs-all.c */, 61 | 944223441D1F5EF20016E082 /* sqlite */, 62 | 9497BD121C53D9D600ADD79F /* Products */, 63 | ); 64 | sourceTree = ""; 65 | }; 66 | 9497BD121C53D9D600ADD79F /* Products */ = { 67 | isa = PBXGroup; 68 | children = ( 69 | 9497BD111C53D9D600ADD79F /* libsqlite.a */, 70 | ); 71 | name = Products; 72 | sourceTree = ""; 73 | }; 74 | /* End PBXGroup section */ 75 | 76 | /* Begin PBXHeadersBuildPhase section */ 77 | 9497BD0F1C53D9D600ADD79F /* Headers */ = { 78 | isa = PBXHeadersBuildPhase; 79 | buildActionMask = 2147483647; 80 | files = ( 81 | 944223611D20445D0016E082 /* cevfs.h in Headers */, 82 | ); 83 | runOnlyForDeploymentPostprocessing = 0; 84 | }; 85 | /* End PBXHeadersBuildPhase section */ 86 | 87 | /* Begin PBXNativeTarget section */ 88 | 9497BD101C53D9D600ADD79F /* sqlite */ = { 89 | isa = PBXNativeTarget; 90 | buildConfigurationList = 9497BD151C53D9D600ADD79F /* Build configuration list for PBXNativeTarget "sqlite" */; 91 | buildPhases = ( 92 | 9497BD0D1C53D9D600ADD79F /* Sources */, 93 | 9497BD0E1C53D9D600ADD79F /* Frameworks */, 94 | 9497BD0F1C53D9D600ADD79F /* Headers */, 95 | ); 96 | buildRules = ( 97 | ); 98 | dependencies = ( 99 | ); 100 | name = sqlite; 101 | productName = sqlite; 102 | productReference = 9497BD111C53D9D600ADD79F /* libsqlite.a */; 103 | productType = "com.apple.product-type.library.static"; 104 | }; 105 | /* End PBXNativeTarget section */ 106 | 107 | /* Begin PBXProject section */ 108 | 9497BD091C53D9D600ADD79F /* Project object */ = { 109 | isa = PBXProject; 110 | attributes = { 111 | LastUpgradeCheck = 0730; 112 | ORGANIZATIONNAME = "Murage Inc."; 113 | TargetAttributes = { 114 | 9497BD101C53D9D600ADD79F = { 115 | CreatedOnToolsVersion = 7.2; 116 | }; 117 | }; 118 | }; 119 | buildConfigurationList = 9497BD0C1C53D9D600ADD79F /* Build configuration list for PBXProject "cevfs" */; 120 | compatibilityVersion = "Xcode 3.2"; 121 | developmentRegion = English; 122 | hasScannedForEncodings = 0; 123 | knownRegions = ( 124 | en, 125 | ); 126 | mainGroup = 9497BD081C53D9D600ADD79F; 127 | productRefGroup = 9497BD121C53D9D600ADD79F /* Products */; 128 | projectDirPath = ""; 129 | projectRoot = ""; 130 | targets = ( 131 | 9497BD101C53D9D600ADD79F /* sqlite */, 132 | ); 133 | }; 134 | /* End PBXProject section */ 135 | 136 | /* Begin PBXSourcesBuildPhase section */ 137 | 9497BD0D1C53D9D600ADD79F /* Sources */ = { 138 | isa = PBXSourcesBuildPhase; 139 | buildActionMask = 2147483647; 140 | files = ( 141 | 9442235E1D2027C60016E082 /* cevfs-all.c in Sources */, 142 | ); 143 | runOnlyForDeploymentPostprocessing = 0; 144 | }; 145 | /* End PBXSourcesBuildPhase section */ 146 | 147 | /* Begin XCBuildConfiguration section */ 148 | 9497BD131C53D9D600ADD79F /* Debug */ = { 149 | isa = XCBuildConfiguration; 150 | buildSettings = { 151 | ALWAYS_SEARCH_USER_PATHS = NO; 152 | CLANG_CXX_LANGUAGE_STANDARD = "gnu++0x"; 153 | CLANG_CXX_LIBRARY = "libc++"; 154 | CLANG_ENABLE_MODULES = YES; 155 | CLANG_ENABLE_OBJC_ARC = YES; 156 | CLANG_WARN_BOOL_CONVERSION = YES; 157 | CLANG_WARN_CONSTANT_CONVERSION = YES; 158 | CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; 159 | CLANG_WARN_EMPTY_BODY = YES; 160 | CLANG_WARN_ENUM_CONVERSION = YES; 161 | CLANG_WARN_INT_CONVERSION = YES; 162 | CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; 163 | CLANG_WARN_UNREACHABLE_CODE = YES; 164 | CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; 165 | CODE_SIGN_IDENTITY = "-"; 166 | COPY_PHASE_STRIP = NO; 167 | DEBUG_INFORMATION_FORMAT = dwarf; 168 | ENABLE_STRICT_OBJC_MSGSEND = YES; 169 | ENABLE_TESTABILITY = YES; 170 | GCC_C_LANGUAGE_STANDARD = gnu99; 171 | GCC_DYNAMIC_NO_PIC = NO; 172 | GCC_NO_COMMON_BLOCKS = YES; 173 | GCC_OPTIMIZATION_LEVEL = 0; 174 | GCC_WARN_64_TO_32_BIT_CONVERSION = YES; 175 | GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; 176 | GCC_WARN_UNDECLARED_SELECTOR = YES; 177 | GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; 178 | GCC_WARN_UNUSED_FUNCTION = YES; 179 | GCC_WARN_UNUSED_VARIABLE = YES; 180 | MACOSX_DEPLOYMENT_TARGET = 10.11; 181 | MTL_ENABLE_DEBUG_INFO = YES; 182 | ONLY_ACTIVE_ARCH = YES; 183 | SDKROOT = macosx; 184 | USER_HEADER_SEARCH_PATHS = sqlite; 185 | }; 186 | name = Debug; 187 | }; 188 | 9497BD141C53D9D600ADD79F /* Release */ = { 189 | isa = XCBuildConfiguration; 190 | buildSettings = { 191 | ALWAYS_SEARCH_USER_PATHS = NO; 192 | CLANG_CXX_LANGUAGE_STANDARD = "gnu++0x"; 193 | CLANG_CXX_LIBRARY = "libc++"; 194 | CLANG_ENABLE_MODULES = YES; 195 | CLANG_ENABLE_OBJC_ARC = YES; 196 | CLANG_WARN_BOOL_CONVERSION = YES; 197 | CLANG_WARN_CONSTANT_CONVERSION = YES; 198 | CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; 199 | CLANG_WARN_EMPTY_BODY = YES; 200 | CLANG_WARN_ENUM_CONVERSION = YES; 201 | CLANG_WARN_INT_CONVERSION = YES; 202 | CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; 203 | CLANG_WARN_UNREACHABLE_CODE = YES; 204 | CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; 205 | CODE_SIGN_IDENTITY = "-"; 206 | COPY_PHASE_STRIP = NO; 207 | DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym"; 208 | ENABLE_NS_ASSERTIONS = NO; 209 | ENABLE_STRICT_OBJC_MSGSEND = YES; 210 | GCC_C_LANGUAGE_STANDARD = gnu99; 211 | GCC_NO_COMMON_BLOCKS = YES; 212 | GCC_WARN_64_TO_32_BIT_CONVERSION = YES; 213 | GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; 214 | GCC_WARN_UNDECLARED_SELECTOR = YES; 215 | GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; 216 | GCC_WARN_UNUSED_FUNCTION = YES; 217 | GCC_WARN_UNUSED_VARIABLE = YES; 218 | MACOSX_DEPLOYMENT_TARGET = 10.11; 219 | MTL_ENABLE_DEBUG_INFO = NO; 220 | SDKROOT = macosx; 221 | USER_HEADER_SEARCH_PATHS = sqlite; 222 | }; 223 | name = Release; 224 | }; 225 | 9497BD161C53D9D600ADD79F /* Debug */ = { 226 | isa = XCBuildConfiguration; 227 | buildSettings = { 228 | COMBINE_HIDPI_IMAGES = YES; 229 | EXECUTABLE_PREFIX = lib; 230 | GCC_PREPROCESSOR_DEFINITIONS = ( 231 | "DEBUG=1", 232 | "SQLITE_DEBUG=1", 233 | "SQLITE_THREADSAFE=0", 234 | "SQLITE_CORE=1", 235 | "SQLITE_OS_UNIX=1", 236 | "HAVE_READLINE=0", 237 | "HAVE_EDITLINE=0", 238 | "SQLITE_TEMP_STORE=1", 239 | "$(inherited)", 240 | ); 241 | PRODUCT_NAME = "$(TARGET_NAME)"; 242 | }; 243 | name = Debug; 244 | }; 245 | 9497BD171C53D9D600ADD79F /* Release */ = { 246 | isa = XCBuildConfiguration; 247 | buildSettings = { 248 | COMBINE_HIDPI_IMAGES = YES; 249 | EXECUTABLE_PREFIX = lib; 250 | GCC_PREPROCESSOR_DEFINITIONS = ( 251 | "SQLITE_THREADSAFE=0", 252 | SQLITE_CORE, 253 | "SQLITE_OS_UNIX=1", 254 | "HAVE_READLINE=0", 255 | "HAVE_EDITLINE=0", 256 | "SQLITE_TEMP_STORE=1", 257 | "$(inherited)", 258 | ); 259 | PRODUCT_NAME = "$(TARGET_NAME)"; 260 | }; 261 | name = Release; 262 | }; 263 | /* End XCBuildConfiguration section */ 264 | 265 | /* Begin XCConfigurationList section */ 266 | 9497BD0C1C53D9D600ADD79F /* Build configuration list for PBXProject "cevfs" */ = { 267 | isa = XCConfigurationList; 268 | buildConfigurations = ( 269 | 9497BD131C53D9D600ADD79F /* Debug */, 270 | 9497BD141C53D9D600ADD79F /* Release */, 271 | ); 272 | defaultConfigurationIsVisible = 0; 273 | defaultConfigurationName = Release; 274 | }; 275 | 9497BD151C53D9D600ADD79F /* Build configuration list for PBXNativeTarget "sqlite" */ = { 276 | isa = XCConfigurationList; 277 | buildConfigurations = ( 278 | 9497BD161C53D9D600ADD79F /* Debug */, 279 | 9497BD171C53D9D600ADD79F /* Release */, 280 | ); 281 | defaultConfigurationIsVisible = 0; 282 | defaultConfigurationName = Release; 283 | }; 284 | /* End XCConfigurationList section */ 285 | }; 286 | rootObject = 9497BD091C53D9D600ADD79F /* Project object */; 287 | } 288 | -------------------------------------------------------------------------------- /cevfs/sqlite/readme.md: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/techrah/sqlite3-compression-encryption-vfs/8fabfc5d775e7078c1c27eee6817d09af8423e35/cevfs/sqlite/readme.md -------------------------------------------------------------------------------- /cevfs_build/cevfs_build.c: -------------------------------------------------------------------------------- 1 | /** 2 | CEVFS - Compression & Encryption VFS 3 | cevfs_build 4 | 5 | Copyright (c) 2016 Ryan Homer, Murage Inc. 6 | 7 | MIT License 8 | 9 | Permission is hereby granted, free of charge, to any person obtaining a copy 10 | of this software and associated documentation files (the "Software"), to deal 11 | in the Software without restriction, including without limitation the rights 12 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 13 | copies of the Software, and to permit persons to whom the Software is 14 | furnished to do so, subject to the following conditions: 15 | 16 | The above copyright notice and this permission notice shall be included in all 17 | copies or substantial portions of the Software. 18 | 19 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 20 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 21 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 22 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 23 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 24 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 25 | SOFTWARE. 26 | */ 27 | 28 | #include 29 | #include 30 | #include 31 | #include "cevfs.h" 32 | #include "xMethods.c" 33 | 34 | extern const char *fileTail(const char *z); 35 | typedef unsigned char u8; 36 | 37 | /* 38 | ** Taken from SQLite source code. 39 | ** Check to see if this machine uses EBCDIC. (Yes, believe it or 40 | ** not, there are still machines out there that use EBCDIC.) 41 | */ 42 | #if 'A' == '\301' 43 | # define SQLITE_EBCDIC 1 44 | #else 45 | # define SQLITE_ASCII 1 46 | #endif 47 | 48 | /* 49 | ** Taken from SQLite source code. 50 | ** Translate a single byte of Hex into an integer. 51 | ** This routine only works if h really is a valid hexadecimal 52 | ** character: 0..9a..fA..F 53 | */ 54 | static u8 hexToInt(int h){ 55 | assert( (h>='0' && h<='9') || (h>='a' && h<='f') || (h>='A' && h<='F') ); 56 | #ifdef SQLITE_ASCII 57 | h += 9*(1&(h>>6)); 58 | #endif 59 | #ifdef SQLITE_EBCDIC 60 | h += 9*(1&~(h>>4)); 61 | #endif 62 | return (u8)(h & 0xf); 63 | } 64 | 65 | /* 66 | ** Taken from SQLite source code. 67 | ** Convert a BLOB literal of the form "x'hhhhhh'" into its binary 68 | ** value. Return a pointer to its binary value. Space to hold the 69 | ** binary value has been obtained from malloc and must be freed by 70 | ** the calling routine. 71 | */ 72 | static void *hexToBlob(const char *z, int n){ 73 | char *zBlob; 74 | int i; 75 | 76 | zBlob = (char *)malloc(n/2 + 1); 77 | n--; 78 | if( zBlob ){ 79 | for(i=0; i\n"); 95 | printf(" VFS_NAME: Name of VFS to embed in database file.\n"); 96 | printf(" KEY: Encryption key in the form: x'...'\n"); 97 | return EXIT_FAILURE; 98 | } 99 | 100 | int rc; 101 | 102 | // Convert encryption key string to hex blob. 103 | // This assumes that the key is in the form of x'' 104 | // You should, of course, implement proper error checking. 105 | const char *key = argv[4]+2; 106 | char *keyBytes = hexToBlob(key, (int)strlen(key)-1); 107 | 108 | ctx.pKey = keyBytes; // 32-bit encryption hex key 109 | ctx.nKeySz = kCCKeySizeAES256; // key size in bytes 110 | ctx.nIvSz = kCCBlockSizeAES128; // size of IV in bytes 111 | 112 | // You can use the VFS name to determine how you set up your xMethods, 113 | // so we pass the VFS name as a command line parameter as well. 114 | rc = cevfs_build(argv[1], argv[2], argv[3], &ctx, cevfsAutoDetect); 115 | return rc; 116 | } 117 | -------------------------------------------------------------------------------- /cevfs_build/cevfs_build.xcodeproj/project.pbxproj: -------------------------------------------------------------------------------- 1 | // !$*UTF8*$! 2 | { 3 | archiveVersion = 1; 4 | classes = { 5 | }; 6 | objectVersion = 46; 7 | objects = { 8 | 9 | /* Begin PBXBuildFile section */ 10 | 944223791D20A7E00016E082 /* cevfs_build.c in Sources */ = {isa = PBXBuildFile; fileRef = 944223781D20A7E00016E082 /* cevfs_build.c */; }; 11 | 9442237B1D20BCAE0016E082 /* libsqlite.a in Frameworks */ = {isa = PBXBuildFile; fileRef = 9442237A1D20BCAE0016E082 /* libsqlite.a */; }; 12 | /* End PBXBuildFile section */ 13 | 14 | /* Begin PBXCopyFilesBuildPhase section */ 15 | 944223691D20A59B0016E082 /* CopyFiles */ = { 16 | isa = PBXCopyFilesBuildPhase; 17 | buildActionMask = 2147483647; 18 | dstPath = /usr/share/man/man1/; 19 | dstSubfolderSpec = 0; 20 | files = ( 21 | ); 22 | runOnlyForDeploymentPostprocessing = 1; 23 | }; 24 | /* End PBXCopyFilesBuildPhase section */ 25 | 26 | /* Begin PBXFileReference section */ 27 | 9442236B1D20A59B0016E082 /* cevfs_build */ = {isa = PBXFileReference; explicitFileType = "compiled.mach-o.executable"; includeInIndex = 0; path = cevfs_build; sourceTree = BUILT_PRODUCTS_DIR; }; 28 | 944223751D20A6870016E082 /* xMethods.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = xMethods.c; sourceTree = ""; }; 29 | 944223781D20A7E00016E082 /* cevfs_build.c */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.c; path = cevfs_build.c; sourceTree = ""; }; 30 | 9442237A1D20BCAE0016E082 /* libsqlite.a */ = {isa = PBXFileReference; lastKnownFileType = archive.ar; name = libsqlite.a; path = "../../../Library/Developer/Xcode/DerivedData/cevfs-hfabimsbqwgxjbecetgifrofdoaw/Build/Products/Debug/libsqlite.a"; sourceTree = ""; }; 31 | /* End PBXFileReference section */ 32 | 33 | /* Begin PBXFrameworksBuildPhase section */ 34 | 944223681D20A59B0016E082 /* Frameworks */ = { 35 | isa = PBXFrameworksBuildPhase; 36 | buildActionMask = 2147483647; 37 | files = ( 38 | 9442237B1D20BCAE0016E082 /* libsqlite.a in Frameworks */, 39 | ); 40 | runOnlyForDeploymentPostprocessing = 0; 41 | }; 42 | /* End PBXFrameworksBuildPhase section */ 43 | 44 | /* Begin PBXGroup section */ 45 | 944223621D20A59B0016E082 = { 46 | isa = PBXGroup; 47 | children = ( 48 | 944223781D20A7E00016E082 /* cevfs_build.c */, 49 | 944223751D20A6870016E082 /* xMethods.c */, 50 | 9442237E1D20BD300016E082 /* Frameworks */, 51 | 9442236C1D20A59B0016E082 /* Products */, 52 | ); 53 | sourceTree = ""; 54 | }; 55 | 9442236C1D20A59B0016E082 /* Products */ = { 56 | isa = PBXGroup; 57 | children = ( 58 | 9442236B1D20A59B0016E082 /* cevfs_build */, 59 | ); 60 | name = Products; 61 | sourceTree = ""; 62 | }; 63 | 9442237E1D20BD300016E082 /* Frameworks */ = { 64 | isa = PBXGroup; 65 | children = ( 66 | 9442237A1D20BCAE0016E082 /* libsqlite.a */, 67 | ); 68 | name = Frameworks; 69 | sourceTree = ""; 70 | }; 71 | /* End PBXGroup section */ 72 | 73 | /* Begin PBXNativeTarget section */ 74 | 9442236A1D20A59B0016E082 /* cevfs_build */ = { 75 | isa = PBXNativeTarget; 76 | buildConfigurationList = 944223721D20A59B0016E082 /* Build configuration list for PBXNativeTarget "cevfs_build" */; 77 | buildPhases = ( 78 | 944223671D20A59B0016E082 /* Sources */, 79 | 944223681D20A59B0016E082 /* Frameworks */, 80 | 944223691D20A59B0016E082 /* CopyFiles */, 81 | ); 82 | buildRules = ( 83 | ); 84 | dependencies = ( 85 | ); 86 | name = cevfs_build; 87 | productName = cevfs_build; 88 | productReference = 9442236B1D20A59B0016E082 /* cevfs_build */; 89 | productType = "com.apple.product-type.tool"; 90 | }; 91 | /* End PBXNativeTarget section */ 92 | 93 | /* Begin PBXProject section */ 94 | 944223631D20A59B0016E082 /* Project object */ = { 95 | isa = PBXProject; 96 | attributes = { 97 | LastUpgradeCheck = 0730; 98 | ORGANIZATIONNAME = "Murage Inc."; 99 | TargetAttributes = { 100 | 9442236A1D20A59B0016E082 = { 101 | CreatedOnToolsVersion = 7.3.1; 102 | }; 103 | }; 104 | }; 105 | buildConfigurationList = 944223661D20A59B0016E082 /* Build configuration list for PBXProject "cevfs_build" */; 106 | compatibilityVersion = "Xcode 3.2"; 107 | developmentRegion = English; 108 | hasScannedForEncodings = 0; 109 | knownRegions = ( 110 | en, 111 | ); 112 | mainGroup = 944223621D20A59B0016E082; 113 | productRefGroup = 9442236C1D20A59B0016E082 /* Products */; 114 | projectDirPath = ""; 115 | projectRoot = ""; 116 | targets = ( 117 | 9442236A1D20A59B0016E082 /* cevfs_build */, 118 | ); 119 | }; 120 | /* End PBXProject section */ 121 | 122 | /* Begin PBXSourcesBuildPhase section */ 123 | 944223671D20A59B0016E082 /* Sources */ = { 124 | isa = PBXSourcesBuildPhase; 125 | buildActionMask = 2147483647; 126 | files = ( 127 | 944223791D20A7E00016E082 /* cevfs_build.c in Sources */, 128 | ); 129 | runOnlyForDeploymentPostprocessing = 0; 130 | }; 131 | /* End PBXSourcesBuildPhase section */ 132 | 133 | /* Begin XCBuildConfiguration section */ 134 | 944223701D20A59B0016E082 /* Debug */ = { 135 | isa = XCBuildConfiguration; 136 | buildSettings = { 137 | CLANG_ANALYZER_NONNULL = YES; 138 | CLANG_CXX_LANGUAGE_STANDARD = "gnu++0x"; 139 | CLANG_CXX_LIBRARY = "libc++"; 140 | CLANG_ENABLE_MODULES = YES; 141 | CLANG_ENABLE_OBJC_ARC = YES; 142 | CLANG_WARN_BOOL_CONVERSION = YES; 143 | CLANG_WARN_CONSTANT_CONVERSION = YES; 144 | CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; 145 | CLANG_WARN_EMPTY_BODY = YES; 146 | CLANG_WARN_ENUM_CONVERSION = YES; 147 | CLANG_WARN_INT_CONVERSION = YES; 148 | CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; 149 | CLANG_WARN_UNREACHABLE_CODE = YES; 150 | CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; 151 | CODE_SIGN_IDENTITY = "-"; 152 | COPY_PHASE_STRIP = NO; 153 | DEBUG_INFORMATION_FORMAT = dwarf; 154 | ENABLE_STRICT_OBJC_MSGSEND = YES; 155 | ENABLE_TESTABILITY = YES; 156 | GCC_C_LANGUAGE_STANDARD = gnu99; 157 | GCC_DYNAMIC_NO_PIC = NO; 158 | GCC_NO_COMMON_BLOCKS = YES; 159 | GCC_OPTIMIZATION_LEVEL = 0; 160 | GCC_PREPROCESSOR_DEFINITIONS = ( 161 | "DEBUG=1", 162 | "$(inherited)", 163 | ); 164 | GCC_WARN_64_TO_32_BIT_CONVERSION = YES; 165 | GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; 166 | GCC_WARN_UNDECLARED_SELECTOR = YES; 167 | GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; 168 | GCC_WARN_UNUSED_FUNCTION = YES; 169 | GCC_WARN_UNUSED_VARIABLE = YES; 170 | MACOSX_DEPLOYMENT_TARGET = 10.11; 171 | MTL_ENABLE_DEBUG_INFO = YES; 172 | ONLY_ACTIVE_ARCH = YES; 173 | OTHER_LDFLAGS = "-lz"; 174 | SDKROOT = macosx; 175 | USER_HEADER_SEARCH_PATHS = "../cevfs/**"; 176 | }; 177 | name = Debug; 178 | }; 179 | 944223711D20A59B0016E082 /* Release */ = { 180 | isa = XCBuildConfiguration; 181 | buildSettings = { 182 | CLANG_ANALYZER_NONNULL = YES; 183 | CLANG_CXX_LANGUAGE_STANDARD = "gnu++0x"; 184 | CLANG_CXX_LIBRARY = "libc++"; 185 | CLANG_ENABLE_MODULES = YES; 186 | CLANG_ENABLE_OBJC_ARC = YES; 187 | CLANG_WARN_BOOL_CONVERSION = YES; 188 | CLANG_WARN_CONSTANT_CONVERSION = YES; 189 | CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; 190 | CLANG_WARN_EMPTY_BODY = YES; 191 | CLANG_WARN_ENUM_CONVERSION = YES; 192 | CLANG_WARN_INT_CONVERSION = YES; 193 | CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; 194 | CLANG_WARN_UNREACHABLE_CODE = YES; 195 | CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; 196 | CODE_SIGN_IDENTITY = "-"; 197 | COPY_PHASE_STRIP = NO; 198 | DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym"; 199 | ENABLE_NS_ASSERTIONS = NO; 200 | ENABLE_STRICT_OBJC_MSGSEND = YES; 201 | GCC_C_LANGUAGE_STANDARD = gnu99; 202 | GCC_NO_COMMON_BLOCKS = YES; 203 | GCC_WARN_64_TO_32_BIT_CONVERSION = YES; 204 | GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; 205 | GCC_WARN_UNDECLARED_SELECTOR = YES; 206 | GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; 207 | GCC_WARN_UNUSED_FUNCTION = YES; 208 | GCC_WARN_UNUSED_VARIABLE = YES; 209 | MACOSX_DEPLOYMENT_TARGET = 10.11; 210 | MTL_ENABLE_DEBUG_INFO = NO; 211 | OTHER_LDFLAGS = "-lz"; 212 | SDKROOT = macosx; 213 | USER_HEADER_SEARCH_PATHS = "../cevfs/**"; 214 | }; 215 | name = Release; 216 | }; 217 | 944223731D20A59B0016E082 /* Debug */ = { 218 | isa = XCBuildConfiguration; 219 | buildSettings = { 220 | PRODUCT_NAME = "$(TARGET_NAME)"; 221 | }; 222 | name = Debug; 223 | }; 224 | 944223741D20A59B0016E082 /* Release */ = { 225 | isa = XCBuildConfiguration; 226 | buildSettings = { 227 | PRODUCT_NAME = "$(TARGET_NAME)"; 228 | }; 229 | name = Release; 230 | }; 231 | /* End XCBuildConfiguration section */ 232 | 233 | /* Begin XCConfigurationList section */ 234 | 944223661D20A59B0016E082 /* Build configuration list for PBXProject "cevfs_build" */ = { 235 | isa = XCConfigurationList; 236 | buildConfigurations = ( 237 | 944223701D20A59B0016E082 /* Debug */, 238 | 944223711D20A59B0016E082 /* Release */, 239 | ); 240 | defaultConfigurationIsVisible = 0; 241 | defaultConfigurationName = Release; 242 | }; 243 | 944223721D20A59B0016E082 /* Build configuration list for PBXNativeTarget "cevfs_build" */ = { 244 | isa = XCConfigurationList; 245 | buildConfigurations = ( 246 | 944223731D20A59B0016E082 /* Debug */, 247 | 944223741D20A59B0016E082 /* Release */, 248 | ); 249 | defaultConfigurationIsVisible = 0; 250 | defaultConfigurationName = Release; 251 | }; 252 | /* End XCConfigurationList section */ 253 | }; 254 | rootObject = 944223631D20A59B0016E082 /* Project object */; 255 | } 256 | -------------------------------------------------------------------------------- /cevfs_build/cevfs_mod.c: -------------------------------------------------------------------------------- 1 | /** 2 | CEVFS - Compression & Encryption VFS 3 | cevfs_mod (module activation) 4 | 5 | Copyright (c) 2016 Ryan Homer, Murage Inc. 6 | 7 | MIT License 8 | 9 | Permission is hereby granted, free of charge, to any person obtaining a copy 10 | of this software and associated documentation files (the "Software"), to deal 11 | in the Software without restriction, including without limitation the rights 12 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 13 | copies of the Software, and to permit persons to whom the Software is 14 | furnished to do so, subject to the following conditions: 15 | 16 | The above copyright notice and this permission notice shall be included in all 17 | copies or substantial portions of the Software. 18 | 19 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 20 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 21 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 22 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 23 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 24 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 25 | SOFTWARE. 26 | */ 27 | 28 | #include "xMethods.c" 29 | 30 | struct context ctx; 31 | 32 | /* 33 | ** Taken from SQLite source code. 34 | ** Check to see if this machine uses EBCDIC. (Yes, believe it or 35 | ** not, there are still machines out there that use EBCDIC.) 36 | */ 37 | #if 'A' == '\301' 38 | # define SQLITE_EBCDIC 1 39 | #else 40 | # define SQLITE_ASCII 1 41 | #endif 42 | 43 | /* 44 | ** Taken from SQLite source code. 45 | ** Translate a single byte of Hex into an integer. 46 | ** This routine only works if h really is a valid hexadecimal 47 | ** character: 0..9a..fA..F 48 | */ 49 | static u8 hexToInt(int h){ 50 | assert( (h>='0' && h<='9') || (h>='a' && h<='f') || (h>='A' && h<='F') ); 51 | #ifdef SQLITE_ASCII 52 | h += 9*(1&(h>>6)); 53 | #endif 54 | #ifdef SQLITE_EBCDIC 55 | h += 9*(1&~(h>>4)); 56 | #endif 57 | return (u8)(h & 0xf); 58 | } 59 | 60 | /* 61 | ** Taken from SQLite source code. 62 | ** Convert a BLOB literal of the form "x'hhhhhh'" into its binary 63 | ** value. Return a pointer to its binary value. Space to hold the 64 | ** binary value has been obtained from malloc and must be freed by 65 | ** the calling routine. 66 | */ 67 | static void *hexToBlob(const char *z, int n){ 68 | char *zBlob; 69 | int i; 70 | 71 | zBlob = (char *)malloc(n/2 + 1); 72 | n--; 73 | if( zBlob ){ 74 | for(i=0; i 10 | #include 11 | #include 12 | #include 13 | 14 | struct context { 15 | void *pKey; 16 | int nKeySz; 17 | int nIvSz; 18 | }; 19 | 20 | static void random_bytes(unsigned char *buf, int num){ 21 | int i; 22 | int j = num/4; 23 | uint32_t *dwbuf = (uint32_t *)buf; 24 | 25 | srandomdev(); 26 | for( i=0; inIvSz); 64 | 65 | /* According to CCCryptor manpage: "For block ciphers, the output size will always be less than or 66 | equal to the input size plus the size of one block." However, there seems to be a bug as normally 67 | CCCrypt fails with error code kCCBufferTooSmall when the output buffer size is too small, can 68 | crash when size is exactly input size plus size of one block. It works with just 1 more byte. 69 | REF: https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man3/CCCrypt.3cc.html */ 70 | size_t nOutSz = nDataInSize+kCCBlockSizeAES128+1; 71 | *ppDataOut = sqlite3_malloc((int)nOutSz); 72 | 73 | CCCryptorStatus ccStatus = CCCrypt( 74 | kCCEncrypt, // enc/dec 75 | kCCAlgorithmAES128, // algorithm 76 | kCCOptionPKCS7Padding, // options: kCCOptionPKCS7Padding, kCCOptionECBMode, 0 = no padding 77 | pCtx->pKey, // 256-bit (32-byte) key 78 | pCtx->nKeySz, // key length (bytes) 79 | pIvOut, // const void *iv 80 | pDataIn, // const void *dataIn 81 | nDataInSize, // data-in length 82 | *ppDataOut, // dataOut; result is written here. 83 | nOutSz, // The size of the dataOut buffer in bytes 84 | nDataSizeOut // On successful return, the number of bytes written to dataOut. 85 | ); 86 | return (ccStatus==kCCSuccess); 87 | } 88 | 89 | int cevfsDecrypt(void *pDecryptCtx, const void *pDataIn, size_t nDataInSize, const void *pIvIn, void *pDataOut, size_t nDataBufferSizeOut, size_t *nDataSizeOut) { 90 | struct context *pCtx = (struct context *)pDecryptCtx; 91 | CCCryptorStatus ccStatus = CCCrypt( 92 | kCCDecrypt, // enc/dec 93 | kCCAlgorithmAES128, // algorithm 94 | kCCOptionPKCS7Padding, // options: kCCOptionPKCS7Padding, kCCOptionECBMode, 0 = no padding 95 | pCtx->pKey, // 256-bit (32-byte) key 96 | pCtx->nKeySz, // key length (bytes) 97 | pIvIn, // const void *iv 98 | pDataIn, // const void *dataIn 99 | nDataInSize, // data-in length 100 | pDataOut, // dataOut; result is written here. 101 | nDataBufferSizeOut, // The size of the dataOut buffer in bytes 102 | nDataSizeOut // On successful return, the number of bytes written to dataOut. 103 | ); 104 | return (ccStatus==kCCSuccess); 105 | } 106 | 107 | int cevfsAutoDetect(void *pCtx, const char *zFile, const char *zHdr, size_t *pEncIvSz, CevfsMethods *pMethods) { 108 | *pEncIvSz = kCCBlockSizeAES128; 109 | pMethods->xCompressBound = cevfsCompressBound; 110 | pMethods->xCompress = cevfsCompress;; 111 | pMethods->xUncompress = cevfsUncompress; 112 | pMethods->xEncrypt = cevfsEncrypt; 113 | pMethods->xDecrypt = cevfsDecrypt; 114 | return true; 115 | } 116 | -------------------------------------------------------------------------------- /cevfs_example/cevfs_example.xcodeproj/project.pbxproj: -------------------------------------------------------------------------------- 1 | // !$*UTF8*$! 2 | { 3 | archiveVersion = 1; 4 | classes = { 5 | }; 6 | objectVersion = 46; 7 | objects = { 8 | 9 | /* Begin PBXBuildFile section */ 10 | 944223601D2042D00016E082 /* libsqlite.a in Frameworks */ = {isa = PBXBuildFile; fileRef = 9442235F1D2042D00016E082 /* libsqlite.a */; }; 11 | 9497BD021C53D96600ADD79F /* main.c in Sources */ = {isa = PBXBuildFile; fileRef = 9497BD011C53D96600ADD79F /* main.c */; }; 12 | /* End PBXBuildFile section */ 13 | 14 | /* Begin PBXCopyFilesBuildPhase section */ 15 | 9497BCFC1C53D96600ADD79F /* CopyFiles */ = { 16 | isa = PBXCopyFilesBuildPhase; 17 | buildActionMask = 2147483647; 18 | dstPath = /usr/share/man/man1/; 19 | dstSubfolderSpec = 0; 20 | files = ( 21 | ); 22 | runOnlyForDeploymentPostprocessing = 1; 23 | }; 24 | /* End PBXCopyFilesBuildPhase section */ 25 | 26 | /* Begin PBXFileReference section */ 27 | 9442235F1D2042D00016E082 /* libsqlite.a */ = {isa = PBXFileReference; lastKnownFileType = archive.ar; name = libsqlite.a; path = "../../../Library/Developer/Xcode/DerivedData/cevfs-hfabimsbqwgxjbecetgifrofdoaw/Build/Products/Debug/libsqlite.a"; sourceTree = ""; }; 28 | 9497BCFE1C53D96600ADD79F /* cevfs_example */ = {isa = PBXFileReference; explicitFileType = "compiled.mach-o.executable"; includeInIndex = 0; path = cevfs_example; sourceTree = BUILT_PRODUCTS_DIR; }; 29 | 9497BD011C53D96600ADD79F /* main.c */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.c; path = main.c; sourceTree = ""; }; 30 | /* End PBXFileReference section */ 31 | 32 | /* Begin PBXFrameworksBuildPhase section */ 33 | 9497BCFB1C53D96600ADD79F /* Frameworks */ = { 34 | isa = PBXFrameworksBuildPhase; 35 | buildActionMask = 2147483647; 36 | files = ( 37 | 944223601D2042D00016E082 /* libsqlite.a in Frameworks */, 38 | ); 39 | runOnlyForDeploymentPostprocessing = 0; 40 | }; 41 | /* End PBXFrameworksBuildPhase section */ 42 | 43 | /* Begin PBXGroup section */ 44 | 9497BCF51C53D96600ADD79F = { 45 | isa = PBXGroup; 46 | children = ( 47 | 9497BD011C53D96600ADD79F /* main.c */, 48 | 9497BE1D1C53E04D00ADD79F /* Frameworks */, 49 | 9497BCFF1C53D96600ADD79F /* Products */, 50 | ); 51 | sourceTree = ""; 52 | }; 53 | 9497BCFF1C53D96600ADD79F /* Products */ = { 54 | isa = PBXGroup; 55 | children = ( 56 | 9497BCFE1C53D96600ADD79F /* cevfs_example */, 57 | ); 58 | name = Products; 59 | sourceTree = ""; 60 | }; 61 | 9497BE1D1C53E04D00ADD79F /* Frameworks */ = { 62 | isa = PBXGroup; 63 | children = ( 64 | 9442235F1D2042D00016E082 /* libsqlite.a */, 65 | ); 66 | name = Frameworks; 67 | sourceTree = ""; 68 | }; 69 | /* End PBXGroup section */ 70 | 71 | /* Begin PBXNativeTarget section */ 72 | 9497BCFD1C53D96600ADD79F /* cevfs_example */ = { 73 | isa = PBXNativeTarget; 74 | buildConfigurationList = 9497BD051C53D96600ADD79F /* Build configuration list for PBXNativeTarget "cevfs_example" */; 75 | buildPhases = ( 76 | 9497BCFA1C53D96600ADD79F /* Sources */, 77 | 9497BCFB1C53D96600ADD79F /* Frameworks */, 78 | 9497BCFC1C53D96600ADD79F /* CopyFiles */, 79 | ); 80 | buildRules = ( 81 | ); 82 | dependencies = ( 83 | ); 84 | name = cevfs_example; 85 | productName = sqlite_compress_test; 86 | productReference = 9497BCFE1C53D96600ADD79F /* cevfs_example */; 87 | productType = "com.apple.product-type.tool"; 88 | }; 89 | /* End PBXNativeTarget section */ 90 | 91 | /* Begin PBXProject section */ 92 | 9497BCF61C53D96600ADD79F /* Project object */ = { 93 | isa = PBXProject; 94 | attributes = { 95 | LastUpgradeCheck = 0720; 96 | ORGANIZATIONNAME = "Murage Inc."; 97 | TargetAttributes = { 98 | 9497BCFD1C53D96600ADD79F = { 99 | CreatedOnToolsVersion = 7.2; 100 | }; 101 | }; 102 | }; 103 | buildConfigurationList = 9497BCF91C53D96600ADD79F /* Build configuration list for PBXProject "cevfs_example" */; 104 | compatibilityVersion = "Xcode 3.2"; 105 | developmentRegion = English; 106 | hasScannedForEncodings = 0; 107 | knownRegions = ( 108 | en, 109 | ); 110 | mainGroup = 9497BCF51C53D96600ADD79F; 111 | productRefGroup = 9497BCFF1C53D96600ADD79F /* Products */; 112 | projectDirPath = ""; 113 | projectRoot = ""; 114 | targets = ( 115 | 9497BCFD1C53D96600ADD79F /* cevfs_example */, 116 | ); 117 | }; 118 | /* End PBXProject section */ 119 | 120 | /* Begin PBXSourcesBuildPhase section */ 121 | 9497BCFA1C53D96600ADD79F /* Sources */ = { 122 | isa = PBXSourcesBuildPhase; 123 | buildActionMask = 2147483647; 124 | files = ( 125 | 9497BD021C53D96600ADD79F /* main.c in Sources */, 126 | ); 127 | runOnlyForDeploymentPostprocessing = 0; 128 | }; 129 | /* End PBXSourcesBuildPhase section */ 130 | 131 | /* Begin XCBuildConfiguration section */ 132 | 9497BD031C53D96600ADD79F /* Debug */ = { 133 | isa = XCBuildConfiguration; 134 | buildSettings = { 135 | ALWAYS_SEARCH_USER_PATHS = YES; 136 | CLANG_CXX_LANGUAGE_STANDARD = "gnu++0x"; 137 | CLANG_CXX_LIBRARY = "libc++"; 138 | CLANG_ENABLE_MODULES = YES; 139 | CLANG_ENABLE_OBJC_ARC = YES; 140 | CLANG_WARN_BOOL_CONVERSION = YES; 141 | CLANG_WARN_CONSTANT_CONVERSION = YES; 142 | CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; 143 | CLANG_WARN_EMPTY_BODY = YES; 144 | CLANG_WARN_ENUM_CONVERSION = YES; 145 | CLANG_WARN_INT_CONVERSION = YES; 146 | CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; 147 | CLANG_WARN_UNREACHABLE_CODE = YES; 148 | CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; 149 | CODE_SIGN_IDENTITY = "-"; 150 | COPY_PHASE_STRIP = NO; 151 | DEBUG_INFORMATION_FORMAT = dwarf; 152 | ENABLE_STRICT_OBJC_MSGSEND = YES; 153 | ENABLE_TESTABILITY = YES; 154 | GCC_C_LANGUAGE_STANDARD = gnu99; 155 | GCC_DYNAMIC_NO_PIC = NO; 156 | GCC_NO_COMMON_BLOCKS = YES; 157 | GCC_OPTIMIZATION_LEVEL = 0; 158 | GCC_PREPROCESSOR_DEFINITIONS = ( 159 | "DEBUG=1", 160 | SQLITE_ENABLE_ZIPVFS, 161 | "$(inherited)", 162 | ); 163 | GCC_WARN_64_TO_32_BIT_CONVERSION = YES; 164 | GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; 165 | GCC_WARN_UNDECLARED_SELECTOR = YES; 166 | GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; 167 | GCC_WARN_UNUSED_FUNCTION = YES; 168 | GCC_WARN_UNUSED_VARIABLE = YES; 169 | HEADER_SEARCH_PATHS = "$(PROJECT_DIR)"; 170 | MACOSX_DEPLOYMENT_TARGET = 10.11; 171 | MTL_ENABLE_DEBUG_INFO = YES; 172 | ONLY_ACTIVE_ARCH = YES; 173 | OTHER_LDFLAGS = "-lz"; 174 | SDKROOT = macosx; 175 | USER_HEADER_SEARCH_PATHS = "$(BUILT_PRODUCTS_DIR)/**"; 176 | }; 177 | name = Debug; 178 | }; 179 | 9497BD041C53D96600ADD79F /* Release */ = { 180 | isa = XCBuildConfiguration; 181 | buildSettings = { 182 | ALWAYS_SEARCH_USER_PATHS = YES; 183 | CLANG_CXX_LANGUAGE_STANDARD = "gnu++0x"; 184 | CLANG_CXX_LIBRARY = "libc++"; 185 | CLANG_ENABLE_MODULES = YES; 186 | CLANG_ENABLE_OBJC_ARC = YES; 187 | CLANG_WARN_BOOL_CONVERSION = YES; 188 | CLANG_WARN_CONSTANT_CONVERSION = YES; 189 | CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR; 190 | CLANG_WARN_EMPTY_BODY = YES; 191 | CLANG_WARN_ENUM_CONVERSION = YES; 192 | CLANG_WARN_INT_CONVERSION = YES; 193 | CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR; 194 | CLANG_WARN_UNREACHABLE_CODE = YES; 195 | CLANG_WARN__DUPLICATE_METHOD_MATCH = YES; 196 | CODE_SIGN_IDENTITY = "-"; 197 | COPY_PHASE_STRIP = NO; 198 | DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym"; 199 | ENABLE_NS_ASSERTIONS = NO; 200 | ENABLE_STRICT_OBJC_MSGSEND = YES; 201 | GCC_C_LANGUAGE_STANDARD = gnu99; 202 | GCC_NO_COMMON_BLOCKS = YES; 203 | GCC_WARN_64_TO_32_BIT_CONVERSION = YES; 204 | GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR; 205 | GCC_WARN_UNDECLARED_SELECTOR = YES; 206 | GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE; 207 | GCC_WARN_UNUSED_FUNCTION = YES; 208 | GCC_WARN_UNUSED_VARIABLE = YES; 209 | HEADER_SEARCH_PATHS = "$(PROJECT_DIR)"; 210 | MACOSX_DEPLOYMENT_TARGET = 10.11; 211 | MTL_ENABLE_DEBUG_INFO = NO; 212 | OTHER_LDFLAGS = "-lz"; 213 | SDKROOT = macosx; 214 | USER_HEADER_SEARCH_PATHS = "$(BUILT_PRODUCTS_DIR)/**"; 215 | }; 216 | name = Release; 217 | }; 218 | 9497BD061C53D96600ADD79F /* Debug */ = { 219 | isa = XCBuildConfiguration; 220 | buildSettings = { 221 | GCC_PREPROCESSOR_DEFINITIONS = ( 222 | "DEBUG=1", 223 | SQLITE_ENABLE_ZIPVFS, 224 | "$(inherited)", 225 | ); 226 | PRODUCT_NAME = "$(TARGET_NAME)"; 227 | USER_HEADER_SEARCH_PATHS = "../cevfs/** $(BUILT_PRODUCTS_DIR)/**"; 228 | }; 229 | name = Debug; 230 | }; 231 | 9497BD071C53D96600ADD79F /* Release */ = { 232 | isa = XCBuildConfiguration; 233 | buildSettings = { 234 | PRODUCT_NAME = "$(TARGET_NAME)"; 235 | USER_HEADER_SEARCH_PATHS = "../cevfs/** $(BUILT_PRODUCTS_DIR)/**"; 236 | }; 237 | name = Release; 238 | }; 239 | /* End XCBuildConfiguration section */ 240 | 241 | /* Begin XCConfigurationList section */ 242 | 9497BCF91C53D96600ADD79F /* Build configuration list for PBXProject "cevfs_example" */ = { 243 | isa = XCConfigurationList; 244 | buildConfigurations = ( 245 | 9497BD031C53D96600ADD79F /* Debug */, 246 | 9497BD041C53D96600ADD79F /* Release */, 247 | ); 248 | defaultConfigurationIsVisible = 0; 249 | defaultConfigurationName = Release; 250 | }; 251 | 9497BD051C53D96600ADD79F /* Build configuration list for PBXNativeTarget "cevfs_example" */ = { 252 | isa = XCConfigurationList; 253 | buildConfigurations = ( 254 | 9497BD061C53D96600ADD79F /* Debug */, 255 | 9497BD071C53D96600ADD79F /* Release */, 256 | ); 257 | defaultConfigurationIsVisible = 0; 258 | defaultConfigurationName = Release; 259 | }; 260 | /* End XCConfigurationList section */ 261 | }; 262 | rootObject = 9497BCF61C53D96600ADD79F /* Project object */; 263 | } 264 | -------------------------------------------------------------------------------- /cevfs_example/main.c: -------------------------------------------------------------------------------- 1 | // 2 | // main.c 3 | // sqlite_compress_test 4 | // 5 | // Created by Ryan Homer on 23/1/2016. 6 | // Copyright © 2016 Murage Inc. All rights reserved. 7 | // 8 | 9 | #include 10 | #include 11 | #include 12 | #include 13 | #include 14 | #include 15 | #include 16 | 17 | #include "cevfs.h" 18 | #include "sqlite3.h" 19 | 20 | #define SQL_TEST_WRITE 1 21 | #define SQL_TEST_READ 1 22 | #define SQL_DEBUG_OUTPUT 1 23 | #define VFS_NAME "example" 24 | 25 | static const char *dbFile = "test.db"; 26 | static const char *dbUri = "file:test.db?block_size=2048"; // lower level page size 27 | 28 | static int callback(void *NotUsed, int argc, char **argv, char **azColName){ 29 | int i; 30 | for(i=0; inIvSz); 92 | 93 | /* According to CCCryptor manpage: "For block ciphers, the output size will always be less than or 94 | equal to the input size plus the size of one block." However, there seems to be a bug as normally 95 | CCCrypt fails with error code kCCBufferTooSmall when the output buffer size is too small, can 96 | crash when size is exactly input size plus size of one block. It works with just 1 more byte. 97 | REF: https://developer.apple.com/library/ios/documentation/System/Conceptual/ManPages_iPhoneOS/man3/CCCrypt.3cc.html */ 98 | size_t nOutSz = nDataInSize+kCCBlockSizeAES128+1; 99 | *ppDataOut = sqlite3_malloc((int)nOutSz); 100 | 101 | CCCryptorStatus ccStatus = CCCrypt( 102 | kCCEncrypt, // enc/dec 103 | kCCAlgorithmAES128, // algorithm 104 | kCCOptionPKCS7Padding, // options: kCCOptionPKCS7Padding, kCCOptionECBMode, 0 = no padding 105 | pCtx->pKey, // 256-bit (32-byte) key 106 | pCtx->nKeySz, // key length (bytes) 107 | pIvOut, // const void *iv 108 | pDataIn, // const void *dataIn 109 | nDataInSize, // data-in length 110 | *ppDataOut, // dataOut; result is written here. 111 | nOutSz, // The size of the dataOut buffer in bytes 112 | nDataSizeOut // On successful return, the number of bytes written to dataOut. 113 | ); 114 | return (ccStatus==kCCSuccess); 115 | } 116 | 117 | int cevfsDecrypt(void *pDecryptCtx, 118 | const void *pDataIn, 119 | size_t nDataInSize, 120 | const void *pIvIn, 121 | void *pDataOut, 122 | size_t nDataBufferSizeOut, 123 | size_t *nDataSizeOut 124 | ){ 125 | struct context *pCtx = (struct context *)pDecryptCtx; 126 | CCCryptorStatus ccStatus = CCCrypt( 127 | kCCDecrypt, // enc/dec 128 | kCCAlgorithmAES128, // algorithm 129 | kCCOptionPKCS7Padding, // options: kCCOptionPKCS7Padding, kCCOptionECBMode, 0 = no padding 130 | pCtx->pKey, // 256-bit (32-byte) key 131 | pCtx->nKeySz, // key length (bytes) 132 | pIvIn, // const void *iv 133 | pDataIn, // const void *dataIn 134 | nDataInSize, // data-in length 135 | pDataOut, // dataOut; result is written here. 136 | nDataBufferSizeOut, // The size of the dataOut buffer in bytes 137 | nDataSizeOut // On successful return, the number of bytes written to dataOut. 138 | ); 139 | return (ccStatus==kCCSuccess); 140 | } 141 | 142 | int cevfsAutoDetect(void *pCtx, const char *zFile, const char *zHdr, size_t *pEncIvSz, CevfsMethods *pMethods) { 143 | *pEncIvSz = kCCBlockSizeAES128; 144 | pMethods->xCompressBound = cevfsCompressBound; 145 | pMethods->xCompress = cevfsCompress;; 146 | pMethods->xUncompress = cevfsUncompress; 147 | pMethods->xEncrypt = cevfsEncrypt; 148 | pMethods->xDecrypt = cevfsDecrypt; 149 | return SQLITE_OK; 150 | } 151 | 152 | void removeDbFile(const char *dbFile){ 153 | char zFile[256]; 154 | char buf[256]; 155 | strcpy(zFile, dbFile); 156 | char *p = strstr(zFile, "?"); 157 | if( p ) *p = '\0'; 158 | sprintf(buf, "%s-journal", zFile); 159 | remove(zFile); 160 | remove(buf); 161 | } 162 | 163 | int testWrite(char* zErrMsg){ 164 | sqlite3 *db; 165 | int rc; 166 | removeDbFile(dbFile); 167 | if( (rc = sqlite3_open_v2(dbUri, &db, SQLITE_OPEN_READWRITE|SQLITE_OPEN_CREATE, VFS_NAME))==SQLITE_OK ){ 168 | /** THE UPPER PAGER (usual pager for DB) IS NOT NEEDED. CEVFS WILL USE ITS OWN (LOWER) PAGER. */ 169 | rc = sqlite3_exec(db, "PRAGMA journal_mode=OFF;", NULL, 0, 0); 170 | 171 | /** ADJUST UPPER PAGER SIZE USING PRAGMA. Set CEVFS (lower) page size in URI. */ 172 | //rc = sqlite3_exec(db, "PRAGMA page_size=2048", NULL, 0, 0); // upper level page size 173 | 174 | if( (rc = sqlite3_exec(db, "CREATE TABLE IF NOT EXISTS dict (key int, entry text, definition text);" \ 175 | "CREATE INDEX IF NOT EXISTS ix_key ON dict(key);" \ 176 | "CREATE INDEX IF NOT EXISTS ix_dict ON dict(entry);", NULL, 0, &zErrMsg))==SQLITE_OK 177 | ){ 178 | char buf[256]; 179 | if( (rc = sqlite3_exec(db, "BEGIN TRANSACTION", NULL, 0, 0)) == SQLITE_OK ){ 180 | for (int i=0; rc==SQLITE_OK && i<100000; i++) { 181 | long key = random(); 182 | snprintf(buf, sizeof(buf), "insert into dict values('%ld', 'This is a test #%03d', 'Trying to compress this string #%03d')", key, i, i); 183 | rc = sqlite3_exec(db, buf, NULL, 0, &zErrMsg); 184 | } 185 | if( rc == SQLITE_OK ){ 186 | rc = sqlite3_exec(db, "COMMIT", NULL, 0, 0); 187 | }else{ 188 | rc = sqlite3_exec(db, "ROLLBACK TRANSACTION", NULL, 0, 0); 189 | } 190 | } 191 | } 192 | 193 | if( sqlite3_errcode(db) == CEVFS_ERROR ){ 194 | switch( sqlite3_extended_errcode(db) ){ 195 | case CEVFS_ERROR_PAGE_SIZE_TOO_SMALL: 196 | break; 197 | case CEVFS_ERROR_MALFORMED_KEY: 198 | break; 199 | case CEVFS_ERROR_EXT_VERSION_TOO_OLD: 200 | break; 201 | case CEVFS_ERROR_VFS_ALREADY_EXISTS: 202 | break; 203 | case CEVFS_ERROR_COMPRESSION_FAILED: 204 | break; 205 | case CEVFS_ERROR_DECOMPRESSION_FAILED: 206 | break; 207 | case CEVFS_ERROR_ENCRYPTION_FAILED: 208 | break; 209 | case CEVFS_ERROR_DECRYPTION_FAILED: 210 | break; 211 | } 212 | } 213 | 214 | if( rc!=SQLITE_OK ) sqlite3_close(db); 215 | else rc = sqlite3_close(db); 216 | } 217 | return rc; 218 | } 219 | 220 | int testRead(char *zErrMsg){ 221 | sqlite3 *db; 222 | int rc; 223 | if( (rc = sqlite3_open_v2(dbFile, &db, SQLITE_OPEN_READONLY, VFS_NAME))==SQLITE_OK ){ 224 | if( (rc = sqlite3_exec(db, "select * from dict where key>1000000000 and key<=1020000000 order by entry;", callback, 0, &zErrMsg))==SQLITE_OK){ 225 | } 226 | } 227 | if( rc==CEVFS_ERROR ){ 228 | if( sqlite3_extended_errcode(db) == CEVFS_ERROR_EXT_VERSION_TOO_OLD ){ 229 | zErrMsg = "You need a newer version of CEVFS"; 230 | } 231 | } 232 | sqlite3_close(db); 233 | return rc; 234 | } 235 | 236 | /* 237 | ** MAIN 238 | */ 239 | int main(int argc, const char * argv[]) { 240 | char *zErrMsg = 0; 241 | int rc; 242 | 243 | time_t t; 244 | srand((unsigned)time(&t)); 245 | 246 | #if SQL_DEBUG_OUTPUT 247 | FILE *f = stdout; 248 | #else 249 | FILE *f = fopen("/dev/null", "w"); 250 | #endif 251 | 252 | // This context will be passed to xAutoDetect and the functions defined within. 253 | struct context ctx; 254 | 255 | // Sample encryption key 256 | char keyBytes[4][8] = { 257 | { 0x8F, 0x22, 0xC9, 0xBA, 0xFE, 0x11, 0x3C, 0xF0 }, 258 | { 0xAF, 0x66, 0x22, 0xC5, 0x73, 0x2E, 0x84, 0xBD }, 259 | { 0x31, 0x16, 0x19, 0x83, 0x8A, 0x6F, 0xAA, 0x24 }, 260 | { 0x71, 0xD8, 0x6C, 0x32, 0x99, 0xCA, 0x29, 0x2A } 261 | }; 262 | 263 | ctx.pKey = keyBytes; // 32-bit encryption hex key 264 | ctx.nKeySz = kCCKeySizeAES256; // key size in bytes 265 | ctx.nIvSz = kCCBlockSizeAES128; // size of IV in bytes 266 | 267 | if( (rc = cevfs_create_vfs(VFS_NAME, NULL, &ctx, cevfsAutoDetect, 0))==SQLITE_OK ){ 268 | rc = SQLITE_OK; 269 | 270 | #if SQL_TEST_WRITE 271 | rc = testWrite(zErrMsg); 272 | #endif 273 | 274 | #if SQL_TEST_READ 275 | if( rc==SQLITE_OK ){ 276 | rc = testRead(zErrMsg); 277 | } 278 | #endif 279 | } 280 | 281 | if( rc != SQLITE_OK ){ 282 | if( zErrMsg ) fprintf(stderr, "SQL error: %s\n", zErrMsg); 283 | else fprintf(stderr, "SQL error: %d\n", rc); 284 | sqlite3_free(zErrMsg); 285 | }else{ 286 | fputs("Done\n", stdout); 287 | } 288 | fclose(f); 289 | cevfs_destroy_vfs(VFS_NAME); 290 | return 0; 291 | } 292 | --------------------------------------------------------------------------------