├── .DS_Store ├── Gemfile ├── Gemfile.lock ├── README.md ├── data └── db.bin ├── include ├── .DS_Store ├── TextTable.h ├── bpt.h └── predefined.h ├── makefile ├── md_image ├── .DS_Store ├── 1.png ├── 10.png ├── 11.png ├── 12.png ├── 13.png ├── 14.png ├── 15.png ├── 16.png ├── 17.png ├── 18.png ├── 2.png ├── 3.gif ├── 4.gif ├── 5.jpg ├── 5.png ├── 6.png ├── 7.png ├── 8-a.png ├── 8.png ├── 9.png ├── Bplustreebuild.gif.crdownload ├── b+.gif ├── bb.gif ├── btreebuild.gif.crdownload └── image-20180831232232550.png ├── spec └── main_spec.rb └── src ├── .DS_Store ├── bpt.cc └── duck_db.cpp /.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/.DS_Store -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | # gem 'github-pages', group: :jekyll_plugins 3 | # gem "jekyll-theme-minimal" 4 | gem "rspec" 5 | -------------------------------------------------------------------------------- /Gemfile.lock: -------------------------------------------------------------------------------- 1 | GEM 2 | remote: https://rubygems.org/ 3 | specs: 4 | diff-lcs (1.3) 5 | rspec (3.8.0) 6 | rspec-core (~> 3.8.0) 7 | rspec-expectations (~> 3.8.0) 8 | rspec-mocks (~> 3.8.0) 9 | rspec-core (3.8.0) 10 | rspec-support (~> 3.8.0) 11 | rspec-expectations (3.8.1) 12 | diff-lcs (>= 1.2.0, < 2.0) 13 | rspec-support (~> 3.8.0) 14 | rspec-mocks (3.8.0) 15 | diff-lcs (>= 1.2.0, < 2.0) 16 | rspec-support (~> 3.8.0) 17 | rspec-support (3.8.0) 18 | 19 | PLATFORMS 20 | ruby 21 | 22 | DEPENDENCIES 23 | rspec 24 | 25 | BUNDLED WITH 26 | 1.16.4 27 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## c/c++ build a simple b+tree RDMS（利用c/c++ 开发基于B+树的小型关系型数据库 ） 2 | 3 | 4 | 开发环境：ubuntu 5 | 6 | 1. 输入`make`进行编译，输入`./duck_db`即可运行； 7 | 2. `make clean`清除文件。 8 | 3. 数据库二进制文件存的目录：./data/db.bin 9 | 10 | 11 | 12 | [TOC] 13 | 14 | #### 目的 15 | 16 | > “What I cannot create, I do not understand.” – [Richard Feynman](https://en.m.wikiquote.org/wiki/Richard_Feynman) 17 | 18 | 正如这句名言，理解一个事物最好的办法就是亲自设计制作它，本文将介绍一个简单的关系型数据库系统（类似MySQL、sqlite等）的开发过程，用于理解关系型数据库基本工作原理，我称它为duck_db。 19 | 20 | 帮助界面： 21 | 22 | ![image-20180902192805872](./md_image/1.png) 23 | 24 | 主要功能特色如下： 25 | 26 | 1. 使用C/C++开发； 27 | 2. 已实现基本的CURD操作，使用控制台SQL： 28 | 1. 新建记录（create），操作SQL：**insert db {index} {name} {age} {email};** 29 | 2. 更新记录（update），操作语句：**update db {name} {age} {email} where id={index};** 30 | 3. 单条读取记录（read），操作语句：**select * from db where id={index};** 31 | 4. 范围读取（read in range），操作语句：**select * from db where id in({minIndex},{maxIndex});** 32 | 5. 删除记录（delete），操作语句：**delete from db where id ={index};** 33 | 3. 底层使用B+树（B+ TREE）构建索引； 34 | 4. 利用二进制存储数据表； 35 | 5. 暂不支持自定义表结构，数据库表结构固定为：id，name，age，email，如图： 36 | 37 | ![image-20180902195643745](./md_image/2.png) 38 | 39 | #### 过程 40 | 41 | 下图是sqlite的架构图： 42 | 43 | ![3](./md_image/3.gif) 44 | 45 | 参考上图，主要开发步骤如下： 46 | 47 | 1. 创建一个控制台对话交互程序（REPL：read-execute-print loop）； 48 | 2. 创建一个简单的词法分析器用来解析SQL语句； 49 | 3. 编写CURD函数实现数据库的增删改查操作； 50 | 4. 创建一个b+树索引引擎，进行数据库的索引和磁盘读写操作，数据表将以二进制的形式存储。 51 | 52 | ##### Step 1：REPL（read-execute-print loop） 53 | 54 | 使用一个字符数组（*userCommand）接受用户输入，并且不断比对关键字，用于识别SQL语句指令： 55 | 56 | 代码片段： 57 | 58 | ```c++ 59 | // REPL 60 | void selectCommand(){ 61 | char *userCommand = new char[256]; 62 | while(true){ 63 | // 获取用户输入 64 | cin.getline(userCommand,256); 65 | if(strcmp(userCommand,".exit") == 0){ 66 | // 退出系统 67 | break; 68 | }else if(strcmp(userCommand,".help") == 0){ 69 | // 帮助 70 | }else if(strcmp(userCommand,".reset") == 0){ 71 | // 重置数据库 72 | }else if(strncmp(userCommand,"insert",6) == 0){ 73 | // 插入记录 74 | }else if(strncmp(userCommand,"delete",6) == 0){ 75 | // 删除记录 76 | }else if(strncmp(userCommand,"select",6) == 0){ 77 | // 读取记录 78 | }else if(strncmp(userCommand,"update",6) == 0){ 79 | // 更新记录 80 | }else{ 81 | // 错误信息 82 | } 83 | } 84 | 85 | } 86 | ``` 87 | ##### Step 2：词法分析器 88 | 89 | 使用sscanf函数，实现基本的SQL解析，如： 90 | 91 | ```c++ 92 | sscanf(userCommand,"insert db %d %s %d %s;",keyIndex, insertData->name,&(insertData->age),insertData->email); 93 | ``` 94 | 95 | 表示解析类似如下SQL语句，其他SQL操作类似： 96 | 97 | ```sql 98 | insert db 3 username 28 user@gmail.com; 99 | ``` 100 | 101 | ##### Step 3：表数据结构 102 | 103 | 暂不支持自定义表结构，数据库表结构固定为：id（主键），name，age，email： 104 | 105 | ```c++ 106 | struct value_t{ 107 | char name[256]; // 姓名 108 | int age; // 年龄 109 | char email[256]; // email 110 | }; 111 | ``` 112 | 113 | ##### Step 4：CURD函数原型 114 | 115 | CURD函数原型如下，我们将稍后完善这些函数： 116 | 117 | ```c++ 118 | // 插入记录 119 | int insertRecord(); 120 | // 删除记录 121 | int deleteRecord(); 122 | // 搜索记录(根据索引) 123 | int searchRecord(); 124 | // 搜索记录(范围搜索) 125 | int searchAll(); 126 | // 更新记录 127 | int updateRecord(); 128 | ``` 129 | 130 | ##### Step 5：B树和B+树 131 | 132 | ###### 一、索引 133 | 134 | 数据库一般会操作海量的数据，这些数据都是存在磁盘上的。查询是数据库的最主要功能之一，我们可以将查询的过程类比在牛津词典中查单词。我们都希望查询数据的速度能尽可能的快。 135 | 136 | ![image-20180904094621292](./md_image/5.png) 137 | 138 | 上图是《牛津高阶词典》，共收录了8万多个单词，假设我需要查询“hash”这个单词，我们有两种方式：一是从 A->H 一个个遍历；二是按图示红圈的方式先找到H，再看单词第二个字母，再看第三个......显然我们一般用的是第二种方式，而红圈内的字母即索引（index）。 139 | 140 | MySQL官方对索引的定义为：索引（Index）是帮助MySQL高效获取数据的数据结构。提取句子主干，就可以得到索引的本质：索引是数据结构。 141 | 142 | 最基本的查询算法当然是[顺序查找](http://en.wikipedia.org/wiki/Linear_search)（linear search），这种复杂度为O(n)的算法在数据量很大时显然是糟糕的，随着计算机科学的发展，我们先后发明了[二分查找](http://en.wikipedia.org/wiki/Binary_search_algorithm)（binary search）、[二叉树查找](http://en.wikipedia.org/wiki/Binary_search_tree)（binary tree search）等算法。每种查找算法都只能应用于特定的数据结构之上，例如二分查找要求被检索数据有序，而二叉树查找只能应用于[二叉查找树](http://en.wikipedia.org/wiki/Binary_search_tree)上，但是数据本身的组织结构不可能完全满足各种数据结构（例如，理论上不可能同时将两列都按顺序进行组织），所以，在数据之外，数据库系统还维护着满足特定查找算法的数据结构，这些数据结构以某种方式引用（指向）数据，这样就可以在这些数据结构上实现高级查找算法。这种数据结构，就是索引。 143 | 144 | ![img](./md_image/6.png) 145 | 146 | 上图展示了一种可能的索引方式。左边是数据表，一共有两列七条记录，最左边的是数据记录的物理地址（注意逻辑上相邻的记录在磁盘上也并不是一定物理相邻的）。为了加快Col2的查找，可以维护一个右边所示的二叉查找树，每个节点分别包含索引键值和一个指向对应数据记录物理地址的指针，这样就可以运用二叉查找在的复杂度内获取到相应数据。 147 | 148 | | | Unsorted Array of rows | Sorted Array of rows | Tree of nodes | 149 | | ------------- | ---------------------- | -------------------- | -------------------------------- | 150 | | Pages contain | only data | only data | metadata, primary keys, and data | 151 | | Rows per page | more | more | fewer | 152 | | Insertion | O(1) | O(n) | O(log(n)) | 153 | | Deletion | O(n) | O(n) | O(log(n)) | 154 | | Lookup by id | O(n) | O(log(n)) | O(log(n) | 155 | 156 | 目前大部分数据库系统及文件系统都采用B-Tree或其变种B+Tree作为索引结构，没有使用二叉查找树或其进化品种[红黑树](http://en.wikipedia.org/wiki/Red-black_tree)（red-black tree）的原因请参见这篇文章：http://www.cnblogs.com/serendipity-fly/p/9300360.html 157 | 158 | ###### 二、B树和B+树 159 | 160 | 维基百科对B树的定义为“在计算机科学中，B树（B-tree）是一种树状数据结构，它能够存储数据、对其进行排序并允许以O(log n)的时间复杂度运行进行查找、顺序读取、插入和删除的数据结构。B树，概括来说是一个节点可以拥有多于2个子节点的二叉查找树。与自平衡二叉查找树不同，B-树为系统最优化**大块数据的读和写操作**。B-tree算法减少定位记录时所经历的中间过程，从而加快存取速度。普遍运用在**数据库**和**文件系统**。” 161 | 162 | **B树** 163 | 164 | **B 树**可以看作是对2-3查找树的一种扩展，即他允许每个节点有M-1个子节点。 165 | 166 | - 根节点至少有两个子节点； 167 | - 每个节点有M-1个key，并且以升序排列； 168 | - 位于M-1和M key的子节点的值位于M-1 和M key对应的Value之间； 169 | - 其它节点至少有M/2个子节点； 170 | 171 | 下图是一个M=4 阶的B树： 172 | 173 | ![7](./md_image/7.png) 174 | 175 | 下面是往B树中依次插入 176 | 177 | **6 10 4 14 5 11 15 3 2 12 1 7 8 8 6 3 6 21 5 15 15 6 32 23 45 65 7 8 6 5 4** 178 | 179 | 的演示动画： 180 | 181 | ![bb](./md_image/bb.gif) 182 | 183 | **B+树** 184 | 185 | **B+**树是对B树的一种变形树，它与B树的差异在于： 186 | 187 | - 有k个子结点的结点必然有k个关键码； 188 | - 非叶结点仅具有索引作用，跟记录有关的信息均存放在叶结点中。 189 | - 树的所有叶结点构成一个有序链表，可以按照关键码排序的次序遍历全部记录。 190 | 191 | 如下图，是一个B+树： 192 | 193 | ![9](./md_image/9.png) 194 | 195 | 下面是往B+树中依次插入 196 | 197 | **6 10 4 14 5 11 15 3 2 12 1 7 8 8 6 3 6 21 5 15 15 6 32 23 45 65 7 8 6 5 4** 198 | 199 | 的演示动画： 200 | 201 | ![b+](./md_image/b+.gif) 202 | 203 | B和B+树的区别在于，B+树的非叶子结点只包含导航信息，不包含实际的值，所有的叶子结点和相连的节点使用链表相连，便于区间查找和遍历。 204 | 205 | **B+ 树的优点在于：** 206 | 207 | - 由于B+树在内部节点上不包含数据信息，因此在内存页中能够存放更多的key。 数据存放的更加紧密，具有更好的空间局部性。因此访问叶子节点上关联的数据也具有更好的缓存命中率。 208 | - B+树的叶子结点都是相链的，因此对整棵树的便利只需要一次线性遍历叶子结点即可。而且由于数据顺序排列并且相连，所以便于区间查找和搜索。而B树则需要进行每一层的递归遍历。相邻的元素可能在内存中不相邻，所以缓存命中性没有B+树好。 209 | 210 | 但是B树也有优点，其优点在于，由于B树的每一个节点都包含key和value，因此经常访问的元素可能离根节点更近，因此访问也更迅速。下面是B 树和B+树的区别图： 211 | 212 | ![8](./md_image/8.png) 213 | 214 | ###### 三、MySQL的MyISAM索引和InnoDB索引 215 | 216 | **MyISAM索引** 217 | 218 | MyISAM引擎使用B+Tree作为索引结构，叶节点的data域存放的是数据记录的地址。下图是MyISAM索引的原理图： 219 | 220 | ![8-a](./md_image/8-a.png) 221 | 222 | 223 | 224 | 这里设表一共有三列，假设我们以Col1为主键，则上图是一个MyISAM表的主索引（Primary key）示意。可以看出MyISAM的索引文件仅仅保存数据记录的地址。 225 | 226 | **InnoDB索引** 227 | 228 | 虽然InnoDB也使用B+Tree作为索引结构，但具体实现方式却与MyISAM截然不同。 229 | 230 | 第一个重大区别是InnoDB的数据文件本身就是索引文件。从上文知道，MyISAM索引文件和数据文件是分离的，索引文件仅保存数据记录的地址。而在InnoDB中，表数据文件本身就是按B+Tree组织的一个索引结构，这棵树的叶节点data域保存了完整的数据记录。这个索引的key是数据表的主键，因此InnoDB表数据文件本身就是主索引。 231 | 232 | ![10](./md_image/10.png) 233 | 234 | 235 | 236 | **本文将模拟InnoDB引擎：** 237 | 238 | 因B+数实现逻辑较复杂，下面代码是函数原型，完整代码请查看文末地址： 239 | 240 | ```c++ 241 | // 构造函数 242 | bplus_tree(const char *path, bool force_empty = false); 243 | /* abstract operations */ 244 | // 搜索 245 | int search(const key_t& key, value_t *value) const; 246 | // 范围搜索 247 | int search_range(key_t *left, const key_t &right, 248 | value_t *values, size_t max, bool *next = NULL) const; 249 | // 删除 250 | int remove(const key_t& key); 251 | // 新增记录 252 | int insert(const key_t& key, value_t value); 253 | // 更新 254 | int update(const key_t& key, value_t value); 255 | ``` 256 | ###### 四、文件IO 257 | 258 | B+tree的叶节点data域保存了完整的数据记录。我们需要将数据存入磁盘，构建相应的函数进行文件IO操作。下面是部分代码： 259 | 260 | ```c++ 261 | /* read block from disk */ 262 | template 263 | int map(T *block, off_t offset) const 264 | { 265 | return map(block, offset, sizeof(T)); 266 | } 267 | /* write block to disk */ 268 | template 269 | int unmap(T *block, off_t offset) const 270 | { 271 | return unmap(block, offset, sizeof(T)); 272 | } 273 | ``` 274 | 275 | ##### Step 6：完善CURD函数，初始化系统 276 | 277 | 以插入记录函数（C操作）为例： 278 | 279 | ```c++ 280 | // insert 281 | int insertRecord(bplus_tree *treePtr,int *index, value_t *values){ 282 | bpt::key_t key; 283 | //转换格式 284 | intToKeyT(&key,index); 285 | return (*treePtr).insert(key, *values); 286 | } 287 | ``` 288 | 289 | 函数参数分别为`bplus_tree *treePtr,int *index, value_t *values`，分别为B+树指针，索引指针，插入记录的参数指针。 290 | 291 | 再建立一个启动函数`initialSystem() ` 给 `main()` 调用： 292 | 293 | ```C++ 294 | // 创建一个B+树指针 295 | bplus_tree *duck_db_ptr; 296 | // initial 297 | void initialSystem(){ 298 | // step 1 : 打印帮助信息 299 | printHelpMess(); 300 | // step 2 : 从文件初始化B+树，如果磁盘没有文件，将自动创建新的 301 | bplus_tree duck_db(dbFileName, (!is_file_exist(dbFileName))); 302 | // 传递地址给指针 303 | duck_db_ptr = &duck_db; 304 | // step 3 : 进入REPL SQL命令解析程序(insert,delete,update,search) 305 | selectCommand(); 306 | } 307 | ``` 308 | 309 | main函数 310 | 311 | ``` c++ 312 | int main(int argc, char *argv[]) 313 | { 314 | // 启动系统 315 | initialSystem(); 316 | } 317 | ``` 318 | 319 | #### 测试 320 | 321 | 使用[rspec](http://rspec.info/)测试duck_db，我们将插入5000条数据： 322 | 323 | ```ruby 324 | it 'inserting test ' do 325 | script = (1..5000).map do |i| 326 | "insert db #{i} user#{i} #{i*12} person#{i}@example.com" 327 | end 328 | script << ".exit" 329 | result = run_script(script) 330 | expect(result.last(2)).to match_array([ 331 | ">insert", 332 | "> bye!", 333 | ]) 334 | end 335 | ``` 336 | 337 | 输入`bundle exec rspec`，结果如下： 338 | 339 | ![image-20180906161404599](./md_image/12.png) 340 | 341 | 共耗时28.71秒，平均插入每条记录需要0.005742秒，并产生了5.1MB的数据文件： 342 | 343 | ![image-20180906161613057](./md_image/13.png) 344 | 345 | 随机查询一条数据： 346 | 347 | ![image-20180906162224569](./md_image/14.png) 348 | 349 | ![image-20180906162308633](./md_image/15.png) 350 | 351 | 打印一个范围： 352 | 353 | ![image-20180906162543967](./md_image/16.png) 354 | 355 | 更新一条id=2634的记录： 356 | 357 | ![image-20180906162915686](./md_image/17.png) 358 | 359 | 删除id=4265的记录： 360 | 361 | ![image-20180906163031482](./md_image/18.png) 362 | 363 | #### 评价 364 | 365 | duck_db利用B+树实现了数据库基本的CURD操作，但是距离真实可用的数据库还有很大的差距，比如： 366 | 367 | 1. 无法创建自定义表； 368 | 2. 不支持事物处理，IO优化； 369 | 3. 不支持远程登录数据库，只可以本地使用； 370 | 4. CURD高级功能如函数，约束，连表操作等不支持； 371 | 5. 更多高级特性等。 372 | 373 | #### 完整代码 374 | 375 | 项目代码已托管至GitHub：https://github.com/enpeizhao/duck_db 376 | 377 | -------------------------------------------------------------------------------- /data/db.bin: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/data/db.bin -------------------------------------------------------------------------------- /include/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/include/.DS_Store -------------------------------------------------------------------------------- /include/TextTable.h: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | class TextTable { 8 | 9 | public: 10 | enum class Alignment { LEFT, RIGHT }; 11 | typedef std::vector< std::string > Row; 12 | TextTable( char horizontal = '-', char vertical = '|', char corner = '+' ) : 13 | _horizontal( horizontal ), 14 | _vertical( vertical ), 15 | _corner( corner ) 16 | {} 17 | 18 | void setAlignment( unsigned i, Alignment alignment ) 19 | { 20 | _alignment[ i ] = alignment; 21 | } 22 | 23 | Alignment alignment( unsigned i ) const 24 | { return _alignment[ i ]; } 25 | 26 | char vertical() const 27 | { return _vertical; } 28 | 29 | char horizontal() const 30 | { return _horizontal; } 31 | 32 | void add( std::string const & content ) 33 | { 34 | _current.push_back( content ); 35 | } 36 | 37 | void endOfRow() 38 | { 39 | _rows.push_back( _current ); 40 | _current.assign( 0, "" ); 41 | } 42 | 43 | template 44 | void addRow( Iterator begin, Iterator end ) 45 | { 46 | for( auto i = begin; i != end; ++i ) { 47 | add( * i ); 48 | } 49 | endOfRow(); 50 | } 51 | 52 | template 53 | void addRow( Container const & container ) 54 | { 55 | addRow( container.begin(), container.end() ); 56 | } 57 | 58 | std::vector< Row > const & rows() const 59 | { 60 | return _rows; 61 | } 62 | 63 | void setup() const 64 | { 65 | determineWidths(); 66 | setupAlignment(); 67 | } 68 | 69 | std::string ruler() const 70 | { 71 | std::string result; 72 | result += _corner; 73 | for( auto width = _width.begin(); width != _width.end(); ++ width ) { 74 | result += repeat( * width, _horizontal ); 75 | result += _corner; 76 | } 77 | 78 | return result; 79 | } 80 | 81 | int width( unsigned i ) const 82 | { return _width[ i ]; } 83 | 84 | private: 85 | char _horizontal; 86 | char _vertical; 87 | char _corner; 88 | Row _current; 89 | std::vector< Row > _rows; 90 | std::vector< unsigned > mutable _width; 91 | std::map< unsigned, Alignment > mutable _alignment; 92 | 93 | static std::string repeat( unsigned times, char c ) 94 | { 95 | std::string result; 96 | for( ; times > 0; -- times ) 97 | result += c; 98 | 99 | return result; 100 | } 101 | 102 | unsigned columns() const 103 | { 104 | return _rows[ 0 ].size(); 105 | } 106 | 107 | void determineWidths() const 108 | { 109 | _width.assign( columns(), 0 ); 110 | for ( auto rowIterator = _rows.begin(); rowIterator != _rows.end(); ++ rowIterator ) { 111 | Row const & row = * rowIterator; 112 | for ( unsigned i = 0; i < row.size(); ++i ) { 113 | _width[ i ] = _width[ i ] > row[ i ].size() ? _width[ i ] : row[ i ].size(); 114 | } 115 | } 116 | } 117 | 118 | void setupAlignment() const 119 | { 120 | for ( unsigned i = 0; i < columns(); ++i ) { 121 | if ( _alignment.find( i ) == _alignment.end() ) { 122 | _alignment[ i ] = Alignment::LEFT; 123 | } 124 | } 125 | } 126 | }; 127 | 128 | std::ostream & operator<<( std::ostream & stream, TextTable const & table ) 129 | { 130 | table.setup(); 131 | stream << table.ruler() << "\n"; 132 | for ( auto rowIterator = table.rows().begin(); rowIterator != table.rows().end(); ++ rowIterator ) { 133 | TextTable::Row const & row = * rowIterator; 134 | stream << table.vertical(); 135 | for ( unsigned i = 0; i < row.size(); ++i ) { 136 | auto alignment = table.alignment( i ) == TextTable::Alignment::LEFT ? std::left : std::right; 137 | stream << std::setw( table.width( i ) ) << alignment << row[ i ]; 138 | stream << table.vertical(); 139 | } 140 | stream << "\n"; 141 | stream << table.ruler() << "\n"; 142 | } 143 | 144 | return stream; 145 | } 146 | -------------------------------------------------------------------------------- /include/bpt.h: -------------------------------------------------------------------------------- 1 | #ifndef BPT_H 2 | #define BPT_H 3 | 4 | #include 5 | #include 6 | #include 7 | #include 8 | 9 | #ifndef UNIT_TEST 10 | #include "predefined.h" 11 | #else 12 | // #include "unit_test_predefined.h" 13 | #endif 14 | 15 | namespace bpt { 16 | 17 | /* offsets */ 18 | #define OFFSET_META 0 19 | #define OFFSET_BLOCK OFFSET_META + sizeof(meta_t) 20 | #define SIZE_NO_CHILDREN sizeof(leaf_node_t) - BP_ORDER * sizeof(record_t) 21 | 22 | /* meta information of B+ tree */ 23 | typedef struct { 24 | size_t order; /* `order` of B+ tree */ 25 | size_t value_size; /* size of value */ 26 | size_t key_size; /* size of key */ 27 | size_t internal_node_num; /* how many internal nodes */ 28 | size_t leaf_node_num; /* how many leafs */ 29 | size_t height; /* height of tree (exclude leafs) */ 30 | off_t slot; /* where to store new block */ 31 | off_t root_offset; /* where is the root of internal nodes */ 32 | off_t leaf_offset; /* where is the first leaf */ 33 | } meta_t; 34 | 35 | /* internal nodes' index segment */ 36 | struct index_t { 37 | key_t key; 38 | off_t child; /* child's offset */ 39 | }; 40 | 41 | /*** 42 | * internal node block 43 | ***/ 44 | struct internal_node_t { 45 | typedef index_t * child_t; 46 | 47 | off_t parent; /* parent node offset */ 48 | off_t next; 49 | off_t prev; 50 | size_t n; /* how many children */ 51 | index_t children[BP_ORDER]; 52 | }; 53 | 54 | /* the final record of value */ 55 | struct record_t { 56 | key_t key; 57 | value_t value; 58 | }; 59 | 60 | /* leaf node block */ 61 | struct leaf_node_t { 62 | typedef record_t *child_t; 63 | 64 | off_t parent; /* parent node offset */ 65 | off_t next; 66 | off_t prev; 67 | size_t n; 68 | record_t children[BP_ORDER]; 69 | }; 70 | 71 | /* the encapulated B+ tree */ 72 | class bplus_tree { 73 | public: 74 | bplus_tree(const char *path, bool force_empty = false); 75 | 76 | /* abstract operations */ 77 | int search(const key_t& key, value_t *value) const; 78 | 79 | int search_range(key_t *left, const key_t &right, 80 | value_t *values, size_t max, bool *next = NULL) const; 81 | int remove(const key_t& key); 82 | int insert(const key_t& key, value_t value); 83 | int update(const key_t& key, value_t value); 84 | meta_t get_meta() const { 85 | return meta; 86 | }; 87 | 88 | #ifndef UNIT_TEST 89 | private: 90 | #else 91 | public: 92 | #endif 93 | char path[512]; 94 | meta_t meta; 95 | 96 | /* init empty tree */ 97 | void init_from_empty(); 98 | 99 | /* find index */ 100 | off_t search_index(const key_t &key) const; 101 | 102 | /* find leaf */ 103 | off_t search_leaf(off_t index, const key_t &key) const; 104 | off_t search_leaf(const key_t &key) const 105 | { 106 | return search_leaf(search_index(key), key); 107 | } 108 | 109 | /* remove internal node */ 110 | void remove_from_index(off_t offset, internal_node_t &node, 111 | const key_t &key); 112 | 113 | /* borrow one key from other internal node */ 114 | bool borrow_key(bool from_right, internal_node_t &borrower, 115 | off_t offset); 116 | 117 | /* borrow one record from other leaf */ 118 | bool borrow_key(bool from_right, leaf_node_t &borrower); 119 | 120 | /* change one's parent key to another key */ 121 | void change_parent_child(off_t parent, const key_t &o, const key_t &n); 122 | 123 | /* merge right leaf to left leaf */ 124 | void merge_leafs(leaf_node_t *left, leaf_node_t *right); 125 | 126 | void merge_keys(index_t *where, internal_node_t &left, 127 | internal_node_t &right); 128 | 129 | /* insert into leaf without split */ 130 | void insert_record_no_split(leaf_node_t *leaf, 131 | const key_t &key, const value_t &value); 132 | 133 | /* add key to the internal node */ 134 | void insert_key_to_index(off_t offset, const key_t &key, 135 | off_t value, off_t after); 136 | void insert_key_to_index_no_split(internal_node_t &node, const key_t &key, 137 | off_t value); 138 | 139 | /* change children's parent */ 140 | void reset_index_children_parent(index_t *begin, index_t *end, 141 | off_t parent); 142 | 143 | template 144 | void node_create(off_t offset, T *node, T *next); 145 | 146 | template 147 | void node_remove(T *prev, T *node); 148 | 149 | /* multi-level file open/close */ 150 | mutable FILE *fp; 151 | mutable int fp_level; 152 | void open_file(const char *mode = "rb+") const 153 | { 154 | // `rb+` will make sure we can write everywhere without truncating file 155 | if (fp_level == 0) 156 | fp = fopen(path, mode); 157 | 158 | ++fp_level; 159 | } 160 | 161 | void close_file() const 162 | { 163 | if (fp_level == 1) 164 | fclose(fp); 165 | 166 | --fp_level; 167 | } 168 | 169 | /* alloc from disk */ 170 | off_t alloc(size_t size) 171 | { 172 | off_t slot = meta.slot; 173 | meta.slot += size; 174 | return slot; 175 | } 176 | 177 | off_t alloc(leaf_node_t *leaf) 178 | { 179 | leaf->n = 0; 180 | meta.leaf_node_num++; 181 | return alloc(sizeof(leaf_node_t)); 182 | } 183 | 184 | off_t alloc(internal_node_t *node) 185 | { 186 | node->n = 1; 187 | meta.internal_node_num++; 188 | return alloc(sizeof(internal_node_t)); 189 | } 190 | 191 | void unalloc(leaf_node_t *leaf, off_t offset) 192 | { 193 | --meta.leaf_node_num; 194 | } 195 | 196 | void unalloc(internal_node_t *node, off_t offset) 197 | { 198 | --meta.internal_node_num; 199 | } 200 | 201 | /* read block from disk */ 202 | int map(void *block, off_t offset, size_t size) const 203 | { 204 | open_file(); 205 | fseek(fp, offset, SEEK_SET); 206 | size_t rd = fread(block, size, 1, fp); 207 | close_file(); 208 | 209 | return rd - 1; 210 | } 211 | 212 | template 213 | int map(T *block, off_t offset) const 214 | { 215 | return map(block, offset, sizeof(T)); 216 | } 217 | 218 | /* write block to disk */ 219 | int unmap(void *block, off_t offset, size_t size) const 220 | { 221 | open_file(); 222 | fseek(fp, offset, SEEK_SET); 223 | size_t wd = fwrite(block, size, 1, fp); 224 | close_file(); 225 | 226 | return wd - 1; 227 | } 228 | template 229 | int unmap(T *block, off_t offset) const 230 | { 231 | return unmap(block, offset, sizeof(T)); 232 | } 233 | }; 234 | 235 | } 236 | 237 | #endif /* end of BPT_H */ 238 | -------------------------------------------------------------------------------- /include/predefined.h: -------------------------------------------------------------------------------- 1 | #ifndef PREDEFINED_H 2 | #define PREDEFINED_H 3 | 4 | #include 5 | 6 | namespace bpt { 7 | 8 | /* predefined B+ info */ 9 | #define BP_ORDER 50 10 | 11 | /* key/value type */ 12 | 13 | struct value_t{ 14 | char name[256]; 15 | int age; 16 | char email[256]; 17 | }; 18 | 19 | // typedef int value_t; 20 | struct key_t { 21 | char k[16]; 22 | 23 | key_t(const char *str = "") 24 | { 25 | bzero(k, sizeof(k)); 26 | strcpy(k, str); 27 | } 28 | }; 29 | 30 | inline int keycmp(const key_t &a, const key_t &b) { 31 | int x = strlen(a.k) - strlen(b.k); 32 | return x == 0 ? strcmp(a.k, b.k) : x; 33 | } 34 | 35 | #define OPERATOR_KEYCMP(type) \ 36 | bool operator< (const key_t &l, const type &r) {\ 37 | return keycmp(l, r.key) < 0;\ 38 | }\ 39 | bool operator< (const type &l, const key_t &r) {\ 40 | return keycmp(l.key, r) < 0;\ 41 | }\ 42 | bool operator== (const key_t &l, const type &r) {\ 43 | return keycmp(l, r.key) == 0;\ 44 | }\ 45 | bool operator== (const type &l, const key_t &r) {\ 46 | return keycmp(l.key, r) == 0;\ 47 | } 48 | 49 | } 50 | 51 | #endif /* end of PREDEFINED_H */ 52 | -------------------------------------------------------------------------------- /makefile: -------------------------------------------------------------------------------- 1 | 2 | CXX = g++ 3 | 4 | SRC_DIR = ./src/ 5 | TARGET = duck_db 6 | OBJ = bpt.o duck_db.o 7 | 8 | $(TARGET):$(OBJ) 9 | $(CXX) -o $(TARGET) $(OBJ) 10 | rm -rf $(OBJ) 11 | 12 | bpt.o: 13 | $(CXX) -c $(SRC_DIR)bpt.cc 14 | 15 | duck_db.o: 16 | $(CXX) -c $(SRC_DIR)duck_db.cpp 17 | 18 | clean: 19 | rm -rf $(OBJ) $(TARGET) -------------------------------------------------------------------------------- /md_image/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/.DS_Store -------------------------------------------------------------------------------- /md_image/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/1.png -------------------------------------------------------------------------------- /md_image/10.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/10.png -------------------------------------------------------------------------------- /md_image/11.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/11.png -------------------------------------------------------------------------------- /md_image/12.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/12.png -------------------------------------------------------------------------------- /md_image/13.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/13.png -------------------------------------------------------------------------------- /md_image/14.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/14.png -------------------------------------------------------------------------------- /md_image/15.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/15.png -------------------------------------------------------------------------------- /md_image/16.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/16.png -------------------------------------------------------------------------------- /md_image/17.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/17.png -------------------------------------------------------------------------------- /md_image/18.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/18.png -------------------------------------------------------------------------------- /md_image/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/2.png -------------------------------------------------------------------------------- /md_image/3.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/3.gif -------------------------------------------------------------------------------- /md_image/4.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/4.gif -------------------------------------------------------------------------------- /md_image/5.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/5.jpg -------------------------------------------------------------------------------- /md_image/5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/5.png -------------------------------------------------------------------------------- /md_image/6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/6.png -------------------------------------------------------------------------------- /md_image/7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/7.png -------------------------------------------------------------------------------- /md_image/8-a.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/8-a.png -------------------------------------------------------------------------------- /md_image/8.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/8.png -------------------------------------------------------------------------------- /md_image/9.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/9.png -------------------------------------------------------------------------------- /md_image/Bplustreebuild.gif.crdownload: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/Bplustreebuild.gif.crdownload -------------------------------------------------------------------------------- /md_image/b+.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/b+.gif -------------------------------------------------------------------------------- /md_image/bb.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/bb.gif -------------------------------------------------------------------------------- /md_image/btreebuild.gif.crdownload: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/btreebuild.gif.crdownload -------------------------------------------------------------------------------- /md_image/image-20180831232232550.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/md_image/image-20180831232232550.png -------------------------------------------------------------------------------- /spec/main_spec.rb: -------------------------------------------------------------------------------- 1 | describe 'database' do 2 | before do 3 | `rm -rf ./data/db.bin` 4 | end 5 | 6 | def run_script(commands) 7 | raw_output = nil 8 | IO.popen("./duck_db", "r+") do |pipe| 9 | commands.each do |command| 10 | begin 11 | pipe.puts command 12 | rescue Errno::EPIPE 13 | break 14 | end 15 | end 16 | 17 | pipe.close_write 18 | 19 | # Read entire output 20 | raw_output = pipe.gets(nil) 21 | end 22 | raw_output.split("\n") 23 | end 24 | 25 | 26 | it 'insert test' do 27 | script = (1..5000).map do |i| 28 | "insert db #{i} user#{i} #{i*12} person#{i}@example.com" 29 | end 30 | script << ".exit" 31 | result = run_script(script) 32 | expect(result.last(2)).to match_array([ 33 | ">insert", 34 | "> bye!", 35 | ]) 36 | end 37 | end -------------------------------------------------------------------------------- /src/.DS_Store: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/enpeizhao/duck_db/32303274554a1ebe5c9aa04ce45499ca052c49fa/src/.DS_Store -------------------------------------------------------------------------------- /src/bpt.cc: -------------------------------------------------------------------------------- 1 | #include "../include/bpt.h" 2 | 3 | #include 4 | 5 | #include 6 | #include 7 | using std::swap; 8 | using std::binary_search; 9 | using std::lower_bound; 10 | using std::upper_bound; 11 | 12 | namespace bpt { 13 | 14 | /* custom compare operator for STL algorithms */ 15 | OPERATOR_KEYCMP(index_t) 16 | OPERATOR_KEYCMP(record_t) 17 | 18 | /* helper iterating function */ 19 | template 20 | inline typename T::child_t begin(T &node) { 21 | return node.children; 22 | } 23 | template 24 | inline typename T::child_t end(T &node) { 25 | return node.children + node.n; 26 | } 27 | 28 | /* helper searching function */ 29 | inline index_t *find(internal_node_t &node, const key_t &key) { 30 | return upper_bound(begin(node), end(node) - 1, key); 31 | } 32 | inline record_t *find(leaf_node_t &node, const key_t &key) { 33 | return lower_bound(begin(node), end(node), key); 34 | } 35 | 36 | bplus_tree::bplus_tree(const char *p, bool force_empty) 37 | : fp(NULL), fp_level(0) 38 | { 39 | bzero(path, sizeof(path)); 40 | strcpy(path, p); 41 | 42 | if (!force_empty) 43 | // read tree from file 44 | if (map(&meta, OFFSET_META) != 0) 45 | force_empty = true; 46 | 47 | if (force_empty) { 48 | open_file("w+"); // truncate file 49 | 50 | // create empty tree if file doesn't exist 51 | init_from_empty(); 52 | close_file(); 53 | } 54 | } 55 | 56 | int bplus_tree::search(const key_t& key, value_t *value) const 57 | { 58 | leaf_node_t leaf; 59 | map(&leaf, search_leaf(key)); 60 | 61 | // finding the record 62 | record_t *record = find(leaf, key); 63 | if (record != leaf.children + leaf.n) { 64 | // always return the lower bound 65 | *value = record->value; 66 | 67 | return keycmp(record->key, key); 68 | } else { 69 | return -1; 70 | } 71 | } 72 | 73 | int bplus_tree::search_range(key_t *left, const key_t &right, 74 | value_t *values, size_t max, bool *next) const 75 | { 76 | if (left == NULL || keycmp(*left, right) > 0) 77 | return -1; 78 | 79 | off_t off_left = search_leaf(*left); 80 | off_t off_right = search_leaf(right); 81 | off_t off = off_left; 82 | size_t i = 0; 83 | record_t *b, *e; 84 | 85 | leaf_node_t leaf; 86 | while (off != off_right && off != 0 && i < max) { 87 | map(&leaf, off); 88 | 89 | // start point 90 | if (off_left == off) 91 | b = find(leaf, *left); 92 | else 93 | b = begin(leaf); 94 | 95 | // copy 96 | e = leaf.children + leaf.n; 97 | for (; b != e && i < max; ++b, ++i) 98 | values[i] = b->value; 99 | 100 | off = leaf.next; 101 | } 102 | 103 | // the last leaf 104 | if (i < max) { 105 | map(&leaf, off_right); 106 | 107 | b = find(leaf, *left); 108 | e = upper_bound(begin(leaf), end(leaf), right); 109 | for (; b != e && i < max; ++b, ++i) 110 | values[i] = b->value; 111 | } 112 | 113 | // mark for next iteration 114 | if (next != NULL) { 115 | if (i == max && b != e) { 116 | *next = true; 117 | *left = b->key; 118 | } else { 119 | *next = false; 120 | } 121 | } 122 | 123 | return i; 124 | } 125 | 126 | int bplus_tree::remove(const key_t& key) 127 | { 128 | internal_node_t parent; 129 | leaf_node_t leaf; 130 | 131 | // find parent node 132 | off_t parent_off = search_index(key); 133 | map(&parent, parent_off); 134 | 135 | // find current node 136 | index_t *where = find(parent, key); 137 | off_t offset = where->child; 138 | map(&leaf, offset); 139 | 140 | // verify 141 | if (!binary_search(begin(leaf), end(leaf), key)) 142 | return -1; 143 | 144 | size_t min_n = meta.leaf_node_num == 1 ? 0 : meta.order / 2; 145 | assert(leaf.n >= min_n && leaf.n <= meta.order); 146 | 147 | // delete the key 148 | record_t *to_delete = find(leaf, key); 149 | std::copy(to_delete + 1, end(leaf), to_delete); 150 | leaf.n--; 151 | 152 | // merge or borrow 153 | if (leaf.n < min_n) { 154 | // first borrow from left 155 | bool borrowed = false; 156 | if (leaf.prev != 0) 157 | borrowed = borrow_key(false, leaf); 158 | 159 | // then borrow from right 160 | if (!borrowed && leaf.next != 0) 161 | borrowed = borrow_key(true, leaf); 162 | 163 | // finally we merge 164 | if (!borrowed) { 165 | assert(leaf.next != 0 || leaf.prev != 0); 166 | 167 | key_t index_key; 168 | 169 | if (where == end(parent) - 1) { 170 | // if leaf is last element then merge | prev | leaf | 171 | assert(leaf.prev != 0); 172 | leaf_node_t prev; 173 | map(&prev, leaf.prev); 174 | index_key = begin(prev)->key; 175 | 176 | merge_leafs(&prev, &leaf); 177 | node_remove(&prev, &leaf); 178 | unmap(&prev, leaf.prev); 179 | } else { 180 | // else merge | leaf | next | 181 | assert(leaf.next != 0); 182 | leaf_node_t next; 183 | map(&next, leaf.next); 184 | index_key = begin(leaf)->key; 185 | 186 | merge_leafs(&leaf, &next); 187 | node_remove(&leaf, &next); 188 | unmap(&leaf, offset); 189 | } 190 | 191 | // remove parent's key 192 | remove_from_index(parent_off, parent, index_key); 193 | } else { 194 | unmap(&leaf, offset); 195 | } 196 | } else { 197 | unmap(&leaf, offset); 198 | } 199 | 200 | return 0; 201 | } 202 | 203 | int bplus_tree::insert(const key_t& key, value_t value) 204 | { 205 | off_t parent = search_index(key); 206 | off_t offset = search_leaf(parent, key); 207 | leaf_node_t leaf; 208 | map(&leaf, offset); 209 | 210 | // check if we have the same key 211 | if (binary_search(begin(leaf), end(leaf), key)) 212 | return 1; 213 | 214 | if (leaf.n == meta.order) { 215 | // split when full 216 | 217 | // new sibling leaf 218 | leaf_node_t new_leaf; 219 | node_create(offset, &leaf, &new_leaf); 220 | 221 | // find even split point 222 | size_t point = leaf.n / 2; 223 | bool place_right = keycmp(key, leaf.children[point].key) > 0; 224 | if (place_right) 225 | ++point; 226 | 227 | // split 228 | std::copy(leaf.children + point, leaf.children + leaf.n, 229 | new_leaf.children); 230 | new_leaf.n = leaf.n - point; 231 | leaf.n = point; 232 | 233 | // which part do we put the key 234 | if (place_right) 235 | insert_record_no_split(&new_leaf, key, value); 236 | else 237 | insert_record_no_split(&leaf, key, value); 238 | 239 | // save leafs 240 | unmap(&leaf, offset); 241 | unmap(&new_leaf, leaf.next); 242 | 243 | // insert new index key 244 | insert_key_to_index(parent, new_leaf.children[0].key, 245 | offset, leaf.next); 246 | } else { 247 | insert_record_no_split(&leaf, key, value); 248 | unmap(&leaf, offset); 249 | } 250 | 251 | return 0; 252 | } 253 | 254 | int bplus_tree::update(const key_t& key, value_t value) 255 | { 256 | off_t offset = search_leaf(key); 257 | leaf_node_t leaf; 258 | map(&leaf, offset); 259 | 260 | record_t *record = find(leaf, key); 261 | if (record != leaf.children + leaf.n) 262 | if (keycmp(key, record->key) == 0) { 263 | record->value = value; 264 | unmap(&leaf, offset); 265 | 266 | return 0; 267 | } else { 268 | return 1; 269 | } 270 | else 271 | return -1; 272 | } 273 | 274 | void bplus_tree::remove_from_index(off_t offset, internal_node_t &node, 275 | const key_t &key) 276 | { 277 | size_t min_n = meta.root_offset == offset ? 1 : meta.order / 2; 278 | assert(node.n >= min_n && node.n <= meta.order); 279 | 280 | // remove key 281 | key_t index_key = begin(node)->key; 282 | index_t *to_delete = find(node, key); 283 | if (to_delete != end(node)) { 284 | (to_delete + 1)->child = to_delete->child; 285 | std::copy(to_delete + 1, end(node), to_delete); 286 | } 287 | node.n--; 288 | 289 | // remove to only one key 290 | if (node.n == 1 && meta.root_offset == offset && 291 | meta.internal_node_num != 1) 292 | { 293 | unalloc(&node, meta.root_offset); 294 | meta.height--; 295 | meta.root_offset = node.children[0].child; 296 | unmap(&meta, OFFSET_META); 297 | return; 298 | } 299 | 300 | // merge or borrow 301 | if (node.n < min_n) { 302 | internal_node_t parent; 303 | map(&parent, node.parent); 304 | 305 | // first borrow from left 306 | bool borrowed = false; 307 | if (offset != begin(parent)->child) 308 | borrowed = borrow_key(false, node, offset); 309 | 310 | // then borrow from right 311 | if (!borrowed && offset != (end(parent) - 1)->child) 312 | borrowed = borrow_key(true, node, offset); 313 | 314 | // finally we merge 315 | if (!borrowed) { 316 | assert(node.next != 0 || node.prev != 0); 317 | 318 | if (offset == (end(parent) - 1)->child) { 319 | // if leaf is last element then merge | prev | leaf | 320 | assert(node.prev != 0); 321 | internal_node_t prev; 322 | map(&prev, node.prev); 323 | 324 | // merge 325 | index_t *where = find(parent, begin(prev)->key); 326 | reset_index_children_parent(begin(node), end(node), node.prev); 327 | merge_keys(where, prev, node); 328 | unmap(&prev, node.prev); 329 | } else { 330 | // else merge | leaf | next | 331 | assert(node.next != 0); 332 | internal_node_t next; 333 | map(&next, node.next); 334 | 335 | // merge 336 | index_t *where = find(parent, index_key); 337 | reset_index_children_parent(begin(next), end(next), offset); 338 | merge_keys(where, node, next); 339 | unmap(&node, offset); 340 | } 341 | 342 | // remove parent's key 343 | remove_from_index(node.parent, parent, index_key); 344 | } else { 345 | unmap(&node, offset); 346 | } 347 | } else { 348 | unmap(&node, offset); 349 | } 350 | } 351 | 352 | bool bplus_tree::borrow_key(bool from_right, internal_node_t &borrower, 353 | off_t offset) 354 | { 355 | typedef typename internal_node_t::child_t child_t; 356 | 357 | off_t lender_off = from_right ? borrower.next : borrower.prev; 358 | internal_node_t lender; 359 | map(&lender, lender_off); 360 | 361 | assert(lender.n >= meta.order / 2); 362 | if (lender.n != meta.order / 2) { 363 | child_t where_to_lend, where_to_put; 364 | 365 | internal_node_t parent; 366 | 367 | // swap keys, draw on paper to see why 368 | if (from_right) { 369 | where_to_lend = begin(lender); 370 | where_to_put = end(borrower); 371 | 372 | map(&parent, borrower.parent); 373 | child_t where = lower_bound(begin(parent), end(parent) - 1, 374 | (end(borrower) -1)->key); 375 | where->key = where_to_lend->key; 376 | unmap(&parent, borrower.parent); 377 | } else { 378 | where_to_lend = end(lender) - 1; 379 | where_to_put = begin(borrower); 380 | 381 | map(&parent, lender.parent); 382 | child_t where = find(parent, begin(lender)->key); 383 | where_to_put->key = where->key; 384 | where->key = (where_to_lend - 1)->key; 385 | unmap(&parent, lender.parent); 386 | } 387 | 388 | // store 389 | std::copy_backward(where_to_put, end(borrower), end(borrower) + 1); 390 | *where_to_put = *where_to_lend; 391 | borrower.n++; 392 | 393 | // erase 394 | reset_index_children_parent(where_to_lend, where_to_lend + 1, offset); 395 | std::copy(where_to_lend + 1, end(lender), where_to_lend); 396 | lender.n--; 397 | unmap(&lender, lender_off); 398 | return true; 399 | } 400 | 401 | return false; 402 | } 403 | 404 | bool bplus_tree::borrow_key(bool from_right, leaf_node_t &borrower) 405 | { 406 | off_t lender_off = from_right ? borrower.next : borrower.prev; 407 | leaf_node_t lender; 408 | map(&lender, lender_off); 409 | 410 | assert(lender.n >= meta.order / 2); 411 | if (lender.n != meta.order / 2) { 412 | typename leaf_node_t::child_t where_to_lend, where_to_put; 413 | 414 | // decide offset and update parent's index key 415 | if (from_right) { 416 | where_to_lend = begin(lender); 417 | where_to_put = end(borrower); 418 | change_parent_child(borrower.parent, begin(borrower)->key, 419 | lender.children[1].key); 420 | } else { 421 | where_to_lend = end(lender) - 1; 422 | where_to_put = begin(borrower); 423 | change_parent_child(lender.parent, begin(lender)->key, 424 | where_to_lend->key); 425 | } 426 | 427 | // store 428 | std::copy_backward(where_to_put, end(borrower), end(borrower) + 1); 429 | *where_to_put = *where_to_lend; 430 | borrower.n++; 431 | 432 | // erase 433 | std::copy(where_to_lend + 1, end(lender), where_to_lend); 434 | lender.n--; 435 | unmap(&lender, lender_off); 436 | return true; 437 | } 438 | 439 | return false; 440 | } 441 | 442 | void bplus_tree::change_parent_child(off_t parent, const key_t &o, 443 | const key_t &n) 444 | { 445 | internal_node_t node; 446 | map(&node, parent); 447 | 448 | index_t *w = find(node, o); 449 | assert(w != node.children + node.n); 450 | 451 | w->key = n; 452 | unmap(&node, parent); 453 | if (w == node.children + node.n - 1) { 454 | change_parent_child(node.parent, o, n); 455 | } 456 | } 457 | 458 | void bplus_tree::merge_leafs(leaf_node_t *left, leaf_node_t *right) 459 | { 460 | std::copy(begin(*right), end(*right), end(*left)); 461 | left->n += right->n; 462 | } 463 | 464 | void bplus_tree::merge_keys(index_t *where, 465 | internal_node_t &node, internal_node_t &next) 466 | { 467 | //(end(node) - 1)->key = where->key; 468 | //where->key = (end(next) - 1)->key; 469 | std::copy(begin(next), end(next), end(node)); 470 | node.n += next.n; 471 | node_remove(&node, &next); 472 | } 473 | 474 | void bplus_tree::insert_record_no_split(leaf_node_t *leaf, 475 | const key_t &key, const value_t &value) 476 | { 477 | record_t *where = upper_bound(begin(*leaf), end(*leaf), key); 478 | std::copy_backward(where, end(*leaf), end(*leaf) + 1); 479 | 480 | where->key = key; 481 | where->value = value; 482 | leaf->n++; 483 | } 484 | 485 | void bplus_tree::insert_key_to_index(off_t offset, const key_t &key, 486 | off_t old, off_t after) 487 | { 488 | if (offset == 0) { 489 | // create new root node 490 | internal_node_t root; 491 | root.next = root.prev = root.parent = 0; 492 | meta.root_offset = alloc(&root); 493 | meta.height++; 494 | 495 | // insert `old` and `after` 496 | root.n = 2; 497 | root.children[0].key = key; 498 | root.children[0].child = old; 499 | root.children[1].child = after; 500 | 501 | unmap(&meta, OFFSET_META); 502 | unmap(&root, meta.root_offset); 503 | 504 | // update children's parent 505 | reset_index_children_parent(begin(root), end(root), 506 | meta.root_offset); 507 | return; 508 | } 509 | 510 | internal_node_t node; 511 | map(&node, offset); 512 | assert(node.n <= meta.order); 513 | 514 | if (node.n == meta.order) { 515 | // split when full 516 | 517 | internal_node_t new_node; 518 | node_create(offset, &node, &new_node); 519 | 520 | // find even split point 521 | size_t point = (node.n - 1) / 2; 522 | bool place_right = keycmp(key, node.children[point].key) > 0; 523 | if (place_right) 524 | ++point; 525 | 526 | // prevent the `key` being the right `middle_key` 527 | // example: insert 48 into |42|45| 6| | 528 | if (place_right && keycmp(key, node.children[point].key) < 0) 529 | point--; 530 | 531 | key_t middle_key = node.children[point].key; 532 | 533 | // split 534 | std::copy(begin(node) + point + 1, end(node), begin(new_node)); 535 | new_node.n = node.n - point - 1; 536 | node.n = point + 1; 537 | 538 | // put the new key 539 | if (place_right) 540 | insert_key_to_index_no_split(new_node, key, after); 541 | else 542 | insert_key_to_index_no_split(node, key, after); 543 | 544 | unmap(&node, offset); 545 | unmap(&new_node, node.next); 546 | 547 | // update children's parent 548 | reset_index_children_parent(begin(new_node), end(new_node), node.next); 549 | 550 | // give the middle key to the parent 551 | // note: middle key's child is reserved 552 | insert_key_to_index(node.parent, middle_key, offset, node.next); 553 | } else { 554 | insert_key_to_index_no_split(node, key, after); 555 | unmap(&node, offset); 556 | } 557 | } 558 | 559 | void bplus_tree::insert_key_to_index_no_split(internal_node_t &node, 560 | const key_t &key, off_t value) 561 | { 562 | index_t *where = upper_bound(begin(node), end(node) - 1, key); 563 | 564 | // move later index forward 565 | std::copy_backward(where, end(node), end(node) + 1); 566 | 567 | // insert this key 568 | where->key = key; 569 | where->child = (where + 1)->child; 570 | (where + 1)->child = value; 571 | 572 | node.n++; 573 | } 574 | 575 | void bplus_tree::reset_index_children_parent(index_t *begin, index_t *end, 576 | off_t parent) 577 | { 578 | // this function can change both internal_node_t and leaf_node_t's parent 579 | // field, but we should ensure that: 580 | // 1. sizeof(internal_node_t) <= sizeof(leaf_node_t) 581 | // 2. parent field is placed in the beginning and have same size 582 | internal_node_t node; 583 | while (begin != end) { 584 | map(&node, begin->child); 585 | node.parent = parent; 586 | unmap(&node, begin->child, SIZE_NO_CHILDREN); 587 | ++begin; 588 | } 589 | } 590 | 591 | off_t bplus_tree::search_index(const key_t &key) const 592 | { 593 | off_t org = meta.root_offset; 594 | int height = meta.height; 595 | while (height > 1) { 596 | internal_node_t node; 597 | map(&node, org); 598 | 599 | index_t *i = upper_bound(begin(node), end(node) - 1, key); 600 | org = i->child; 601 | --height; 602 | } 603 | 604 | return org; 605 | } 606 | 607 | off_t bplus_tree::search_leaf(off_t index, const key_t &key) const 608 | { 609 | internal_node_t node; 610 | map(&node, index); 611 | 612 | index_t *i = upper_bound(begin(node), end(node) - 1, key); 613 | return i->child; 614 | } 615 | 616 | template 617 | void bplus_tree::node_create(off_t offset, T *node, T *next) 618 | { 619 | // new sibling node 620 | next->parent = node->parent; 621 | next->next = node->next; 622 | next->prev = offset; 623 | node->next = alloc(next); 624 | // update next node's prev 625 | if (next->next != 0) { 626 | T old_next; 627 | map(&old_next, next->next, SIZE_NO_CHILDREN); 628 | old_next.prev = node->next; 629 | unmap(&old_next, next->next, SIZE_NO_CHILDREN); 630 | } 631 | unmap(&meta, OFFSET_META); 632 | } 633 | 634 | template 635 | void bplus_tree::node_remove(T *prev, T *node) 636 | { 637 | unalloc(node, prev->next); 638 | prev->next = node->next; 639 | if (node->next != 0) { 640 | T next; 641 | map(&next, node->next, SIZE_NO_CHILDREN); 642 | next.prev = node->prev; 643 | unmap(&next, node->next, SIZE_NO_CHILDREN); 644 | } 645 | unmap(&meta, OFFSET_META); 646 | } 647 | 648 | void bplus_tree::init_from_empty() 649 | { 650 | // init default meta 651 | bzero(&meta, sizeof(meta_t)); 652 | meta.order = BP_ORDER; 653 | meta.value_size = sizeof(value_t); 654 | meta.key_size = sizeof(key_t); 655 | meta.height = 1; 656 | meta.slot = OFFSET_BLOCK; 657 | 658 | // init root node 659 | internal_node_t root; 660 | root.next = root.prev = root.parent = 0; 661 | meta.root_offset = alloc(&root); 662 | 663 | // init empty leaf 664 | leaf_node_t leaf; 665 | leaf.next = leaf.prev = 0; 666 | leaf.parent = meta.root_offset; 667 | meta.leaf_offset = root.children[0].child = alloc(&leaf); 668 | 669 | // save 670 | unmap(&meta, OFFSET_META); 671 | unmap(&root, meta.root_offset); 672 | unmap(&leaf, root.children[0].child); 673 | } 674 | 675 | } 676 | -------------------------------------------------------------------------------- /src/duck_db.cpp: -------------------------------------------------------------------------------- 1 | // a journey of a thousand miles begins with a single step 2 | // author: enpeizhao 3 | // blog: www.enpeizhao.com 4 | 5 | #include "../include/bpt.h" 6 | #include "../include/TextTable.h" 7 | 8 | #include 9 | #include 10 | #include 11 | #include 12 | #include 13 | 14 | using namespace bpt; 15 | using namespace std; 16 | 17 | 18 | const char *errorMessage = "> your input is invalid,print \".help\" for more infomation!\n"; 19 | const char *nextLineHeader ="> "; 20 | const char *exitMessage = "> bye!\n"; 21 | const char *dbFileName = "./data/db.bin"; 22 | 23 | 24 | clock_t startTime,finishTime; 25 | 26 | // function prototype 27 | void printHelpMess(); 28 | void selectCommand(); 29 | int insertRecord(bplus_tree *treePtr,int *key, value_t *values); 30 | int deleteRecord(bplus_tree *treePtr,int *index); 31 | int searchRecord(bplus_tree *treePtr,int *index, value_t *return_val); 32 | int searchAll(bplus_tree *treePtr,int *i_start, int *i_end); 33 | int updateRecord(bplus_tree *treePtr,int *key, value_t *values); 34 | void printTable(int *index, value_t *values); 35 | void intToKeyT(bpt::key_t *a,int *b); 36 | bool is_file_exist(const char *fileName); 37 | double durationTime(clock_t *f,clock_t *s); 38 | 39 | 40 | bplus_tree *duck_db_ptr; 41 | 42 | // initial 43 | void initialSystem(){ 44 | // step 1 : print help message 45 | printHelpMess(); 46 | // step 2 : initial database from file 47 | bplus_tree duck_db(dbFileName, (!is_file_exist(dbFileName))); 48 | duck_db_ptr = &duck_db; 49 | // step 3 : REPL select commdand (insert,delete,update,search) 50 | selectCommand(); 51 | 52 | } 53 | // print help message 54 | void printHelpMess(){ 55 | cout << "*********************************************************************************************"<name,&(insertData->age),insertData->email); 104 | 105 | if(okNum < 3){ 106 | 107 | cout << errorMessage<< nextLineHeader; 108 | 109 | }else{ 110 | 111 | startTime = clock(); 112 | 113 | int return_code = insertRecord(duck_db_ptr,keyIndex,insertData); 114 | 115 | finishTime = clock(); 116 | 117 | if (return_code == 0){ 118 | // cout << ">insert\n"; 119 | cout << "> executed insert index:"<< *keyIndex << ", time : "<< durationTime(&finishTime,&startTime) <<" seconds\n"<< nextLineHeader; 120 | }else if(return_code == 1){ 121 | cout << "> failed: already exist index:"<< *keyIndex << "\n"<< nextLineHeader; 122 | }else{ 123 | cout << "> failed!\n"<< nextLineHeader; 124 | } 125 | } 126 | 127 | 128 | }else if(strncmp(userCommand,"delete",6) == 0){ 129 | 130 | int *keyIndex = new int; 131 | 132 | int okNum = sscanf(userCommand,"delete from db where id=%d;", keyIndex); 133 | 134 | if(okNum < 1){ 135 | cout << errorMessage<< nextLineHeader; 136 | }else{ 137 | startTime = clock(); 138 | 139 | int return_code = deleteRecord(duck_db_ptr,keyIndex); 140 | 141 | finishTime = clock(); 142 | 143 | if (return_code == 0){ 144 | cout << "> executed delete index:"<< *keyIndex << ", time : "<< durationTime(&finishTime,&startTime) <<" seconds\n"<< nextLineHeader; 145 | }else if(return_code == -1){ 146 | cout << "> failed ! no index:"<< *keyIndex << "\n"<< nextLineHeader; 147 | }else{ 148 | cout << "> failed!\n"<< nextLineHeader; 149 | } 150 | } 151 | 152 | 153 | }else if(strncmp(userCommand,"select",6) == 0){ 154 | 155 | if( ! strstr (userCommand,"=")){ 156 | 157 | int i_start,i_end; 158 | 159 | int okNum = sscanf(userCommand,"select * from db where id in(%d,%d);", &i_start,&i_end); 160 | 161 | if(okNum < 2){ 162 | cout << errorMessage<< nextLineHeader; 163 | }else{ 164 | startTime = clock(); 165 | 166 | searchAll(duck_db_ptr,&i_start, &i_end); 167 | 168 | finishTime = clock(); 169 | cout << "> executed search, time : "<< durationTime(&finishTime,&startTime) <<" seconds\n"<< nextLineHeader; 170 | } 171 | 172 | 173 | }else{ 174 | 175 | int *keyIndex = new int; 176 | int okNum = sscanf(userCommand,"select * from db where id=%d;", keyIndex); 177 | 178 | if(okNum < 1){ 179 | cout << errorMessage<< nextLineHeader; 180 | }else{ 181 | 182 | value_t *return_val = new value_t; 183 | startTime = clock(); 184 | 185 | int return_code = searchRecord(duck_db_ptr,keyIndex,return_val); 186 | 187 | finishTime = clock(); 188 | 189 | if (return_code != 0){ 190 | cout << "> index:"<< *keyIndex << " doesn't exist, time : "<< durationTime(&finishTime,&startTime) <<" seconds\n"<< nextLineHeader; 191 | }else{ 192 | printTable( keyIndex , return_val); 193 | cout << "> executed search, time : "<< durationTime(&finishTime,&startTime) <<" seconds\n"<< nextLineHeader; 194 | 195 | } 196 | } 197 | } 198 | 199 | 200 | }else if(strncmp(userCommand,"update",6) == 0){ 201 | 202 | int *keyIndex = new int; 203 | value_t *updateData = new value_t; 204 | 205 | int okNum = sscanf(userCommand,"update db %s %d %s where id=%d;", 206 | updateData->name,&(updateData->age),updateData->email,keyIndex); 207 | 208 | if(okNum < 3){ 209 | cout << errorMessage<< nextLineHeader; 210 | }else{ 211 | startTime = clock(); 212 | 213 | int return_code = updateRecord(duck_db_ptr,keyIndex,updateData); 214 | 215 | finishTime = clock(); 216 | 217 | if (return_code == 0){ 218 | cout << "> executed update index:"<< *keyIndex << ", time : "<< durationTime(&finishTime,&startTime) <<" seconds\n"<< nextLineHeader; 219 | }else{ 220 | cout << "> failed! no index:"<< *keyIndex << ", time : "<< durationTime(&finishTime,&startTime) <<" seconds\n"<< nextLineHeader; 221 | } 222 | } 223 | 224 | } 225 | else{ 226 | cout << errorMessage<< nextLineHeader; 227 | } 228 | } 229 | 230 | 231 | 232 | } 233 | 234 | // insert 235 | int insertRecord(bplus_tree *treePtr,int *index, value_t *values){ 236 | 237 | bpt::key_t key; 238 | intToKeyT(&key,index); 239 | return (*treePtr).insert(key, *values); 240 | } 241 | 242 | // delete 243 | int deleteRecord(bplus_tree *treePtr,int *index){ 244 | 245 | bpt::key_t key; 246 | intToKeyT(&key,index); 247 | 248 | return (*treePtr).remove(key); 249 | } 250 | 251 | // update 252 | int updateRecord(bplus_tree *treePtr,int *index, value_t *values){ 253 | bpt::key_t key; 254 | intToKeyT(&key,index); 255 | return (*treePtr).update(key, *values); 256 | } 257 | 258 | // search by index 259 | int searchRecord(bplus_tree *treePtr,int *index, value_t *return_val){ 260 | bpt::key_t key; 261 | intToKeyT(&key,index); 262 | return (*treePtr).search(key, return_val); 263 | } 264 | // search all 265 | int searchAll(bplus_tree *treePtr,int *start, int *end){ 266 | 267 | TextTable t( '-', '|', '+' ); 268 | 269 | t.add( " id " ); 270 | t.add( " name " ); 271 | t.add( " age " ); 272 | t.add( " email " ); 273 | t.endOfRow(); 274 | 275 | bpt::key_t key; 276 | value_t *return_val = new value_t; 277 | 278 | for (int i = *start; i <= *end; ++i) 279 | { 280 | 281 | intToKeyT(&key,&i); 282 | 283 | int return_code = (*treePtr).search(key, return_val); 284 | switch(return_code){ 285 | case -1: 286 | // no exist 287 | break; 288 | case 0: 289 | // find 290 | t.add( to_string(i) ); 291 | t.add( return_val ->name ); 292 | t.add( to_string(return_val ->age)); 293 | t.add( return_val ->email ); 294 | t.endOfRow(); 295 | break; 296 | case 1: 297 | // deleted 298 | break; 299 | } 300 | 301 | 302 | } 303 | cout << t << endl; 304 | 305 | } 306 | // print table 307 | void printTable(int *index, value_t *values){ 308 | 309 | 310 | TextTable t( '-', '|', '+' ); 311 | 312 | t.add( " id " ); 313 | t.add( " name " ); 314 | t.add( " age " ); 315 | t.add( " email " ); 316 | t.endOfRow(); 317 | 318 | t.add( to_string(*index) ); 319 | t.add( values ->name ); 320 | t.add( to_string(values ->age)); 321 | t.add( values ->email ); 322 | t.endOfRow(); 323 | 324 | cout << t << endl; 325 | } 326 | // int to key_t 327 | void intToKeyT(bpt::key_t *a,int *b){ 328 | char key[16] = { 0 }; 329 | sprintf(key, "%d", *b); 330 | *a = key; 331 | } 332 | 333 | bool is_file_exist(const char *fileName) 334 | { 335 | ifstream ifile(fileName); 336 | return ifile.good(); 337 | } 338 | 339 | double durationTime(clock_t *f,clock_t *s){ 340 | return (double)(*f - *s) / CLOCKS_PER_SEC; 341 | } 342 | 343 | 344 | int main(int argc, char *argv[]) 345 | { 346 | initialSystem(); 347 | 348 | } 349 | --------------------------------------------------------------------------------