{{ site.description | default: site.github.project_tagline }}
20 | 21 | 22 | 23 | {% if site.github.is_project_page %} 24 |View on GitHub (pull requests welcome)
25 | {% endif %} 26 | 27 | {% if site.github.is_user_page %} 28 | 29 | {% endif %} 30 | 31 | {% if site.show_downloads %} 32 |This project is no longer under active development. You can read more here. But if you'd like to keep learning how to make your own SQLite clone from scratch, or one of many other projects like Docker, Redis, Git or BitTorrent, try CodeCrafters.
41 |
44 |
15 |
16 | If you use my referral link, I get a commision.
--------------------------------------------------------------------------------
/_parts/part2.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: Part 2 - World's Simplest SQL Compiler and Virtual Machine
3 | date: 2017-08-31
4 | ---
5 |
6 | We're making a clone of sqlite. The "front-end" of sqlite is a SQL compiler that parses a string and outputs an internal representation called bytecode.
7 |
8 | This bytecode is passed to the virtual machine, which executes it.
9 |
10 | {% include image.html url="assets/images/arch2.gif" description="SQLite Architecture (https://www.sqlite.org/arch.html)" %}
11 |
12 | Breaking things into two steps like this has a couple advantages:
13 | - Reduces the complexity of each part (e.g. virtual machine does not worry about syntax errors)
14 | - Allows compiling common queries once and caching the bytecode for improved performance
15 |
16 | With this in mind, let's refactor our `main` function and support two new keywords in the process:
17 |
18 | ```diff
19 | int main(int argc, char* argv[]) {
20 | InputBuffer* input_buffer = new_input_buffer();
21 | while (true) {
22 | print_prompt();
23 | read_input(input_buffer);
24 |
25 | - if (strcmp(input_buffer->buffer, ".exit") == 0) {
26 | - exit(EXIT_SUCCESS);
27 | - } else {
28 | - printf("Unrecognized command '%s'.\n", input_buffer->buffer);
29 | + if (input_buffer->buffer[0] == '.') {
30 | + switch (do_meta_command(input_buffer)) {
31 | + case (META_COMMAND_SUCCESS):
32 | + continue;
33 | + case (META_COMMAND_UNRECOGNIZED_COMMAND):
34 | + printf("Unrecognized command '%s'\n", input_buffer->buffer);
35 | + continue;
36 | + }
37 | }
38 | +
39 | + Statement statement;
40 | + switch (prepare_statement(input_buffer, &statement)) {
41 | + case (PREPARE_SUCCESS):
42 | + break;
43 | + case (PREPARE_UNRECOGNIZED_STATEMENT):
44 | + printf("Unrecognized keyword at start of '%s'.\n",
45 | + input_buffer->buffer);
46 | + continue;
47 | + }
48 | +
49 | + execute_statement(&statement);
50 | + printf("Executed.\n");
51 | }
52 | }
53 | ```
54 |
55 | Non-SQL statements like `.exit` are called "meta-commands". They all start with a dot, so we check for them and handle them in a separate function.
56 |
57 | Next, we add a step that converts the line of input into our internal representation of a statement. This is our hacky version of the sqlite front-end.
58 |
59 | Lastly, we pass the prepared statement to `execute_statement`. This function will eventually become our virtual machine.
60 |
61 | Notice that two of our new functions return enums indicating success or failure:
62 |
63 | ```c
64 | typedef enum {
65 | META_COMMAND_SUCCESS,
66 | META_COMMAND_UNRECOGNIZED_COMMAND
67 | } MetaCommandResult;
68 |
69 | typedef enum { PREPARE_SUCCESS, PREPARE_UNRECOGNIZED_STATEMENT } PrepareResult;
70 | ```
71 |
72 | "Unrecognized statement"? That seems a bit like an exception. I prefer not to use exceptions (and C doesn't even support them), so I'm using enum result codes wherever practical. The C compiler will complain if my switch statement doesn't handle a member of the enum, so we can feel a little more confident we handle every result of a function. Expect more result codes to be added in the future.
73 |
74 | `do_meta_command` is just a wrapper for existing functionality that leaves room for more commands:
75 |
76 | ```c
77 | MetaCommandResult do_meta_command(InputBuffer* input_buffer) {
78 | if (strcmp(input_buffer->buffer, ".exit") == 0) {
79 | exit(EXIT_SUCCESS);
80 | } else {
81 | return META_COMMAND_UNRECOGNIZED_COMMAND;
82 | }
83 | }
84 | ```
85 |
86 | Our "prepared statement" right now just contains an enum with two possible values. It will contain more data as we allow parameters in statements:
87 |
88 | ```c
89 | typedef enum { STATEMENT_INSERT, STATEMENT_SELECT } StatementType;
90 |
91 | typedef struct {
92 | StatementType type;
93 | } Statement;
94 | ```
95 |
96 | `prepare_statement` (our "SQL Compiler") does not understand SQL right now. In fact, it only understands two words:
97 | ```c
98 | PrepareResult prepare_statement(InputBuffer* input_buffer,
99 | Statement* statement) {
100 | if (strncmp(input_buffer->buffer, "insert", 6) == 0) {
101 | statement->type = STATEMENT_INSERT;
102 | return PREPARE_SUCCESS;
103 | }
104 | if (strcmp(input_buffer->buffer, "select") == 0) {
105 | statement->type = STATEMENT_SELECT;
106 | return PREPARE_SUCCESS;
107 | }
108 |
109 | return PREPARE_UNRECOGNIZED_STATEMENT;
110 | }
111 | ```
112 |
113 | Note that we use `strncmp` for "insert" since the "insert" keyword will be followed by data. (e.g. `insert 1 cstack foo@bar.com`)
114 |
115 | Lastly, `execute_statement` contains a few stubs:
116 | ```c
117 | void execute_statement(Statement* statement) {
118 | switch (statement->type) {
119 | case (STATEMENT_INSERT):
120 | printf("This is where we would do an insert.\n");
121 | break;
122 | case (STATEMENT_SELECT):
123 | printf("This is where we would do a select.\n");
124 | break;
125 | }
126 | }
127 | ```
128 |
129 | Note that it doesn't return any error codes because there's nothing that could go wrong yet.
130 |
131 | With these refactors, we now recognize two new keywords!
132 | ```command-line
133 | ~ ./db
134 | db > insert foo bar
135 | This is where we would do an insert.
136 | Executed.
137 | db > delete foo
138 | Unrecognized keyword at start of 'delete foo'.
139 | db > select
140 | This is where we would do a select.
141 | Executed.
142 | db > .tables
143 | Unrecognized command '.tables'
144 | db > .exit
145 | ~
146 | ```
147 |
148 | The skeleton of our database is taking shape... wouldn't it be nice if it stored data? In the next part, we'll implement `insert` and `select`, creating the world's worst data store. In the mean time, here's the entire diff from this part:
149 |
150 | ```diff
151 | @@ -10,6 +10,23 @@ struct InputBuffer_t {
152 | } InputBuffer;
153 |
154 | +typedef enum {
155 | + META_COMMAND_SUCCESS,
156 | + META_COMMAND_UNRECOGNIZED_COMMAND
157 | +} MetaCommandResult;
158 | +
159 | +typedef enum { PREPARE_SUCCESS, PREPARE_UNRECOGNIZED_STATEMENT } PrepareResult;
160 | +
161 | +typedef enum { STATEMENT_INSERT, STATEMENT_SELECT } StatementType;
162 | +
163 | +typedef struct {
164 | + StatementType type;
165 | +} Statement;
166 | +
167 | InputBuffer* new_input_buffer() {
168 | InputBuffer* input_buffer = malloc(sizeof(InputBuffer));
169 | input_buffer->buffer = NULL;
170 | @@ -40,17 +57,67 @@ void close_input_buffer(InputBuffer* input_buffer) {
171 | free(input_buffer);
172 | }
173 |
174 | +MetaCommandResult do_meta_command(InputBuffer* input_buffer) {
175 | + if (strcmp(input_buffer->buffer, ".exit") == 0) {
176 | + close_input_buffer(input_buffer);
177 | + exit(EXIT_SUCCESS);
178 | + } else {
179 | + return META_COMMAND_UNRECOGNIZED_COMMAND;
180 | + }
181 | +}
182 | +
183 | +PrepareResult prepare_statement(InputBuffer* input_buffer,
184 | + Statement* statement) {
185 | + if (strncmp(input_buffer->buffer, "insert", 6) == 0) {
186 | + statement->type = STATEMENT_INSERT;
187 | + return PREPARE_SUCCESS;
188 | + }
189 | + if (strcmp(input_buffer->buffer, "select") == 0) {
190 | + statement->type = STATEMENT_SELECT;
191 | + return PREPARE_SUCCESS;
192 | + }
193 | +
194 | + return PREPARE_UNRECOGNIZED_STATEMENT;
195 | +}
196 | +
197 | +void execute_statement(Statement* statement) {
198 | + switch (statement->type) {
199 | + case (STATEMENT_INSERT):
200 | + printf("This is where we would do an insert.\n");
201 | + break;
202 | + case (STATEMENT_SELECT):
203 | + printf("This is where we would do a select.\n");
204 | + break;
205 | + }
206 | +}
207 | +
208 | int main(int argc, char* argv[]) {
209 | InputBuffer* input_buffer = new_input_buffer();
210 | while (true) {
211 | print_prompt();
212 | read_input(input_buffer);
213 |
214 | - if (strcmp(input_buffer->buffer, ".exit") == 0) {
215 | - close_input_buffer(input_buffer);
216 | - exit(EXIT_SUCCESS);
217 | - } else {
218 | - printf("Unrecognized command '%s'.\n", input_buffer->buffer);
219 | + if (input_buffer->buffer[0] == '.') {
220 | + switch (do_meta_command(input_buffer)) {
221 | + case (META_COMMAND_SUCCESS):
222 | + continue;
223 | + case (META_COMMAND_UNRECOGNIZED_COMMAND):
224 | + printf("Unrecognized command '%s'\n", input_buffer->buffer);
225 | + continue;
226 | + }
227 | }
228 | +
229 | + Statement statement;
230 | + switch (prepare_statement(input_buffer, &statement)) {
231 | + case (PREPARE_SUCCESS):
232 | + break;
233 | + case (PREPARE_UNRECOGNIZED_STATEMENT):
234 | + printf("Unrecognized keyword at start of '%s'.\n",
235 | + input_buffer->buffer);
236 | + continue;
237 | + }
238 | +
239 | + execute_statement(&statement);
240 | + printf("Executed.\n");
241 | }
242 | }
243 | ```
244 |
--------------------------------------------------------------------------------
/_parts/part3.md:
--------------------------------------------------------------------------------
1 | ---
2 | title: Part 3 - An In-Memory, Append-Only, Single-Table Database
3 | date: 2017-09-01
4 | ---
5 |
6 | We're going to start small by putting a lot of limitations on our database. For now, it will:
7 |
8 | - support two operations: inserting a row and printing all rows
9 | - reside only in memory (no persistence to disk)
10 | - support a single, hard-coded table
11 |
12 | Our hard-coded table is going to store users and look like this:
13 |
14 | | column | type |
15 | |----------|--------------|
16 | | id | integer |
17 | | username | varchar(32) |
18 | | email | varchar(255) |
19 |
20 | This is a simple schema, but it gets us to support multiple data types and multiple sizes of text data types.
21 |
22 | `insert` statements are now going to look like this:
23 |
24 | ```
25 | insert 1 cstack foo@bar.com
26 | ```
27 |
28 | That means we need to upgrade our `prepare_statement` function to parse arguments
29 |
30 | ```diff
31 | if (strncmp(input_buffer->buffer, "insert", 6) == 0) {
32 | statement->type = STATEMENT_INSERT;
33 | + int args_assigned = sscanf(
34 | + input_buffer->buffer, "insert %d %s %s", &(statement->row_to_insert.id),
35 | + statement->row_to_insert.username, statement->row_to_insert.email);
36 | + if (args_assigned < 3) {
37 | + return PREPARE_SYNTAX_ERROR;
38 | + }
39 | return PREPARE_SUCCESS;
40 | }
41 | if (strcmp(input_buffer->buffer, "select") == 0) {
42 | ```
43 |
44 | We store those parsed arguments into a new `Row` data structure inside the statement object:
45 |
46 | ```diff
47 | +#define COLUMN_USERNAME_SIZE 32
48 | +#define COLUMN_EMAIL_SIZE 255
49 | +typedef struct {
50 | + uint32_t id;
51 | + char username[COLUMN_USERNAME_SIZE];
52 | + char email[COLUMN_EMAIL_SIZE];
53 | +} Row;
54 | +
55 | typedef struct {
56 | StatementType type;
57 | + Row row_to_insert; // only used by insert statement
58 | } Statement;
59 | ```
60 |
61 | Now we need to copy that data into some data structure representing the table. SQLite uses a B-tree for fast lookups, inserts and deletes. We'll start with something simpler. Like a B-tree, it will group rows into pages, but instead of arranging those pages as a tree it will arrange them as an array.
62 |
63 | Here's my plan:
64 |
65 | - Store rows in blocks of memory called pages
66 | - Each page stores as many rows as it can fit
67 | - Rows are serialized into a compact representation with each page
68 | - Pages are only allocated as needed
69 | - Keep a fixed-size array of pointers to pages
70 |
71 | First we'll define the compact representation of a row:
72 | ```diff
73 | +#define size_of_attribute(Struct, Attribute) sizeof(((Struct*)0)->Attribute)
74 | +
75 | +const uint32_t ID_SIZE = size_of_attribute(Row, id);
76 | +const uint32_t USERNAME_SIZE = size_of_attribute(Row, username);
77 | +const uint32_t EMAIL_SIZE = size_of_attribute(Row, email);
78 | +const uint32_t ID_OFFSET = 0;
79 | +const uint32_t USERNAME_OFFSET = ID_OFFSET + ID_SIZE;
80 | +const uint32_t EMAIL_OFFSET = USERNAME_OFFSET + USERNAME_SIZE;
81 | +const uint32_t ROW_SIZE = ID_SIZE + USERNAME_SIZE + EMAIL_SIZE;
82 | ```
83 |
84 | This means the layout of a serialized row will look like this:
85 |
86 | | column | size (bytes) | offset |
87 | |----------|--------------|--------------|
88 | | id | 4 | 0 |
89 | | username | 32 | 4 |
90 | | email | 255 | 36 |
91 | | total | 291 | |
92 |
93 | We also need code to convert to and from the compact representation.
94 | ```diff
95 | +void serialize_row(Row* source, void* destination) {
96 | + memcpy(destination + ID_OFFSET, &(source->id), ID_SIZE);
97 | + memcpy(destination + USERNAME_OFFSET, &(source->username), USERNAME_SIZE);
98 | + memcpy(destination + EMAIL_OFFSET, &(source->email), EMAIL_SIZE);
99 | +}
100 | +
101 | +void deserialize_row(void* source, Row* destination) {
102 | + memcpy(&(destination->id), source + ID_OFFSET, ID_SIZE);
103 | + memcpy(&(destination->username), source + USERNAME_OFFSET, USERNAME_SIZE);
104 | + memcpy(&(destination->email), source + EMAIL_OFFSET, EMAIL_SIZE);
105 | +}
106 | ```
107 |
108 | Next, a `Table` structure that points to pages of rows and keeps track of how many rows there are:
109 | ```diff
110 | +const uint32_t PAGE_SIZE = 4096;
111 | +#define TABLE_MAX_PAGES 100
112 | +const uint32_t ROWS_PER_PAGE = PAGE_SIZE / ROW_SIZE;
113 | +const uint32_t TABLE_MAX_ROWS = ROWS_PER_PAGE * TABLE_MAX_PAGES;
114 | +
115 | +typedef struct {
116 | + uint32_t num_rows;
117 | + void* pages[TABLE_MAX_PAGES];
118 | +} Table;
119 | ```
120 |
121 | I'm making our page size 4 kilobytes because it's the same size as a page used in the virtual memory systems of most computer architectures. This means one page in our database corresponds to one page used by the operating system. The operating system will move pages in and out of memory as whole units instead of breaking them up.
122 |
123 | I'm setting an arbitrary limit of 100 pages that we will allocate. When we switch to a tree structure, our database's maximum size will only be limited by the maximum size of a file. (Although we'll still limit how many pages we keep in memory at once)
124 |
125 | Rows should not cross page boundaries. Since pages probably won't exist next to each other in memory, this assumption makes it easier to read/write rows.
126 |
127 | Speaking of which, here is how we figure out where to read/write in memory for a particular row:
128 | ```diff
129 | +void* row_slot(Table* table, uint32_t row_num) {
130 | + uint32_t page_num = row_num / ROWS_PER_PAGE;
131 | + void* page = table->pages[page_num];
132 | + if (page == NULL) {
133 | + // Allocate memory only when we try to access page
134 | + page = table->pages[page_num] = malloc(PAGE_SIZE);
135 | + }
136 | + uint32_t row_offset = row_num % ROWS_PER_PAGE;
137 | + uint32_t byte_offset = row_offset * ROW_SIZE;
138 | + return page + byte_offset;
139 | +}
140 | ```
141 |
142 | Now we can make `execute_statement` read/write from our table structure:
143 | ```diff
144 | -void execute_statement(Statement* statement) {
145 | +ExecuteResult execute_insert(Statement* statement, Table* table) {
146 | + if (table->num_rows >= TABLE_MAX_ROWS) {
147 | + return EXECUTE_TABLE_FULL;
148 | + }
149 | +
150 | + Row* row_to_insert = &(statement->row_to_insert);
151 | +
152 | + serialize_row(row_to_insert, row_slot(table, table->num_rows));
153 | + table->num_rows += 1;
154 | +
155 | + return EXECUTE_SUCCESS;
156 | +}
157 | +
158 | +ExecuteResult execute_select(Statement* statement, Table* table) {
159 | + Row row;
160 | + for (uint32_t i = 0; i < table->num_rows; i++) {
161 | + deserialize_row(row_slot(table, i), &row);
162 | + print_row(&row);
163 | + }
164 | + return EXECUTE_SUCCESS;
165 | +}
166 | +
167 | +ExecuteResult execute_statement(Statement* statement, Table* table) {
168 | switch (statement->type) {
169 | case (STATEMENT_INSERT):
170 | - printf("This is where we would do an insert.\n");
171 | - break;
172 | + return execute_insert(statement, table);
173 | case (STATEMENT_SELECT):
174 | - printf("This is where we would do a select.\n");
175 | - break;
176 | + return execute_select(statement, table);
177 | }
178 | }
179 | ```
180 |
181 | Lastly, we need to initialize the table, create the respective
182 | memory release function and handle a few more error cases:
183 |
184 | ```diff
185 | + Table* new_table() {
186 | + Table* table = (Table*)malloc(sizeof(Table));
187 | + table->num_rows = 0;
188 | + for (uint32_t i = 0; i < TABLE_MAX_PAGES; i++) {
189 | + table->pages[i] = NULL;
190 | + }
191 | + return table;
192 | +}
193 | +
194 | +void free_table(Table* table) {
195 | + for (int i = 0; table->pages[i]; i++) {
196 | + free(table->pages[i]);
197 | + }
198 | + free(table);
199 | +}
200 | ```
201 | ```diff
202 | int main(int argc, char* argv[]) {
203 | + Table* table = new_table();
204 | InputBuffer* input_buffer = new_input_buffer();
205 | while (true) {
206 | print_prompt();
207 | @@ -105,13 +203,22 @@ int main(int argc, char* argv[]) {
208 | switch (prepare_statement(input_buffer, &statement)) {
209 | case (PREPARE_SUCCESS):
210 | break;
211 | + case (PREPARE_SYNTAX_ERROR):
212 | + printf("Syntax error. Could not parse statement.\n");
213 | + continue;
214 | case (PREPARE_UNRECOGNIZED_STATEMENT):
215 | printf("Unrecognized keyword at start of '%s'.\n",
216 | input_buffer->buffer);
217 | continue;
218 | }
219 |
220 | - execute_statement(&statement);
221 | - printf("Executed.\n");
222 | + switch (execute_statement(&statement, table)) {
223 | + case (EXECUTE_SUCCESS):
224 | + printf("Executed.\n");
225 | + break;
226 | + case (EXECUTE_TABLE_FULL):
227 | + printf("Error: Table full.\n");
228 | + break;
229 | + }
230 | }
231 | }
232 | ```
233 |
234 | With those changes we can actually save data in our database!
235 | ```command-line
236 | ~ ./db
237 | db > insert 1 cstack foo@bar.com
238 | Executed.
239 | db > insert 2 bob bob@example.com
240 | Executed.
241 | db > select
242 | (1, cstack, foo@bar.com)
243 | (2, bob, bob@example.com)
244 | Executed.
245 | db > insert foo bar 1
246 | Syntax error. Could not parse statement.
247 | db > .exit
248 | ~
249 | ```
250 |
251 | Now would be a great time to write some tests, for a couple reasons:
252 | - We're planning to dramatically change the data structure storing our table, and tests would catch regressions.
253 | - There are a couple edge cases we haven't tested manually (e.g. filling up the table)
254 |
255 | We'll address those issues in the next part. For now, here's the complete diff from this part:
256 | ```diff
257 | @@ -2,6 +2,7 @@
258 | #include