├── .github └── workflows │ └── build.yml ├── .gitignore ├── CONTRIBUTING.md ├── LICENSE ├── Makefile ├── README.md ├── disk_store.go ├── disk_store_test.go ├── format.go ├── format_test.go ├── go.mod ├── hints.md ├── memory_store.go ├── memory_store_test.go ├── store.go └── workshop.md /.github/workflows/build.yml: -------------------------------------------------------------------------------- 1 | name: build 2 | 3 | on: 4 | push: 5 | branches: 6 | - 'master' 7 | pull_request: {} 8 | 9 | jobs: 10 | tests: 11 | runs-on: ${{ matrix.os }} 12 | strategy: 13 | matrix: 14 | os: [ ubuntu-latest, macos-latest, windows-latest ] 15 | steps: 16 | - uses: actions/checkout@v3 17 | - uses: actions/setup-go@v3 18 | - name: tests 19 | run: | 20 | make test 21 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Binaries for programs and plugins 2 | *.exe 3 | *.exe~ 4 | *.dll 5 | *.so 6 | *.dylib 7 | 8 | # Test binary, built with `go test -c` 9 | *.test 10 | 11 | # Output of the go coverage tool, specifically when used with LiteIDE 12 | *.out 13 | 14 | # Dependency directories (remove the comment below to include it) 15 | vendor/ 16 | 17 | # coverage report file 18 | coverage.txt -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contribution Guidelines 2 | 3 | I am happy to accept PRs on CaskDB. For short bug/typo fixes, feel free to open a PR directly. 4 | 5 | I accept PRs related to the challenges mentioned in the different levels. Please open a separate PR for each challenge. The shorter PR is better. It is easier to review and catch bugs. 6 | 7 | For any new feature (or the level challenges) in the db, please open a GitHub issue and discuss your design/changes first. 8 | 9 | Do *NOT* club multiple things into a single PR. This adds unnecessary work for me, and it will take way longer for me to review. 10 | - CaskDB is an educational project. PRs become a great learning point for someone new to navigate the codebase and add a new feature. 11 | - Your PRs will be a great stepping stone for someone else who is new. 12 | 13 | Thank you! 14 | 15 | ## Branches 16 | 17 | - `start-here` contains all the base challenges and test cases 18 | - `master` implements the base challenges 19 | - `final` implements challenges from different levels 20 | 21 | If your PR is a bug/typo fix, open a PR with `master` as the base. I will backport the changes to `start-here` and `final` once merged. 22 | 23 | If you are implementing something new, open a PR with `final` as the base. 24 | 25 | ## "I am new; how do I get started?" 26 | 27 | Pick any challenge from any level and open a GitHub issue to discuss. I am happy to provide more resources/research papers to understand a particular concept. 28 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Avinash Sajjanshetty 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | build: 2 | go build 3 | 4 | test: 5 | go test -v ./... 6 | 7 | lint: 8 | go fmt ./... 9 | 10 | coverage: 11 | go test -coverprofile=coverage.txt ./... 12 | 13 | html: coverage 14 | go tool cover -html=coverage.txt -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ![logo](https://github.com/avinassh/py-caskdb/raw/master/assets/logo.svg) 2 | # CaskDB - Disk based Log Structured Hash Table Store 3 | 4 | ![made-with-go](https://img.shields.io/badge/Made%20with-Go-1f425f.svg) 5 | [![build](https://github.com/avinassh/py-caskdb/actions/workflows/build.yml/badge.svg)](https://github.com/avinassh/go-caskdb/actions/workflows/build.yml) 6 | [![codecov](https://codecov.io/gh/avinassh/py-caskdb/branch/master/graph/badge.svg?token=9SA8Q4L7AZ)](https://codecov.io/gh/avinassh/py-caskdb) 7 | ![GitHub License](https://img.shields.io/github/license/avinassh/go-caskdb) 8 | [![twitter@iavins](https://img.shields.io/twitter/follow/iavins?style=social)](https://twitter.com/iavins) 9 | 10 | ![architecture](https://user-images.githubusercontent.com/640792/167299554-0fc44510-d500-4347-b680-258e224646fa.png) 11 | 12 | CaskDB is a disk-based, embedded, persistent, key-value store based on the [Riak's bitcask paper](https://riak.com/assets/bitcask-intro.pdf), written in Go. It is more focused on the educational capabilities than using it in production. The file format is platform, machine, and programming language independent. Say, the database file created from Go on macOS should be compatible with Rust on Windows. 13 | 14 | This project aims to help anyone, even a beginner in databases, build a persistent database in a few hours. There are no external dependencies; only the Go standard library is enough. 15 | 16 | If you are interested in writing the database yourself, head to the workshop section. 17 | 18 | ## Features 19 | - Low latency for reads and writes 20 | - High throughput 21 | - Easy to back up / restore 22 | - Simple and easy to understand 23 | - Store data much larger than the RAM 24 | 25 | ## Limitations 26 | Most of the following limitations are of CaskDB. However, there are some due to design constraints by the Bitcask paper. 27 | 28 | - Single file stores all data, and deleted keys still take up the space 29 | - CaskDB does not offer range scans 30 | - CaskDB requires keeping all the keys in the internal memory. With a lot of keys, RAM usage will be high 31 | - Slow startup time since it needs to load all the keys in memory 32 | 33 | ## Community 34 | 35 | [![CaskDB Discord](https://img.shields.io/discord/851000331721900053)](https://discord.gg/HzthUYkrPp) 36 | 37 | Consider joining the Discord community to build and learn KV Store with peers. 38 | 39 | 40 | ## Dependencies 41 | CaskDB does not require any external libraries to run. Go standard library is enough. 42 | 43 | ## Installation 44 | ```shell 45 | go get github.com/avinassh/go-caskdb 46 | ``` 47 | 48 | ## Usage 49 | 50 | ```go 51 | store, _ := NewDiskStore("books.db") 52 | store.Set("othello", "shakespeare") 53 | author := store.Get("othello") 54 | ``` 55 | 56 | ## Cask DB (Python) 57 | This project is a Go version of the [same project in Python](https://github.com/avinassh/py-caskdb). 58 | 59 | ## Prerequisites 60 | The workshop is for intermediate-advanced programmers. Knowing basics of Go helps, and you can build the database in any language you wish. 61 | 62 | Not sure where you stand? You are ready if you have done the following in any language: 63 | - If you have used a dictionary or hash table data structure 64 | - Converting an object (class, struct, or dict) to JSON and converting JSON back to the things 65 | - Open a file to write or read anything. A common task is dumping a dictionary contents to disk and reading back 66 | 67 | ## Workshop 68 | **NOTE:** I don't have any [workshops](workshop.md) scheduled shortly. [Follow me on Twitter](https://twitter.com/iavins/) for updates. [Drop me an email](http://scr.im/avii) if you wish to arrange a workshop for your team/company. 69 | 70 | CaskDB comes with a full test suite and a wide range of tools to help you write a database quickly. [A Github action](https://github.com/avinassh/go-caskdb/blob/master/.github/workflows/build.yml) is present with an automated tests runner. Fork the repo, push the code, and pass the tests! 71 | 72 | Throughout the workshop, you will implement the following: 73 | - Serialiser methods take a bunch of objects and serialise them into bytes. Also, the procedures take a bunch of bytes and deserialise them back to the things. 74 | - Come up with a data format with a header and data to store the bytes on the disk. The header would contain metadata like timestamp, key size, and value. 75 | - Store and retrieve data from the disk 76 | - Read an existing CaskDB file to load all keys 77 | 78 | ### Tasks 79 | 1. Read [the paper](https://riak.com/assets/bitcask-intro.pdf). Fork this repo and checkout the `start-here` branch 80 | 2. Implement the fixed-sized header, which can encode timestamp (uint, 4 bytes), key size (uint, 4 bytes), value size (uint, 4 bytes) together 81 | 3. Implement the key, value serialisers, and pass the tests from `format_test.go` 82 | 4. Figure out how to store the data on disk and the row pointer in the memory. Implement the get/set operations. Tests for the same are in `disk_store_test.go` 83 | 5. Code from the task #2 and #3 should be enough to read an existing CaskDB file and load the keys into memory 84 | 85 | Run `make test` to run the tests locally. Push the code to Github, and tests will run on different OS: ubuntu, mac, and windows. 86 | 87 | Not sure how to proceed? Then check the [hints](hints.md) file which contains more details on the tasks and hints. 88 | 89 | ### Hints 90 | - Not sure how to come up with a file format? Read the comment in the [format file](format.go) 91 | 92 | ## What next? 93 | I often get questions about what is next after the basic implementation. Here are some challenges (with different levels of difficulties) 94 | 95 | ### Level 1: 96 | - Crash safety: the bitcask paper stores CRC in the row, and while fetching the row back, it verifies the data 97 | - Key deletion: CaskDB does not have a delete API. Read the paper and implement it 98 | - Instead of using a hash table, use a data structure like the red-black tree to support range scans 99 | - CaskDB accepts only strings as keys and values. Make it generic and take other data structures like int or bytes. 100 | 101 | ### Level 2: 102 | - Hint file to improve the startup time. The paper has more details on it 103 | - Implement an internal cache which stores some of the key-value pairs. You may explore and experiment with different cache eviction strategies like LRU, LFU, FIFO etc. 104 | - Split the data into multiple files when the files hit a specific capacity 105 | 106 | ### Level 3: 107 | - Support for multiple processes 108 | - Garbage collector: keys which got updated and deleted remain in the file and take up space. Write a garbage collector to remove such stale data 109 | - Add SQL query engine layer 110 | - Store JSON in values and explore making CaskDB as a document database like Mongo 111 | - Make CaskDB distributed by exploring algorithms like raft, paxos, or consistent hashing 112 | 113 | ## Line Count 114 | 115 | ```shell 116 | $ tokei -f format.go disk_store.go 117 | =============================================================================== 118 | Language Files Lines Code Comments Blanks 119 | =============================================================================== 120 | Go 2 320 133 168 19 121 | ------------------------------------------------------------------------------- 122 | format.go 111 35 67 9 123 | disk_store.go 209 98 101 10 124 | =============================================================================== 125 | Total 2 320 133 168 19 126 | =============================================================================== 127 | ``` 128 | 129 | ## Contributing 130 | All contributions are welcome. Please check [CONTRIBUTING.md](CONTRIBUTING.md) for more details. 131 | 132 | ## Community Contributions 133 | 134 | | Author | Feature | PR | 135 | |-------------------------------------------------|-----------|----------------------------------------------------| 136 | | [PaulisMatrix](https://github.com/PaulisMatrix) | Delete Op | [#6](https://github.com/avinassh/go-caskdb/pull/6) | 137 | | [PaulisMatrix](https://github.com/PaulisMatrix) | Checksum | [#7](https://github.com/avinassh/go-caskdb/pull/7) | 138 | 139 | ## License 140 | The MIT license. Please check `LICENSE` for more details. 141 | -------------------------------------------------------------------------------- /disk_store.go: -------------------------------------------------------------------------------- 1 | package caskdb 2 | 3 | import ( 4 | "errors" 5 | "fmt" 6 | "io" 7 | "io/fs" 8 | "os" 9 | "time" 10 | ) 11 | 12 | // defaultWhence helps us with `file.Seek` method to move our cursor to certain byte offset for read 13 | // or write operations. The method takes two parameters file.Seek(offset, whence). 14 | // The offset says the byte offset and whence says the direction: 15 | // 16 | // whence 0 - beginning of the file 17 | // whence 1 - current cursor position 18 | // whence 2 - end of the file 19 | // 20 | // read more about it here: 21 | // https://pkg.go.dev/os#File.Seek 22 | const defaultWhence = 0 23 | 24 | // DiskStore is a Log-Structured Hash Table as described in the BitCask paper. We 25 | // keep appending the data to a file, like a log. DiskStorage maintains an in-memory 26 | // hash table called KeyDir, which keeps the row's location on the disk. 27 | // 28 | // The idea is simple yet brilliant: 29 | // - Write the record to the disk 30 | // - Update the internal hash table to point to that byte offset 31 | // - Whenever we get a read request, check the internal hash table for the address, 32 | // fetch that and return 33 | // 34 | // KeyDir does not store values, only their locations. 35 | // 36 | // The above approach solves a lot of problems: 37 | // - Writes are insanely fast since you are just appending to the file 38 | // - Reads are insanely fast since you do only one disk seek. In B-Tree backed 39 | // storage, there could be 2-3 disk seeks 40 | // 41 | // However, there are drawbacks too: 42 | // - We need to maintain an in-memory hash table KeyDir. A database with a large 43 | // number of keys would require more RAM 44 | // - Since we need to build the KeyDir at initialisation, it will affect the startup 45 | // time too 46 | // - Deleted keys need to be purged from the file to reduce the file size 47 | // 48 | // Read the paper for more details: https://riak.com/assets/bitcask-intro.pdf 49 | // 50 | // DiskStore provides two simple operations to get and set key value pairs. Both key 51 | // and value need to be of string type, and all the data is persisted to disk. 52 | // During startup, DiskStorage loads all the existing KV pair metadata, and it will 53 | // throw an error if the file is invalid or corrupt. 54 | // 55 | // Note that if the database file is large, the initialisation will take time 56 | // accordingly. The initialisation is also a blocking operation; till it is completed, 57 | // we cannot use the database. 58 | // 59 | // Typical usage example: 60 | // 61 | // store, _ := NewDiskStore("books.db") 62 | // store.Set("othello", "shakespeare") 63 | // author := store.Get("othello") 64 | type DiskStore struct { 65 | // file object pointing the file_name 66 | file *os.File 67 | // current cursor position in the file where the data can be written 68 | writePosition int 69 | // keyDir is a map of key and KeyEntry being the value. KeyEntry contains the position 70 | // of the byte offset in the file where the value exists. key_dir map acts as in-memory 71 | // index to fetch the values quickly from the disk 72 | keyDir map[string]KeyEntry 73 | } 74 | 75 | func isFileExists(fileName string) bool { 76 | // https://stackoverflow.com/a/12518877 77 | if _, err := os.Stat(fileName); err == nil || errors.Is(err, fs.ErrExist) { 78 | return true 79 | } 80 | return false 81 | } 82 | 83 | func NewDiskStore(fileName string) (*DiskStore, error) { 84 | ds := &DiskStore{keyDir: make(map[string]KeyEntry)} 85 | // if the file exists already, then we will load the key_dir 86 | if isFileExists(fileName) { 87 | ds.initKeyDir(fileName) 88 | } 89 | // we open the file in following modes: 90 | // os.O_APPEND - says that the writes are append only. 91 | // os.O_RDWR - says we can read and write to the file 92 | // os.O_CREATE - creates the file if it does not exist 93 | file, err := os.OpenFile(fileName, os.O_APPEND|os.O_RDWR|os.O_CREATE, 0666) 94 | if err != nil { 95 | return nil, err 96 | } 97 | ds.file = file 98 | return ds, nil 99 | } 100 | 101 | func (d *DiskStore) Get(key string) string { 102 | // Get retrieves the value from the disk and returns. If the key does not 103 | // exist then it returns an empty string 104 | // 105 | // How get works? 106 | // 1. Check if there is any KeyEntry record for the key in keyDir 107 | // 2. Return an empty string if key doesn't exist 108 | // 3. If it exists, then read KeyEntry.totalSize bytes starting from the 109 | // KeyEntry.position from the disk 110 | // 4. Decode the bytes into valid KV pair and return the value 111 | // 112 | kEntry, ok := d.keyDir[key] 113 | if !ok { 114 | return "" 115 | } 116 | // move the current pointer to the right offset 117 | // TODO: handle errors 118 | d.file.Seek(int64(kEntry.position), defaultWhence) 119 | data := make([]byte, kEntry.totalSize) 120 | // TODO: handle errors 121 | _, err := io.ReadFull(d.file, data) 122 | if err != nil { 123 | panic("read error") 124 | } 125 | _, _, value := decodeKV(data) 126 | return value 127 | } 128 | 129 | func (d *DiskStore) Set(key string, value string) { 130 | // Set stores the key and value on the disk 131 | // 132 | // The steps to save a KV to disk is simple: 133 | // 1. Encode the KV into bytes 134 | // 2. Write the bytes to disk by appending to the file 135 | // 3. Update KeyDir with the KeyEntry of this key 136 | timestamp := uint32(time.Now().Unix()) 137 | size, data := encodeKV(timestamp, key, value) 138 | d.write(data) 139 | d.keyDir[key] = NewKeyEntry(timestamp, uint32(d.writePosition), uint32(size)) 140 | // update last write position, so that next record can be written from this point 141 | d.writePosition += size 142 | } 143 | 144 | func (d *DiskStore) Close() bool { 145 | // before we close the file, we need to safely write the contents in the buffers 146 | // to the disk. Check documentation of DiskStore.write() to understand 147 | // following the operations 148 | // TODO: handle errors 149 | d.file.Sync() 150 | if err := d.file.Close(); err != nil { 151 | // TODO: log the error 152 | return false 153 | } 154 | return true 155 | } 156 | 157 | func (d *DiskStore) write(data []byte) { 158 | // saving stuff to a file reliably is hard! 159 | // if you would like to explore and learn more, then 160 | // start from here: https://danluu.com/file-consistency/ 161 | // and read this too: https://lwn.net/Articles/457667/ 162 | if _, err := d.file.Write(data); err != nil { 163 | panic(err) 164 | } 165 | // calling fsync after every write is important, this assures that our writes 166 | // are actually persisted to the disk 167 | if err := d.file.Sync(); err != nil { 168 | panic(err) 169 | } 170 | } 171 | 172 | func (d *DiskStore) initKeyDir(existingFile string) { 173 | // we will initialise the keyDir by reading the contents of the file, record by 174 | // record. As we read each record, we will also update our keyDir with the 175 | // corresponding KeyEntry 176 | // 177 | // NOTE: this method is a blocking one, if the DB size is yuge then it will take 178 | // a lot of time to startup 179 | file, _ := os.Open(existingFile) 180 | defer file.Close() 181 | for { 182 | header := make([]byte, headerSize) 183 | _, err := io.ReadFull(file, header) 184 | if err == io.EOF { 185 | break 186 | } 187 | // TODO: handle errors 188 | if err != nil { 189 | break 190 | } 191 | timestamp, keySize, valueSize := decodeHeader(header) 192 | key := make([]byte, keySize) 193 | value := make([]byte, valueSize) 194 | _, err = io.ReadFull(file, key) 195 | // TODO: handle errors 196 | if err != nil { 197 | break 198 | } 199 | _, err = io.ReadFull(file, value) 200 | // TODO: handle errors 201 | if err != nil { 202 | break 203 | } 204 | totalSize := headerSize + keySize + valueSize 205 | d.keyDir[string(key)] = NewKeyEntry(timestamp, uint32(d.writePosition), totalSize) 206 | d.writePosition += int(totalSize) 207 | fmt.Printf("loaded key=%s, value=%s\n", key, value) 208 | } 209 | } 210 | -------------------------------------------------------------------------------- /disk_store_test.go: -------------------------------------------------------------------------------- 1 | package caskdb 2 | 3 | import ( 4 | "os" 5 | "testing" 6 | ) 7 | 8 | func TestDiskStore_Get(t *testing.T) { 9 | store, err := NewDiskStore("test.db") 10 | if err != nil { 11 | t.Fatalf("failed to create disk store: %v", err) 12 | } 13 | defer os.Remove("test.db") 14 | store.Set("name", "jojo") 15 | if val := store.Get("name"); val != "jojo" { 16 | t.Errorf("Get() = %v, want %v", val, "jojo") 17 | } 18 | } 19 | 20 | func TestDiskStore_GetInvalid(t *testing.T) { 21 | store, err := NewDiskStore("test.db") 22 | if err != nil { 23 | t.Fatalf("failed to create disk store: %v", err) 24 | } 25 | defer os.Remove("test.db") 26 | if val := store.Get("some key"); val != "" { 27 | t.Errorf("Get() = %v, want %v", val, "") 28 | } 29 | } 30 | 31 | func TestDiskStore_SetWithPersistence(t *testing.T) { 32 | store, err := NewDiskStore("test.db") 33 | if err != nil { 34 | t.Fatalf("failed to create disk store: %v", err) 35 | } 36 | defer os.Remove("test.db") 37 | 38 | tests := map[string]string{ 39 | "crime and punishment": "dostoevsky", 40 | "anna karenina": "tolstoy", 41 | "war and peace": "tolstoy", 42 | "hamlet": "shakespeare", 43 | "othello": "shakespeare", 44 | "brave new world": "huxley", 45 | "dune": "frank herbert", 46 | } 47 | for key, val := range tests { 48 | store.Set(key, val) 49 | if store.Get(key) != val { 50 | t.Errorf("Get() = %v, want %v", store.Get(key), val) 51 | } 52 | } 53 | store.Close() 54 | store, err = NewDiskStore("test.db") 55 | if err != nil { 56 | t.Fatalf("failed to create disk store: %v", err) 57 | } 58 | for key, val := range tests { 59 | if store.Get(key) != val { 60 | t.Errorf("Get() = %v, want %v", store.Get(key), val) 61 | } 62 | } 63 | store.Close() 64 | } 65 | 66 | func TestDiskStore_Delete(t *testing.T) { 67 | store, err := NewDiskStore("test.db") 68 | if err != nil { 69 | t.Fatalf("failed to create disk store: %v", err) 70 | } 71 | defer os.Remove("test.db") 72 | 73 | tests := map[string]string{ 74 | "crime and punishment": "dostoevsky", 75 | "anna karenina": "tolstoy", 76 | "war and peace": "tolstoy", 77 | "hamlet": "shakespeare", 78 | "othello": "shakespeare", 79 | "brave new world": "huxley", 80 | "dune": "frank herbert", 81 | } 82 | for key, val := range tests { 83 | store.Set(key, val) 84 | } 85 | for key, _ := range tests { 86 | store.Set(key, "") 87 | } 88 | store.Set("end", "yes") 89 | store.Close() 90 | 91 | store, err = NewDiskStore("test.db") 92 | if err != nil { 93 | t.Fatalf("failed to create disk store: %v", err) 94 | } 95 | for key := range tests { 96 | if store.Get(key) != "" { 97 | t.Errorf("Get() = %v, want '' (empty)", store.Get(key)) 98 | } 99 | } 100 | if store.Get("end") != "yes" { 101 | t.Errorf("Get() = %v, want %v", store.Get("end"), "yes") 102 | } 103 | store.Close() 104 | } 105 | -------------------------------------------------------------------------------- /format.go: -------------------------------------------------------------------------------- 1 | package caskdb 2 | 3 | // format file provides encode/decode functions for serialisation and deserialisation 4 | // operations 5 | // 6 | // format methods are generic and does not have any disk or memory specific code. 7 | // 8 | // The disk storage deals with bytes; you cannot just store a string or object without 9 | // converting it to bytes. The programming languages provide abstractions where you 10 | // don't have to think about all this when storing things in memory (i.e. RAM). 11 | // Consider the following example where you are storing stuff in a hash table: 12 | // 13 | // books = {} 14 | // books["hamlet"] = "shakespeare" 15 | // books["anna karenina"] = "tolstoy" 16 | // 17 | // In the above, the language deals with all the complexities: 18 | // 19 | // - allocating space on the RAM so that it can store data of `books` 20 | // - whenever you add data to `books`, convert that to bytes and keep it in the memory 21 | // - whenever the size of `books` increases, move that to somewhere in the RAM so that 22 | // we can add new items 23 | // 24 | // Unfortunately, when it comes to disks, we have to do all this by ourselves, write 25 | // code which can allocate space, convert objects to/from bytes and many other operations. 26 | // 27 | // This file has two functions which help us with serialisation of data. 28 | // 29 | // encodeKV - takes the key value pair and encodes them into bytes 30 | // decodeKV - takes a bunch of bytes and decodes them into key value pairs 31 | // 32 | //**workshop note** 33 | // 34 | //For the workshop, the functions will have the following signature: 35 | // 36 | // func encodeKV(timestamp uint32, key string, value string) (int, []byte) 37 | // func decodeKV(data []byte) (uint32, string, string) 38 | 39 | import "encoding/binary" 40 | 41 | // headerSize specifies the total header size. Our key value pair, when stored on disk 42 | // looks like this: 43 | // 44 | // ┌───────────┬──────────┬────────────┬─────┬───────┐ 45 | // │ timestamp │ key_size │ value_size │ key │ value │ 46 | // └───────────┴──────────┴────────────┴─────┴───────┘ 47 | // 48 | // This is analogous to a typical database's row (or a record). The total length of 49 | // the row is variable, depending on the contents of the key and value. 50 | // 51 | // The first three fields form the header: 52 | // 53 | // ┌───────────────┬──────────────┬────────────────┐ 54 | // │ timestamp(4B) │ key_size(4B) │ value_size(4B) │ 55 | // └───────────────┴──────────────┴────────────────┘ 56 | // 57 | // These three fields store unsigned integers of size 4 bytes, giving our header a 58 | // fixed length of 12 bytes. Timestamp field stores the time the record we 59 | // inserted in unix epoch seconds. Key size and value size fields store the length of 60 | // bytes occupied by the key and value. The maximum integer 61 | // stored by 4 bytes is 4,294,967,295 (2 ** 32 - 1), roughly ~4.2GB. So, the size of 62 | // each key or value cannot exceed this. Theoretically, a single row can be as large 63 | // as ~8.4GB. 64 | const headerSize = 12 65 | 66 | // KeyEntry keeps the metadata about the KV, specially the position of 67 | // the byte offset in the file. Whenever we insert/update a key, we create a new 68 | // KeyEntry object and insert that into keyDir. 69 | type KeyEntry struct { 70 | // Timestamp at which we wrote the KV pair to the disk. The value 71 | // is current time in seconds since the epoch. 72 | timestamp uint32 73 | // The position is the byte offset in the file where the data 74 | // exists 75 | position uint32 76 | // Total size of bytes of the value. We use this value to know 77 | // how many bytes we need to read from the file 78 | totalSize uint32 79 | } 80 | 81 | func NewKeyEntry(timestamp uint32, position uint32, totalSize uint32) KeyEntry { 82 | return KeyEntry{timestamp, position, totalSize} 83 | } 84 | 85 | func encodeHeader(timestamp uint32, keySize uint32, valueSize uint32) []byte { 86 | header := make([]byte, headerSize) 87 | binary.LittleEndian.PutUint32(header[0:4], timestamp) 88 | binary.LittleEndian.PutUint32(header[4:8], keySize) 89 | binary.LittleEndian.PutUint32(header[8:12], valueSize) 90 | return header 91 | } 92 | 93 | func decodeHeader(header []byte) (uint32, uint32, uint32) { 94 | timestamp := binary.LittleEndian.Uint32(header[0:4]) 95 | keySize := binary.LittleEndian.Uint32(header[4:8]) 96 | valueSize := binary.LittleEndian.Uint32(header[8:12]) 97 | return timestamp, keySize, valueSize 98 | } 99 | 100 | func encodeKV(timestamp uint32, key string, value string) (int, []byte) { 101 | header := encodeHeader(timestamp, uint32(len(key)), uint32(len(value))) 102 | data := append([]byte(key), []byte(value)...) 103 | return headerSize + len(data), append(header, data...) 104 | } 105 | 106 | func decodeKV(data []byte) (uint32, string, string) { 107 | timestamp, keySize, valueSize := decodeHeader(data[0:headerSize]) 108 | key := string(data[headerSize : headerSize+keySize]) 109 | value := string(data[headerSize+keySize : headerSize+keySize+valueSize]) 110 | return timestamp, key, value 111 | } 112 | -------------------------------------------------------------------------------- /format_test.go: -------------------------------------------------------------------------------- 1 | package caskdb 2 | 3 | import ( 4 | "testing" 5 | ) 6 | 7 | func Test_encodeHeader(t *testing.T) { 8 | tests := []struct { 9 | timestamp uint32 10 | keySize uint32 11 | valueSize uint32 12 | }{ 13 | {10, 10, 10}, 14 | {0, 0, 0}, 15 | {10000, 10000, 10000}, 16 | } 17 | for _, tt := range tests { 18 | data := encodeHeader(tt.timestamp, tt.keySize, tt.valueSize) 19 | timestamp, keySize, valueSize := decodeHeader(data) 20 | if timestamp != tt.timestamp { 21 | t.Errorf("encodeHeader() timestamp = %v, want %v", timestamp, tt.timestamp) 22 | } 23 | if keySize != tt.keySize { 24 | t.Errorf("encodeHeader() keySize = %v, want %v", keySize, tt.keySize) 25 | } 26 | if valueSize != tt.valueSize { 27 | t.Errorf("encodeHeader() valueSize = %v, want %v", valueSize, tt.valueSize) 28 | } 29 | } 30 | } 31 | 32 | func Test_encodeKV(t *testing.T) { 33 | tests := []struct { 34 | timestamp uint32 35 | key string 36 | value string 37 | size int 38 | }{ 39 | {10, "hello", "world", headerSize + 10}, 40 | {0, "", "", headerSize}, 41 | {100, "🔑", "", headerSize + 4}, 42 | } 43 | for _, tt := range tests { 44 | size, data := encodeKV(tt.timestamp, tt.key, tt.value) 45 | timestamp, key, value := decodeKV(data) 46 | if timestamp != tt.timestamp { 47 | t.Errorf("encodeKV() timestamp = %v, want %v", timestamp, tt.timestamp) 48 | } 49 | if key != tt.key { 50 | t.Errorf("encodeKV() key = %v, want %v", key, tt.key) 51 | } 52 | if value != tt.value { 53 | t.Errorf("encodeKV() value = %v, want %v", value, tt.value) 54 | } 55 | if size != tt.size { 56 | t.Errorf("encodeKV() size = %v, want %v", size, tt.size) 57 | } 58 | } 59 | } 60 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/avinassh/go-caskdb 2 | 3 | go 1.19 4 | -------------------------------------------------------------------------------- /hints.md: -------------------------------------------------------------------------------- 1 | # Hints 2 | 3 | _contains spoilers, proceed with caution!_ 4 | 5 | ## Tests 6 | Below is the order in which you should pass the tests: 7 | 8 | 1. `Test_encodeHeader` 9 | 2. `Test_encodeKV` 10 | 3. `TestDiskStore_Set` 11 | 4. `TestDiskStore_Get` and `TestDiskStore_GetInvalid` 12 | 5. `TestDiskStore_SetWithPersistence` 13 | 14 | ## Tasks 15 | 16 | I have relisted the tasks here again, but with more details. If you have difficulty understanding the steps, the following should help you. 17 | 18 | ### Read the paper 19 | The word 'paper' might scare you, but [Bitcask's paper](https://riak.com/assets/bitcask-intro.pdf) is very approachable. It is only six pages long, half of them being diagrams. 20 | 21 | ### Header 22 | 23 | | Test | Test_encodeHeader | 24 | |------|-------------------| 25 | 26 | The next step is to implement a fixed-sized header similar to Bitcask. Every record in our DB contains a header holding metadata and helps our DB read values from the disk. The DB will read a bunch of bytes from the disk, so we need information on how many bytes to read and from which byte offset. 27 | 28 | **Some more details:** 29 | 30 | The header contains three fields timestamp, key size, and value size. 31 | 32 | | Field | Type | Size | 33 | |------------|------|------| 34 | | timestamp | int | 4B | 35 | | key_size | int | 4B | 36 | | value_size | int | 4B | 37 | 38 | We need to implement a function which takes all these three fields and serialises them to bytes. The function signature looks like this: 39 | 40 | ```go 41 | func encodeHeader(timestamp uint32, keySize uint32, valueSize uint32) 42 | ``` 43 | 44 | Then we also need to write the opposite of the above: 45 | 46 | ```go 47 | func decodeHeader(header []byte) (uint32, uint32, uint32) 48 | ``` 49 | 50 | **More Hints:** 51 | - Read this [comment](https://github.com/avinassh/go-caskdb/blob/0ae4fab/format.go#L3,L37) to understand why do we need serialiser methods 52 | - Not sure how to come up with a file format? Read the comment in the [format module](https://github.com/avinassh/go-caskdb/blob/0ae4fab/format.go#L41,L63) 53 | 54 | ### Key Value Serialisers 55 | 56 | | Test | Test_encodeKV | 57 | |------|---------------| 58 | 59 | Now we will write encode and decode methods for key and value. 60 | 61 | The method signatures: 62 | ```go 63 | func encodeKV(timestamp uint32, key string, value string) (int, []byte) 64 | func decodeKV(data []byte) (uint32, string, string) 65 | ``` 66 | 67 | Note that `encodeKV` method returns the bytes and the bytes' size. 68 | 69 | ### Storing to Disk 70 | 71 | | Test | TestDiskStore_Set | 72 | |------|-------------------| 73 | 74 | This step involves figuring out the persistence layer, saving the data to the disk, and keeping the pointer to the inserted record in the memory. 75 | 76 | So, implement the `DiskStore.Set` class in `disk_store.go` 77 | 78 | **Hints:** 79 | - Some meta info on the DiskStore and inner workings of the DiskStore are [here](https://github.com/avinassh/go-caskdb/blob/0ae4fab/disk_store.go#L24,L63). 80 | 81 | ### Start up tasks 82 | 83 | | Test | TestDiskStore_SetWithPersistence | 84 | |------|----------------------------------| 85 | 86 | DiskStore is a persistent key-value store, so we need to load the existing keys into the `keyDir` at the start of the database. 87 | -------------------------------------------------------------------------------- /memory_store.go: -------------------------------------------------------------------------------- 1 | package caskdb 2 | 3 | type MemoryStore struct { 4 | data map[string]string 5 | } 6 | 7 | func NewMemoryStore() *MemoryStore { 8 | return &MemoryStore{make(map[string]string)} 9 | } 10 | 11 | func (m *MemoryStore) Get(key string) string { 12 | return m.data[key] 13 | } 14 | 15 | func (m *MemoryStore) Set(key string, value string) { 16 | m.data[key] = value 17 | } 18 | 19 | func (m *MemoryStore) Close() bool { 20 | return true 21 | } 22 | -------------------------------------------------------------------------------- /memory_store_test.go: -------------------------------------------------------------------------------- 1 | package caskdb 2 | 3 | import "testing" 4 | 5 | func TestMemoryStore_Get(t *testing.T) { 6 | store := NewMemoryStore() 7 | store.Set("name", "jojo") 8 | if val := store.Get("name"); val != "jojo" { 9 | t.Errorf("Get() = %v, want %v", val, "jojo") 10 | } 11 | } 12 | 13 | func TestMemoryStore_InvalidGet(t *testing.T) { 14 | store := NewMemoryStore() 15 | if val := store.Get("some rando key"); val != "" { 16 | t.Errorf("Get() = %v, want %v", val, "") 17 | } 18 | } 19 | 20 | func TestMemoryStore_Close(t *testing.T) { 21 | store := NewMemoryStore() 22 | if !store.Close() { 23 | t.Errorf("Close() failed") 24 | } 25 | } 26 | -------------------------------------------------------------------------------- /store.go: -------------------------------------------------------------------------------- 1 | package caskdb 2 | 3 | type Store interface { 4 | Get(key string) string 5 | Set(key string, value string) 6 | Close() bool 7 | } 8 | -------------------------------------------------------------------------------- /workshop.md: -------------------------------------------------------------------------------- 1 | # Workshop 2 | 3 | I don't have any workshops scheduled shortly. [Follow me on Twitter](https://twitter.com/iavins/) for updates. [Drop me an email](http://scr.im/avii) if you wish to arrange a workshop for your team/company. 4 | 5 | A typical workshop session is 4 hours and can span over two days. It is a hands-on session where I will live code with you. Though I will code in Python, you / your team may pick any programming language you are comfortable with. We will also pair/mob program together. 6 | 7 | | Session | Duration | 8 | |---------------------------------------|----------| 9 | | The building blocks of the database | 1 Hour | 10 | | Coffee break | 5 Min | 11 | | Live coding | 2 Hour | 12 | | Break | 10 Min | 13 | | Some Level 1 challenges | 30 Min | 14 | | [Optional] Covering the Bitcask Paper | 30 Min | 15 | | [Optional] Discussion of Level 2, 3 | 30 Min | --------------------------------------------------------------------------------