├── LICENSE ├── README.md └── leveldb ├── diagrams ├── LICENSE.txt ├── README.txt ├── comments_for_diagram_improvement.txt ├── computing_account_balance_for_bitcoin.drawio ├── computing_account_balance_for_bitcoin.png ├── eth_database_simple(2).xml ├── ethereum_database.jpg ├── ethereum_database.xml ├── patricia_trie_adding.jpg ├── patricia_trie_adding.xml ├── patricia_trie_adding_wood_wooden.xml ├── patricia_trie_adding_wooden.jpg ├── sending_a_partial_bitcoin.drawio └── sending_a_partial_bitcoin.png ├── images ├── accounts.png ├── block_root_key_values.png ├── consensus_after.png ├── consensus_before.png ├── console.png ├── console_status.png ├── eth_database_simple.jpg ├── ethereum_read_stream.png ├── ethereum_read_stream_first_output.png ├── first_transaction.png ├── state_trie_diagram.png ├── three_accounts.png └── two_accounts.png ├── javascript ├── print_state_trie.js ├── print_state_trie_keys.js ├── print_transaction_trie_keys.js ├── reading_ethereum_leveldb_using_nodejs_and_patricia_trie.js ├── reading_ethereum_leveldb_using_nodejs_level_key_read.js ├── reading_ethereum_leveldb_using_nodejs_level_stream_read.js ├── reading_leveldb_using_nodejs_level_key_read.js ├── reading_leveldb_using_nodejs_level_stream_read.js ├── reading_leveldb_using_nodejs_level_value_read.js ├── testing_leveldb_batch_processing_using_nodejs_level.js ├── testing_leveldb_json_processing_using_nodejs_level.js └── testing_leveldb_using_nodejs_level.js ├── leveldb.md └── scratch_pad.md /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2018 Timothy McCallum 2 | This work is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/4.0/. 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | The aim of this work is to provide as much information, as possible, in relation to how Ethereum stores its transactions, contracts, account balances and more. 2 | 3 | As part of this work, we: 4 | - begin with an overview of what blockchain "state" means 5 | - provide an overview of the popular blockchain database software, leveldb 6 | - introduce the Patricia Trie data structure which Ethereum uses exclusively 7 | - learn how Ethereum implements the Patricia Trie data structure inside leveldb 8 | - learn about Ethereum's leveldb structure 9 | - provide a step by step guide on installing a testbed for Ethereum and leveldb (using Ubuntu Linux 16.04LTS) 10 | - provide justification for using an Ethereum private network for the testbed (as apposed to the mainnet or testnets) 11 | - configure and run Ethereum on a private network 12 | - configure Ethereum to mine on the private network 13 | - execute transactions and smart contracts on the Ethereum private network 14 | - explore Ethereum's application data storage layer, leveldb, before and after activity to discover how it works 15 | 16 | Please begin [here](https://github.com/tpmccallum/ethereum_database_research_and_testing/blob/master/leveldb/leveldb.md) 17 | -------------------------------------------------------------------------------- /leveldb/diagrams/LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2018 Timothy McCallum 2 | This diagram is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/4.0/. 3 | -------------------------------------------------------------------------------- /leveldb/diagrams/README.txt: -------------------------------------------------------------------------------- 1 | The diagrams here are currently in draft format. Please review and comment on the diagrams as you see fit. 2 | 3 | Whilst the ethereum_database.jpeg file can be opened natively, the ethereum_database.xml diagram file can be opened and edited using the following free online software < https://www.draw.io/ > 4 | 5 | Please make any adjustments as you wish and create a pull request if you would like to improve the information and/or the diagram. 6 | 7 | -------------------------------------------------------------------------------- /leveldb/diagrams/comments_for_diagram_improvement.txt: -------------------------------------------------------------------------------- 1 | 2 | -------------------------------------------------------------------------------- /leveldb/diagrams/computing_account_balance_for_bitcoin.drawio: -------------------------------------------------------------------------------- 1 | 5VhNj5swFPw1HDcy5is5hmTbatVWlXKoejTggLUGI2OaZH99n4NJIBBpK2W3dJNL7PGz/TwzfglYzirff5akzL6JhHILo2RvOWsLY9sOMHxp5NAg/sJpgFSyxASdgQ17oQZEBq1ZQqteoBKCK1b2wVgUBY1VDyNSil0/bCt4f9eSpHQAbGLCh+hPlqisQec4OONfKEuzdmfbXzQjOWmDzUmqjCRi14GcR8tZSSFU08r3K8o1eS0vzbxPV0ZPiUlaqNdMWFW/np7CtSBpsNiTNVp+t18ejDq/Ca/NgU2y6tAyAHmXuhkTCSuFu4wpuilJrLEdiA5YpnIOPRuaZj0qFd1fTdQ+HR98Q0VOlTxAiJngIMOYsYztLWZeg+w6EpigrMt+O5MY1dPT4mdioGG4+QuenAFPSOeEIhUPCIOTqz4plZLima4EFxKQQhQQGW4Z5xcQ4SwtNNXAFAU81DwyMOPSDOQsSfQ2oypIURcJ1adANxIC94XAaCiDOyIDfisV3Am61b10q4/+sVe9e/SqO/TqSNF4V7f6E3SrN8HaGtyjX72p1db5BN3qT662Lu7Rq/70amv7eNARArgBFjDCV+Ro3VvnfBkr0aX1K4ko/yEqppjQ9EZCKZFDANcDIYmf0yOprUwWdrbHz4g0SlzcA1ErzgpQvX1AudW/M9vrqxIML4f3rpfDHvnF80IrCGezmd4vBw4hiSW0m1uj8QDCYDuf67sSSWil6kiQT3LNZBFV5bGP/qNVvPXHrQau268Gp8e1ju9Gi0HwZr4bVuVWRNbRsFG1BSKmYsGKh5hDDOKsUnVRlZoDEwmpRJezAeus+GEV9i/q/XxEYXybcg/d8yuQ41jnRZLz+Ac= -------------------------------------------------------------------------------- /leveldb/diagrams/computing_account_balance_for_bitcoin.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/diagrams/computing_account_balance_for_bitcoin.png -------------------------------------------------------------------------------- /leveldb/diagrams/eth_database_simple(2).xml: -------------------------------------------------------------------------------- 1 | 7VrRcps6EP0aP9YDEmB4jNO4fbh35k7TmT7LIIMmgBgh1879+q6MZANSxskYJ01TP8SwkpA452i1u84M31b7L4I0xb88o+UMedl+hj/PEIq9BfxVhsfOEERhZ8gFyzqTfzLcs/+pNnraumUZbQcdJeelZM3QmPK6pqkc2IgQfDfstuHlcNaG5NQy3KektK0/WCYL/Vqhd7J/pSwvzMy+p1vWJH3IBd/Wer4ZwpvDp2uuiHmW7t8WJOO7ngnfzfCt4Fx2V9X+lpYKWgNbN271ROtx3YLW8jkDUNKN+EnKLTVLPixMPhowDq9D1QBvhpe7gkl635BUte6AfbAVsirhzodLwSWRjNdwm6ju9oL0Gn9SIem+Z9IL/EJ5RaV4hC661aCutYRw0N3vTsygSC+76LMSaxUSrYb8+OgTInChQXEDlPgOgKISpliu4SJXF3eyoIJuK8V/ydOHtCCsNr1ggmPH40hxsozABkzkENFWCv5Ab3nJBVhqXkPP5YaV5chESpYr2FNAmoJ9qRBmoOkb3VCxLFPTOCkckjwBa3E8YM03HqDHmh87WMPxBKSF51UNO69Rly2r85LeKKcBL54xAf6k0++OtvK84F1MqIcZv+HNsTFpP+fNw3jwSaZB3I9HG8V40z7kxlH1IY+8CSBfTOxIHLg+AYgDticxQs/BCF0JoyP4706XF2Fu6TJOHLoMroW5y4H/jsIMzoN0PWFGf4UJR8/CjiyuKMz34jGD8yBdTZj+h/SYgXfeGSRXg/x9OMzABHlHjOLX1GXgAGmUFnwXpG5Jp0HkfReMunOCsxnAFLs4iQZo+XFko+U70MKToOWKx0dowTeplHLqddu8yb13D6krfXuqxpFAtHhNqlyRwFmqADouVE3lrcEL/VGsGdk6d3nOabBznegj7L7RlLJGtpch9XY1gQulPfTZOHQUBYwC+/SgSeiJXfQMlNx2Mv6m6m/HEt0fSAQ2QwwRHraIcJXUTLnwIh7QMwLfMyEFaZuu7Lthe4XLINpr+Va9wyQZAh764iCxffHCoVdjuwwnl16fiH1Zdaho9zEaK0zypmf9h6xp+R9vmQZtzaXkFXQoVcPyWMY2Oj4VsvHyMNmNocBz8aHX87mQUpXsb9Rbo1Wa1WjOUl5vGOhZzFOYEa0yIgl8KTu4xRXfbFhKP6k64AoFJ0OQfAJI5k2dT0FsOPREQWhH2KYSOXV10n9O0f033QCRyQ5eYwPgFyR/fzfASzfAkNjQkdZfbQNgbBHbS5Za+2eS7kD+U09jf3Qaw/FscRE4gtZJoiJsZ7I6D/poJKBRSGTXE65Hgp0gQ6LA6CFR+Gg8jELTxHZME/EAt6df2g9tvf9mwHe/AA== -------------------------------------------------------------------------------- /leveldb/diagrams/ethereum_database.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/diagrams/ethereum_database.jpg -------------------------------------------------------------------------------- /leveldb/diagrams/ethereum_database.xml: -------------------------------------------------------------------------------- 1 | 7Vvfc5s4EP5rPHf3EI/4YQyPTtJcO02nmSa9e8wIkI0mgHxCTpz+9bcSyDYgxyTBSes6DzEsK5D2+3a1K8TAOcuWf3M8T76wmKQDG8XLgXM+sG3LRy78SMljKQkcVApmnMaV0lpwTX+QSqjVFjQmRU1RMJYKOq8LI5bnJBI1GeacPdTVpiytP3WOZ6QluI5w2pb+S2ORlFJ/hNbyj4TOEv1kC1VXQhzdzThb5NXzBrYzVX/l5Qzre1X6RYJj9rAhcj4MnDPOmCiPsuUZSaVttdnKdhdbrq76zUkuujSwg7LFPU4XRHdZdUw8amOo4RDZAA2c04eECnI9x5G8+gDwgywRWQpnFhxyJrCgLIfTQKpXtydckOXWPlqrkQOjCMuI4I+gUjU4scZVm4pNjld18mGNzRhVsmQTF39ccaLiw2x187VN4KAyyxYT7bbQDpvgYl6SdEqX0o59GMVyUc0mvuu1bOKjtkm07DUW8XZbBHg9l4c0U562aQ05bAquNknpTNJEsPmG9BKHJL1iBa1IFDIhWAYKqbxwunKvM5Yyrp6lHcw5VQ+baGMjk+Wr/pwnQshQMpFjti+iOLeHFILJlALT+TCCJ9oXMRYYfqS8gF82ndKInDg+HNvuWuAGJ5btD+f5rA9YvTrVfeS3YHX8Nqxa9hpY/f6JHlMOghLJgi3kCPZB/sBz34z8HQLmkfz9kD+w2zFtX+TXCUoNVy+FJ5yGKYvu/lswQUAyE4rC5ZWY3svHVliCVGpJuMhUrM90K8FxXmDlDsU3OcPr+/PmfdeSmEo7L1Lx2EWbLOEZ5wq53cozXFzSjHbqBuh+L4BDHVQTXCRd9FI2K+CfJNtu5YzmhHdTXH7s2IECEhXSFYd8kYXdelDSvWsnIDwSOhed+QChxfmeRykkx120ZWLdZXQsjzopCpoRsFs276QMqWB6/iwCb7pIF/1FZ1NoiQxt0mtbUpOXN+I6RLaSOJzdER2DwXYyxk9pmjZEG2HfNG82p4KMxrF80imDK9NUVQUJyEjez6QZ1OdMD7ViqzU2zJlju4dJ07G2RtfnxdBaq1L0NU8hOsA9AMCq9JjIMdQmuRkVySKs5jYxzyIwfLqQJ0QkhJNFditnvBAX5JaTgmAeJbc4j28F8J3K2e1CcELkOHEhZCS4SMk9SeNQEQrPOM6e4KGZdaEW3CS0kNVyeR9pN3mawgSbq6iLZAHGZRGcyIGecQIDvVeHLMuUr6CJEJyGC2mAk+sEWAG8upMq7hAARJ9y6HWu7IMlMpflzTcdQPdmCJdvmESLkgeJjSq055JKbKo6sdm9gX2mVAs5kyirrzOLqp9R2ckh49KOVTuZVYSPJ4XMMlQXL4bd7Wd2zA2H6+qjmnfQJYnqE15pcuF6ZfwaH33U6x8mt0Qjg186Pbil+5xktiRQei0YL9PaHcWAwdpqgSjw5ZEc8DjoJ7KN7HpoM60P2Mhgw14KgtETmaNm7GfyaPK0AySyhmS0e8nGNSHSByDbJ5uV5f9R134rSDzt7e8Bid2CZBLHMNEa5kxLhkAUmgqTT5AMCTql5kz8UIFr+pKLnLcDzmkBV9aXYGm+iMSCyxSjIJyC0X6oXIXm8O/b5dXhAuK5fgOQt/Mkrw3IizNpWcdB8RcW8+fUgM2GIU7xC5sWZS7RtfhtNo9YTDYL7W5Z4i9avnmNJMd126SzjEkO6oN2o620m6sQUClqlqHpyGtzDi1H/nA4xMR3NzBTNzj4IN6Ab2S34QvcPcWMsQG8faf5kotVnu/b/dgQbuk3jGi1jLi/RN8zvSD53RN9CznBTkz2Nhma6tdjpm9ZzvjdMBm3y+HPJIrwHcjskXcSlutEato+WACQ1Yj2I9OyjgGBUR8ItOtflY4jkkPGRPWruoM0vPWuht+eJK2izrV86wQ6N5xuiUrNrPfPL4TfpbLNFRacRhRXzf86XBxbK3qGbMk40ffxKnhsSpdaMJbp0RHIZ6Zsrt9evNgfkoexvu35dW8I2nuCzFlvH5Wf357Pj1nv2LJ2IbKv/Mo/rm6bEPFR8G6I2AZEnJuv518P19xNB7B0FfgW9jYtgR64vf3GNGo5b8hv037f5g6G9cadY070zOnc8Mp/bxlRYJzN+3kRMMPFFUD40radNyM225Zj2UKXX3S1fzxu+LvfsY5dFcCvYkmHHONDtW8K1NTWuSjB8s1fF6c/XNf266h57f3Lls4Wevdsd6dnH89/tnNULo2eNBdGy412RbVWBBMj+UNuuVPfhdkoZ3E73uk6tqD5LCUT+RHcoP4BxgMpxBZHeLqs7cExrMbuU8dr54uOY4hnXh/hzLgu95NR4XjeyVUG7bcIelcqqblHJa3lpe/tSOrTVP1BKRraQe0PjbRK9UEsGlpeP+632i262mdqKNdMu0x7cT9j/fCT0evQz7d7jWK/Xs1+bw/po7wKmpsExu2pxuqH63C6/lZbXdv4IN758D8= -------------------------------------------------------------------------------- /leveldb/diagrams/patricia_trie_adding.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/diagrams/patricia_trie_adding.jpg -------------------------------------------------------------------------------- /leveldb/diagrams/patricia_trie_adding.xml: -------------------------------------------------------------------------------- 1 | 5Vldc6MgFP01eRfwIz42aXf3ZWd3Ng99pkqUKZEMkibdX78YQQOaiZtJnLbmIQMHucA5eO8FZ2i5OXwXeJv/5ClhM+ilhxl6nEEY+7H6r4D3Ggj8qAYyQdMaAi2won+JBj2N7mhKSutByTmTdGuDCS8KkkgLw0Lwvf3YmjN71C3OSAdYJZh10WeayrxG54HX4j8IzXIzMvB0ywtOXjPBd4UebwbR+virmzfY2NLPlzlO+f4EQk8ztBScy7q0OSwJq6g1tNX9vp1pbeYtSCGHdIB1hzfMdsTM+Dgv+W64UB0U7aqy2OdUktUWJ1XLXgmvsFxumKoBVcTlttZiTQ9E2V+sKWNLzrhQWMGLyoQejghJDmenDBoi1P4ifEOkeFeP6A4IhXUXvbciTeW+FcrolJ9oZDCst0bWGG7pUQXNUD9baABbRfpQbUBVSxguS5rYJNUdSNrZfxeXP3B9gjAs6Zttvm/ReoTfnKqBW3ZDz2IXQIe3ku9EQnSv053lGoK2Ieg5hiQWGZEdQ0cNmmUPkiWegCyew6ZxqP8rC3T0RTC6lyzGmX9lXSCa23RG8DpdQGwb8gN4N13A19cFRHaQ8OMr3xfgRJsgut/7En59XRAMHF38K8NLGDi6+HfTZQLxBYbwNn4MebahO/oxOCS+fLDkFTjh1/e12x8he/2MyT50/EXjP8bga0i6/9H4mtt8NWnFGHz5PXyFTI2weFGFrCo8G0QZa8AOq2r90qavlIK/EoevHgoxo1lROWHFF1H4omKTqoP9g27Y0DRl5/Q6HtwrdR69G51VTa5l5PDDjhx+jxzuoesqOYLLcvyZlhxgHtveJO56k7vJ0ZfdOXL8mpYcMLCdFQzBeHJEl+V4nJYcwIm1ARxPDvQZcznnTqY5goxxEznkDP/B+EL+3OELjsfXZ8x9584dhvmqMgZffbnv1KMVsuXwRnSPA1LriUUrBBxvArtvx43kUNX2i1x9UdF+9URP/wA= -------------------------------------------------------------------------------- /leveldb/diagrams/patricia_trie_adding_wood_wooden.xml: -------------------------------------------------------------------------------- 1 | 5Vldc+IgFP01edyZBEjUx6rd7stud7Y702eaYMIUg0Ow2v76hQRUSNzNWD+61gcnHOAC59ycXGMAJ/P1ncCL4jvPCAtAmK0DOA0AGIYD9a2B1wZASdwAuaBZA0Vb4IG+EQOGBl3SjFTOQMk5k3ThgikvS5JKB8NC8JU7bMaZu+oC56QFPKSYtdFHmsnCHCsOt/g3QvPCrhyFpucJp8+54MvSrBcAOKs/Tfcc21hmfFXgjK92IHgbwIngXDZX8/WEME2tpa2Z93VP72bfgpSyzwTQTHjBbEnsjut9yVfLhZqgaFeN8aqgkjwscKp7Vkp4hRVyzlQrUpe4WjRazOiaqPjjGWVswhkXCit5qUO092e2/EKEJOsdyOz3jvA5keJVDTG9ECbNFJNbA0PlaiuU1anY0chi2KRGvgm8pUddGIa62YI92CqzG52AqpUyXFU0dUnaSwDJnJRsH7/n+QRhWNIXN5G7Dm1W+Mmp2smW3SR02I2Ax1vFlyIlZtZuZvmBgBsIhF4giUVOZCtQrcHm2L1ksa5xzbqA0dClEw0O0wXEbiAIBifTZXT9ukAQe7qgA++XJPZ0QafSBfS5Xz6Y6wPPTqA14zPYfi/f/2h8Dd10iqx/nIMv1MFXwtQK4yd1keuLx3sLqWgbtEWrOq10+auk4M/EI6yDQ8xoXmpXURwShY81d1SVeDemY06zjO0TrC7htDzT8EhViy2JrR4oaemBOvTwH78H6RH/W49f0yCehJ9LE4CA691J+x45lSbwf6y8Y9+Dwdk8BXZ5sJfD99PPlb/Q1hqXyN+j1XZeIdfBycVqu+HIzfdk5IboW9sh4NbcCEE30PFqO3SZ2q6VtL1ktPRE0KMnOpuv2KX+5iu3P97xbBSkom/4qR6g7/uFFrnecDwO4qlmeSl51bwCi3ZMhpGZ7LAYybVIldKMlvlv3Zh+QcfRAYaun8BRW4dT+Yl9FF+1nySR5yfDw/wEAjcQQl6gI/rJZX77vCuPwdD3E3Q+P+n47VObxzUZBYq8dxX2QblDMDyOUajm9m14k7/bfxzg7R8= -------------------------------------------------------------------------------- /leveldb/diagrams/patricia_trie_adding_wooden.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/diagrams/patricia_trie_adding_wooden.jpg -------------------------------------------------------------------------------- /leveldb/diagrams/sending_a_partial_bitcoin.drawio: -------------------------------------------------------------------------------- 1 | 5VnRbpswFP2aPK4CO1B4bNKunbRp1Tpp26MDbvBqcGRMk/TrZwebAIaSdmWpsjxE9uVybZ9z7rWBCZynm2uOVskXFmM6AU68mcDLCQAhBPJfGbalwfPC0rDkJC5N7t5wR56wNjraWpAY5w1HwRgVZNU0RizLcCQaNsQ5Wzfd7hltjrpCS2wZ7iJEbesPEouktAbgfG+/wWSZmJFdX68vRcZZryRPUMzWNRO8msA5Z0yUrXQzx1RhZ3Ap7/vYc7WaGMeZOOSG5Ed4/c13n2bw5pKi3/NttqYf/DLKI6KFXrCerNgaBDgrshirIM4EztYJEfhuhSJ1dS0pl7ZEpFT2XNnU4TAXeNM7T7davVQNZikWfCtd9A0g0IBpxbi+7q/3+Fc+SR1744g058sq9h4W2dDIvAAl10JpxhYWUJLglWoWKb2IBOMSDgUEkWL6jBaY3rKcCMIy6bJgQrBUOlB1YYaih+UO5jmj6j4ZDd7vfrUYF5Qs1b2CtTBnhaAkw/MqA5y3IcINvDOvSUUYGkuNDM/r4MIZiwvP4kICI9V46mz4jnMIF115MRoXwXD1wFl8ocqw7GUsk8ZZjPJkV07cJnB4Q8RPhZdcVtn7VbtyudFQ7jrbBq44tmp4C1U5JVbwCA9VQhv9AWyNjWOKBHlsTqMLbz3CLSNyghW5UwjOQuBUP7dBNXRbBJaL0THqJb8dFrQ0YzZfE0ggvsTCCrQTQwXC6/URWvr4zlGWo2iXeMD5lK0KYUlG5ohoSiMXnD1gk45aR/eE0pYJ6aSMJPuYd2RrSuJYDdO5jTU3uhF2Mmh4rYnKnXaoCoy2kTkDjHwtxElT0s6Id0CJfbjo3csixON/cgarKo4pHOdBx2bTeQgbbbNxwTsEanreAiqYWjB1yckbDaWphZJcgbMQ0ekmNYRNEiTkB5EwXk7bh1R1qjlxGtq5ABz/yDTY54/3VzHA9MgVA9hngv9BrOY9jaHBPz+uWAEcFmvtYSqiKM9J1CXP4cehoz3m+EET9HDaDHHogw2A4fOBRn6wAXaBPzmyqjgGY/hKstqHSStQD1kSPbStua2UQ94/4b5xeuflPOsvG+UM3lY5B7xxHVKOeVNi2sPvSVTnFnMiZ69q74venRxLf5458P+t/rzQbwWCo+ivb5zeeQXP+r9Uf7K7/5hQuu+/yMCrPw== -------------------------------------------------------------------------------- /leveldb/diagrams/sending_a_partial_bitcoin.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/diagrams/sending_a_partial_bitcoin.png -------------------------------------------------------------------------------- /leveldb/images/accounts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/images/accounts.png -------------------------------------------------------------------------------- /leveldb/images/block_root_key_values.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/images/block_root_key_values.png -------------------------------------------------------------------------------- /leveldb/images/consensus_after.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/images/consensus_after.png -------------------------------------------------------------------------------- /leveldb/images/consensus_before.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/images/consensus_before.png -------------------------------------------------------------------------------- /leveldb/images/console.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/images/console.png -------------------------------------------------------------------------------- /leveldb/images/console_status.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/images/console_status.png -------------------------------------------------------------------------------- /leveldb/images/eth_database_simple.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/images/eth_database_simple.jpg -------------------------------------------------------------------------------- /leveldb/images/ethereum_read_stream.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/images/ethereum_read_stream.png -------------------------------------------------------------------------------- /leveldb/images/ethereum_read_stream_first_output.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/images/ethereum_read_stream_first_output.png -------------------------------------------------------------------------------- /leveldb/images/first_transaction.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/images/first_transaction.png -------------------------------------------------------------------------------- /leveldb/images/state_trie_diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/images/state_trie_diagram.png -------------------------------------------------------------------------------- /leveldb/images/three_accounts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/images/three_accounts.png -------------------------------------------------------------------------------- /leveldb/images/two_accounts.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/tpmccallum/ethereum_database_research_and_testing/9a3cfce88c9d73f9661a9b8db051af39adb2c04d/leveldb/images/two_accounts.png -------------------------------------------------------------------------------- /leveldb/javascript/print_state_trie.js: -------------------------------------------------------------------------------- 1 | //Run the following first 2 | //npm install levelup leveldown rlp merkle-patricia-tree assert --save 3 | 4 | //Just getting the requirements 5 | var Trie = require('merkle-patricia-tree/secure'); 6 | var levelup = require('levelup'); 7 | var leveldown = require('leveldown'); 8 | var RLP = require('rlp'); 9 | var assert = require('assert'); 10 | 11 | //Connecting to the leveldb database 12 | var db = levelup(leveldown('/home/timothymccallum/gethDataDir/geth/chaindata')); 13 | 14 | //Obtaining the state root (of the latest block) using this command web3.eth.getBlock(web3.eth.defaultBlock).stateRoot 15 | var root = '0x8c77785e3e9171715dd34117b047dffe44575c32ede59bde39fbf5dc074f2976'; 16 | 17 | //Creating a trie object of the merkle-patricia-tree library 18 | var trie = new Trie(db, root); 19 | 20 | //Creating a nodejs stream object so that we can access the data 21 | var stream = trie.createReadStream() 22 | 23 | //Turning on the stream (because the node js stream is set to pause by default) 24 | stream.on('data', function (data){ 25 | //Capture the RLP serialised data structure 26 | var arrayOfStateValues = RLP.decode(data.value); 27 | //Loop through the data of the 3 accounts in the array to obtain the nonce, balance, storageRoot and codeHash for each one 28 | for (i = 0; i < arrayOfStateValues.length; i++){ 29 | console.log(RLP.decode(arrayOfStateValues[i], [skipRemainderCheck=false])); 30 | console.log(""); 31 | } 32 | 33 | }); -------------------------------------------------------------------------------- /leveldb/javascript/print_state_trie_keys.js: -------------------------------------------------------------------------------- 1 | //Run the following first 2 | //npm install levelup leveldown rlp merkle-patricia-tree assert --save 3 | 4 | //Just getting the requirements 5 | var Trie = require('merkle-patricia-tree/secure'); 6 | var levelup = require('levelup'); 7 | var leveldown = require('leveldown'); 8 | var RLP = require('rlp'); 9 | var assert = require('assert'); 10 | 11 | //Connecting to the leveldb database 12 | var db = levelup(leveldown('/home/timothymccallum/gethDataDir/geth/chaindata')); 13 | 14 | 15 | //Obtaining the state root (of the latest block) using this command web3.eth.getBlock(web3.eth.defaultBlock).stateRoot 16 | //block 2 at the time 17 | //var root = '0x55b3b55cac8793a38209f410541ee93926cd58e2d49f5b82e0e4501428c1a823'; 18 | //block 13 at time time 19 | //var root = '0x73f842ee4f51b59cbcfa31d213720bae992f0679cc0f27dc9eb16f891ab08e11'; 20 | //block 23 at the time 21 | var root = '0x8c77785e3e9171715dd34117b047dffe44575c32ede59bde39fbf5dc074f2976'; 22 | //Creating a trie object of the merkle-patricia-tree library 23 | var trie = new Trie(db, root); 24 | 25 | //Creating a nodejs stream object so that we can access the data 26 | var stream = trie.createReadStream() 27 | 28 | //Turning on the stream (because the node js stream is set to pause by default) 29 | stream.on('data', function (data){ 30 | //printing out the keys of the "state trie" 31 | var arrayOfStateValues = RLP.decode(data.value); 32 | for (i = 0; i < arrayOfStateValues.length; i++){ console.log(RLP.decode(arrayOfStateValues[i], [skipRemainderCheck=false])); console.log("");} 33 | 34 | 35 | }); 36 | 37 | 38 | /* 39 | var arrayLength = arrayOfStateValues[0].length; 40 | console.log("This array has " + arrayLength + " elements"); 41 | for (var i = 0; i < arrayLength; i++) { 42 | console.log("Item " + i); 43 | console.log(arrayOfStateValues[0][i]); 44 | console.log(" "); 45 | } 46 | */ 47 | 48 | -------------------------------------------------------------------------------- /leveldb/javascript/print_transaction_trie_keys.js: -------------------------------------------------------------------------------- 1 | //Run the following first 2 | //npm install levelup leveldown rlp merkle-patricia-tree assert --save 3 | 4 | //Just getting the requirements 5 | var Trie = require('merkle-patricia-tree/secure'); 6 | var levelup = require('levelup'); 7 | var leveldown = require('leveldown'); 8 | var RLP = require('rlp'); 9 | var assert = require('assert'); 10 | console.log("begin"); 11 | 12 | //Connecting to the leveldb database 13 | var db = levelup(leveldown('/home/timothymccallum/gethDataDir/geth/chaindata')); 14 | 15 | //Obtaining the state root (of the latest block) using this command web3.eth.getBlock(web3.eth.defaultBlock).stateRoot 16 | //block 2 at the time 17 | //var root = '0x55b3b55cac8793a38209f410541ee93926cd58e2d49f5b82e0e4501428c1a823'; 18 | //block 13 at time time 19 | //var root = '0x73f842ee4f51b59cbcfa31d213720bae992f0679cc0f27dc9eb16f891ab08e11'; 20 | //block 23 at the time 21 | //block 14 which holds the transaction 22 | //var root = '0x4988f58d335e5fc3b9c35c6282b80d7053889c9991d18c287f0e5ca5050a967e'; 23 | //block 15 which has no transactions but is after the previous block which held the transaction 24 | var root = '0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421' 25 | //Creating a trie object of the merkle-patricia-tree library 26 | var trie = new Trie(db, root); 27 | 28 | //Creating a nodejs stream object so that we can access the data 29 | var stream = trie.createReadStream() 30 | 31 | //Turning on the stream (because the node js stream is set to pause by default) 32 | stream.on('data', function (data){ 33 | //printing out the keys of the "state trie" 34 | console.log("keys"); 35 | console.log(data.key); 36 | console.log("values"); 37 | console.log(data.value); 38 | }); 39 | 40 | //Activity 41 | 42 | //Created 2 accounts which generated two addresses 43 | //0x19c4b28272fc737f7d3483dd18ab176ca59fd7c8 - coinbase 44 | //0xbe4e1f1a934fbcbe6ffedc16dfba377974d3a568 [1] 45 | 46 | //mined about 2 blocks 47 | 48 | //ran this file and say 2 account keys in the state trie 49 | 50 | //Created a 3rd account which generated the following single address 51 | //0x87fb6767c16161d05ba948dd2f6f47e73e9d53c2 52 | 53 | //Mined about 13 blocks 54 | 55 | //ran this file and still only say 2 account keys in the state trie 56 | 57 | //sent eth at block 13 from coinbase to account [2] 58 | //eth.sendTransaction({from:eth.coinbase, to:eth.accounts[2], value: web3.toWei(0.05, "ether")}) 59 | //This generated the following transaction id 60 | //0x45a0ccb4363d56f6d179ea78324fbcdf57d162dfec7051a0dd86eca8305674b8 61 | 62 | //Mined 10 more blocks and sitting at block 23 63 | 64 | //ran this file again 65 | 66 | 67 | 68 | 69 | //87fb6767c16161d05ba948dd2f6f47e73e9d53c2 should have zero funds 70 | -------------------------------------------------------------------------------- /leveldb/javascript/reading_ethereum_leveldb_using_nodejs_and_patricia_trie.js: -------------------------------------------------------------------------------- 1 | //The following is an example of how to add and retrieve json data from leveldb via the nodejs high level leveldb wrapper 2 | //Firstly reference the node library 3 | /* 4 | var level = require('level') 5 | var rlp = require('rlp') 6 | var trie = require('merkle-patricia-tree'); 7 | var db = level('/home/timothymccallum/gethDataDir/geth/chaindata') 8 | 9 | 10 | //Genesis state root 11 | //Result from typing eth.getBlock(0).stateRoot 12 | var genesis_block_state_root = "0xe403429877a3fd13eb3da17503af458dd0741ec3b617f2eaf2338ca945edb0e2"; 13 | //Create new trie 14 | var trieGen = new trie(db, genesis_block_state_root); 15 | var address1 = '0x77fd1acbd74fab8dc821fe5c88598c7b4b906fd4' 16 | trieGen.get(address1, function (err, info) { 17 | console.log(info); 18 | var decoded = rlp.decode(info); 19 | console.log(decoded); 20 | }); 21 | console.log("Finished with genesis block now!") 22 | 23 | //Using latest block now 24 | console.log("Using latest block now!") 25 | var latest_block_state_root = '0x71fbdf3c0de93ec8562628cedcc22636f66e9ccd68ae30f7c7dc7a47038f7342' 26 | var trieLatest = new trie(db, latest_block_state_root); 27 | var address2 = '0xb7714bd73152a27939914c644fb7471966378626' 28 | trieLatest.get(address2, function (err, info) { 29 | console.log(info); 30 | var decoded = rlp.decode(info); 31 | console.log(decoded); 32 | }); 33 | */ 34 | 35 | var Trie = require('merkle-patricia-tree'); 36 | var rlp = require('rlp'); 37 | var level = require('level'); 38 | var db = level('/home/timothymccallum/gethDataDir/geth/lightchaindata'); 39 | 40 | //the genesis state root 41 | var root = '0xe403429877a3fd13eb3da17503af458dd0741ec3b617f2eaf2338ca945edb0e2'; 42 | var trie = new Trie(db, root); 43 | 44 | //gav's address 45 | var gav = new Buffer('0xb7714bd73152a27939914c644fb7471966378626', 'hex'); 46 | 47 | trie.get(gav, function (err, val) { 48 | var decoded = rlp.decode(val); 49 | console.log(decoded); 50 | }); 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | /* 61 | var stream = db.createReadStream() 62 | stream.on('data', function (data){ 63 | var base64data = new Buffer(genesis_block_state_root, 'binary').toString('base64'); 64 | console.log(base64data); 65 | }); 66 | 67 | 68 | var stream = db.createReadStream() 69 | stream.on('data', function (data){ 70 | var base64data = new Buffer(genesis_block_state_root, 'binary').toString('base64'); 71 | console.log(base64data); 72 | }); 73 | */ 74 | 75 | -------------------------------------------------------------------------------- /leveldb/javascript/reading_ethereum_leveldb_using_nodejs_level_key_read.js: -------------------------------------------------------------------------------- 1 | //The following is an example of how to add and retrieve json data from leveldb via the nodejs high level leveldb wrapper 2 | //Firstly reference the node library 3 | var level = require('level') 4 | 5 | var db = level('/home/timothymccallum/gethDataDir/geth/lightchaindata') 6 | 7 | var keyStream = db.createKeyStream() 8 | keyStream.on('data', function (data){ 9 | console.log("key=", data); 10 | }); 11 | 12 | -------------------------------------------------------------------------------- /leveldb/javascript/reading_ethereum_leveldb_using_nodejs_level_stream_read.js: -------------------------------------------------------------------------------- 1 | //The following is an example of how to add and retrieve json data from leveldb via the nodejs high level leveldb wrapper 2 | //Firstly reference the node library 3 | var level = require('level') 4 | var rlp = require('rlp') 5 | 6 | var db = level('/home/timothymccallum/gethDataDir/geth/chaindata') 7 | 8 | //Genesis state root 9 | //Result from typing eth.getBlock(0).stateRoot 10 | var genesis_block_state_root = "0xe403429877a3fd13eb3da17503af458dd0741ec3b617f2eaf2338ca945edb0e2"; 11 | 12 | var stream = db.createReadStream() 13 | stream.on('data', function (data){ 14 | var base64data = new Buffer(genesis_block_state_root, 'binary').toString('base64'); 15 | console.log(base64data); 16 | }); 17 | 18 | 19 | var stream = db.createReadStream() 20 | stream.on('data', function (data){ 21 | var base64data = new Buffer(genesis_block_state_root, 'binary').toString('base64'); 22 | console.log(base64data); 23 | }); 24 | 25 | 26 | -------------------------------------------------------------------------------- /leveldb/javascript/reading_leveldb_using_nodejs_level_key_read.js: -------------------------------------------------------------------------------- 1 | //The following is an example of how to add and retrieve json data from leveldb via the nodejs high level leveldb wrapper 2 | //Firstly reference the node library 3 | var level = require('level') 4 | 5 | var db = level('./tpmccallum') 6 | 7 | var keyStream = db.createKeyStream() 8 | keyStream.on('data', function (data){ 9 | console.log("key=", data); 10 | }); -------------------------------------------------------------------------------- /leveldb/javascript/reading_leveldb_using_nodejs_level_stream_read.js: -------------------------------------------------------------------------------- 1 | //The following is an example of how to add and retrieve json data from leveldb via the nodejs high level leveldb wrapper 2 | //Firstly reference the node library 3 | var level = require('level') 4 | 5 | var db = level('./tpmccallum') 6 | 7 | var stream = db.createReadStream() 8 | stream.on('data', function (data){ 9 | console.log("key=", data.key + ", value=" + data.value); 10 | }); 11 | 12 | 13 | -------------------------------------------------------------------------------- /leveldb/javascript/reading_leveldb_using_nodejs_level_value_read.js: -------------------------------------------------------------------------------- 1 | //The following is an example of how to add and retrieve json data from leveldb via the nodejs high level leveldb wrapper 2 | //Firstly reference the node library 3 | var level = require('level') 4 | 5 | var db = level('./tpmccallum') 6 | 7 | var valueStream = db.createValueStream() 8 | valueStream.on('data', function (data){ 9 | console.log("value=", data); 10 | }); -------------------------------------------------------------------------------- /leveldb/javascript/testing_leveldb_batch_processing_using_nodejs_level.js: -------------------------------------------------------------------------------- 1 | //The following is an example of how to add and retrieve json data from leveldb via the nodejs high level leveldb wrapper 2 | //Firstly reference the node library 3 | var level = require('level') 4 | //create database and specify encoding as json 5 | var db = level('./tpmccallum', {valueEncoding: 'json'}) 6 | 7 | //Testing leveldb batch processing by passing in a pre written array variable called operations 8 | console.log("Trying batch processing now ..."); 9 | console.log("Creating array"); 10 | var operations = [ 11 | {type:'put', key:'dataKeyOps1', value:'dataValueOps1'}, 12 | {type:'put', key:'dataKeyOps2', value:'dataValueOps2'}, 13 | {type:'put', key:'dataKeyOps3', value:'dataValueOps3'}, 14 | {type:'put', key:'dataKeyOps4', value:'dataValueOps4'} 15 | ] 16 | console.log("Array created OK"); 17 | console.log("Performing batch operation by passing in the array") 18 | db.batch(operations, function(err){ 19 | if(err != null){ 20 | console.log("Error" + err); 21 | } 22 | else{ 23 | console.log("Data added successfully, OK!"); 24 | } 25 | 26 | }) 27 | 28 | //This outputs 29 | //key= dataKeyOps1, value="dataValueOps1" 30 | //key= dataKeyOps2, value="dataValueOps2" 31 | //key= dataKeyOps3, value="dataValueOps3" 32 | //key= dataKeyOps4, value="dataValueOps4" 33 | -------------------------------------------------------------------------------- /leveldb/javascript/testing_leveldb_json_processing_using_nodejs_level.js: -------------------------------------------------------------------------------- 1 | //The following is an example of how to add and retrieve json data from leveldb via the nodejs high level leveldb wrapper 2 | //Firstly reference the node library 3 | var level = require('level') 4 | //create database and specify encoding as json 5 | var db = level('./tpmccallum', {valueEncoding: 'json'}) 6 | 7 | //Testing leveldb batch processing by passing in a pre written array variable called operations 8 | console.log("Trying batch processing now ..."); 9 | 10 | var complexJson = { 11 | firstName: "John", 12 | lastName: "Smith", 13 | pets: ["dog", "cat"], 14 | address: {number: "123", street:"main street", city:"new york"} 15 | } 16 | 17 | db.put('complexJson', complexJson, function(err){ 18 | if(err != null){ 19 | console.log(err); 20 | } 21 | else 22 | { 23 | console.log("Data has been added"); 24 | } 25 | 26 | }) 27 | 28 | //This outputs 29 | //key= complexJson, value={"firstName":"John","lastName":"Smith","pets":["dog","cat"],"address":{"number":"123","street":"main street","city":"new york"}} 30 | -------------------------------------------------------------------------------- /leveldb/javascript/testing_leveldb_using_nodejs_level.js: -------------------------------------------------------------------------------- 1 | //The following is an example of how to add and retrieve json data from leveldb via the nodejs high level leveldb wrapper 2 | //Firstly reference the node library 3 | var level = require('level') 4 | //create database and specify encoding as json 5 | var db = level('./tpmccallum', {valueEncoding: 'json'}) 6 | //just a reusable variable 7 | var quote = '"' 8 | console.log("About to add the data"); 9 | //put data into the database 10 | db.put('dataKey', {name:'jsonData', type:'json'}, function(err) { 11 | console.log("Data added OK!"); 12 | console.log("About to retrieve the data"); 13 | //get data from the database 14 | db.get('dataKey', function(err, dataKey){ 15 | console.log("The data is " + quote + dataKey.type + quote); 16 | console.log("Data retrieved OK!"); 17 | }) 18 | }) 19 | 20 | -------------------------------------------------------------------------------- /leveldb/leveldb.md: -------------------------------------------------------------------------------- 1 | # The Ethereum world - how its data is stored 2 | 3 | The aim of this post is to provide as much information, as possible, in relation to how Ethereum stores its transactions, contracts, account balances and more. 4 | 5 | In this post we: 6 | - begin with an overview of what blockchain "state" means 7 | - provide an overview of the popular blockchain database software, leveldb 8 | - introduce the Patricia Trie data structure which Ethereum uses exclusively 9 | - learn how Ethereum implements the Patricia Trie data structure inside leveldb 10 | - learn about Ethereum's leveldb structure 11 | - provide a step by step guide on installing a test-bed for Ethereum and leveldb (using Ubuntu Linux 16.04LTS) 12 | - provide justification for using an Ethereum private network for the test-bed (as apposed to the main-net or testnets) 13 | - configure and run Ethereum on a private network 14 | - configure Ethereum to mine on the private network 15 | - execute transactions and smart contracts on the Ethereum private network 16 | - explore Ethereum's application data storage layer, leveldb, before and after activity to discover how it works 17 | 18 | # Blockchain "state" 19 | Ethereum is a transaction-based "state" machine; a technology on which all transaction based state machine concepts may be built [1]. The Ethereum blockchain begins life with a genesis state. From the genesis block onward, activities such as transactions, contracts and mining continually change the state of the Ethereum blockchain. In Ethereum, an example of this would be an account balance (stored in the state trie) changing when a transaction, in relation to that account, takes place. 20 | 21 | The state is not stored in the blocks of the Ethereum blockchain. The blocks function as a journal; a record of transactions. One of the key features of the blockchain is immutability. Once blocks are mined they are never updated. It makes sense then, that permanent data like mined transactions would be stored separately from data like ephemeral account balances. We will learn more about this when we cover Ethereum's data structure. 22 | 23 | # Ethereum's blockchain databases 24 | 25 | The main Ethereum clients use two different database software solutions to store their data. Ethereum's Rust client Parity uses rocksdb. Whereas Ethereum's Go, C++ and Python clients all use leveldb. 26 | 27 | ## Ethereum and rocksdb 28 | 29 | Rocksdb is out of scope for this post. This will be covered at a later date. For now let's explore how 3 of the 4 major Ethereum clients utilise leveldb. 30 | 31 | ## Ethereum and leveldb 32 | 33 | LevelDB is an open source Google key-value storage library which provides, amongst other things, forward and backward iterations over data, ordered mapping from string keys to string values, custom comparison functions and automatic compression. The data is automatically compressed using “Snappy” an open source Google compression/decompression library. Whilst Snappy does not aim for maximum compression, it aims for very high speeds. Leveldb is an important storage and retrieval mechanism which manages the state of the Ethereum network. As such, leveldb is a dependency for the most popular Ethereum clients (nodes) such as go-ethereum, cpp-ethereum and pyethereum. 34 | 35 | So how does Ethereum utilise leveldb? 36 | 37 | # The Trie (or Tree) 38 | A trie (or tree) is a data structure. Whilst the implementation of the trie data structure can be done on disk (using database software such as leveldb) it is important to note that there is a difference between traversing a trie and simply looking at the flat key/value database. 39 | 40 | As a general example, if we were wanting to traverse a trie for the word dog, we might start at the root node of the tree and find the first character of the hexadecimal representation of "d" for dog. The letter "d" has a hexadecimal representation of 64, and as such we would start at the index of "6" and continue to follow the path downwards by progressing to index 4 in the next node and so on. In a general sense this is quite inefficient because we might find our path being extremely deep, when storing longer strings. To address this Ethereum specifically use a "Practical Algorithm To Retrieve Information Coded In Alphanumeric" (PATRICIA) Trie as apposed to say a radix trie. All of the merkle tries in Ethereum use a Merkle Patricia Trie. 41 | 42 | ## Merkle Patricia Trie 43 | 44 | Every interaction between Ethereum and its database involves a deterministic hash. In other words every put, update and delete function performed on the trie is done so, using a deterministic cryptographic hash. Further, the cryptographic hash of the node is used as the unique pointer of each node in Ethereum's database. This of course works due to the fact that a deterministic hash (a one-way has like sha3) provides a consistent hash in relation to the data which was used to generate the hash. 45 | 46 | The use of hashing also provides cryptographic data integrity; any changes to the data, at any level, would change the root hash. This provides a fast data integrity mechanism. 47 | 48 | In Ethereum, a single merkle Patricia trie node is either: 49 | - an empty string (referred to as NULL) 50 | - an array which contains 17 items (referred to as a branch) 51 | - an array which contains 2 items (referred to as a leaf) 52 | - an array which contains 2 items (referred to as an extension) 53 | 54 | # Ethereum's database layout 55 | 56 | ## State trie - the one and only 57 | 58 | There is one, and one only, global state trie in Ethereum. The state trie's root node can be used as a secure and unique identity for the entire state trie; the state trie's root node is cryptographically dependent on all internal state trie data. A "value" in the global state trie is created by encoding the following account details of an Ethereum account (using the Recursive-Length Prefix encoding (RLP) method): 59 | - nonce 60 | - balance 61 | - storageRoot 62 | - codeHash 63 | 64 | The global state trie is constantly updated. 65 | 66 | ## Storage trie - where the contract data lives 67 | 68 | A storage trie is where all of the contract data lives. Each Ethereum account has its own storage trie. 69 | 70 | ## Transaction trie - one per block 71 | 72 | Each Ethereum block has its own separate transaction trie. A block contains many transactions. The order of the transactions in a block are of course decided by the miner who assembles the block. The path to a specific transaction in the transaction trie, is via (the RLP encoding of) the index of where the transaction sits in the block. Mined blocks are never updated; the position of the transaction in a block is never changed. This means that once you locate a transaction in a block's transaction trie, you can return to the same path over and over to retrieve the same result. 73 | 74 | # Installing Ethereum 75 | 76 | We are going to install Ethereum on a private network, create accounts, perform transactions and then analyse the application data layer in order to improve our understand of how Ethereum works. We will be doing this on a Linux Ubuntu 16.04 LTS installation. 77 | 78 | ## Housekeeping 79 | 80 | Type the following commands to update the system as best we can. 81 | 82 | ` 83 | sudo apt-get update 84 | ` 85 | 86 | ` 87 | sudo apt-get upgrade 88 | ` 89 | 90 | ` 91 | sudo apt-get install -y build-essential 92 | ` 93 | 94 | ## Installing Go 95 | 96 | To get the Go package type the following. 97 | 98 | ` 99 | cd ~ 100 | ` 101 | 102 | ` 103 | wget https://dl.google.com/go/go1.9.2.linux-amd64.tar.gz 104 | ` 105 | 106 | Then verify the download by checking that the sha256sum results from the following command match that of the relevant sha256 checksum section on the < https://golang.org/dl/ > website. 107 | 108 | ` 109 | sha256sum go1.9.2.linux-amd64.tar.gz 110 | ` 111 | 112 | To unpack the package, type the following (using sudo). 113 | 114 | ` 115 | sudo tar -C /usr/local -xzf go1.9.2.linux-amd64.tar.gz 116 | ` 117 | 118 | To ensure that the Go path is set for your user profile. Add the following line to the end of your ~/.profile file. 119 | 120 | ` 121 | export PATH="$PATH:/usr/local/go/bin" 122 | ` 123 | 124 | Log out and in (or reboot) to ensure the environment variables have taken. 125 | Test Go with the following command (confirm the version). 126 | 127 | ` 128 | go version 129 | ` 130 | 131 | The output should show the version. 132 | 133 | ` 134 | go version go1.9.2 linux/amd64 135 | ` 136 | 137 | ## Installing Git 138 | 139 | ` 140 | sudo apt-get install git 141 | ` 142 | 143 | ## Fetching Geth 144 | 145 | Change to home directory. 146 | 147 | ` 148 | cd ~ 149 | ` 150 | 151 | Fetch the Geth software. 152 | 153 | ` 154 | git clone https://github.com/ethereum/go-ethereum.git 155 | ` 156 | 157 | ### Mining difficulty 158 | 159 | As this will be a private network, we do not want the Ethereum installation to constantly increase the difficulty setting. We are just performing some testing, in a closed and private network, and as such want the quickest mining time available. 160 | 161 | To set the difficulty at 1 statically, we need to open the following file in the source code (before compiling). 162 | 163 | ` 164 | go-ethereum/consensus/ethash/consensus.go 165 | ` 166 | 167 | Once we have this file open for editing, we need to go to the calcDifficulty function (currently on line 298). 168 | 169 | ` 170 | https://github.com/ethereum/go-ethereum/blob/02aeb3d76652a4c0451e5c3734e6881aefe46249/consensus/ethash/consensus.go#L298 171 | ` 172 | 173 | This function controls which difficulty adjustment function is called. 174 | 175 | ![default Ethereum consensus file](./images/consensus_before.png) 176 | 177 | In order to guarantee the lowest difficulty level (on an ongoing basis), we will need to strip out the case statement and simply return big.NewInt(1) for the entire function's execution. 178 | 179 | ` 180 | return big.NewInt(1) 181 | ` 182 | 183 | ![modified Ethereum consensus file](./images/consensus_after.png) 184 | 185 | 186 | **NOTE:** Additional configuration and commands are required in relation to accounts, networks and configuration. Please hold off running any commands until reading the next few paragraphs. 187 | 188 | # Explaining Ethereum networks 189 | 190 | ## Main network 191 | 192 | For peer-to-peer nodes to interact on the same network they have to have the identical protocol version and the right network ID. In order for peer nodes to connect to the main Ethereum network, nothing extra is required; simply starting the Ethereum software will suffice. From an experimental, testing perspective there are two main drawbacks to using the main Ethereum network. Firstly, of course starting Ethereum with the default network settings will result in your peer node downloading the entire Ethereum blockchain. This takes a significant amount of time and requires a lot of storage on your local disk. Secondly any activity on the main Ethereum network requires “gas”. Whilst this gas is obtainable, it involves purchasing and depositing ETH into your account. You may not want to do this just for experimental testing. 193 | 194 | ## Test network 195 | 196 | There are other ways to start your Ethereum node. You may want to use an Ethereum test network like Ropsten as this will provide the flexibility to participate on the network without purchasing ETH. Testnet ETH can be obtained without having to pay real money. This solves one problem, however using a testnet like Ropsten also requires you to download the Ropsten testnet blockchain. This again, takes time and requires local disk space. 197 | 198 | ## Private network 199 | 200 | One final alternative is to run a private network. A private network requires negligible local disk storage and can be operated without having to purchase ETH with real money. So what constitutes a private Ethereum network? 201 | 202 | “An Ethereum network is a private network if the nodes are not connected to the main network nodes. In this context private only means reserved or isolated, rather than protected or secure.”[9] 203 | 204 | As mentioned on the Ethereum GitHub page a private network is neither protected or secure. In addition, setting up a private network requires a little bit of extra configuration. You can alleviate the security aspect by simply running the private Ethereum network behind a properly configured firewall. With security aside, all that is left if to perform is the extra configuration. 205 | 206 | # Running a private Ethereum network 207 | 208 | The following paragraphs will help with configuring and running the Ethereum private network. 209 | 210 | ## Network ID 211 | 212 | The network ID for the main Ethereum network is 1. If started in default mode, the code will automatically connect to the main Ethereum network by default. 213 | 214 | Alternatively, the network ID can be passed into the command line when starting Go Ethereum code, using the following flag. 215 | ` 216 | -- networkid 217 | ` 218 | 219 | Obviously, in order to keep other nodes from connecting to your private Ethereum test network it would be in your best interests to a) secure the network using a properly configured firewall and b) choose a unique network ID 220 | 221 | ## Account 222 | 223 | In a moment we will be using a once off mechanism to fund our account. Before we use that mechanism we need to create create a data directory for Ethereum, create our new accounts and save those account details for use on future steps. 224 | Create an Ethereum data directory by typing the following commands. 225 | 226 | ` 227 | cd ~ 228 | ` 229 | 230 | ` 231 | mkdir gethDataDir 232 | ` 233 | 234 | Then create a new account by typing the following command. 235 | 236 | ` 237 | geth account new --datadir ~/gethDataDir 238 | ` 239 | 240 | The output from the above command will produce a public account address (after you provide a password). Record and save this address for later use. 241 | 242 | ``` 243 | Your new account is locked with a password. Please give a password. Do not forget this password. 244 | Passphrase: 245 | Repeat passphrase: 246 | Address: {77fd1acbd74fab8dc821fe5c88598c7b4b906fd4} 247 | ``` 248 | 249 | **It is recommended that you do this twice, that way you will have to addresses which you can send ETH back and forth to** 250 | 251 | ## Genesis block 252 | 253 | When running Ethereum for the first time, if the default settings are used, the blockchain will start at the "hard coded" genesis block (first block in the public main net blockchain). From this point onward the code will find peers and synchronise until the Ethereum instance which you are running is up to date. Being synchronised, or up to date, means that you are storing everything from the genesis block, right up, to the most recent block, locally. Ethereum's blockchain is almost 45GB in size and so for our purpose of testing this is not desirable. 254 | 255 | Instead, we can create our own genesis block. This must be done before starting Ethereum. 256 | 257 | ## Create genesis.json 258 | 259 | Create a file such as below, and save it with the name "genesis.json" into the ~/gethDataDir directory. 260 | 261 | **Note** the "alloc" section where we paste in the two account addresses from above. 262 | **Note** the "chainId" section which is set to 15 (and not 1). If you remember, above in the **Network ID** section, 1 is the main Ethereum network. 263 | 264 | ``` 265 | { 266 | "config": { 267 | "chainId": 15, 268 | "homesteadBlock": 0, 269 | "eip155Block": 0, 270 | "eip158Block": 0 271 | }, 272 | "difficulty": "1", 273 | "gasLimit": "2100000", 274 | "alloc": { 275 | "77fd1acbd74fab8dc821fe5c88598c7b4b906fd4": { "balance": "300000" }, 276 | "b7714bd73152a27939914c644fb7471966378626": { "balance": "400000" } 277 | } 278 | } 279 | ``` 280 | 281 | ## Initialize the blockchain 282 | 283 | Type the following initialisation command, which references the genesis and its settings. 284 | 285 | ` 286 | geth --datadir ~/gethDataDir init genesis.json 287 | ` 288 | 289 | ## Starting Ethereum 290 | 291 | Type the following command to start Ethereum. Please note that the networkid is the same as the "chainId" in the genesis.json file 292 | 293 | ` 294 | geth --datadir ~/gethDataDir --networkid 15 295 | ` 296 | 297 | ## Starting the console 298 | 299 | If we open a new terminal and type the following command, we can then use the Geth Javascript API to interact with Ethereum. 300 | 301 | ` 302 | geth attach ipc:gethDataDir/geth.ipc 303 | ` 304 | 305 | ![console picture](./images/console.png) 306 | 307 | ## Using the console 308 | 309 | The following command will list the Ethereum accounts which we recently created. 310 | 311 | ` 312 | eth.accounts 313 | ` 314 | 315 | ![console picture](./images/accounts.png) 316 | 317 | The following commands will list details about your: 318 | - account balances 319 | - coinbase address (where the mining rewards are sent) 320 | - mining status (true/false) 321 | 322 | ![console picture](./images/console_status.png) 323 | 324 | # Mining the Ethereum private network 325 | 326 | To mine the Ethereum private network, simply type the following command. Please note the specific parameters which we provide such as datadir and networkid. These are to ensure the mining takes place on the correct blockchain/network. 327 | 328 | ` 329 | geth --mine --minerthreads=1 --datadir ~/gethDataDir --networkid 15 330 | ` 331 | 332 | # Transactions and smart contracts 333 | 334 | Transactions and smart contracts soon. 335 | 336 | # Analysing the Ethereum database 337 | 338 | 339 | ## Web3 340 | As we mentioned previously there are many Merkle Patricia Tries (referenced in **each** block) within the Ethereum blockchain: 341 | 342 | - State Trie 343 | - Storage Trie 344 | - Transaction Trie 345 | - Receipts Trie 346 | 347 | To reference a particular Merkle Patricia Trie in a particular block we need to obtain its root hash, as a reference. The following commands are an example of how to obtain the root hashes of the state, transaction and receipt tries in the genesis block (block 0). 348 | 349 | ` 350 | web3.eth.getBlock(0).stateRoot 351 | ` 352 | 353 | ` 354 | web3.eth.getBlock(0).transactionsRoot 355 | ` 356 | 357 | ` 358 | web3.eth.getBlock(0).receiptsRoot 359 | ` 360 | 361 | For example 362 | ![block key value image](./images/block_root_key_values.png) 363 | 364 | **Note:** If you would like the root hashes of the **latest** block (instead of the genesis block), please use the following command. 365 | 366 | ` 367 | web3.eth.getBlock(web3.eth.blockNumber).stateRoot 368 | ` 369 | 370 | 371 | ## Installing npm, node and level 372 | 373 | We will be using nodejs and level to inspect the leveldb database. 374 | 375 | With reference to the following site https://github.com/nodesource/distributions to install nodejs and npm please use the following commands. 376 | 377 | ` 378 | curl -sL https://deb.nodesource.com/setup_9.x | sudo -E bash - 379 | sudo apt-get install -y nodejs 380 | ` 381 | 382 | ` 383 | sudo apt-get install nodejs 384 | ` 385 | 386 | You can verify the versions by typing the following commands. 387 | 388 | ` 389 | npm -v 390 | ` 391 | 392 | For nodejs 393 | 394 | ` 395 | nodejs -v 396 | ` 397 | 398 | To install level and other requirements, please use the following command. 399 | 400 | ` 401 | npm install levelup leveldown rlp merkle-patricia-tree assert --save 402 | ` 403 | 404 | The nodejs console can be started by typing. 405 | 406 | ` 407 | nodejs 408 | ` 409 | 410 | The javascript files in this repository can be run using the following syntax. 411 | 412 | ` 413 | nodejs name_of_file.js 414 | ` 415 | 416 | # Ethereum's database logic 417 | 418 | ## State Trie 419 | 420 | The state trie contains a key and value pair for every account which exists on the Ethereum network. The "key" is a single 160 bit identifier (the address of an Ethereum account). The "value", is a serialised data structure (containing the nonce, balance, storageRoot and codeHash) of an Ethereum account. 421 | 422 | A Keccak-256-bit hash of state trie's root node (a fixed length hash of the entire Ethereum state) is stored in the block header, under the value of "stateRoot". Storing only a single root hash in the block (as opposed to storing the entire list of all accounts, balances and so forth) provides two enormous benefits. Firstly, it keeps the size of the blocks in the blockchain down. Secondly, capturing the cryptographic hash of the state trie at every block height provides an opportunity for light clients to quickly join and trust the network. Light clients like Internet of Things (IoT) devices and mobile phone payment applications are not able to store the entire Ethereum blockchain and all of its databases (like a full node would do). Light client devices do not have the storage, memory or computing power to do much else than perform a quick check against a state trie root hash followed by a quick validity request to a full Ethereum node. 423 | 424 | ![state trie diagram](./images/state_trie_diagram.png) 425 | 426 | 427 | 428 | You can print a list of the Etherem account keys from the state root by executing the following script. 429 | 430 | ``` 431 | //Just getting the requirements 432 | var Trie = require('merkle-patricia-tree/secure'); 433 | var levelup = require('levelup'); 434 | var leveldown = require('leveldown'); 435 | var RLP = require('rlp'); 436 | var assert = require('assert'); 437 | 438 | //Connecting to the leveldb database 439 | var db = levelup(leveldown('/home/timothymccallum/gethDataDir/geth/chaindata')); 440 | 441 | //Adding the "stateRoot" value from the block so that we can inspect the state root at that block height. 442 | var root = '0x8c77785e3e9171715dd34117b047dffe44575c32ede59bde39fbf5dc074f2976'; 443 | 444 | //Creating a trie object of the merkle-patricia-tree library 445 | var trie = new Trie(db, root); 446 | 447 | //Creating a nodejs stream object so that we can access the data 448 | var stream = trie.createReadStream() 449 | 450 | //Turning on the stream (because the node js stream is set to pause by default) 451 | stream.on('data', function (data){ 452 | //printing out the keys of the "state trie" 453 | console.log(data.key); 454 | }); 455 | ``` 456 | 457 | ![List of Ethereum accounts in the state trie](./images/three_accounts.png) 458 | 459 | Interestingly, accounts in Ethereum are only added to the state trie once a transaction has taken place (in relation to that specific account). For example, just creating a new account using "geth account new" will not include that account in the state trie; even after many blocks have been mined. 460 | 461 | However, if a successful transaction (one which is included in a mined block) is recorded against that account, then that account will then appear in the state trie. This is clever logic which protects against a malicious attacker continously create new accounts (in the command line at no cost to the attacker). Such an activity would bloat the state trie. 462 | 463 | # References 464 | [1] Wood, G., 2014. Ethereum: A secure decentralised generalised transaction ledger. Ethereum Project Yellow Paper, 151. 465 | -------------------------------------------------------------------------------- /leveldb/scratch_pad.md: -------------------------------------------------------------------------------- 1 | # draft notes put aside which may or not be useful in the long run 2 | 3 | Please note, Ethereum must not be running when trying to access the database using nodejs. There will be a lock in place which will stop you from accessing the data. 4 | 5 | The following code creates a read stream and allows us to retrieve all key/value pairs from the leveldb database 6 | 7 | ![read stream code example](./images/ethereum_read_stream.png) 8 | 9 | As we mentioned above Ethereum uses RLP to encode the values. This is evident in the output which we just received. 10 | 11 | ![read stream code example](./images/ethereum_read_stream_first_output.png) 12 | 13 | We are now going to provide node references and then traverse the paths of the **state** trie, **storage** trie and **transaction** trie in order to obtain the **state** data, **contract** data and **transaction** data respectively. 14 | 15 | From this point, we can make changes to all of these areas and re-inspect the data. 16 | --------------------------------------------------------------------------------