├── .github ├── ISSUE_TEMPLATE │ ├── blog-post-topic-review.md │ ├── bug_report.md │ ├── feature_request.md │ ├── plain--blank-issue.md │ └── plain-issue.md └── workflows │ └── hugo-deploy.yml ├── CONTRIBUTING.md ├── LICENSE ├── README.md ├── archetypes ├── blog.md └── default.md ├── config.toml ├── content ├── _index.md └── blog │ ├── bip158-deep-dive.md │ ├── bitcoin-core-usdt-support.md │ ├── example-post.md │ └── schnorr-basics.md ├── layouts ├── _default │ ├── baseof.html │ ├── list.html │ ├── rss.xml │ └── single.html ├── index.html ├── partials │ ├── card-small.html │ ├── card.html │ ├── footer.html │ ├── head.html │ ├── mathjax_support.html │ ├── navbar.html │ └── prev-next.html ├── shortcodes │ └── center-figure.html └── tags │ └── list.html ├── static ├── CNAME ├── bitcoin.pdf ├── css │ ├── bootstrap.min.css │ ├── index.css │ └── syntax.css ├── favicon.png ├── img │ ├── footer │ │ ├── github.svg │ │ ├── gmail.svg │ │ ├── rss.svg │ │ └── twitter.svg │ ├── logo-512.png │ ├── logo-large.png │ └── og-image.png ├── post-data │ ├── compact-block-filters │ │ ├── bitcoinExample.png │ │ ├── cover.png │ │ ├── dense.jpeg │ │ ├── quotient.png │ │ ├── remainder.png │ │ └── sparse.png │ ├── example-post │ │ ├── header.png │ │ └── header.svg │ ├── schnorr-basics │ │ ├── sig-overview.png │ │ ├── sig-overview.svg │ │ └── xpub-derivation.svg │ └── usdt-support-core │ │ └── header.png └── robots.txt └── svg-source ├── logo.svg └── og-image.svg /.github/ISSUE_TEMPLATE/blog-post-topic-review.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Blog post topic review 3 | about: Get feedback on your blog post topic. Does it fit for this blog? 4 | title: "[Topic Review]" 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | ## Summary 11 | 12 | Summarize your blog post in three to five sentences. 13 | 14 | ## Outline 15 | 16 | Provide a rough outline of the outline of your blog post. 17 | 18 | ## Links and other material 19 | 20 | Provide links to your project, your projects documentation, or other material that helps a reviewer. 21 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: "[bug]" 5 | labels: bug 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. Go to '...' 16 | 2. Click on '....' 17 | 3. Scroll down to '....' 18 | 4. See error 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Screenshots** 24 | If applicable, add screenshots to help explain your problem. 25 | 26 | **Desktop (please complete the following information):** 27 | - OS: [e.g. iOS] 28 | - Browser [e.g. chrome, safari] 29 | - Version [e.g. 22] 30 | 31 | **Smartphone (please complete the following information):** 32 | - Device: [e.g. iPhone6] 33 | - OS: [e.g. iOS8.1] 34 | - Browser [e.g. stock browser, safari] 35 | - Version [e.g. 22] 36 | 37 | **Additional context** 38 | Add any other context about the problem here. 39 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. 21 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/plain--blank-issue.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Plain, blank issue 3 | about: Just a plain and blank issue. 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | 11 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/plain-issue.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Plain Issue 3 | about: '' 4 | title: '' 5 | labels: '' 6 | assignees: '' 7 | 8 | --- 9 | 10 | 11 | -------------------------------------------------------------------------------- /.github/workflows/hugo-deploy.yml: -------------------------------------------------------------------------------- 1 | name: GitHub Pages 2 | 3 | on: 4 | push: 5 | branches: [ main ] 6 | 7 | jobs: 8 | build-deploy: 9 | runs-on: ubuntu-20.04 10 | steps: 11 | - uses: actions/checkout@v1 12 | 13 | - name: Setup Hugo 14 | uses: peaceiris/actions-hugo@v2 15 | with: 16 | hugo-version: '0.88.1' 17 | 18 | - name: Build site 19 | run: hugo --minify 20 | 21 | - name: Deploy site to GH pages 22 | uses: peaceiris/actions-gh-pages@v2 23 | env: 24 | ACTIONS_DEPLOY_KEY: ${{ secrets.ACTIONS_DEPLOY_KEY }} 25 | PUBLISH_BRANCH: gh-pages 26 | PUBLISH_DIR: ./public 27 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | 2 | # Contributing to the bitcoin-dev blog 3 | 4 | 1. Find a technical Bitcoin topic you want to write about 5 | - This can, for example, be about a Bitcoin open-source project you work on, or a technical topic you've recently dove into. 6 | - If you are not sure if the topic of your post is suitable for this blog, you can get feedback by opening a [topic-review issue]. 7 | 2. Write the post (or crosspost it from your own blog!) 8 | - The contents of a post lives under `content/blog/.md`. 9 | - These files contain a few lines of YAML front matter with, for example, the title and author information. 10 | - If you have [`hugo`] installed on your system, you can use `hugo new blog/blog-post-title.md` to create a new markdown file from a [template]. 11 | - If you don't have [`hugo`] installed on your system, you can copy an existing blog post, delete the content, and edit the front matter. 12 | - There is an example blog post showing how to format text and how to embed code, images, and videos. The post is not listed, but can be found [here][example-post] and the [source here][example-source]. 13 | 3. We strive for high-quality blog posts. 14 | - The post doesn't have to be perfect but should contain very few formatting, spelling, grammar, or technical mistakes. Try asking other people to review your content first. 15 | - Your post should arouse interest in the project or topic. 16 | - We recommended to self-advertise. Include links to your project for interested developers to learn more and to reach out. 17 | - Keep your blog post roughly between two and six pages. Feel free to split a post up into multiple, independent parts. 18 | 4. We follow a few strict rules. 19 | - Price-talk, altcoins, and politics are off-topic. This is a technical, bitcoin-only blog. 20 | - Posts you submit should be your original content. You must own the copyright or have permission from the owner to use the material. 21 | - Posts (text and images) are published under an [Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)] license once merged. 22 | - Cross-posting from your blog is welcome. Feel free to link to the site where the post first appeared. 23 | 5. Open a PR to the [dev-bitcoin/blog repo] and wait for review. 24 | 25 | 26 | 27 | [topic-review issue]: https://github.com/dev-bitcoin/blog/issues/new?template=blog-post-topic-review.md&title=%5BTopic+Review%5D 28 | [`hugo`]: https://gohugo.io/ 29 | [template]: https://github.com/dev-bitcoin/blog/blob/main/archetypes/blog.md 30 | [example-post]: https://bitcoin-dev.blog/blog/example-post/ 31 | [example-source]: https://github.com/dev-bitcoin/blog/blob/main/content/blog/example-post.md?plain=1 32 | [dev-bitcoin/blog repo]: https://github.com/dev-bitcoin/blog 33 | 34 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2021 bitcoin-dev 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # bitcoin-dev blog 2 | 3 | 4 | ------------------------- 5 | ### Run Google Colab 6 | 7 | https://colab.research.google.com/drive/1OShIMVcFZ_khsUIBOIV1lzrqAGo1gfm_?usp=sharing 8 | 9 | ------------------------- 10 | 11 | 12 | 13 | 14 | Technical and open-content blog posts by Bitcoin developers. 15 | 16 | To learn more about submitting a blog post, see [CONTRIBUTING.md](CONTRIBUTING.md). 17 | 18 | 19 | ---- 20 | 21 | | | Donation Address | 22 | | --- | --- | 23 | | ♥ __BTC__ | 1Lw2kh9WzCActXSGHxyypGLkqQZfxDpw8v | 24 | | ♥ __ETH__ | 0xaBd66CF90898517573f19184b3297d651f7b90bf | 25 | -------------------------------------------------------------------------------- /archetypes/blog.md: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | title: "{{ replace .Name "-" " " | title }}" 4 | 5 | # Subtitle (optional) 6 | subtitle: "" 7 | 8 | # Date of the blog post. Keep in mind that you won't be able to see future blog post without the "--buildFuture" option in Hugo. 9 | date: {{ .Date }} 10 | 11 | # Blog post authors 12 | authors: 13 | - name: "" # required! 14 | github: "" # optional 15 | twitter: "" # optional 16 | # co-author: 17 | #- name: "" # required! 18 | # github: "" # optional 19 | # twitter: "" # optional 20 | 21 | # Credit where credit is due. You are encouraged to link to, e.g., your own blog 22 | # where you first published the post. This is optional though. 23 | appearedfirston: # optional 24 | label: "" # domain of your blog. E.g.: "bitcoin-dev.blog". Without "https://". 25 | url: "" # full url to your blog post. E.g.: "https://bitcoin-dev.blog/blog/example-post/" 26 | 27 | tags: 28 | #- "Bitcoin Core" 29 | #- "C++" 30 | #- "Rust" 31 | #- "GUI" 32 | 33 | # Header image (required!) 34 | # Open-Graph image (og:image) that will be displayed in e.g. the twitter card. 35 | # Also shown on the main page where blog posts are listed. 36 | # Create a directory for your images in the "static/post-data/" folder with 37 | # the title of your blog post. 38 | images: 39 | - /post-data//header.png 40 | 41 | # MathJax support 42 | # Set this to true if you plan to use MathJax for LaTeX formulars in your blog post. 43 | mathjax: false 44 | 45 | --- 46 | 47 | 48 | FIXME: A short summary of this post before the 'more'-tag. 49 | Keep it below 100 words. 50 | 51 | 52 | 53 | FIXME: Your blog post. 54 | See the example post for e.g. ways to include tweets, videos, code, and more. 55 | 56 | -------------------------------------------------------------------------------- /archetypes/default.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "{{ replace .Name "-" " " | title }}" 3 | date: {{ .Date }} 4 | draft: true 5 | --- 6 | 7 | -------------------------------------------------------------------------------- /config.toml: -------------------------------------------------------------------------------- 1 | baseURL = "https://bitcoin-dev.blog" 2 | languageCode = "en-us" 3 | title = "bitcoin-dev blog" 4 | 5 | [params] 6 | images = [ "/img/og-image.png" ] 7 | description = "Bitcoin developers blogging about their open-source projects. A technical open-content blog." 8 | 9 | [taxonomies] 10 | tag = 'tags' 11 | 12 | [Social] 13 | twitter = "bitcoindevblog" 14 | 15 | [outputFormats] 16 | [outputFormats.RSS] 17 | mediatype = "application/rss" 18 | baseName = "feed" 19 | 20 | [privacy] 21 | [privacy.disqus] 22 | disable = true 23 | [privacy.googleAnalytics] 24 | disable = true 25 | [privacy.instagram] 26 | disable = true 27 | 28 | [privacy.twitter] 29 | # Do-Not-Track 30 | enableDNT = true 31 | 32 | [privacy.vimeo] 33 | # Do-Not-Track 34 | enableDNT = true 35 | 36 | [privacy.youtube] 37 | # YouTube won’t store information about visitors on your website unless the user plays the embedded video. 38 | privacyEnhanced = true 39 | 40 | -------------------------------------------------------------------------------- /content/_index.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "bitcoin-dev blog" 3 | date: 2021-10-20T00:00:00+00:00 4 | draft: false 5 | --- 6 | 7 | The bitcoin-dev blog is a technical, open-content blog covering topics and projects Bitcoin developers work on. 8 | 9 | These blog posts are for new contributors exploring the breadth of existing topics and projects, 10 | for seasoned developers wanting to get short updates, 11 | for our pseudonymous and anonymous friends who can't or don't want to attend CoreDev, BitDevs, other meetups, and conferences, 12 | for lurkers, and the loud Bitcoin-Twitter voices. 13 | Posts are written by new Bitcoin developers diving deep into a topic, 14 | by designers making the technical side of Bitcoin more accessible, by students, 15 | and by long-time Bitcoin Core contributors implementing complex new features. 16 | **Everyone can submit a PR for a blog post covering a technical Bitcoin topic or an open-source Bitcoin project.** 17 | 18 | 19 | You can support open-source Bitcoin developers via [Bitcoin Developer Donation Portal], [Brink], and [OpenSats]. 20 | 21 | [Bitcoin Developer Donation Portal]: https://bitcoindevlist.com/ 22 | [Brink]: https://brink.dev/donate 23 | [OpenSats]: https://opensats.org/ 24 | 25 | 26 | If you want to submit a blog post, please read the contributing guidelines in [CONTRIBUTING.md][contributing]. 27 | If you are not sure if your topic fits, ask for feedback [in a GitHub issue][topic-feedback]. 28 | Cross-posts from your own blog are welcome! 29 | 30 | [contributing]: https://github.com/dev-bitcoin/blog/blob/main/CONTRIBUTING.md 31 | [topic-feedback]: https://github.com/dev-bitcoin/blog/issues/new?template=blog-post-topic-review.md&title=%5BTopic+Review%5D 32 | -------------------------------------------------------------------------------- /content/blog/bip158-deep-dive.md: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | title: "Compact Block Filters Deep Dive (BIP 158)" 4 | subtitle: "A technical explanation of the workings of compact block filters and 5 | Golomb-Rice Coding." 6 | date: 2021-11-13T09:32:00+02:00 7 | authors: 8 | - name: Elle Mouton 9 | github: ellemouton 10 | twitter: ellemouton 11 | tags: 12 | - "BIP158" 13 | - "Compact block filters" 14 | - "Bitcoin light client" 15 | 16 | images: 17 | - "/post-data/compact-block-filters/cover.png" 18 | 19 | appearedfirston: 20 | label: "ellemouton.com" 21 | url: "https://ellemouton.com/posts/bip158/" 22 | 23 | --- 24 | 25 | In this post, I will briefly describe the needs of a bitcoin light client and 26 | why compact block filters satisfy these needs better than Bloom filters do. Then 27 | I will dive into exactly how compact block filters work and will follow this 28 | with a step-by-step guide for constructing such a filter from a testnet block. 29 | 30 | #### The purpose of block filters 31 | 32 | A bitcoin light client is software that can back a bitcoin wallet without 33 | storing the blockchain. This means that it needs to be able to broadcast 34 | transactions to the network, but most importantly, it must be able to pick up 35 | when there is a new transaction that is relevant to the wallet it is backing. 36 | There are two ways a transaction becomes relevant to a wallet: either it is 37 | sending money to the wallet (creating a new output to a wallet address), or it 38 | is spending one of the UTXOs that the wallet owns. 39 | 40 | #### What was wrong with Bloom filters? 41 | 42 | Before [BIP 158] came along, the most widely used method for light clients was 43 | to use Bloom filters[^bloomfilters] as described in [BIP 37]. With a bloom 44 | filter, you take all the objects you are interested in (script pub keys spent or 45 | created), hash them a couple times and add the result of each to a bit map 46 | called a Bloom filter. This filter represents what you are interested in. You 47 | would then send this filter to a trusted bitcoin node and ask them to send you 48 | anything that matches your filter. The problem with this is that it is not very 49 | private since you are revealing some information to the bitcoin node you are 50 | sending this filter to. They can start getting an idea of the transactions you 51 | are interested in as well as the ones you are definitely not interested in. They 52 | can also just decide not to send you a transaction that matches the filter. So 53 | as you can see, it isn’t great for the light client. But it is also not great 54 | for the bitcoin node serving the light client. Each time you send them a filter, 55 | they have to load the relevant block from disk and determine which transactions 56 | match your filter. You could just spam them with fake filters and effectively 57 | DOS them. It takes very little energy to create a filter and lots to respond to 58 | it. 59 | 60 | #### Introducing Compact Block Filters: 61 | 62 | Ok, take two. What we want is: 63 | - More privacy 64 | - Less asymmetry in the client - server work load. Ie, the server should be 65 | required to do way less work. 66 | - Less trust. The light client shouldn't need to worry about the server holding 67 | back relevant transactions. 68 | 69 | With compact block filters, the server (full node) will for each block construct 70 | a deterministic filter that includes all the objects in the block. This filter 71 | can be calculated once and persisted. If light clients request a filter for a 72 | block, there is no asymmetry since the server won't have to do any more work 73 | than the client had to do when making the request. A light client can also 74 | choose to download the filters from multiple sources to ensure they match and 75 | can always download the full block and check for itself if the filter that the 76 | server provided was indeed correct given the block's contents. Another bonus is 77 | that this is way more private. The light client no longer sends a fingerprint of 78 | the data it is interested in to the server. And so it becomes way more difficult 79 | to analyse the light client's activity. The light client gets these filters from 80 | the server and checks for itself if any of its objects match what is seen in the 81 | filter, and if it does match, then the light client asks for the full block. One 82 | thing to note with this way of doing things is that full nodes serving the light 83 | clients will need to persist these filters, and the light clients might also want 84 | to persist a few filters and so it is important that the filters are as small as 85 | possible (hence the name, compact block filters). 86 | 87 | Cool! Now we get to the cool stuff. How is this filter created? What does it 88 | look like? 89 | 90 | What do we want? 91 | - We want to put fingerprints of certain objects in the filter so that when 92 | clients are looking to see if a block maybe contains info relevant to them, 93 | they can take all their objects and check if the filter matches on those 94 | objects. 95 | - We want the filters to be as small as possible. 96 | - Effectively we want to sort of summarise some of the block info… in a size 97 | much much smaller than the block. 98 | 99 | The info included in the basic filter is: every transaction's input's 100 | scriptPubKey being spent and every transaction's output's scriptPubKey being 101 | created. So something like this: 102 | ```go 103 | objects = {spk1, spk2, spk3, spk4, ..., spkN} // A list of N scriptPubKeys. 104 | ``` 105 | 106 | Technically we could just stop here and say this list of scriptPubKeys is our 107 | filter. It is a condensed version of what is in the blockchain and contains the 108 | info the light client needs. With this list, they could tell with 100% certainty 109 | if something they are interested in is in the block. But it is still pretty big. 110 | So the next step is all about making this list as compact as possible. This is 111 | where things get insanely cool. 112 | 113 | First, we convert each object into a number in a range such that the object 114 | numbers are uniformly distributed in that range: Let's say we have 10 objects (N 115 | = 10), then we have some function that turns each of the objects into a number. 116 | Let’s say we chose the range [0, 10] since we have 10 objects. Now the 117 | hashing-plus-convert-to-number function we use will take each object and produce 118 | a number in the space from [0, 10]. It is uniformly distributed in this space. 119 | That means that, after ordering them, we will get (in the very, very ideal case) 120 | something like this: 121 | 122 | {{< center-figure src="/post-data/compact-block-filters/dense.jpeg" caption="" 123 | height=250 width=250 >}} 124 | 125 | First of all, wow that is so great cause we have drastically decreased the size 126 | of an objects fingerprint. Each one is just a number now. Ok so, let this be 127 | our new filter: 128 | 129 | ```go 130 | numbers := {1,2,3,4,5,6,7,8,9,10} 131 | ``` 132 | 133 | Now a light client downloads the filter and wants to see if one of the objects 134 | they are looking for is matched in this filter. All they need to do is take 135 | their objects and do the same hashing-plus-convert-to-number scheme and check if 136 | any of the numbers are in the filter. What is the problem? The filter has a 137 | number for each possible number in the space! Meaning that absolutely any object 138 | will match on this filter. In other words, the false-positive rate of this 139 | filter is 1. This is no good. We have lost too much info on our quest to 140 | compress the data in the filter. What we need is a higher false-positive (fp) 141 | rate. Ok so let’s say we want a false positive rate of 5. Then what we want is 142 | to have our objects be mapped uniformly to a space of [0, 50]: 143 | 144 | {{< center-figure src="/post-data/compact-block-filters/sparse.png" caption="" 145 | height=250 width=250 >}} 146 | 147 | This is starting to look a bit better. If I am a client downloading this filter 148 | and I check if my objects are maybe in the filter, there will be a 1/5 chance 149 | that if it matches, it is a false positive. Great, so now we have mapped 10 150 | objects to numbers between 0 & 50. This new list of numbers is our filter. 151 | Again, we could stop here… but we can compress this even further!! 152 | 153 | We have this list of ordered numbers that we know are distributed uniformly 154 | across this space between [0, 50]. We know that there are 10 items in the list. 155 | What this means is that we can deduce that the most likely _difference_ between 156 | each of the numbers in this ordered list is about 5. In general, if we have N 157 | items and a false positive rate of M, then the space will be of size N * M. So 158 | the numbers in the space can range from 0 to N * M, but the difference between 159 | each number (once ordered) will be roughly M. M will definitely be a smaller 160 | number to store than a number in the N * M space. So what we can do is instead 161 | of storing each number, we can instead store the difference of each successive 162 | number. In the above case, this would mean that instead of storing `[0, 5, 10, 163 | 15, 20, 25, 30, 35, 40, 45, 50]`, we just store `[0, 5, 5, 5, 5, 5, 5, 5, 5, 5]` 164 | and then it is trivial to reconstruct the original list. As you can gather, 165 | storing the number 50 requires way more bits than storing the number 5. But why 166 | stop there? We can compress this even further! 167 | 168 | This is where Golomb-Rice Coding comes in. This encoding works well for a list 169 | of numbers that will all very likely be close to some number. This is what we 170 | have! We have a list of numbers that will all very likely be close to 5 (or, in 171 | general, close to our FP rate of M) and so taking the quotient of any number in 172 | the list with that number (dividing each number by 5 and ignoring the remainder) 173 | will very likely be 0 (if the number is slightly less than 5) or 1 if the number 174 | is slightly more than 5. The quotient could be 2, 3, etc, but the likelihood 175 | decreases a lot. Great! So we can take advantage of this knowledge and say that 176 | we will encode a small quotient with the smallest number of bits that we can and 177 | use more bits to encode larger, unlikely quotients. Then we also need to encode 178 | the remainders (since we want to be able to reconstruct the values exactly), and 179 | these will always be numbers between [0, M-1] (in our case, [0, 4]). For 180 | encoding the quotients, we use the following mapping: 181 | 182 | {{< center-figure src="/post-data/compact-block-filters/quotient.png" caption="" height=125 width=125 >}} 183 | 184 | The mapping above is easy to read: The number of `1`s indicates the quotient we 185 | are encoding, and the `0` indicates the end of the quotient encoding. So for each 186 | number in our list, we encode the quotient using the above table, and then we 187 | convert the remainder to binary using the number of bits needed to encode the 188 | maximum of M-1. In our case, that is 3 bits. Here is a table showing the 189 | encoding of the possible remainders in our example: 190 | 191 | {{< center-figure src="/post-data/compact-block-filters/remainder.png" caption="" height=125 width=125 >}} 192 | 193 | So, in our ideal case example, our list of `[0, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]` 194 | can be encoded as follows: 195 | ```go 196 | 0000 10000 10000 10000 10000 10000 10000 10000 10000 10000 197 | ``` 198 | 199 | Before we move on to a more realistic example, let’s see if we can reconstruct 200 | our original list from this filter. 201 | 202 | Ok so we have: “0000100001000010000100001000010000100001000010000”. We know how 203 | Golomb-Rice Coding encodes quotients and we also know that M is 5 (since this 204 | will be public knowledge known to everyone using this filter construction). 205 | Since we know M is 5, we know that 3 bits will be used to encode the remainders. 206 | So we can take our filter and produce the following quotient-remainder tuples: 207 | 208 | ```go 209 | [(0, 0), (1, 0), (1, 0), (1, 0), (1, 0), (1, 0), (1, 0), (1, 0), (1, 0), (1, 0)] 210 | ``` 211 | 212 | We know that the quotients were produced by dividing the numbers by M (5), so we can 213 | reconstruct these: 214 | 215 | ```go 216 | [0, 5, 5, 5, 5, 5, 5, 5, 5, 5] 217 | ``` 218 | 219 | And we know that this list represents differences of numbers, so we can 220 | reconstruct the OG list: 221 | 222 | ```go 223 | [0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50] 224 | ``` 225 | 226 | #### A more realistic example 227 | 228 | We will now try to construct a filter from an actual Bitcoin testnet block. I'm 229 | going to use block [2101914]. Let’s see what its actual filter is: 230 | 231 | ``` 232 | $ bitcoin-cli getblockhash 2101914 233 | 000000000000002c06f9afaf2b2b066d4f814ff60cfbc4df55840975a00e035c 234 | 235 | $ bitcoin-cli getblockfilter 000000000000002c06f9afaf2b2b066d4f814ff60cfbc4df55840975a00e035c 236 | ``` 237 | ```json 238 | { 239 | "filter": "5571d126b85aa79c9de56d55995aa292de0484b830680a735793a8c2260113148421279906f800c3b8c94ff37681fb1fd230482518c52df57437864023833f2f801639692646ddcd7976ae4f2e2a1ef58c79b3aed6a705415255e362581692831374a5e5e70d5501cdc0a52095206a15cd2eb98ac980c22466e6945a65a5b0b0c5b32aa1e0cda2545da2c4345e049b614fcad80b9dc9c903788163822f4361bbb8755b79c276b1cf7952148de1e5ee0a92f6d70c4f522aa6877558f62b34b56ade12fa2e61023abf3e570937bf379722bc1b0dc06ffa1c5835bb651b9346a270", 240 | "header": "8d0cd8353342930209ac7027be150e679bbc7c65cc62bb8392968e43a6ea3bfe" 241 | } 242 | ``` 243 | 244 | Ok shweeeeeet, let’s see if we can reconstruct this filter from the block. 245 | 246 | The full code for this can be found in this [github repo]. I will just show some 247 | pseudo code snippets here. The beef of the code is the function called 248 | `constructFilter`, which takes in a bitcoin client that can be used to make calls 249 | to bitcoind and the block in question. The function looks something like this: 250 | 251 | 252 | ```go 253 | func constructFilter(bc *bitcoind.Bitcoind, block bitcoind.Block) ([]byte, error) { 254 | // 1. Collect all the objects from the block that we want to add to the filter 255 | 256 | // 2. Convert all the objects to numbers and sort them. 257 | 258 | // 3. Get the differences between the sorted numbers 259 | 260 | // 4. Encode these differences using Golomb-Rice Coding 261 | } 262 | ``` 263 | 264 | Ok so step 1 is to collect all the objects from the block that we want to add to 265 | the filter. From the BIP, we know that these objects are all the scriptPubKeys 266 | being spent as well as all the scriptPubKeys of each output. Some extra rules 267 | from the BIP are that we skip the input for the coinbase transaction (since it 268 | is empty and meaningless), and we skip any OP_RETURN outputs. We also 269 | de-duplicate the data. So if there are two identical scriptPubKeys, we only 270 | include one in the filter. 271 | 272 | 273 | ```go 274 | // The list of objects we want to include in our filter. These will be 275 | // every scriptPubKey being spent as well as each output's scriptPubKey. 276 | // We use a map so that we can dedup any duplicate scriptPubKeys. 277 | objects := make(map[string] struct{}) 278 | 279 | // Loop over every transaction in the block. 280 | for i, tx := range block.Tx { 281 | 282 | // Add the scriptPubKey of each of the transaction's outputs 283 | // and add those to our list of objects. 284 | for _, txOut := range tx.Vout { 285 | scriptPubKey := txOut.ScriptPubKey 286 | 287 | if len(scriptPubKey) == 0 { 288 | continue 289 | } 290 | 291 | // We don't add the output if it is an OP_RETURN (0x6a). 292 | if spk[0] == 0x6a { 293 | continue 294 | } 295 | 296 | objects[skpStr] = struct{}{} 297 | } 298 | 299 | // We don't add the inputs of the coinbase transaction. 300 | if i == 0 { 301 | continue 302 | } 303 | 304 | // For each input, go and fetch the scriptPubKey that it is 305 | // spending. 306 | for _, txIn := range tx.Vin { 307 | prevTx, err := bc.GetRawTransaction(txIn.Txid) 308 | if err != nil { 309 | return nil, err 310 | } 311 | 312 | scriptPubKey := prevTx.Vout[txIn.Vout].ScriptPubKey 313 | 314 | if len(scriptPubKey) == 0 { 315 | continue 316 | } 317 | 318 | objects[spkStr] = struct{}{} 319 | } 320 | } 321 | 322 | ``` 323 | 324 | Ok great, we have all the objects we care about. And now we can also define the 325 | variable N to be the length of the `objects` map. In this example, N is 85. 326 | 327 | The next step is to convert each of the objects to numbers spread uniformly 328 | across a range. Remember that this range depends on the false-positive rate we 329 | want. BIP158 defines the constant M to be 784931. This means that we want every 330 | 1/784931 matches to be a false-positive. As we did in our earlier example, we 331 | take this fp rate of M and multiply it by N to get the range that we want all 332 | our numbers to lie in. We define this as F where F = M\*N. In our case, we have 333 | 85 objects, and so F=66719135. I am not going to go into the details of the 334 | function used to map our objects to numbers (you can check out the details of 335 | this in the code in the linked repo). All you need to know for now is that it 336 | takes in an object, the constant F, which defines the range that it needs to map 337 | the object to, and a key which is the block hash. Once we have all the numbers, 338 | we sort the list in ascending order, and then we also create a new list called 339 | `differences` which will hold the differences between each sequential number in 340 | the sorted `numbers` list. 341 | 342 | ```go 343 | numbers := make([]uint64, 0, N) 344 | 345 | // Iterate over all the objects, convert them to numbers lying uniformly in the 346 | // range [0, F] and add them to the `numbers` list. 347 | for o := range objects { 348 | // Using the given key, max number (F) and object bytes (o), 349 | // convert the object to a number between 0 and F. 350 | v := convertToNumber(b, F, key) 351 | 352 | numbers = append(numbers, v) 353 | } 354 | 355 | // Sort the numbers. 356 | sort.Slice(numbers, func(i, j int) bool { return numbers[i] < numbers[j] }) 357 | 358 | // Convert the list of numbers to a list of differences. 359 | differences := make([]uint64, N) 360 | for i, num := range numbers { 361 | if i == 0 { 362 | differences[i] = num 363 | continue 364 | } 365 | 366 | differences[i] = num - numbers[i-1] 367 | } 368 | ``` 369 | 370 | Awesome! Here is a graph showing the values in the `numbers` and `differences` 371 | lists: 372 | 373 | {{< center-figure src="/post-data/compact-block-filters/bitcoinExample.png" caption="" height=600 width=600 >}} 374 | 375 | As you can see, the 85 numbers are really nicely uniformly distributed across 376 | the space! And this results in the values in the `differences` list being pretty 377 | small. 378 | 379 | The last step now is to use Golomb-Rice Coding to encode this `differences` 380 | list. Recall from the earlier explanation that we need to divide each difference 381 | by its most likely value and then we encode that quotient along with the 382 | remainder. In my earlier example, I said that this most-likely value would be the 383 | M that we choose and that the remainder would then lie in the range [0, M]. 384 | However, this is not what is done in the BIP as it was found[^golombloss] that 385 | this is, in fact, not the ideal way to choose the Golomb-Rice coder parameter when 386 | trying to optimize for the smallest possible size of the final encoded filter. 387 | And so, instead of using M, a new constant of P is defined and P^2 is used as the 388 | Golomb-Rice parameter. P is defined as 19. This means that each difference value 389 | is divided by 2^19 to get the quotient and remainder and the remainder is then 390 | encoded in binary in 19 bits. 391 | 392 | ```go 393 | filter := bstream.NewBStreamWriter(0) 394 | 395 | // For each number in the differences list, calculate the quotient and 396 | // remainder after dividing by 2^P. 397 | for _, d := range differences { 398 | q := math.Floor(float64(d)/math.Exp2(float64(P))) 399 | r := d - uint64(math.Exp2(float64(P))*q) 400 | 401 | // Encode the quotient. 402 | for i := 0; i < int(q); i++ { 403 | filter.WriteBit(true) 404 | } 405 | filter.WriteBit(false) 406 | 407 | filter.WriteBits(r, P) 408 | } 409 | ``` 410 | 411 | Great stuff! Now when we print out this filter, we get: 412 | 413 | ``` 414 | 71d126b85aa79c9de56d55995aa292de0484b830680a735793a8c2260113148421279906f800c3b8c94ff37681fb1fd230482518c52df57437864023833f2f801639692646ddcd7976ae4f2e2a1ef58c79b3aed6a705415255e362581692831374a5e5e70d5501cdc0a52095206a15cd2eb98ac980c22466e6945a65a5b0b0c5b32aa1e0cda2545da2c4345e049b614fcad80b9dc9c903788163822f4361bbb8755b79c276b1cf7952148de1e5ee0a92f6d70c4f522aa6877558f62b34b56ade12fa2e61023abf3e570937bf379722bc1b0dc06ffa1c5835bb651b9346a270 415 | ``` 416 | 417 | Apart from the first two bytes, this matches the filter we got from bitcoind 418 | exactly! Why the 2 byte difference? The BIP says that the N value needs to be 419 | encoded in CompactSize format and appended to the front of the filter so that it 420 | can be decoded by the receiver. This is done as follows: 421 | 422 | ```go 423 | fd := filter.Bytes() 424 | 425 | var buffer bytes.Buffer 426 | buffer.Grow(wire.VarIntSerializeSize(uint64(N)) + len(fd)) 427 | 428 | err = wire.WriteVarInt(&buffer, 0, uint64(N)) 429 | if err != nil { 430 | return nil, err 431 | } 432 | 433 | _, err = buffer.Write(fd) 434 | if err != nil { 435 | return nil, err 436 | } 437 | ``` 438 | 439 | If we print out the filter now, we get one that matches the one we got from 440 | bitcoind exactly: 441 | 442 | ``` 443 | 5571d126b85aa79c9de56d55995aa292de0484b830680a735793a8c2260113148421279906f800c3b8c94ff37681fb1fd230482518c52df57437864023833f2f801639692646ddcd7976ae4f2e2a1ef58c79b3aed6a705415255e362581692831374a5e5e70d5501cdc0a52095206a15cd2eb98ac980c22466e6945a65a5b0b0c5b32aa1e0cda2545da2c4345e049b614fcad80b9dc9c903788163822f4361bbb8755b79c276b1cf7952148de1e5ee0a92f6d70c4f522aa6877558f62b34b56ade12fa2e61023abf3e570937bf379722bc1b0dc06ffa1c5835bb651b9346a270 444 | ``` 445 | 446 | Yay! 447 | 448 | However, from my understanding, there is no need to add N to the filter. If you 449 | know the value of P, then you can figure out the value of N. Let’s do this now 450 | by seeing if we can take the filter above, and reconstruct the original list of 451 | numbers: 452 | 453 | ```go 454 | b := bstream.NewBStreamReader(filter) 455 | var ( 456 | numbers []uint64 457 | prevNum uint64 458 | ) 459 | 460 | for { 461 | 462 | // Read a quotient from the stream. Read until we encounter 463 | // a '0' bit indicating the end of the quotient. The number of 464 | // '1's we encounter before reaching the '0' defines the 465 | // quotient. 466 | var q uint64 467 | c, err := b.ReadBit() 468 | if err != nil { 469 | return err 470 | } 471 | 472 | for c { 473 | q++ 474 | c, err = b.ReadBit() 475 | if errors.Is(err, io.EOF) { 476 | break 477 | } else if err != nil { 478 | return err 479 | } 480 | } 481 | 482 | // The following P bits are the remainder encoded as binary. 483 | r, err := b.ReadBits(P) 484 | if errors.Is(err, io.EOF) { 485 | break 486 | } else if err != nil { 487 | return err 488 | } 489 | 490 | n := q*uint64(math.Exp2(float64(P))) + r 491 | 492 | num := n + prevNum 493 | numbers = append(numbers, num) 494 | prevNum = num 495 | } 496 | 497 | fmt.Println(numbers) 498 | ``` 499 | 500 | The above produces the same list of numbers that we had before and we were able 501 | to reconstruct this without the knowledge of N. So I am not sure why it was 502 | decided that N should be added to the filter. If anyone knows why it was 503 | required to add N to the filter, please let me know! 504 | 505 | Cool, that was fun! Thanks for reading. This is a [cross-post] from my 506 | [website], where you can find many more Bitcoin and Lightning related technical 507 | posts. Until next time... Yeeeeet! 508 | 509 | [github repo]: https://github.com/ellemouton/bip158Example 510 | [cross-post]: https://www.ellemouton.com/blog/view/9 511 | [website]: https://www.ellemouton.com 512 | [BIP 158]: https://github.com/bitcoin/bips/blob/master/bip-0158.mediawiki 513 | [BIP 37]: https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki 514 | [2101914]: https://blockstream.info/testnet/block/000000000000002c06f9afaf2b2b066d4f814ff60cfbc4df55840975a00e035c 515 | 516 | 517 | [^bloomfilters]: https://en.wikipedia.org/wiki/Bloom_filter 518 | [^bip37]: https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki 519 | [^repo]: https://github.com/ellemouton/bip158Example 520 | [^golombloss]: https://gist.github.com/sipa/576d5f09c3b86c3b1b75598d799fc845 521 | 522 | 523 | -------------------------------------------------------------------------------- /content/blog/bitcoin-core-usdt-support.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Userspace, Statically Defined Tracing support for Bitcoin Core" 3 | date: 2021-08-30T09:00:00+02:00 4 | authors: 5 | - name: "0xB10C" 6 | github: "0xB10C" 7 | twitter: "0xB10C" 8 | tags: 9 | - "Bitcoin Core" 10 | - "USDT tracepoints" 11 | appearedfirston: 12 | label: b10c.me 13 | url: "https://b10c.me/blog/008-bitcoin-core-usdt-support/" 14 | images: 15 | - /post-data/usdt-support-core/header.png 16 | --- 17 | 18 | _This report updates on what 0xB10C, [Coinbase Crypto Community Fund grant recipient], 19 | has been working on over the first half of his year-long Bitcoin development grant. 20 | This specifically covers his work on Userspace, Statically Defined Tracing support 21 | for Bitcoin Core. This report was published on [0xB10Cs blog] and the [Coinbase blog] too._ 22 | 23 | [Coinbase Crypto Community Fund grant recipient]: https://blog.coinbase.com/announcing-our-first-bitcoin-core-developer-grants-3d88559db068 24 | [Coinbase blog]: https://blog.coinbase.com/userspace-statically-defined-tracing-support-for-bitcoin-core-e4076cd3e07 25 | [0xB10Cs blog]: https://b10c.me/blog/008-bitcoin-core-usdt-support/ 26 | 27 | 28 | 29 | The reference implementation to the Bitcoin protocol rules, Bitcoin Core, is 30 | the most widely used software to interact with the Bitcoin network. Bitcoin 31 | Core is, however, a black box to most users. While information can be queried 32 | via the RPC interface or searched in the debug log, there is no defined interface 33 | for real-time insights into process internals. Yet, some users could benefit 34 | from more observability into their node. Hobbyists and companies running Bitcoin 35 | Core in production want to include their nodes in their real-time monitoring. 36 | Developers need visibility into test deployments to evaluate, review, debug, 37 | and benchmark changes. Researchers want to observe and analyze the behavior 38 | of nodes on the peer-to-peer network. Exchanges and other services handling 39 | large sums of bitcoin want to detect attacks and other anomalies early. 40 | 41 | 42 | ### Peeking inside with Userspace, Statically Defined Tracing 43 | 44 | 45 | 46 | The [eBPF] technology present in the Linux kernel can be used for 47 | observability into userspace applications. The technology allows 48 | running a small, sandboxed program in the Linux kernel, which can hook 49 | into predefined tracepoints in running processes. Once hooked into a 50 | tracepoint, the program is executed each time the tracepoint is reached. 51 | Tracepoints can pass data, for example, application state. Tracing scripts 52 | can further process the data. The practice of hooking into tracepoints in 53 | userspace applications is known as Userspace, Statically Defined Tracing 54 | (USDT). For example, these tracepoints are also included in PostgreSQL, 55 | MySQL, Python, NodeJS, Ruby, PHP, and libraries like libc, libpthread, 56 | and libvirt. 57 | 58 | [eBPF]: https://ebpf.io 59 | 60 | 61 | 62 | The static tracepoints can be leveraged by Bitcoin Core users wishing for 63 | more insights into their node. Adding USDT support [did not require intrusive 64 | changes], and no custom tooling had to be written. When not used, the performance 65 | impact of the tracepoints is minimal to non-existent. Only privileged processes 66 | can hook into the tracepoints, no information leaks to other processes on the host. 67 | These properties make Userspace, Statically Defined Tracing a good fit for Bitcoin 68 | Core. 69 | 70 | [did not require intrusive changes]: https://github.com/bitcoin/bitcoin/pull/19866 71 | 72 | 73 | 74 | 75 | For example, I [placed two tracepoints] in the peer-to-peer message handling 76 | code of Bitcoin Core. For each inbound and outbound P2P message, the tracepoints 77 | pass information about the peer, the connection, and the message. This data can 78 | be filtered and processed by tracing scripts. As a demo, I have built a P2P Monitor 79 | that shows the communication between two peers in real-time. Users can find this 80 | script alongside other [USDT examples] in the `contrib/tracing/` directory of the 81 | Bitcoin Core repository. 82 | 83 | {{< tweet 1396889721859715077 >}} 84 | 85 | [placed two tracepoints]: https://github.com/bitcoin/bitcoin/commit/4224dec22baa66547303840707cf1d4f15a49b20 86 | [USDT examples]: https://github.com/bitcoin/bitcoin/tree/master/contrib/tracing 87 | 88 | ### Use-cases for Userspace, Statically Defined Tracing 89 | 90 | I list some use-cases for Userspace, Statically Defined Tracing 91 | I have thought about or worked on. With only three tracepoints merged, 92 | there is plenty of room for developers to add new tracepoints and 93 | get creative with tracing scripts. [Issue #20981] contains discussion 94 | and ideas for additional tracepoints that can be implemented. 95 | 96 | [Issue #20981]: https://github.com/bitcoin/bitcoin/issues/20981 97 | 98 | 99 | Researchers and developers can use the P2P message tracepoints to 100 | monitor P2P network anomalies in real-time. One example could be 101 | detecting the recent `addr` message flooding as reported in this 102 | [bitcointalk.org post]. The messages were announcing random IP 103 | addresses not belonging to nodes on the Bitcoin network. The flooding 104 | has been [covered][paper] in detail by Grundmann and Baumstark. They 105 | discuss that the attacker could obtain the number of connected peers and 106 | learn about other addresses, including Tor addresses, the node is listening 107 | on. This would reduce the privacy of the node operator. It's important to 108 | stay vigilant to these attacks, discuss them, and then, if needed, react 109 | to them. 110 | 111 | [bitcointalk.org post]: https://bitcointalk.org/index.php?topic=5348856.0 112 | [paper]: https://arxiv.org/abs/2108.00815 113 | 114 | Similarly, I have been instrumenting the Bitcoin Core network address manager 115 | with tracepoints. The addrman keeps track of gossiped network addresses for 116 | potential outbound peers connections a node makes. It's designed to be resiliant 117 | against [Eclipse Attacks], where a node only has connections to peers controlled 118 | by the attacker. The attacker can choose which information to feed to the node, 119 | enabling, for example, double-spending attacks. Information about the addresses 120 | in the addrman might help detect the build-up of an eclipse attack when combined 121 | with other data. 122 | 123 | Additionally, these addrman tracepoints can be helpful during debugging and 124 | code review. To showcase this, I build a tool that visualizes the addresses 125 | in the addrman data structure based on the data submitted to the tracepoints. 126 | 127 | {{< tweet 1407019872681332742 >}} 128 | 129 | [Eclipse Attacks]: https://cs-people.bu.edu/heilman/eclipse/ 130 | 131 | 132 | 133 | A Prometheus metric exporter can also build on top of the tracepoints 134 | without requiring additional code in Bitcoin Core. There already exist 135 | RPC-based Prometheus exporters and projects like [Statoshi]. However, 136 | RPC-based exporters are limited by the information exposed via the RPC 137 | interface, and Statoshi is large a patch-set that requires maintenance 138 | on each Bitcoin Core release. I have published an experimental USDT-based 139 | exporter called [bitcoind-observer] that hooks into the three currently 140 | merged tracepoints and serves metrics in the Prometheus format. The exporter 141 | can be used by everyone currently running a Bitcoin Core node compiled with 142 | USDT support. A demo is available on [bitcoind.observer]. I've recently used 143 | this to benchmark the bandwidth usage of [an implementation] of [Erlay: 144 | Bandwidth-Efficient Transaction Relay for Bitcoin]. Initial results can be 145 | found [here][erlay-results]. 146 | 147 | [Statoshi]: https://statoshi.info/ 148 | [bitcoind-observer]: https://github.com/0xb10c/bitcoind-observer 149 | [bitcoind.observer]: https://bitcoind.observer 150 | [an implementation]: https://github.com/bitcoin/bitcoin/pull/21515 151 | [Erlay: Bandwidth-Efficient Transaction Relay for Bitcoin]: https://arxiv.org/abs/1905.10518 152 | [erlay-results]: https://github.com/naumenkogs/txrelaysim/issues/8#issuecomment-903255752 153 | 154 | The already existing tracepoint `validation:block_connected` can be used 155 | to benchmarking block validation. This allows, for example, to compare 156 | the initial block download performance between different patches and can 157 | aid in detecting performance improvements and regressions. For example, 158 | the [bitcoinperf] project might benefit from such tracepoints. I've used 159 | the tracepoint to [benchmark] Martin Ankerls pull request [#22702]. If 160 | merged, the changes he proposes would result in a substantial block 161 | validation speed up and reduction in memory usage. 162 | 163 | [bitcoinperf]: https://github.com/chaincodelabs/bitcoinperf 164 | [benchmark]: https://github.com/bitcoin/bitcoin/pull/22702#issuecomment-900662089 165 | [#22702]: https://github.com/bitcoin/bitcoin/pull/22702 166 | 167 | ### Next steps 168 | 169 | I will collect further ideas for tracepoints and implement them alongside 170 | example tracing scripts and more tooling. This will also involve 171 | communicating with other Bitcoin and Bitcoin Core developers about 172 | which tracepoints could be helpful in their projects. An example is 173 | Antoine Riard's [cross-layer anomaly detection watchdog] which he 174 | initially proposed as a new, internal module to Bitcoin Core. However, 175 | many of the required events and metrics can be collected by hooking into 176 | tracepoints. This means the watchdog could be an external runtime, which 177 | would speed up the watchdog development and requires less code and 178 | maintenance on the Bitcoin Core side. 179 | 180 | [cross-layer anomaly detection watchdog]: https://github.com/bitcoin/bitcoin/pull/18987 181 | 182 | If everything goes according to plan, the v23.0 release of Bitcoin Core, 183 | expected in early 2022, will include the first set of tracepoints. A 184 | goal is to enable USDT support in release builds by default, which still needs 185 | some work. Additionally, the tracepoint API should be semi-stable and thus 186 | needs testing. 187 | 188 | 189 | --- 190 | 191 | **In short**: I have been adding tracepoints to Bitcoin Core that users can hook 192 | into to get insights into the internal state. The tracepoints are based on Linux 193 | kernel technology and do not require intrusive changes or custom tooling. The 194 | groundwork is done. Now further tracepoints can be added, and tooling can be written. 195 | 196 | -------------------------------------------------------------------------------- /content/blog/example-post.md: -------------------------------------------------------------------------------- 1 | --- 2 | 3 | title: "This is an example post" 4 | subtitle: "Showcasing how to write your own bitcoin-dev.blog post." 5 | date: 2021-10-20T00:00:00+00:00 6 | 7 | # This isn't really needed for this blog post, but should show-case 8 | # that you can and are encuraged to link to, e.g., your own blog 9 | # where the post was first published. 10 | appearedfirston: 11 | label: bitcoin-dev.blog 12 | url: "https://bitcoin-dev.blog/blog/example-post/" 13 | 14 | authors: 15 | - name: "0xB10C" 16 | github: "0xb10c" 17 | twitter: "twitter" 18 | - name: "Co-Author-1" 19 | github: "bitcoin" 20 | twitter: "bitcoincoreorg" 21 | images: 22 | - "/post-data/example-post/header.png" 23 | 24 | # enables MathJax support for this blog post 25 | # Only set this if you are using MathJax in your post. 26 | mathjax: true 27 | 28 | # never list this example on the blog 29 | _build: 30 | list: never 31 | 32 | --- 33 | 34 | Here is the description of the example blog post. 35 | You'd want to summarize the contents of the following post. 36 | Don't let this get too long, though! 37 | 38 | 39 | 40 | 41 | 42 | Here the blog post starts. 43 | You can find the [source] on GitHub. 44 | Please read the [CONTRIBUTING.md] before you start writing a post for this blog. 45 | 46 | [CONTRIBUTING.md]: https://github.com/dev-bitcoin/blog/blob/main/CONTRIBUTING.md 47 | [source]: https://github.com/dev-bitcoin/blog/blob/main/content/blog/example-post.md?plain=1 48 | 49 | #### Text 50 | 51 | You can use markdown formatting. 52 | 53 | - **Bold text.** 54 | - _Italic text._ 55 | - *More italic text.* 56 | - ~~Strikethrough text.~~ 57 | 58 | 59 | #### Links 60 | 61 | Markdown lets us include links in three formats. 62 | [Inline-style links](https://your-looooong-url.btc) can often get very long and make reading the markdown source harder. 63 | [Reference-style links] are shorter and thus preferred. 64 | 65 | [Reference-style links]: https://your-looooong-url.btc 66 | 67 | #### Images 68 | 69 | For images, we could use markdown image tags too. 70 | However, it's recommended to use the custom `center-figure` [Hugo shortcode]. 71 | This is based on the built-in [`figure`] shortcode but automatically centers the image and captions. 72 | 73 | [Hugo shortcode]: https://gohugo.io/content-management/shortcodes/ 74 | [`figure`]: https://gohugo.io/content-management/shortcodes/#figure 75 | 76 | ``` 77 | {{}} 78 | ``` 79 | produces 80 | 81 | {{< center-figure src="/img/logo-512.png" caption="The bitcoin-dev.blog logo." height=128 width=128 >}} 82 | 83 | Generally, including images in the repository is preferred to linking to external images. 84 | Check if you have permission to include the image. 85 | 86 | #### Code 87 | 88 | The built-in shortcode [`highlight`] can be used to showcase code. 89 | 90 | ``` 91 | {{}} 92 | let mut builder = wallet_bob.build_tx(); 93 | builder 94 | .add_recipient(addr.script_pubkey(), 60_000) 95 | .add_foreign_utxo(alice_outpoint, alice_psbt_input, satisfaction_weight)?; 96 | {{}} 97 | ``` 98 | 99 | produces 100 | 101 | 102 | {{< highlight rust >}} 103 | let mut builder = wallet_bob.build_tx(); 104 | builder 105 | .add_recipient(addr.script_pubkey(), 60_000) 106 | .add_foreign_utxo(alice_outpoint, alice_psbt_input, satisfaction_weight)?; 107 | {{< /highlight >}} 108 | 109 | 110 | [`highlight`]: https://gohugo.io/content-management/shortcodes/#highlight 111 | 112 | #### Footnotes 113 | 114 | We can include footnotes[^this-is-a-footnote-ref] by using `[^ref]` where `ref` is reference to a `[^ref]: `. 115 | 116 | [^this-is-a-footnote-ref]: This is a footnote. 117 | 118 | 119 | #### Tweets 120 | 121 | Hugo offers a built-in tweet shortcode too. 122 | 123 | ``` 124 | {{}} 125 | ``` 126 | 127 | {{< tweet 1110302988 >}} 128 | 129 | 130 | #### Videos 131 | 132 | Vimeo and YouTube videos can be embedded with the [`vimeo`] and [`youtube`] tags. 133 | 134 | [`vimeo`]: https://gohugo.io/content-management/shortcodes/#vimeo 135 | 136 | [`youtube`]: https://gohugo.io/content-management/shortcodes/#youtube 137 | 138 | `{{}}` produces 139 | {{< vimeo 412058509 >}} 140 | 141 | 142 | `{{}}` 143 | produces 144 | 145 | 146 | {{< youtube Ey0xAcN11zk >}} 147 | 148 | --- 149 | 150 | #### $\LaTeX$ via MathJax 151 | 152 | $\LaTeX$ formulars can be embeeded with [MathJax](https://www.mathjax.org/). 153 | MathJax is disabled by default, but can be enabled by setting `mathjax: true` in the front-matter. 154 | See the front-matter of this post for an example. 155 | 156 | In-line LaTeX can be written as $1 + 1 = 3$. This also works in headings. 157 | Due to Hugo's MarkDown parser, multi-line formulars [need six backslashes](https://github.com/wowchemy/wowchemy-hugo-themes/issues/291#issuecomment-334746889) (`\\\\\\\\\\\\`) between new lines. 158 | See the example below. 159 | 160 | $$ 161 | \begin{align} 162 | p & : \text{private key} \\\\\\ 163 | P = pG & : \text{public key} \\\\\\ 164 | m & : \text{message} \\\\\\ 165 | r & : \text{random nonce} \\\\\\ 166 | R = rG & : \text{nonce commitment} 167 | \end{align} 168 | $$ 169 | -------------------------------------------------------------------------------- /content/blog/schnorr-basics.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Schnorr basics" 3 | subtitle: "Explaining Schnorr signatures with simplified maths" 4 | date: "2022-01-03" 5 | authors: 6 | - name: Kalle Rosenbaum 7 | github: kallerosenbaum 8 | twitter: kallerosenbaum 9 | tags: 10 | - "Schnorr" 11 | - "signature" 12 | - "BIP340" 13 | mathjax: true 14 | images: 15 | - "/post-data/schnorr-basics/sig-overview.png" 16 | appearedfirston: 17 | label: "popeller.io" 18 | url: "https://popeller.io/schnorr-basics" 19 | --- 20 | 21 | If you're having a difficult time wrapping your head around Schnorr 22 | signatures, you're not alone. In this post, I attempt to explain 23 | Schnorr signatures at a level that I myself appreciate, and 24 | hopefully, you'll find it valuable too. 25 | 26 | 27 | 28 | ## What's Schnorr? 29 | 30 | Schnorr is a new signature scheme in Bitcoin that got activated in the 31 | taproot upgrade. Schnorr has some nice properties listed in 32 | [BIP340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki#motivation), but I won't reiterate those here. 33 | 34 | ## Signing and verifying 35 | 36 | A signature scheme consists of two actions, signing and verification 37 | as the following diagram shows. 38 | 39 | {{< center-figure 40 | src="/post-data/schnorr-basics/sig-overview.svg" 41 | caption="Left side: signing using a message and a private key. Right side: verification that the message (the cat) was signed with the correct private key." 42 | height=80% 43 | width=80% >}} 44 | 45 | The signer has a private key and a message to sign (for example, a 46 | Bitcoin transaction or a cat picture) and produces a signature. The 47 | verifier has the public key corresponding to the private key that the 48 | signer used, the message, and the signature. The verification process 49 | makes sure that the signature was created with the correct private 50 | key without knowing the private key. 51 | 52 | If you want a more in-depth explanation, please visit [the second half of chapter 2 of Grokking Bitcoin](https://rosenbaum.se/book/grokking-bitcoin-2.html#_digital_signatures). 53 | 54 | ### Signer 55 | 56 | We're now going to see how a Schnorr signature is created. It 57 | represents the left side of the diagram above. I assume that you're 58 | familiar with how a key pair is created using a random number 59 | generator and an elliptic curve. If not, I suggest you read 60 | [section 4.8 of Grokking Bitcoin](https://rosenbaum.se/book/grokking-bitcoin-4.html#public-key-math). 61 | 62 | Suppose that you want to make a signature for the cat picture. The 63 | picture is the message, $m$, to sign. Your private key is $p$ which belongs to your public key $P$. 64 | 65 | The first thing you'll do is to draw a random number, $r$, that we'll 66 | call the nonce (for "Number Once"). Then you'll treat $r$ as if it was 67 | a private key, which means that you can generate the corresponding 68 | public key by multiplying it with $G$, which is the _generator 69 | point_. At this stage, you have the following: 70 | 71 | $$ 72 | \begin{align} 73 | p & : \text{private key} \\\\ 74 | P = pG & : \text{public key} \\\\ 75 | m & : \text{message} \\\\ 76 | r & : \text{random nonce} \\\\ 77 | R = rG & : \text{nonce commitment} 78 | \end{align} 79 | $$ 80 | 81 | $R$ is your _nonce commitment_ which will become the first part of 82 | the final signature, and $r$ must remain secret 83 | ([explained later](#why-r-secret)) and never be reused (also 84 | [explained later](#nonce-reuse)). You've prepared everything you need 85 | to make the signature. You'll do it in two steps. Step 1 is to 86 | calculate a so-called _challenge hash_, $e$: 87 | 88 | $$ 89 | e = H(R || P || m) 90 | $$ 91 | 92 | The challenge hash is the hash of the challenge, which is the 93 | concatenation of $R$, $P$, and $m$. These components will all be 94 | available to the verifier. Using the challenge hash, you can now do step 2: 95 | Calculate the _challenge response_, or simply _response_, $s$, which 96 | is the second part of the signature (Thanks to Anthony Towns, 97 | Ruben Somsen, and nothingmuch on Twitter for 98 | [their help with the name _response_](https://twitter.com/kallerosenbaum/status/1472515231050080266).): 99 | 100 | $$ 101 | s = r + ep \tag{1} \label{respeqn} 102 | $$ 103 | 104 | Finally, your signature is 105 | 106 | $$ 107 | (R,s) 108 | $$ 109 | 110 | You send the cat picture, $m$, and your signature $(R,s)$ to your 111 | friend, Fred. 112 | 113 | ### Verifier 114 | 115 | Fred wants to make sure the cat picture hasn't been compromised during 116 | transfer and that it really originates from you, the only one with 117 | access to your private key $p$. He has access to $P$, $m$, $R$, and 118 | $s$. Of course, he also has access to $G$ because that's a widely 119 | known constant. From this information, he can calculate the challenge 120 | hash and verify that the _verification equation_ balances: 121 | 122 | $$ 123 | \begin{align} 124 | e & = H(R || P || m) \\\\ 125 | sG & = R + eP \tag{2} \label{vereqn} 126 | \end{align} 127 | $$ 128 | 129 | **If this equation balances, Fred can be sure that the signature was 130 | made with $p$**. Note that the verification equation, equation 131 | $(\ref{vereqn})$, is the response equation, $(\ref{respeqn})$, where 132 | both sides are multiplied by $G$. Starting with the response equation, 133 | we get 134 | 135 | $$ 136 | \begin{align} 137 | &s = r + ep \iff sG = (r+ep)G \\\\ 138 | &\iff sG = rG + epG \iff sG = R + eP 139 | \end{align} 140 | $$ 141 | 142 | As you can see: If the response equation holds, the verification 143 | equation holds. Likewise, if the verification equation holds, the 144 | response equation also holds. Thus, when Fred verifies the equation 145 | based on points on the elliptic curve, the verification equation, he 146 | also implicitly verifies that the response equation, based on scalars, 147 | holds. 148 | 149 | When Fred has verified the signature, he can enjoy the cat picture, 150 | fully confident that it's actually the same picture as you sent him. 151 | 152 | ## Why is $r$ secret? 153 | 154 | You might wonder why the nonce $r$ must be kept secret. You might even 155 | wonder why it's needed at all? Let's start with the latter. Let's remove $r$ from the process and see what happens. 156 | 157 | You would create the signature as follows: 158 | 159 | $$ 160 | \begin{array}{} 161 | e = H(P || m) \\\\ 162 | s = ep 163 | \end{array} 164 | $$ 165 | 166 | The signature would consist only of $s$. Fred would then verify your signature as: 167 | 168 | $$ 169 | \begin{array}{} 170 | e = H(P || m) \\\\ 171 | sG = eP 172 | \end{array} 173 | $$ 174 | 175 | That equation holds, but it would also allow Fred to extract the 176 | private key $p$ since he knows both $s$ and $e$. He'll take the 177 | response equation and solve it for p: 178 | 179 | $$ 180 | s = ep \iff p = \frac{s}{e} 181 | $$ 182 | 183 | OK, we need the nonce to prevent Fred from figuring out your private 184 | key, but why must we keep the nonce secret? Why the hassle of using 185 | the nonce commitment, instead of the nonce itself? 186 | 187 | It's for the same reason. Suppose that that the nonce, $r$, was made 188 | available to Fred, then he could figure out $p$ by: 189 | 190 | $$ 191 | \begin{array}{} 192 | e = H(R || P || m) \\\\ 193 | s = r + ep \iff p = \frac{s-r}{e} 194 | \end{array} 195 | $$ 196 | 197 | So Fred could calculate $p$ by subtracting $r$ from $s$ and dividing 198 | the result by $e$. 199 | 200 | By revealing just the nonce commitment, $R$, to Fred, we make sure 201 | that Fred can't calculate $p$ while at the same time allowing him to 202 | verify that $p$ was used to generate the signature. 203 | 204 | ## Don't reuse nonces 205 | 206 | Even if you keep the nonce secret, you may still leak your private key 207 | if you use the nonce twice for the same private key. Suppose that you 208 | make two signatures with the same nonce and private key as follows: 209 | 210 | $$ 211 | \begin{array}{lr} 212 | e = H(R || P || m) & e' = H(R || P || m')\\\\ 213 | s = r + ep & s' = r + e'p 214 | \end{array} 215 | $$ 216 | 217 | Then you give the signatures $(R,s)$ and $(R,s')$ to the verifier. The 218 | verifier can then use simple arithmetic to calculate your private 219 | key. He can set up an equation system with two equations and two 220 | unknown as follows: 221 | 222 | $$ 223 | \begin{align} 224 | s &= r + ep\\\\ 225 | s' &= r + e'p 226 | \end{align} 227 | $$ 228 | 229 | This is solvable for $p$ by subtracting $s'$ from $s$: 230 | 231 | $$ 232 | \begin{align} 233 | &s-s'=r+ep-r-e'p \\\\ 234 | &=(e-e')p \implies p=\frac{s-s'}{e-e'} 235 | \end{align} 236 | $$ 237 | 238 | As you can see, the verifier will be able to extract the private 239 | key. Lesson learned: Don't reuse nonces. 240 | 241 | If the nonce is reused, but for different private keys, $p$ and $p'$, 242 | the above equation system wouldn't be solvable because you'd have 243 | three unknowns, $p$, $p'$, and $r$, but only _two_ equations. 244 | 245 | ## What's with the challenge? 246 | 247 | The challenge hash, $e$, is the hash of the challenge $R||P||m$. Why 248 | do we use this particular challenge? Let's look at the three components separately. 249 | 250 | ### $m$ 251 | 252 | The message to sign is $m$, so it's really important that $m$ is 253 | somehow committed to by the signature. If we'd remove $m$ from the 254 | challenge, the "signature" would be valid for any message. 255 | 256 | ### $R$ 257 | 258 | (Thanks to [waxwing](https://x0f.org/@waxwing/107491024866468566) and 259 | [A J Towns](https://mastodon.social/@ajtowns/107491605500890702) for 260 | their help in sorting this out.) 261 | 262 | To make sure that no one but the owner of the private key can create a 263 | signature, the challenge must contain the nonce commitment 264 | $R$. Suppose that the challenge didn't include the nonce commitment, 265 | then a signature can be trivially forged by anyone with access to the 266 | public key $P$. They can make up an arbitrary $s$ and do: 267 | 268 | $$ 269 | \begin{align} 270 | &e = H(P||m)\\\\ 271 | &s = \text{any number} \\\\ 272 | &sG=R+eP \implies R=sG-eP 273 | \end{align} 274 | $$ 275 | 276 | The last equation is the "verification equation", but solved for 277 | $R$. The right side of that equation contains only known variables, 278 | $s$, $e$, and $P$. Thus the signature $(R,s)$ is valid. 279 | 280 | With $R$ in the challenge, it's impossible to solve the 281 | verification equation for $R$ because $R$ is part of the challenge 282 | $e$. It's hard to find an $R$ such that $R=sG-H(R||P||m)P$. 283 | 284 | ### $P$ 285 | 286 | Let's finally see what $P$ is doing in the challenge. Suppose that we 287 | didn't have $P$ in the challenge, $e=H(R||m)$, and that the signature 288 | $(R,s)$ is valid for public key $P$ and message $m$. Then the 289 | signature $(R,s')=(R,s+ex)$, where $x$ is an arbitrary number, would 290 | be valid for a public key $P'=P+xG$ and message $m$. Let's look at 291 | why: 292 | 293 | $$ 294 | \begin{align} 295 | &e = H(R || m) \\\\ 296 | &s'G=R+eP' \iff (s+ex)G=R+e(P+xG) \iff \\\\ 297 | &sG+exG=R+eP+exG \iff sG=R+eP+exG-exG \iff \\\\ 298 | &sG=R+eP 299 | \end{align} 300 | $$ 301 | 302 | This is known as a related-key attack. If you're familiar with how 303 | extended public key derivation in 304 | [BIP32](https://github.com/bitcoin/bips/blob/master/bip-0032.mediawiki) 305 | works, you might see the potential danger. For a refresher, here's the 306 | general idea: 307 | 308 | {{< center-figure 309 | src="/post-data/schnorr-basics/xpub-derivation.svg" 310 | caption="Deriving a child extended public key from a parent extended public key, The orange paper strips are known as chain code, described in detail in Grokking Bitcoin. You just need to know that they're 256-bit numbers." 311 | height=80% 312 | width=80% >}} 313 | 314 | This means that if an attacker knows the parent extended public key 315 | (xpub), and a valid signature for a child key, then the attacker can 316 | use this trick to forge signatures for the parent xpub, as well as any 317 | child xpubs that can be derived from the parent xpub. This is a 318 | problem not only for BIP32, but for many schemes using public key 319 | addition somehow, for example, Taproot 320 | ([BIP341](https://github.com/bitcoin/bips/blob/master/bip-0341.mediawiki#constructing-and-spending-taproot-outputs)). For 321 | a bit more details, please visit 322 | [the Design section of BIP340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki#Design). 323 | 324 | ## Next steps 325 | 326 | In the next post, I'll show how Schnorr signatures can be used in a 327 | multisignature setting to produce a signature that looks just like a 328 | normal single signature. This is very practical in Bitcoin because it 329 | reduces resource requirements for verifying the blockchain. 330 | 331 | 332 | _This is a cross-post from [my personal blog](https://popeller.io/schnorr-basics)._ 333 | -------------------------------------------------------------------------------- /layouts/_default/baseof.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | {{- partial "head.html" . -}} 4 | 5 | {{- partial "navbar.html" . -}} 6 |
7 | {{- block "main" . }}{{- end }} 8 |
9 | {{- partial "footer.html" . -}} 10 | 11 | 12 | -------------------------------------------------------------------------------- /layouts/_default/list.html: -------------------------------------------------------------------------------- 1 | {{ define "main" }} 2 |
3 |
4 |
5 |

{{.Title}}

6 | {{.Content}} 7 | {{ range .Pages }} 8 | {{ partial "card.html" . }} 9 | {{ end }} 10 |
11 |
12 |
13 | {{ end }} 14 | -------------------------------------------------------------------------------- /layouts/_default/rss.xml: -------------------------------------------------------------------------------- 1 | {{- $pctx := . -}} 2 | {{- if .IsHome -}}{{ $pctx = .Site }}{{- end -}} 3 | {{- $pages := $pctx.RegularPages -}} 4 | {{- $limit := .Site.Config.Services.RSS.Limit -}} 5 | {{- if ge $limit 1 -}} 6 | {{- $pages = $pages | first $limit -}} 7 | {{- end -}} 8 | {{- printf "" | safeHTML }} 9 | 10 | 11 | {{ if eq .Title .Site.Title }}{{ .Site.Title }}{{ else }}{{ with .Title }}{{.}} on {{ end }}{{ .Site.Title }}{{ end }} 12 | {{ .Permalink }} 13 | Recent content {{ if ne .Title .Site.Title }}{{ with .Title }}in {{.}} {{ end }}{{ end }}on {{ .Site.Title }} 14 | Hugo -- gohugo.io{{ with .Site.LanguageCode }} 15 | {{.}}{{end}}{{ with .Site.Author.email }} 16 | {{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}{{end}}{{ with .Site.Author.email }} 17 | {{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}{{end}}{{ with .Site.Copyright }} 18 | {{.}}{{end}}{{ if not .Date.IsZero }} 19 | {{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }}{{ end }} 20 | {{ with .OutputFormats.Get "RSS" }} 21 | {{ printf "" .Permalink .MediaType | safeHTML }} 22 | {{ end }} 23 | {{ range $pages }} 24 | 25 | {{ .Title }} 26 | {{ .Permalink }} 27 | {{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }} 28 | {{ with .Site.Author.email }}{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}{{end}} 29 | {{ .Permalink }} 30 | {{ .Content | html }} 31 | 32 | {{ end }} 33 | 34 | -------------------------------------------------------------------------------- /layouts/_default/single.html: -------------------------------------------------------------------------------- 1 | {{ define "main" }} 2 |
3 |
4 |
5 |
6 | 7 | 8 |

9 | {{ .Title }} 10 |

11 |
{{ .Params.Subtitle }}
12 | 13 | published {{ .PublishDate.Format "January 2, 2006" }} 14 | 15 | 16 | by 17 | {{ range $index, $author := .Params.authors }} 18 | {{ if (gt $index 0) }} and {{ end }} 19 | {{ .name }} 20 | {{ if isset . "github" }} 21 | 22 | link to @{{ .github }} on GitHub 23 | 24 | {{ end }} 25 | {{ if isset . "twitter" }} 26 | 27 | link to @{{ .twitter }} on Twitter 28 | 29 | {{ end }} 30 | {{ end }} 31 | 32 | {{ with .Params.tags }} 33 |
34 | 35 | tagged with 36 | {{ $last_tag_index := sub (len . ) 1 }} 37 | {{ range $index, $tag := .}} 38 | {{ . }}{{ if (ne $index ($last_tag_index)) }}, {{ end }} 39 | {{ end }} 40 | 41 | {{ end }} 42 | {{ with .Params.appearedfirston }} 43 |
44 | 45 | appeared first on {{ .label }} 46 | 47 | {{ end }} 48 |
49 |
50 |
51 | {{ .Content }} 52 |
53 | 54 |
55 | 61 | 69 |
70 |
71 | All text and images in this work are licensed under a Creative Commons Attribution-ShareAlike 4.0 International License 72 | Creative Commons License 73 |
74 |
75 |
76 |
77 | {{ partial "prev-next.html" . }} 78 | {{ end }} 79 | -------------------------------------------------------------------------------- /layouts/index.html: -------------------------------------------------------------------------------- 1 | {{ define "main" }} 2 | 3 |
4 |
5 |
6 | {{.Content}} 7 |
8 |
9 |
10 |

Recent Posts

11 | 12 | {{ $pages := (.Site.GetPage "/blog").Pages }} 13 | {{ range (sort $pages "Params.date" "desc")}} 14 | {{ partial "card.html" . }} 15 | {{ end }} 16 |
17 |
18 |
19 | {{ end }} 20 | -------------------------------------------------------------------------------- /layouts/partials/card-small.html: -------------------------------------------------------------------------------- 1 |
2 |
3 |
4 | {{ if .Params.images }} 5 | Image for {{.Title}} 6 | {{ end }} 7 | 8 |
9 |
10 |
11 | 12 | {{ .PublishDate.Format "January 2, 2006" }} 13 | 14 |

{{.Title}} 15 |

16 |

17 | {{ .Summary }} 18 |

19 |
20 | 21 |
22 |
23 |
24 | -------------------------------------------------------------------------------- /layouts/partials/card.html: -------------------------------------------------------------------------------- 1 |
2 |
3 | {{ if .Params.images }} 4 |
5 | Image for {{.Title}} 6 | 7 |
8 | {{ end }} 9 |
10 |
11 | {{ if isset . "PublishDate" }} 12 | 13 | {{ .PublishDate.Format "January 2, 2006" }} 14 | 15 | {{ end }} 16 |

{{.Title}} 17 |

18 |

19 | {{ truncate 300 .Summary }} 20 |

21 |
22 | 23 |
24 |
25 |
26 | -------------------------------------------------------------------------------- /layouts/partials/footer.html: -------------------------------------------------------------------------------- 1 | 24 | -------------------------------------------------------------------------------- /layouts/partials/head.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | {{.Title}} 7 | 8 | {{ template "_internal/opengraph.html" . }} {{ template "_internal/twitter_cards.html" . }} 9 | 10 | {{ range .AlternativeOutputFormats -}} 11 | {{ printf `` .Rel .MediaType.Type .Permalink $.Site.Title | safeHTML }} 12 | {{ end -}} 13 | 14 | 16 | 17 | 18 | 19 | 20 | 21 | {{ if not .Site.IsServer }} 22 | 23 | 26 | {{ end }} 27 | 28 | {{ if .Params.mathjax }}{{ partial "mathjax_support.html" . }}{{ end }} 29 | 30 | 31 | -------------------------------------------------------------------------------- /layouts/partials/mathjax_support.html: -------------------------------------------------------------------------------- 1 | 20 | 21 | 22 | 23 | -------------------------------------------------------------------------------- /layouts/partials/navbar.html: -------------------------------------------------------------------------------- 1 | 56 | -------------------------------------------------------------------------------- /layouts/partials/prev-next.html: -------------------------------------------------------------------------------- 1 |
2 |
3 | 4 | {{ if (.NextInSection) }} 5 |
6 |

Next

7 | {{ if and (.PrevInSection) (.NextInSection) }} 8 | {{ partial "card-small.html" .NextInSection }} 9 | {{ else }} 10 | {{ partial "card.html" .NextInSection }} 11 | {{ end }} 12 |
13 | {{ end }} 14 | 15 | {{ if (.PrevInSection) }} 16 |
17 |

Previous

18 | {{ if and (.PrevInSection) (.NextInSection) }} 19 | {{ partial "card-small.html" .PrevInSection }} 20 | {{ else }} 21 | {{ partial "card.html" .PrevInSection }} 22 | {{ end }} 23 |
24 | {{ end }} 25 | 26 |
27 |
-------------------------------------------------------------------------------- /layouts/shortcodes/center-figure.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | {{- if .Get "link" -}} 5 | 6 | {{- end -}} 7 | {{ with .Get 14 | {{- if .Get "link" }}{{ end -}} 15 | {{- if or (or (.Get "title") (.Get "caption")) (.Get "attr") -}} 16 |
17 | {{ with (.Get "title") -}} 18 |

{{ . }}

19 | {{- end -}} 20 | {{- if or (.Get "caption") (.Get "attr") -}}

21 |

22 | {{- .Get "caption" | markdownify -}} 23 | {{- with .Get "attrlink" }} 24 | 25 | {{- end -}} 26 | {{- .Get "attr" | markdownify -}} 27 | {{- if .Get "attrlink" }}{{ end }} 28 |
29 |

30 | {{- end }} 31 |
32 | {{- end }} 33 | 34 | -------------------------------------------------------------------------------- /layouts/tags/list.html: -------------------------------------------------------------------------------- 1 | {{ define "main" }} 2 |
3 |
4 |
5 |

Posts tagged with "{{.Title}}"

6 | {{.Content}} 7 | {{ range .Pages }} 8 | {{ partial "card.html" . }} 9 | {{ end }} 10 |
11 |
12 |
13 | {{ end }} 14 | -------------------------------------------------------------------------------- /static/CNAME: -------------------------------------------------------------------------------- 1 | bitcoin-dev.blog 2 | -------------------------------------------------------------------------------- /static/bitcoin.pdf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/bitcoin.pdf -------------------------------------------------------------------------------- /static/css/index.css: -------------------------------------------------------------------------------- 1 | :root { 2 | --body-bg: var(--bs-light); 3 | --body-color: var(--bs-dark); 4 | --article-bg: #00000007; 5 | --transparent-img-bg: #0000; 6 | } 7 | 8 | html.light { 9 | --body-bg: var(--bs-light); 10 | --body-color: var(--bs-dark); 11 | --article-bg: #00000007; 12 | --transparent-img-bg: #0000; 13 | } 14 | html.dark { 15 | --body-bg: var(--bs-dark); 16 | --body-color: var(--bs-light); 17 | --article-bg: #ffffff07; 18 | --transparent-img-bg: #fffd; 19 | } 20 | 21 | .dark-light-mode-symbol { 22 | font-size: 1.4em; 23 | color: var(--body-color); 24 | } 25 | 26 | html.dark pre code { 27 | color: var(--bs-pink); 28 | } 29 | html.dark .image-to-invert { 30 | filter: invert(100%); 31 | } 32 | html.dark a { 33 | color: var(--bs-orange); 34 | } 35 | 36 | body { 37 | font-family: sans-serif; 38 | background: var(--body-bg); 39 | color: var(--body-color); 40 | } 41 | 42 | a { 43 | text-decoration: none; 44 | } 45 | a.navbar-brand { 46 | color: var(--body-color) !important; 47 | } 48 | 49 | #light-dark-mode-switch.form-check-input:checked { 50 | background-color: var(--bs-dark); 51 | border-color: var(--bs-gray); 52 | } 53 | 54 | .content-text p { 55 | font-size: 1em; 56 | text-align: justify; 57 | } 58 | 59 | article, .card, nav { 60 | background: var(--article-bg); 61 | } 62 | 63 | h1, h2, h3, h4, h5, h6 { 64 | font-family: "Ubuntu"; 65 | font-weight: 600; 66 | } 67 | 68 | .twitter-tweet-rendered { 69 | display:block; 70 | margin-left:auto; 71 | margin-right:auto; 72 | } 73 | 74 | .highlight pre { 75 | padding: 1rem; 76 | } 77 | 78 | .figure-center-img { 79 | background-color: var(--transparent-img-bg); 80 | } 81 | 82 | /* MathJax */ 83 | 84 | code.has-jax { 85 | -webkit-font-smoothing: antialiased; 86 | background: inherit !important; 87 | border: none !important; 88 | font-size: 100%; 89 | } 90 | 91 | -------------------------------------------------------------------------------- /static/css/syntax.css: -------------------------------------------------------------------------------- 1 | /* Background */ .chroma { background-color: #ffffff } 2 | /* Error */ .chroma .err { color: #a61717; background-color: #e3d2d2 } 3 | /* LineTableTD */ .chroma .lntd { vertical-align: top; padding: 0; margin: 0; border: 0; } 4 | /* LineTable */ .chroma .lntable { border-spacing: 0; padding: 0; margin: 0; border: 0; width: auto; overflow: auto; display: block; } 5 | /* LineHighlight */ .chroma .hl { display: block; width: 100%;background-color: #ffffcc } 6 | /* LineNumbersTable */ .chroma .lnt { margin-right: 0.4em; padding: 0 0.4em 0 0.4em;color: #7f7f7f } 7 | /* LineNumbers */ .chroma .ln { margin-right: 0.4em; padding: 0 0.4em 0 0.4em;color: #7f7f7f } 8 | /* Keyword */ .chroma .k { color: #008800; font-weight: bold } 9 | /* KeywordConstant */ .chroma .kc { color: #008800; font-weight: bold } 10 | /* KeywordDeclaration */ .chroma .kd { color: #008800; font-weight: bold } 11 | /* KeywordNamespace */ .chroma .kn { color: #008800; font-weight: bold } 12 | /* KeywordPseudo */ .chroma .kp { color: #008800 } 13 | /* KeywordReserved */ .chroma .kr { color: #008800; font-weight: bold } 14 | /* KeywordType */ .chroma .kt { color: #888888; font-weight: bold } 15 | /* NameAttribute */ .chroma .na { color: #336699 } 16 | /* NameBuiltin */ .chroma .nb { color: #003388 } 17 | /* NameClass */ .chroma .nc { color: #bb0066; font-weight: bold } 18 | /* NameConstant */ .chroma .no { color: #003366; font-weight: bold } 19 | /* NameDecorator */ .chroma .nd { color: #555555 } 20 | /* NameException */ .chroma .ne { color: #bb0066; font-weight: bold } 21 | /* NameFunction */ .chroma .nf { color: #0066bb; font-weight: bold } 22 | /* NameLabel */ .chroma .nl { color: #336699; font-style: italic } 23 | /* NameNamespace */ .chroma .nn { color: #bb0066; font-weight: bold } 24 | /* NameProperty */ .chroma .py { color: #336699; font-weight: bold } 25 | /* NameTag */ .chroma .nt { color: #bb0066; font-weight: bold } 26 | /* NameVariable */ .chroma .nv { color: #336699 } 27 | /* NameVariableClass */ .chroma .vc { color: #336699 } 28 | /* NameVariableGlobal */ .chroma .vg { color: #dd7700 } 29 | /* NameVariableInstance */ .chroma .vi { color: #3333bb } 30 | /* LiteralString */ .chroma .s { color: #dd2200; background-color: #fff0f0 } 31 | /* LiteralStringAffix */ .chroma .sa { color: #dd2200; background-color: #fff0f0 } 32 | /* LiteralStringBacktick */ .chroma .sb { color: #dd2200; background-color: #fff0f0 } 33 | /* LiteralStringChar */ .chroma .sc { color: #dd2200; background-color: #fff0f0 } 34 | /* LiteralStringDelimiter */ .chroma .dl { color: #dd2200; background-color: #fff0f0 } 35 | /* LiteralStringDoc */ .chroma .sd { color: #dd2200; background-color: #fff0f0 } 36 | /* LiteralStringDouble */ .chroma .s2 { color: #dd2200; background-color: #fff0f0 } 37 | /* LiteralStringEscape */ .chroma .se { color: #0044dd; background-color: #fff0f0 } 38 | /* LiteralStringHeredoc */ .chroma .sh { color: #dd2200; background-color: #fff0f0 } 39 | /* LiteralStringInterpol */ .chroma .si { color: #3333bb; background-color: #fff0f0 } 40 | /* LiteralStringOther */ .chroma .sx { color: #22bb22; background-color: #f0fff0 } 41 | /* LiteralStringRegex */ .chroma .sr { color: #008800; background-color: #fff0ff } 42 | /* LiteralStringSingle */ .chroma .s1 { color: #dd2200; background-color: #fff0f0 } 43 | /* LiteralStringSymbol */ .chroma .ss { color: #aa6600; background-color: #fff0f0 } 44 | /* LiteralNumber */ .chroma .m { color: #0000dd; font-weight: bold } 45 | /* LiteralNumberBin */ .chroma .mb { color: #0000dd; font-weight: bold } 46 | /* LiteralNumberFloat */ .chroma .mf { color: #0000dd; font-weight: bold } 47 | /* LiteralNumberHex */ .chroma .mh { color: #0000dd; font-weight: bold } 48 | /* LiteralNumberInteger */ .chroma .mi { color: #0000dd; font-weight: bold } 49 | /* LiteralNumberIntegerLong */ .chroma .il { color: #0000dd; font-weight: bold } 50 | /* LiteralNumberOct */ .chroma .mo { color: #0000dd; font-weight: bold } 51 | /* OperatorWord */ .chroma .ow { color: #008800 } 52 | /* Comment */ .chroma .c { color: #888888 } 53 | /* CommentHashbang */ .chroma .ch { color: #888888 } 54 | /* CommentMultiline */ .chroma .cm { color: #888888 } 55 | /* CommentSingle */ .chroma .c1 { color: #888888 } 56 | /* CommentSpecial */ .chroma .cs { color: #cc0000; background-color: #fff0f0; font-weight: bold } 57 | /* CommentPreproc */ .chroma .cp { color: #cc0000; font-weight: bold } 58 | /* CommentPreprocFile */ .chroma .cpf { color: #cc0000; font-weight: bold } 59 | /* GenericDeleted */ .chroma .gd { color: #000000; background-color: #ffdddd } 60 | /* GenericEmph */ .chroma .ge { font-style: italic } 61 | /* GenericError */ .chroma .gr { color: #aa0000 } 62 | /* GenericHeading */ .chroma .gh { color: #333333 } 63 | /* GenericInserted */ .chroma .gi { color: #000000; background-color: #ddffdd } 64 | /* GenericOutput */ .chroma .go { color: #888888 } 65 | /* GenericPrompt */ .chroma .gp { color: #555555 } 66 | /* GenericStrong */ .chroma .gs { font-weight: bold } 67 | /* GenericSubheading */ .chroma .gu { color: #666666 } 68 | /* GenericTraceback */ .chroma .gt { color: #aa0000 } 69 | /* GenericUnderline */ .chroma .gl { text-decoration: underline } 70 | /* TextWhitespace */ .chroma .w { color: #bbbbbb } 71 | -------------------------------------------------------------------------------- /static/favicon.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/favicon.png -------------------------------------------------------------------------------- /static/img/footer/github.svg: -------------------------------------------------------------------------------- 1 | GitHub icon 2 | -------------------------------------------------------------------------------- /static/img/footer/gmail.svg: -------------------------------------------------------------------------------- 1 | Gmail icon 2 | -------------------------------------------------------------------------------- /static/img/footer/rss.svg: -------------------------------------------------------------------------------- 1 | RSS icon 2 | -------------------------------------------------------------------------------- /static/img/footer/twitter.svg: -------------------------------------------------------------------------------- 1 | Twitter icon 2 | -------------------------------------------------------------------------------- /static/img/logo-512.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/img/logo-512.png -------------------------------------------------------------------------------- /static/img/logo-large.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/img/logo-large.png -------------------------------------------------------------------------------- /static/img/og-image.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/img/og-image.png -------------------------------------------------------------------------------- /static/post-data/compact-block-filters/bitcoinExample.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/post-data/compact-block-filters/bitcoinExample.png -------------------------------------------------------------------------------- /static/post-data/compact-block-filters/cover.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/post-data/compact-block-filters/cover.png -------------------------------------------------------------------------------- /static/post-data/compact-block-filters/dense.jpeg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/post-data/compact-block-filters/dense.jpeg -------------------------------------------------------------------------------- /static/post-data/compact-block-filters/quotient.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/post-data/compact-block-filters/quotient.png -------------------------------------------------------------------------------- /static/post-data/compact-block-filters/remainder.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/post-data/compact-block-filters/remainder.png -------------------------------------------------------------------------------- /static/post-data/compact-block-filters/sparse.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/post-data/compact-block-filters/sparse.png -------------------------------------------------------------------------------- /static/post-data/example-post/header.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/post-data/example-post/header.png -------------------------------------------------------------------------------- /static/post-data/example-post/header.svg: -------------------------------------------------------------------------------- 1 | 2 | 20 | 22 | 28 | 29 | 49 | 51 | 52 | 54 | image/svg+xml 55 | 57 | 58 | 59 | 60 | 61 | 65 | Example 75 | Post 79 | 80 | 81 | -------------------------------------------------------------------------------- /static/post-data/schnorr-basics/sig-overview.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/post-data/schnorr-basics/sig-overview.png -------------------------------------------------------------------------------- /static/post-data/usdt-support-core/header.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/demining/dev-bitcoin-Google-Colab/ac836f1514dc02246fc9eb6bdc98fd06935904da/static/post-data/usdt-support-core/header.png -------------------------------------------------------------------------------- /static/robots.txt: -------------------------------------------------------------------------------- 1 | User-Agent: * 2 | Allow: / 3 | -------------------------------------------------------------------------------- /svg-source/logo.svg: -------------------------------------------------------------------------------- 1 | 2 | 19 | 21 | 22 | 24 | image/svg+xml 25 | 27 | 28 | 29 | 30 | 31 | 33 | 54 | 57 | 60 | 63 | 68 | itcoin 78 | 79 | dev 89 | blog 99 | 100 | 101 | 102 | -------------------------------------------------------------------------------- /svg-source/og-image.svg: -------------------------------------------------------------------------------- 1 | 2 | 19 | 21 | 22 | 24 | image/svg+xml 25 | 27 | 28 | 29 | 30 | 31 | 33 | 54 | 57 | 64 | 67 | 70 | 75 | itcoin 85 | 86 | dev 96 | blog 106 | 107 | 108 | 109 | --------------------------------------------------------------------------------