15 |
16 | [mempool.observer](https://mempool.observer) visualizes various statistics around my Bitcoin memory pool (mempool).
17 | Seemingly stuck and longtime-unconfirmed transactions can be quite annoying for users transacting on the Bitcoin network.
18 | The idea of mempool.observer is to provide users with information about unconfirmed transactions and transaction fees.
19 |
20 | ## Project Structure
21 |
22 | Folder Structure
23 | ```
24 | memo/
25 | ├── api/ # Go code that compiles to a binary API returning JSON from a Redis instance
26 | ├── memod/ # Go code that compiles to a binary worker daemon writing data to a Redis instance
27 | └── www/ # Statically served HTML, JS and CSS files
28 | ```
29 |
30 | There exists a overview of my [infrastructure setup](https://www.plectica.com/maps/RCXWDOYD9) for mempool.observer.
31 |
32 | ## Project History
33 |
34 | I've started building the first version of mempool.observer mid 2017 as my first Bitcoin related project.
35 | I was (and still am) motivated by presumably Greg Maxwell's words:
36 |
37 | >"What's going to happen to Bitcoin?" is the wrong question. The right question is "What are you going to contribute?" — [Greg Maxwell](https://github.com/gmaxwell)
38 |
39 | Later this year the bitcoin transaction fees rose and I had quite some traffic.
40 | The high fees were caused by a huge transaction flood as the price rose to $20k.
41 | I regularly had problems with long running scripts due to querying and processing the huge mempool on a low end VPS.
42 | Due to time constrains I wasn't able to improve the performance.
43 | This resulted in mempool.observer version 1 dieing the not-maintained-death sometime in 2018.
44 |
45 | I've focused full time on Bitcoin in spring 2019 and spend a part of that time to work on version 2.
46 | Version 2 is a full rewrite of mempool.observer - only the idea, license and the quote from Maxwell remained.
47 | The goal is to offer way more than version 1 did, but build on a foundation with performance and maintainability in mind.
48 | I'm open for ideas and feedback.
49 |
50 | ## Self-Hosting
51 |
52 | Self-hosting memo is possible, but there is no detailed setup and maintenance documentation written yet.
53 | You might need to do some exploration on your own to get everything working.
54 | Keep in mind, that you need a customized Bitcoin Core version to run the Bitcoin Transaction Monitor.
55 |
56 | ## Licence
57 | This project and all it's files are licensed under a GNU Affero General Public License.
58 |
59 |
60 | ----
61 |
62 | | | Donation Address |
63 | | --- | --- |
64 | | ♥ __BTC__ | 1Lw2kh9WzCActXSGHxyypGLkqQZfxDpw8v |
65 | | ♥ __ETH__ | 0xaBd66CF90898517573f19184b3297d651f7b90bf |
66 |
--------------------------------------------------------------------------------
/api/api.go:
--------------------------------------------------------------------------------
1 | package main
2 |
3 | import (
4 | "encoding/json"
5 | "fmt"
6 | "net/http"
7 | "strconv"
8 |
9 | "github.com/0xb10c/memo/api/cache"
10 | "github.com/0xb10c/memo/api/config"
11 | "github.com/0xb10c/memo/api/database"
12 | "github.com/gin-contrib/cors"
13 | "github.com/gin-contrib/gzip"
14 | "github.com/gin-gonic/gin"
15 | )
16 |
17 | func main() {
18 |
19 | if config.GetBool("api.production") {
20 | gin.SetMode(gin.ReleaseMode)
21 | }
22 |
23 | router := gin.Default()
24 | corsConfig := cors.DefaultConfig()
25 | if config.GetBool("api.production") {
26 | corsConfig.AllowOrigins = []string{"https://mempool.observer/"}
27 | } else {
28 | corsConfig.AllowOrigins = []string{"*"}
29 | }
30 |
31 | router.Use(cors.New(corsConfig))
32 | router.Use(gzip.Gzip(gzip.DefaultCompression))
33 |
34 | err := database.SetupDatabase()
35 | if err != nil {
36 | panic(fmt.Errorf("Failed to setup database: %v", err))
37 | }
38 |
39 | go cache.SetupCache()
40 |
41 | api := router.Group("/api")
42 | {
43 | api.GET("/mempool", getMempool)
44 | api.GET("/recentBlocks", getRecentBlocks)
45 | api.GET("/historicalMempool/:timeframe/:by", getHistoricalMempool)
46 | api.GET("/transactionStats", getTransactionStats)
47 | api.GET("/getMempoolEntries", getCachedMempoolEntries)
48 | api.GET("/getRecentFeerateAPIData", getRecentFeerateAPIEntries)
49 | api.GET("/getBlockEntries", getBlockEntries)
50 | }
51 |
52 | portString := ":" + config.GetString("api.port")
53 | router.Run(portString)
54 | }
55 |
56 | func getMempool(c *gin.Context) {
57 |
58 | timestamp, byCount, megabyteMarkersJSON, mempoolSize, err := database.GetMempool()
59 | if err != nil {
60 | fmt.Println(err.Error())
61 | c.JSON(http.StatusInternalServerError, gin.H{
62 | "error": "Database error",
63 | })
64 | return
65 | }
66 |
67 | // Possible REFACTOR: write to database as blob not JSON String to
68 | // skip the marshalling when writing and unmarshalling when reading
69 | // from the database
70 | var feerateMap map[int]int
71 | json.Unmarshal([]byte(byCount), &feerateMap)
72 |
73 | var megabyteMarkers []int
74 | json.Unmarshal([]byte(megabyteMarkersJSON), &megabyteMarkers)
75 |
76 | c.JSON(http.StatusOK, gin.H{
77 | "timestamp": timestamp.Unix(),
78 | "feerateMap": feerateMap,
79 | "megabyteMarkers": megabyteMarkers,
80 | "mempoolSize": mempoolSize,
81 | })
82 | }
83 |
84 | func getRecentBlocks(c *gin.Context) {
85 |
86 | blocks, err := database.GetRecentBlocks()
87 | if err != nil {
88 | fmt.Println(err.Error())
89 | c.JSON(http.StatusInternalServerError, gin.H{
90 | "error": "Database error",
91 | })
92 | return
93 | }
94 |
95 | c.JSON(http.StatusOK, blocks)
96 | }
97 |
98 | func getBlockEntries(c *gin.Context) {
99 | blocks, err := database.GetBlockEntries()
100 | if err != nil {
101 | fmt.Println(err.Error())
102 | c.JSON(http.StatusInternalServerError, gin.H{
103 | "error": "Database error",
104 | })
105 | return
106 | }
107 |
108 | c.JSON(http.StatusOK, blocks)
109 | }
110 |
111 | func getHistoricalMempool(c *gin.Context) {
112 | timeframe, err := strconv.ParseInt(c.Param("timeframe"), 10, 0)
113 | if err != nil {
114 | fmt.Println(err.Error())
115 | c.JSON(http.StatusInternalServerError, gin.H{
116 | "error": "Invalid input error",
117 | })
118 | return
119 | }
120 |
121 | by := c.Param("by")
122 | if by != "byCount" && by != "byFee" && by != "bySize" {
123 | c.JSON(http.StatusInternalServerError, gin.H{
124 | "error": "Invalid input error",
125 | })
126 | return
127 | }
128 |
129 | mempoolStates, err := database.GetHistorical(int(timeframe), by)
130 | if err != nil {
131 | fmt.Println(err.Error())
132 | c.JSON(http.StatusInternalServerError, gin.H{
133 | "error": "Database error",
134 | })
135 | return
136 | }
137 |
138 | c.JSON(http.StatusOK, mempoolStates)
139 | }
140 |
141 | func getTransactionStats(c *gin.Context) {
142 | tss, err := database.GetTransactionStats()
143 | if err != nil {
144 | fmt.Println(err.Error())
145 | c.JSON(http.StatusInternalServerError, gin.H{
146 | "error": "Database error",
147 | })
148 | return
149 | }
150 |
151 | c.JSON(http.StatusOK, tss)
152 | }
153 |
154 | func getMempoolEntries(c *gin.Context) {
155 | mes, err := database.GetMempoolEntries()
156 | if err != nil {
157 | fmt.Println(err.Error())
158 | c.JSON(http.StatusInternalServerError, gin.H{
159 | "error": "Database error",
160 | })
161 | return
162 | }
163 |
164 | c.JSON(http.StatusOK, mes)
165 | }
166 |
167 | func getRecentFeerateAPIEntries(c *gin.Context) {
168 |
169 | entries, err := database.GetRecentFeerateAPIEntries()
170 | if err != nil {
171 | fmt.Println(err.Error())
172 | c.JSON(http.StatusInternalServerError, gin.H{
173 | "error": "Database error",
174 | })
175 | return
176 | }
177 |
178 | c.JSON(http.StatusOK, entries)
179 | }
180 |
181 | func getCachedMempoolEntries(c *gin.Context) {
182 | entries, err := database.GetMempoolEntriesCache()
183 | if err != nil {
184 | fmt.Println(err.Error())
185 | c.JSON(http.StatusInternalServerError, gin.H{
186 | "error": "Database error",
187 | })
188 | return
189 | }
190 | c.Header("Content-Type", "application/json; charset=utf-8")
191 | c.String(http.StatusOK, entries)
192 | }
193 |
--------------------------------------------------------------------------------
/api/cache/cache.go:
--------------------------------------------------------------------------------
1 | package cache
2 |
3 | import (
4 | "encoding/json"
5 | "log"
6 |
7 | "github.com/0xb10c/memo/api/database"
8 | "github.com/jasonlvhit/gocron"
9 | )
10 |
11 | // SetupCache sets up caching scripts
12 | func SetupCache() {
13 | SetupMempoolEntriesCacher()
14 | }
15 |
16 | // SetupMempoolEntriesCacher sets up a periodic GetMempoolEntries() fetch job
17 | func SetupMempoolEntriesCacher() {
18 | cacheMempoolEntries()
19 |
20 | fetchInterval := uint64(30)
21 | s := gocron.NewScheduler()
22 | s.Every(fetchInterval).Seconds().Do(cacheMempoolEntries)
23 | log.Printf("Setup GetMempoolEntries() cacher to run every %d seconds.\n", fetchInterval)
24 | <-s.Start()
25 | defer s.Clear()
26 | }
27 |
28 | func cacheMempoolEntries() {
29 | log.Printf("Caching mempool entries.\n")
30 | mes, err := database.GetMempoolEntries()
31 | if err != nil {
32 | log.Printf("Error getting mempool entries %v.\n", err)
33 | }
34 |
35 | mesJSON, err := json.Marshal(mes)
36 | if err != nil {
37 | log.Printf("Error marshalling mempool entries %v.\n", err)
38 | }
39 |
40 | err = database.SetMempoolEntriesCache(string(mesJSON))
41 | if err != nil {
42 | log.Printf("Could not cache mempool entries %v.\n", err)
43 | }
44 |
45 | }
46 |
--------------------------------------------------------------------------------
/api/config.example.toml:
--------------------------------------------------------------------------------
1 | [api]
2 | port = "23485" # this needs to be a string
3 | production = true
4 |
5 |
6 | [database]
7 | host = "" # FIXME:
8 | user = "" # FIXME:
9 | passwd = "" # FIXME:
10 | name = "" # FIXME:
11 | connection = "tcp"
12 |
--------------------------------------------------------------------------------
/api/config/config.go:
--------------------------------------------------------------------------------
1 | package config
2 |
3 | import (
4 | "fmt"
5 |
6 | "github.com/spf13/viper"
7 | )
8 |
9 | // logger/logger.go can't be used here to log,
10 | // since logger.go itself depends on the config
11 | // to read `log.enableTrace`.
12 |
13 | func init() {
14 | setDefaults()
15 |
16 | viper.SetConfigType("toml")
17 | viper.SetConfigName("config")
18 | viper.AddConfigPath(".")
19 | err := viper.ReadInConfig()
20 | if err != nil {
21 | fmt.Println("ERROR: Error reading config file: ", err)
22 | }
23 | }
24 |
25 | func setDefaults() {
26 | viper.SetDefault("redis.host", "localhost")
27 | viper.SetDefault("redis.connection", "redis")
28 | viper.SetDefault("api.port", "23485")
29 | viper.SetDefault("api.production", true)
30 | }
31 |
32 | // GetInt returns a config property as int
33 | func GetInt(property string) int {
34 | result := viper.GetInt(property)
35 | if result == 0 {
36 | fmt.Println("WARN: Property " + property + " is 0. Is this not set?")
37 | }
38 | return result
39 | }
40 |
41 | // GetString returns a config property as string
42 | func GetString(property string) string {
43 | result := viper.GetString(property)
44 | if result == "" {
45 | fmt.Println("WARN: Property " + property + " is \"\". Is this not set?")
46 | }
47 | return result
48 | }
49 |
50 | // GetBool returns a config property as bool
51 | func GetBool(property string) bool {
52 | return viper.GetBool(property)
53 | }
54 |
--------------------------------------------------------------------------------
/api/database/database.go:
--------------------------------------------------------------------------------
1 | package database
2 |
3 | import (
4 | "errors"
5 | "strconv"
6 | "time"
7 |
8 | "github.com/0xb10c/memo/api/config"
9 | "github.com/0xb10c/memo/api/types"
10 | "github.com/gomodule/redigo/redis"
11 | jsoniter "github.com/json-iterator/go"
12 | )
13 |
14 | var Pool *redis.Pool
15 | var json = jsoniter.ConfigCompatibleWithStandardLibrary
16 |
17 | func newPool() *redis.Pool {
18 | dbUser := config.GetString("redis.user")
19 | dbPasswd := config.GetString("redis.passwd")
20 | dbHost := config.GetString("redis.host")
21 | dbPort := config.GetString("redis.port")
22 | dbConnection := config.GetString("redis.connection")
23 |
24 | connectionString := dbConnection + "://" + dbUser + ":" + dbPasswd + "@" + dbHost + ":" + dbPort
25 |
26 | return &redis.Pool{
27 | MaxIdle: 80,
28 | MaxActive: 12000, // max number of connections
29 | Dial: func() (redis.Conn, error) {
30 |
31 | c, err := redis.DialURL(connectionString)
32 | if err != nil {
33 | return c, err
34 | }
35 | return c, err
36 | },
37 | }
38 | }
39 |
40 | func SetupDatabase() error {
41 | Pool = newPool()
42 |
43 | c := Pool.Get() // get a new connection
44 | defer c.Close()
45 |
46 | _, err := c.Do("PING")
47 | if err != nil {
48 | return err
49 | }
50 |
51 | return nil
52 | }
53 |
54 | // GetMempool gets the current mempool from the database
55 | func GetMempool() (timestamp time.Time, feerateMapJSON string, megabyteMarkersJSON string, mempoolSize int, err error) {
56 | c := Pool.Get()
57 | defer c.Close()
58 |
59 | prefix := "currentMempool"
60 |
61 | response, err := redis.Strings(c.Do("MGET", prefix+":utcTimestamp", prefix+":feerateMap", prefix+":megabyteMarkers", prefix+":mempoolSizeInByte"))
62 | if err != nil {
63 | return
64 | }
65 |
66 | if n, err := strconv.Atoi(response[0]); err == nil {
67 | timestamp = time.Unix(int64(n), 0)
68 | } else {
69 | timestamp = time.Unix(0, 0)
70 | }
71 |
72 | feerateMapJSON = response[1]
73 | megabyteMarkersJSON = response[2]
74 |
75 | if n, err := strconv.Atoi(response[3]); err == nil {
76 | mempoolSize = n
77 | } else {
78 | mempoolSize = 0
79 | }
80 |
81 | return
82 | }
83 |
84 | // GetRecentBlocks returns the 10 most recent blocks.
85 | func GetRecentBlocks() (blocks []types.RecentBlock, err error) {
86 | c := Pool.Get()
87 | defer c.Close()
88 |
89 | reJSON, err := redis.Strings(c.Do("LRANGE", "recentBlocks", 0, 9))
90 | if err != nil {
91 | return
92 | }
93 |
94 | blocks = make([]types.RecentBlock, 0)
95 | for index := range reJSON {
96 | block := types.RecentBlock{}
97 | err = json.Unmarshal([]byte(reJSON[index]), &block)
98 | if err != nil {
99 | return
100 | }
101 | blocks = append(blocks, block)
102 | }
103 |
104 | return
105 | }
106 |
107 |
108 | // GetBlockEntries returns the 20 most recent blocks with short TXIDs.
109 | func GetBlockEntries() (blocks []types.BlockEntry, err error) {
110 | c := Pool.Get()
111 | defer c.Close()
112 |
113 | beJSON, err := redis.Strings(c.Do("LRANGE", "blockEntries", 0, 20))
114 | if err != nil {
115 | return
116 | }
117 |
118 | blocks = make([]types.BlockEntry, 0)
119 | for index := range beJSON {
120 | block := types.BlockEntry{}
121 | err = json.Unmarshal([]byte(beJSON[index]), &block)
122 | if err != nil {
123 | return
124 | }
125 | blocks = append(blocks, block)
126 | }
127 |
128 | return
129 | }
130 |
131 | type MempoolState struct {
132 | time time.Time
133 | Timestamp int64 `json:"timestamp"`
134 | dataInBucketsJSON string
135 | DataInBuckets []float64 `json:"dataInBuckets"`
136 | }
137 |
138 | func GetHistorical(timeframe int, by string) (hmds []types.HistoricalMempoolData, err error) {
139 | c := Pool.Get()
140 | defer c.Close()
141 |
142 | var timeSelector string
143 | switch timeframe {
144 | case 1:
145 | timeSelector = "historicalMempool1"
146 | case 2:
147 | timeSelector = "historicalMempool2"
148 | case 3:
149 | timeSelector = "historicalMempool3"
150 | case 4:
151 | timeSelector = "historicalMempool4"
152 | case 5:
153 | timeSelector = "historicalMempool5"
154 | case 6:
155 | timeSelector = "historicalMempool6"
156 | default:
157 | return hmds, errors.New("Invalid input")
158 | }
159 |
160 | var bySelector string
161 | switch by {
162 | case "byCount":
163 | bySelector = "countInBuckets"
164 | case "byFee":
165 | bySelector = "feeInBuckets"
166 | case "bySize":
167 | bySelector = "sizeInBuckets"
168 | default:
169 | return hmds, errors.New("Invalid input")
170 | }
171 |
172 | hmdsJSON, err := redis.Strings(c.Do("LRANGE", timeSelector+":"+bySelector, 0, 29))
173 | if err != nil {
174 | return
175 | }
176 |
177 | hmds = make([]types.HistoricalMempoolData, 0)
178 | for index := range hmdsJSON {
179 | hmd := types.HistoricalMempoolData{}
180 | err = json.Unmarshal([]byte(hmdsJSON[index]), &hmd)
181 | if err != nil {
182 | return
183 | }
184 | hmds = append(hmds, hmd)
185 | }
186 |
187 | return
188 | }
189 |
190 | // GetTransactionStats gets the Transaction Stats data from the database
191 | func GetTransactionStats() (tss []types.TransactionStat, err error) {
192 | c := Pool.Get()
193 | defer c.Close()
194 |
195 | tssJSON, err := redis.Strings(c.Do("LRANGE", "transactionStats", 0, 180))
196 | if err != nil {
197 | return
198 | }
199 |
200 | type transactionStat struct {
201 | SegwitCount int `json:"segwitCount"`
202 | RbfCount int `json:"rbfCount"`
203 | TxCount int `json:"txCount"`
204 | Timestamp int64 `json:"timestamp"`
205 | }
206 |
207 | tss = make([]types.TransactionStat, 0)
208 | for index := range tssJSON {
209 | ts := types.TransactionStat{}
210 | err = json.Unmarshal([]byte(tssJSON[index]), &ts)
211 | if err != nil {
212 | return
213 | }
214 | tss = append(tss, ts)
215 | }
216 |
217 | return
218 | }
219 |
220 | // GetMempoolEntries gets the last x mempool Entries from the database
221 | func GetMempoolEntries() (mes []types.MempoolEntry, err error) {
222 | c := Pool.Get()
223 | defer c.Close()
224 |
225 | // gets recent entries from 0 to 19999 (20k)
226 | mesJSON, err := redis.Strings(c.Do("ZREVRANGE", "mempoolEntries", 0, 29999))
227 | if err != nil {
228 | return
229 | }
230 |
231 | mes = make([]types.MempoolEntry, 0)
232 | for index := range mesJSON {
233 | me := types.MempoolEntry{}
234 | err = json.Unmarshal([]byte(mesJSON[index]), &me)
235 | if err != nil {
236 | return
237 | }
238 | mes = append(mes, me)
239 | }
240 |
241 | return
242 | }
243 |
244 | // SetMempoolEntriesCache SETs the response of a recent GetMempoolEntries() as a cache
245 | func SetMempoolEntriesCache(mesJSON string) (err error) {
246 | c := Pool.Get()
247 | defer c.Close()
248 |
249 | _, err = c.Do("SET", "cache:mempoolEntries", mesJSON)
250 | if err != nil {
251 | return err
252 | }
253 | return nil
254 | }
255 |
256 | // GetMempoolEntriesCache GET the cached response of a recent GetMempoolEntries() call
257 | func GetMempoolEntriesCache() (mesJSON string, err error) {
258 | c := Pool.Get()
259 | defer c.Close()
260 |
261 | mesJSON, err = redis.String(c.Do("GET", "cache:mempoolEntries"))
262 | if err != nil {
263 | return
264 | }
265 | return
266 | }
267 |
268 | // GetRecentFeerateAPIEntries returns the recent feeRate API entrys from Redis
269 | func GetRecentFeerateAPIEntries() (entries []types.FeeRateAPIEntry, err error) {
270 | c := Pool.Get()
271 | defer c.Close()
272 |
273 | entrysJSON, err := redis.Strings(c.Do("LRANGE", "feerateAPIEntries", 0, 100))
274 | if err != nil {
275 | return
276 | }
277 |
278 | entries = make([]types.FeeRateAPIEntry, 0)
279 | for index := range entrysJSON {
280 | entry := types.FeeRateAPIEntry{}
281 | err = json.Unmarshal([]byte(entrysJSON[index]), &entry)
282 | if err != nil {
283 | return
284 | }
285 | entries = append(entries, entry)
286 | }
287 |
288 | return
289 | }
290 |
--------------------------------------------------------------------------------
/api/go.mod:
--------------------------------------------------------------------------------
1 | module github.com/0xb10c/memo/api
2 |
3 | go 1.12
4 |
5 | replace github.com/ugorji/go v1.1.4 => github.com/ugorji/go/codec v0.0.0-20190204201341-e444a5086c43
6 |
7 | require (
8 | github.com/gin-contrib/cors v1.3.0
9 | github.com/gin-contrib/gzip v0.0.1
10 | github.com/gin-gonic/gin v1.4.0
11 | github.com/gomodule/redigo v2.0.0+incompatible
12 | github.com/jasonlvhit/gocron v0.0.0-20190920201010-985d45da66c5
13 | github.com/json-iterator/go v1.1.7
14 | github.com/spf13/viper v1.4.0
15 | )
16 |
--------------------------------------------------------------------------------
/api/go.sum:
--------------------------------------------------------------------------------
1 | cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
2 | github.com/0xb10c/memo/memod v0.0.0-20190711122239-5b1322de098a h1:EnDDgtJI7EuiH6yRkSDBzg/DsfQ4gFoWttlvOqmh8ZU=
3 | github.com/0xb10c/memo/memod v0.0.0-20190711122239-5b1322de098a/go.mod h1:9XotoI7MUQZqk1xvrmEewddYx4cm1M65H9kHQLNDOfU=
4 | github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
5 | github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
6 | github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII=
7 | github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
8 | github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
9 | github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
10 | github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
11 | github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
12 | github.com/btcsuite/btcd v0.0.0-20190427004231-96897255fd17/go.mod h1:3J08xEfcugPacsc34/LKRU2yO7YmuT8yt28J8k2+rrI=
13 | github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA=
14 | github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg=
15 | github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd/go.mod h1:HHNXQzUsZCxOoE+CPiyCTO6x34Zs86zZUiwtpXoGdtg=
16 | github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd/go.mod h1:F+uVaaLLH7j4eDXPRvw78tMflu7Ie2bzYOH4Y8rRKBY=
17 | github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc=
18 | github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY=
19 | github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs=
20 | github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
21 | github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
22 | github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
23 | github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
24 | github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
25 | github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
26 | github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
27 | github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
28 | github.com/davecgh/go-spew v0.0.0-20171005155431-ecdeabc65495/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
29 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
30 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
31 | github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
32 | github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
33 | github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
34 | github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
35 | github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
36 | github.com/gin-contrib/cors v1.3.0 h1:PolezCc89peu+NgkIWt9OB01Kbzt6IP0J/JvkG6xxlg=
37 | github.com/gin-contrib/cors v1.3.0/go.mod h1:artPvLlhkF7oG06nK8v3U8TNz6IeX+w1uzCSEId5/Vc=
38 | github.com/gin-contrib/gzip v0.0.1 h1:ezvKOL6jH+jlzdHNE4h9h8q8uMpDQjyl0NN0Jd7jozc=
39 | github.com/gin-contrib/gzip v0.0.1/go.mod h1:fGBJBCdt6qCZuCAOwWuFhBB4OOq9EFqlo5dEaFhhu5w=
40 | github.com/gin-contrib/sse v0.0.0-20170109093832-22d885f9ecc7/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s=
41 | github.com/gin-contrib/sse v0.0.0-20190301062529-5545eab6dad3 h1:t8FVkw33L+wilf2QiWkw0UV77qRpcH/JHPKGpKa2E8g=
42 | github.com/gin-contrib/sse v0.0.0-20190301062529-5545eab6dad3/go.mod h1:VJ0WA2NBN22VlZ2dKZQPAPnyWw5XTlK1KymzLKsr59s=
43 | github.com/gin-gonic/gin v1.3.0/go.mod h1:7cKuhb5qV2ggCFctp2fJQ+ErvciLZrIeoOSOm6mUr7Y=
44 | github.com/gin-gonic/gin v1.4.0 h1:3tMoCCfM7ppqsR0ptz/wi1impNpT7/9wQtMZ8lr1mCQ=
45 | github.com/gin-gonic/gin v1.4.0/go.mod h1:OW2EZn3DO8Ln9oIKOvM++LBO+5UPHJJDH72/q/3rZdM=
46 | github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
47 | github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
48 | github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
49 | github.com/go-sql-driver/mysql v1.4.1/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w=
50 | github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
51 | github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
52 | github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
53 | github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
54 | github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
55 | github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
56 | github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
57 | github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg=
58 | github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
59 | github.com/gomodule/redigo v2.0.0+incompatible h1:K/R+8tc58AaqLkqG2Ol3Qk+DR/TlNuhuh457pBFPtt0=
60 | github.com/gomodule/redigo v2.0.0+incompatible/go.mod h1:B4C85qUVwatsJoIUNIfCRsp7qO0iAmpGFZ4EELWSbC4=
61 | github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
62 | github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
63 | github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
64 | github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
65 | github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
66 | github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
67 | github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
68 | github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
69 | github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
70 | github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
71 | github.com/jasonlvhit/gocron v0.0.0-20190402024347-5bcdd9fcfa9b/go.mod h1:rwi/esz/h+4oWLhbWWK7f6dtmgLzxeZhnwGr7MCsTNk=
72 | github.com/jasonlvhit/gocron v0.0.0-20190807165158-02e46f9ad554 h1:Qbote/akfC8hawzAVG4Z3D5VcXvAcINmQl1jOxqnLBs=
73 | github.com/jasonlvhit/gocron v0.0.0-20190807165158-02e46f9ad554/go.mod h1:rwi/esz/h+4oWLhbWWK7f6dtmgLzxeZhnwGr7MCsTNk=
74 | github.com/jasonlvhit/gocron v0.0.0-20190920201010-985d45da66c5 h1:m1t5VsnIRS9HY+X/NReDtPvLriGcMx3Foc8a97Ogyyk=
75 | github.com/jasonlvhit/gocron v0.0.0-20190920201010-985d45da66c5/go.mod h1:rwi/esz/h+4oWLhbWWK7f6dtmgLzxeZhnwGr7MCsTNk=
76 | github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
77 | github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
78 | github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlTRt3OuAQ=
79 | github.com/json-iterator/go v1.1.5/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
80 | github.com/json-iterator/go v1.1.6 h1:MrUvLMLTMxbqFJ9kzlvat/rYZqZnW3u4wkLzWTaFwKs=
81 | github.com/json-iterator/go v1.1.6/go.mod h1:+SdeFBvtyEkXs7REEP0seUULqWtbJapLOCVDaaPEHmU=
82 | github.com/json-iterator/go v1.1.7 h1:KfgG9LzI+pYjr4xvmz/5H4FXjokeP+rlHLhv3iH62Fo=
83 | github.com/json-iterator/go v1.1.7/go.mod h1:KdQUCv79m/52Kvf8AW2vK1V8akMuk1QjK/uOdHXbAo4=
84 | github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
85 | github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
86 | github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
87 | github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4=
88 | github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
89 | github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
90 | github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
91 | github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
92 | github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
93 | github.com/magiconair/properties v1.8.0 h1:LLgXmsheXeRoUOBOjtwPQCWIYqM/LU1ayDtDePerRcY=
94 | github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
95 | github.com/mattn/go-isatty v0.0.4/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4=
96 | github.com/mattn/go-isatty v0.0.7 h1:UvyT9uN+3r7yLEYSlJsbQGdsaB/a0DlgWP3pql6iwOc=
97 | github.com/mattn/go-isatty v0.0.7/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
98 | github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
99 | github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE=
100 | github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
101 | github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
102 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
103 | github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
104 | github.com/modern-go/reflect2 v0.0.0-20180701023420-4b7aa43c6742/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
105 | github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
106 | github.com/modern-go/reflect2 v1.0.1/go.mod h1:bx2lNnkwVCuqBIxFjflWJWanXIb3RllmbCylyMrvgv0=
107 | github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
108 | github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
109 | github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
110 | github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
111 | github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
112 | github.com/pebbe/zmq4 v1.0.0/go.mod h1:7N4y5R18zBiu3l0vajMUWQgZyjv464prE8RCyBcmnZM=
113 | github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
114 | github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
115 | github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
116 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
117 | github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
118 | github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
119 | github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
120 | github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
121 | github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
122 | github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
123 | github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
124 | github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
125 | github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
126 | github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
127 | github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
128 | github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
129 | github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
130 | github.com/spf13/afero v1.1.2 h1:m8/z1t7/fwjysjQRYbP0RD+bUIF/8tJwPdEZsI83ACI=
131 | github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
132 | github.com/spf13/cast v1.3.0 h1:oget//CVOEoFewqQxwr0Ej5yjygnqGkvggSE/gB35Q8=
133 | github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
134 | github.com/spf13/jwalterweatherman v1.0.0 h1:XHEdyB+EcvlqZamSM4ZOMGlc93t6AcsBEu9Gc1vn7yk=
135 | github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
136 | github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
137 | github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
138 | github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
139 | github.com/spf13/viper v1.4.0 h1:yXHLWeravcrgGyFSyCgdYpXQ9dR9c/WED3pg1RhxqEU=
140 | github.com/spf13/viper v1.4.0/go.mod h1:PTJ7Z/lr49W6bUbkmS1V3by4uWynFiR9p7+dSq/yZzE=
141 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
142 | github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
143 | github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
144 | github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
145 | github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
146 | github.com/ugorji/go v1.1.2/go.mod h1:hnLbHMwcvSihnDhEfx2/BzKp2xb0Y+ErdfYcrs9tkJQ=
147 | github.com/ugorji/go/codec v0.0.0-20181022190402-e5e69e061d4f h1:y3Vj7GoDdcBkxFa2RUUFKM25TrBbWVDnjRDI0u975zQ=
148 | github.com/ugorji/go/codec v0.0.0-20181022190402-e5e69e061d4f/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
149 | github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8 h1:3SVOIvH7Ae1KRYyQWRjXWJEA9sS/c/pjvH++55Gr648=
150 | github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
151 | github.com/ugorji/go/codec v0.0.0-20190204201341-e444a5086c43 h1:BasDe+IErOQKrMVXab7UayvSlIpiyGwRvuX3EKYY7UA=
152 | github.com/ugorji/go/codec v0.0.0-20190204201341-e444a5086c43/go.mod h1:iT03XoTwV7xq/+UGwKO3UbC1nNNlopQiY61beSdrtOA=
153 | github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
154 | github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
155 | go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
156 | go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
157 | go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
158 | go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
159 | golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
160 | golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
161 | golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
162 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
163 | golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
164 | golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
165 | golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
166 | golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
167 | golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
168 | golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
169 | golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
170 | golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
171 | golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
172 | golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
173 | golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
174 | golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
175 | golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
176 | golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
177 | golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
178 | golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
179 | golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
180 | golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
181 | golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
182 | golang.org/x/sys v0.0.0-20181228144115-9a3f9b0469bb/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
183 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
184 | golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
185 | golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b h1:ag/x1USPSsqHud38I9BAC88qdNLDHHtQ4mlgQIZPPNA=
186 | golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
187 | golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
188 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
189 | golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
190 | golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
191 | golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
192 | golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
193 | google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
194 | google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
195 | google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
196 | google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
197 | gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
198 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
199 | gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
200 | gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
201 | gopkg.in/go-playground/assert.v1 v1.2.1/go.mod h1:9RXL0bg/zibRAgZUYszZSwO/z8Y/a8bDuhia5mkpMnE=
202 | gopkg.in/go-playground/validator.v8 v8.18.2 h1:lFB4DoMU6B626w8ny76MV7VX6W2VHct2GVOI3xgiMrQ=
203 | gopkg.in/go-playground/validator.v8 v8.18.2/go.mod h1:RX2a/7Ha8BgOhfk7j780h4/u/RRjR0eouCJSH80/M2Y=
204 | gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
205 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
206 | gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
207 | gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
208 | gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
209 | gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
210 | honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
211 |
--------------------------------------------------------------------------------
/api/memo-api.service:
--------------------------------------------------------------------------------
1 | [Unit]
2 | Description=memo-api - mempool observer API
3 | Wants=network.target
4 | After=network.target
5 |
6 | [Service]
7 | Type=simple
8 | Restart=on-failure
9 | RestartSec=5s
10 | ExecStart=/opt/memo/api/api
11 | WorkingDirectory=/opt/memo/api/
12 |
13 |
14 | [Install]
15 | WantedBy=multi-user.target
--------------------------------------------------------------------------------
/api/types/database.go:
--------------------------------------------------------------------------------
1 | package types
2 |
3 | type RecentBlock struct {
4 | Height int `json:"height"`
5 | Size int `json:"size"`
6 | Timestamp int64 `json:"timestamp"`
7 | TxCount int `json:"txCount"`
8 | Weight int `json:"weight"`
9 | }
10 |
11 | type HistoricalMempoolData struct {
12 | DataInBuckets interface{} `json:"dataInBuckets"`
13 | Timestamp int64 `json:"timestamp"`
14 | }
15 |
16 | type TransactionStat struct {
17 | SegwitCount int `json:"segwitCount"`
18 | RbfCount int `json:"rbfCount"`
19 | TxCount int `json:"txCount"`
20 | Timestamp int64 `json:"timestamp"`
21 | }
22 |
23 | type MempoolEntry struct {
24 | EntryTime int64 `json:"entryTime"`
25 | TxID string `json:"txid"`
26 | Fee int64 `json:"fee"`
27 | Size int64 `json:"size"`
28 | Version int32 `json:"version"`
29 | InputCount int `json:"inputCount"`
30 | OutputCount int `json:"outputCount"`
31 | Locktime uint32 `json:"locktime"`
32 | OutputSum int64 `json:"outputValue"`
33 | SpendsSegWit bool `json:"spendsSegWit"`
34 | SpendsMultisig bool `json:"spendsMultisig"`
35 | IsBIP69 bool `json:"isBIP69"`
36 | SignalsRBF bool `json:"signalsRBF"`
37 | OPReturnData string `json:"opreturnData"`
38 | OPReturnLength int `json:"opreturnLength"`
39 | Multisig map[string]int `json:"multisigsSpend"`
40 | Spends map[string]int `json:"spends"`
41 | PaysTo map[string]int `json:"paysTo"`
42 | }
43 |
44 | // BlockEntry holds the height, the first-seen timestamp and
45 | // shortended TXIDs. It's used in the Bitcoin Transaction Monitor
46 | // to mark transactions by block they were included in.
47 | type BlockEntry struct {
48 | Height int `json:"height"`
49 | Timestamp int64 `json:"timestamp"`
50 | ShortTXIDs []string `json:"shortTXIDs"`
51 | }
52 |
--------------------------------------------------------------------------------
/api/types/feerates.go:
--------------------------------------------------------------------------------
1 | package types
2 |
3 | // The external fee APIs return different estimate counts for which different structs are used
4 | // FeeAPIResponse1 for one estimate, FeeAPIResponse2 for two and FeeAPIResponse3 for three estimates.
5 |
6 | type FeeAPIResponse1 struct {
7 | HighFee float64 `json:"high"`
8 | }
9 |
10 | type FeeAPIResponse2 struct {
11 | HighFee float64 `json:"high"`
12 | MedFee float64 `json:"med"`
13 | }
14 |
15 | type FeeAPIResponse3 struct {
16 | HighFee float64 `json:"high"`
17 | MedFee float64 `json:"med"`
18 | LowFee float64 `json:"low"`
19 | }
20 |
21 | type FeeRateAPIEntry struct {
22 | Timestamp int64 `json:"timestamp"`
23 | BTCCom FeeAPIResponse1 `json:"btccom"`
24 | BlockchairCom FeeAPIResponse1 `json:"blockchaircom"`
25 | BlockchainInfo FeeAPIResponse2 `json:"blockchaininfo"`
26 | EarnCom FeeAPIResponse3 `json:"earncom"`
27 | BitgoCom FeeAPIResponse3 `json:"bitgocom"`
28 | BlockcypherCom FeeAPIResponse3 `json:"blockcyphercom"`
29 | BitpayCom FeeAPIResponse3 `json:"bitpaycom"`
30 | WasabiWalletIoEcon FeeAPIResponse3 `json:"wasabiwalletioEcon"`
31 | WasabiWalletIoCons FeeAPIResponse3 `json:"wasabiwalletioCons"`
32 | TrezorIo FeeAPIResponse3 `json:"trezorio"`
33 | LedgerCom FeeAPIResponse3 `json:"ledgercom"`
34 | MyceliumIo FeeAPIResponse3 `json:"myceliumio"`
35 | BitcoinerLive FeeAPIResponse3 `json:"bitcoinerlive"`
36 | BlockstreamInfo FeeAPIResponse3 `json:"blockstreaminfo"`
37 | }
38 |
--------------------------------------------------------------------------------
/api/types/mempool.go:
--------------------------------------------------------------------------------
1 | package types
2 |
3 | /*
4 | type Fees struct {
5 | Base float64 `json:"base"`
6 | Modified float64 `json:"modified"`
7 | Ancestor float64 `json:"ancestor"`
8 | Descendant float64 `json:"descendant"`
9 | }
10 |
11 | type Transaction struct {
12 | Fees Fees `json:"fees"`
13 | Size int `json:"size"`
14 | Fee float64 `json:"fee"`
15 | Modifiedfee float64 `json:"modifiedfee"`
16 | Time int `json:"time"`
17 | Height int `json:"height"`
18 | Descendantcount int `json:"descendantcount"`
19 | Descendantsize int `json:"descendantsize"`
20 | Descendantfees int `json:"descendantfees"`
21 | Ancestorcount int `json:"ancestorcount"`
22 | Ancestorsize int `json:"ancestorsize"`
23 | Ancestorfees int `json:"ancestorfees"`
24 | Wtxid string `json:"wtxid"`
25 | Depends []interface{} `json:"depends"`
26 | Spentby []interface{} `json:"spentby"`
27 | Bip125Replaceable bool `json:"bip125-replaceable"`
28 | }
29 | */
30 |
31 | // PartialTransaction is a part-struct of `Transaction` which contains
32 | // only the for-now-used values to be more memory efficient
33 | type PartialTransaction struct {
34 | Size int `json:"vsize"`
35 | Fee float64 `json:"fee"`
36 | Time int `json:"time"`
37 | Wtxid string `json:"wtxid"`
38 | Bip125Replaceable bool `json:"bip125-replaceable"`
39 | }
40 |
--------------------------------------------------------------------------------
/memod/app/app.go:
--------------------------------------------------------------------------------
1 | package app
2 |
3 | import (
4 | "os"
5 | "os/signal"
6 | "syscall"
7 |
8 | "github.com/0xb10c/memo/memod/config"
9 | "github.com/0xb10c/memo/memod/database"
10 | "github.com/0xb10c/memo/memod/fetcher"
11 | "github.com/0xb10c/memo/memod/logger"
12 | "github.com/0xb10c/memo/memod/zmq"
13 | )
14 |
15 | // Run starts the memo deamon
16 | func Run() {
17 |
18 | exitSignals := make(chan os.Signal, 1)
19 | shouldExit := make(chan bool, 1)
20 | noError := true
21 |
22 | signal.Notify(exitSignals, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
23 | go handleExitSig(exitSignals, shouldExit)
24 |
25 | err := database.SetupRedis()
26 | if err != nil {
27 | logger.Error.Printf("Failed to setup Redis database connection: %s", err.Error())
28 | shouldExit <- true
29 | noError = false
30 | }
31 |
32 | if config.GetBool("zmq.saveMempoolEntries.enable") {
33 | sqlDB, err := database.SetupSQLite()
34 | if err != nil {
35 | logger.Error.Printf("Failed to setup SQLite database connection: %s", err.Error())
36 | shouldExit <- true
37 | noError = false
38 | }
39 | defer sqlDB.Close()
40 | }
41 |
42 | if noError {
43 | startWorkers()
44 | }
45 |
46 | <-shouldExit // wait till memod should exit
47 | logger.Info.Println("Memod exiting")
48 | }
49 |
50 | // handles exit signals
51 | func handleExitSig(exitSignals chan os.Signal, shouldExit chan bool) {
52 | sig := <-exitSignals
53 | logger.Info.Println("Received signal", sig)
54 | shouldExit <- true
55 | }
56 |
57 | func startWorkers() {
58 |
59 | if config.GetBool("mempool.enable") {
60 | logger.Info.Println("Starting with mempool fetching enabled")
61 | go fetcher.SetupMempoolFetcher()
62 | }
63 |
64 | if config.GetBool("feeratefetcher.enable") {
65 | logger.Info.Println("Starting with feerate API fetching enabled")
66 | go fetcher.SetupFeerateAPIFetcher()
67 | }
68 |
69 | if config.GetBool("zmq.enable") {
70 | logger.Info.Println("Starting with ZMQ interface enabled")
71 | go zmq.SetupZMQ()
72 | }
73 |
74 | }
75 |
--------------------------------------------------------------------------------
/memod/config.example.toml:
--------------------------------------------------------------------------------
1 | # config.toml
2 | # This example config contains all avaliable configuration options.
3 | # Options are assigned their default or left blank with a FIXME: mark.
4 |
5 | [redis]
6 | host = "" # FIXME:
7 | port = "" # FIXME:
8 | user = "" # FIXME:
9 | passwd = "" # FIXME:
10 | connection = "redis"
11 |
12 | # Needed when fetching via REST interface
13 | [bitcoind.rest]
14 | protocol = "http"
15 | host = "localhost"
16 | port = "8332"
17 | responseTimeout = 30 # seconds
18 |
19 | # Needed when fetching via JSON-RPC interface
20 | [bitcoind.jsonrpc]
21 | protocol = "http"
22 | host = "localhost"
23 | port = "8332" # regtest 18443 # testnet 18332
24 | username = ""
25 | password = ""
26 | responseTimeout = 30 # seconds
27 |
28 | [mempool]
29 | enable = false # enable mempool fetching
30 | fetchInterface = "REST" # fetch mempool via REST or JSON-RPC
31 | fetchInterval = 60 # seconds
32 | callSaveMempool = true # calls the savemempool RPC on average after every fourth fetch
33 | [mempool.processing]
34 | processCurrentMempool = true
35 | processHistoricalMempool = true
36 | processTransactionStats = true
37 |
38 | [feeratefetcher]
39 | enable = false
40 | fetchInterval = 180
41 |
42 | [log]
43 | enableTrace = false
44 | colorizeOutput = true
45 |
46 | [zmq]
47 | enable = false # enable the zmq interface
48 | host = "localhost" # currently only tcp connections are supported
49 | port = "28332"
50 | [zmq.subscribeTo]
51 | rawTx = false
52 | rawBlock = true # needed to write recentBlocks to database
53 | hashTx = false
54 | hashBlock = false
55 | rawTx2 = false # this needs a custom bitcoind patch https://github.com/0xB10C/bitcoin/tree/2019-06-rawtx2-zmq-for-memod to work
56 |
57 |
--------------------------------------------------------------------------------
/memod/config/config.go:
--------------------------------------------------------------------------------
1 | package config
2 |
3 | import (
4 | "flag"
5 | "fmt"
6 |
7 | "github.com/spf13/viper"
8 | )
9 |
10 | // logger/logger.go can't be used here to log,
11 | // since logger.go itself depends on the config
12 | // to read `log.enableTrace`.
13 |
14 | func init() {
15 | // do not try to load a config file when running tests
16 | if flag.Lookup("test.v") != nil {
17 | return
18 | }
19 |
20 | setDefaults()
21 |
22 | viper.SetConfigType("toml")
23 | viper.SetConfigName("config")
24 | viper.AddConfigPath(".")
25 | err := viper.ReadInConfig()
26 | if err != nil {
27 | fmt.Println("ERROR: Error reading config file: ", err)
28 | }
29 | }
30 |
31 | func setDefaults() {
32 | viper.SetDefault("redis.host", "localhost")
33 | viper.SetDefault("redis.connection", "redis")
34 |
35 | viper.SetDefault("bitcoind.rest.protocol", "http")
36 | viper.SetDefault("bitcoind.rest.host", "localhost")
37 | viper.SetDefault("bitcoind.rest.port", "8332")
38 | viper.SetDefault("bitcoind.rest.responseTimeout", 30)
39 |
40 | viper.SetDefault("bitcoind.jsonrpc.protocol", "http")
41 | viper.SetDefault("bitcoind.jsonrpc.host", "localhost")
42 | viper.SetDefault("bitcoind.jsonrpc.port", "8332")
43 | viper.SetDefault("bitcoind.jsonrpc.responseTimeout", 30)
44 |
45 | viper.SetDefault("mempool.enable", false)
46 | viper.SetDefault("mempool.fetchInterface", "REST")
47 | viper.SetDefault("mempool.callSaveMempool", true)
48 | viper.SetDefault("mempool.processing.processCurrentMempool", false)
49 | viper.SetDefault("mempool.processing.processHistoricalMempool", false)
50 | viper.SetDefault("mempool.processing.processTransactionStats", false)
51 | viper.SetDefault("mempool.fetchInterval", 60)
52 |
53 | viper.SetDefault("feeratefetcher.enable", false)
54 | viper.SetDefault("feeratefetcher.fetchInterval", 180)
55 |
56 | viper.SetDefault("log.enableTrace", false)
57 | viper.SetDefault("log.colorizeOutput", true)
58 |
59 | viper.SetDefault("zmq.enable", false)
60 | viper.SetDefault("zmq.host", "localhost")
61 | viper.SetDefault("zmq.port", 28332)
62 | viper.SetDefault("zmq.subscribeTo.rawTx", false)
63 | viper.SetDefault("zmq.subscribeTo.rawBlock", false)
64 | viper.SetDefault("zmq.subscribeTo.hashTx", false)
65 | viper.SetDefault("zmq.subscribeTo.hashBlock", false)
66 | viper.SetDefault("zmq.saveMempoolEntries.enable", false) // saves mempool entries to a SQLite database
67 | viper.SetDefault("zmq.saveMempoolEntries.dbPath", "/tmp/mempool-entries.sqlite")
68 | }
69 |
70 | // GetInt returns a config property as int
71 | func GetInt(property string) int {
72 | result := viper.GetInt(property)
73 | if result == 0 {
74 | fmt.Println("WARN: Property " + property + " is 0. Is this not set?")
75 | }
76 | return result
77 | }
78 |
79 | // GetString returns a config property as string
80 | func GetString(property string) string {
81 | result := viper.GetString(property)
82 | if result == "" {
83 | fmt.Println("WARN: Property " + property + " is \"\". Is this not set?")
84 | }
85 | return result
86 | }
87 |
88 | // GetBool returns a config property as bool
89 | func GetBool(property string) bool {
90 | return viper.GetBool(property)
91 | }
92 |
--------------------------------------------------------------------------------
/memod/database/database.go:
--------------------------------------------------------------------------------
1 | package database
2 |
3 | import (
4 | "database/sql"
5 |
6 | "github.com/gomodule/redigo/redis"
7 | _ "github.com/mattn/go-sqlite3"
8 |
9 | "github.com/0xb10c/memo/memod/config"
10 | "github.com/0xb10c/memo/memod/logger"
11 | )
12 |
13 | var (
14 | // Pool is a pool of redis connections
15 | Pool *redis.Pool
16 | // SQLiteDB is a open SQLite database
17 | SQLiteDB *sql.DB
18 | )
19 |
20 | func newPool() *redis.Pool {
21 | dbUser := config.GetString("redis.user")
22 | dbPasswd := config.GetString("redis.passwd")
23 | dbHost := config.GetString("redis.host")
24 | dbPort := config.GetString("redis.port")
25 | dbConnection := config.GetString("redis.connection")
26 |
27 | connectionString := dbConnection + "://" + dbUser + ":" + dbPasswd + "@" + dbHost + ":" + dbPort
28 |
29 | return &redis.Pool{
30 | MaxIdle: 40,
31 | MaxActive: 1200, // max number of connections
32 | Dial: func() (redis.Conn, error) {
33 | c, err := redis.DialURL(connectionString)
34 | if err != nil {
35 | return c, err
36 | }
37 | return c, err
38 | },
39 | }
40 | }
41 |
42 | // SetupRedis sets up a new Redis Pool
43 | func SetupRedis() (err error) {
44 | Pool = newPool()
45 |
46 | c := Pool.Get() // get a new connection
47 | defer c.Close()
48 |
49 | _, err = c.Do("PING")
50 | if err != nil {
51 | return err
52 | }
53 |
54 | logger.Info.Println("Setup redis database connection pool")
55 |
56 | return nil
57 | }
58 |
59 | // SetupSQLite sets up the SQLite Database used for persitently saving mempool entries
60 | func SetupSQLite() (db *sql.DB, err error) {
61 | filePath := config.GetString("zmq.saveMempoolEntries.dbPath")
62 | db, err = sql.Open("sqlite3", filePath)
63 | if err != nil {
64 | return nil, err
65 | }
66 |
67 | err = db.Ping()
68 | if err != nil {
69 | return nil, err
70 | }
71 |
72 | SQLiteDB = db
73 | createMempoolEntryTableIfNotExists()
74 | logger.Info.Println("Setup SQLite database in " + filePath)
75 |
76 | return db, nil
77 | }
78 |
79 | func createMempoolEntryTableIfNotExists() {
80 | statement := `CREATE TABLE IF NOT EXISTS mempoolEntries (
81 | entryTime INT NOT NULL,
82 | txid TEXT NOT NULL,
83 | fee INT NOT NULL,
84 | size INT NOT NULL,
85 | version INT NOT NULL,
86 | inputs INT NOT NULL,
87 | outputs INT NOT NULL,
88 | locktime INT NOT NULL,
89 | outSum INT NOT NULL,
90 | spendsSegWit BOOLEAN NOT NULL,
91 | spendsMultisig BOOLEAN NOT NULL,
92 | bip69compliant BOOLEAN NOT NULL,
93 | signalsRBF BOOLEAN NOT NULL,
94 | spends TEXT NOT NULL,
95 | paysto TEXT NOT NULL,
96 | multisigs TEXT,
97 | opreturndata TEXT,
98 | PRIMARY KEY (entryTime, txid)
99 | )`
100 |
101 | _, err := SQLiteDB.Exec(statement)
102 | if err != nil {
103 | logger.Error.Printf("Cound not create table mempoolEntries: %v.\n", err)
104 | return
105 | }
106 | }
107 |
--------------------------------------------------------------------------------
/memod/database/reader.go:
--------------------------------------------------------------------------------
1 | package database
2 |
3 | import (
4 | "fmt"
5 | "strconv"
6 | "time"
7 |
8 | "github.com/0xb10c/memo/memod/logger"
9 | "github.com/gomodule/redigo/redis"
10 | )
11 |
12 | type NeedsUpdate struct {
13 | Update2h bool
14 | Update12h bool
15 | Update48h bool
16 | Update7d bool
17 | Update30d bool
18 | Update180d bool
19 | }
20 |
21 | func ReadHistoricalMempoolNeedUpdate() (nu NeedsUpdate, err error) {
22 | defer logger.TrackTime(time.Now(), "ReadHistroricalMempoolNeedUpdate()")
23 | c := Pool.Get()
24 | defer c.Close()
25 |
26 | lastUpdatedTimesStrings, err := redis.Strings(c.Do("MGET", "historicalMempool1:lastUpdated", "historicalMempool2:lastUpdated", "historicalMempool3:lastUpdated", "historicalMempool4:lastUpdated", "historicalMempool5:lastUpdated", "historicalMempool6:lastUpdated"))
27 | if err != nil {
28 | return
29 | }
30 |
31 | // convert responses from string to int64 unix timestamp and
32 | // calculate the time difference between now and the last updated time
33 | lastUpdatedTimeDiffs := make([]time.Duration, 0)
34 | for _, lastUpdatedString := range lastUpdatedTimesStrings {
35 | if n, err := strconv.Atoi(lastUpdatedString); err == nil {
36 | lastUpdatedTime := time.Unix(int64(n), 0)
37 | timeDiff := time.Duration(time.Now().Unix()-lastUpdatedTime.Unix()) * time.Second
38 | lastUpdatedTimeDiffs = append(lastUpdatedTimeDiffs, timeDiff)
39 | } else {
40 | fmt.Println(lastUpdatedString, "is not an integer.")
41 | lastUpdatedTimeDiffs = append(lastUpdatedTimeDiffs, time.Duration(1000)*time.Hour)
42 | }
43 | }
44 |
45 | // Update 2h data every 4 minutes
46 | if lastUpdatedTimeDiffs[0] >= 4*time.Minute {
47 | nu.Update2h = true
48 | logger.Trace.Println("2h Historical Mempool data needs to be updated")
49 | }
50 |
51 | // Update 12h data every 24 minutes
52 | if lastUpdatedTimeDiffs[1] >= 24*time.Minute {
53 | nu.Update12h = true
54 | logger.Trace.Println("12h Historical Mempool data needs to be updated")
55 | }
56 |
57 | // Update 48h data every 96 minutes
58 | if lastUpdatedTimeDiffs[2] >= 96*time.Minute {
59 | nu.Update48h = true
60 | logger.Trace.Println("48h Historical Mempool data needs to be updated")
61 | }
62 |
63 | // Update 7d data every 336 minutes
64 | if lastUpdatedTimeDiffs[3] >= 336*time.Minute {
65 | nu.Update7d = true
66 | logger.Trace.Println("7d Historical Mempool data needs to be updated")
67 | }
68 |
69 | // Update 30d data every 1440 minutes
70 | if lastUpdatedTimeDiffs[4] >= 1440*time.Minute {
71 | nu.Update30d = true
72 | logger.Trace.Println("30d Historical Mempool data needs to be updated")
73 | }
74 |
75 | // Update 180d data every 8640 minutes
76 | if lastUpdatedTimeDiffs[5] >= 8640*time.Minute {
77 | nu.Update180d = true
78 | logger.Trace.Println("180d Historical Mempool data needs to be updated")
79 | }
80 |
81 | return
82 | }
83 |
--------------------------------------------------------------------------------
/memod/database/writer.go:
--------------------------------------------------------------------------------
1 | package database
2 |
3 | import (
4 | "encoding/json"
5 | "fmt"
6 | "math/rand"
7 | "strconv"
8 | "time"
9 |
10 | "github.com/0xb10c/memo/memod/config"
11 |
12 | "github.com/gomodule/redigo/redis"
13 |
14 | "github.com/0xb10c/memo/memod/logger"
15 | "github.com/0xb10c/memo/memod/types"
16 | )
17 |
18 | // WriteCurrentMempoolData writes the current mempool data into the database
19 | func WriteCurrentMempoolData(feerateMap map[int]int, mempoolSizeInByte int, megabyteMarkers []int) error {
20 | defer logger.TrackTime(time.Now(), "writeCurrentMempoolData()")
21 | c := Pool.Get()
22 | defer c.Close()
23 |
24 | feerateMapJSON, err := json.Marshal(feerateMap)
25 | if err != nil {
26 | return err
27 | }
28 |
29 | megabyteMarkersJSON, err := json.Marshal(megabyteMarkers)
30 | if err != nil {
31 | return err
32 | }
33 |
34 | prefix := "currentMempool"
35 |
36 | c.Send("MULTI")
37 | c.Send("SET", prefix+":feerateMap", feerateMapJSON)
38 | c.Send("SET", prefix+":mempoolSizeInByte", mempoolSizeInByte)
39 | c.Send("SET", prefix+":megabyteMarkers", megabyteMarkersJSON)
40 | c.Send("SET", prefix+":utcTimestamp", time.Now().Unix())
41 | _, err = c.Do("EXEC")
42 | if err != nil {
43 | return err
44 | }
45 |
46 | return nil
47 | }
48 |
49 | // WriteNewBlockData writes data for a new block into the database
50 | func WriteNewBlockData(height int, numTx int, sizeWithWitness int, weight int) error {
51 | defer logger.TrackTime(time.Now(), "writeNewBlockData()")
52 | c := Pool.Get()
53 | defer c.Close()
54 | listName := "recentBlocks"
55 |
56 | rb := types.RecentBlock{Height: height, Size: sizeWithWitness, Timestamp: time.Now().Unix(), TxCount: numTx, Weight: weight}
57 |
58 | rbJSON, err := json.Marshal(rb)
59 | if err != nil {
60 | return err
61 | }
62 |
63 | _, err = c.Do("LPUSH", listName, rbJSON)
64 | if err != nil {
65 | return err
66 | }
67 |
68 | return nil
69 | }
70 |
71 | // WriteNewBlockEntry writes a BlockEntry into the Redis database.
72 | func WriteNewBlockEntry(height int, shortTXIDs []string) error {
73 | defer logger.TrackTime(time.Now(), "WriteNewBlockEntry()")
74 | c := Pool.Get()
75 | defer c.Close()
76 | listName := "blockEntries"
77 |
78 | be := types.BlockEntry{Height: height, Timestamp: time.Now().Unix(), ShortTXIDs: shortTXIDs}
79 |
80 | beJSON, err := json.Marshal(be)
81 | if err != nil {
82 | return err
83 | }
84 |
85 | _, err = c.Do("LPUSH", listName, beJSON)
86 | if err != nil {
87 | return err
88 | }
89 |
90 | // only keep the last ~200 blocks
91 | // check 10% of all insertions
92 | if rand.Intn(10) == 4 {
93 | _, err := c.Do("LTRIM", listName, "0", 200)
94 | if err != nil {
95 | return fmt.Errorf("could not do LTRIM on %s: %s", listName, err.Error())
96 | }
97 | }
98 |
99 | return nil
100 | }
101 |
102 | // WriteHistoricalMempoolData writes the histoical mempool data into the database
103 | func WriteHistoricalMempoolData(countInBuckets []int, feeInBuckets []float64, sizeInBuckets []int, timeframe int) error {
104 | defer logger.TrackTime(time.Now(), "WriteHistoricalMempoolData()")
105 | c := Pool.Get()
106 | defer c.Close()
107 |
108 | countInBucketsJSON, err := json.Marshal(types.HistoricalMempoolData{DataInBuckets: countInBuckets, Timestamp: time.Now().Unix()})
109 | if err != nil {
110 | return err
111 | }
112 |
113 | feeInBucketsJSON, err := json.Marshal(types.HistoricalMempoolData{DataInBuckets: feeInBuckets, Timestamp: time.Now().Unix()})
114 | if err != nil {
115 | return err
116 | }
117 |
118 | sizeInBucketsJSON, err := json.Marshal(types.HistoricalMempoolData{DataInBuckets: sizeInBuckets, Timestamp: time.Now().Unix()})
119 | if err != nil {
120 | return err
121 | }
122 |
123 | listName := "historicalMempool" + strconv.Itoa(timeframe)
124 |
125 | _, err = c.Do("LPUSH", listName+":countInBuckets", countInBucketsJSON)
126 | if err != nil {
127 | return err
128 | }
129 |
130 | _, err = c.Do("LPUSH", listName+":feeInBuckets", feeInBucketsJSON)
131 | if err != nil {
132 | return err
133 | }
134 |
135 | _, err = c.Do("LPUSH", listName+":sizeInBuckets", sizeInBucketsJSON)
136 | if err != nil {
137 | return err
138 | }
139 |
140 | _, err = c.Do("SET", listName+":lastUpdated", time.Now().Unix())
141 |
142 | return nil
143 | }
144 |
145 | // WriteCurrentTransactionStats writes the current transaction stats into the database
146 | func WriteCurrentTransactionStats(segwitCount int, rbfCount int, txCount int) error {
147 | defer logger.TrackTime(time.Now(), "WriteCurrentTransactionStats()")
148 | c := Pool.Get()
149 | defer c.Close()
150 |
151 | ts := types.TransactionStat{SegwitCount: segwitCount, RbfCount: rbfCount, TxCount: txCount, Timestamp: time.Now().Unix()}
152 | tsJSON, err := json.Marshal(ts)
153 | if err != nil {
154 | return err
155 | }
156 |
157 | listName := "transactionStats"
158 |
159 | _, err = c.Do("LPUSH", listName, tsJSON)
160 | if err != nil {
161 | return err
162 | }
163 |
164 | return nil
165 | }
166 |
167 | // WriteMempoolEntries writes a txid and it's feerate to the database
168 | func WriteMempoolEntries(me types.MempoolEntry) error {
169 | //defer logger.TrackTime(time.Now(), "WriteMempoolEntries()")
170 | c := Pool.Get()
171 | defer c.Close()
172 |
173 | meJSON, err := json.Marshal(me)
174 | if err != nil {
175 | return fmt.Errorf("could not marshal the mempoolEntry to JSON: %s", err.Error())
176 | }
177 |
178 | listName := "mempoolEntries"
179 |
180 | // insert the mempool entry into a redis sorted list
181 | // the list is sorted by timestamps in ascending order.
182 | _, err = c.Do("ZADD", listName, me.EntryTime, meJSON)
183 | if err != nil {
184 | return fmt.Errorf("could not ZADD the mempoolEntry JSON to %s: %s", listName, err.Error())
185 | }
186 |
187 | if config.GetBool("zmq.saveMempoolEntries.enable") {
188 | err = writeMempoolEntriesSQLite(me)
189 | if err != nil {
190 | return fmt.Errorf("could not write the mempoolEntry to SQLite %s", err.Error())
191 | }
192 | }
193 |
194 | // only keep the last ~500k transactions
195 | // check every 0.1% of all insertions
196 | if rand.Intn(1000) == 42 {
197 | count, err := redis.Int64(c.Do("ZCOUNT", listName, "-inf", "+inf"))
198 | if err != nil {
199 | return fmt.Errorf("could not do ZCOUNT on %s: %s", listName, err.Error())
200 | }
201 | if count > 500000 {
202 | removeIndex := count - 500000
203 | _, err := c.Do("ZREMRANGEBYRANK", listName, "0", removeIndex)
204 | if err != nil {
205 | return fmt.Errorf("could not do ZREMRANGEBYRANK on %s: %s", listName, err.Error())
206 | }
207 | }
208 | }
209 |
210 | return nil
211 | }
212 |
213 | func writeMempoolEntriesSQLite(me types.MempoolEntry) error {
214 | spendsJSON, err := json.Marshal(me.Spends)
215 | if err != nil {
216 | return fmt.Errorf("could not marshal mempoolEntry.Spends to JSON: %s", err.Error())
217 | }
218 |
219 | paystoJSON, err := json.Marshal(me.PaysTo)
220 | if err != nil {
221 | return fmt.Errorf("could not marshal mempoolEntry.PaysTo to JSON: %s", err.Error())
222 | }
223 |
224 | multisigJSON, err := json.Marshal(me.Multisig)
225 | if err != nil {
226 | return fmt.Errorf("could not marshal mempoolEntry.Multisig to JSON: %s", err.Error())
227 | }
228 |
229 | _, err = SQLiteDB.Exec("INSERT INTO mempoolEntries(entryTime, txid, fee, size, version, inputs, outputs, locktime, outSum, spendsSegWit, spendsMultisig, bip69compliant, signalsRBF, spends, paysto, multisigs, opreturndata) VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
230 | me.EntryTime, me.TxID, me.Fee, me.Size, me.Version, me.InputCount, me.OutputCount, me.Locktime, me.OutputSum, me.SpendsSegWit, me.SpendsMultisig, me.IsBIP69, me.SignalsRBF, spendsJSON, paystoJSON, multisigJSON, me.OPReturnData)
231 | if err != nil {
232 | return fmt.Errorf("error while writing to SQLite: %s", err.Error())
233 | }
234 | return nil
235 | }
236 |
237 | // WriteFeerateAPIEntry writes a new feerate API entry into the database
238 | func WriteFeerateAPIEntry(entry types.FeeRateAPIEntry) error {
239 | defer logger.TrackTime(time.Now(), "WriteFeerateAPIEntry()")
240 | c := Pool.Get()
241 | defer c.Close()
242 | listName := "feerateAPIEntries"
243 |
244 | entryJSON, err := json.Marshal(entry)
245 | if err != nil {
246 | return err
247 | }
248 |
249 | _, err = c.Do("LPUSH", listName, entryJSON)
250 | if err != nil {
251 | return err
252 | }
253 |
254 | return nil
255 | }
256 |
--------------------------------------------------------------------------------
/memod/encoder/encoder.go:
--------------------------------------------------------------------------------
1 | package encoder
2 |
3 | import (
4 | "encoding/json"
5 | "time"
6 |
7 | "github.com/0xb10c/memo/memod/logger"
8 | "github.com/0xb10c/memo/memod/types"
9 | )
10 |
11 | /* decodes and encodes */
12 |
13 | // DecodeFetchedMempoolBody decode the Body of the JSON response as a map of PartialTransactions
14 | func DecodeFetchedMempoolBody(body []byte) (map[string]types.PartialTransaction, error) {
15 | defer logger.TrackTime(time.Now(), "decodeFetchedMempoolBody()")
16 |
17 | var mempool map[string]types.PartialTransaction
18 | err := json.Unmarshal(body, &mempool)
19 | if err != nil {
20 | return nil, err
21 | }
22 |
23 | return mempool, nil
24 | }
25 |
--------------------------------------------------------------------------------
/memod/fetcher/feerate.go:
--------------------------------------------------------------------------------
1 | package fetcher
2 |
3 | import (
4 | "crypto/tls"
5 | "crypto/x509"
6 | "encoding/base64"
7 | "net/http"
8 | "time"
9 |
10 | "github.com/0xb10c/memo/memod/config"
11 | "github.com/0xb10c/memo/memod/database"
12 | "github.com/0xb10c/memo/memod/logger"
13 | "github.com/0xb10c/memo/memod/types"
14 | "github.com/jasonlvhit/gocron"
15 | "github.com/tidwall/gjson"
16 | )
17 |
18 | // SetupFeerateAPIFetcher sets up a periodic feerate API fetch job
19 | func SetupFeerateAPIFetcher() {
20 | feerateFetchInterval := config.GetInt("feeratefetcher.fetchInterval")
21 | s := gocron.NewScheduler()
22 | s.Every(uint64(feerateFetchInterval)).Seconds().Do(getFeerates)
23 | logger.Info.Println("Setup feerate API fetcher. First fetch in", feerateFetchInterval, "seconds")
24 | <-s.Start()
25 | defer s.Clear()
26 | }
27 |
28 | func getFeerates() {
29 | cBTCCom := make(chan types.FeeAPIResponse1)
30 | cBlockchairCom := make(chan types.FeeAPIResponse1)
31 |
32 | cBlockchainInfo := make(chan types.FeeAPIResponse2)
33 |
34 | cEarnCom := make(chan types.FeeAPIResponse3)
35 | cBitgoCom := make(chan types.FeeAPIResponse3)
36 | cBlockcypherCom := make(chan types.FeeAPIResponse3)
37 | cBitpayCom := make(chan types.FeeAPIResponse3)
38 | cWasabiWalletIoEcon := make(chan types.FeeAPIResponse3)
39 | cWasabiWalletIoCons := make(chan types.FeeAPIResponse3)
40 | cTrezorIo := make(chan types.FeeAPIResponse3)
41 | cLedgerCom := make(chan types.FeeAPIResponse3)
42 | cMyceliumIo := make(chan types.FeeAPIResponse3)
43 | cBitcoinerLive := make(chan types.FeeAPIResponse3)
44 | cBlockstreamInfo := make(chan types.FeeAPIResponse3)
45 |
46 | go getBTCCom(cBTCCom)
47 | go getBlockchairCom(cBlockchairCom)
48 | go getBlockchainInfo(cBlockchainInfo)
49 | go getEarnCom(cEarnCom)
50 | go getBitgoCom(cBitgoCom)
51 | go getBlockcypherCom(cBlockcypherCom)
52 | go getBitpayCom(cBitpayCom)
53 | go getWasabiWalletIo(cWasabiWalletIoEcon, cWasabiWalletIoCons)
54 | go getTrezorIo(cTrezorIo)
55 | go getLedgerCom(cLedgerCom)
56 | go getMyceliumIo(cMyceliumIo)
57 | go getBitcoinerLive(cBitcoinerLive)
58 | go getBlockstreamInfo(cBlockstreamInfo)
59 |
60 | respBTCCom := <-cBTCCom
61 | respBlockchairCom := <-cBlockchairCom
62 | respBlockchainInfo := <-cBlockchainInfo
63 | respEarnCom := <-cEarnCom
64 | respBitgoCom := <-cBitgoCom
65 | respBlockcypherCom := <-cBlockcypherCom
66 | respBitpayCom := <-cBitpayCom
67 | respWasabiWalletIoEcon := <-cWasabiWalletIoEcon
68 | respWasabiWalletIoCons := <-cWasabiWalletIoCons
69 | respTrezorIo := <-cTrezorIo
70 | respLedgerCom := <-cLedgerCom
71 | respMyceliumIo := <-cMyceliumIo
72 | respBitcoinerLive := <-cBitcoinerLive
73 | respBlockstreamInfo := <-cBlockstreamInfo
74 |
75 | entry := types.FeeRateAPIEntry{
76 | Timestamp: time.Now().Unix(),
77 | BTCCom: respBTCCom,
78 | BlockchairCom: respBlockchairCom,
79 | BlockchainInfo: respBlockchainInfo,
80 | EarnCom: respEarnCom,
81 | BitgoCom: respBitgoCom,
82 | BlockcypherCom: respBlockcypherCom,
83 | BitpayCom: respBitpayCom,
84 | WasabiWalletIoEcon: respWasabiWalletIoEcon,
85 | WasabiWalletIoCons: respWasabiWalletIoCons,
86 | TrezorIo: respTrezorIo,
87 | LedgerCom: respLedgerCom,
88 | MyceliumIo: respMyceliumIo,
89 | BitcoinerLive: respBitcoinerLive,
90 | BlockstreamInfo: respBlockstreamInfo,
91 | }
92 |
93 | database.WriteFeerateAPIEntry(entry)
94 | }
95 |
96 | func getEarnCom(cEarnCom chan types.FeeAPIResponse3) {
97 | url := "https://bitcoinfees.earn.com/api/v1/fees/recommended" // response: {"fastestFee":50,"halfHourFee":50,"hourFee":42}
98 | body, err := makeHTTPGETReq(url, 5)
99 | if err != nil {
100 | logger.Error.Printf("Could not fetch bitcoinfees.earn.com: %v", err.Error())
101 | }
102 | result := gjson.GetMany(string(body), "fastestFee", "halfHourFee", "hourFee")
103 | highFee, medFee, lowFee := result[0].Float(), result[1].Float(), result[2].Float()
104 |
105 | cEarnCom <- types.FeeAPIResponse3{highFee, medFee, lowFee}
106 | }
107 |
108 | func getBitgoCom(cBitgoCom chan types.FeeAPIResponse3) {
109 | url := "https://www.bitgo.com/api/v2/btc/tx/fee"
110 | body, err := makeHTTPGETReq(url, 5)
111 | if err != nil {
112 | logger.Error.Printf("Could not fetch bitgo.com: %v", err.Error())
113 | cBitgoCom <- types.FeeAPIResponse3{0, 0, 0}
114 | return
115 | }
116 | result := gjson.GetMany(string(body), "feeByBlockTarget.2", "feeByBlockTarget.4", "feeByBlockTarget.6")
117 | highFee, medFee, lowFee := result[0].Float()/1000, result[1].Float()/1000, result[2].Float()/1000
118 |
119 | cBitgoCom <- types.FeeAPIResponse3{highFee, medFee, lowFee}
120 | }
121 |
122 | func getBTCCom(cBTCCom chan types.FeeAPIResponse1) {
123 | url := "https://btc.com/service/fees/distribution"
124 | body, err := makeHTTPGETReq(url, 5)
125 | if err != nil {
126 | logger.Error.Printf("Could not fetch btc.com: %v", err.Error())
127 | cBTCCom <- types.FeeAPIResponse1{0}
128 | return
129 | }
130 | result := gjson.Get(string(body), "fees_recommended.one_block_fee")
131 | highFee := result.Float()
132 |
133 | cBTCCom <- types.FeeAPIResponse1{highFee}
134 | }
135 |
136 | func getBlockcypherCom(cBlockcypherCom chan types.FeeAPIResponse3) {
137 | url := "https://api.blockcypher.com/v1/btc/main"
138 | body, err := makeHTTPGETReq(url, 5)
139 | if err != nil {
140 | logger.Error.Printf("Could not fetch api.blockcypher.com: %v", err.Error())
141 | cBlockcypherCom <- types.FeeAPIResponse3{0, 0, 0}
142 | return
143 | }
144 | result := gjson.GetMany(string(body), "high_fee_per_kb", "medium_fee_per_kb", "low_fee_per_kb")
145 | highFee, medFee, lowFee := result[0].Float()/1000, result[1].Float()/1000, result[2].Float()/1000
146 |
147 | cBlockcypherCom <- types.FeeAPIResponse3{highFee, medFee, lowFee}
148 | }
149 |
150 | func getBlockchainInfo(cBlockchainInfo chan types.FeeAPIResponse2) {
151 | url := "https://api.blockchain.info/mempool/fees"
152 | body, err := makeHTTPGETReq(url, 5)
153 | if err != nil {
154 | logger.Error.Printf("Could not fetch api.blockchain.info: %v", err.Error())
155 | cBlockchainInfo <- types.FeeAPIResponse2{0, 0}
156 | return
157 | }
158 | result := gjson.GetMany(string(body), "priority", "regular")
159 | highFee, medFee := result[0].Float(), result[1].Float()
160 |
161 | cBlockchainInfo <- types.FeeAPIResponse2{highFee, medFee}
162 | }
163 |
164 | func getBlockchairCom(cBlockchairCom chan types.FeeAPIResponse1) {
165 | url := "https://api.blockchair.com/bitcoin/stats"
166 | body, err := makeHTTPGETReq(url, 5)
167 | if err != nil {
168 | logger.Error.Printf("Could not fetch api.blockchair.com: %v", err.Error())
169 | cBlockchairCom <- types.FeeAPIResponse1{0}
170 | }
171 | result := gjson.Get(string(body), "data.suggested_transaction_fee_per_byte_sat")
172 | highFee := result.Float()
173 |
174 | cBlockchairCom <- types.FeeAPIResponse1{highFee}
175 | }
176 |
177 | func getBitpayCom(cBitpayCom chan types.FeeAPIResponse3) {
178 | url := "https://insight.bitpay.com/api/utils/estimatefee?nbBlocks=2,3,6"
179 | body, err := makeHTTPGETReq(url, 5)
180 | if err != nil {
181 | logger.Error.Printf("Could not fetch insight.bitpay.com: %v", err.Error())
182 | cBitpayCom <- types.FeeAPIResponse3{0, 0, 0}
183 | return
184 | }
185 | result := gjson.GetMany(string(body), "2", "3", "6")
186 | highFee, medFee, lowFee := result[0].Float()*100000, result[1].Float()*100000, result[2].Float()*100000
187 | cBitpayCom <- types.FeeAPIResponse3{highFee, medFee, lowFee}
188 | }
189 |
190 | func getWasabiWalletIo(cWasabiWalletIoEcon chan types.FeeAPIResponse3, cWasabiWalletIoCons chan types.FeeAPIResponse3) {
191 | url := "https://wasabiwallet.io/api/v3/btc/Blockchain/fees/2,4,6"
192 | body, err := makeHTTPGETReq(url, 5)
193 | if err != nil {
194 | logger.Error.Printf("Could not fetch wasabiwallet.io: %v", err.Error())
195 | cWasabiWalletIoEcon <- types.FeeAPIResponse3{0, 0, 0}
196 | cWasabiWalletIoCons <- types.FeeAPIResponse3{0, 0, 0}
197 | return
198 | }
199 | resultEconomical := gjson.GetMany(string(body), "2.economical", "4.economical", "6.economical")
200 | resultConservative := gjson.GetMany(string(body), "2.conservative", "4.conservative", "6.conservative")
201 | highFeeEconomical, medFeeEconomical, lowFeeEconomical := resultEconomical[0].Float(), resultEconomical[1].Float(), resultEconomical[2].Float()
202 | highFeeConservative, medFeeConservative, lowFeeConservative := resultConservative[0].Float(), resultConservative[1].Float(), resultConservative[2].Float()
203 |
204 | cWasabiWalletIoEcon <- types.FeeAPIResponse3{highFeeEconomical, medFeeEconomical, lowFeeEconomical}
205 | cWasabiWalletIoCons <- types.FeeAPIResponse3{highFeeConservative, medFeeConservative, lowFeeConservative}
206 | }
207 |
208 | func getTrezorIo(cTrezorIo chan types.FeeAPIResponse3) {
209 |
210 | // Since the Trezor API is publicly accessible, but not publicly advertised I don't
211 | // want to have the plain text url crawable on GitHub. It's encoded as base64.
212 | urlPrefixBytes, _ := base64.StdEncoding.DecodeString("aHR0cHM6Ly9idGMxLnRyZXpvci5pby9hcGkvdjEvZXN0aW1hdGVmZWUv")
213 | urlPrefix := string(urlPrefixBytes)
214 |
215 | bodyBlocks2, err := makeHTTPGETReq(urlPrefix+"2", 5)
216 | if err != nil {
217 | logger.Error.Printf("Could not fetch trezor.io: %v", err.Error())
218 | cTrezorIo <- types.FeeAPIResponse3{0, 0, 0}
219 | return
220 | }
221 |
222 | bodyBlocks4, err := makeHTTPGETReq(urlPrefix+"4", 5)
223 | if err != nil {
224 | logger.Error.Printf("Could not fetch trezor.io: %v", err.Error())
225 | cTrezorIo <- types.FeeAPIResponse3{0, 0, 0}
226 | return
227 | }
228 |
229 | bodyBlocks6, err := makeHTTPGETReq(urlPrefix+"6", 5)
230 | if err != nil {
231 | logger.Error.Printf("Could not fetch trezor.io: %v", err.Error())
232 | cTrezorIo <- types.FeeAPIResponse3{0, 0, 0}
233 | return
234 | }
235 |
236 | resultBlocks2 := gjson.Get(string(bodyBlocks2), "result")
237 | resultBlocks4 := gjson.Get(string(bodyBlocks4), "result")
238 | resultBlocks6 := gjson.Get(string(bodyBlocks6), "result")
239 |
240 | highFee := resultBlocks2.Float() * 100000
241 | medFee := resultBlocks4.Float() * 100000
242 | lowFee := resultBlocks6.Float() * 100000
243 |
244 | cTrezorIo <- types.FeeAPIResponse3{highFee, medFee, lowFee}
245 | }
246 |
247 | func getMyceliumIo(cMyceliumIo chan types.FeeAPIResponse3) {
248 |
249 | // Since the Mycelium API is publicly accessible, but not publicly advertised I don't
250 | // want to have the plain text url crawable on GitHub. It's encoded as base64.
251 | urlBytes, _ := base64.StdEncoding.DecodeString("aHR0cHM6Ly9td3MyMC5teWNlbGl1bS5jb20vd2FwaS93YXBpL2dldE1pbmVyRmVlRXN0aW1hdGlvbnM=")
252 | url := string(urlBytes)
253 |
254 | // Mycelium uses a self signed cert. It's appended it to a copy of the cert store and trust it to query their API.
255 | rootCAs, _ := x509.SystemCertPool()
256 | if rootCAs == nil {
257 | rootCAs = x509.NewCertPool()
258 | }
259 |
260 | // Append our cert to a copy of our system cert pool / trust store
261 | if ok := rootCAs.AppendCertsFromPEM(getMyceliumFeeAPICert()); !ok {
262 | logger.Error.Printf("Could not append Mycelium self-singed cert to trust store.")
263 | cMyceliumIo <- types.FeeAPIResponse3{0, 0, 0}
264 | return
265 | }
266 |
267 | // Trust the augmented cert pool in our client
268 | config := &tls.Config{
269 | RootCAs: rootCAs,
270 | InsecureSkipVerify: true, // The https certificate mycelium uses expired on 13.08.2019.
271 | }
272 | tr := &http.Transport{TLSClientConfig: config}
273 | client := &http.Client{Transport: tr}
274 |
275 | resp, err := client.Post(url, "application/json", nil)
276 | if err != nil {
277 | logger.Error.Printf("Could not POST to mycelium.com: %v", err.Error())
278 | cMyceliumIo <- types.FeeAPIResponse3{0, 0, 0}
279 | return
280 | }
281 | defer resp.Body.Close()
282 |
283 | body, err := readResponseBody(resp)
284 | if err != nil {
285 | logger.Error.Printf("Could not read response body from mycelium.com: %v", err.Error())
286 | cMyceliumIo <- types.FeeAPIResponse3{0, 0, 0}
287 | return
288 | }
289 |
290 | result := gjson.GetMany(string(body), "r.feeEstimation.feeForNBlocks.2", "r.feeEstimation.feeForNBlocks.4", "r.feeEstimation.feeForNBlocks.10")
291 | highFee, medFee, lowFee := result[0].Float()/1000, result[1].Float()/1000, result[2].Float()/1000
292 |
293 | cMyceliumIo <- types.FeeAPIResponse3{highFee, medFee, lowFee}
294 | }
295 |
296 | func getBitcoinerLive(cBitcoinerLive chan types.FeeAPIResponse3) {
297 | url := "https://bitcoiner.live/api/fees/estimates/latest"
298 | body, err := makeHTTPGETReq(url, 5)
299 | if err != nil {
300 | logger.Error.Printf("Could not fetch bitcoiner.live: %v", err.Error())
301 | cBitcoinerLive <- types.FeeAPIResponse3{0, 0, 0}
302 | return
303 | }
304 | result := gjson.GetMany(string(body), "estimates.30.sat_per_vbyte", "estimates.60.sat_per_vbyte", "estimates.120.sat_per_vbyte")
305 | highFee, medFee, lowFee := result[0].Float(), result[1].Float(), result[2].Float()
306 | cBitcoinerLive <- types.FeeAPIResponse3{highFee, medFee, lowFee}
307 | }
308 |
309 | func getBlockstreamInfo(cBlockstreamInfo chan types.FeeAPIResponse3) {
310 | url := "https://blockstream.info/api/fee-estimates"
311 | body, err := makeHTTPGETReq(url, 5)
312 | if err != nil {
313 | logger.Error.Printf("Could not fetch blockstream.info: %v", err.Error())
314 | cBlockstreamInfo <- types.FeeAPIResponse3{0, 0, 0}
315 | return
316 | }
317 | result := gjson.GetMany(string(body), "2", "3", "6")
318 | highFee, medFee, lowFee := result[0].Float(), result[1].Float(), result[2].Float()
319 | cBlockstreamInfo <- types.FeeAPIResponse3{highFee, medFee, lowFee}
320 | }
321 |
322 | func getLedgerCom(cLedgerCom chan types.FeeAPIResponse3) {
323 | // Since the Ledger Live API is publicly accessible, but not publicly advertised I don't
324 | // want to have the plain text url crawable on GitHub. It's encoded as base64.
325 | urlBytes, _ := base64.StdEncoding.DecodeString("aHR0cHM6Ly9leHBsb3JlcnMuYXBpLmxpdmUubGVkZ2VyLmNvbS9ibG9ja2NoYWluL3YyL2J0Yy9mZWVz")
326 | url := string(urlBytes)
327 | body, err := makeHTTPGETReq(url, 5)
328 | if err != nil {
329 | logger.Error.Printf("Could not fetch ledger.com: %v", err.Error())
330 | cLedgerCom <- types.FeeAPIResponse3{0, 0, 0}
331 | return
332 | }
333 | result := gjson.GetMany(string(body), "1", "3", "6")
334 | highFee, medFee, lowFee := result[0].Float()/1000, result[1].Float()/1000, result[2].Float()/1000
335 | cLedgerCom <- types.FeeAPIResponse3{highFee, medFee, lowFee}
336 | }
337 |
--------------------------------------------------------------------------------
/memod/fetcher/mempool.go:
--------------------------------------------------------------------------------
1 | package fetcher
2 |
3 | /* fetches */
4 |
5 | import (
6 | "errors"
7 | "fmt"
8 | "math/rand"
9 | "net/http"
10 | "strings"
11 | "time"
12 |
13 | "github.com/0xb10c/memo/memod/config"
14 | "github.com/0xb10c/memo/memod/encoder"
15 | "github.com/0xb10c/memo/memod/logger"
16 | "github.com/0xb10c/memo/memod/processor"
17 |
18 | "github.com/jasonlvhit/gocron"
19 | )
20 |
21 | // SetupMempoolFetcher sets up a periodic mempool fetch job
22 | func SetupMempoolFetcher() {
23 | mempoolFetchInterval := config.GetInt("mempool.fetchInterval")
24 | s := gocron.NewScheduler()
25 | s.Every(uint64(mempoolFetchInterval)).Seconds().Do(getMempool)
26 | logger.Info.Println("Setup mempool fetcher. First fetch in", mempoolFetchInterval, "seconds")
27 | <-s.Start()
28 | defer s.Clear()
29 | }
30 |
31 | func getMempool() {
32 |
33 | body, err := fetchMempool()
34 | if err != nil {
35 | logger.Error.Printf("Could not fetch mempool: %v", err.Error())
36 | }
37 |
38 | mempool, err := encoder.DecodeFetchedMempoolBody(body) // decode fetched response body
39 | if err != nil {
40 | logger.Error.Printf("Failed to decode response body as JSON: %s", err.Error())
41 | return // we return here to stop the execution
42 | }
43 |
44 | processor.ProcessMempool(mempool)
45 |
46 | if config.GetBool("mempool.callSaveMempool") {
47 | if rand.Intn(100) <= 25 { // Only call savemempool every 4th call
48 | err = saveMempoolJSONRPC()
49 | if err != nil {
50 | logger.Error.Printf("Failed to save mempool: %s", err.Error())
51 | }
52 | }
53 | }
54 | }
55 |
56 | func fetchMempool() (body []byte, err error) {
57 | if config.GetString("mempool.fetchInterface") == "REST" {
58 |
59 | body, err := fetchMempoolFromREST()
60 | if err != nil {
61 | return nil, fmt.Errorf("Could not fetch mempool from REST: %s", err.Error())
62 | }
63 |
64 | return body, nil
65 |
66 | } else if config.GetString("mempool.fetchInterface") == "JSON-RPC" {
67 |
68 | body, err := fetchMempoolFromJSONRPC()
69 | if err != nil {
70 | return nil, fmt.Errorf("Could not fetch mempool from JSON-RPC: %s", err.Error())
71 | }
72 |
73 | return body, nil
74 |
75 | } else {
76 | return nil, errors.New("Unknown interface " + config.GetString("mempool.fetchInterface"))
77 | }
78 | }
79 |
80 | // fetches the current mempool from the Bitcoin Core REST API
81 | func fetchMempoolFromREST() (body []byte, err error) {
82 | defer logger.TrackTime(time.Now(), "fetchMempoolFromREST()")
83 |
84 | urlPrefix := config.GetString("bitcoind.rest.protocol") +
85 | "://" + config.GetString("bitcoind.rest.host") +
86 | ":" + config.GetString("bitcoind.rest.port")
87 | const urlSuffix = "/rest/mempool/contents.json"
88 |
89 | logger.Trace.Println("Fetching mempool contents from ", urlPrefix+urlSuffix)
90 |
91 | body, err = makeHTTPGETReq(urlPrefix+urlSuffix, config.GetInt("bitcoind.rest.responseTimeout"))
92 | if err != nil {
93 | return
94 | }
95 |
96 | return
97 | }
98 |
99 | func fetchMempoolFromJSONRPC() ([]byte, error) {
100 | defer logger.TrackTime(time.Now(), "fetchMempoolFromJSONRPC()")
101 | logger.Trace.Println("Fetching mempool via JSON-RPC")
102 |
103 | rpcURL := getJSONRPCURL()
104 |
105 | timeout := time.Duration(config.GetInt("bitcoind.jsonrpc.responseTimeout")) * time.Second
106 | client := http.Client{
107 | Timeout: timeout,
108 | }
109 |
110 | bodyReq := strings.NewReader("{\"jsonrpc\": \"1.0\", \"id\":\"memod-via-rpc\", \"method\": \"getrawmempool\", \"params\": [true] }")
111 | req, err := http.NewRequest("POST", rpcURL, bodyReq)
112 | if err != nil {
113 | return nil, err
114 | }
115 | req.Header.Set("Content-Type", "text/plain")
116 |
117 | resp, err := client.Do(req)
118 | if err != nil {
119 | return nil, err
120 | }
121 | defer resp.Body.Close()
122 |
123 | if resp.StatusCode != http.StatusOK {
124 | return nil, fmt.Errorf("JSON-RPC Request failed with status code %d %s", resp.StatusCode, http.StatusText(resp.StatusCode))
125 | }
126 |
127 | body, err := readResponseBody(resp)
128 | if err != nil {
129 | return nil, err
130 | }
131 |
132 | // The JSON-RPC response is encapsulated in a JSON result object
133 | // like {"result":{...},"error":null,"id":"memod-via-rpc"}
134 | // The REST response isn't. We remove this here.
135 | body = body[10 : len(body)-36]
136 |
137 | return body, nil
138 | }
139 |
140 | func getJSONRPCURL() (rpcURL string) {
141 | return config.GetString("bitcoind.jsonrpc.protocol") +
142 | "://" + config.GetString("bitcoind.jsonrpc.username") +
143 | ":" + config.GetString("bitcoind.jsonrpc.password") +
144 | "@" + config.GetString("bitcoind.jsonrpc.host") +
145 | ":" + config.GetString("bitcoind.jsonrpc.port")
146 | }
147 |
148 | func saveMempoolJSONRPC() (err error) {
149 | defer logger.TrackTime(time.Now(), "saveMempoolJSONRPC()")
150 | logger.Trace.Println("Saving mempool via JSON-RPC")
151 |
152 | client := http.Client{Timeout: 5 * time.Second}
153 | rpcURL := getJSONRPCURL()
154 |
155 | bodyReq := strings.NewReader("{\"jsonrpc\": \"1.0\", \"id\":\"memod-via-rpc\", \"method\": \"savemempool\" }")
156 | req, err := http.NewRequest("POST", rpcURL, bodyReq)
157 | if err != nil {
158 | return
159 | }
160 | req.Header.Set("Content-Type", "text/plain")
161 |
162 | resp, err := client.Do(req)
163 | if err != nil {
164 | return err
165 | }
166 | defer resp.Body.Close()
167 |
168 | if resp.StatusCode != http.StatusOK {
169 | return fmt.Errorf("JSON-RPC Request failed with status code %d %s", resp.StatusCode, http.StatusText(resp.StatusCode))
170 | }
171 |
172 | body, err := readResponseBody(resp)
173 | if err != nil {
174 | return err
175 | }
176 |
177 | if string(body) != "{\"result\":null,\"error\":null,\"id\":\"memod-via-rpc\"}\n" {
178 | return fmt.Errorf("JSON-RPC Request failed with response %s", string(body))
179 | }
180 |
181 | return nil
182 | }
183 |
--------------------------------------------------------------------------------
/memod/fetcher/util.go:
--------------------------------------------------------------------------------
1 | package fetcher
2 |
3 | import (
4 | "io/ioutil"
5 | "net/http"
6 | "time"
7 |
8 | "github.com/0xb10c/memo/memod/logger"
9 | )
10 |
11 | // read the response body
12 | func readResponseBody(resp *http.Response) ([]byte, error) {
13 | body, err := ioutil.ReadAll(resp.Body)
14 | if err != nil {
15 | logger.Error.Println(err.Error())
16 | }
17 | defer resp.Body.Close()
18 |
19 | return body, nil
20 | }
21 |
22 | func makeHTTPGETReq(url string, timeout int) (body []byte, err error) {
23 | client := http.Client{Timeout: time.Duration(timeout) * time.Second}
24 | resp, err := client.Get(url)
25 | if err != nil {
26 | return
27 | }
28 |
29 | body, err = readResponseBody(resp)
30 | if err != nil {
31 | return
32 | }
33 |
34 | return
35 | }
36 |
37 | func getMyceliumFeeAPICert() (cert []byte) {
38 |
39 | // Mycelium uses a self signed certificate for their feerate API.
40 | // This cert is stored here.
41 |
42 | certAsPem := `-----BEGIN CERTIFICATE-----
43 | MIIDTTCCAjWgAwIBAgIJAOudLU3vQBxpMA0GCSqGSIb3DQEBCwUAMD0xGzAZBgNV
44 | BAMMEm13czIwLm15Y2VsaXVtLmNvbTERMA8GA1UECgwITXljZWxpdW0xCzAJBgNV
45 | BAYTAlhYMB4XDTE2MTExNjE1MTExMFoXDTE5MDgxMzE1MTExMFowPTEbMBkGA1UE
46 | AwwSbXdzMjAubXljZWxpdW0uY29tMREwDwYDVQQKDAhNeWNlbGl1bTELMAkGA1UE
47 | BhMCWFgwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC7jTD9MLV/k4y2
48 | qJH+aLfJU3pdMmdlciAz1L3T03rT6QDOaT4eanGqpeYNqO/udp8GOFzvnJJ1qXU9
49 | XI3zYOVwK+m/3JBG5B4olQjibkahqi4yterxxXjSzGP5apG//9kfKPx8Q2P47EG+
50 | wt2cdVF2WPicK5+42D7QPwM3cEcohPzzHHaVB0tFt9bcFgooGCIlCz/mo7rD3PsK
51 | nxmdx/0T3Tyh8iLCuLh/PqcLPWKPtpgy3vo3W9gxVnfZj0KR8qMMbQV8KjjCR74s
52 | l3Mfs3bfSKuJKLxK/qElu3BZJRZ6CZzodSb6n0+s9qHrwfJ2FjNvj7jzACv+lTU6
53 | DHGgRd3rAgMBAAGjUDBOMB0GA1UdDgQWBBT2Di9Vrc35R3LSmUfQbPzJrFo6PTAf
54 | BgNVHSMEGDAWgBT2Di9Vrc35R3LSmUfQbPzJrFo6PTAMBgNVHRMEBTADAQH/MA0G
55 | CSqGSIb3DQEBCwUAA4IBAQBFVqonJzLPL/5OY+yy/AJnMqscgGNMiKn9lMi9xU1H
56 | 2mx1Tk4ziJgfT7OtoPIwTPMkqGLRfh6gGQnuePmvuG9MrjNNYBEPB0/esvVOws8V
57 | wgYESOC6b4uGyLCv79gyUQFQgwo7CgMxs1vltZEk3DUx1y6eiHGyLiSEE5fmxLQY
58 | xUGiv1w/ZWSlqDiYRl7BdjVtVDxqXyPgcBIW9+k+iRhae1M/nCB8ZpU2IR8x2IjC
59 | fDm1IEMFno98J0hAFyHPVBXsXWLKuDRNbRAWJpe2TmK8G4/MxwgOS41RASTG6icK
60 | mFjasRwenevsfGl0e5Nsth0ToynsQzuO3Tv2NzQbYgOG
61 | -----END CERTIFICATE-----`
62 |
63 | return []byte(certAsPem)
64 | }
65 |
--------------------------------------------------------------------------------
/memod/go.mod:
--------------------------------------------------------------------------------
1 | module github.com/0xb10c/memo/memod
2 |
3 | go 1.12
4 |
5 | require (
6 | github.com/0xb10c/rawtx v1.0.0
7 | github.com/btcsuite/btcd v0.0.0-20190824003749-130ea5bddde3
8 | github.com/gomodule/redigo v2.0.0+incompatible
9 | github.com/jasonlvhit/gocron v0.0.0-20190920201010-985d45da66c5
10 | github.com/mattn/go-sqlite3 v1.11.0
11 | github.com/pebbe/zmq4 v1.0.0
12 | github.com/spf13/viper v1.4.0
13 | github.com/tidwall/gjson v1.3.2
14 | )
15 |
--------------------------------------------------------------------------------
/memod/go.sum:
--------------------------------------------------------------------------------
1 | cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
2 | github.com/0xb10c/rawtx v0.0.0-20190916163152-442e95d15078 h1:L5O290+Ee7NZ2e5imjzuFVpQc9U10lTHpvpYlkhlCgI=
3 | github.com/0xb10c/rawtx v0.0.0-20190916163152-442e95d15078/go.mod h1:io8NRwsYR1B6kq/zjtIuC8srkAWez/jEEcdgRfqA8E4=
4 | github.com/0xb10c/rawtx v0.0.0-20190928143028-49c49b945cec h1:RIpvk7XXc6LqnVt6wmvUs0PjwXj6VPsOpGw8p5Zb0x4=
5 | github.com/0xb10c/rawtx v0.0.0-20190928143028-49c49b945cec/go.mod h1:io8NRwsYR1B6kq/zjtIuC8srkAWez/jEEcdgRfqA8E4=
6 | github.com/0xb10c/rawtx v1.0.0 h1:kowu7Fw5k8i5biShkaerQyRN61H6oYnwuVpIUT97xyw=
7 | github.com/0xb10c/rawtx v1.0.0/go.mod h1:io8NRwsYR1B6kq/zjtIuC8srkAWez/jEEcdgRfqA8E4=
8 | github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
9 | github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
10 | github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
11 | github.com/aead/siphash v1.0.1/go.mod h1:Nywa3cDsYNNK3gaciGTWPwHt0wlpNV15vwmswBAUSII=
12 | github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
13 | github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
14 | github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
15 | github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
16 | github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
17 | github.com/btcsuite/btcd v0.0.0-20190824003749-130ea5bddde3 h1:A/EVblehb75cUgXA5njHPn0kLAsykn6mJGz7rnmW5W0=
18 | github.com/btcsuite/btcd v0.0.0-20190824003749-130ea5bddde3/go.mod h1:3J08xEfcugPacsc34/LKRU2yO7YmuT8yt28J8k2+rrI=
19 | github.com/btcsuite/btclog v0.0.0-20170628155309-84c8d2346e9f/go.mod h1:TdznJufoqS23FtqVCzL0ZqgP5MqXbb4fg/WgDys70nA=
20 | github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d h1:yJzD/yFppdVCf6ApMkVy8cUxV0XrxdP9rVf6D87/Mng=
21 | github.com/btcsuite/btcutil v0.0.0-20190425235716-9e5f4b9a998d/go.mod h1:+5NJ2+qvTyV9exUAL/rxXi3DcLg2Ts+ymUAY5y4NvMg=
22 | github.com/btcsuite/go-socks v0.0.0-20170105172521-4720035b7bfd/go.mod h1:HHNXQzUsZCxOoE+CPiyCTO6x34Zs86zZUiwtpXoGdtg=
23 | github.com/btcsuite/goleveldb v0.0.0-20160330041536-7834afc9e8cd/go.mod h1:F+uVaaLLH7j4eDXPRvw78tMflu7Ie2bzYOH4Y8rRKBY=
24 | github.com/btcsuite/snappy-go v0.0.0-20151229074030-0bdef8d06723/go.mod h1:8woku9dyThutzjeg+3xrA5iCpBRH8XEEg3lh6TiUghc=
25 | github.com/btcsuite/websocket v0.0.0-20150119174127-31079b680792/go.mod h1:ghJtEyQwv5/p4Mg4C0fgbePVuGr935/5ddU9Z3TmDRY=
26 | github.com/btcsuite/winsvc v1.0.0/go.mod h1:jsenWakMcC0zFBFurPLEAyrnc/teJEM1O46fmI40EZs=
27 | github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
28 | github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
29 | github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
30 | github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
31 | github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
32 | github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
33 | github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
34 | github.com/davecgh/go-spew v0.0.0-20171005155431-ecdeabc65495/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
35 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
36 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
37 | github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
38 | github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
39 | github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
40 | github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
41 | github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
42 | github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
43 | github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
44 | github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
45 | github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
46 | github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
47 | github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
48 | github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
49 | github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
50 | github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
51 | github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
52 | github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
53 | github.com/gomodule/redigo v2.0.0+incompatible h1:K/R+8tc58AaqLkqG2Ol3Qk+DR/TlNuhuh457pBFPtt0=
54 | github.com/gomodule/redigo v2.0.0+incompatible/go.mod h1:B4C85qUVwatsJoIUNIfCRsp7qO0iAmpGFZ4EELWSbC4=
55 | github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
56 | github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
57 | github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
58 | github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
59 | github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
60 | github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
61 | github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
62 | github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
63 | github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
64 | github.com/jasonlvhit/gocron v0.0.0-20190829164038-7ef3bafdc25c h1:Pe7FHOh2zH0MReaY0ksqwECozI8O80IwpAXBmBxdq8s=
65 | github.com/jasonlvhit/gocron v0.0.0-20190829164038-7ef3bafdc25c/go.mod h1:rwi/esz/h+4oWLhbWWK7f6dtmgLzxeZhnwGr7MCsTNk=
66 | github.com/jasonlvhit/gocron v0.0.0-20190920201010-985d45da66c5 h1:m1t5VsnIRS9HY+X/NReDtPvLriGcMx3Foc8a97Ogyyk=
67 | github.com/jasonlvhit/gocron v0.0.0-20190920201010-985d45da66c5/go.mod h1:rwi/esz/h+4oWLhbWWK7f6dtmgLzxeZhnwGr7MCsTNk=
68 | github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
69 | github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
70 | github.com/jrick/logrotate v1.0.0/go.mod h1:LNinyqDIJnpAur+b8yyulnQw/wDuN1+BYKlTRt3OuAQ=
71 | github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
72 | github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
73 | github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
74 | github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4=
75 | github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
76 | github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
77 | github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
78 | github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
79 | github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
80 | github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
81 | github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
82 | github.com/magiconair/properties v1.8.0 h1:LLgXmsheXeRoUOBOjtwPQCWIYqM/LU1ayDtDePerRcY=
83 | github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
84 | github.com/mattn/go-sqlite3 v1.11.0 h1:LDdKkqtYlom37fkvqs8rMPFKAMe8+SgjbwZ6ex1/A/Q=
85 | github.com/mattn/go-sqlite3 v1.11.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
86 | github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
87 | github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE=
88 | github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
89 | github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
90 | github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
91 | github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
92 | github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
93 | github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
94 | github.com/pebbe/zmq4 v1.0.0 h1:D+MSmPpqkL5PSSmnh8g51ogirUCyemThuZzLW7Nrt78=
95 | github.com/pebbe/zmq4 v1.0.0/go.mod h1:7N4y5R18zBiu3l0vajMUWQgZyjv464prE8RCyBcmnZM=
96 | github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
97 | github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
98 | github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
99 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
100 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
101 | github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
102 | github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
103 | github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
104 | github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
105 | github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
106 | github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
107 | github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
108 | github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
109 | github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
110 | github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
111 | github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
112 | github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
113 | github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
114 | github.com/spf13/afero v1.1.2 h1:m8/z1t7/fwjysjQRYbP0RD+bUIF/8tJwPdEZsI83ACI=
115 | github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
116 | github.com/spf13/cast v1.3.0 h1:oget//CVOEoFewqQxwr0Ej5yjygnqGkvggSE/gB35Q8=
117 | github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
118 | github.com/spf13/jwalterweatherman v1.0.0 h1:XHEdyB+EcvlqZamSM4ZOMGlc93t6AcsBEu9Gc1vn7yk=
119 | github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
120 | github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
121 | github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
122 | github.com/spf13/viper v1.4.0 h1:yXHLWeravcrgGyFSyCgdYpXQ9dR9c/WED3pg1RhxqEU=
123 | github.com/spf13/viper v1.4.0/go.mod h1:PTJ7Z/lr49W6bUbkmS1V3by4uWynFiR9p7+dSq/yZzE=
124 | github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
125 | github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w=
126 | github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
127 | github.com/tidwall/gjson v1.3.2 h1:+7p3qQFaH3fOMXAJSrdZwGKcOO/lYdGS0HqGhPqDdTI=
128 | github.com/tidwall/gjson v1.3.2/go.mod h1:P256ACg0Mn+j1RXIDXoss50DeIABTYK1PULOJHhxOls=
129 | github.com/tidwall/match v1.0.1 h1:PnKP62LPNxHKTwvHHZZzdOAOCtsJTjo6dZLCwpKm5xc=
130 | github.com/tidwall/match v1.0.1/go.mod h1:LujAq0jyVjBy028G1WhWfIzbpQfMO8bBZ6Tyb0+pL9E=
131 | github.com/tidwall/pretty v1.0.0 h1:HsD+QiTn7sK6flMKIvNmpqz1qrpP3Ps6jOKIKMooyg4=
132 | github.com/tidwall/pretty v1.0.0/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
133 | github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
134 | github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
135 | github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
136 | github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
137 | go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
138 | go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
139 | go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
140 | go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
141 | golang.org/x/crypto v0.0.0-20170930174604-9419663f5a44/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
142 | golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
143 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
144 | golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
145 | golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
146 | golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
147 | golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
148 | golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
149 | golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
150 | golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
151 | golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
152 | golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
153 | golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
154 | golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
155 | golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
156 | golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
157 | golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
158 | golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
159 | golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
160 | golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
161 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a h1:1BGLXjeY4akVXGgbC9HugT3Jv3hCI0z56oJR5vAMgBU=
162 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
163 | golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
164 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
165 | golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
166 | golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
167 | golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
168 | golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
169 | google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
170 | google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
171 | google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
172 | google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
173 | gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
174 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
175 | gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
176 | gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
177 | gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
178 | gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
179 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
180 | gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
181 | gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
182 | gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
183 | gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
184 | honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
185 |
--------------------------------------------------------------------------------
/memod/logger/color.go:
--------------------------------------------------------------------------------
1 | package logger
2 |
3 | import (
4 | "github.com/0xb10c/memo/memod/config"
5 | )
6 |
7 | const prefix = "\033["
8 | const resetAll = prefix + "0m"
9 | const resetFontColor = prefix + "39m"
10 |
11 | func colorIsEnabled() bool {
12 | return config.GetBool("log.colorizeOutput")
13 | }
14 |
15 | // Red colorizes the input red
16 | func Red(input string) string {
17 | if colorIsEnabled() {
18 | return prefix + "31m" + input + resetFontColor
19 | }
20 | return input
21 | }
22 |
23 | // Blue colorizes the input red
24 | func Blue(input string) string {
25 | if colorIsEnabled() {
26 | return prefix + "36m" + input + resetFontColor
27 | }
28 | return input
29 | }
30 |
31 | // Yellow colorizes the input red
32 | func Yellow(input string) string {
33 | if colorIsEnabled() {
34 | return prefix + "93m" + input + resetFontColor
35 | }
36 | return input
37 | }
38 |
39 | // Dim colorizes the input red
40 | func Dim(input string) string {
41 | if colorIsEnabled() {
42 | return prefix + "2m" + input + prefix + "22m"
43 | }
44 | return input
45 | }
46 |
--------------------------------------------------------------------------------
/memod/logger/logger.go:
--------------------------------------------------------------------------------
1 | package logger
2 |
3 | import (
4 | "flag"
5 | "io/ioutil"
6 | "log"
7 | "os"
8 | "time"
9 |
10 | "github.com/0xb10c/memo/memod/config"
11 | )
12 |
13 | var (
14 | // Trace logs trace messages
15 | Trace *log.Logger
16 | // Info logs info messages
17 | Info *log.Logger
18 | // Warning logs warning messages
19 | Warning *log.Logger
20 | // Error logs error messages
21 | Error *log.Logger
22 | )
23 |
24 | func init() {
25 | // do not init a logger when running tests
26 | if flag.Lookup("test.v") != nil {
27 | return
28 | }
29 |
30 | traceHandle := ioutil.Discard
31 | if config.GetBool("log.enableTrace") {
32 | traceHandle = os.Stdout
33 | }
34 |
35 | infoHandle := os.Stdout
36 | warningHandle := os.Stdout
37 | errorHandle := os.Stderr
38 |
39 | logFlags := log.Ldate | log.Ltime //log.Lshortfile
40 |
41 | Trace = log.New(traceHandle, Dim("TRACE: "), logFlags)
42 | Info = log.New(infoHandle, Blue("INFO: "), logFlags)
43 | Warning = log.New(warningHandle, Yellow("WARN: "), logFlags)
44 | Error = log.New(errorHandle, Red("ERROR: "), logFlags)
45 |
46 | Info.Println("Setup logger. Logging Trace:", config.GetBool("log.enableTrace"))
47 | }
48 |
49 | // TrackTime tracks the time a function takes till return and logs it to Trace
50 | func TrackTime(start time.Time, funcname string) {
51 | // do not print tracked time in tests
52 | if flag.Lookup("test.v") != nil {
53 | return
54 | }
55 | elapsed := time.Since(start)
56 | Trace.Println(funcname + " took " + elapsed.String())
57 | }
58 |
--------------------------------------------------------------------------------
/memod/memod.go:
--------------------------------------------------------------------------------
1 | package main
2 |
3 | import (
4 | app "github.com/0xb10c/memo/memod/app"
5 | )
6 |
7 | func main() {
8 | app.Run()
9 | }
10 |
--------------------------------------------------------------------------------
/memod/memod.service:
--------------------------------------------------------------------------------
1 | [Unit]
2 | Description=memod - mempool observer deamon
3 | Wants=network.target
4 | After=network.target
5 |
6 | [Service]
7 | Type=simple
8 | Restart=on-failure
9 | RestartSec=5s
10 | ExecStart=/opt/memo/memod/memod
11 | WorkingDirectory=/opt/memo/memod/
12 | # FIXME: Add your user setup here
13 |
14 |
15 | [Install]
16 | WantedBy=multi-user.target
--------------------------------------------------------------------------------
/memod/processor/stat_generator.go:
--------------------------------------------------------------------------------
1 | package processor
2 |
3 | /* processes the mempool and generates statistics */
4 |
5 | import (
6 | "sort"
7 | "time"
8 |
9 | "github.com/0xb10c/memo/memod/config"
10 | "github.com/0xb10c/memo/memod/database"
11 | "github.com/0xb10c/memo/memod/logger"
12 | "github.com/0xb10c/memo/memod/types"
13 | )
14 |
15 | // cMEGABYTE: size of one megabyte in byte
16 | const cMEGABYTE = 1000000
17 |
18 | // cSATOSHIPERBITCOIN: satoshi per bitcoin
19 | const cSATOSHIPERBITCOIN = 100000000
20 |
21 | // ProcessMempool retrives the mempool and starts various processing functions on it
22 | func ProcessMempool(mempool map[string]types.PartialTransaction) {
23 | if config.GetBool("mempool.processing.processHistoricalMempool") {
24 | go historicalMempool(mempool)
25 | }
26 |
27 | if config.GetBool("mempool.processing.processCurrentMempool") {
28 | go currentMempool(mempool) // start _current mempool_ stat generation in a goroutine
29 | }
30 |
31 | if config.GetBool("mempool.processing.processTransactionStats") {
32 | go transactionStatsMempool(mempool)
33 | }
34 |
35 | }
36 |
37 | func currentMempool(mempool map[string]types.PartialTransaction) {
38 | feerateMap, mempoolSizeInByte, megabyteMarkers := generateCurrentMempoolStats(mempool)
39 |
40 | err := database.WriteCurrentMempoolData(feerateMap, mempoolSizeInByte, megabyteMarkers)
41 | if err != nil {
42 | logger.Error.Printf("Failed to write Current Mempool to database: %s", err.Error())
43 | }
44 |
45 | logger.Info.Println("Success writing Current Mempool to database.")
46 | }
47 |
48 | func historicalMempool(mempool map[string]types.PartialTransaction) {
49 |
50 | const timeframe2h = 1
51 | const timeframe12h = 2
52 | const timeframe48h = 3
53 | const timeframe7d = 4
54 | const timeframe30d = 5
55 | const timeframe180d = 6
56 |
57 | needsUpdate, err := database.ReadHistoricalMempoolNeedUpdate()
58 | if err != nil {
59 | logger.Error.Printf("Failed to get Needs Update data from database: %s", err.Error())
60 | }
61 |
62 | if needsUpdate.Update2h || needsUpdate.Update12h || needsUpdate.Update48h || needsUpdate.Update7d || needsUpdate.Update30d || needsUpdate.Update180d {
63 |
64 | countInBuckets, feeInBuckets, sizeInBuckets := generateHistoricalMempoolStats(mempool)
65 |
66 | if needsUpdate.Update2h {
67 | logger.Info.Println("Writing 2h Historical Mempool data.")
68 | err = database.WriteHistoricalMempoolData(countInBuckets, feeInBuckets, sizeInBuckets, timeframe2h)
69 | if err != nil {
70 | logger.Error.Printf("Failed to write Historical Mempool to database: %s", err.Error())
71 | return
72 | }
73 | }
74 |
75 | if needsUpdate.Update12h {
76 | logger.Info.Println("Writing 12h Historical Mempool data.")
77 | err = database.WriteHistoricalMempoolData(countInBuckets, feeInBuckets, sizeInBuckets, timeframe12h)
78 | if err != nil {
79 | logger.Error.Printf("Failed to write Historical Mempool to database: %s", err.Error())
80 | return
81 | }
82 | }
83 |
84 | if needsUpdate.Update48h {
85 | logger.Info.Println("Writing 48h Historical Mempool data.")
86 | err = database.WriteHistoricalMempoolData(countInBuckets, feeInBuckets, sizeInBuckets, timeframe48h)
87 | if err != nil {
88 | logger.Error.Printf("Failed to write Historical Mempool to database: %s", err.Error())
89 | return
90 | }
91 | }
92 |
93 | if needsUpdate.Update7d {
94 | logger.Info.Println("Writing 7d Historical Mempool data.")
95 | err = database.WriteHistoricalMempoolData(countInBuckets, feeInBuckets, sizeInBuckets, timeframe7d)
96 | if err != nil {
97 | logger.Error.Printf("Failed to write Historical Mempool to database: %s", err.Error())
98 | return
99 | }
100 | }
101 |
102 | if needsUpdate.Update30d {
103 | logger.Info.Println("Writing 30d Historical Mempool data.")
104 | err = database.WriteHistoricalMempoolData(countInBuckets, feeInBuckets, sizeInBuckets, timeframe30d)
105 | if err != nil {
106 | logger.Error.Printf("Failed to write Historical Mempool to database: %s", err.Error())
107 | return
108 | }
109 | }
110 |
111 | if needsUpdate.Update180d {
112 | logger.Info.Println("Writing 180d Historical Mempool data.")
113 | err = database.WriteHistoricalMempoolData(countInBuckets, feeInBuckets, sizeInBuckets, timeframe180d)
114 | if err != nil {
115 | logger.Error.Printf("Failed to write Historical Mempool to database: %s", err.Error())
116 | return
117 | }
118 | }
119 |
120 | logger.Info.Println("Success writing Historical Mempool to database.")
121 | }
122 | }
123 |
124 | func transactionStatsMempool(mempool map[string]types.PartialTransaction) {
125 | segwitCount, rbfCount, txCount := generateTransactionStats(mempool)
126 |
127 | err := database.WriteCurrentTransactionStats(segwitCount, rbfCount, txCount)
128 | if err != nil {
129 | logger.Error.Printf("Failed to write Transaction Stats to database: %s", err.Error())
130 | return
131 | }
132 |
133 | logger.Info.Println("Success writing Transaction Stats to database.")
134 | }
135 |
136 | /* generateCurrentMempoolStats()
137 | This function generates the _Current Mempool_ data. Which is:
138 | - The size of the transactions in the mempool `mempoolSizeInByte`
139 | - A map mapping the transaction count to the feerate (as a whole
140 | needsUpdatember). Named `feerateMap`.
141 | - A list positions in the mempool (tx count), when sorted by
142 | feerate, which each mark one megabyte worth of transactions.
143 | Positions starting from the top. Named `megabyteMarkers`.
144 | */
145 | func generateCurrentMempoolStats(mempool map[string]types.PartialTransaction) (map[int]int, int, []int) {
146 | defer logger.TrackTime(time.Now(), "generateCurrentMempoolStats()")
147 |
148 | // this represents a entry in a mempool list (memlist).
149 | // The memlist can be sorted by feerate making it to a
150 | // memqueue.
151 | type memlistEntry struct {
152 | feerate float64
153 | size int
154 | }
155 |
156 | mempoolPos := 0
157 | mempoolSizeInByte := 0
158 | var megabyteMarkers []int
159 | feerateMap := make(map[int]int)
160 | memlist := make([]memlistEntry, len(mempool))
161 |
162 | for _, tx := range mempool {
163 | mempoolSizeInByte += tx.Size
164 | feerate := tx.Fee * cSATOSHIPERBITCOIN / float64(tx.Size)
165 | feerateMap[int(feerate)]++
166 |
167 | memlist[mempoolPos] = memlistEntry{feerate: feerate, size: tx.Size}
168 | mempoolPos++
169 | }
170 |
171 | // sort the list of memlistEntry's by highest feerate first
172 | sort.Slice(memlist, func(i, j int) bool {
173 | return memlist[i].feerate > memlist[j].feerate
174 | })
175 |
176 | memlistPos := len(memlist)
177 | megabyteBucket := cMEGABYTE
178 | for _, entry := range memlist {
179 | if megabyteBucket-entry.size > 0 { // if entry.size fits in the bucket
180 | megabyteBucket = megabyteBucket - entry.size
181 | memlistPos--
182 | } else { // if entry.size doesn't fit in the bucket
183 | megabyteBucket = cMEGABYTE - entry.size // start a new megabyte bucket mineedsUpdates the current entry.size
184 | megabyteMarkers = append(megabyteMarkers, memlistPos) // append current position to the megabyteMarkers list
185 | memlistPos--
186 | }
187 | }
188 |
189 | return feerateMap, mempoolSizeInByte, megabyteMarkers
190 | }
191 |
192 | var feerateBuckets = [40]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 15, 18, 22, 27, 33, 41, 50, 62, 76, 93, 114, 140, 172, 212, 261, 321, 395, 486, 598, 736, 905, 1113, 1369, 1684, 2071, 2547, 3133, 3854, 3855}
193 |
194 | // generates a list of counts of transactions representing the count in a feerate bucket
195 | func generateHistoricalMempoolStats(mempool map[string]types.PartialTransaction) (countInBuckets []int, feeInBuckets []float64, sizeInBuckets []int) {
196 |
197 | countInBuckets = make([]int, len(feerateBuckets), len(feerateBuckets))
198 | feeInBuckets = make([]float64, len(feerateBuckets), len(feerateBuckets))
199 | sizeInBuckets = make([]int, len(feerateBuckets), len(feerateBuckets))
200 |
201 | for _, tx := range mempool {
202 | feerate := tx.Fee * cSATOSHIPERBITCOIN / float64(tx.Size)
203 | bucketIndex := findBucketForFeerate(feerate)
204 | countInBuckets[bucketIndex]++
205 | feeInBuckets[bucketIndex] += tx.Fee
206 | sizeInBuckets[bucketIndex] += tx.Size
207 | }
208 |
209 | return
210 | }
211 |
212 | // finds the bucket index for a given feerate in feerateBuckets. the last bucket is a catch all larger-equal.
213 | // given a feerate of 19 it gives the bucket index for 22. (since it's bigger than 18)
214 | // buckets should be read like "in between or equal feerate [index-1] [index]"
215 | func findBucketForFeerate(feerate float64) int {
216 | i := sort.Search(len(feerateBuckets), func(i int) bool { return feerateBuckets[i] >= int(feerate) })
217 | if i < len(feerateBuckets) && feerateBuckets[i] >= int(feerate) {
218 | return i
219 | }
220 | return len(feerateBuckets) - 1
221 | }
222 |
223 | func generateTransactionStats(mempool map[string]types.PartialTransaction) (segwitCount int, rbfCount int, txCount int) {
224 |
225 | for txid, tx := range mempool {
226 | if txid != tx.Wtxid {
227 | segwitCount++
228 | }
229 | if tx.Bip125Replaceable {
230 | rbfCount++
231 | }
232 | }
233 |
234 | txCount = len(mempool)
235 |
236 | return
237 | }
238 |
--------------------------------------------------------------------------------
/memod/processor/stat_generator_test.go:
--------------------------------------------------------------------------------
1 | package processor
2 |
3 | import "testing"
4 |
5 | func TestFindBucketForFeerate(t *testing.T) {
6 |
7 | var feerateBuckets = [40]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 15, 18, 22, 27, 33, 41, 50, 62, 76, 93, 114, 140, 172, 212, 261, 321, 395, 486, 598, 736, 905, 1113, 1369, 1684, 2071, 2547, 3133, 3854, 3855}
8 |
9 | // test a feerate of 0
10 | t1 := findBucketForFeerate(0)
11 | if t1 != 0 {
12 | t.Errorf("findBucketForFeerate(0) = %d; want 0", t1)
13 | }
14 |
15 | // test a feerate of 1
16 | t2 := findBucketForFeerate(1)
17 | if t2 != 0 {
18 | t.Errorf("findBucketForFeerate(1) = %d; want 0", t1)
19 | }
20 |
21 | // test a feerate of 2
22 | t3 := findBucketForFeerate(2)
23 | if t3 != 1 {
24 | t.Errorf("findBucketForFeerate(2) = %d; want 1", t3)
25 | }
26 |
27 | // test a feerate of 18
28 | t4 := findBucketForFeerate(18)
29 | if t4 != 12 {
30 | t.Errorf("findBucketForFeerate(18) = %d; want 12", t4)
31 | }
32 |
33 | // test a feerate of 19
34 | t5 := findBucketForFeerate(19)
35 | if t5 != 13 {
36 | t.Errorf("findBucketForFeerate(19) = %d; want 12, bucket = %d", t5, feerateBuckets[t5])
37 | }
38 |
39 | // test a feerate of -1
40 | t6 := findBucketForFeerate(-1)
41 | if t6 != 0 {
42 | t.Errorf("findBucketForFeerate(-1) = %d; want 0, bucket = %d", t6, feerateBuckets[t6])
43 | }
44 |
45 | // test a feerate of 19
46 | t10 := findBucketForFeerate(4000)
47 | if t10 != 39 {
48 | t.Errorf("findBucketForFeerate(4000) = %d; want 39", t10)
49 | }
50 | }
51 |
--------------------------------------------------------------------------------
/memod/processor/zmq_handler.go:
--------------------------------------------------------------------------------
1 | package processor
2 |
3 | import (
4 | "encoding/binary"
5 | "strconv"
6 | "strings"
7 | "time"
8 |
9 | "github.com/0xb10c/memo/memod/database"
10 | "github.com/0xb10c/memo/memod/logger"
11 | "github.com/0xb10c/memo/memod/types"
12 | "github.com/btcsuite/btcd/wire"
13 |
14 | "github.com/0xb10c/rawtx"
15 | )
16 |
17 | // This file includes the functions to handle zmq events
18 | // and their subfunctions.
19 |
20 | /* ----- RAW BLOCK handling ----- */
21 |
22 | // HandleRawBlock handles a raw incoming zmq block
23 | func HandleRawBlock(payload string) {
24 | block, err := deserializeRawBlock(payload)
25 | if err != nil {
26 | logger.Error.Printf("Error handling raw block: %v", err)
27 | }
28 |
29 | height := getBlockHeightFromCoinbase(block.Transactions[0])
30 | numTx := len(block.Transactions)
31 | sizeWithWitness := block.SerializeSize()
32 | sizeWithoutWitness := block.SerializeSizeStripped()
33 | weight := sizeWithWitness + sizeWithoutWitness*3
34 |
35 | err = database.WriteNewBlockData(height, numTx, sizeWithWitness, weight)
36 | if err != nil {
37 | logger.Error.Printf("Error writing block to database: %v", err)
38 | }
39 |
40 | // According to https://rusnak.io/longest-txid-prefix-collision-in-bitcoin/ (updated 2019)
41 | // the longest TXID collision is 15 hex bytes long. Choosing a short txid of 16 here should
42 | // be way more than enough. Compared to the blog post, TXIDs are here only compared to the 30k
43 | // transactions displayed in the Bitcoin Transaction Monitor.
44 | const shortTXIDLength int = 16
45 | shortTXIDs := make([]string, 0, len(block.Transactions))
46 | for _, tx := range block.Transactions {
47 | shortTXIDs = append(shortTXIDs, tx.TxHash().String()[0:shortTXIDLength])
48 | }
49 |
50 | err = database.WriteNewBlockEntry(height, shortTXIDs)
51 | if err != nil {
52 | logger.Error.Printf("Error writing block entries to database: %v", err)
53 | }
54 |
55 | logger.Info.Printf("Success writing new block %d with %d transactions, size %d, weight %d", height, numTx, sizeWithWitness, weight)
56 | }
57 |
58 | func deserializeRawBlock(rawBlock string) (block *wire.MsgBlock, err error) {
59 | defer logger.TrackTime(time.Now(), "deserializeRawBlock()")
60 | blockHeader := wire.BlockHeader{}
61 | block = wire.NewMsgBlock(&blockHeader)
62 | r := strings.NewReader(rawBlock)
63 | err = block.Deserialize(r)
64 | if err != nil {
65 | return
66 | }
67 | return
68 | }
69 |
70 | func getBlockHeightFromCoinbase(coinbase *wire.MsgTx) (height int) {
71 | defer logger.TrackTime(time.Now(), "getBlockHeightFromCoinbase()")
72 | // To get the block height we look into the coinbase transaction
73 | // (the first transaction in a block). The scriptsig of the coin-
74 | // base transaction starts with the height. **The first byte sets
75 | // height length**. (only true for blocks with BIP34, but this can
76 | // be ignored here, since we probably don't work with older blocks)
77 | heightLength := coinbase.TxIn[0].SignatureScript[0]
78 |
79 | // we get the bytes from pos 1 till pos heightLength + 1 since the
80 | // second parameter is exclusive in Go
81 | heightLittleEndian := coinbase.TxIn[0].SignatureScript[1 : heightLength+1]
82 |
83 | // since we want the block height in a int32 (4 byte) and big
84 | // endian we first add padding (at the end, since it's little
85 | // endian) and then convert it to big endian.
86 | for i := heightLength; i < 4; i++ {
87 | heightLittleEndian = append(heightLittleEndian, 0x0)
88 | }
89 |
90 | height = int(binary.LittleEndian.Uint32(heightLittleEndian))
91 | return
92 | }
93 |
94 | /* ----- HASH BLOCK handling ----- */
95 |
96 | // HandleHashBlock handles a incoming zmq block hash
97 | func HandleHashBlock(payload string) {
98 | logger.Warning.Println("HandleHashBlock() not Implemented")
99 | }
100 |
101 | /* ----- RAW TX handling ----- */
102 |
103 | // HandleRawTx handles a incoming zmq raw tx
104 | func HandleRawTx(payload string) {
105 | logger.Warning.Println("HandleRawTx() not Implemented")
106 | }
107 |
108 | // HandleRawTxWithSizeAndFee handles the special rawtx2 zmq message
109 | // which contains a tx, it's size and it's fee as 64 bit ints
110 | func HandleRawTxWithSizeAndFee(payload string) {
111 | payloadLength := len(payload)
112 |
113 | tx, err := rawtx.DeserializeRawTxBytes([]byte(payload[0 : payloadLength-16]))
114 | if err != nil {
115 | logger.Error.Printf("Error handling raw tx: %v", err)
116 | }
117 |
118 | sizeBytes := []byte(payload[payloadLength-16 : payloadLength-8])
119 | feeBytes := []byte(payload[payloadLength-8 : payloadLength])
120 |
121 | sizeInByte := int64(binary.LittleEndian.Uint64(sizeBytes))
122 | feeInSat := int64(binary.LittleEndian.Uint64(feeBytes))
123 |
124 | me := types.MempoolEntry{}
125 | me.EntryTime = time.Now().Unix()
126 | me.TxID = tx.Hash
127 | me.Fee = feeInSat
128 | me.Size = sizeInByte
129 | me.Version = tx.Version
130 | me.InputCount = tx.GetNumInputs()
131 | me.OutputCount = tx.GetNumOutputs()
132 | me.Locktime = tx.GetLocktime()
133 | me.OutputSum = tx.GetOutputSum()
134 | me.SpendsSegWit = tx.IsSpendingSegWit()
135 | me.SpendsMultisig = tx.IsSpendingMultisig()
136 | me.IsBIP69 = tx.IsBIP69Compliant()
137 | me.SignalsRBF = tx.IsExplicitlyRBFSignaling()
138 |
139 | me.Spends = make(map[string]int)
140 | me.PaysTo = make(map[string]int)
141 |
142 | if me.SpendsMultisig {
143 | me.Multisig = make(map[string]int)
144 | }
145 |
146 | for _, in := range tx.Inputs {
147 | inputType := in.GetType()
148 | me.Spends[inputType.String()]++
149 | switch inputType {
150 | case rawtx.InP2SH:
151 | isMultisig, m, n := in.GetP2SHRedeemScript().IsMultisigScript()
152 | if isMultisig {
153 | me.Multisig[strconv.Itoa(m)+"-of-"+strconv.Itoa(n)]++
154 | }
155 | case rawtx.InP2SH_P2WSH:
156 | isMultisig, m, n := in.GetNestedP2WSHRedeemScript().IsMultisigScript()
157 | if isMultisig {
158 | me.Multisig[strconv.Itoa(m)+"-of-"+strconv.Itoa(n)]++
159 | }
160 | case rawtx.InP2WSH:
161 | isMultisig, m, n := in.GetP2WSHRedeemScript().IsMultisigScript()
162 | if isMultisig {
163 | me.Multisig[strconv.Itoa(m)+"-of-"+strconv.Itoa(n)]++
164 | }
165 | case rawtx.InUNKNOWN:
166 | logger.Warning.Printf("Tx with the id %s categorized as Spends UNKNOWN.", me.TxID)
167 | }
168 | }
169 |
170 | for _, out := range tx.Outputs {
171 | outputType := out.GetType()
172 | me.PaysTo[outputType.String()]++
173 |
174 | switch outputType {
175 | case rawtx.OutOPRETURN:
176 | var isOPRETURN, data = out.GetOPReturnData()
177 | if isOPRETURN {
178 | me.OPReturnData = string(data.PushedData)
179 | me.OPReturnLength = len(data.PushedData)
180 | }
181 | case rawtx.OutUNKNOWN:
182 | logger.Warning.Printf("Tx with the id %s categorized as PaysTo UNKNOWN.", me.TxID)
183 | }
184 | }
185 |
186 | err = database.WriteMempoolEntries(me)
187 | if err != nil {
188 | logger.Error.Printf("Error handling raw tx: %v", err)
189 | }
190 | }
191 |
192 | /* ----- HASH TX handling ----- */
193 |
194 | // HandleHashTx handles incoming zmq tx hashes
195 | func HandleHashTx(payload string) {
196 | logger.Warning.Println("HandleHashTx() not Implemented")
197 | }
198 |
--------------------------------------------------------------------------------
/memod/types/database.go:
--------------------------------------------------------------------------------
1 | package types
2 |
3 | type RecentBlock struct {
4 | Height int `json:"height"`
5 | Size int `json:"size"`
6 | Timestamp int64 `json:"timestamp"`
7 | TxCount int `json:"txCount"`
8 | Weight int `json:"weight"`
9 | }
10 |
11 | type HistoricalMempoolData struct {
12 | DataInBuckets interface{} `json:"dataInBuckets"`
13 | Timestamp int64 `json:"timestamp"`
14 | }
15 |
16 | type TransactionStat struct {
17 | SegwitCount int `json:"segwitCount"`
18 | RbfCount int `json:"rbfCount"`
19 | TxCount int `json:"txCount"`
20 | Timestamp int64 `json:"timestamp"`
21 | }
22 |
23 | type MempoolEntry struct {
24 | EntryTime int64 `json:"entryTime"`
25 | TxID string `json:"txid"`
26 | Fee int64 `json:"fee"`
27 | Size int64 `json:"size"`
28 | Version int32 `json:"version"`
29 | InputCount int `json:"inputCount"`
30 | OutputCount int `json:"outputCount"`
31 | Locktime uint32 `json:"locktime"`
32 | OutputSum int64 `json:"outputValue"`
33 | SpendsSegWit bool `json:"spendsSegWit"`
34 | SpendsMultisig bool `json:"spendsMultisig"`
35 | IsBIP69 bool `json:"isBIP69"`
36 | SignalsRBF bool `json:"signalsRBF"`
37 | OPReturnData string `json:"opreturnData"`
38 | OPReturnLength int `json:"opreturnLength"`
39 | Multisig map[string]int `json:"multisigsSpend"`
40 | Spends map[string]int `json:"spends"`
41 | PaysTo map[string]int `json:"paysTo"`
42 | }
43 |
44 | // BlockEntry holds the height, the first-seen timestamp and
45 | // shortended TXIDs. It's used in the Bitcoin Transaction Monitor
46 | // to mark transactions by block they were included in.
47 | type BlockEntry struct {
48 | Height int `json:"height"`
49 | Timestamp int64 `json:"timestamp"`
50 | ShortTXIDs []string `json:"shortTXIDs"`
51 | }
52 |
--------------------------------------------------------------------------------
/memod/types/feerates.go:
--------------------------------------------------------------------------------
1 | package types
2 |
3 | // The external fee APIs return different estimate counts for which different structs are used
4 | // FeeAPIResponse1 for one estimate, FeeAPIResponse2 for two and FeeAPIResponse3 for three estimates.
5 |
6 | type FeeAPIResponse1 struct {
7 | HighFee float64 `json:"high"`
8 | }
9 |
10 | type FeeAPIResponse2 struct {
11 | HighFee float64 `json:"high"`
12 | MedFee float64 `json:"med"`
13 | }
14 |
15 | type FeeAPIResponse3 struct {
16 | HighFee float64 `json:"high"`
17 | MedFee float64 `json:"med"`
18 | LowFee float64 `json:"low"`
19 | }
20 |
21 | type FeeRateAPIEntry struct {
22 | Timestamp int64 `json:"timestamp"`
23 | BTCCom FeeAPIResponse1 `json:"btccom"`
24 | BlockchairCom FeeAPIResponse1 `json:"blockchaircom"`
25 | BlockchainInfo FeeAPIResponse2 `json:"blockchaininfo"`
26 | EarnCom FeeAPIResponse3 `json:"earncom"`
27 | BitgoCom FeeAPIResponse3 `json:"bitgocom"`
28 | BlockcypherCom FeeAPIResponse3 `json:"blockcyphercom"`
29 | BitpayCom FeeAPIResponse3 `json:"bitpaycom"`
30 | WasabiWalletIoEcon FeeAPIResponse3 `json:"wasabiwalletioEcon"`
31 | WasabiWalletIoCons FeeAPIResponse3 `json:"wasabiwalletioCons"`
32 | TrezorIo FeeAPIResponse3 `json:"trezorio"`
33 | LedgerCom FeeAPIResponse3 `json:"ledgercom"`
34 | MyceliumIo FeeAPIResponse3 `json:"myceliumio"`
35 | BitcoinerLive FeeAPIResponse3 `json:"bitcoinerlive"`
36 | BlockstreamInfo FeeAPIResponse3 `json:"blockstreaminfo"`
37 | }
38 |
--------------------------------------------------------------------------------
/memod/types/mempool.go:
--------------------------------------------------------------------------------
1 | package types
2 |
3 | /*
4 | type Fees struct {
5 | Base float64 `json:"base"`
6 | Modified float64 `json:"modified"`
7 | Ancestor float64 `json:"ancestor"`
8 | Descendant float64 `json:"descendant"`
9 | }
10 |
11 | type Transaction struct {
12 | Fees Fees `json:"fees"`
13 | Size int `json:"size"`
14 | Fee float64 `json:"fee"`
15 | Modifiedfee float64 `json:"modifiedfee"`
16 | Time int `json:"time"`
17 | Height int `json:"height"`
18 | Descendantcount int `json:"descendantcount"`
19 | Descendantsize int `json:"descendantsize"`
20 | Descendantfees int `json:"descendantfees"`
21 | Ancestorcount int `json:"ancestorcount"`
22 | Ancestorsize int `json:"ancestorsize"`
23 | Ancestorfees int `json:"ancestorfees"`
24 | Wtxid string `json:"wtxid"`
25 | Depends []interface{} `json:"depends"`
26 | Spentby []interface{} `json:"spentby"`
27 | Bip125Replaceable bool `json:"bip125-replaceable"`
28 | }
29 | */
30 |
31 | // PartialTransaction is a part-struct of `Transaction` which contains
32 | // only the for-now-used values to be more memory efficient
33 | type PartialTransaction struct {
34 | Size int `json:"vsize"`
35 | Fee float64 `json:"fee"`
36 | Time int `json:"time"`
37 | Wtxid string `json:"wtxid"`
38 | Bip125Replaceable bool `json:"bip125-replaceable"`
39 | }
40 |
--------------------------------------------------------------------------------
/memod/zmq/zmq.go:
--------------------------------------------------------------------------------
1 | package zmq
2 |
3 | import (
4 | "github.com/0xb10c/memo/memod/config"
5 | "github.com/0xb10c/memo/memod/logger"
6 | "github.com/0xb10c/memo/memod/processor"
7 |
8 | "github.com/pebbe/zmq4"
9 | )
10 |
11 | const rawBlock string = "rawblock"
12 | const hashBlock string = "hashblock"
13 | const rawTx string = "rawtx"
14 | const rawTx2 string = "rawtx2"
15 | const hashTx string = "hashtx"
16 |
17 | func SetupZMQ() {
18 |
19 | zmqHost := config.GetString("zmq.host")
20 | zmqPort := config.GetString("zmq.port")
21 | connectionString := "tcp://" + zmqHost + ":" + zmqPort
22 |
23 | subscriber, _ := zmq4.NewSocket(zmq4.SUB)
24 | subscriber.Connect(connectionString)
25 |
26 | if config.GetBool("zmq.subscribeTo.rawTx") {
27 | subscriber.SetSubscribe(rawTx)
28 | }
29 | if config.GetBool("zmq.subscribeTo.rawTx2") {
30 | subscriber.SetSubscribe(rawTx2)
31 | }
32 | if config.GetBool("zmq.subscribeTo.hashTx") {
33 | subscriber.SetSubscribe(hashTx)
34 | }
35 | if config.GetBool("zmq.subscribeTo.rawBlock") {
36 | subscriber.SetSubscribe(rawBlock)
37 | }
38 | if config.GetBool("zmq.subscribeTo.hashBlock") {
39 | subscriber.SetSubscribe(hashBlock)
40 | }
41 |
42 | defer subscriber.Close() // cancel subscribe
43 |
44 | loopZMQ(subscriber)
45 | }
46 |
47 | func loopZMQ(subscriber *zmq4.Socket) {
48 | for {
49 | msg, err := subscriber.RecvMessage(0)
50 | if err != nil {
51 | logger.Error.Println(err)
52 | }
53 | handleZMQMessage(msg)
54 | }
55 | }
56 |
57 | func handleZMQMessage(zmqMessage []string) {
58 | topic := zmqMessage[0]
59 | payload := zmqMessage[1]
60 |
61 | switch topic {
62 | case rawBlock:
63 | go processor.HandleRawBlock(payload)
64 | case hashBlock:
65 | go processor.HandleHashBlock(payload)
66 | case rawTx:
67 | go processor.HandleRawTx(payload)
68 | case rawTx2:
69 | go processor.HandleRawTxWithSizeAndFee(payload)
70 | case hashTx:
71 | go processor.HandleHashTx(payload)
72 | default:
73 | logger.Warning.Println("Unhandled ZMQ topic", topic)
74 | }
75 | }
76 |
--------------------------------------------------------------------------------
/www/404.html:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
10 |
12 |
13 |
14 |
16 |
17 |
18 |
19 |
20 |
21 |
22 |
23 | 
40 | 109 | Bitcoin Transaction Monitor 110 |
111 |112 | Whenever you, an exchange or somebody else sends a Bitcoin transaction, it gets broadcasted to all nodes in the Bitcoin network. 113 | Each broadcast transaction is represented by a dot on the scatterplot below. 114 | The transactions are arranged on the x-axis by the time of arrival at my Bitcoin node. 115 | The y-axis represents the feerate (fee per size) the transaction pays. 116 |
117 |118 | The plot reveals activity patterns of wallets, exchanges and users transacting on the Bitcoin network. 119 | Some patterns are only visible on certain days or at certain times. 120 | To reduce the noise you can apply filters, set the dot radius and highlight transactions based on their properties. 121 | Additionally feerate estimates from various sources can be overlayed. 122 | Hovering over a transaction reveals more information about that transaction, and clicking opens a new tab with the transaction in a blockchain explorer. 123 |
124 | 125 |126 | I've written a FAQ addressing some additional questions you might have. 127 |
128 |145 | 0 transactions loaded from the last 0 minutes
146 | 0 drawn (0%) with out-of-bounds
147 |
Radius
156 | 168 |The radius is calculated based on transaction vsize.
169 |Highlight
174 | 186 |No transactions are highlighted.
187 |Feerate Estimator
192 | 208 |No feerate estimator overlayed.
209 |Filter
214 |215 | You can filter out transactions based on their properties. 216 | By default, all filters are inactive, which results in all transaction being shown. 217 |
218 |