├── .github └── PULL_REQUEST_TEMPLATE.md ├── CONTRIBUTING.md ├── LICENSE └── README.md /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | Please check if what you want to add to `awesome-go-storage` list meets [quality standards](https://github.com/gostor/awesome-go-storage/blob/master/CONTRIBUTING.md#quality-standard) before sending pull request. Thanks! 2 | 3 | **Please provide package links to:** 4 | - github.com repo: 5 | - godoc.org: 6 | - goreportcard.com: 7 | - coverage service link (gocover, coveralls etc.): 8 | 9 | 10 | **Note**: that new categories can be added only when there are 3 packages or more. 11 | 12 | **Make sure that you've checked the boxes below before you submit PR:** 13 | - [ ] I have added my package in alphabetical order 14 | - [ ] I know that this package was not listed before 15 | - [ ] I have added godoc link to the repo and to my pull request 16 | - [ ] I have added coverage service link to the repo and to my pull request 17 | - [ ] I have added goreportcard link to the repo and to my pull request 18 | - [ ] I have read [Contribution guidelines](https://github.com/gostor/awesome-go-storage/blob/master/CONTRIBUTING.md#contribution-guidelines) and [Quality standard](https://github.com/gostor/awesome-go-storage/blob/master/CONTRIBUTING.md#quality-standard). 19 | 20 | Thanks for your PR, you're awesome! :+1: 21 | -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | This resource was made by the Go community and wouldn't be possible without you! We appreciate and recognize [all contributors](https://github.com/avelino/awesome-go/graphs/contributors). 2 | 3 | Join us on IRC at **#awesome-go** on freenode [web access](http://webchat.freenode.net/?channels=awesome-go) 4 | 5 | 6 | # Contribution Guidelines 7 | 8 | - **To add, remove, or change things on the list:** Submit a pull request 9 | 10 | To set this list apart from and compliment the excellent [Go wiki Projects page](https://golang.org/wiki/Projects), awesome-go is a specially curated list for high-quality, actively maintained Go packages and resources. 11 | 12 | - List items should be sorted *alphabetically*. 13 | - Each item should be limited to one link 14 | - The link should be the name of the package or project 15 | - Descriptions should be clear, concise, and non-promotional 16 | - Descriptions should follow the link, on the same line 17 | - Needs library 3 (minimum) to create a new category. 18 | 19 | Please contribute links to packages/projects you have used or are familiar with. This will help ensure high-quality entries. 20 | 21 | If you removed our PR template you can find it [here](https://github.com/avelino/awesome-go/blob/master/.github/PULL_REQUEST_TEMPLATE.md). 22 | 23 | 24 | ## Quality standard 25 | 26 | To be on the list, project repositories should adhere to these quality standards (http://goreportcard.com/report/**github_user**/**github_repo**): 27 | 28 | - Code functions as documented and expected 29 | - Generally useful to the wider community of Go programmers 30 | - Actively maintained 31 | - Regular, recent commits 32 | - Or, for finished projects, issues and pull requests are responded to 33 | - Stable or progressing toward stable 34 | - Thoroughly documented (README, godoc comments, etc.) 35 | - Tests, where practical. If library/program testable, than coverage should be >= 80% for non-data-related packages and >=90% for data related packages. **Notice**: the tests will be reviewed too. We will check your coverage manually if your package's coverage is just a benchmark results. 36 | 37 | 38 | ## Reporting issues 39 | 40 | Please open an issue if you would like to discuss anything that could be improved or have suggestions for making the list a more valuable resource. We realize sometimes packages fall into abandonment or have breaking builds for extended periods of time, so if you see that, feel free to change its listing or let us know. We also realize that sometimes projects are just going through transitions or are more experimental in nature. These can still be cool, but we can indicate them as transitory or experimental. 41 | 42 | Removal changes will not be applied until they have been pending for a minimum of 1 week (7 days). This grace window benefits projects that may be going through a temporary transition but are otherwise worthy of being on the list. 43 | 44 | Thanks everyone! 45 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 Thiago Avelino 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Awesome Go Storage [![Awesome](https://cdn.jsdelivr.net/gh/sindresorhus/awesome@d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome) 2 | 3 | A curated list of awesome Go storage projects and libraries. Inspired by [awesome-go](https://github.com/avelino/awesome-go). 4 | 5 | 6 | ### Contributing 7 | 8 | Please take a quick gander at the [contribution guidelines](https://github.com/gostor/awesome-go-storage/blob/master/CONTRIBUTING.md) first. Thanks to all [contributors](https://github.com/gostor/awesome-go-storage/graphs/contributors); you rock! 9 | 10 | #### *If you see a package or project here that is no longer maintained or is not a good fit, please submit a pull request to improve this file. Thank you!* 11 | 12 | 13 | ### Contents 14 | 15 | - [Awesome Go Storage](#awesome-go-storage) 16 | - [Storage Server](#storage-server) 17 | - [Key-Value Store](#key-value-store) 18 | - [File System](#file-system) 19 | - [Database](#database) 20 | - [Database Drivers](#database-drivers) 21 | 22 | 23 | ## Storage Server 24 | 25 | *Storage Servers implemented in Go.* 26 | 27 | * [minio](https://github.com/minio/minio) - Minio is an open source object storage server compatible with Amazon S3 APIs. 28 | * [rclone](https://github.com/ncw/rclone) - "rsync for cloud storage" - Google Drive, Amazon Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Cloudfile… 29 | * [perkeep](https://github.com/perkeep/perkeep) - Perkeep is your personal storage system for life: a way of storing, syncing, sharing, modelling and backing up content. 30 | * [s3git](https://github.com/s3git/s3git) - Git for Cloud Storage. Distributed Version Control for Data. 31 | * [storj](https://github.com/storj/storj) - Decentralized cloud object storage that is affordable, easy to use, private, and secure. 32 | * [rook](https://github.com/rook/rook) - Open, Cloud Native, and Universal Distributed Storage. 33 | * [longhorn](https://github.com/rancher/longhorn) - Longhorn is an open source persistent block storage server delivered via containers. 34 | 35 | ## Key-Value Store 36 | 37 | *Key-Value Store implemented in Go.* 38 | 39 | * [Bitcask](https://git.mills.io/prologic/bitcask) - Bitcask is an embeddable, persistent and fast key-value (KV) database written in pure Go with predictable read/write performance, low latency and high throughput thanks to the bitcask on-disk layout (LSM+WAL). 40 | * [Bitraft](https://git.mills.io/prologic/bitraft) - Bitraft is a distributed raft-based key/value store with a Redis-compatible protocol. It uses [Bitcask](https://git.mills.io/prologic/bitcask) for high performance/throughout and low latency. 41 | * [BadgerDB](https://github.com/dgraph-io/badger) - BadgerDB is an embeddable, persistent, simple and fast key-value (KV) database written in pure Go. It's meant to be a performant alternative to non-Go-based key-value stores like RocksDB. 42 | * [column](https://github.com/kelindar/column) - Embeddable, columnar, in-memory store with bitmap indexing, transaction and zero-alloc quering. 43 | * [consul](https://github.com/hashicorp/consul) - Distributed consistent replicated key-value store for service discovery and configuration. 44 | * [diskv](https://github.com/peterbourgon/diskv) - A disk-backed key-value store. 45 | * [etcd](https://github.com/coreos/etcd) - Distributed reliable key-value store for the most critical data of a distributed system. 46 | * [go-cache](https://github.com/patrickmn/go-cache) - An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications. 47 | * [IceFireDB](https://github.com/IceFireDB/IceFireDB) - Distributed disk storage database using the Raft and Redis protocols. 48 | * [nutsdb](https://github.com/xujiajun/nutsdb) - Nutsdb is a simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set. 49 | * [pogreb](https://github.com/akrylysov/pogreb) - Embedded key-value store for read-heavy workloads. 50 | * [rosedb](https://github.com/roseduan/rosedb) - A fast, stable and embedded k-v database in pure Golang, supports string, list, hash, set, sorted set. 51 | * [LotusDB](https://github.com/flower-corp/lotusdb) - Fast k/v storage compatible with lsm tree and b+tree. 52 | 53 | ## File System 54 | 55 | *File Systems implemented in Go.* 56 | 57 | * [afero](https://github.com/spf13/afero) - A FileSystem Abstraction System for Go 58 | * [fsnotify](https://github.com/fsnotify/fsnotify) - Cross-platform file system notifications for Go. 59 | * [juicefs](https://github.com/juicedata/juicefs) - A distributed POSIX file system built on top of Redis and S3. 60 | * [gcsfuse](https://github.com/GoogleCloudPlatform/gcsfuse) - A user-space file system for interacting with Google Cloud Storage. 61 | * [git-lfs](https://github.com/git-lfs/git-lfs) - Git extension for versioning large files. 62 | * [go-systemd](https://github.com/coreos/go-systemd) - Go bindings to systemd socket activation, journal, D-Bus, and unit files. 63 | * [goofys](https://github.com/kahing/goofys) - A high-performance, POSIX-ish Amazon S3 file system written in Go. 64 | * [minikeyvalue](https://github.com/geohot/minikeyvalue) - A ~1000 line distributed key value store. 65 | * [seaweedfs](https://github.com/chrislusf/seaweedfs) - SeaweedFS is a simple and highly scalable distributed file system for small files. 66 | * [svfs](https://github.com/ovh/svfs) - A virtual file system over Openstack Swift built upon fuse. 67 | 68 | 69 | ## Database 70 | 71 | *Databases implemented in Go.* 72 | 73 | * [BigCache](https://github.com/allegro/bigcache) - Efficient key/value cache for gigabytes of data. 74 | * [bolt](https://github.com/boltdb/bolt) - A low-level key/value database for Go. This original version by Ben Johnson has been marked as unmaintained and forked by [etcd-io bbolt](https://github.com/etcd-io/bbolt). 75 | * [buntdb](https://github.com/tidwall/buntdb) - A fast, embeddable, in-memory key/value database for Go with custom indexing and spatial support. 76 | * [cache2go](https://github.com/muesli/cache2go) - An in-memory key:value cache which supports automatic invalidation based on timeouts. 77 | * [clover](https://github.com/ostafen/clover) - A lightweight document-oriented NoSQL database written in pure Golang. 78 | * [cockroach](https://github.com/cockroachdb/cockroach) - A Scalable, Geo-Replicated, Transactional Datastore 79 | * [couchcache](https://github.com/codingsince1985/couchcache) - A RESTful caching micro-service backed by Couchbase server. 80 | * [CovenantSQL](https://github.com/CovenantSQL/CovenantSQL) - A SQL Database with Blockchain features. 81 | * [dgraph](https://github.com/dgraph-io/dgraph) - Scalable, Distributed, Low Latency, High Throughput Graph Database. 82 | * [diskv](https://github.com/peterbourgon/diskv) - A home-grown disk-backed key-value store. 83 | * [dolt](https://github.com/dolthub/dolt) - A MySQL-compatible database with Git-style version control. The first SQL database you can branch and merge. 84 | * [eliasdb](https://github.com/krotik/eliasdb) - Dependency-free, transactional graph database with REST API, phrase search and SQL-like query language. 85 | * [emitter](https://github.com/emitter-io/emitter) - Scalable, low-latency, distributed & secure pub/sub database with time-series message storage, suitable for IoT, gaming, apps and real-time web. 86 | * [forestdb](https://github.com/couchbase/goforestdb) - Go bindings for ForestDB. 87 | * [frostdb](https://github.com/polarsignals/frostdb/) - Embeddable column database written in Go. 88 | * [GCache](https://github.com/bluele/gcache) - Cache library with support for expirable Cache, LFU, LRU and ARC. 89 | * [geocache](https://github.com/melihmucuk/geocache) - An in-memory cache that is suitable for geolocation based applications. 90 | * [go-cache](https://github.com/pmylund/go-cache) - An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications. 91 | * [godis](https://github.com/hdt3213/godis) - A Golang implemented high-performance Redis server and cluster 92 | * [goleveldb](https://github.com/syndtr/goleveldb) - An implementation of the [LevelDB](https://github.com/google/leveldb) key/value database in the Go. 93 | * [groupcache](https://github.com/golang/groupcache) - Groupcache is a caching and cache-filling library, intended as a replacement for memcached in many cases. 94 | * [immudb](https://github.com/codenotary/immudb) - Database with built-in cryptographic proof and verification. Can operate as a key-value store or as relational database (SQL). 95 | * [influxdb](https://github.com/influxdb/influxdb) - Scalable datastore for metrics, events, and real-time analytics 96 | * [ledisdb](https://github.com/siddontang/ledisdb) - Ledisdb is a high performance NoSQL like Redis based on LevelDB. 97 | * [levigo](https://github.com/jmhodges/levigo) - Levigo is a Go wrapper for LevelDB. 98 | * [moss](https://github.com/couchbase/moss) - Moss is a simple LSM key-value storage engine written in 100% Go. 99 | * [noms](https://github.com/attic-labs/noms) - The versioned, forkable, syncable database. 100 | * [objectbox-go](https://github.com/objectbox/objectbox-go) - Embedded Object Database (NoSQL) with Go API. 101 | * [piladb](https://github.com/fern4lvarez/piladb) - Lightweight RESTful database engine based on stack data structures. 102 | * [pREST](https://github.com/nuveo/prest) - Serve a RESTful API from any PostgreSQL database. 103 | * [prometheus](https://github.com/prometheus/prometheus) - Monitoring system and time series database. 104 | * [rqlite](https://github.com/rqlite/rqlite) - The lightweight, distributed, relational database built on SQLite. 105 | * [scribble](https://github.com/nanobox-io/golang-scribble) - A tiny flat file JSON store. 106 | * [SpiceDB](https://github.com/authzed/spicedb) - A [Zanzibar](https://research.google/pubs/pub48190/)-inspired database that stores, computes, and validates application permissions with support for multiple database backends. 107 | * [tidb](https://github.com/pingcap/tidb) - TiDB is a distributed SQL database. Inspired by the design of Google F1. 108 | * [tiedot](https://github.com/HouzuoGuo/tiedot) - Your NoSQL database powered by Golang. 109 | * [Tile38](https://github.com/tidwall/tile38) - A geolocation DB with spatial index and realtime geofencing. 110 | * [VictoriaMetrics](https://github.com/VictoriaMetrics/VictoriaMetrics) - Time series database designed for monitoring, compatible with PromQL (from Promethueus) and InfluxQL (from Influxdb). 111 | 112 | *Database schema migration.* 113 | 114 | * [darwin](https://github.com/GuiaBolso/darwin) - Database schema evolution library for Go 115 | * [goose](https://github.com/steinbacher/goose) - Database migration tool. You can manage your database's evolution by creating incremental SQL or Go scripts. 116 | * [gormigrate](https://github.com/go-gormigrate/gormigrate) - Database schema migration helper for Gorm ORM. 117 | * [migrate](https://github.com/golang-migrate/migrate) - Database migration handling in Golang support MySQL, PostgreSQL, Cassandra, and SQLite. 118 | * [pravasan](https://github.com/pravasan/pravasan) - Simple Migration tool - currently for MySQL but planning to support soon for Postgres, SQLite, MongoDB, etc., 119 | * [soda](https://github.com/markbates/pop/tree/master/soda) - Database migration, creation, ORM, etc... for MySQL, PostgreSQL, and SQLite. 120 | * [sql-migrate](https://github.com/rubenv/sql-migrate) - Database migration tool. Allows embedding migrations into the application using go-bindata. 121 | 122 | *Database tools.* 123 | 124 | * [go-mysql](https://github.com/siddontang/go-mysql) - A go toolset to handle MySQL protocol and replication. 125 | * [go-mysql-elasticsearch](https://github.com/siddontang/go-mysql-elasticsearch) - Sync your MySQL data into Elasticsearch automatically. 126 | * [kingshard](https://github.com/flike/kingshard) - kingshard is a high performance proxy for MySQL powered by Golang. 127 | * [myreplication](https://github.com/2tvenom/myreplication) - MySql binary log replication listener. Support statement and row based replication. 128 | * [orchestrator](https://github.com/openark/orchestrator) - MySQL replication topology manager & visualizer 129 | * [pgweb](https://github.com/sosedoff/pgweb) - A web-based PostgreSQL database browser 130 | * [vitess](https://github.com/youtube/vitess) - vitess provides servers and tools which facilitate scaling of MySQL databases for large scale web services. 131 | * [usql](https://github.com/xo/usql) - universal command-line interface for SQL databases 132 | 133 | *SQL query builder, libraries for building and using SQL.* 134 | 135 | * [buildsqlx](https://github.com/arthurkushman/buildsqlx) - Go Database query builder library 136 | * [dat](https://github.com/mgutz/dat) - Go Postgres Data Access Toolkit 137 | * [Dotsql](https://github.com/gchaincl/dotsql) - Go library that helps you keep sql files in one place and use it with ease. 138 | * [goqu](https://github.com/doug-martin/goqu) - An idiomatic SQL builder and query library. 139 | * [grimoire](https://github.com/Fs02/grimoire) - Grimoire is a database access layer and validation for golang. (Support: MySQL, PostgreSQL and SQLite3). 140 | * [igor](https://github.com/galeone/igor) - Abstraction layer for PostgreSQL that supports advanced functionality and uses gorm-like syntax. 141 | * [ozzo-dbx](https://github.com/go-ozzo/ozzo-dbx) - Powerful data retrieval methods as well as DB-agnostic query building capabilities. 142 | * [SQLBoiler](https://github.com/volatiletech/sqlboiler) - a tool to generate a Go ORM tailored to your database schema. It is a "database-first" ORM as opposed to "code-first", and you must first create your database. 143 | * [sqrl](https://github.com/elgris/sqrl) - SQL query builder, fork of Squirrel with improved performance. 144 | * [Squirrel](https://github.com/Masterminds/squirrel) - Go library that helps you build SQL queries. 145 | * [xo](https://github.com/knq/xo) - Generate idiomatic Go code for databases based on existing schema definitions or custom queries supporting PostgreSQL, MySQL, SQLite, Oracle, and Microsoft SQL Server. 146 | 147 | 148 | ## Database Drivers 149 | 150 | *Libraries for connecting and operating databases.* 151 | 152 | * Relational Databases 153 | * [bgc](https://github.com/viant/bgc) - Datastore Connectivity for BigQuery for go. 154 | * [firebirdsql](https://github.com/nakagami/firebirdsql) - Firebird RDBMS SQL driver for Go 155 | * [go-adodb](https://github.com/mattn/go-adodb) - Microsoft ActiveX Object DataBase driver for go that using database/sql. 156 | * [go-bqstreamer](https://github.com/rounds/go-bqstreamer) - BigQuery fast and concurrent stream insert. 157 | * [go-mssqldb](https://github.com/denisenkom/go-mssqldb) - Microsoft MSSQL driver in go language. 158 | * [go-oci8](https://github.com/mattn/go-oci8) - Oracle driver for go that using database/sql. 159 | * [go-sql-driver/mysql](https://github.com/go-sql-driver/mysql) - MySQL driver for Go. 160 | * [go-sqlite3](https://github.com/mattn/go-sqlite3) - SQLite3 driver for go that using database/sql. 161 | * [gofreetds](https://github.com/minus5/gofreetds) Microsoft MSSQL driver. Go wrapper over [FreeTDS](http://www.freetds.org). 162 | * [pgx](https://github.com/jackc/pgx) - PostgreSQL driver supporting features beyond those exposed by database/sql. 163 | * [pq](https://github.com/lib/pq) - Pure Go Postgres driver for database/sql. 164 | 165 | * NoSQL Databases 166 | * [aerospike-client-go](https://github.com/aerospike/aerospike-client-go) - Aerospike client in Go language. 167 | * [arangolite](https://github.com/solher/arangolite) - Lightweight golang driver for ArangoDB. 168 | * [asc](https://github.com/viant/asc) - Datastore Connectivity for Aerospike for go. 169 | * [cayley](https://github.com/google/cayley) - A graph database with support for multiple backends. 170 | * [dsc](https://github.com/viant/dsc) - Datastore connectivity for SQL, NoSQL, structured files. 171 | * [dynago](https://github.com/underarmour/dynago) - Dynago is a principle of least surprise client for DynamoDB 172 | * [go-couchbase](https://github.com/couchbase/go-couchbase) - Couchbase client in Go 173 | * [go-couchdb](https://github.com/fjl/go-couchdb) - Yet another CouchDB HTTP API wrapper for Go 174 | * [gocb](https://github.com/couchbase/gocb) - Official Couchbase Go SDK 175 | * [gocql](http://gocql.github.io) - A Go language driver for Apache Cassandra. 176 | * [gomemcache](https://github.com/bradfitz/gomemcache/) - memcache client library for the Go programming language. 177 | * [gorethink](https://github.com/GoRethink/gorethink) - Go language driver for RethinkDB 178 | * [goriak](https://github.com/zegl/goriak) - Go language driver for Riak KV 179 | * [mongo-go-driver](https://github.com/mongodb/mongo-go-driver) - The Go driver for MongoDB 180 | * [neo4j](https://github.com/cihangir/neo4j) - Neo4j Rest API Bindings for Golang 181 | * [Neo4j-GO](https://github.com/davemeehan/Neo4j-GO) - Neo4j REST Client in golang. 182 | * [neoism](https://github.com/jmcvetta/neoism) - Neo4j client for Golang 183 | * [redigo](https://github.com/gomodule/redigo) - Redigo is a Go client for the Redis database. 184 | * [redis](https://github.com/go-redis/redis) - Redis client for Golang 185 | * [redis](https://github.com/hoisie/redis) - A simple, powerful Redis client for Go. 186 | * [redeo](https://github.com/bsm/redeo) - Redis-protocol compatible TCP servers/services. 187 | 188 | * Search and Analytic Databases 189 | * [bleve](https://github.com/blevesearch/bleve) - A modern text indexing library for go. 190 | * [elastic](https://github.com/olivere/elastic) - Elasticsearch client for Go. 191 | * [elastigo](https://github.com/mattbaird/elastigo) - A Elasticsearch client library. 192 | * [skizze](https://github.com/seiflotfy/skizze) - A probabilistic data-structures service and storage. 193 | 194 | * Multiple Backends 195 | * [Fiber Storage](https://github.com/gofiber/storage) - Premade storage drivers that implement the [Storage](https://github.com/gofiber/storage/blob/main/storage.go) interface. 196 | * [gokv](https://github.com/philippgille/gokv) - A simple key-value store interface and many implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more) 197 | --------------------------------------------------------------------------------