├── LICENSE ├── README.md ├── crypt.go ├── example └── main.go └── http_test.go /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 David Pennington 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Encryption and Decryption 2 | 3 | This package is a simple AES-CTR encryption wrapper with SHA512 HMAC authentication. I wrote it to handle large blobs of data that would not fit into memory (or would take to much memory). Examples include files and client-to-client uploads. The assumption is that this will be used with public/private key cryptography where the AES password (and HMAC password) will be strong and random providing a strong security guarantee. 4 | 5 | I also wanted this to be [easy to](https://gist.github.com/AndiDittrich/4629e7db04819244e843) implement [in Javascript](https://stackoverflow.com/questions/36909746/aes-ctr-encrypt-in-cryptojs-and-decrypt-in-go-lang) for client-to-client communication via electron or react-native. 6 | 7 | ## Benchmarks 8 | 9 | Included the example folder is a benchmark of encrypting an decrypting a 500MB stream of data. I get over 100MB/sec on my local computer using two cores. 10 | 11 | go get github.com/Xeoncross/go-aesctr-with-hmac 12 | cd $GOPATH/src/github.com/Xeoncross/go-aesctr-with-hmac/example 13 | go run main.go 14 | 15 | ## Using passwords 16 | 17 | If using passwords to encrypt things I recommend you use this the "decrypto" AES-CTR + HMAC + scrypt password strengthening implementation found in [odeke-em/drive](https://github.com/sselph/drive/tree/master/src/dcrypto). It might be slower (and uses a temp file) but is worth it for the security gains. Human-passwords aren't safe to use alone. 18 | 19 | ## Encrypting small blobs 20 | 21 | If the data you are encrypting is small and easily fits into memory then you should use GCM. GCM is [nice and simple to use](https://github.com/gtank/cryptopasta/blob/master/encrypt.go) if your data is small. 22 | 23 | ## Encrypting a Media stream 24 | 25 | If you need to encrypt video/audio stream, then a more complex chunked version of GCM is for you. https://github.com/minio/sio (D.A.R.E. v2) provides a way to break data up into chunks that can be decrypted as they arrive and used without waiting for the rest of the stream to finish arriving. 26 | 27 | # Warning 28 | 29 | I am not a cryptographer. However, this implementation has very few moving parts all of which are written by real cryptographers and used as described. 30 | 31 | 32 | ## Reference 33 | 34 | - [Cryptography lessons-learned](https://security.stackexchange.com/questions/2202/lessons-learned-and-misconceptions-regarding-encryption-and-cryptology) 35 | - [Symmetric Security (NaCI, GCM, CTR)](https://leanpub.com/gocrypto/read#leanpub-auto-chapter-3-symmetric-security) 36 | - https://www.imperialviolet.org/2014/06/27/streamingencryption.html 37 | - [streaming encryption with AES-OFB](https://golang.org/src/crypto/cipher/example_test.go#L335) 38 | - [Encrypt then MAC](http://www.daemonology.net/blog/2009-06-24-encrypt-then-mac.html) 39 | - [AES OFB vs CRT](https://security.stackexchange.com/questions/27776/block-chaining-modes-to-avoid/27780#27780) 40 | - [How do you encrypt large streams?](https://stackoverflow.com/questions/49546567/how-do-you-encrypt-large-files-byte-streams-in-go/49546791?noredirect=1#comment86134522_49546791) 41 | - [Authenticated Encryption "AHEAD"](https://en.wikipedia.org/wiki/Authenticated_encryption) 42 | - [PBKDF2 is redundant for bits from a CSPRNG (for use in AES)](https://crypto.stackexchange.com/questions/14842/is-it-overkill-to-run-a-key-generated-by-openssl-through-pbkdf2) 43 | - [AES-256-GCM basic implementation](https://gist.github.com/cannium/c167a19030f2a3c6adbb5a5174bea3ff) 44 | - [Golang AES-CFB encrypted TCP stream ](https://gist.github.com/raincious/96bb69414859e7ea0abfdb177ee97a1f) 45 | - https://github.com/SermoDigital/boxer/blob/master/boxer.go 46 | - [AES-256-GCM in C using OpenSSL for iPhone](https://gist.github.com/eliburke/24f06a1590d572e86a01504e1b38b27f) 47 | - [AES-CTR + HMAC + RFC2898 key derivation (Go)](https://github.com/xeodou/aesf) 48 | -------------------------------------------------------------------------------- /crypt.go: -------------------------------------------------------------------------------- 1 | package aesctr 2 | 3 | import ( 4 | "bufio" 5 | "crypto/aes" 6 | "crypto/cipher" 7 | "crypto/hmac" 8 | "crypto/rand" 9 | "crypto/sha512" 10 | "encoding/binary" 11 | "errors" 12 | "io" 13 | ) 14 | 15 | const BUFFER_SIZE int = 16 * 1024 16 | const IV_SIZE int = 16 17 | const V1 byte = 0x1 18 | const hmacSize = sha512.Size 19 | 20 | // ErrInvalidHMAC for authentication failure 21 | var ErrInvalidHMAC = errors.New("Invalid HMAC") 22 | 23 | // Encrypt the stream using the given AES-CTR and SHA512-HMAC key 24 | func Encrypt(in io.Reader, out io.Writer, keyAes, keyHmac []byte) (err error) { 25 | 26 | iv := make([]byte, IV_SIZE) 27 | _, err = rand.Read(iv) 28 | if err != nil { 29 | return err 30 | } 31 | 32 | AES, err := aes.NewCipher(keyAes) 33 | if err != nil { 34 | return err 35 | } 36 | 37 | ctr := cipher.NewCTR(AES, iv) 38 | HMAC := hmac.New(sha512.New, keyHmac) // https://golang.org/pkg/crypto/hmac/#New 39 | 40 | // Version 41 | _, err = out.Write([]byte{V1}) 42 | if err != nil { 43 | return 44 | } 45 | 46 | w := io.MultiWriter(out, HMAC) 47 | 48 | _, err = w.Write(iv) 49 | if err != nil { 50 | return 51 | } 52 | 53 | buf := make([]byte, BUFFER_SIZE) 54 | for { 55 | n, err := in.Read(buf) 56 | if err != nil && err != io.EOF { 57 | return err 58 | } 59 | 60 | if n != 0 { 61 | outBuf := make([]byte, n) 62 | ctr.XORKeyStream(outBuf, buf[:n]) 63 | _, err = w.Write(outBuf) 64 | if err != nil { 65 | return err 66 | } 67 | } 68 | 69 | if err == io.EOF { 70 | break 71 | } 72 | } 73 | 74 | _, err = out.Write(HMAC.Sum(nil)) 75 | 76 | return err 77 | } 78 | 79 | // Decrypt the stream and verify HMAC using the given AES-CTR and SHA512-HMAC key 80 | // Do not trust the out io.Writer contents until the function returns the result 81 | // of validating the ending HMAC hash. 82 | func Decrypt(in io.Reader, out io.Writer, keyAes, keyHmac []byte) (err error) { 83 | 84 | // Read version (up to 0-255) 85 | var version int8 86 | err = binary.Read(in, binary.LittleEndian, &version) 87 | if err != nil { 88 | return 89 | } 90 | 91 | iv := make([]byte, IV_SIZE) 92 | _, err = io.ReadFull(in, iv) 93 | if err != nil { 94 | return 95 | } 96 | 97 | AES, err := aes.NewCipher(keyAes) 98 | if err != nil { 99 | return 100 | } 101 | 102 | ctr := cipher.NewCTR(AES, iv) 103 | h := hmac.New(sha512.New, keyHmac) 104 | h.Write(iv) 105 | mac := make([]byte, hmacSize) 106 | 107 | w := out 108 | 109 | buf := bufio.NewReaderSize(in, BUFFER_SIZE) 110 | var limit int 111 | var b []byte 112 | for { 113 | b, err = buf.Peek(BUFFER_SIZE) 114 | if err != nil && err != io.EOF { 115 | return 116 | } 117 | 118 | limit = len(b) - hmacSize 119 | 120 | // We reached the end 121 | if err == io.EOF { 122 | 123 | left := buf.Buffered() 124 | if left < hmacSize { 125 | return errors.New("not enough left") 126 | } 127 | 128 | copy(mac, b[left-hmacSize:left]) 129 | 130 | if left == hmacSize { 131 | break 132 | } 133 | } 134 | 135 | h.Write(b[:limit]) 136 | 137 | // We always leave at least hmacSize bytes left in the buffer 138 | // That way, our next Peek() might be EOF, but we will still have enough 139 | outBuf := make([]byte, int64(limit)) 140 | _, err = buf.Read(b[:limit]) 141 | if err != nil { 142 | return 143 | } 144 | ctr.XORKeyStream(outBuf, b[:limit]) 145 | _, err = w.Write(outBuf) 146 | if err != nil { 147 | return 148 | } 149 | 150 | if err == io.EOF { 151 | break 152 | } 153 | } 154 | 155 | if !hmac.Equal(mac, h.Sum(nil)) { 156 | return ErrInvalidHMAC 157 | } 158 | 159 | return nil 160 | } 161 | -------------------------------------------------------------------------------- /example/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "encoding/hex" 5 | "fmt" 6 | "io" 7 | "io/ioutil" 8 | "log" 9 | "strings" 10 | "sync/atomic" 11 | "time" 12 | 13 | aesctr "github.com/Xeoncross/go-aesctr-with-hmac" 14 | ) 15 | 16 | // 17 | // For the demo 18 | // 19 | 20 | type devZero byte 21 | 22 | func (z devZero) Read(b []byte) (int, error) { 23 | for i := range b { 24 | b[i] = byte(z) 25 | } 26 | return len(b), nil 27 | } 28 | 29 | func mockDataSrc(size int64) io.Reader { 30 | fmt.Printf("dev/zero of size %d (%d MB)\n", size, size/1024/1024) 31 | var z devZero 32 | return io.LimitReader(z, size) 33 | } 34 | 35 | // WriteCounter counts the number of bytes written to it. 36 | type WriteCounter struct { 37 | total int64 // Total # of bytes transferred 38 | recent int64 // Used for per-second/minute/etc.. reports 39 | } 40 | 41 | func (wc *WriteCounter) Write(p []byte) (int, error) { 42 | n := len(p) 43 | atomic.AddInt64(&wc.total, int64(n)) 44 | atomic.AddInt64(&wc.recent, int64(n)) 45 | return n, nil 46 | } 47 | 48 | func (wc *WriteCounter) Total() int64 { 49 | return atomic.LoadInt64(&wc.total) 50 | } 51 | 52 | func (wc *WriteCounter) Recent() (n int64) { 53 | n = atomic.LoadInt64(&wc.recent) 54 | atomic.StoreInt64(&wc.recent, int64(0)) 55 | return n 56 | } 57 | 58 | func main() { 59 | 60 | var err error 61 | 62 | // Could be a file, TCP connection, etc... 63 | in := mockDataSrc(2 << 28) // ~536MB 64 | // in := strings.NewReader(strings.Repeat("Hello World! 012345\n", 4)) 65 | 66 | out := ioutil.Discard // throw it away 67 | // out := os.Stdout // dump it to the console 68 | // out := new(bytes.Buffer) 69 | 70 | // Process flow 71 | // in -> encrypt -> (pw -> pr) -> (decrypt) -> out 72 | 73 | // To pipe encryption to decryption 74 | pr, pw := io.Pipe() 75 | 76 | // This is our logger innstance to see the whole byte stream (demo below) 77 | // var b bytes.Buffer 78 | // writer := bufio.NewWriter(&b) 79 | // r := io.TeeReader(pr, writer) 80 | 81 | // In real life we would generate a random key 82 | // aesKey := make([]byte, 32) 83 | // _, err = rand.Read(aesKey) 84 | // if err != nil { 85 | // return err 86 | // } 87 | 88 | // But this isn't real life now is it? 89 | keyAes, _ := hex.DecodeString(strings.Repeat("6368616e676520746869732070617373", 2)) 90 | keyHmac := keyAes // don't do this either, use a different key or kids may die 91 | 92 | // writing without a reader will deadlock so write in a goroutine 93 | go func() { 94 | err = aesctr.Encrypt(in, pw, keyAes, keyHmac) 95 | if err != nil { 96 | log.Fatal(err) 97 | } 98 | 99 | pw.Close() 100 | }() 101 | 102 | // Log how much data passed through 103 | wc := &WriteCounter{} 104 | finalwriter := io.MultiWriter(wc, out) 105 | 106 | go func() { 107 | for { 108 | select { 109 | case <-time.After(time.Second): 110 | fmt.Printf("Encrypted and Decrypted %10d MB/s of %10d MB\n", wc.Recent()/1024/1024, wc.Total()/1024/1024) 111 | } 112 | } 113 | }() 114 | 115 | err = aesctr.Decrypt(pr, finalwriter, keyAes, keyHmac) 116 | 117 | // writer.Flush() 118 | // x := b.Bytes() 119 | // fmt.Println("Version", x[:1]) 120 | // fmt.Println("IV ", x[1:IV_SIZE+1]) 121 | // fmt.Println("B ", x[IV_SIZE+1:len(x)-hmacSize]) 122 | // fmt.Println("MAC", x[len(x)-hmacSize:]) 123 | // // fmt.Println("MAC", hex.EncodeToString(x[len(x)-hmacSize:])) 124 | 125 | if err != nil { 126 | log.Fatal(err) 127 | } 128 | 129 | } 130 | -------------------------------------------------------------------------------- /http_test.go: -------------------------------------------------------------------------------- 1 | package aesctr 2 | 3 | import ( 4 | "bytes" 5 | "io" 6 | "log" 7 | "net/http" 8 | "net/http/httptest" 9 | "os" 10 | "strings" 11 | "testing" 12 | ) 13 | 14 | // var aesKey = make([]byte, 32) 15 | var aesKey = []byte{139, 46, 150, 181, 48, 123, 170, 178, 55, 133, 209, 214, 35, 46, 101, 16 | 32, 231, 24, 92, 86, 3, 41, 59, 198, 221, 2, 193, 66, 26, 100, 154, 147} 17 | var hmacKey = aesKey 18 | 19 | var encryptHandler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { 20 | 21 | // r.Body -> pipeWriter -> {Encrypt} -> pipeReader -> ResponseWriter 22 | pipeReader, pipeWriter := io.Pipe() 23 | 24 | go func() { 25 | // Write to stdout so we can see what is being read 26 | tee := io.TeeReader(r.Body, os.Stdout) 27 | 28 | err := Encrypt(tee, pipeWriter, aesKey, hmacKey) 29 | if err != nil { 30 | log.Fatal(err) 31 | } 32 | pipeWriter.Close() // Nothing else to write 33 | }() 34 | 35 | // Let the client know what we encrypted by sending it back 36 | _, err := io.Copy(w, pipeReader) 37 | if err != nil { 38 | log.Fatal(err) 39 | } 40 | }) 41 | 42 | func TestHTTPClient(t *testing.T) { 43 | 44 | secretMessage := "My Secret Message" 45 | 46 | // Expected output 47 | in := strings.NewReader(secretMessage) 48 | out := &bytes.Buffer{} 49 | 50 | err := Encrypt(in, out, aesKey, hmacKey) 51 | if err != nil { 52 | t.Fatal(err) 53 | } 54 | 55 | // Create the request and call the HTTP handler 56 | req, err := http.NewRequest("POST", "ABC", strings.NewReader(secretMessage)) 57 | if err != nil { 58 | t.Fatalf("err: %v", err) 59 | } 60 | 61 | resp := httptest.NewRecorder() 62 | encryptHandler(resp, req) 63 | 64 | result := resp.Body.String() 65 | 66 | if result != out.String() { 67 | t.Logf("\nWant: %v\nGot: %v", out.Bytes(), resp.Body.Bytes()) 68 | t.Fail() 69 | } 70 | 71 | } 72 | --------------------------------------------------------------------------------