├── README.md ├── episode1 └── post.md ├── episode10 ├── marshaling_test.go └── post.md ├── episode11 ├── api_expressions.go ├── expressions_test.go ├── item.go ├── post.md ├── strings_expressions.go └── template.yml ├── episode12 ├── orders_test.go ├── post.md └── template.yml ├── episode2 ├── post.md ├── putget_test.go └── template.yml ├── episode3 ├── post.md ├── queries_test.go └── template.yml ├── episode4 ├── gsi.png ├── lsi.png ├── post.md ├── queries_test.go ├── table.png └── template.yml ├── episode5 ├── mapper.go ├── mapper_test.go ├── post.md └── template.yml ├── episode6 ├── mapper.go ├── mapper_test.go ├── post.md └── template.yml ├── episode7 └── post.md ├── episode8 ├── post.md ├── v1 │ ├── sensors │ │ ├── sensors.go │ │ └── sensors_test.go │ └── template.yml └── v2 │ ├── sensors │ ├── sensors.go │ └── sensors_test.go │ └── template.yml ├── episode9 ├── post.md ├── template.yml ├── toggle.go └── toggle_test.go ├── go.mod ├── go.sum └── pkg └── dynamo ├── setup.go ├── setup_test.go ├── table.go ├── table_test.go └── testdata └── template.yml /README.md: -------------------------------------------------------------------------------- 1 | # DynamoDB with Go 2 | 3 | :warning: :warning: :warning: 4 | 5 | DynamoDB with Go was updated to use new (v2) version of AWS SDK for Go. If you want to play with older version 6 | just checkout branch `go-sdk-v1`. 7 | 8 | :warning: :warning: :warning: 9 | 10 | Series of posts on how to use DynamoDB with Go SDK. 11 | 12 | Each episode has it's directory with text of the post and runnable code. 13 | 14 | ## Table of contents 15 | - [Episode #1 - Setup](./episode1/post.md) 16 | - [Episode #2 - Put & Get](./episode2/post.md) 17 | - [Episode #3 - Composite Primary Keys](./episode3/post.md) 18 | - [Episode #4 - Indices ](./episode4/post.md) 19 | - [Episode #5 - Legacy IDs mapping](./episode5/post.md) 20 | - [Episode #6 - Legacy IDs mapping with transactions](./episode6/post.md) 21 | - [Episode #7 - Modelling hierarchical data with Single Table Design](./episode7/post.md) 22 | - [Episode #8 - Implement hierarchical data with Single Table Design](./episode8/post.md) 23 | - [Episode #9 - Switching the toggle, toggling the switch](./episode9/post.md) 24 | - [Episode #10 - Gotcha with empty slices](./episode10/post.md) 25 | - [Episode #11 - Expressions API](./episode11/post.md) 26 | - [Episode #12 - Condition on other item from item collection](./episode12/post.md) 27 | 28 | ## Prerequisites 29 | 1. Golang (1.14 or higher) installed 30 | 2. Docker (19.03 or higher) installed 31 | 32 | ## Running code 33 | 34 | 1. Run local DynamoDB in separate terminal 35 | ``` 36 | docker run --rm -p 8000:8000 amazon/dynamodb-local 37 | ``` 38 | 39 | 2. Execute tests 40 | ``` 41 | go test ./... 42 | ``` -------------------------------------------------------------------------------- /episode1/post.md: -------------------------------------------------------------------------------- 1 | # DynamoDB with Go #1 - Setup 2 | 3 | This post begins short series that aims to explore Go API used when interacting with DynamoDB database. 4 | Such combination of Go and DynamoDB is being used in serverless applications - specifically ones that run 5 | on the GoLD stack, where Go is the runtime for the AWS Lambda and DynamoDB is a database of choice. 6 | 7 | Make sure to read [Introduction to the GoLD stack](https://dev.to/prozz/introduction-to-the-gold-stack-5b66) 8 | by [@prozz](https://twitter.com/prozz). 9 | 10 | Throughout the series we are going to learn how to use the API in a convenient way. We are going to show 11 | some popular use cases, we are going to learn tips, tricks, and we are going to fight gotchas 12 | of that API. 13 | 14 | ## [Setting up the stage](#setting-up-stage) 15 | 16 | The goal of this very first post in the series is to setup environment for us. At the end of 17 | this post I would like to run simple API call that returns connection to the DynamoDB that I 18 | can play with. Before that happens we need to have the following dependencies. 19 | 20 | 1. Golang 1.x (I am using 1.15) installed 21 | 2. Docker 19.03 (or higher) installed 22 | 23 | Next step is to clone the repository. 24 | ``` 25 | git clone git@github.com:jbszczepaniak/dynamodb-with-go.git 26 | ``` 27 | 28 | That's it. Now we can run local DynamoDB. 29 | 30 | ``` 31 | docker run --rm -p 8000:8000 amazon/dynamodb-local 32 | ``` 33 | 34 | This will take entire terminal session. The advantage is that after you are done playing 35 | with DynamoDB - you will remember to shut it down. If you want to run DynamoDB in the background 36 | add `-d` parameter to the `docker run` command. Either way - since you are running the local 37 | version of DynamoDB you can go to the directory where repository was cloned and run. 38 | 39 | ``` 40 | go test ./... -v 41 | ``` 42 | 43 | The idea for this series is for you to always be able to run container with DynamoDB and execute 44 | test suite. Having working examples and being able to play with them is excellent opportunity to learn! 45 | 46 | ## [Creating DynamoDB tables](#creating-tables) 47 | 48 | This series will be driven by tests. We are going to setup DynamoDB table, act on it in some ways 49 | and verify what happened. We already have environment up and running. Now we need to have code that will 50 | create the DynamoDB tables for us. I created dynamo `package` that provides `SetupTable` test helper 51 | that takes path to the CloudFormation template file and table name and creates that table for us. 52 | 53 | Let me show you test that demonstrates usage of the [`SetupTable`](../pkg/dynamo/setup_test.go). 54 | 55 | ```go 56 | ctx := context.Background() 57 | db, cleanup := SetupTable(t, ctx, "PartitionKeyTable", "./testdata/template.yml") 58 | ``` 59 | 60 | `PartitionKeyTable` is the name of the DynamoDB table that is defined in the [`template.yml`](../pkg/dynamo/testdata/template.yml) 61 | file. File itself follows format of CloudFormation templates. 62 | 63 | `SetupTable` returns `db` - connection to the database, and `cleanup` method which needs to be called 64 | after every test. You cannot have many tables with the same name in DynamoDB - we need to clean them up. 65 | 66 | ```go 67 | out, err := db.DescribeTable(ctx, &dynamodb.DescribeTableInput{ 68 | TableName: aws.String("PartitionKeyTable"), 69 | }) 70 | assert.NoError(t, err) 71 | assert.Equal(t, "PartitionKeyTable", *out.Table.TableName) 72 | assert.Equal(t, "pk", *out.Table.AttributeDefinitions[0].AttributeName) 73 | ``` 74 | 75 | Next piece of the test asks DynamoDB about the table we've just created. Upon receiving the answer - we check 76 | whether table name and name of the Partition Key matches specification from CloudFormation template. We will 77 | talk about different types of keys in following episodes of the series. 78 | 79 | ```go 80 | cleanup() 81 | _, err = db.DescribeTable(ctx, &dynamodb.DescribeTableInput{ 82 | TableName: aws.String("PartitionKeyTable"), 83 | }) 84 | 85 | var notfound *types.ResourceNotFoundException 86 | assert.True(t, errors.As(err, ¬found)) 87 | ``` 88 | 89 | At the end of the test we run the `cleanup()` and verify that DynamoDB doesn't anything about it anymore. 90 | 91 | ## [Summary](#summary) 92 | 93 | We've just prepared ourselves for the journey of exploring Go API for DynamoDB. We can create DynamoDB tables 94 | out of CloudFormation templates in our local instance of DynamoDB that runs inside Docker. We can now run tests against 95 | the local instance demonstrating various aspects of the API. 96 | -------------------------------------------------------------------------------- /episode10/marshaling_test.go: -------------------------------------------------------------------------------- 1 | package episode10 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue" 7 | "github.com/davecgh/go-spew/spew" 8 | "github.com/stretchr/testify/assert" 9 | ) 10 | 11 | func TestSlicesMarshaling(t *testing.T) { 12 | 13 | t.Run("regular way", func(t *testing.T) { 14 | t.Skip("this test fails") 15 | attrs, err := attributevalue.Marshal([]string{}) 16 | spew.Dump(attrs) 17 | assert.NoError(t, err) 18 | 19 | var s []string 20 | err = attributevalue.Unmarshal(attrs, &s) 21 | assert.NoError(t, err) 22 | 23 | assert.NotNil(t, s) // fails 24 | assert.Len(t, s, 0) 25 | }) 26 | 27 | t.Run("new way", func(t *testing.T) { 28 | e := attributevalue.NewEncoder(func(opt *attributevalue.EncoderOptions) { 29 | opt.NullEmptySets = true 30 | }) 31 | attrs, err := e.Encode([]string{}) 32 | assert.NoError(t, err) 33 | 34 | var s []string 35 | err = attributevalue.Unmarshal(attrs, &s) 36 | assert.NoError(t, err) 37 | assert.NotNil(t, s) 38 | assert.Len(t, s, 0) 39 | }) 40 | } 41 | -------------------------------------------------------------------------------- /episode10/post.md: -------------------------------------------------------------------------------- 1 | # DynamoDB with Go #10 2 | 3 | This time we are going to cover the gotcha of AWS SDK for Go that has bitten me recently. It's about storing empty slices in the DynamoDB. 4 | 5 | ## [Different flavours of empty slices](#different-falvors-of-empty-slices) 6 | First of all - what is the slice? It is three things: 7 | 1. pointer to the underlying data, 8 | 2. length of the slice, 9 | 3. capacity of the slice. 10 | 11 | These three things make a slice. If an empty slice means the slice with length of 0 then we can create such entity in 12 | at least three different ways. 13 | 14 | ### [1. Zero value slice](#zero-value-slice) 15 | Zero value of a slice is the `nil`. It has capacity of 0, length of 0 and no underlying array. 16 | ```go 17 | var stringSlice []string 18 | ``` 19 | ### [2. Short declaration](#short-declaration) 20 | Short declaration produces slice with capacity of 0, length of 0 and __pointer to the underlying data__. 21 | ```go 22 | stringSlice := []string{} 23 | ``` 24 | 25 | ### [3. Make](#make) 26 | Make function similarly to short declaration produces slice with capacity of 0, length of 0 27 | and __pointer to the underlying data__. 28 | ```go 29 | stringSlice := make([]string, 0) 30 | ``` 31 | 32 | Each method produces slice for which condition `len(stringSlice) == 0` holds true. However, condition `stringSlice == nil` is true only for zero value slice. It is in the spirit of Go for zero value to be useful. Indeed, zero value slice is useful. Sometimes however distinction between `nil` slice and non-`nil` empty slice can be handy. Let's imagine a case where you want to collect information about professional experience from the user. Professional experience can be modelled as a slice of workplaces. You might want to distinguish three different situations. 33 | 1. user didn't provide professional experience yet, 34 | 2. user marked *I do not have any professional experience yet* checkbox, 35 | 3. user provided a list of his past professional endeavors. 36 | 37 | Modelling the difference between cases 1. and 2. can be done with distinction of `nil` slice and non-`nil` empty slice. Luckily for us when marshaling into JSON - Go distinguishes between these cases. 38 | 39 | ```go 40 | var stringSlice []string 41 | json.Marshal(stringSlice) // Marshals into `null` 42 | ``` 43 | 44 | ```go 45 | stringSlice := []string{} 46 | json.Marshal(stringSlice) // Marshals into `[]` 47 | ``` 48 | 49 | ## [Empty slices in the DynamoDB](#empty-slices-dynamo) 50 | 51 | What happens when we try to save an item into the DynamoDB when one of the attributes is empty slice? 52 | 53 | There is no problem with zero value slice. You can save `nil` slice into DynamoDB and after retrieving it, it will be unmarshalled as a `nil` slice. A problem occurs when using empty non-`nil` slices. Look at the test. 54 | 55 | ```go 56 | t.Run("regular way", func(t *testing.T) { 57 | t.Skip("this test fails") 58 | attrs, err := attributevalue.Marshal([]string{}) 59 | assert.NoError(t, err) 60 | 61 | var s []string 62 | err = attributevalue.Unmarshal(attrs, &s) 63 | assert.NoError(t, err) 64 | 65 | assert.NotNil(t, s) // fails 66 | assert.Len(t, s, 0) 67 | }) 68 | ``` 69 | 70 | This failing test shows exactly what is the problem. I mean the problem isn't with the DynamoDB, but rather with how AWS SDK for Go treats slices. The DynamoDB is 100% capable of distinguishing between empty lists and `NULL` values. In order to see that let's look at `attrs`variable from the example above. 71 | 72 | ```go 73 | (*dynamodb.AttributeValue)(0xc0000d5ea0)({ 74 | NULL: true 75 | }) 76 | ``` 77 | This is what `attributevalue.Marshal` function did to non-`nil` empty slice. It changed into `NULL`. Actually as you can see `NULL` is a `bool` field on `dynamodb.AttributeValue` type. This is just how AttributeValue represents something that is `NULL`. Let's keep that in mind. 78 | 79 | What we really want to do if we care about distinction between `nil` slice and non-`nil` empty slice is to use the custom encoder and decoder and set the `NullEmptySets` option that will preserve empty list. 80 | 81 | ```go 82 | t.Run("new way", func(t *testing.T) { 83 | e := attributevalue.NewEncoder(func(opt *attributevalue.EncoderOptions) { 84 | opt.NullEmptySets = true 85 | }) 86 | attrs, err := e.Encode([]string{}) 87 | assert.NoError(t, err) 88 | 89 | var s []string 90 | err = attributevalue.Unmarshal(attrs, &s) 91 | assert.NoError(t, err) 92 | assert.NotNil(t, s) 93 | assert.Len(t, s, 0) 94 | }) 95 | ``` 96 | 97 | This test passes. Let's see how `attrs` variable looks like. 98 | ```go 99 | (*dynamodb.AttributeValue)(0xc00024c140)({ 100 | L: [] 101 | }) 102 | ``` 103 | This is truly empty list. This is what we want! Do we have a success? Not yet... 104 | 105 | The `attributevalue` package is used to transform application types into attribute values - which are types that the DynamoDB understands. It is used when performing for example `PutItem` operation or `GetItem` operation. 106 | 107 | But we have also `UpdateItem` operations and typically we construct them with expression API. Let's look at the example. 108 | ```go 109 | expr, err := expression.NewBuilder(). 110 | WithUpdate(expression.Set(expression.Name("attr_name"), expression.Value([]string{}))). 111 | Build() 112 | ``` 113 | This piece of code is an expresion that should update `attr_name` to the value of `[]string{}` which is empty non-`nil` slice. Let's print that expression and observe what are we dealing with. 114 | 115 | ```go 116 | (expression.Expression) { 117 | expressionMap: (map[expression.expressionType]string) (len=1) { 118 | (expression.expressionType) (len=6) "update": (string) (len=12) "SET #0 = :0\n" 119 | }, 120 | namesMap: (map[string]*string) (len=1) { 121 | (string) (len=2) "#0": (*string)(0xc000051990)((len=3) "123") 122 | }, 123 | valuesMap: (map[string]*dynamodb.AttributeValue) (len=1) { 124 | (string) (len=2) ":0": (*dynamodb.AttributeValue)(0xc0002245a0)({ 125 | NULL: true 126 | }) 127 | } 128 | } 129 | ``` 130 | 131 | This is how update expression looks under the hood. We have `expressionMap`, `namesMap` and `valuesMap` parameters. Expression itself is `SET #0 = :0`. Path `#0` is substituted with `"123"` and `":0"` is substituted with `{NULL: true}`. We never really talked about how AWS SDK encodes expressions into the DynamoDB format because we have the expression API, and we didn't need to think about actual representation. At least until now. As you can see, even though we used `expression.Value([]string{})` it was transformed into the DynamoDB `NULL` value which isn't very good because we didn't want to have nil slice there. As a matter of fact expression API will always transform any slice with length of 0 into `NULL` value. Are we helpless though? No, we are not. Instead of simply using `expression.Value` we need to construct that value by hand. 132 | 133 | ```go 134 | v := expression.Value((&dynamodb.AttributeValue{}).SetL([]*dynamodb.AttributeValue{})) 135 | expr, _ := expression.NewBuilder().WithUpdate(expression.Set(expression.Name("attr_name"), v)).Build() 136 | ``` 137 | 138 | This isn't very pretty, but it works. Now we get: 139 | ```go 140 | (expression.Expression) { 141 | expressionMap: (map[expression.expressionType]string) (len=1) { 142 | (expression.expressionType) (len=6) "update": (string) (len=12) "SET #0 = :0\n" 143 | }, 144 | namesMap: (map[string]*string) (len=1) { 145 | (string) (len=2) "#0": (*string)(0xc000051a20)((len=9) "attr_name") 146 | }, 147 | valuesMap: (map[string]*dynamodb.AttributeValue) (len=1) { 148 | (string) (len=2) ":0": (*dynamodb.AttributeValue)(0xc000222780)({ 149 | L: [] 150 | }) 151 | } 152 | } 153 | ``` 154 | The expression is aware of the empty list. 155 | 156 | ## [Summary](#summary) 157 | Sometimes we need to get our hands dirty and tinker with the internals of the API in order to get things done. But from now on - we'll have complete confidence that we know what is going on with our slices when we use them with Dynamo. -------------------------------------------------------------------------------- /episode11/api_expressions.go: -------------------------------------------------------------------------------- 1 | package episode12 2 | 3 | import ( 4 | "context" 5 | "errors" 6 | "fmt" 7 | 8 | "github.com/aws/aws-sdk-go-v2/aws" 9 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue" 10 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression" 11 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 12 | "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" 13 | ) 14 | 15 | func GetItemCollectionV2(ctx context.Context, db *dynamodb.Client, table, pk string) ([]Item, error) { 16 | expr, err := expression.NewBuilder(). 17 | WithKeyCondition(expression.KeyEqual(expression.Key("pk"), expression.Value(pk))). 18 | Build() 19 | 20 | if err != nil { 21 | return nil, err 22 | } 23 | out, err := db.Query(ctx, &dynamodb.QueryInput{ 24 | KeyConditionExpression: expr.KeyCondition(), 25 | ExpressionAttributeNames: expr.Names(), 26 | ExpressionAttributeValues: expr.Values(), 27 | TableName: aws.String(table), 28 | }) 29 | if err != nil { 30 | return nil, err 31 | } 32 | 33 | var items []Item 34 | err = attributevalue.UnmarshalListOfMaps(out.Items, &items) 35 | if err != nil { 36 | return nil, err 37 | } 38 | return items, nil 39 | } 40 | 41 | func UpdateAWhenBAndUnsetBV2(ctx context.Context, db *dynamodb.Client, table string, k Key, newA, whenB string) (Item, error) { 42 | marshaledKey, err := attributevalue.MarshalMap(k) 43 | if err != nil { 44 | return Item{}, err 45 | } 46 | 47 | expr, err := expression.NewBuilder(). 48 | WithCondition(expression.Equal(expression.Name("b"), expression.Value(whenB))). 49 | WithUpdate(expression. 50 | Set(expression.Name("a"), expression.Value(newA)). 51 | Remove(expression.Name("b"))). 52 | Build() 53 | if err != nil { 54 | return Item{}, err 55 | } 56 | out, err := db.UpdateItem(ctx, &dynamodb.UpdateItemInput{ 57 | ConditionExpression: expr.Condition(), 58 | ExpressionAttributeNames: expr.Names(), 59 | ExpressionAttributeValues: expr.Values(), 60 | UpdateExpression: expr.Update(), 61 | Key: marshaledKey, 62 | ReturnValues: types.ReturnValueAllNew, 63 | TableName: aws.String(table), 64 | }) 65 | if err != nil { 66 | var conditionFailed *types.ConditionalCheckFailedException 67 | if errors.As(err, &conditionFailed) { 68 | return Item{}, fmt.Errorf("b is not %s, aborting update", whenB) 69 | } 70 | return Item{}, err 71 | } 72 | var i Item 73 | err = attributevalue.UnmarshalMap(out.Attributes, &i) 74 | if err != nil { 75 | return Item{}, err 76 | } 77 | return i, nil 78 | } 79 | 80 | func PutIfNotExistsV2(ctx context.Context, db *dynamodb.Client, table string, k Key) error { 81 | marshaledKey, err := attributevalue.MarshalMap(k) 82 | if err != nil { 83 | return err 84 | } 85 | 86 | expr, err := expression.NewBuilder(). 87 | WithCondition(expression.AttributeNotExists(expression.Name("pk"))). 88 | Build() 89 | if err != nil { 90 | return err 91 | } 92 | 93 | _, err = db.PutItem(ctx, &dynamodb.PutItemInput{ 94 | ConditionExpression: expr.Condition(), 95 | ExpressionAttributeNames: expr.Names(), 96 | Item: marshaledKey, 97 | TableName: aws.String(table), 98 | }) 99 | 100 | if err != nil { 101 | var conditionFailed *types.ConditionalCheckFailedException 102 | if errors.As(err, &conditionFailed) { 103 | return errors.New("Item with this Key already exists") 104 | } 105 | return err 106 | } 107 | 108 | return nil 109 | } 110 | -------------------------------------------------------------------------------- /episode11/expressions_test.go: -------------------------------------------------------------------------------- 1 | package episode12 2 | 3 | import ( 4 | "context" 5 | "dynamodb-with-go/pkg/dynamo" 6 | "testing" 7 | 8 | "github.com/aws/aws-sdk-go-v2/aws" 9 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue" 10 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 11 | 12 | "github.com/stretchr/testify/assert" 13 | ) 14 | 15 | func insert(ctx context.Context, db *dynamodb.Client, table string, items ...Item) { 16 | for _, i := range items { 17 | attrs, err := attributevalue.MarshalMap(i) 18 | if err != nil { 19 | panic(err) 20 | } 21 | 22 | _, err = db.PutItem(ctx, &dynamodb.PutItemInput{ 23 | Item: attrs, 24 | TableName: aws.String(table), 25 | }) 26 | if err != nil { 27 | panic(err) 28 | } 29 | } 30 | } 31 | 32 | func TestExpressions(t *testing.T) { 33 | item1 := Item{Key: Key{PK: "1", SK: "1"}, A: "foo", B: "bar"} 34 | item2 := Item{Key: Key{PK: "1", SK: "2"}, A: "foo", B: "baz"} 35 | 36 | t.Run("v1 - get whole item collection", func(t *testing.T) { 37 | ctx := context.Background() 38 | tableName := "ATable" 39 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 40 | defer cleanup() 41 | insert(ctx, db, tableName, item1, item2) 42 | 43 | collection, err := GetItemCollectionV1(ctx, db, tableName, "1") 44 | assert.NoError(t, err) 45 | assert.Subset(t, collection, []Item{item1, item2}) 46 | assert.Len(t, collection, 2) 47 | }) 48 | 49 | t.Run("v2 - get whole item collection", func(t *testing.T) { 50 | ctx := context.Background() 51 | tableName := "ATable" 52 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 53 | defer cleanup() 54 | insert(ctx, db, tableName, item1, item2) 55 | 56 | collection, err := GetItemCollectionV2(ctx, db, tableName, "1") 57 | assert.NoError(t, err) 58 | assert.Subset(t, collection, []Item{item1, item2}) 59 | assert.Len(t, collection, 2) 60 | }) 61 | 62 | t.Run("v1 - update A and unset B but only if B is set to `baz`", func(t *testing.T) { 63 | ctx := context.Background() 64 | tableName := "ATable" 65 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 66 | defer cleanup() 67 | insert(ctx, db, tableName, item1, item2) 68 | 69 | updated, err := UpdateAWhenBAndUnsetBV1(ctx, db, tableName, Key{PK: "1", SK: "1"}, "newA", "baz") 70 | if assert.Error(t, err) { 71 | assert.Equal(t, "b is not baz, aborting update", err.Error()) 72 | } 73 | assert.Empty(t, updated) 74 | 75 | updated, err = UpdateAWhenBAndUnsetBV1(ctx, db, tableName, Key{PK: "1", SK: "2"}, "newA", "baz") 76 | assert.NoError(t, err) 77 | assert.Equal(t, "newA", updated.A) 78 | assert.Empty(t, updated.B) 79 | }) 80 | 81 | t.Run("v2 - update A and unset B but only if B is set to `baz`", func(t *testing.T) { 82 | ctx := context.Background() 83 | tableName := "ATable" 84 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 85 | defer cleanup() 86 | insert(ctx, db, tableName, item1, item2) 87 | 88 | updated, err := UpdateAWhenBAndUnsetBV2(ctx, db, tableName, Key{PK: "1", SK: "1"}, "newA", "baz") 89 | if assert.Error(t, err) { 90 | assert.Equal(t, "b is not baz, aborting update", err.Error()) 91 | } 92 | assert.Empty(t, updated) 93 | 94 | updated, err = UpdateAWhenBAndUnsetBV2(ctx, db, tableName, Key{PK: "1", SK: "2"}, "newA", "baz") 95 | assert.NoError(t, err) 96 | assert.Equal(t, "newA", updated.A) 97 | assert.Empty(t, updated.B) 98 | }) 99 | 100 | t.Run("v1 - put if doesn't exist", func(t *testing.T) { 101 | ctx := context.Background() 102 | tableName := "ATable" 103 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 104 | defer cleanup() 105 | insert(ctx, db, tableName, item1, item2) 106 | 107 | err := PutIfNotExistsV1(ctx, db, tableName, Key{PK: "1", SK: "2"}) 108 | if assert.Error(t, err) { 109 | assert.Equal(t, "Item with this Key already exists", err.Error()) 110 | } 111 | 112 | err = PutIfNotExistsV1(ctx, db, tableName, Key{PK: "10", SK: "20"}) 113 | assert.NoError(t, err) 114 | }) 115 | 116 | t.Run("v2 - put if doesn't exist", func(t *testing.T) { 117 | ctx := context.Background() 118 | tableName := "ATable" 119 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 120 | defer cleanup() 121 | insert(ctx, db, tableName, item1, item2) 122 | 123 | err := PutIfNotExistsV2(ctx, db, tableName, Key{PK: "1", SK: "2"}) 124 | if assert.Error(t, err) { 125 | assert.Equal(t, "Item with this Key already exists", err.Error()) 126 | } 127 | 128 | err = PutIfNotExistsV2(ctx, db, tableName, Key{PK: "10", SK: "20"}) 129 | assert.NoError(t, err) 130 | 131 | }) 132 | } 133 | -------------------------------------------------------------------------------- /episode11/item.go: -------------------------------------------------------------------------------- 1 | package episode12 2 | 3 | type Key struct { 4 | PK string `dynamodbav:"pk"` 5 | SK string `dynamodbav:"sk"` 6 | } 7 | 8 | type Item struct { 9 | Key 10 | A string `dynamodbav:"a"` 11 | B string `dynamodbav:"b"` 12 | } 13 | -------------------------------------------------------------------------------- /episode11/post.md: -------------------------------------------------------------------------------- 1 | # DynamoDB with Go #11 2 | 3 | I am advocating for using the expression API from the first episode of the DynamoDB with Go. In this episode I would like to show you why I enjoy using them so much. We are going to examine three examples. Each of them is going to be implemented using plain text expressions and with expressions API. I hope that after examining these three comparisons you'll be convinced that expression API is the way to go. 4 | 5 | ## [Example #1 - Get item collection ](#example1) 6 | When table uses Composite Primary Key (meaning that it has both PK and SK), an item collection is bunch of items that share the same Partition Key. In order to fetch the item collection we need to use DynamoDB Query with Key Condition. Let's start with a test. 7 | 8 | ```go 9 | t.Run("v1 - get whole item collection", func(t *testing.T) { 10 | ctx := context.Background() 11 | tableName := "ATable" 12 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 13 | defer cleanup() 14 | insert(ctx, db, tableName, item1, item2) 15 | 16 | collection, err := GetItemCollectionV1(ctx, db, tableName, "1") 17 | assert.NoError(t, err) 18 | assert.Subset(t, collection, []Item{item1, item2}) 19 | assert.Len(t, collection, 2) 20 | }) 21 | ``` 22 | 23 | I am using `insert` helper that sets up the stage for me. We are going to begin each test with this one. 24 | 25 | > I would like to draw your attention to the fact that `GetItemCollection` function has `V1` prefix. We also have version 2 of the function. For this episode I'll set a convention where all functions that use traditional string based expressions end with `V1` and are located in [strings_expressions.go](./strings_expressions.go) file. Functions that use expression API can be found in [api_expressions.go](./api_expressions.go) and end with `V2` prefix. 26 | 27 | Let's compare implementations. 28 | 29 | ```go 30 | out, err := db.Query(ctx, &dynamodb.QueryInput{ 31 | KeyConditionExpression: aws.String("#key = :value"), 32 | ExpressionAttributeNames: map[string]string{ 33 | "#key": "pk", 34 | }, 35 | ExpressionAttributeValues: map[string]types.AttributeValue{ 36 | ":value": &types.AttributeValueMemberS{Value: pk}, 37 | }, 38 | TableName: aws.String(table), 39 | }) 40 | ``` 41 | 42 | vs 43 | 44 | ```go 45 | expr, err := expression.NewBuilder(). 46 | WithKeyCondition(expression.KeyEqual(expression.Key("pk"), expression.Value(pk))). 47 | Build() 48 | 49 | out, err := db.Query(ctx, &dynamodb.QueryInput{ 50 | KeyConditionExpression: expr.KeyCondition(), 51 | ExpressionAttributeNames: expr.Names(), 52 | ExpressionAttributeValues: expr.Values(), 53 | TableName: aws.String(table), 54 | }) 55 | ``` 56 | 57 | Certainly second version hides from us details that we need to think about when constructing expressions ourselves. 58 | 59 | What we really want to do in this example is to construct condition that checks equality. Attribute `"pk"` should be equal to the variable `pk`. Doing it directly is risky because we cannot use reserved keywords of the DynamoDB. This is why we create aliases `#key` and `:value` which are later mapped to real values, `#key` becomes `"pk"`, and `:value` becomes contents of variable `pk`. When using expressions API, we don't need to build these by hand, the API is doing it automatically. Let's examine the expression that is built by the expression API. 60 | 61 | ```go 62 | (expression.Expression) { 63 | expressionMap: (map[expression.expressionType]string) (len=1) { 64 | (expression.expressionType) (len=12) "keyCondition": (string) (len=7) "#0 = :0" 65 | }, 66 | namesMap: (map[string]*string) (len=1) { 67 | (string) (len=2) "#0": (*string)(0xc0002b63b0)((len=2) "pk") 68 | }, 69 | valuesMap: (map[string]*dynamodb.AttributeValue) (len=1) { 70 | (string) (len=2) ":0": (*dynamodb.AttributeValue)(0xc0002768c0)({ 71 | S: "1" 72 | }) 73 | } 74 | } 75 | ``` 76 | 77 | It is really similar. The only difference is that expression API creates aliases meaningless for humans like `#0` and `:0`. Note that alias for key has to start with `:` and alias for value with `#`. 78 | 79 | > Further parts of both versions of implementations are identical - so I am skipping them here. You can examine them in the repository. The same goes for second and third example. I am going to show only expressions - as they are the only parts that differ. 80 | 81 | ## [Example #2 - "update A and unset B but only if B is set to `baz`"](#example2) 82 | 83 | Taken out of context doesn't really make sens, but this scenario is inspired by real life case. For given item that has attributes `A` and `B` I want to update `A` to some value and unset `B` but only if `B` is set to `baz`. 84 | 85 | Let's jump into the test. 86 | 87 | ```go 88 | t.Run("v1 - update A and unset B but only if B is set to `baz`", func(t *testing.T) { 89 | ctx := context.Background() 90 | tableName := "ATable" 91 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 92 | defer cleanup() 93 | insert(ctx, db, tableName, item1, item2) 94 | 95 | updated, err := UpdateAWhenBAndUnsetBV1(ctx, db, tableName, Key{PK: "1", SK: "1"}, "newA", "baz") 96 | if assert.Error(t, err) { 97 | assert.Equal(t, "b is not baz, aborting update", err.Error()) 98 | } 99 | assert.Empty(t, updated) 100 | 101 | updated, err = UpdateAWhenBAndUnsetBV1(ctx, db, tableName, Key{PK: "1", SK: "2"}, "newA", "baz") 102 | assert.NoError(t, err) 103 | assert.Equal(t, "newA", updated.A) 104 | assert.Empty(t, updated.B) 105 | }) 106 | ``` 107 | It turns out that `item1` has `B` set to `bar` thus it's not updated, `item2` on the other hand has `B` set to `baz` and it is updated. 108 | 109 | Let's compare implementations. 110 | 111 | ```go 112 | out, err := db.UpdateItem(ctx, &dynamodb.UpdateItemInput{ 113 | ConditionExpression: aws.String("#b = :b"), 114 | ExpressionAttributeNames: map[string]string{ 115 | "#b": "b", 116 | "#a": "a", 117 | }, 118 | ExpressionAttributeValues: map[string]types.AttributeValue{ 119 | ":b": &types.AttributeValueMemberS{Value: whenB}, 120 | ":a": &types.AttributeValueMemberS{Value: newA}, 121 | }, 122 | Key: marshaledKey, 123 | ReturnValues: types.ReturnValueAllNew, 124 | TableName: aws.String(table), 125 | UpdateExpression: aws.String("REMOVE #b SET #a = :a"), 126 | }) 127 | ``` 128 | 129 | vs. 130 | 131 | ```go 132 | expr, err := expression.NewBuilder(). 133 | WithCondition(expression.Equal(expression.Name("b"), expression.Value(whenB))). 134 | WithUpdate(expression. 135 | Set(expression.Name("a"), expression.Value(newA)). 136 | Remove(expression.Name("b"))). 137 | Build() 138 | 139 | out, err := db.UpdateItem(ctx, &dynamodb.UpdateItemInput{ 140 | ConditionExpression: expr.Condition(), 141 | ExpressionAttributeNames: expr.Names(), 142 | ExpressionAttributeValues: expr.Values(), 143 | UpdateExpression: expr.Update(), 144 | Key: marshaledKey, 145 | ReturnValues: types.ReturnValueAllNew, 146 | TableName: aws.String(table), 147 | }) 148 | ``` 149 | 150 | I think that second approach is better because it is less error prone and gives more clarity (which is 100% subjective by the way). This example is fairly simple. The more fields you want to update, and more complex your conditions are - the more you'll appreciate type systems of Golang when constructing expressions with the expressions API. 151 | 152 | On a side note, notice how expression builder allows you to mix both condition expressions and update expressions. 153 | 154 | ## [Example #3 - Put if doesn't exist](#example3) 155 | 156 | By default `PutItem` operation overwrites an item if it already exists. It is very common to make sure to not overwrite that, and the tool to do that is the condition expression. Here is the test. 157 | 158 | ```go 159 | t.Run("v1 - put if doesn't exist", func(t *testing.T) { 160 | ctx := context.Background() 161 | tableName := "ATable" 162 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 163 | defer cleanup() 164 | insert(ctx, db, tableName, item1, item2) 165 | 166 | err := PutIfNotExistsV1(ctx, db, tableName, Key{PK: "1", SK: "2"}) 167 | if assert.Error(t, err) { 168 | assert.Equal(t, "Item with this Key already exists", err.Error()) 169 | } 170 | 171 | err = PutIfNotExistsV1(ctx, db, tableName, Key{PK: "10", SK: "20"}) 172 | assert.NoError(t, err) 173 | }) 174 | ``` 175 | 176 | Item with PK=1 and SK=2 was already inserted to the DynamoDB, thus cannot be inserted again and test fails. For PK=10 and SK=20 operation succeeds. Let's compare implementations. 177 | 178 | ```go 179 | _, err = db.PutItem(ctx, &dynamodb.PutItemInput{ 180 | ConditionExpression: aws.String("attribute_not_exists(#pk)"), 181 | ExpressionAttributeNames: map[string]string{ 182 | "#pk": "pk", 183 | }, 184 | Item: marshaledKey, 185 | TableName: aws.String(table), 186 | }) 187 | ``` 188 | 189 | vs. 190 | 191 | ```go 192 | expr, err := expression.NewBuilder(). 193 | WithCondition(expression.AttributeNotExists(expression.Name("pk"))). 194 | Build() 195 | 196 | _, err = db.PutItem(ctx, &dynamodb.PutItemInput{ 197 | ConditionExpression: expr.Condition(), 198 | ExpressionAttributeNames: expr.Names(), 199 | Item: marshaledKey, 200 | TableName: aws.String(table), 201 | }) 202 | ``` 203 | 204 | If I am being honest - I kind of like first version. It is concise, you immediately know what is going. The thing is that I think that conventions matter, and I'd rather stick with one way of doing things. Other thing is that even though `attribute_not_exists(#pk)` is cute - it is so simple to make mistake - and you don't have any autocompletion when writing it by hand. 205 | 206 | ## [Summary](#summary) 207 | I think that ability to write expressions by hand matters. I believe that this ability helps along the way when you're trying to figure out when your query breaks. Having said that - when you know what is what - I think in day to day work it is better to stick with expression API as it very convenient and less error prone than plain text expressions. 208 | -------------------------------------------------------------------------------- /episode11/strings_expressions.go: -------------------------------------------------------------------------------- 1 | package episode12 2 | 3 | import ( 4 | "context" 5 | "errors" 6 | "fmt" 7 | 8 | "github.com/aws/aws-sdk-go-v2/aws" 9 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue" 10 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 11 | "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" 12 | ) 13 | 14 | func GetItemCollectionV1(ctx context.Context, db *dynamodb.Client, table, pk string) ([]Item, error) { 15 | out, err := db.Query(ctx, &dynamodb.QueryInput{ 16 | KeyConditionExpression: aws.String("#key = :value"), 17 | ExpressionAttributeNames: map[string]string{ 18 | "#key": "pk", 19 | }, 20 | ExpressionAttributeValues: map[string]types.AttributeValue{ 21 | ":value": &types.AttributeValueMemberS{Value: pk}, 22 | }, 23 | TableName: aws.String(table), 24 | }) 25 | if err != nil { 26 | return nil, err 27 | } 28 | 29 | var items []Item 30 | err = attributevalue.UnmarshalListOfMaps(out.Items, &items) 31 | if err != nil { 32 | return nil, err 33 | } 34 | return items, nil 35 | } 36 | 37 | func UpdateAWhenBAndUnsetBV1(ctx context.Context, db *dynamodb.Client, table string, k Key, newA, whenB string) (Item, error) { 38 | marshaledKey, err := attributevalue.MarshalMap(k) 39 | if err != nil { 40 | return Item{}, err 41 | } 42 | 43 | out, err := db.UpdateItem(ctx, &dynamodb.UpdateItemInput{ 44 | ConditionExpression: aws.String("#b = :b"), 45 | ExpressionAttributeNames: map[string]string{ 46 | "#b": "b", 47 | "#a": "a", 48 | }, 49 | ExpressionAttributeValues: map[string]types.AttributeValue{ 50 | ":b": &types.AttributeValueMemberS{Value: whenB}, 51 | ":a": &types.AttributeValueMemberS{Value: newA}, 52 | }, 53 | Key: marshaledKey, 54 | ReturnValues: types.ReturnValueAllNew, 55 | TableName: aws.String(table), 56 | UpdateExpression: aws.String("REMOVE #b SET #a = :a"), 57 | }) 58 | if err != nil { 59 | var conditionFailed *types.ConditionalCheckFailedException 60 | if errors.As(err, &conditionFailed) { 61 | return Item{}, fmt.Errorf("b is not %s, aborting update", whenB) 62 | } 63 | return Item{}, err 64 | } 65 | var i Item 66 | err = attributevalue.UnmarshalMap(out.Attributes, &i) 67 | if err != nil { 68 | return Item{}, err 69 | } 70 | return i, nil 71 | } 72 | 73 | func PutIfNotExistsV1(ctx context.Context, db *dynamodb.Client, table string, k Key) error { 74 | marshaledKey, err := attributevalue.MarshalMap(k) 75 | if err != nil { 76 | return err 77 | } 78 | 79 | _, err = db.PutItem(ctx, &dynamodb.PutItemInput{ 80 | ConditionExpression: aws.String("attribute_not_exists(#pk)"), 81 | ExpressionAttributeNames: map[string]string{ 82 | "#pk": "pk", 83 | }, 84 | Item: marshaledKey, 85 | TableName: aws.String(table), 86 | }) 87 | 88 | if err != nil { 89 | var conditionFailed *types.ConditionalCheckFailedException 90 | if errors.As(err, &conditionFailed) { 91 | return errors.New("Item with this Key already exists") 92 | } 93 | return err 94 | } 95 | 96 | return nil 97 | } 98 | -------------------------------------------------------------------------------- /episode11/template.yml: -------------------------------------------------------------------------------- 1 | Resources: 2 | ATable: 3 | Type: AWS::DynamoDB::Table 4 | Properties: 5 | AttributeDefinitions: 6 | - AttributeName: pk 7 | AttributeType: S 8 | - AttributeName: sk 9 | AttributeType: S 10 | KeySchema: 11 | - AttributeName: pk 12 | KeyType: HASH 13 | - AttributeName: sk 14 | KeyType: RANGE 15 | BillingMode: PAY_PER_REQUEST 16 | TableName: ATable 17 | -------------------------------------------------------------------------------- /episode12/orders_test.go: -------------------------------------------------------------------------------- 1 | package episode13 2 | 3 | import ( 4 | "dynamodb-with-go/pkg/dynamo" 5 | "errors" 6 | "testing" 7 | 8 | "context" 9 | 10 | "github.com/aws/aws-sdk-go-v2/aws" 11 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression" 12 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 13 | "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" 14 | "github.com/stretchr/testify/assert" 15 | ) 16 | 17 | func TestInsertingOrderFailsBecauseUserDoesNotExist(t *testing.T) { 18 | ctx := context.Background() 19 | tableName := "ATable" 20 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 21 | defer cleanup() 22 | 23 | expr, err := expression.NewBuilder(). 24 | WithCondition(expression.AttributeExists(expression.Name("pk"))). 25 | WithUpdate(expression.Add(expression.Name("orders_count"), expression.Value(1))). 26 | Build() 27 | assert.NoError(t, err) 28 | 29 | _, err = db.TransactWriteItems(ctx, &dynamodb.TransactWriteItemsInput{ 30 | TransactItems: []types.TransactWriteItem{ 31 | { 32 | Put: &types.Put{ 33 | Item: map[string]types.AttributeValue{ 34 | "pk": &types.AttributeValueMemberS{Value: "1234"}, 35 | "sk": &types.AttributeValueMemberS{Value: "ORDER#2017-03-04 00:00:00 +0000 UTC"}, 36 | }, 37 | TableName: aws.String(tableName), 38 | }, 39 | }, 40 | { 41 | Update: &types.Update{ 42 | ConditionExpression: expr.Condition(), 43 | ExpressionAttributeValues: expr.Values(), 44 | ExpressionAttributeNames: expr.Names(), 45 | UpdateExpression: expr.Update(), 46 | Key: map[string]types.AttributeValue{ 47 | "pk": &types.AttributeValueMemberS{Value: "1234"}, 48 | "sk": &types.AttributeValueMemberS{Value: "USERINFO"}, 49 | }, 50 | TableName: aws.String(tableName), 51 | }, 52 | }, 53 | }, 54 | }) 55 | 56 | assert.Error(t, err) 57 | var transactionCancelled *types.TransactionCanceledException 58 | assert.True(t, errors.As(err, &transactionCancelled)) 59 | } 60 | 61 | func TestInsertingOrderSucceedsBecauseUserExists(t *testing.T) { 62 | ctx := context.Background() 63 | tableName := "ATable" 64 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 65 | defer cleanup() 66 | 67 | _, err := db.PutItem(ctx, &dynamodb.PutItemInput{ 68 | Item: map[string]types.AttributeValue{ 69 | "pk": &types.AttributeValueMemberS{Value: "1234"}, 70 | "sk": &types.AttributeValueMemberS{Value: "USERINFO"}, 71 | }, 72 | TableName: aws.String(tableName), 73 | }) 74 | assert.NoError(t, err) 75 | 76 | expr, err := expression.NewBuilder(). 77 | WithCondition(expression.AttributeExists(expression.Name("pk"))). 78 | WithUpdate(expression.Add(expression.Name("orders_count"), expression.Value(1))). 79 | Build() 80 | assert.NoError(t, err) 81 | 82 | _, err = db.TransactWriteItems(ctx, &dynamodb.TransactWriteItemsInput{ 83 | TransactItems: []types.TransactWriteItem{ 84 | { 85 | Put: &types.Put{ 86 | Item: map[string]types.AttributeValue{ 87 | "pk": &types.AttributeValueMemberS{Value: "1234"}, 88 | "sk": &types.AttributeValueMemberS{Value: "ORDER#2017-03-04 00:00:00 +0000 UTC"}, 89 | }, 90 | TableName: aws.String(tableName), 91 | }, 92 | }, 93 | { 94 | Update: &types.Update{ 95 | ConditionExpression: expr.Condition(), 96 | ExpressionAttributeValues: expr.Values(), 97 | ExpressionAttributeNames: expr.Names(), 98 | UpdateExpression: expr.Update(), 99 | Key: map[string]types.AttributeValue{ 100 | "pk": &types.AttributeValueMemberS{Value: "1234"}, 101 | "sk": &types.AttributeValueMemberS{Value: "USERINFO"}, 102 | }, 103 | TableName: aws.String(tableName), 104 | }, 105 | }, 106 | }, 107 | }) 108 | assert.NoError(t, err) 109 | } 110 | -------------------------------------------------------------------------------- /episode12/post.md: -------------------------------------------------------------------------------- 1 | # DynamoDB with Go #12 2 | 3 | Item collection is a set of attributes that share the same Partition Key. A common pattern is to model one to many relationships with item collections where PK represents commonality among items and SK distinguishes types of items. 4 | 5 | As an example we can take information about user and orders placed by the user. 6 | 7 | | PK | SK | 8 | | --- | ---- | 9 | | 1234 | USERINFO | 10 | | 1234 | ORDER#2017-03-04 00:00:00 +0000 UTC| 11 | | 1234 | ORDER#2017-02-04 00:00:00 +0000 UTC| 12 | 13 | This is Single Table Design. We put different types of things into single table so that we can satisfy access patterns with minimal interactions with DynamoDB table. In this example we can display on a single page information about user with the latest orders while doing single query. PK represents ID of the user, and SK depending on the case represents basic user information or the order. 14 | 15 | Now, that we've established that querying data gets simpler with Single Table Design, let us consider inserting data. 16 | 17 | We need to be able to register a user. This use case is fairly simple. We need to make sure, that user with given ID doesn't exist in our system yet. The tool to do just that is to use expression like this one. 18 | 19 | ```go 20 | expr, err := expression.NewBuilder(). 21 | WithCondition(expression.AttributeNotExists(expression.Name("pk"))). 22 | Build() 23 | ``` 24 | 25 | When we have a user, we can place orders. Here is the thing. I would like to be able to place orders only for existing users. I would like to do something so that test passes. We want to get error when placing an order for the user that doesn't exist yet. 26 | 27 | ```go 28 | func TestInsertingOrderFailsBecauseUserDoesNotExist(t *testing.T) { 29 | ctx := context.Background() 30 | tableName := "ATable" 31 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 32 | defer cleanup() 33 | 34 | _, err := db.PutItem(ctx, &dynamodb.PutItemInput{ 35 | Item: map[string]*dynamodb.AttributeValue{ 36 | "pk": &types.AttributeValueMemberS{Value:" 1234"}, 37 | "sk": &types.AttributeValueMemberS{Value: "ORDER#2017-03-04 00:00:00 +0000 UTC"}, 38 | }, 39 | TableName: aws.String(tableName), 40 | }) 41 | assert.Error(t, err) 42 | } 43 | ``` 44 | 45 | Don't consider this a real world unit test. I am using `_test.go` file merely for executing queries. Anyway, this test fails. There is no error. Can we write simple condition that makes sure that user for which we want to place order exists? 46 | 47 | Unfortunately, we can't do this _easily_. Writing conditions that relate to other items in the Item Collection is impossible. In order to make sure that user exists while introducing new order we need to use the transaction. 48 | 49 | ```go 50 | func TestInsertingOrderFailsBecauseUserDoesNotExist(t *testing.T) { 51 | ctx := context.Background() 52 | tableName := "ATable" 53 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 54 | defer cleanup() 55 | 56 | expr, err := expression.NewBuilder(). 57 | WithCondition(expression.AttributeExists(expression.Name("pk"))). 58 | WithUpdate(expression.Add(expression.Name("orders_count"), expression.Value(1))). 59 | Build() 60 | assert.NoError(t, err) 61 | 62 | _, err = db.TransactWriteItems(ctx, &dynamodb.TransactWriteItemsInput{ 63 | TransactItems: []types.TransactWriteItem{ 64 | { 65 | Put: &types.Put{ 66 | Item: map[string]types.AttributeValue{ 67 | "pk": &types.AttributeValueMemberS{Value: "1234"}, 68 | "sk": &types.AttributeValueMemberS{Value: "ORDER#2017-03-04 00:00:00 +0000 UTC"}, 69 | }, 70 | TableName: aws.String(tableName), 71 | }, 72 | }, 73 | { 74 | Update: &dynamodb.Update{ 75 | ConditionExpression: expr.Condition(), 76 | ExpressionAttributeValues: expr.Values(), 77 | ExpressionAttributeNames: expr.Names(), 78 | UpdateExpression: expr.Update(), 79 | Key: map[string]types.AttributeValue{ 80 | "pk": &types.AttributeValueMemberS{Value: "1234"}, 81 | "sk": &types.AttributeValueMemberS{Value: "USERINFO"}, 82 | }, 83 | TableName: aws.String(tableName), 84 | }, 85 | }, 86 | }, 87 | }) 88 | 89 | assert.Error(t, err) 90 | var transactionCancelled *types.TransactionCanceledException 91 | assert.True(t, errors.As(err, &transactionCancelled)) 92 | } 93 | ``` 94 | 95 | Transaction has two items. First one (Put) is what we really want to do here which is placing new order. Additionally, there is an update for item that represents information about the user. I did it so that I can put condition on that update operation. The condition says exactly what we wanted. Transaction will succeed only for existing user. 96 | 97 | You can notice however that I am increasing `orders_count` which wasn't talked about as a requirement for this example. It doesn't really matter what gets updated, but you cannot perform empty update. We could choose to update some helper attribute to `""` but I wanted to make it useful. We are killing two birds with one stone. We get orders count for free. 98 | 99 | Takeaway from this episode is the following one. 100 | > If you want to base the condition of your operation on other item - you need to use transaction. 101 | -------------------------------------------------------------------------------- /episode12/template.yml: -------------------------------------------------------------------------------- 1 | Resources: 2 | ATable: 3 | Type: AWS::DynamoDB::Table 4 | Properties: 5 | AttributeDefinitions: 6 | - AttributeName: pk 7 | AttributeType: S 8 | - AttributeName: sk 9 | AttributeType: S 10 | KeySchema: 11 | - AttributeName: pk 12 | KeyType: HASH 13 | - AttributeName: sk 14 | KeyType: RANGE 15 | BillingMode: PAY_PER_REQUEST 16 | TableName: ATable 17 | -------------------------------------------------------------------------------- /episode2/post.md: -------------------------------------------------------------------------------- 1 | # DynamoDB with Go #2 - Put & Get 2 | Today we are going to do the simplest thing you could imagine with the DynamoDB. First we are going to put something in, then we will take it out. It seems too easy and not worth reading about but bear with me for a moment. 3 | 4 | In the first episode of the series we successfully created environment in which we are going to play with DynamoDB. You can find code for this episode 5 | in [episode2](.) directory. 6 | 7 | ## [Database layout](#database-layout) 8 | 9 | With the environment ready to go we can start solving problems. Our _problem_ for today is to save and retrieve basic information about Order. Order has three properties. 10 | - `id` - string 11 | - `price` - number 12 | - `is_shipped` - boolean 13 | 14 | In order to define the database layout I am using CloudFormation - assembly language for AWS infrastructure. You can 15 | create DynamoDB tables via different channels for example using AWS CLI or with AWS console. I chose AWS CloudFormation 16 | because if you work with serverless applications using Serverless Application Model (SAM) or Serverless framework - this 17 | is how you are going to define your tables. 18 | 19 | Let’s see how it looks. 20 | 21 | ```yaml 22 | AWSTemplateFormatVersion: "2010-09-09" 23 | Resources: 24 | OrdersTable: 25 | Type: AWS::DynamoDB::Table 26 | Properties: 27 | AttributeDefinitions: 28 | - AttributeName: id 29 | AttributeType: S 30 | KeySchema: 31 | - AttributeName: id 32 | KeyType: HASH 33 | BillingMode: PAY_PER_REQUEST 34 | TableName: OrdersTable 35 | 36 | ``` 37 | 38 | First of all the table is called __OrdersTable__. Next, let’s focus on __AttributeDefinitions__. _Attributes_ in DynamoDB are fields or properties that you store inside an _item_. In the template we defined the `id` attribute of type string. 39 | 40 | _Items_ are identified uniquely by their _keys_ - which are defined in the __KeySchema__ section. Key `id` is of type __HASH__ which is also referred to as the __Partition Key__. I won’t dive into types of keys at this time. For now, you need to know that Partition Key a.k.a. HASH Key uniquely identifies item in the DynamoDB table. 41 | 42 | ### Why `price` and `is_shipped` attributes aren’t defined? 43 | In DynamoDB we need to define only the attributes that are part of the key. This is NoSQL world and we don’t need to specify each and every attribute of the item. 44 | 45 | ## [Let’s see some code already!](#lets-see-code) 46 | There you go. This will be our order definition. Notice `dynamodbav` struct tag which specifies how to serialize a given struct field. By the way __av__ in dynamodbav stands for attribute value. 47 | 48 | ```go 49 | type Order struct { 50 | ID string `dynamodbav:"id"` 51 | Price int `dynamodbav:"price"` 52 | IsShipped bool `dynamodbav:"is_shipped"` 53 | } 54 | ``` 55 | Let’s start with DynamoDB connection setup: 56 | 57 | ```go 58 | func TestPutGet(t *testing.T) { 59 | ctx := context.Background() 60 | tableName := "OrdersTable" 61 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 62 | defer cleanup() 63 | ``` 64 | Note that we defer calling `cleanup` function. This method removes the table that we created calling the `SetupTable`. 65 | 66 | Now we need to prepare data before inserting it into DynamoDB. 67 | 68 | ```go 69 | order := Order{ID: "12-34", Price: 22, IsShipped: false} 70 | avs, err := attributevalue.MarshalMap(order) 71 | assert.NoError(t, err) 72 | ``` 73 | 74 | Thanks to `dynamodbav` struct tags on the `Order`, `MarshalMap` function knows how to marshal struct into structure that DynamoDB understands. We are finally ready to insert something into DB. 75 | 76 | ```go 77 | _, err = db.PutItem(ctx, &dynamodb.PutItemInput{ 78 | TableName: aws.String(tableName), 79 | Item: avs, 80 | }) 81 | assert.NoError(t, err) 82 | ``` 83 | 84 | We are using DynamoDB __PutItem__ operation which creates new item or replaces old item with the same key. First parameter is `context` which is used for cancellation. Second argument is `dynamodb.PutItemInput`. For every call to the AWS that SDK supports, you may expect this pattern: 85 | - `APICall` - function to call 86 | - `APICallInput` - argument for the function 87 | - `APICallOutput` - return value of the function 88 | 89 | One thing to notice is that `table` is wrapped in call to `aws.String` function. This is because in many places SDK accepts type `pointer to type` instead of just `type` and this wrapper makes that conversion. 90 | 91 | Notice that first return value from the SDK call is being ignored. We don't really need it here. Only thing we want to know at this point is that we didn't get any errors. 92 | 93 | ## [Get order back from DynamoDB](#get-order-back) 94 | 95 | ```go 96 | out, err := db.GetItem(ctx, &dynamodb.GetItemInput{ 97 | Key: map[string]types.AttributeValue{ 98 | "id": &types.AttributeValueMemberS{ 99 | Value: "12-34", 100 | }, 101 | }, 102 | TableName: aws.String(tableName), 103 | }) 104 | assert.NoError(t, err) 105 | ``` 106 | 107 | Many pieces here are similar. There is `APICall`, and `APICallInput` elements that match pattern I showed you before. `TableName` parameter in the input is exactly the same. 108 | Since we want to get item, we need to provide the key. This is where I find SDK cumbersome. Constructing keys looks a little bit off, but it is what it is. It is a map because key can be more complicated than what we have here. Remember how we defined `id` in the `template.yml`? It was of type "S" which is string. We need to specify that in the key as well when talking with the DynamoDB. 109 | 110 | Last steps we need to perform are deserializing whatever we got from DynamoDB, and just to be sure - comparing results with what was put in. 111 | 112 | ```go 113 | var queried Order 114 | err = attributevalue.UnmarshalMap(out.Item, &queried) 115 | assert.NoError(t, err) 116 | assert.Equal(t, Order{ID: "12-34", Price: 22, IsShipped: false}, queried) 117 | ``` 118 | 119 | ## [Summary](#summary) 120 | Let me recap what we did today: 121 | 122 | 1. we defined database layout, 123 | 2. we marshaled a struct into DynamoDB item, 124 | 3. we put an item into DynamoDB, 125 | 4. we got item out of Dynamo, 126 | 5. we unmarshaled an item back into struct. 127 | 128 | 129 | Make sure to clone the [repository](https://github.com/jbszczepaniak/dynamodb-with-go) and play with the code. Code related to this episode is in [episode2](.) directory in the repository. 130 | -------------------------------------------------------------------------------- /episode2/putget_test.go: -------------------------------------------------------------------------------- 1 | package episode2 2 | 3 | import ( 4 | "context" 5 | "dynamodb-with-go/pkg/dynamo" 6 | "testing" 7 | 8 | "github.com/aws/aws-sdk-go-v2/aws" 9 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue" 10 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 11 | "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" 12 | "github.com/stretchr/testify/assert" 13 | ) 14 | 15 | type Order struct { 16 | ID string `dynamodbav:"id"` 17 | Price int `dynamodbav:"price"` 18 | IsShipped bool `dynamodbav:"is_shipped"` 19 | } 20 | 21 | func TestPutGet(t *testing.T) { 22 | ctx := context.Background() 23 | tableName := "OrdersTable" 24 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 25 | defer cleanup() 26 | 27 | order := Order{ID: "12-34", Price: 22, IsShipped: false} 28 | avs, err := attributevalue.MarshalMap(order) 29 | assert.NoError(t, err) 30 | 31 | _, err = db.PutItem(ctx, &dynamodb.PutItemInput{ 32 | TableName: aws.String(tableName), 33 | Item: avs, 34 | }) 35 | assert.NoError(t, err) 36 | 37 | out, err := db.GetItem(ctx, &dynamodb.GetItemInput{ 38 | Key: map[string]types.AttributeValue{ 39 | "id": &types.AttributeValueMemberS{ 40 | Value: "12-34", 41 | }, 42 | }, 43 | TableName: aws.String(tableName), 44 | }) 45 | assert.NoError(t, err) 46 | 47 | var queried Order 48 | err = attributevalue.UnmarshalMap(out.Item, &queried) 49 | assert.NoError(t, err) 50 | assert.Equal(t, Order{ID: "12-34", Price: 22, IsShipped: false}, queried) 51 | 52 | } 53 | -------------------------------------------------------------------------------- /episode2/template.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: "2010-09-09" 2 | Resources: 3 | OrdersTable: 4 | Type: AWS::DynamoDB::Table 5 | Properties: 6 | AttributeDefinitions: 7 | - AttributeName: id 8 | AttributeType: S 9 | KeySchema: 10 | - AttributeName: id 11 | KeyType: HASH 12 | BillingMode: PAY_PER_REQUEST 13 | TableName: OrdersTable 14 | -------------------------------------------------------------------------------- /episode3/post.md: -------------------------------------------------------------------------------- 1 | # DynamoDB with Go #3 2 | ## Use this 4 tricks to build filesystem in DynamoDB under 15 minutes! 3 | After reading previous episode you could have an impression that DynamoDB is just simple key-value store. I would like to straighten things out because DynamoDB is much more than that. Last time I also mentioned that keys can be more complicated. Get ready to dive into the topic of keys in DynamoDB with Go #3! 4 | 5 | ## [Let's build filesystem](#lets-build-filesystem) 6 | 7 | Maybe it won't be full fledged filesystem, but I would like to model tree-like structure (width depth of 1, nested directories not allowed) where inside a directory there are many files. Moreover, I would like to query this filesystem in two ways: 8 | 1. Give me single file from given directory, 9 | 2. Give me all files from given directory. 10 | 11 | These are my __access patterns__. I want to model my table in a way that will allow me to perform such queries. 12 | 13 | ## [Composite Primary Key](#composite-primary-key) 14 | 15 | __Composite Primary Key__ consists of __Partition Key__ and __Sort Key__. Without going into details (AWS documentation covers this subject thoroughly), a pair of Partition Key and Sort Key identifies an item in the DynamoDB. Many items can have the same Partition Key, but each of them needs to have a different Sort Key. If you are looking for an item in the table and you already know what is the Partition Key, Sort Key narrows down the search to the specific item. 16 | 17 | If a table is defined only by a Partition Key; each item is recognized uniquely by its Partition Key. If, however, a table is defined with a Composite Primary Key; each item is recognized by pair of Partition and Sort keys. 18 | 19 | ## [Table definition](#table-definition) 20 | 21 | With all that theory in mind, let's figure out what should be a Partition Key and a Sort Key in our filesystem. 22 | 23 | Each item in the table will represent a single file. Additionally, each file must point to its parent directory. As I mentioned, a Sort Key kind of narrows down the search. In this example, knowing already what directory we are looking for, we want to narrow down the search to a single file. 24 | 25 | All that suggests that `directory` should be the Partition Key and `filename` the Sort Key. Let's express it as a CloudFormation template. 26 | 27 | ```yaml 28 | Resources: 29 | FileSystemTable: 30 | Type: AWS::DynamoDB::Table 31 | Properties: 32 | AttributeDefinitions: 33 | - AttributeName: directory 34 | AttributeType: S 35 | - AttributeName: filename 36 | AttributeType: S 37 | KeySchema: 38 | - AttributeName: directory 39 | KeyType: HASH 40 | - AttributeName: filename 41 | KeyType: RANGE 42 | BillingMode: PAY_PER_REQUEST 43 | TableName: FileSystemTable 44 | ``` 45 | We need to define two attributes (`directory` and `filename`), because both of them are part of the __Composite Primary Key__. As you can see there is no Sort Key in the template. There is however `RANGE` key type. Just remember that: 46 | - `HASH` key type corresponds to __Partition Key__ 47 | - `RANGE` key type corresponds to __Sort Key__ 48 | 49 | ## [Moving on to the code](#code) 50 | 51 | This is how a single item in the DynamoDB is going to look. 52 | ```go 53 | type item struct { 54 | Directory string `dynamodbav:"directory"` 55 | Filename string `dynamodbav:"filename"` 56 | Size string `dynamodbav:"size"` 57 | } 58 | ``` 59 | I am going to insert a couple of items to the database so that we have content to query. Code that is doing that is omitted for brevity, you can look it up [here](./queries_test.go). At the end I want to have following table: 60 | 61 | | Directory | Filename | Size | 62 | | --- | ---- | ---- | 63 | | finances | report2017.pdf | 1MB | 64 | | finances | report2018.pdf | 1MB | 65 | | finances | report2019.pdf | 1MB | 66 | | finances | report2020.pdf | 2MB | 67 | | fun | game1 | 4GB | 68 | 69 | ## [Query #1: Give me single file from given directory](#query1) 70 | We need to start with a database setup. 71 | ```go 72 | func TestSingleFileFromDirectory(t *testing.T) { 73 | ctx := context.Background() 74 | tableName := "FileSystemTable" 75 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 76 | defer cleanup() 77 | insert(ctx, db, tableName) 78 | ``` 79 | With a connection to DynamoDB in place and with the testing data inserted, we can move on to the query itself. I want to obtain a single element from the DynamoDB, thus I am going to use `GetItem`. 80 | ```go 81 | out, err := db.GetItem(ctx, &dynamodb.GetItemInput{ 82 | Key: map[string]types.AttributeValue{ 83 | "directory": &types.AttributeValueMemberS{Value: "finances"}, 84 | "filename": &types.AttributeValueMemberS{Value: "report2020.pdf"}, 85 | }, 86 | TableName: aws.String(tableName), 87 | }) 88 | ``` 89 | Note that `Key` consists of two elements: `directory` (Partition Key) and `filename` (Sort Key). Let's make sure that output of the query is really what we think it is: 90 | ```go 91 | var i item 92 | err = attributevalue.UnmarshalMap(out.Item, &i) 93 | assert.NoError(t, err) 94 | assert.Equal(t, item{Directory: "finances", Filename: "report2020.pdf", Size: "2MB"}, i) 95 | ``` 96 | ## [Query #2: Give me whole directory](#query2) 97 | 98 | In this query we cannot use `GetItem` because we want to obtain many items from the DynamoDB. Also when we get a single item we need to know whole composite primary key. Here we want to get all the files from the directory so we know only the Partition Key. Solution to that problem is `Query` method with __Key Condition Expression__. 99 | ```go 100 | expr, err := expression.NewBuilder(). 101 | WithKeyCondition( 102 | expression.KeyEqual(expression.Key("directory"), expression.Value("finances"))). 103 | Build() 104 | assert.NoError(t, err) 105 | 106 | out, err := db.Query(ctx, &dynamodb.QueryInput{ 107 | ExpressionAttributeNames: expr.Names(), 108 | ExpressionAttributeValues: expr.Values(), 109 | KeyConditionExpression: expr.KeyCondition(), 110 | TableName: aws.String(tableName), 111 | }) 112 | assert.NoError(t, err) 113 | assert.Len(t, out.Items, 4) 114 | ``` 115 | 116 | That is a lot of new things, so let me break it down for you. First part is where we construct key condition expression which describes what we really want to query. In our case this is just _"Give me all items whose directory attribute is equal to `finances`"_. I am using an expression builder which simplifies construction of expressions by far. 117 | 118 | In the next step we are using expression inside the query. We need to provide __condition__, __names__, and __values__. In this example, condition is just an equality comparison, where names correspond to names of attributes and values correspond to... their values! 119 | 120 | An expression object gives us easy access to condition, names, and values. As you can see I am using them as parameters to `QueryInput`. 121 | 122 | At the end, I am just checking whether we really have 4 items which are in finances directory. 123 | 124 | ## [Bonus - Query #3 Give me reports before 2019](#query3) 125 | 126 | It turns out that I constructed filenames in a way that makes them sortable. I figured - let's try to use it to our advantage and get all the reports created before 2019. 127 | 128 | Query stays exactly the same. The only thing we need to change is the key condition expression. 129 | 130 | ```go 131 | expr, err := expression.NewBuilder(). 132 | WithKeyCondition( 133 | expression.KeyAnd( 134 | expression.KeyEqual(expression.Key("directory"), expression.Value("finances")), 135 | expression.KeyLessThan(expression.Key("filename"), expression.Value("report2019")))). 136 | Build() 137 | assert.NoError(t, err) 138 | ``` 139 | 140 | We have two conditions that we combine with the AND clause. The first one specifies what is our Partition Key, second one, the Sort Key. `KeyLessThan` makes sure that we will only get `report2018.pdf` and `report2017.pdf`. Let's have a look at the results of the query. 141 | 142 | ```go 143 | var items []item 144 | err = attributevalue.UnmarshalListOfMaps(out.Items, &items) 145 | assert.NoError(t, err) 146 | if assert.Len(t, items, 2) { 147 | assert.Equal(t, "report2017.pdf", items[0].Filename) 148 | assert.Equal(t, "report2018.pdf", items[1].Filename) 149 | } 150 | ``` 151 | 152 | In the first query we used `attributevalue.UnmarshalMap` for unmarshaling single DynamoDB item into the struct. We knew we will get single item. Here we know that there will be one item or more - thus we use `attributevalue.UnmarshalListOfMaps` - which unmarshals the query results into the slice of items. 153 | 154 | Note that I assert that first item is the report from 2017 and second one is from 2018. How am I so sure that items will go back from the DynamoDB in that order? If not told otherwise - DynamoDB will scan items from given Partition in ascending order. Since 2017 comes before 2018 - I know that first item should be from 2017. 155 | 156 | ## [Summary](#summary) 157 | We learned today how to use composite primary keys. Moreover we know how to take advantage of them with Go! That's great! You know what is even better? Playing with the code! Make sure to clone this repository and tinker with it! 158 | 159 | Also we used expression builder to create the DynamoDB expression. Get used to them - we will use them a lot in the future episode! It takes some time to build intuition around using expression builder API, but it's totally worth it! 160 | -------------------------------------------------------------------------------- /episode3/queries_test.go: -------------------------------------------------------------------------------- 1 | package episode3 2 | 3 | import ( 4 | "context" 5 | "dynamodb-with-go/pkg/dynamo" 6 | "testing" 7 | 8 | "github.com/aws/aws-sdk-go-v2/aws" 9 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue" 10 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression" 11 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 12 | "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" 13 | "github.com/stretchr/testify/assert" 14 | ) 15 | 16 | func TestSingleFileFromDirectory(t *testing.T) { 17 | ctx := context.Background() 18 | tableName := "FileSystemTable" 19 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 20 | defer cleanup() 21 | 22 | insert(ctx, db, tableName) 23 | 24 | out, err := db.GetItem(ctx, &dynamodb.GetItemInput{ 25 | Key: map[string]types.AttributeValue{ 26 | "directory": &types.AttributeValueMemberS{Value: "finances"}, 27 | "filename": &types.AttributeValueMemberS{Value: "report2020.pdf"}, 28 | }, 29 | TableName: aws.String(tableName), 30 | }) 31 | assert.NoError(t, err) 32 | 33 | var i item 34 | err = attributevalue.UnmarshalMap(out.Item, &i) 35 | assert.NoError(t, err) 36 | assert.Equal(t, item{Directory: "finances", Filename: "report2020.pdf", Size: "2MB"}, i) 37 | } 38 | 39 | func TestAllFilesFromDirectory(t *testing.T) { 40 | ctx := context.Background() 41 | tableName := "FileSystemTable" 42 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 43 | defer cleanup() 44 | 45 | insert(ctx, db, tableName) 46 | 47 | expr, err := expression.NewBuilder(). 48 | WithKeyCondition( 49 | expression.KeyEqual(expression.Key("directory"), expression.Value("finances"))). 50 | Build() 51 | assert.NoError(t, err) 52 | 53 | out, err := db.Query(ctx, &dynamodb.QueryInput{ 54 | ExpressionAttributeNames: expr.Names(), 55 | ExpressionAttributeValues: expr.Values(), 56 | KeyConditionExpression: expr.KeyCondition(), 57 | TableName: aws.String(tableName), 58 | }) 59 | assert.NoError(t, err) 60 | assert.Len(t, out.Items, 4) 61 | } 62 | 63 | func TestAllReportsBefore2019(t *testing.T) { 64 | ctx := context.Background() 65 | tableName := "FileSystemTable" 66 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 67 | defer cleanup() 68 | 69 | insert(ctx, db, tableName) 70 | 71 | expr, err := expression.NewBuilder(). 72 | WithKeyCondition( 73 | expression.KeyAnd( 74 | expression.KeyEqual(expression.Key("directory"), expression.Value("finances")), 75 | expression.KeyLessThan(expression.Key("filename"), expression.Value("report2019")))). 76 | Build() 77 | assert.NoError(t, err) 78 | 79 | out, err := db.Query(ctx, &dynamodb.QueryInput{ 80 | ExpressionAttributeNames: expr.Names(), 81 | ExpressionAttributeValues: expr.Values(), 82 | KeyConditionExpression: expr.KeyCondition(), 83 | TableName: aws.String(tableName), 84 | }) 85 | assert.NoError(t, err) 86 | var items []item 87 | err = attributevalue.UnmarshalListOfMaps(out.Items, &items) 88 | assert.NoError(t, err) 89 | if assert.Len(t, items, 2) { 90 | assert.Equal(t, "report2017.pdf", items[0].Filename) 91 | assert.Equal(t, "report2018.pdf", items[1].Filename) 92 | } 93 | } 94 | 95 | type item struct { 96 | Directory string `dynamodbav:"directory"` 97 | Filename string `dynamodbav:"filename"` 98 | Size string `dynamodbav:"size"` 99 | } 100 | 101 | func insert(ctx context.Context, db *dynamodb.Client, tableName string) { 102 | item1 := item{Directory: "finances", Filename: "report2017.pdf", Size: "1MB"} 103 | item2 := item{Directory: "finances", Filename: "report2018.pdf", Size: "1MB"} 104 | item3 := item{Directory: "finances", Filename: "report2019.pdf", Size: "1MB"} 105 | item4 := item{Directory: "finances", Filename: "report2020.pdf", Size: "2MB"} 106 | item5 := item{Directory: "fun", Filename: "game1", Size: "4GB"} 107 | 108 | for _, item := range []item{item1, item2, item3, item4, item5} { 109 | attrs, _ := attributevalue.MarshalMap(&item) 110 | db.PutItem(ctx, &dynamodb.PutItemInput{ 111 | TableName: aws.String(tableName), 112 | Item: attrs, 113 | }) 114 | } 115 | } 116 | -------------------------------------------------------------------------------- /episode3/template.yml: -------------------------------------------------------------------------------- 1 | Resources: 2 | FileSystemTable: 3 | Type: AWS::DynamoDB::Table 4 | Properties: 5 | AttributeDefinitions: 6 | - AttributeName: directory 7 | AttributeType: S 8 | - AttributeName: filename 9 | AttributeType: S 10 | KeySchema: 11 | - AttributeName: directory 12 | KeyType: HASH 13 | - AttributeName: filename 14 | KeyType: RANGE 15 | BillingMode: PAY_PER_REQUEST 16 | TableName: FileSystemTable 17 | -------------------------------------------------------------------------------- /episode4/gsi.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jbszczepaniak/dynamodb-with-go/105b29d02cde311361e77ca1f750bf174e7ea118/episode4/gsi.png -------------------------------------------------------------------------------- /episode4/lsi.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jbszczepaniak/dynamodb-with-go/105b29d02cde311361e77ca1f750bf174e7ea118/episode4/lsi.png -------------------------------------------------------------------------------- /episode4/post.md: -------------------------------------------------------------------------------- 1 | # DynamoDB with Go #4 2 | 3 | Last time we looked at how Composite Primary Keys can improve our ability to query the DynamoDB. This time we'll focus on how indices can help us even more with our access patterns. 4 | 5 | In episode #3 of DynamoDB we built an oversimplified filesystem model. Let me share with you a single item from that filesystem so that we are on the same page on how it looked. 6 | 7 | | Directory | Filename | Size | 8 | | --- | ---- | ---- | 9 | | finances | report2017.pdf | 1MB | 10 | 11 | If you recall we were able to query the filesystem in a following ways: 12 | - Give me a single file from a directory 13 | - Give me the whole directory 14 | 15 | Thanks to the fact that some of the files have year in their names we are able to get files older/younger than given date. [Look here how we did it](../episode3/post.md#query3). 16 | 17 | As we are testing this on staging, product person looks up our shoulder and sees that we are querying based on the creation time and she absolutely loves it. Except for one thing. 18 | 19 | ## You cannot really expect from our clients to put creation date of the file in the name 20 | 21 | Now we have additional requirement. We need to be able to sort files inside directory based on their creation date. This is new **access pattern**. 22 | 23 | This isn't SQL, we cannot simply use `WHERE` and `ORDER BY` clauses and query just anything without additional work. The tool for this job in DynamoDB is an **index**. 24 | 25 | ## [Indices](#indices) 26 | 27 | Table in the DynamoDB can have only single pair of main Partition and Sort Keys. It's called Primary Key. They are important because they are entry point for queries. 28 | 29 | But why are they the entry point? In order to answer that question let's think of how DynamoDB stores its data. First of all there is a partition key which is used to determine _on which partition data is stored._ The DynamoDB calculates hash function based on partition key and finds that partition. All items that share the same partition key (so called item collection) are stored within single B-Tree data structure. What is important about B-Tree is that it allows to query data in sorted order. Sort Key is important here because it is the criteria by which items are sorted in the tree. Let's have a look. 30 | 31 | ![table](./table.png) 32 | 33 | I used Binary Search Tree instead of B-Tree to simplify but they share the same property of being sortable and this is enough to build your intuition around the topic. 34 | 35 | Data that corresponds to image above is presented here: 36 | 37 | | Directory (PK) | Filename (SK) | Size | CreatedAt | 38 | | --- | ---- | ---- | --------- | 39 | | photos | bike.png | 1.2MB | 2017-03-04 00:00:00 +0000 UTC | 40 | | photos | apartment.jpg | 4MB | 2018-06-25 00:00:00 +0000 UTC | 41 | | photos | grandpa.png | 3MB | 2019-04-01 00:00:00 +0000 UTC | 42 | | photos | kids.png | 3MB | 2020-01-10 00:00:00 +0000 UTC | 43 | 44 | Our entry point here is the ability to get items from `photos` directory and sort them by filename. This is exactly reflected in the image because tree is constructed in a way that allow to sort only by `filename`. If we would like to have different entry point - we need to introduce an index. 45 | 46 | **Indices enhance ability to query the table by introducing additional Partition Keys and Sort Keys.** 47 | 48 | DynamoDB has two types of indices: Local Secondary Indices (LSI) and Global Secondary Indices (GSI). *Secondary* means that they are an addition to Primary Key. 49 | 50 | ### [Local Secondary Index](#LSI) 51 | LSI has the same Partition Key as Primary Key but different Sort Key. This allows us to sort items by additional attribute. In order to do that DynamoDB has to store additional - reorganized tree. Each sorting pattern needs to have it's own tree. 52 | 53 | Our additional access pattern was sorting by `created_at` attribute. This means additional tree sorted by `created_at`. Let's have a look at the image to warp our heads around that. 54 | 55 | ![lsi](./lsi.png) 56 | 57 | LSI needs to have the same Partition Key as the table itself - this is why it's *local*. We can use the same partition to find both Primary Sort Key and Sort Key for the LSI. Important limitation of LSI is that it needs to be created when the table is created. There can be up to 5 LSIs on single table. 58 | 59 | ### [Global Secondary Index](#GSI) 60 | GSI is different. It allows to have arbitrary Partition Key and Sort Key for the same table. Let's construct an index where Partition Key is `size` and `filename` is Sort Key. This would allow to do query that would group photos by their size. 61 | 62 | ![gsi](./gsi.png) 63 | 64 | We no longer have the possibility to keep an index close to Primary Key because Partition Key on the index is completely different. We need to construct a new data structure. That's why this index is global - it lives somewhere else so to speak. GSIs are important because they allow to query the table in just any way you imagine. As opposed to LSIs, GSIs can be created anytime in the table lifetime. There can be up to 20 GSIs per table. 65 | 66 | ## [Database layout](#database-layout) 67 | 68 | We will concentrate on solving our business problem and we will create LSI that allows to sort by `created_at`. 69 | 70 | ```yaml 71 | Resources: 72 | FileSystemTable: 73 | Type: AWS::DynamoDB::Table 74 | Properties: 75 | AttributeDefinitions: 76 | - AttributeName: directory 77 | AttributeType: S 78 | - AttributeName: filename 79 | AttributeType: S 80 | - AttributeName: created_at 81 | AttributeType: S 82 | KeySchema: 83 | - AttributeName: directory 84 | KeyType: HASH 85 | - AttributeName: filename 86 | KeyType: RANGE 87 | LocalSecondaryIndexes: 88 | - IndexName: ByCreatedAt 89 | KeySchema: 90 | - AttributeName: directory 91 | KeyType: HASH 92 | - AttributeName: created_at 93 | KeyType: RANGE 94 | Projection: 95 | ProjectionType: ALL 96 | BillingMode: PAY_PER_REQUEST 97 | TableName: FileSystemTable 98 | ``` 99 | 100 | This is very similar to what we had before in the third epiode, there are only two differences. First difference is that we have new big section `LocalSecondaryIndexes` where an index lives. I called it `ByCreatedAt` because this is essentially what I want to do - I want to query by the creation time. If we look inside we will see something very similar to the Composite Primary Key definition. There is `KeySchema` with `RANGE` and `HASH` keys. Second difference is that there is new attribute definition. Because `RANGE` key of the `CreatedAt` index uses `created_at` attribute it has to be defined. 101 | 102 | ```go 103 | type item struct { 104 | Directory string `dynamodbav:"directory"` 105 | Filename string `dynamodbav:"filename"` 106 | Size string `dynamodbav:"size"` 107 | CreatedAt time.Time `dynamodbav:"created_at"` 108 | } 109 | ``` 110 | 111 | The `item` changed as well. It has new attribute - `CreatedAt`. 112 | 113 | ## [Query #1 - Photos taken from 2019](#query1) 114 | 115 | Initial setup for all queries will be the same. It's just: 116 | 117 | ```go 118 | ctx := context.Background() 119 | tableName := "FileSystemTable" 120 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 121 | defer cleanup() 122 | ``` 123 | 124 | Now let's build key condition expression. 125 | 126 | ```go 127 | expr, err := expression.NewBuilder(). 128 | WithKeyCondition( 129 | expression.KeyAnd( 130 | expression.KeyEqual(expression.Key("directory"), expression.Value("photos")), 131 | expression.KeyGreaterThanEqual(expression.Key("created_at"), expression.Value(time.Date(2019, 1, 1, 0, 0, 0, 0, time.UTC))))). 132 | Build() 133 | ``` 134 | 135 | I need to provide the exact directory for the query. Additionally I am using `KeyGreaterThanEqual` in order to take advantage of the index. From perspective of this expression however the index is not visible. Hypothetically with table where `directory` is the Partition Key and `created_at` is the Sort Key it would be valid expression as well. The point here is that we cannot have many sort keys on the table - we need to use indices for that - but it kind of works as additional sort key and it doesn't affect expression very much. 136 | 137 | ```go 138 | out, err := db.Query(ctx, &dynamodb.QueryInput{ 139 | ExpressionAttributeNames: expr.Names(), 140 | ExpressionAttributeValues: expr.Values(), 141 | KeyConditionExpression: expr.KeyCondition(), 142 | TableName: aws.String(table), 143 | IndexName: aws.String("ByCreatedAt"), 144 | }) 145 | assert.Len(t, out.Items, 2) 146 | ``` 147 | 148 | As you can see we need to specify that we are using the `ByCreatedAt` index. If we don't specify that - the DynamoDB will complain that key condition expression is missing its sort key (`filename`) and has other field instead (`created_at`). At the end I am just checking if result consists of 2 items. 149 | 150 | ## [Query #2 - Photos taken from 2017 to 2018](#query2) 151 | 152 | The query looks exactly the same as in first query (hence I'im skipping it), but the Key Condition Expression used to construct this query uses new operator which I wanted to show you. 153 | 154 | ```go 155 | expr, err := expression.NewBuilder(). 156 | WithKeyCondition( 157 | expression.KeyAnd( 158 | expression.KeyEqual(expression.Key("directory"), expression.Value("photos")), 159 | expression.KeyBetween(expression.Key("created_at"), 160 | expression.Value(time.Date(2017, 1, 1, 0, 0, 0, 0, time.UTC)), 161 | expression.Value(time.Date(2019, 1, 1, 0, 0, 0, 0, time.UTC))))). 162 | Build() 163 | ``` 164 | It's that simple - you just need to set lower and upper bounds for your search and that's all - you get photos from 2017 to 2018. 165 | 166 | ## [Query #3 - Newest photo](#query3) 167 | 168 | Because an index stores items in the order - sorted by its Sort Key (of the index) - we can access data in a new way. We need to have the Key Condition Expression that will specify only the Partition Key - so that we are in photos directory. 169 | 170 | ```go 171 | expr, err := expression.NewBuilder(). 172 | WithKeyCondition(expression.KeyEqual(expression.Key("directory"), expression.Value("photos"))). 173 | Build() 174 | ``` 175 | 176 | As the Key Condition Expression got simpler, the query itself got a little bit more complicated. 177 | 178 | ```go 179 | out, err := db.Query(ctx, &dynamodb.QueryInput{ 180 | ExpressionAttributeNames: expr.Names(), 181 | ExpressionAttributeValues: expr.Values(), 182 | KeyConditionExpression: expr.KeyCondition(), 183 | TableName: aws.String(table), 184 | IndexName: aws.String("ByCreatedAt"), 185 | ScanIndexForward: aws.Bool(false), 186 | Limit: aws.Int32(1), 187 | }) 188 | ``` 189 | 190 | First of all we need to specify that we wan't to traverse index backwards because by default index keeps data in ascending order - we do it with the `ScanIndexForward` parameter. Second of all we want to limit query results to single item. Without that bit - we would return all of the photos from newest to oldest one. 191 | 192 | At the end we just verify whether item we obtained is really the newest one. 193 | 194 | ```go 195 | var items []item 196 | err = attributevalue.UnmarshalListOfMaps(out.Items, &items) 197 | assert.Equal(t, 2020, items[0].CreatedAt.Year()) 198 | ``` 199 | 200 | ## [Summary](#summary) 201 | This episode was about indices. First we built an intuition on how they work and then we used Local Secondary Index to query by additional attribute. 202 | 203 | Indices are esessential concept in the DynamoDB and we will see more of them when working with more complicated data models. 204 | 205 | Once again - I am inviting you to clone this repository and play with queries by yourself! -------------------------------------------------------------------------------- /episode4/queries_test.go: -------------------------------------------------------------------------------- 1 | package episode4 2 | 3 | import ( 4 | "context" 5 | "dynamodb-with-go/pkg/dynamo" 6 | "testing" 7 | "time" 8 | 9 | "github.com/aws/aws-sdk-go-v2/aws" 10 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue" 11 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression" 12 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 13 | "github.com/stretchr/testify/assert" 14 | ) 15 | 16 | func TestPhotosYoungerThan(t *testing.T) { 17 | ctx := context.Background() 18 | tableName := "FileSystemTable" 19 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 20 | defer cleanup() 21 | 22 | insert(ctx, db, tableName) 23 | 24 | expr, err := expression.NewBuilder(). 25 | WithKeyCondition( 26 | expression.KeyAnd( 27 | expression.KeyEqual(expression.Key("directory"), expression.Value("photos")), 28 | expression.KeyGreaterThanEqual( 29 | expression.Key("created_at"), 30 | expression.Value(time.Date(2019, 1, 1, 0, 0, 0, 0, time.UTC))))). 31 | Build() 32 | assert.NoError(t, err) 33 | 34 | out, err := db.Query(ctx, &dynamodb.QueryInput{ 35 | ExpressionAttributeNames: expr.Names(), 36 | ExpressionAttributeValues: expr.Values(), 37 | KeyConditionExpression: expr.KeyCondition(), 38 | TableName: aws.String(tableName), 39 | IndexName: aws.String("ByCreatedAt"), 40 | }) 41 | assert.NoError(t, err) 42 | assert.Len(t, out.Items, 2) 43 | } 44 | 45 | func TestPhotosFromTimeRange(t *testing.T) { 46 | ctx := context.Background() 47 | tableName := "FileSystemTable" 48 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 49 | defer cleanup() 50 | 51 | insert(ctx, db, tableName) 52 | 53 | expr, err := expression.NewBuilder(). 54 | WithKeyCondition( 55 | expression.KeyAnd( 56 | expression.KeyEqual(expression.Key("directory"), expression.Value("photos")), 57 | expression.KeyBetween(expression.Key("created_at"), 58 | expression.Value(time.Date(2017, 1, 1, 0, 0, 0, 0, time.UTC)), 59 | expression.Value(time.Date(2019, 1, 1, 0, 0, 0, 0, time.UTC))))). 60 | Build() 61 | assert.NoError(t, err) 62 | 63 | out, err := db.Query(ctx, &dynamodb.QueryInput{ 64 | ExpressionAttributeNames: expr.Names(), 65 | ExpressionAttributeValues: expr.Values(), 66 | KeyConditionExpression: expr.KeyCondition(), 67 | TableName: aws.String(tableName), 68 | IndexName: aws.String("ByCreatedAt"), 69 | }) 70 | assert.NoError(t, err) 71 | assert.Len(t, out.Items, 2) 72 | } 73 | 74 | func TestNewestPhoto(t *testing.T) { 75 | ctx := context.Background() 76 | tableName := "FileSystemTable" 77 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 78 | defer cleanup() 79 | 80 | insert(ctx, db, tableName) 81 | 82 | expr, err := expression.NewBuilder(). 83 | WithKeyCondition(expression.KeyEqual(expression.Key("directory"), expression.Value("photos"))). 84 | Build() 85 | assert.NoError(t, err) 86 | 87 | out, err := db.Query(ctx, &dynamodb.QueryInput{ 88 | ExpressionAttributeNames: expr.Names(), 89 | ExpressionAttributeValues: expr.Values(), 90 | KeyConditionExpression: expr.KeyCondition(), 91 | TableName: aws.String(tableName), 92 | IndexName: aws.String("ByCreatedAt"), 93 | ScanIndexForward: aws.Bool(false), 94 | Limit: aws.Int32(1), 95 | }) 96 | assert.NoError(t, err) 97 | 98 | var items []item 99 | err = attributevalue.UnmarshalListOfMaps(out.Items, &items) 100 | assert.NoError(t, err) 101 | 102 | assert.Equal(t, 2020, items[0].CreatedAt.Year()) 103 | } 104 | 105 | type item struct { 106 | Directory string `dynamodbav:"directory"` 107 | Filename string `dynamodbav:"filename"` 108 | Size string `dynamodbav:"size"` 109 | CreatedAt time.Time `dynamodbav:"created_at"` 110 | } 111 | 112 | func insert(ctx context.Context, db *dynamodb.Client, tableName string) { 113 | item1 := item{Directory: "photos", Filename: "bike.png", Size: "1.2MB", CreatedAt: time.Date(2017, 3, 4, 0, 0, 0, 0, time.UTC)} 114 | item2 := item{Directory: "photos", Filename: "apartment.jpg", Size: "4MB", CreatedAt: time.Date(2018, 6, 25, 0, 0, 0, 0, time.UTC)} 115 | item3 := item{Directory: "photos", Filename: "grandpa.png", Size: "3MB", CreatedAt: time.Date(2019, 4, 1, 0, 0, 0, 0, time.UTC)} 116 | item4 := item{Directory: "photos", Filename: "kids.png", Size: "3MB", CreatedAt: time.Date(2020, 1, 10, 0, 0, 0, 0, time.UTC)} 117 | 118 | for _, item := range []item{item1, item2, item3, item4} { 119 | attrs, _ := attributevalue.MarshalMap(&item) 120 | db.PutItem(ctx, &dynamodb.PutItemInput{ 121 | TableName: aws.String(tableName), 122 | Item: attrs, 123 | }) 124 | } 125 | } 126 | -------------------------------------------------------------------------------- /episode4/table.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jbszczepaniak/dynamodb-with-go/105b29d02cde311361e77ca1f750bf174e7ea118/episode4/table.png -------------------------------------------------------------------------------- /episode4/template.yml: -------------------------------------------------------------------------------- 1 | Resources: 2 | FileSystemTable: 3 | Type: AWS::DynamoDB::Table 4 | Properties: 5 | AttributeDefinitions: 6 | - AttributeName: directory 7 | AttributeType: S 8 | - AttributeName: filename 9 | AttributeType: S 10 | - AttributeName: created_at 11 | AttributeType: S 12 | KeySchema: 13 | - AttributeName: directory 14 | KeyType: HASH 15 | - AttributeName: filename 16 | KeyType: RANGE 17 | LocalSecondaryIndexes: 18 | - IndexName: ByCreatedAt 19 | KeySchema: 20 | - AttributeName: directory 21 | KeyType: HASH 22 | - AttributeName: created_at 23 | KeyType: RANGE 24 | Projection: 25 | ProjectionType: ALL 26 | BillingMode: PAY_PER_REQUEST 27 | TableName: FileSystemTable 28 | -------------------------------------------------------------------------------- /episode5/mapper.go: -------------------------------------------------------------------------------- 1 | package episode5 2 | 3 | import ( 4 | "context" 5 | "errors" 6 | 7 | "github.com/aws/aws-sdk-go-v2/aws" 8 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue" 9 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression" 10 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 11 | "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" 12 | "github.com/google/uuid" 13 | ) 14 | 15 | // Mapper keeps Dynamo dependency. 16 | type Mapper struct { 17 | db *dynamodb.Client 18 | table string 19 | } 20 | 21 | // NewMapper creates instance of Mapper. 22 | func NewMapper(client *dynamodb.Client, table string) *Mapper { 23 | return &Mapper{db: client, table: table} 24 | } 25 | 26 | type mapping struct { 27 | OldID string `dynamodbav:"old_id"` 28 | NewID string `dynamodbav:"new_id"` 29 | } 30 | 31 | // Map generates new ID for old ID or retrieves already created new ID. 32 | func (m *Mapper) Map(ctx context.Context, old string) (string, error) { 33 | idsMapping := mapping{OldID: old, NewID: uuid.New().String()} 34 | attrs, err := attributevalue.MarshalMap(&idsMapping) 35 | if err != nil { 36 | return "", err 37 | } 38 | 39 | expr, err := expression.NewBuilder(). 40 | WithCondition(expression.AttributeNotExists(expression.Name("old_id"))). 41 | Build() 42 | if err != nil { 43 | return "", err 44 | } 45 | 46 | _, err = m.db.PutItem(ctx, &dynamodb.PutItemInput{ 47 | ConditionExpression: expr.Condition(), 48 | ExpressionAttributeNames: expr.Names(), 49 | ExpressionAttributeValues: expr.Values(), 50 | Item: attrs, 51 | TableName: aws.String(m.table), 52 | }) 53 | if err == nil { 54 | return idsMapping.NewID, nil 55 | } 56 | var conditionErr *types.ConditionalCheckFailedException 57 | if !errors.As(err, &conditionErr) { 58 | return "", err 59 | } 60 | 61 | out, err := m.db.GetItem(ctx, &dynamodb.GetItemInput{ 62 | Key: map[string]types.AttributeValue{ 63 | "old_id": &types.AttributeValueMemberS{Value: old}, 64 | }, 65 | TableName: aws.String(m.table), 66 | }) 67 | if err != nil { 68 | return "", err 69 | } 70 | 71 | var ret mapping 72 | attributevalue.UnmarshalMap(out.Item, &ret) 73 | return ret.NewID, nil 74 | } 75 | -------------------------------------------------------------------------------- /episode5/mapper_test.go: -------------------------------------------------------------------------------- 1 | package episode5 2 | 3 | import ( 4 | "context" 5 | "dynamodb-with-go/pkg/dynamo" 6 | "testing" 7 | 8 | "github.com/stretchr/testify/assert" 9 | ) 10 | 11 | func TestMapping(t *testing.T) { 12 | t.Run("generate new ID for each legacy ID", func(t *testing.T) { 13 | ctx := context.Background() 14 | tableName := "LegacyIDsTable" 15 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 16 | defer cleanup() 17 | 18 | mapper := NewMapper(db, tableName) 19 | 20 | first, err := mapper.Map(ctx, "123") 21 | assert.NoError(t, err) 22 | assert.NotEmpty(t, first) 23 | assert.NotEqual(t, "123", first) 24 | 25 | second, err := mapper.Map(ctx, "456") 26 | assert.NoError(t, err) 27 | assert.NotEmpty(t, second) 28 | assert.NotEqual(t, "456", second) 29 | 30 | assert.NotEqual(t, first, second) 31 | }) 32 | 33 | t.Run("do not regenerate ID for the same legacy ID", func(t *testing.T) { 34 | ctx := context.Background() 35 | tableName := "LegacyIDsTable" 36 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 37 | defer cleanup() 38 | 39 | mapper := NewMapper(db, tableName) 40 | 41 | first, err := mapper.Map(ctx, "123") 42 | assert.NoError(t, err) 43 | 44 | second, err := mapper.Map(ctx, "123") 45 | assert.NoError(t, err) 46 | 47 | assert.Equal(t, first, second) 48 | }) 49 | 50 | } 51 | -------------------------------------------------------------------------------- /episode5/post.md: -------------------------------------------------------------------------------- 1 | # DynamoDB with Go #5 2 | 3 | Imagine that you are developing a brand new software that is based on information from the legacy system. The way you integrate with the legacy is through events that you are listening to. Let's say that you receive information about orders from an event. Each order is identifiable via unique id given by the legacy system that is an incrementally increased integer. One of the requirements for you is to process orders in a way that won't reveal total amount of orders. 4 | 5 | It seems that the problem to solve is to map the old id to the new value. One thing to remember here is that we can receive many events about the same order which means that we need to either generate new id or use the id that we already generated for the legacy id. How can we do it with the DynamoDB? 6 | 7 | ## [Table definition](#table-definition) 8 | 9 | ```yaml 10 | Resources: 11 | LegacyIDsTable: 12 | Type: AWS::DynamoDB::Table 13 | Properties: 14 | AttributeDefinitions: 15 | - AttributeName: old_id 16 | AttributeType: S 17 | KeySchema: 18 | - AttributeName: old_id 19 | KeyType: HASH 20 | Projection: 21 | ProjectionType: ALL 22 | BillingMode: PAY_PER_REQUEST 23 | TableName: LegacyIDsTable 24 | ``` 25 | 26 | Very straightforward. Just a single attribute that is also partition key. 27 | 28 | ## [Failing test first](#failing-test-first) 29 | 30 | Let's go here with the TDD approach. I will write a failing test first, then I will try to make it pass! 31 | 32 | ```go 33 | func TestMapping(t *testing.T) { 34 | t.Run("generate new ID for each legacy ID", func(t *testing.T) { 35 | ctx := context.Background() 36 | tableName := "LegacyIDsTable" 37 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 38 | defer cleanup() 39 | 40 | mapper := NewMapper(db, tableName) 41 | 42 | first, err := mapper.Map(ctx, "123") 43 | assert.NoError(t, err) 44 | assert.NotEmpty(t, first) 45 | assert.NotEqual(t, "123", first) 46 | 47 | second, err := mapper.Map(ctx, "456") 48 | assert.NoError(t, err) 49 | assert.NotEmpty(t, second) 50 | assert.NotEqual(t, "456", first) 51 | 52 | assert.NotEqual(t, first, second) 53 | }) 54 | 55 | t.Run("do not regenerate ID for the same legacy ID", func(t *testing.T) { 56 | ctx := context.Background() 57 | tableName := "LegacyIDsTable" 58 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 59 | defer cleanup() 60 | 61 | mapper := NewMapper(db, tableName) 62 | 63 | first, err := mapper.Map(ctx, "123") 64 | assert.NoError(t, err) 65 | 66 | second, err := mapper.Map(ctx, "123") 67 | assert.NoError(t, err) 68 | 69 | assert.Equal(t, first, second) 70 | }) 71 | 72 | } 73 | ``` 74 | 75 | As you can see I have two requirements I want to cover. First of all I need to generate an id for each incoming legacy id that enters my function. Second of all, if I invoke that function twice with the same legacy id it shouldn't regenerate new id. 76 | 77 | ## [Structs setup](#structs-setup) 78 | 79 | Let's take care of basic structs. 80 | 81 | ```go 82 | type Mapper struct { 83 | db dynamodbiface.DynamoDBAPI 84 | table string 85 | } 86 | 87 | type mapping struct { 88 | OldID string `dynamodbav:"old_id"` 89 | NewID string `dynamodbav:"new_id"` 90 | } 91 | ``` 92 | 93 | There is `Mapper` that holds dependencies to the DynamoDB and the `mapping` that will be an item we store in the DynamoDB. Moreover external packages need to create the `Mapper`. 94 | 95 | ```go 96 | func NewMapper(client *dynamodb.Client, table string) *Mapper { 97 | return &Mapper{db: client, table: table} 98 | } 99 | ``` 100 | 101 | ## [Solution](#solution) 102 | 103 | Now let's think of the logic of the `Map` function. If the mapping of an old id and new id already exists in Dynamo - we need to fetch it. If it doesn't, we need to generate new id and save it. At first glance we could just use the `GetItem` and if we get nothing we just use thr `PutItem` to save newly generated id. Unfortunately this approach won't work. Remember that we can receive many events about the same order. They can arrive any time and can be handled concurrently. If two threads of execution will run the `GetItem` in more or less the same time and both will figure out that there is no mapping yet, we will end up with one of the thread overriding the other threads mapping. 104 | 105 | Let's see how can we implement that functionality that will work in the world of concurrent execution. 106 | 107 | ```go 108 | func (m *Mapper) Map(ctx context.Context, old string) (string, error) { 109 | idsMapping := mapping{OldID: old, NewID: uuid.New().String()} 110 | attrs, err := attributevalue.MarshalMap(&idsMapping) 111 | ``` 112 | 113 | At the beginning we create a mapping that has an old id and that generates new id using UUIDv4. Next thing we'll do isn't retrieving an item from Dynamo. Instead we will revert the logic. I want translate into the code the following sentence: __Put into the DynamoDB a mapping, but only if it doesn't exist yet.__ In order to do that, we need to write condition expression. 114 | ```go 115 | expr, err := expression.NewBuilder(). 116 | WithCondition(expression.AttributeNotExists(expression.Name("old_id"))). 117 | Build() 118 | ``` 119 | This expression will make sure that the `PutItem` operation fails if an item with the same partition key as ours with attribute `old_item` already exists. 120 | 121 | ```go 122 | _, err = m.db.PutItem(ctx, &dynamodb.PutItemInput{ 123 | ConditionExpression: expr.Condition(), 124 | ExpressionAttributeNames: expr.Names(), 125 | ExpressionAttributeValues: expr.Values(), 126 | Item: attrs, 127 | TableName: aws.String(m.table), 128 | }) 129 | ``` 130 | 131 | We ignore output of the function, but care about error very much. If there is no error - then we know that there wasn't any mapping like ours before and we just return to the caller newly generated id. 132 | 133 | ```go 134 | if err == nil { 135 | return idsMapping.NewID, nil 136 | } 137 | ``` 138 | 139 | If however there is an error we need to check what type of error that is. If this error tells us that condition failed, it means that mapping already exists and we can do additional `GetItem` operation to retrieve it. If this is any other error, something went terribly wrong. 140 | 141 | ```go 142 | var conditionErr *types.ConditionalCheckFailedException 143 | if !errors.As(err, &conditionErr) { 144 | return "", err 145 | } 146 | ``` 147 | At this point we know that the `PutItem` operation failed because our conditional failed. 148 | 149 | This means that someone before us already mapped the legacy id to new id and we can retrieve it. 150 | 151 | ```go 152 | out, err := m.db.GetItem(ctx, &dynamodb.GetItemInput{ 153 | Key: map[string]types.AttributeValue{ 154 | "old_id": &types.AttributeValueMemberS{Value: old}, 155 | }, 156 | TableName: aws.String(m.table), 157 | }) 158 | if err != nil { 159 | return "", err 160 | } 161 | 162 | var ret mapping 163 | attributevalue.UnmarshalMap(out.Item, &ret) 164 | return ret.NewID, nil 165 | ``` 166 | 167 | # [Summary](#summary) 168 | 169 | We did it - we no longer rely on ids from legacy system and our solution is bulletproof! There is however one downside to it. Only for the first time for given legacy id we will communicate with DynamoDB once. Subsequent calls to `Map` function require two sequential calls to the DynamoDB. Next time we will try to figure out whether we can do something about it. As always I am inviting you to clone this repository and play with queries yourself! -------------------------------------------------------------------------------- /episode5/template.yml: -------------------------------------------------------------------------------- 1 | Resources: 2 | LegacyIDsTable: 3 | Type: AWS::DynamoDB::Table 4 | Properties: 5 | AttributeDefinitions: 6 | - AttributeName: old_id 7 | AttributeType: S 8 | KeySchema: 9 | - AttributeName: old_id 10 | KeyType: HASH 11 | BillingMode: PAY_PER_REQUEST 12 | TableName: LegacyIDsTable 13 | -------------------------------------------------------------------------------- /episode6/mapper.go: -------------------------------------------------------------------------------- 1 | package episode6 2 | 3 | import ( 4 | "context" 5 | "errors" 6 | 7 | "github.com/aws/aws-sdk-go-v2/aws" 8 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue" 9 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression" 10 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 11 | "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" 12 | "github.com/google/uuid" 13 | ) 14 | 15 | // Mapper keeps Dynamo dependency. 16 | type Mapper struct { 17 | db *dynamodb.Client 18 | table string 19 | } 20 | 21 | // NewMapper creates instance of Mapper. 22 | func NewMapper(client *dynamodb.Client, table string) *Mapper { 23 | return &Mapper{db: client, table: table} 24 | } 25 | 26 | type mapping struct { 27 | OldID string `dynamodbav:"old_id"` 28 | NewID string `dynamodbav:"new_id"` 29 | } 30 | 31 | // Map generates new ID for old ID or retrieves already created new ID. 32 | func (m *Mapper) Map(ctx context.Context, old string) (string, error) { 33 | idsMapping := mapping{OldID: old, NewID: uuid.New().String()} 34 | attrs, err := attributevalue.MarshalMap(&idsMapping) 35 | if err != nil { 36 | return "", err 37 | } 38 | 39 | expr, err := expression.NewBuilder(). 40 | WithCondition(expression.AttributeNotExists(expression.Name("old_id"))). 41 | Build() 42 | if err != nil { 43 | return "", err 44 | } 45 | 46 | _, err = m.db.TransactWriteItems(ctx, &dynamodb.TransactWriteItemsInput{ 47 | TransactItems: []types.TransactWriteItem{ 48 | { 49 | Put: &types.Put{ 50 | ConditionExpression: expr.Condition(), 51 | ExpressionAttributeNames: expr.Names(), 52 | ExpressionAttributeValues: expr.Values(), 53 | Item: attrs, 54 | ReturnValuesOnConditionCheckFailure: types.ReturnValuesOnConditionCheckFailure(types.ReturnValueAllOld), 55 | TableName: aws.String(m.table), 56 | }, 57 | }, 58 | }, 59 | }) 60 | 61 | if err == nil { 62 | return idsMapping.NewID, nil 63 | } 64 | 65 | var transactionCanelled *types.TransactionCanceledException 66 | if !errors.As(err, &transactionCanelled) { 67 | return "", err 68 | } 69 | 70 | // ALL_OLD is not empty - mapping exists. 71 | if len(transactionCanelled.CancellationReasons[0].Item) > 0 { 72 | var ret mapping 73 | attributevalue.UnmarshalMap(transactionCanelled.CancellationReasons[0].Item, &ret) 74 | return ret.NewID, nil 75 | } 76 | 77 | return "", err 78 | } 79 | -------------------------------------------------------------------------------- /episode6/mapper_test.go: -------------------------------------------------------------------------------- 1 | package episode6 2 | 3 | import ( 4 | "context" 5 | "dynamodb-with-go/pkg/dynamo" 6 | "testing" 7 | 8 | "github.com/stretchr/testify/assert" 9 | ) 10 | 11 | func TestMapping(t *testing.T) { 12 | t.Run("generate new ID for each legacy ID", func(t *testing.T) { 13 | ctx := context.Background() 14 | tableName := "LegacyIDsTable" 15 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 16 | defer cleanup() 17 | 18 | mapper := NewMapper(db, tableName) 19 | 20 | first, err := mapper.Map(ctx, "123") 21 | assert.NoError(t, err) 22 | assert.NotEmpty(t, first) 23 | assert.NotEqual(t, "123", first) 24 | 25 | second, err := mapper.Map(ctx, "456") 26 | assert.NoError(t, err) 27 | assert.NotEmpty(t, second) 28 | assert.NotEqual(t, "456", second) 29 | 30 | assert.NotEqual(t, first, second) 31 | }) 32 | 33 | t.Run("do not regenerate ID for the same legacy ID", func(t *testing.T) { 34 | ctx := context.Background() 35 | tableName := "LegacyIDsTable" 36 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 37 | defer cleanup() 38 | 39 | mapper := NewMapper(db, tableName) 40 | 41 | first, err := mapper.Map(ctx, "123") 42 | assert.NoError(t, err) 43 | 44 | second, err := mapper.Map(ctx, "123") 45 | assert.NoError(t, err) 46 | 47 | assert.Equal(t, first, second) 48 | }) 49 | 50 | } 51 | -------------------------------------------------------------------------------- /episode6/post.md: -------------------------------------------------------------------------------- 1 | # DynamoDB with Go #6 2 | 3 | I finished last episode with a promise to show you how to approach what we discussed differently. If you want to get most out of this episode I highly recommend you to read [episode #5](../episode5/post.md) first. In a nutshell a problem to solve was to map ids from existing legacy system into new system that will use its own ids schema. 4 | 5 | ## [Previous approach](#previous-approach) 6 | 7 | Initial solution for this problem was to generate the new id for the legacy id and conditionally put the mapping into DB. The condition stated that we don't want to do that if mapping for given legacy id already exists. When condition succeeded we knew that there is no mapping for given legacy id yet and mapping was inserted into the DynamoDB. If however condition failed we knew that mapping already exists, and we need to grab it from the database. 8 | 9 | This approach works very well and is immune to data races. When we call `Map` function for the first time for given legacy id, we need to make single call to the DynamoDB. The condition succeeds, and we return newly generated id. However, subsequent calls to the `Map` with the same id will require two calls to the database. First failing call because of the condition we have in place, and second one that grabs already existing new id from the Dynamo. 10 | 11 | ## [ReturnValues parameter](#return-values-parameter) 12 | 13 | It turns out that there is a parameter in the `PutItem` called `ReturnValues`. It can be set to `NONE` (default) or `ALL_OLD`. `ALL_OLD` means that if `PutItem` has overridden an item, then the response from the DynamoDB will contain content of the item before overriding it. This would be great for us. We would like to know - if we failed - what was in the DynamoDB that caused the failure. That would mean that we don't need to call Dynamo for the second time. 14 | 15 | Unfortunately - this works only if `PutItem` succeeds and in case of conditional check failure we don't get an image of an item before the `PutItem` that failed. 16 | 17 | Since that didn't work - maybe error that we get back that informs us about condition failure, contains more context on why it failed? 18 | 19 | Unfortunately it doesn't. 20 | 21 | We can however use this idea somewhere else to get just what we want. 22 | 23 | ## [Transactions](#transactions) 24 | 25 | In general, transactions are for something else. You use them when you need to make many changes to the database state and all of them need to be successful in order for transaction to complete. If one of the changes in the transaction fails, then whole transaction fails and none of the changes are being made. 26 | 27 | If the transaction fails then the DynamoDB can give you more context of what happened. Let's see how it works. 28 | 29 | ## [Code](#code) 30 | 31 | All the code is the same as in the 5th episode. Only thing that changes is `Map` function. You can observe here also beauty of automated tests that verify only behaviour. We can change implementation as much as we want and tests do not need to change at all. Moreover, they'll tell us whether new approach works or not. 32 | 33 | Let's change `Map` function to use the transaction. 34 | 35 | ```go 36 | idsMapping := mapping{OldID: old, NewID: uuid.New().String()} 37 | attrs, err := attributevalue.MarshalMap(&idsMapping) 38 | if err != nil { 39 | return "", err 40 | } 41 | expr, err := expression.NewBuilder(). 42 | WithCondition(expression.AttributeNotExists(expression.Name("old_id"))). 43 | Build() 44 | if err != nil { 45 | return "", err 46 | } 47 | ``` 48 | When I told you that only `Map` function changes I didn't mean all of it. The beginning stays exactly the same. Just to recap, we create the mapping with the old id and the new id that gets generated for us. Then we construct the condition that fails when we want to put an item that already exists. 49 | 50 | ```go 51 | _, err = m.db.TransactWriteItems(ctx, &dynamodb.TransactWriteItemsInput{ 52 | TransactItems: []types.TransactWriteItem{ 53 | { 54 | Put: &types.Put{ 55 | ConditionExpression: expr.Condition(), 56 | ExpressionAttributeNames: expr.Names(), 57 | ExpressionAttributeValues: expr.Values(), 58 | Item: attrs, 59 | ReturnValuesOnConditionCheckFailure: types.ReturnValuesOnConditionCheckFailure(types.ReturnValueAllOld), 60 | TableName: aws.String(m.table), 61 | }, 62 | }, 63 | }, 64 | }) 65 | ``` 66 | We have a transaction here with single `Put` operation in it. There is also something that is not an option for regular `PutItem` method - `ReturnValuesOnConditionCheckFailure` parameter. Hopefully when condition fails, we will get exactly what we want - which is the new id that already exists. 67 | 68 | The key aspect here is error handling. 69 | ```go 70 | if err == nil { 71 | return idsMapping.NewID, nil 72 | } 73 | ``` 74 | If there is no error it means that condition succeeded, hence this was first time anyone called `Map` with given legacy id, and we just return what we've put into the DynamoDB. 75 | 76 | ```go 77 | var transactionCanelled *types.TransactionCanceledException 78 | if !errors.As(err, &transactionCanelled) { 79 | return "", err 80 | } 81 | ``` 82 | If there is an error we need to check its type, and if it is not `TransactionCanceledException` - something went wrong, and we don't know what it is, so we just return. 83 | 84 | ```go 85 | if len(transactionCanelled.CancellationReasons[0].Item) > 0 { 86 | var ret mapping 87 | attributevalue.UnmarshalMap(transactionCanelled.CancellationReasons[0].Item, &ret) 88 | return ret.NewID, nil 89 | } 90 | ``` 91 | Otherwise, we get `new_id` from `CancellationReasons` and we can return that to the client without calling Dynamo again! 92 | 93 | ## [Summary](#summary) 94 | 95 | We just showed how can we leverage DynamoDB API to give us reason for transaction failure. In our case this means that we can insert into the DynamoDB new mapping or get existing mapping in one step. 96 | 97 | Should you use this approach? Is it better than the previous one? As always it depends. When calling the DynamoDB only once you'll save time spent on request/response round trip. Does it come for free then? Absolutely not. Transaction calls to the DynamoFB are more expensive in terms of Capacity Units you'll pay for them. So in this particular case either you pay more for the DynamoDB transaction call, but make fewer calls in general, or pay less for single call but call DynamoDB more times and spend more time waiting for network calls to the DynamoDB. Having said that I am not recommending any of the approaches. It depends on your needs. I am just showing possible options. -------------------------------------------------------------------------------- /episode6/template.yml: -------------------------------------------------------------------------------- 1 | Resources: 2 | LegacyIDsTable: 3 | Type: AWS::DynamoDB::Table 4 | Properties: 5 | AttributeDefinitions: 6 | - AttributeName: old_id 7 | AttributeType: S 8 | KeySchema: 9 | - AttributeName: old_id 10 | KeyType: HASH 11 | BillingMode: PAY_PER_REQUEST 12 | TableName: LegacyIDsTable 13 | -------------------------------------------------------------------------------- /episode7/post.md: -------------------------------------------------------------------------------- 1 | # DynamoDB with Go #7 2 | 3 | Imagine that you are working for real estate company that offers office space for rent. All the buildings are those smart new ones with a lot of sensors measuring myriad of parameters. You need to design a system that will manage somehow these sensors. 4 | 5 | ## [Access patterns](#access-patterns) 6 | 7 | First things first, before we open IDE we need to know what are we going to do with this application. Let's enumerate all actions that should be possible. 8 | 9 | We need to be able to register the sensor. I want to introduce a new sensor into the system in a following way: 10 | ```go 11 | Register(Sensor{ID: "humidity-sensor-1", City: "Poznan", Building: "A", Floor: "3", Room: "112"}) 12 | ``` 13 | After succesful registration I expect that there will be possibility to write a new sensor reading. 14 | ```go 15 | NewReading(Reading{SensorID: "humidity-sensor-1", Value: "0.67"}) 16 | ``` 17 | Sensor is registered and there are some data points already in the DB. How can we display this to the user of the system? I would like to be able to get sensors from locations. For example, I would like to say: 18 | - give me all sensors in Lisbon, 19 | - give me all sensors in Berlin in building D on 4th floor, 20 | - give me all sensors in Poznań in building A, 2nd floor in room 432. 21 | 22 | I am thinking about the API that would look like this: 23 | ```go 24 | GetSensors(Location{City: "Poznań", Building: "A", Floor: "2"}) 25 | GetSensors(Location{City: "Poznań", Building: "D", Floor: "1", Room: "123"}) 26 | ``` 27 | Let's say I searched for: _"Poznań, Building A"_ and received list of 25 different sensors. In terms of UI - I want to click on given sensor and receive detailed information about the sensor with 10 latest readings of this sensor. I imagine a call: 28 | ```go 29 | Get("carbon-monoxide-sensor-2", 10) 30 | ``` 31 | 32 | It seems simple enough. We have four different functions operating on two different types. 33 | 34 | ## [Table design](#table-design) 35 | 36 | We know WHAT we want achieve. It's time to define the HOW. How can we implement that? I would like to make this episode an exercise for two different things: 37 | 1. Single Table Design, 38 | 2. modelling hierarchical data. 39 | 40 | ### [Single Table Design](#single-table-design) 41 | 42 | The idea of single table design is to be able to fetch different types of entities with single query to the DynamoDB. Since there is no possibility to query many tables at the same time, different types needs to be stored together within single table. 43 | 44 | ### [Modeling hierarchical data](#modeling-hierarchical-data) 45 | 46 | Being able to fetch data on different level of hierarchy requires some thought upfront. We are going to leverage ability to use `begins_with` method on the sort key. 47 | 48 | ### [Sensors Table](#sensors-table) 49 | 50 | | PK | SK | Value | City | Building | Floor | Room | Type | 51 | | --- | ---- | --- | --- | --- | --- | --- | --- | 52 | | SENSOR#S1 | READ#2020-03-01-12:30 | 2 | | | | | | 53 | | SENSOR#S1 | READ#2020-03-01-12:31 | 3 | | | | | | 54 | | SENSOR#S1 | READ#2020-03-01-12:32 | 5 | | | | | | 55 | | SENSOR#S1 | READ#2020-03-01-12:33 | 3 | | | | | | 56 | | SENSOR#S1 | SENSORINFO | | Poznań | A | 2 | 13 | Gas | 57 | 58 | I sketched out first attempt to put data into the table. First thing to notice is that Partition Key and Sort Key don't have a name that conveys domain knowledge. They're just abbreviation and this is because in Single Table Design there are different types of entities, hence PK and SK can mean different things for a different item type. 59 | 60 | This layout allows us to query a sensor and obtain detailed information together with the latest reads from it. This is why we are doing the Single Table Design. We only have to query the DynamoDB once to obtain different types of entities. 61 | 62 | This isn't bad so far, but there is still one feature missing - querying for many sensors that share the same location. As I said before we are going to use `begins_with` on the Sort Key. I want to do queries like these (pseudocode): 63 | 64 | ```pseudocode 65 | Query(PK="Poznań", SKbeginswith="A#-1") -> all sensors from garage in building A in Poznań 66 | Query(PK="Berlin") -> all sensors in Berlin 67 | Query(PK="Lisbon", SKbeginswith="F#3#102") -> all sensors in Lisbon in the building F, room 102 on 3rd floor. 68 | ``` 69 | 70 | In order to do that we need to introduce additional artificial attribute which is a concatenation of different attributes. 71 | 72 | First thing that comes to mind is to simply write something like this to the table: 73 | 74 | | PK | SK | ID | 75 | | --- | ---- | --- | 76 | | CITY#Poznań | LOCATION#A#2#13 | S1 | 77 | 78 | Now I can satisfy the requirement for querying sensors in the given location. One thing to remember though is that we have two items that need to be synchronized. Detailed information about the sensor, and its location. Registering a sensor and changing the location is more complicated with this approach because we need to change two items in the DynamoDB transactionally so that these pieces of information won't diverge. 79 | 80 | The other idea is to use Global Secondary Index (GSI). 81 | 82 | | PK | SK | Value | City | Building | Floor | Room | Type | GSI_PK | GSI_SK | 83 | | --- | ---- | --- | --- | --- | --- | --- | --- | --- | --- | 84 | | SENSOR#S1 | READ#2020-03-01-12:30 | 2 | | | | | | | | 85 | | SENSOR#S1 | READ#2020-03-01-12:31 | 3 | | | | | | | | 86 | | SENSOR#S1 | READ#2020-03-01-12:32 | 5 | | | | | | | | 87 | | SENSOR#S1 | READ#2020-03-01-12:33 | 3 | | | | | | | | 88 | | SENSOR#S1 | SENSORINFO | | Poznań | A | 2 | 13 | Gas | Poznań | A#2#13 | 89 | 90 | Whenever we want to query for sensors in given location we use GSI to do that. Hidden here is yet another concept which is called the Sparse Index. This index is sparse because it contains only some of the items. When querying or scanning that index we won't get any item that has `READ#` prefix in the SK, because these items aren't in this index (because these items don't have value for GSI_PK and GSI_SK attributes). 91 | 92 | Which approach is better? Additional item in a table is very simple approach that just works. One downside it has is that when sensor is being registered or it's location changes - we need to change two items transactionally and programmatic error can cause this data to diverge. On the other hand there is GSI. Its advantage is that we need to change only single item at a time when location changes or when we register new sensor. You need to be aware however that indexes cost extra money and that data is copied to the GSI in a asynchronous way which further means that when reading data from GSI, strongly consistent reads are not an option. All the reads from GSI are eventually consistent. 93 | 94 | # [Summary](#summary) 95 | 96 | I think we've done enough work for now. Let me split this topic into 2 episodes. This one was about __DynamoDB__, next one will be more __with Go__. Nevertheless, to summarize, we know what our access patterns are, and we know how to implement that in two different ways. More over we learned what are cons and pros of each approach. What I propose for the next episode is that first we will write unit tests that define behavior we want to obtain. Then we are going to implement both ways: with additional item and with GSI. -------------------------------------------------------------------------------- /episode8/post.md: -------------------------------------------------------------------------------- 1 | # DynamoDB with Go #8 2 | 3 | This episode is all about implementation. If you didn't read episode 7th - please [do](../episode7/post.md) because we can't move forward without it. Assuming you've already read it, let's express all the use cases we have with help of unit tests. 4 | 5 | ## [Registering a sensor](#registering-a-sensor) 6 | 7 | Registering a sensor gives ability to record new readings of the sensor later on. 8 | 9 | ```go 10 | t.Run("register sensor, get sensor", func(t *testing.T) { 11 | tableName := "SensorsTable" 12 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 13 | defer cleanup() 14 | manager := sensors.NewManager(db, tableName) 15 | 16 | err := manager.Register(ctx, sensor) 17 | assert.NoError(t, err) 18 | 19 | returned, err := manager.Get(ctx, "sensor-1") 20 | assert.NoError(t, err) 21 | assert.Equal(t, sensor, returned) 22 | }) 23 | ``` 24 | 25 | In order to verify that registration went well - sensor is retrieved afterwards. You might wonder - isn't unit testing about testing single units? Isn't the registration __a unit__? Well, registration doesn't really matter if we cannot do anything with it, and a unit is single behavior. At the beginning of the journey of testing my code it was a little strange for me. I was thinking - "but now such test has 2 reasons to break". Thinking in terms of above example - when registration fails and when retrieval fails. Later on, I came to the conclusion that there is superiority of such test over - for example checking what method was called underneath because it is better to test behavior than implementation. This is the key aspect of testing for me because change in implementation shouldn't break tests when behavior doesn't change. Now let's go back to the sensor business. Next behavior we want to cover is inability to register a sensor with the same ID twice. 26 | 27 | ## [Inability to register the same sensor twice](#register-only-once) 28 | 29 | ```go 30 | t.Run("do not allow to register many times", func(t *testing.T) { 31 | tableName := "SensorsTable" 32 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 33 | defer cleanup() 34 | manager := sensors.NewManager(db, tableName) 35 | 36 | err := manager.Register(ctx, sensor) 37 | assert.NoError(t, err) 38 | 39 | err = manager.Register(ctx, sensor) 40 | assert.EqualError(t, err, "already registered") 41 | }) 42 | ``` 43 | 44 | When you try to register again you get slapped with an error. One thing I want to mention is `sensor` variable that I used in both above mentioned snippets. It's just exemplary sensor I've declared on top of the test suite. You can look it up in the repository if you want to. 45 | 46 | ## [Recording a sensor reading](#recording-sensor-reading) 47 | 48 | Since we know how to register a sensor, let's record a reading of that sensor. 49 | 50 | ```go 51 | t.Run("save new reading", func(t *testing.T) { 52 | tableName := "SensorsTable" 53 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 54 | defer cleanup() 55 | manager := sensors.NewManager(db, tableName) 56 | 57 | err := manager.Register(ctx, sensor) 58 | assert.NoError(t, err) 59 | 60 | err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.67", ReadAt: time.Now()}) 61 | assert.NoError(t, err) 62 | 63 | _, latest, err := manager.LatestReadings(ctx, "sensor-1", 1) 64 | assert.NoError(t, err) 65 | assert.Equal(t, "0.67", latest[0].Value) 66 | }) 67 | ``` 68 | 69 | After saving new reading I need to verify somehow that it worked. In order to do that I am using `LatestReadings` method that provides me with one latest reading - which hopefully should be the one that was just saved. 70 | 71 | ## [Retrieving sensor and its last readings](#retrieve-sensor-with-readings) 72 | 73 | Let's explore more the API that we already've seen in previous test. 74 | 75 | ```go 76 | t.Run("get last readings and sensor", func(t *testing.T) { 77 | tableName := "SensorsTable" 78 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 79 | defer cleanup() 80 | manager := sensors.NewManager(db, tableName) 81 | 82 | err := manager.Register(ctx, sensor) 83 | 84 | assert.NoError(t, err) 85 | 86 | err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.3", ReadAt: time.Now().Add(-20 * time.Second)}) 87 | assert.NoError(t, err) 88 | err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.5", ReadAt: time.Now().Add(-10 * time.Second)}) 89 | assert.NoError(t, err) 90 | err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.67", ReadAt: time.Now()}) 91 | assert.NoError(t, err) 92 | 93 | sensor, latest, err := manager.LatestReadings(ctx, "sensor-1", 2) 94 | assert.NoError(t, err) 95 | assert.Len(t, latest, 2) 96 | assert.Equal(t, "0.67", latest[0].Value) 97 | assert.Equal(t, "0.5", latest[1].Value) 98 | assert.Equal(t, "sensor-1", sensor.ID) 99 | }) 100 | ``` 101 | 102 | The point of this test is to show that we are able to fetch a sensor and latest readings of this sensor at the same time. 103 | 104 | ## [Get sensors by location](#get-sensors-by-location) 105 | 106 | ```go 107 | t.Run("get by sensors by location", func(t *testing.T) { 108 | tableName := "SensorsTable" 109 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 110 | defer cleanup() 111 | manager := sensors.NewManager(db, tableName) 112 | 113 | err := manager.Register(ctx, sensors.Sensor{ID: "sensor-1", City: "Poznan", Building: "A", Floor: "1", Room: "2"}) 114 | err = manager.Register(ctx, sensors.Sensor{ID: "sensor-2", City: "Poznan", Building: "A", Floor: "2", Room: "4"}) 115 | err = manager.Register(ctx, sensors.Sensor{ID: "sensor-3", City: "Poznan", Building: "A", Floor: "2", Room: "5"}) 116 | 117 | ids, err := manager.GetSensors(ctx, sensors.Location{City: "Poznan", Building: "A", Floor: "2"}) 118 | assert.NoError(t, err) 119 | assert.Len(t, ids, 2) 120 | assert.Contains(t, ids, "sensor-2") 121 | assert.Contains(t, ids, "sensor-3") 122 | }) 123 | ``` 124 | 125 | This test is the reason why there are two versions of the code in this episode ([version 1](./v1) and [version 2](./v2)). Both versions have the same test suite we've just described but each of them has different implementation. They differ because I wanted yo show you different approaches of handling hierarchical data modeling. 126 | 127 | Before we jump into implementations I wanted to let you know that I am well aware of the fact that the test suite is not complete. There are corner cases that aren't covered, but I hope you'll realize that this is rather the DynamoDB modeling exercise rather than testing exercise. One more thing about testing. Both versions of the code have the same test suite because we are testing behavior - not implementation. I'm repeating myself, but I would like to emphasize the importance of such tests. I know that some times people are discouraged to test their code because whenever they change production code they need to fix a lot of tests too. It's not the case when testing the behavior. That's it. I used the word "behavior" for the last time in this episode. Moving on. 128 | 129 | ## [Implementation - version 1](#implementation-v1) 130 | 131 | I am going to show you first version of implementation and then we are going to jump into the second one and 132 | compare them. Let me remind you what really is the first version. 133 | 134 | | PK | SK | Value | City | Building | Floor | Room | Type | ID | 135 | | --- | ---- | --- | --- | --- | --- | --- | --- | --- | 136 | | SENSOR#S1 | READ#2020-03-01-12:30 | 2 | | | | | | | 137 | | SENSOR#S1 | READ#2020-03-01-12:31 | 3 | | | | | | | 138 | | SENSOR#S1 | READ#2020-03-01-12:32 | 5 | | | | | | | 139 | | SENSOR#S1 | READ#2020-03-01-12:33 | 3 | | | | | | | 140 | | SENSOR#S1 | SENSORINFO | | Poznań | A | 2 | 13 | Gas | | 141 | | CITY#Poznań| LOCATION#A#2#13 | | | | | | | S1 | 142 | 143 | Information about a sensor is broken down into two separate items with different Partition Keys. Additionally, every recording is an item that shares the same PK as main item describing the sensor. 144 | 145 | ### [Registration](#registration) 146 | 147 | As you can see in the layout, when registering we need to write two different items which means that we are going to need transactions. Let's jump into it. 148 | 149 | ```go 150 | func (s *sensorManager) Register(ctx context.Context, sensor Sensor) error { 151 | attrs, err := attributevalue.MarshalMap(sensor.asItem()) 152 | ``` 153 | 154 | What we are doing here is transforming a sensor into something that we can put into the DynamoDB. Before we go any further, let's talk about `Sensor` type and `asItem` method. I differentiate here two different types: `Sensor` which is the public representation of a sensor and additional type `sensorItem` that is concerned only with how sensor is stored in the DynamoDB. This type is unexported because it is only the implementation detail. 155 | 156 | ```go 157 | type Sensor struct { 158 | ID string 159 | City string 160 | Building string 161 | Floor string 162 | Room string 163 | } 164 | 165 | type sensorItem struct { 166 | PK string `dynamodbav:"pk"` 167 | SK string `dynamodbav:"sk"` 168 | ID string `dynamodbav:"id"` 169 | 170 | City string `dynamodbav:"city"` 171 | Building string `dynamodbav:"building"` 172 | Floor string `dynamodbav:"floor"` 173 | Room string `dynamodbav:"room"` 174 | } 175 | ``` 176 | 177 | As you can see `Sensor` knows nothing about underlying implementation. The `asItem` method is a transformation that makes sure that PK and SK are set in a proper way. 178 | 179 | ```go 180 | func (s Sensor) asItem() sensorItem { 181 | return sensorItem{ 182 | City: s.City, 183 | PK: "SENSOR#" + s.ID, 184 | SK: "SENSORINFO", 185 | ID: s.ID, 186 | Building: s.Building, 187 | Floor: s.Floor, 188 | Room: s.Room, 189 | } 190 | } 191 | ``` 192 | 193 | Notice also that I named Partition Key - PK, and Sort Key - SK. This is because we are using Single Table Design and different items have their own meaning of the PK and SK. In this example SK has value `SENSORINFO`. It is a constant value. I am setting this that way so that we are able to distinguish a sensor and its readings. Now, back to the implementation. The sensor is in the format that DynamoDB will understand. Next thing we need to take care of is uniqueness. We cannot register the same sensor twice and in order to achieve that we need a condition. 194 | 195 | ```go 196 | expr, err := expression.NewBuilder().WithCondition(expression.AttributeNotExists(expression.Name("pk"))).Build() 197 | ``` 198 | 199 | What it says is: "I am going to move further with the operation only if DynamoDB doesn't have an item with `pk` that I want to store in this operation". 200 | 201 | ```go 202 | _, err = s.db.TransactWriteItems(ctx, &dynamodb.TransactWriteItemsInput{ 203 | TransactItems: []types.TransactWriteItem{ 204 | { 205 | Put: &types.Put{ 206 | ConditionExpression: expr.Condition(), 207 | ExpressionAttributeNames: expr.Names(), 208 | ExpressionAttributeValues: expr.Values(), 209 | 210 | Item: attrs, 211 | TableName: aws.String(s.table), 212 | }, 213 | }, 214 | { 215 | Put: &types.Put{ 216 | Item: map[string]types.AttributeValue{ 217 | "pk": &types.AttributeValueMemberS{Value: "CITY#" + sensor.City}, 218 | "sk": &types.AttributeValueMemberS{Value: fmt.Sprintf("LOCATION#%s#%s#%s", sensor.Building, sensor.Floor, sensor.Room)}, 219 | "id": &types.AttributeValueMemberS{Value: sensor.ID}, 220 | }, 221 | TableName: aws.String(s.table), 222 | }, 223 | }, 224 | }, 225 | }) 226 | ``` 227 | 228 | We want to put two items into the DynamoDB, sensor itself and the location. First Write Item has a condition that we defined and the other constructs the location. I decided to define it on the fly here because it's not important anywhere else. 229 | 230 | Let's have a look at the error handling. 231 | 232 | ```go 233 | if err != nil { 234 | var transactionCanelled *types.TransactionCanceledException 235 | if errors.As(err, &transactionCanelled) { 236 | return errors.New("already registered") 237 | } 238 | return err 239 | } 240 | return nil 241 | ``` 242 | 243 | It needs to be handled explicitly because we need to verify whether transaction failed because of the failed condition or because something unexpected happened. 244 | 245 | ### [Sensor retrieval](#sensor-retrieval) 246 | 247 | In order to retrieve sensor we need to use proper `SK` and `PK` which means we need to construct proper Composite Primary Key. 248 | 249 | ```go 250 | map[string]types.AttributeValue{ 251 | "pk": &types.AttributeValueMemberS{Value: "SENSOR#" + id}, 252 | "sk": &types.AttributeValueMemberS{Value: "SENSORINFO"}, 253 | } 254 | ``` 255 | The ID needs to have the prefix, and SK needs to be the constant I choose to mark a sensor. If you want to see whole implementation of `Get` method please have a look [here](./v1/sensors.go). There is nothing interesting going on there - just simple data retrieval, so I am not repeating it here. 256 | 257 | ### [Saving a reading](#saving-a-reding) 258 | 259 | Another fairly simple piece of code. It is just a PUT operation of a `Reading`. What is worth to talk about her is how data structure looks like. 260 | 261 | ```go 262 | type Reading struct { 263 | SensorID string 264 | Value string 265 | ReadAt time.Time 266 | } 267 | 268 | type readingItem struct { 269 | SensorID string `dynamodbav:"pk"` 270 | Value string `dynamodbav:"value"` 271 | ReadAt string `dynamodbav:"sk"` 272 | } 273 | ``` 274 | 275 | I used the same pattern as for `Sensor`. There is `Reading` that makes sense in the domain, and there is `readingItem` that defines how implementation is going to look like. 276 | 277 | ```go 278 | func (r Reading) asItem() readingItem { 279 | return readingItem{ 280 | SensorID: "SENSOR#" + r.SensorID, 281 | ReadAt: "READ#" + r.ReadAt.Format(time.RFC3339), 282 | Value: r.Value, 283 | } 284 | } 285 | ``` 286 | This transformation makes sure that `PK` of an item begins with `SENSOR#` prefix. We need that because we want readings of the sensor and sensor itself to be in the same __Item Collection__. Item collection is collection of items that share the same Partition Key. We need that to be able to retrieve sensor and its latest readings with single query. Other thing that is going on here is formatting `SK` of an item in a way that will be sortable by time. 287 | 288 | ### [Retrieving the latest readings and the sensor](#retrieving-latest) 289 | 290 | We will query two item types at the same time. We need some sort of condition. 291 | 292 | ```go 293 | expr, err := expression.NewBuilder().WithKeyCondition(expression.KeyAnd( 294 | expression.KeyEqual(expression.Key("pk"), expression.Value("SENSOR#"+sensorID)), 295 | expression.KeyLessThanEqual(expression.Key("sk"), expression.Value("SENSORINFO")), 296 | )).Build() 297 | ``` 298 | 299 | Let's read it. Attribute `pk` is the ID prefixed with `SENSOR#`. This makes sense - we need to fetch whole item collection. Let's keep reading. Attribute `sk` needs to be less than or equal than `SENSORINFO`. Wait, what? We wanted to fetch the sensor and it's readings. How on earth such condition is going to achieve that? Bare with me. 300 | 301 | | PK | SK | 302 | | --- | ---- | 303 | | SENSOR#S1 | READ#2020-03-01-12:30 | 304 | | SENSOR#S1 | READ#2020-03-01-12:31 | 305 | | SENSOR#S1 | READ#2020-03-01-12:32 | 306 | | SENSOR#S1 | READ#2020-03-01-12:33 | 307 | | SENSOR#S1 | SENSORINFO | 308 | 309 | This is excerpt from the table that I showed you before but containing just Composite Primary Key. Items are sorted in ascending order by default. This means that readings are sorted from oldest to the newest, and after readings there is `SENSORINFO` because `S` comes after `R` in the alphabet. What we want to achieve is to read the data backwards starting from the item with `SENSORINFO` as `SK`. In order to read the data in this way we need to construct a query with parameter `ScanIndexForward` set to false. 310 | 311 | ```go 312 | out, err := s.db.Query(ctx, &dynamodb.QueryInput{ 313 | ExpressionAttributeValues: expr.Values(), 314 | ExpressionAttributeNames: expr.Names(), 315 | KeyConditionExpression: expr.KeyCondition(), 316 | Limit: aws.Int32(last + 1), 317 | ScanIndexForward: aws.Bool(false), 318 | TableName: aws.String(s.table), 319 | }) 320 | ``` 321 | 322 | Also, the limit is set to amount of last readings we want to retrieve increased by one so that we will retrieve information about the sensor as well. 323 | 324 | What is going on at the end of the method is proper unmarshalling items into domain objects. 325 | ```go 326 | var si sensorItem 327 | err = attributevalue.UnmarshalMap(out.Items[0], &si) 328 | 329 | var ri []readingItem 330 | err = attributevalue.UnmarshalListOfMaps(out.Items[1:out.Count], &ri) 331 | 332 | var readings []Reading 333 | for _, r := range ri { 334 | readings = append(readings, r.asReading()) 335 | } 336 | return si.asSensor(), readings, nil 337 | ``` 338 | We know for a fact that `Sensor` is first in the item collection, so it is unmarshalled as the `Sensor`. The rest of the items are treated as `Readings`. 339 | 340 | ### [Get sensors by location](#get-sensors-by-location) 341 | 342 | As you remember in this version of implementation - the location is stored as an additional item. Method `GetSensors` accepts `Location` type that contains `City`, `Building`, `Floor` and `Room`. An item representing the location looks like this: 343 | 344 | | PK | SK | ID | 345 | | --- | ---- | --- | 346 | | CITY#Poznań | LOCATION#A#2#13 | S1 | 347 | 348 | We need to build key condition that will point to `PK` which is just a `City` prefixed with `CITY#` and that has `SK` that begins with certain prefix. Depending on level of location precision - `SK` begins with shorter or longer prefix that specify from where we should get the sensors. 349 | 350 | ```go 351 | expr, err := expression.NewBuilder().WithKeyCondition(expression.KeyAnd( 352 | expression.KeyEqual(expression.Key("pk"), expression.Value("CITY#"+location.City)), 353 | expression.KeyBeginsWith(expression.Key("sk"), location.asPath()), 354 | )).Build() 355 | ``` 356 | 357 | After building the condition expression we need to use it in the query: 358 | 359 | ```go 360 | out, err := s.db.Query(ctx, &dynamodb.QueryInput{ 361 | ExpressionAttributeNames: expr.Names(), 362 | ExpressionAttributeValues: expr.Values(), 363 | KeyConditionExpression: expr.KeyCondition(), 364 | TableName: aws.String(s.table), 365 | }) 366 | ``` 367 | 368 | At the end I just prepare list of IDs that should be returned from the method. 369 | 370 | ```go 371 | var ids []string 372 | for _, item := range out.Items { 373 | var si sensorItem 374 | attributevalue.UnmarshalMap(item, &si) 375 | ids = append(ids, si.ID) 376 | } 377 | return ids, nil 378 | ``` 379 | 380 | This is it. Complete code for first version of implementation is [here](./v1). 381 | 382 | ## [Implementation - version 2](#implementation-v2) 383 | 384 | Second version of the implementation varies a little. The difference lays in how location is stored. In first version queryable location was just additional item. Second version uses Global Secondary Index for that purpose. 385 | 386 | | PK | SK | City | Building | Floor | Room | Type | GSI_PK | GSI_SK | 387 | | --- | ---- | --- | --- | --- | --- | --- | --- | --- | 388 | | SENSOR#S1 | SENSORINFO | Poznań | A | 2 | 13 | Gas | Poznań | A#2#13 | 389 | 390 | Local Secondary Index cannot be used in this scenario because it would need to have the same Partition Key as Primary Key. Because we want to use different Partition Key - we need to use GSI. 391 | 392 | I am going to show you only two methods because only they are different - registration of a sensor and retrieving sensors by the location. 393 | 394 | ### [Registration](#registration-v2) 395 | 396 | `Sensor` type stays exactly the same because the domain sense of it doesn't change with implementation. However `sensorItem` is going to have two additional fields: `GSIPK` and `GSISK`. 397 | 398 | ```go 399 | func (s Sensor) asItem() sensorItem { 400 | return sensorItem{ 401 | City: s.City, 402 | ID: "SENSOR#" + s.ID, 403 | SK: "SENSORINFO", 404 | Building: s.Building, 405 | Floor: s.Floor, 406 | Room: s.Room, 407 | GSIPK: "CITY#" + s.City, 408 | GSISK: fmt.Sprintf("LOCATION#%s#%s#%s", s.Building, s.Floor, s.Room), 409 | } 410 | } 411 | ``` 412 | As you can see `GSIPK` and `GSISK` look exactly the same as the additional `location` item in the first version of implementation. It's the same information but inside`sensorItem`. 413 | 414 | Registration itself holds exactly the same condition as before - which is to make sure that we are not introducing duplicated sensors. What changed is instead of using transactions - we use simple PUT operation. 415 | 416 | ```go 417 | _, err = s.db.PutItem(ctx, &dynamodb.PutItemInput{ 418 | ConditionExpression: expr.Condition(), 419 | ExpressionAttributeNames: expr.Names(), 420 | ExpressionAttributeValues: expr.Values(), 421 | 422 | Item: attrs, 423 | TableName: aws.String(s.table), 424 | }) 425 | ``` 426 | 427 | Frankly speaking registration just got very boring. We transform the `Sensor` into `sensorItem` and drop it into the DynamoDB with a condition. 428 | 429 | ### [Get sensors by location](#get-sensors-by-location-v2) 430 | 431 | This method changed just slightly compared to the first version. Let's have a look at the key condition. 432 | ```go 433 | expr, err := expression.NewBuilder().WithKeyCondition(expression.KeyAnd( 434 | expression.KeyEqual(expression.Key("gsi_pk"), expression.Value("CITY#"+location.City)), 435 | expression.KeyBeginsWith(expression.Key("gsi_sk"), location.asPath()), 436 | )).Build() 437 | ``` 438 | It uses exactly the same mechanism as first version but instead of `pk` and `sk`, we use `gsi_pk` and `gsi_sk` when building key condition expression. What about the query? 439 | ```go 440 | out, err := s.db.Query(ctx, &dynamodb.QueryInput{ 441 | ExpressionAttributeNames: expr.Names(), 442 | ExpressionAttributeValues: expr.Values(), 443 | KeyConditionExpression: expr.KeyCondition(), 444 | TableName: aws.String(s.table), 445 | IndexName: aws.String("ByLocation"), 446 | }) 447 | ``` 448 | It didn't change much either. There is one additional bit which is `IndexName` that we used. This index has `GSI_PK` and `GSI_SK` as its key. 449 | 450 | This is the whole difference between two versions. 451 | 452 | ## Summary 453 | 454 | We covered a lot this time. Let me enumerate concepts that we used to make this work. 455 | 456 | - Single Table Design 457 | - Fetching two different item type with single query 458 | - Modeling hierarchical data in DynamoDB 459 | - Sparse Indexes 460 | - Transactions 461 | 462 | I hope you enjoyed this long journey. Also, I would like to invite you more than ever to fetch this repository and play with examples! -------------------------------------------------------------------------------- /episode8/v1/sensors/sensors.go: -------------------------------------------------------------------------------- 1 | package sensors 2 | 3 | import ( 4 | "context" 5 | "errors" 6 | "fmt" 7 | "strings" 8 | "time" 9 | 10 | "github.com/aws/aws-sdk-go-v2/aws" 11 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue" 12 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression" 13 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 14 | "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" 15 | ) 16 | 17 | type Sensor struct { 18 | ID string 19 | City string 20 | Building string 21 | Floor string 22 | Room string 23 | } 24 | 25 | type Reading struct { 26 | SensorID string 27 | Value string 28 | ReadAt time.Time 29 | } 30 | 31 | type Location struct { 32 | City string 33 | Building string 34 | Floor string 35 | } 36 | 37 | func (l Location) asPath() string { 38 | path := "LOCATION#" 39 | if l.Building == "" { 40 | return path 41 | } 42 | path = path + l.Building + "#" 43 | if l.Floor == "" { 44 | return path 45 | } 46 | return path + l.Floor 47 | } 48 | 49 | func (s Sensor) asItem() sensorItem { 50 | return sensorItem{ 51 | City: s.City, 52 | PK: "SENSOR#" + s.ID, 53 | SK: "SENSORINFO", 54 | ID: s.ID, 55 | Building: s.Building, 56 | Floor: s.Floor, 57 | Room: s.Room, 58 | } 59 | } 60 | 61 | type sensorItem struct { 62 | PK string `dynamodbav:"pk"` 63 | SK string `dynamodbav:"sk"` 64 | ID string `dynamodbav:"id"` 65 | 66 | City string `dynamodbav:"city"` 67 | Building string `dynamodbav:"building"` 68 | Floor string `dynamodbav:"floor"` 69 | Room string `dynamodbav:"room"` 70 | } 71 | 72 | type readingItem struct { 73 | SensorID string `dynamodbav:"pk"` 74 | Value string `dynamodbav:"value"` 75 | ReadAt string `dynamodbav:"sk"` 76 | } 77 | 78 | func (r Reading) asItem() readingItem { 79 | return readingItem{ 80 | SensorID: "SENSOR#" + r.SensorID, 81 | ReadAt: "READ#" + r.ReadAt.Format(time.RFC3339), 82 | Value: r.Value, 83 | } 84 | } 85 | 86 | func (si sensorItem) asSensor() Sensor { 87 | return Sensor{ 88 | ID: si.ID, 89 | City: si.City, 90 | Building: si.Building, 91 | Floor: si.Floor, 92 | Room: si.Room, 93 | } 94 | } 95 | 96 | func (ri readingItem) asReading() Reading { 97 | t, err := time.Parse(time.RFC3339, strings.Split(ri.ReadAt, "#")[1]) 98 | if err != nil { 99 | panic("I would handle that in production") 100 | } 101 | return Reading{ 102 | SensorID: strings.Split(ri.SensorID, "#")[1], 103 | ReadAt: t, 104 | Value: ri.Value, 105 | } 106 | } 107 | 108 | func NewManager(db *dynamodb.Client, table string) *sensorManager { 109 | return &sensorManager{db: db, table: table} 110 | } 111 | 112 | type SensorsManager interface { 113 | Register(ctx context.Context, sensor Sensor) error 114 | Get(ctx context.Context, id string) (Sensor, error) 115 | } 116 | 117 | type sensorManager struct { 118 | db *dynamodb.Client 119 | table string 120 | } 121 | 122 | func (s *sensorManager) Register(ctx context.Context, sensor Sensor) error { 123 | attrs, err := attributevalue.MarshalMap(sensor.asItem()) 124 | if err != nil { 125 | return err 126 | } 127 | expr, err := expression.NewBuilder().WithCondition(expression.AttributeNotExists(expression.Name("pk"))).Build() 128 | if err != nil { 129 | return err 130 | } 131 | 132 | _, err = s.db.TransactWriteItems(ctx, &dynamodb.TransactWriteItemsInput{ 133 | TransactItems: []types.TransactWriteItem{ 134 | { 135 | Put: &types.Put{ 136 | ConditionExpression: expr.Condition(), 137 | ExpressionAttributeNames: expr.Names(), 138 | ExpressionAttributeValues: expr.Values(), 139 | 140 | Item: attrs, 141 | TableName: aws.String(s.table), 142 | }, 143 | }, 144 | { 145 | Put: &types.Put{ 146 | Item: map[string]types.AttributeValue{ 147 | "pk": &types.AttributeValueMemberS{Value: "CITY#" + sensor.City}, 148 | "sk": &types.AttributeValueMemberS{Value: fmt.Sprintf("LOCATION#%s#%s#%s", sensor.Building, sensor.Floor, sensor.Room)}, 149 | "id": &types.AttributeValueMemberS{Value: sensor.ID}, 150 | }, 151 | TableName: aws.String(s.table), 152 | }, 153 | }, 154 | }, 155 | }) 156 | if err != nil { 157 | var transactionCanelled *types.TransactionCanceledException 158 | if errors.As(err, &transactionCanelled) { 159 | return errors.New("already registered") 160 | } 161 | return err 162 | } 163 | return nil 164 | } 165 | 166 | func (s *sensorManager) Get(ctx context.Context, id string) (Sensor, error) { 167 | out, err := s.db.GetItem(ctx, &dynamodb.GetItemInput{ 168 | Key: map[string]types.AttributeValue{ 169 | "pk": &types.AttributeValueMemberS{Value: "SENSOR#" + id}, 170 | "sk": &types.AttributeValueMemberS{Value: "SENSORINFO"}, 171 | }, 172 | TableName: aws.String(s.table), 173 | }) 174 | if err != nil { 175 | return Sensor{}, err 176 | } 177 | 178 | var si sensorItem 179 | err = attributevalue.UnmarshalMap(out.Item, &si) 180 | if err != nil { 181 | return Sensor{}, err 182 | } 183 | return si.asSensor(), nil 184 | } 185 | 186 | func (s *sensorManager) SaveReading(ctx context.Context, reading Reading) error { 187 | attrs, err := attributevalue.MarshalMap(reading.asItem()) 188 | if err != nil { 189 | return err 190 | } 191 | _, err = s.db.PutItem(ctx, &dynamodb.PutItemInput{ 192 | Item: attrs, 193 | TableName: aws.String(s.table), 194 | }) 195 | return err 196 | } 197 | 198 | func (s *sensorManager) LatestReadings(ctx context.Context, sensorID string, last int32) (Sensor, []Reading, error) { 199 | expr, err := expression.NewBuilder().WithKeyCondition(expression.KeyAnd( 200 | expression.KeyEqual(expression.Key("pk"), expression.Value("SENSOR#"+sensorID)), 201 | expression.KeyLessThanEqual(expression.Key("sk"), expression.Value("SENSORINFO")), 202 | )).Build() 203 | if err != nil { 204 | return Sensor{}, nil, err 205 | } 206 | 207 | out, err := s.db.Query(ctx, &dynamodb.QueryInput{ 208 | ExpressionAttributeValues: expr.Values(), 209 | ExpressionAttributeNames: expr.Names(), 210 | KeyConditionExpression: expr.KeyCondition(), 211 | Limit: aws.Int32(last + 1), 212 | ScanIndexForward: aws.Bool(false), 213 | TableName: aws.String(s.table), 214 | }) 215 | if err != nil { 216 | return Sensor{}, nil, err 217 | } 218 | 219 | var si sensorItem 220 | err = attributevalue.UnmarshalMap(out.Items[0], &si) 221 | 222 | var ri []readingItem 223 | err = attributevalue.UnmarshalListOfMaps(out.Items[1:out.Count], &ri) 224 | 225 | var readings []Reading 226 | for _, r := range ri { 227 | readings = append(readings, r.asReading()) 228 | } 229 | return si.asSensor(), readings, nil 230 | } 231 | 232 | func (s *sensorManager) GetSensors(ctx context.Context, location Location) ([]string, error) { 233 | expr, err := expression.NewBuilder().WithKeyCondition(expression.KeyAnd( 234 | expression.KeyEqual(expression.Key("pk"), expression.Value("CITY#"+location.City)), 235 | expression.KeyBeginsWith(expression.Key("sk"), location.asPath()), 236 | )).Build() 237 | if err != nil { 238 | return nil, err 239 | } 240 | 241 | out, err := s.db.Query(ctx, &dynamodb.QueryInput{ 242 | ExpressionAttributeNames: expr.Names(), 243 | ExpressionAttributeValues: expr.Values(), 244 | KeyConditionExpression: expr.KeyCondition(), 245 | TableName: aws.String(s.table), 246 | }) 247 | if err != nil { 248 | return nil, err 249 | } 250 | var ids []string 251 | for _, item := range out.Items { 252 | var si sensorItem 253 | attributevalue.UnmarshalMap(item, &si) 254 | ids = append(ids, si.ID) 255 | } 256 | return ids, nil 257 | } 258 | -------------------------------------------------------------------------------- /episode8/v1/sensors/sensors_test.go: -------------------------------------------------------------------------------- 1 | package sensors_test 2 | 3 | import ( 4 | "context" 5 | "testing" 6 | "time" 7 | 8 | "dynamodb-with-go/episode8/v1/sensors" 9 | "dynamodb-with-go/pkg/dynamo" 10 | 11 | "github.com/stretchr/testify/assert" 12 | ) 13 | 14 | func TestSensors(t *testing.T) { 15 | ctx := context.Background() 16 | 17 | sensor := sensors.Sensor{ 18 | ID: "sensor-1", 19 | City: "Poznan", 20 | Building: "A", 21 | Floor: "1", 22 | Room: "123", 23 | } 24 | 25 | t.Run("register sensor, get sensor", func(t *testing.T) { 26 | tableName := "SensorsTable" 27 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 28 | defer cleanup() 29 | manager := sensors.NewManager(db, tableName) 30 | 31 | err := manager.Register(ctx, sensor) 32 | assert.NoError(t, err) 33 | 34 | returned, err := manager.Get(ctx, "sensor-1") 35 | assert.NoError(t, err) 36 | assert.Equal(t, sensor, returned) 37 | }) 38 | 39 | t.Run("do not allow to register many times", func(t *testing.T) { 40 | tableName := "SensorsTable" 41 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 42 | defer cleanup() 43 | manager := sensors.NewManager(db, tableName) 44 | 45 | err := manager.Register(ctx, sensor) 46 | assert.NoError(t, err) 47 | 48 | err = manager.Register(ctx, sensor) 49 | assert.EqualError(t, err, "already registered") 50 | }) 51 | 52 | t.Run("save new reading", func(t *testing.T) { 53 | tableName := "SensorsTable" 54 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 55 | defer cleanup() 56 | manager := sensors.NewManager(db, tableName) 57 | 58 | err := manager.Register(ctx, sensor) 59 | assert.NoError(t, err) 60 | 61 | err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.67", ReadAt: time.Now()}) 62 | assert.NoError(t, err) 63 | 64 | _, latest, err := manager.LatestReadings(ctx, "sensor-1", 1) 65 | assert.NoError(t, err) 66 | assert.Equal(t, "0.67", latest[0].Value) 67 | }) 68 | 69 | t.Run("get last readings and sensor", func(t *testing.T) { 70 | tableName := "SensorsTable" 71 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 72 | defer cleanup() 73 | manager := sensors.NewManager(db, tableName) 74 | 75 | err := manager.Register(ctx, sensor) 76 | 77 | assert.NoError(t, err) 78 | 79 | err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.3", ReadAt: time.Now().Add(-20 * time.Second)}) 80 | assert.NoError(t, err) 81 | err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.5", ReadAt: time.Now().Add(-10 * time.Second)}) 82 | assert.NoError(t, err) 83 | err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.67", ReadAt: time.Now()}) 84 | assert.NoError(t, err) 85 | 86 | sensor, latest, err := manager.LatestReadings(ctx, "sensor-1", 2) 87 | assert.NoError(t, err) 88 | assert.Len(t, latest, 2) 89 | assert.Equal(t, "0.67", latest[0].Value) 90 | assert.Equal(t, "0.5", latest[1].Value) 91 | assert.Equal(t, "sensor-1", sensor.ID) 92 | }) 93 | 94 | t.Run("get by sensors by location", func(t *testing.T) { 95 | tableName := "SensorsTable" 96 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 97 | defer cleanup() 98 | manager := sensors.NewManager(db, tableName) 99 | 100 | err := manager.Register(ctx, sensors.Sensor{ID: "sensor-1", City: "Poznan", Building: "A", Floor: "1", Room: "2"}) 101 | err = manager.Register(ctx, sensors.Sensor{ID: "sensor-2", City: "Poznan", Building: "A", Floor: "2", Room: "4"}) 102 | err = manager.Register(ctx, sensors.Sensor{ID: "sensor-3", City: "Poznan", Building: "A", Floor: "2", Room: "5"}) 103 | 104 | ids, err := manager.GetSensors(ctx, sensors.Location{City: "Poznan", Building: "A", Floor: "2"}) 105 | assert.NoError(t, err) 106 | assert.Len(t, ids, 2) 107 | assert.Contains(t, ids, "sensor-2") 108 | assert.Contains(t, ids, "sensor-3") 109 | }) 110 | } 111 | -------------------------------------------------------------------------------- /episode8/v1/template.yml: -------------------------------------------------------------------------------- 1 | Resources: 2 | SensorsTable: 3 | Type: AWS::DynamoDB::Table 4 | Properties: 5 | AttributeDefinitions: 6 | - AttributeName: pk 7 | AttributeType: S 8 | - AttributeName: sk 9 | AttributeType: S 10 | KeySchema: 11 | - AttributeName: pk 12 | KeyType: HASH 13 | - AttributeName: sk 14 | KeyType: RANGE 15 | BillingMode: PAY_PER_REQUEST 16 | TableName: SensorsTable 17 | -------------------------------------------------------------------------------- /episode8/v2/sensors/sensors.go: -------------------------------------------------------------------------------- 1 | package sensors 2 | 3 | import ( 4 | "context" 5 | "errors" 6 | "fmt" 7 | "strings" 8 | "time" 9 | 10 | "github.com/aws/aws-sdk-go-v2/aws" 11 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue" 12 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression" 13 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 14 | "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" 15 | ) 16 | 17 | type Sensor struct { 18 | ID string 19 | City string 20 | Building string 21 | Floor string 22 | Room string 23 | } 24 | 25 | type Reading struct { 26 | SensorID string 27 | Value string 28 | ReadAt time.Time 29 | } 30 | 31 | type Location struct { 32 | City string 33 | Building string 34 | Floor string 35 | } 36 | 37 | func (l Location) asPath() string { 38 | path := "LOCATION#" 39 | if l.Building == "" { 40 | return path 41 | } 42 | path = path + l.Building + "#" 43 | if l.Floor == "" { 44 | return path 45 | } 46 | return path + l.Floor 47 | } 48 | 49 | func (s Sensor) asItem() sensorItem { 50 | return sensorItem{ 51 | City: s.City, 52 | ID: "SENSOR#" + s.ID, 53 | SK: "SENSORINFO", 54 | Building: s.Building, 55 | Floor: s.Floor, 56 | Room: s.Room, 57 | GSIPK: "CITY#" + s.City, 58 | GSISK: fmt.Sprintf("LOCATION#%s#%s#%s", s.Building, s.Floor, s.Room), 59 | } 60 | } 61 | 62 | type sensorItem struct { 63 | ID string `dynamodbav:"pk"` 64 | SK string `dynamodbav:"sk"` 65 | 66 | City string `dynamodbav:"city"` 67 | Building string `dynamodbav:"building"` 68 | Floor string `dynamodbav:"floor"` 69 | Room string `dynamodbav:"room"` 70 | 71 | GSIPK string `dynamodbav:"gsi_pk"` 72 | GSISK string `dynamodbav:"gsi_sk"` 73 | } 74 | 75 | type readingItem struct { 76 | SensorID string `dynamodbav:"pk"` 77 | Value string `dynamodbav:"value"` 78 | ReadAt string `dynamodbav:"sk"` 79 | } 80 | 81 | func (r Reading) asItem() readingItem { 82 | return readingItem{ 83 | SensorID: "SENSOR#" + r.SensorID, 84 | ReadAt: "READ#" + r.ReadAt.Format(time.RFC3339), 85 | Value: r.Value, 86 | } 87 | } 88 | 89 | func (si sensorItem) asSensor() Sensor { 90 | return Sensor{ 91 | ID: strings.Split(si.ID, "#")[1], 92 | City: si.City, 93 | Building: si.Building, 94 | Floor: si.Floor, 95 | Room: si.Room, 96 | } 97 | } 98 | 99 | func (ri readingItem) asReading() Reading { 100 | t, err := time.Parse(time.RFC3339, strings.Split(ri.ReadAt, "#")[1]) 101 | if err != nil { 102 | panic("I would handle that in production") 103 | } 104 | return Reading{ 105 | SensorID: strings.Split(ri.SensorID, "#")[1], 106 | ReadAt: t, 107 | Value: ri.Value, 108 | } 109 | } 110 | 111 | func NewManager(db *dynamodb.Client, table string) *sensorManager { 112 | return &sensorManager{db: db, table: table} 113 | } 114 | 115 | type SensorsManager interface { 116 | Register(ctx context.Context, sensor Sensor) error 117 | Get(ctx context.Context, id string) (Sensor, error) 118 | } 119 | 120 | type sensorManager struct { 121 | db *dynamodb.Client 122 | table string 123 | } 124 | 125 | func (s *sensorManager) Register(ctx context.Context, sensor Sensor) error { 126 | attrs, err := attributevalue.MarshalMap(sensor.asItem()) 127 | if err != nil { 128 | return err 129 | } 130 | expr, err := expression.NewBuilder().WithCondition(expression.AttributeNotExists(expression.Name("pk"))).Build() 131 | if err != nil { 132 | return err 133 | } 134 | 135 | _, err = s.db.PutItem(ctx, &dynamodb.PutItemInput{ 136 | ConditionExpression: expr.Condition(), 137 | ExpressionAttributeNames: expr.Names(), 138 | ExpressionAttributeValues: expr.Values(), 139 | 140 | Item: attrs, 141 | TableName: aws.String(s.table), 142 | }) 143 | 144 | if err != nil { 145 | var conditionFailed *types.ConditionalCheckFailedException 146 | if errors.As(err, &conditionFailed) { 147 | return errors.New("already registered") 148 | } 149 | return err 150 | } 151 | return nil 152 | } 153 | 154 | func (s *sensorManager) Get(ctx context.Context, id string) (Sensor, error) { 155 | out, err := s.db.GetItem(ctx, &dynamodb.GetItemInput{ 156 | Key: map[string]types.AttributeValue{ 157 | "pk": &types.AttributeValueMemberS{Value: "SENSOR#" + id}, 158 | "sk": &types.AttributeValueMemberS{Value: "SENSORINFO"}, 159 | }, 160 | TableName: aws.String(s.table), 161 | }) 162 | if err != nil { 163 | return Sensor{}, err 164 | } 165 | 166 | var si sensorItem 167 | err = attributevalue.UnmarshalMap(out.Item, &si) 168 | if err != nil { 169 | return Sensor{}, err 170 | } 171 | return si.asSensor(), nil 172 | } 173 | 174 | func (s *sensorManager) SaveReading(ctx context.Context, reading Reading) error { 175 | attrs, err := attributevalue.MarshalMap(reading.asItem()) 176 | if err != nil { 177 | return err 178 | } 179 | _, err = s.db.PutItem(ctx, &dynamodb.PutItemInput{ 180 | Item: attrs, 181 | TableName: aws.String(s.table), 182 | }) 183 | return err 184 | } 185 | 186 | func (s *sensorManager) LatestReadings(ctx context.Context, sensorID string, last int32) (Sensor, []Reading, error) { 187 | expr, err := expression.NewBuilder().WithKeyCondition(expression.KeyAnd( 188 | expression.KeyEqual(expression.Key("pk"), expression.Value("SENSOR#"+sensorID)), 189 | expression.KeyLessThanEqual(expression.Key("sk"), expression.Value("SENSORINFO")), 190 | )).Build() 191 | if err != nil { 192 | return Sensor{}, nil, err 193 | } 194 | 195 | out, err := s.db.Query(ctx, &dynamodb.QueryInput{ 196 | ExpressionAttributeValues: expr.Values(), 197 | ExpressionAttributeNames: expr.Names(), 198 | KeyConditionExpression: expr.KeyCondition(), 199 | Limit: aws.Int32(last + 1), 200 | ScanIndexForward: aws.Bool(false), 201 | TableName: aws.String(s.table), 202 | }) 203 | if err != nil { 204 | return Sensor{}, nil, err 205 | } 206 | 207 | var si sensorItem 208 | err = attributevalue.UnmarshalMap(out.Items[0], &si) 209 | 210 | var ri []readingItem 211 | err = attributevalue.UnmarshalListOfMaps(out.Items[1:out.Count], &ri) 212 | 213 | var readings []Reading 214 | for _, r := range ri { 215 | readings = append(readings, r.asReading()) 216 | } 217 | return si.asSensor(), readings, nil 218 | } 219 | 220 | func (s *sensorManager) GetSensors(ctx context.Context, location Location) ([]string, error) { 221 | expr, err := expression.NewBuilder().WithKeyCondition(expression.KeyAnd( 222 | expression.KeyEqual(expression.Key("gsi_pk"), expression.Value("CITY#"+location.City)), 223 | expression.KeyBeginsWith(expression.Key("gsi_sk"), location.asPath()), 224 | )).Build() 225 | if err != nil { 226 | return nil, err 227 | } 228 | 229 | out, err := s.db.Query(ctx, &dynamodb.QueryInput{ 230 | ExpressionAttributeNames: expr.Names(), 231 | ExpressionAttributeValues: expr.Values(), 232 | KeyConditionExpression: expr.KeyCondition(), 233 | TableName: aws.String(s.table), 234 | IndexName: aws.String("ByLocation"), 235 | }) 236 | if err != nil { 237 | return nil, err 238 | } 239 | var ids []string 240 | 241 | for _, item := range out.Items { 242 | var si sensorItem 243 | attributevalue.UnmarshalMap(item, &si) 244 | ids = append(ids, strings.TrimLeft(si.ID, "SENSOR#")) 245 | } 246 | return ids, nil 247 | } 248 | -------------------------------------------------------------------------------- /episode8/v2/sensors/sensors_test.go: -------------------------------------------------------------------------------- 1 | package sensors_test 2 | 3 | import ( 4 | "context" 5 | "testing" 6 | "time" 7 | 8 | "dynamodb-with-go/episode8/v2/sensors" 9 | "dynamodb-with-go/pkg/dynamo" 10 | 11 | "github.com/stretchr/testify/assert" 12 | ) 13 | 14 | func TestSensors(t *testing.T) { 15 | ctx := context.Background() 16 | 17 | sensor := sensors.Sensor{ 18 | ID: "sensor-1", 19 | City: "Poznan", 20 | Building: "A", 21 | Floor: "1", 22 | Room: "123", 23 | } 24 | 25 | t.Run("register sensor, get sensor", func(t *testing.T) { 26 | tableName := "SensorsTable" 27 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 28 | defer cleanup() 29 | manager := sensors.NewManager(db, tableName) 30 | 31 | err := manager.Register(ctx, sensor) 32 | assert.NoError(t, err) 33 | 34 | returned, err := manager.Get(ctx, "sensor-1") 35 | assert.NoError(t, err) 36 | assert.Equal(t, sensor, returned) 37 | }) 38 | 39 | t.Run("do not allow to register many times", func(t *testing.T) { 40 | tableName := "SensorsTable" 41 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 42 | defer cleanup() 43 | manager := sensors.NewManager(db, tableName) 44 | 45 | err := manager.Register(ctx, sensor) 46 | assert.NoError(t, err) 47 | 48 | err = manager.Register(ctx, sensor) 49 | assert.EqualError(t, err, "already registered") 50 | }) 51 | 52 | t.Run("save new reading", func(t *testing.T) { 53 | tableName := "SensorsTable" 54 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 55 | defer cleanup() 56 | manager := sensors.NewManager(db, tableName) 57 | 58 | err := manager.Register(ctx, sensor) 59 | assert.NoError(t, err) 60 | 61 | err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.67", ReadAt: time.Now()}) 62 | assert.NoError(t, err) 63 | 64 | _, latest, err := manager.LatestReadings(ctx, "sensor-1", 1) 65 | assert.NoError(t, err) 66 | assert.Equal(t, "0.67", latest[0].Value) 67 | }) 68 | 69 | t.Run("get last readings and sensor", func(t *testing.T) { 70 | tableName := "SensorsTable" 71 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 72 | defer cleanup() 73 | manager := sensors.NewManager(db, tableName) 74 | 75 | err := manager.Register(ctx, sensor) 76 | 77 | assert.NoError(t, err) 78 | 79 | err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.3", ReadAt: time.Now().Add(-20 * time.Second)}) 80 | assert.NoError(t, err) 81 | err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.5", ReadAt: time.Now().Add(-10 * time.Second)}) 82 | assert.NoError(t, err) 83 | err = manager.SaveReading(ctx, sensors.Reading{SensorID: "sensor-1", Value: "0.67", ReadAt: time.Now()}) 84 | assert.NoError(t, err) 85 | 86 | sensor, latest, err := manager.LatestReadings(ctx, "sensor-1", 2) 87 | assert.NoError(t, err) 88 | assert.Len(t, latest, 2) 89 | assert.Equal(t, "0.67", latest[0].Value) 90 | assert.Equal(t, "0.5", latest[1].Value) 91 | assert.Equal(t, "sensor-1", sensor.ID) 92 | }) 93 | 94 | t.Run("get by sensors by location", func(t *testing.T) { 95 | tableName := "SensorsTable" 96 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "../template.yml") 97 | defer cleanup() 98 | manager := sensors.NewManager(db, tableName) 99 | 100 | err := manager.Register(ctx, sensors.Sensor{ID: "sensor-1", City: "Poznan", Building: "A", Floor: "1", Room: "2"}) 101 | err = manager.Register(ctx, sensors.Sensor{ID: "sensor-2", City: "Poznan", Building: "A", Floor: "2", Room: "4"}) 102 | err = manager.Register(ctx, sensors.Sensor{ID: "sensor-3", City: "Poznan", Building: "A", Floor: "2", Room: "5"}) 103 | 104 | ids, err := manager.GetSensors(ctx, sensors.Location{City: "Poznan", Building: "A", Floor: "2"}) 105 | assert.NoError(t, err) 106 | assert.Len(t, ids, 2) 107 | assert.Contains(t, ids, "sensor-2") 108 | assert.Contains(t, ids, "sensor-3") 109 | }) 110 | } 111 | -------------------------------------------------------------------------------- /episode8/v2/template.yml: -------------------------------------------------------------------------------- 1 | 2 | Resources: 3 | SensorsTable: 4 | Type: AWS::DynamoDB::Table 5 | Properties: 6 | AttributeDefinitions: 7 | - AttributeName: pk 8 | AttributeType: S 9 | - AttributeName: sk 10 | AttributeType: S 11 | - AttributeName: gsi_pk 12 | AttributeType: S 13 | - AttributeName: gsi_sk 14 | AttributeType: S 15 | KeySchema: 16 | - AttributeName: pk 17 | KeyType: HASH 18 | - AttributeName: sk 19 | KeyType: RANGE 20 | GlobalSecondaryIndexes: 21 | - IndexName: ByLocation 22 | KeySchema: 23 | - AttributeName: gsi_pk 24 | KeyType: HASH 25 | - AttributeName: gsi_sk 26 | KeyType: RANGE 27 | Projection: 28 | ProjectionType: ALL 29 | BillingMode: PAY_PER_REQUEST 30 | TableName: SensorsTable 31 | -------------------------------------------------------------------------------- /episode9/post.md: -------------------------------------------------------------------------------- 1 | # DynamoDB with Go #9 2 | 3 | Here is the scenario for this episode. There is a toggle, and it can be switched on and off. Whenever toggle is switched, an event is published and we are in charge of consuming that event. Our task is to save switches of state of the toggle and to be able to tell whether it is on or off at the moment. Bad news is that events can arrive out of order - an old event can appear at any time, and we need to be able to reject it. 4 | 5 | It may seem like fictitious example but is 100% real. I dealt with this kind of problem couple of weeks ago. Let me show you how it can be done with the DynamoDB! 6 | 7 | We really have simple access pattern here. We want to obtain the latest state of the toggle. Additionally, we should be able to obtain log of switches that happened. Let's jump into the tests that define what we really want to do. 8 | 9 | ## [Test suite](#test-suite) 10 | 11 | ```go 12 | t.Run("save toggle", func(t *testing.T) { 13 | tableName := "ToggleStateTable" 14 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 15 | defer cleanup() 16 | 17 | toggle := NewToggle(db, tableName) 18 | err := toggle.Save(ctx, Switch{ID: "123", State: true, CreatedAt: time.Now()}) 19 | assert.NoError(t, err) 20 | 21 | s, err := toggle.Latest(ctx, "123") 22 | assert.NoError(t, err) 23 | assert.Equal(t, s.State, true) 24 | }) 25 | ``` 26 | 27 | Can it be simpler? I don't think so - save it first - retrieve later. We want more however, next test proves that we can save many switches, and we can retrieve the latest one. 28 | 29 | ```go 30 | t.Run("save toggles, retrieve latest", func(t *testing.T) { 31 | tableName := "ToggleStateTable" 32 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 33 | defer cleanup() 34 | 35 | toggle := NewToggle(db, tableName) 36 | now := time.Now() 37 | err := toggle.Save(ctx, Switch{ID: "123", State: true, CreatedAt: now}) 38 | assert.NoError(t, err) 39 | 40 | err = toggle.Save(ctx, Switch{ID: "123", State: false, CreatedAt: now.Add(10 * time.Second)}) 41 | assert.NoError(t, err) 42 | 43 | s, err := toggle.Latest(ctx, "123") 44 | assert.NoError(t, err) 45 | assert.Equal(t, s.State, false) 46 | }) 47 | ``` 48 | 49 | Last test shows that out of order events are not taken into account. If an old event arrives, it doesn't influence the latest state. 50 | 51 | ```go 52 | t.Run("drop out of order switch", func(t *testing.T) { 53 | tableName := "ToggleStateTable" 54 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 55 | defer cleanup() 56 | 57 | toggle := NewToggle(db, tableName) 58 | now := time.Now() 59 | err := toggle.Save(ctx, Switch{ID: "123", State: true, CreatedAt: now}) 60 | assert.NoError(t, err) 61 | 62 | err = toggle.Save(ctx, Switch{ID: "123", State: false, CreatedAt: now.Add(-10 * time.Second)}) 63 | assert.NoError(t, err) 64 | 65 | s, err := toggle.Latest(ctx, "123") 66 | assert.NoError(t, err) 67 | assert.Equal(t, s.State, true) 68 | }) 69 | ``` 70 | 71 | This is all we want from `the system`. In the next section I am going to provide you with the solution for this problem that I came up with. If you are able to provide more elegant solution, more efficient one or maybe just ...better, feel free to share it with me! I am keen to look at the problem at a different angle! 72 | 73 | ## [A solution](#a-solution) 74 | 75 | Let's recap. Whenever new `Switch` is consumed by our function I want to append it at the end of my "log" but only if it is the newest one. 76 | 77 | We do not have any table yet! We need to fix that. The table will have Partition Key called `pk`, and Sort Key called `sk`. As you saw in test cases the toggle has ID. Let's use that ID as the PK for every item connected with that toggle. In terms of SK - I want to use it twofold. First of all we will have log items. Each log item will have SK that starts with `READ` prefix followed by time of given switch. There will be additional special item with SK with constant "LATEST_SWITCH". This item will be used both to keep the order when writing a log item and when retrieving the latest switch. 78 | 79 | ## [Coding time](#coding-time) 80 | 81 | ```go 82 | func (t *Toggle) Save(ctx context.Context, s Switch) error { 83 | item := s.asItem() 84 | ``` 85 | 86 | We'll start with keeping details unexported. `Switch` is a public thing, let's not pollute it with implementation details. The DynamoDB on the other hand needs to know how to deal with switches. We can also have different contents of `sk`. Because of that we have `asItem()` method. 87 | 88 | ```go 89 | func (s Switch) asLogItem() switchItem { 90 | return switchItem{ 91 | PK: s.ID, 92 | SK: "SWITCH#" + s.CreatedAt.Format(time.RFC3339Nano), 93 | CreatedAt: s.CreatedAt, 94 | State: s.State, 95 | } 96 | } 97 | ``` 98 | 99 | What is `switchItem`? I am glad you asked. It knows how to marshal/unmarshal items. 100 | 101 | ```go 102 | type switchItem struct { 103 | PK string `dynamodbav:"pk"` 104 | SK string `dynamodbav:"sk"` 105 | 106 | State bool `dynamodbav:"state"` 107 | CreatedAt time.Time `dynamodbav:"created_at"` 108 | } 109 | ``` 110 | 111 | Since we are speaking about marshaling we need to convert our data into the format that the DynamoDB understands. 112 | 113 | ```go 114 | attrs, err := attributevalue.MarshalMap(item) 115 | ``` 116 | 117 | Next thing is the most complicated expression we've ever seen in the DynamoDB with Go because it combines condition and update. 118 | 119 | ```go 120 | expr, err := expression.NewBuilder(). 121 | WithCondition(expression.LessThan(expression.Name("created_at"), expression.Value(item.CreatedAt))). 122 | WithUpdate(expression. 123 | Set(expression.Name("created_at"), expression.Value(item.CreatedAt)). 124 | Set(expression.Name("state"), expression.Value(item.State))). 125 | Build() 126 | ``` 127 | 128 | What is says is 129 | > Please update `created_at` field and `state` field but only if item we are inserting is younger than what is in the DynamoDB. 130 | 131 | ```go 132 | _, err = t.db.TransactWriteItems(ctx, &dynamodb.TransactWriteItemsInput{ 133 | TransactItems: []types.TransactWriteItem{ 134 | { 135 | Update: &types.Update{ 136 | Key: map[string]types.AttributeValue{ 137 | "pk": &types.AttributeValueMemberS{Value: item.PK}, 138 | "sk": &types.AttributeValueMemberS{Value: "LATEST_SWITCH"}, 139 | }, 140 | ExpressionAttributeNames: expr.Names(), 141 | ExpressionAttributeValues: expr.Values(), 142 | ConditionExpression: expr.Condition(), 143 | TableName: aws.String(t.table), 144 | UpdateExpression: expr.Update(), 145 | ReturnValuesOnConditionCheckFailure: types.ReturnValuesOnConditionCheckFailure(types.ReturnValueAllOld), 146 | }, 147 | }, 148 | { 149 | Put: &types.Put{ 150 | Item: attrs, 151 | TableName: aws.String(t.table), 152 | }, 153 | }, 154 | }, 155 | }) 156 | ``` 157 | 158 | We are using transaction because we want to append log entry (Put) and update the latest state of the toggle (Update) but only if the condition holds true for the latest state. 159 | 160 | Let's put in english the condition mixed with this transaction. This is what is says: 161 | > I want to update the `created_at` and the `state` fields on the item that represents the latest state of the switch, but only if what I am trying to put into the DynamoDB is younger than what is already there. Additionally, I want to append the log item (because I want to have history of what happened to that toggle). 162 | 163 | One more thing - `ReturnValuesOnConditionCheckFailure` parameter. It is crucial because it allows us to recognize what happened after transaction failure. It can fail for two reasons: 164 | 1. condition failed - we received out of order event - we need to discard it 165 | 2. database doesn't have any item with given `pk` - it is the first write for given partition key 166 | 167 | ```go 168 | if err == nil { 169 | return nil 170 | } 171 | ``` 172 | 173 | We might not have an error at all - that means that we've just appended new log entry with an update to the latest state. 174 | 175 | ```go 176 | var transactionCanelled *types.TransactionCanceledException 177 | if !errors.As(err, &transactionCanelled) { 178 | return err 179 | } 180 | ``` 181 | 182 | We might have an error that wasn't anticipated. In that case - application blows up, and we have an incident to handle. 183 | 184 | ```go 185 | if len(transactionCanelled.CancellationReasons[0].Item) > 0 { 186 | return nil 187 | } 188 | ``` 189 | 190 | Based on that condition we can reason that we received an out of order event. Why? Because we filled `ReturnValuesOnConditionCheckFailure` parameter with `ALL_OLD` value. This means that when transaction fails the value of `transactionCanelled.CancellationReasons[0].Item` will be an item that was in the DynamoDB before our action. If that value is not empty this means that there is an item in the DynamoDB for given Partition Key. Since we have two reasons for transaction failure we can - by elimination - conclude that we'ce received an out of order event. 191 | 192 | Out of order event isn't saved into the DynamoDB so we exit immediately. Now we need to handle the situation when transaction failed because it's the first time DynamoDB sees such Partition Key. 193 | 194 | ```go 195 | expr, err = expression.NewBuilder(). 196 | WithCondition( 197 | expression.Not(expression.And( 198 | expression.Equal(expression.Name("pk"), expression.Value(item.PK)), 199 | expression.Equal(expression.Name("sk"), expression.Value("LATEST_SWITCH")), 200 | ))).Build() 201 | ``` 202 | 203 | This condition is responsible for making sure that the DynamoDB didn't save `LATEST_SWITCH` for our Partition Key yet. It's a possibility because between failed transaction and now there is time difference. Some other process could have saved such item in between. 204 | 205 | We also need to create an item representing the latest state of the toggle. 206 | 207 | ```go 208 | latestAttrs, err := attributevalue.MarshalMap(s.asLatestItem()) 209 | ``` 210 | 211 | It's similar to `asLogItem` but it sets `sk` to `LATEST_SWITCH`. Next thing we do is yet another transaction. 212 | 213 | ```go 214 | _, err = t.db.TransactWriteItems(ctx, &dynamodb.TransactWriteItemsInput{ 215 | TransactItems: []types.TransactWriteItem{ 216 | { 217 | Put: &types.Put{ 218 | ConditionExpression: expr.Condition(), 219 | ExpressionAttributeNames: expr.Names(), 220 | ExpressionAttributeValues: expr.Values(), 221 | Item: latestAttrs, 222 | TableName: aws.String(t.table), 223 | }, 224 | }, 225 | { 226 | Put: &types.Put{ 227 | Item: attrs, 228 | TableName: aws.String(t.table), 229 | }, 230 | }, 231 | }, 232 | }) 233 | ``` 234 | What we want to achieve here is appending first log item and create the latest state of the toggle which holds the same information as the only log item we've saved. 235 | 236 | What can happen now? 237 | ```go 238 | if err == nil { 239 | return nil 240 | } 241 | ``` 242 | We might have no error - condition passed and both items were saved. This DynamoDB call can fail as well. 243 | ```go 244 | if !errors.As(err, &transactionCanelled) { 245 | return err 246 | } 247 | ``` 248 | If it does - and the reason isn't transaction failure - something wrong happened, and we return with an error. If however transaction failed - first switch for the toggle was saved but not by us. What we can do is to call this whole function again. It is completely save and won't create an infinite loop because the `Switch` is either older or younger than what is saved in the Dynamo. If it is younger it will be saved. If it's older - it will be rejected. 249 | 250 | ```go 251 | return t.Save(ctx, s) 252 | ``` 253 | 254 | Very often - when writing is complex - reading must be trivial. This is the case in here! 255 | ```go 256 | func (t *Toggle) Latest(ctx context.Context, userID string) (Switch, error) { 257 | out, err := t.db.GetItem(ctx, &dynamodb.GetItemInput{ 258 | Key: map[string]types.AttributeValue{ 259 | "pk": &types.AttributeValueMemberS{Value: userID}, 260 | "sk": &types.AttributeValueMemberS{Value: "LATEST_SWITCH"}, 261 | }, 262 | TableName: aws.String(t.table), 263 | }) 264 | if err != nil { 265 | return Switch{}, err 266 | } 267 | if len(out.Item) == 0 { 268 | return Switch{}, errors.New("not found") 269 | } 270 | 271 | var item switchItem 272 | err = attributevalue.UnmarshalMap(out.Item, &item) 273 | return item.asSwitch(), err 274 | } 275 | ``` 276 | 277 | We retrieve the latest state of the toggle, unmarshal it and return it to the client! 278 | 279 | ## Summary 280 | Fairly easy test suite received very complex implementation. If you know how to satisfy the same test suite with simpler implementation - let me know - I would love to know how! Nevertheless, we learned two important pieces of DynamoDB API. First of all we know that expression API can combine both updates and conditions. Other than that - now we know that after transaction fails - we can use `ReturnValuesOnConditionCheckFailure` to obtain more insight on what really happened. -------------------------------------------------------------------------------- /episode9/template.yml: -------------------------------------------------------------------------------- 1 | Resources: 2 | ToggleStateTable: 3 | Type: AWS::DynamoDB::Table 4 | Properties: 5 | AttributeDefinitions: 6 | - AttributeName: pk 7 | AttributeType: S 8 | - AttributeName: sk 9 | AttributeType: S 10 | KeySchema: 11 | - AttributeName: pk 12 | KeyType: HASH 13 | - AttributeName: sk 14 | KeyType: RANGE 15 | BillingMode: PAY_PER_REQUEST 16 | TableName: ToggleStateTable 17 | -------------------------------------------------------------------------------- /episode9/toggle.go: -------------------------------------------------------------------------------- 1 | package episode9 2 | 3 | import ( 4 | "context" 5 | "errors" 6 | "time" 7 | 8 | "github.com/aws/aws-sdk-go-v2/aws" 9 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue" 10 | "github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression" 11 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 12 | "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" 13 | ) 14 | 15 | type Toggle struct { 16 | db *dynamodb.Client 17 | table string 18 | } 19 | 20 | type Switch struct { 21 | ID string 22 | State bool 23 | CreatedAt time.Time 24 | } 25 | 26 | type switchItem struct { 27 | PK string `dynamodbav:"pk"` 28 | SK string `dynamodbav:"sk"` 29 | 30 | State bool `dynamodbav:"state"` 31 | CreatedAt time.Time `dynamodbav:"created_at"` 32 | } 33 | 34 | func (s switchItem) asSwitch() Switch { 35 | return Switch{ 36 | ID: s.PK, 37 | State: s.State, 38 | CreatedAt: s.CreatedAt, 39 | } 40 | } 41 | 42 | func (s Switch) asLogItem() switchItem { 43 | return switchItem{ 44 | PK: s.ID, 45 | SK: "SWITCH#" + s.CreatedAt.Format(time.RFC3339Nano), 46 | CreatedAt: s.CreatedAt, 47 | State: s.State, 48 | } 49 | } 50 | 51 | func (s Switch) asLatestItem() switchItem { 52 | return switchItem{ 53 | PK: s.ID, 54 | SK: "LATEST_SWITCH", 55 | CreatedAt: s.CreatedAt, 56 | State: s.State, 57 | } 58 | } 59 | 60 | func NewToggle(db *dynamodb.Client, table string) *Toggle { 61 | return &Toggle{db: db, table: table} 62 | } 63 | 64 | func (t *Toggle) Save(ctx context.Context, s Switch) error { 65 | item := s.asLogItem() 66 | attrs, err := attributevalue.MarshalMap(item) 67 | if err != nil { 68 | return err 69 | } 70 | 71 | expr, err := expression.NewBuilder(). 72 | WithCondition(expression.LessThan(expression.Name("created_at"), expression.Value(item.CreatedAt))). 73 | WithUpdate(expression. 74 | Set(expression.Name("created_at"), expression.Value(item.CreatedAt)). 75 | Set(expression.Name("state"), expression.Value(item.State))). 76 | Build() 77 | 78 | if err != nil { 79 | return err 80 | } 81 | _, err = t.db.TransactWriteItems(ctx, &dynamodb.TransactWriteItemsInput{ 82 | TransactItems: []types.TransactWriteItem{ 83 | { 84 | Update: &types.Update{ 85 | Key: map[string]types.AttributeValue{ 86 | "pk": &types.AttributeValueMemberS{Value: item.PK}, 87 | "sk": &types.AttributeValueMemberS{Value: "LATEST_SWITCH"}, 88 | }, 89 | ExpressionAttributeNames: expr.Names(), 90 | ExpressionAttributeValues: expr.Values(), 91 | ConditionExpression: expr.Condition(), 92 | TableName: aws.String(t.table), 93 | UpdateExpression: expr.Update(), 94 | ReturnValuesOnConditionCheckFailure: types.ReturnValuesOnConditionCheckFailure(types.ReturnValueAllOld), 95 | }, 96 | }, 97 | { 98 | Put: &types.Put{ 99 | Item: attrs, 100 | TableName: aws.String(t.table), 101 | }, 102 | }, 103 | }, 104 | }) 105 | 106 | if err == nil { 107 | return nil 108 | } 109 | var transactionCanelled *types.TransactionCanceledException 110 | if !errors.As(err, &transactionCanelled) { 111 | return err 112 | } 113 | 114 | if len(transactionCanelled.CancellationReasons[0].Item) > 0 { 115 | return nil 116 | } 117 | 118 | expr, err = expression.NewBuilder(). 119 | WithCondition( 120 | expression.Not(expression.And( 121 | expression.Equal(expression.Name("pk"), expression.Value(item.PK)), 122 | expression.Equal(expression.Name("sk"), expression.Value("LATEST_SWITCH")), 123 | ))).Build() 124 | if err != nil { 125 | return err 126 | } 127 | 128 | latestAttrs, err := attributevalue.MarshalMap(s.asLatestItem()) 129 | 130 | _, err = t.db.TransactWriteItems(ctx, &dynamodb.TransactWriteItemsInput{ 131 | TransactItems: []types.TransactWriteItem{ 132 | { 133 | Put: &types.Put{ 134 | ConditionExpression: expr.Condition(), 135 | ExpressionAttributeNames: expr.Names(), 136 | ExpressionAttributeValues: expr.Values(), 137 | Item: latestAttrs, 138 | TableName: aws.String(t.table), 139 | }, 140 | }, 141 | { 142 | Put: &types.Put{ 143 | Item: attrs, 144 | TableName: aws.String(t.table), 145 | }, 146 | }, 147 | }, 148 | }) 149 | if err == nil { 150 | return nil 151 | } 152 | if !errors.As(err, &transactionCanelled) { 153 | return err 154 | } 155 | 156 | return t.Save(ctx, s) 157 | } 158 | 159 | func (t *Toggle) Latest(ctx context.Context, userID string) (Switch, error) { 160 | out, err := t.db.GetItem(ctx, &dynamodb.GetItemInput{ 161 | Key: map[string]types.AttributeValue{ 162 | "pk": &types.AttributeValueMemberS{Value: userID}, 163 | "sk": &types.AttributeValueMemberS{Value: "LATEST_SWITCH"}, 164 | }, 165 | TableName: aws.String(t.table), 166 | }) 167 | if err != nil { 168 | return Switch{}, err 169 | } 170 | if len(out.Item) == 0 { 171 | return Switch{}, errors.New("not found") 172 | } 173 | 174 | var item switchItem 175 | err = attributevalue.UnmarshalMap(out.Item, &item) 176 | return item.asSwitch(), err 177 | } 178 | -------------------------------------------------------------------------------- /episode9/toggle_test.go: -------------------------------------------------------------------------------- 1 | package episode9 2 | 3 | import ( 4 | "context" 5 | "dynamodb-with-go/pkg/dynamo" 6 | "testing" 7 | "time" 8 | 9 | "github.com/stretchr/testify/assert" 10 | ) 11 | 12 | func TestToggle(t *testing.T) { 13 | ctx := context.Background() 14 | 15 | t.Run("save toggle", func(t *testing.T) { 16 | tableName := "ToggleStateTable" 17 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 18 | defer cleanup() 19 | 20 | toggle := NewToggle(db, tableName) 21 | err := toggle.Save(ctx, Switch{ID: "123", State: true, CreatedAt: time.Now()}) 22 | assert.NoError(t, err) 23 | 24 | s, err := toggle.Latest(ctx, "123") 25 | assert.NoError(t, err) 26 | assert.Equal(t, s.State, true) 27 | }) 28 | 29 | t.Run("save toggles, retrieve latest", func(t *testing.T) { 30 | tableName := "ToggleStateTable" 31 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 32 | defer cleanup() 33 | 34 | toggle := NewToggle(db, tableName) 35 | now := time.Now() 36 | err := toggle.Save(ctx, Switch{ID: "123", State: true, CreatedAt: now}) 37 | assert.NoError(t, err) 38 | 39 | err = toggle.Save(ctx, Switch{ID: "123", State: false, CreatedAt: now.Add(10 * time.Second)}) 40 | assert.NoError(t, err) 41 | 42 | s, err := toggle.Latest(ctx, "123") 43 | assert.NoError(t, err) 44 | assert.Equal(t, s.State, false) 45 | }) 46 | 47 | t.Run("drop out of order switch", func(t *testing.T) { 48 | tableName := "ToggleStateTable" 49 | db, cleanup := dynamo.SetupTable(t, ctx, tableName, "./template.yml") 50 | defer cleanup() 51 | 52 | toggle := NewToggle(db, tableName) 53 | now := time.Now() 54 | err := toggle.Save(ctx, Switch{ID: "123", State: true, CreatedAt: now}) 55 | assert.NoError(t, err) 56 | 57 | err = toggle.Save(ctx, Switch{ID: "123", State: false, CreatedAt: now.Add(-10 * time.Second)}) 58 | assert.NoError(t, err) 59 | 60 | s, err := toggle.Latest(ctx, "123") 61 | assert.NoError(t, err) 62 | assert.Equal(t, s.State, true) 63 | }) 64 | } 65 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module dynamodb-with-go 2 | 3 | go 1.14 4 | 5 | require ( 6 | github.com/aws/aws-sdk-go-v2 v1.2.1 7 | github.com/aws/aws-sdk-go-v2/config v1.1.2 8 | github.com/aws/aws-sdk-go-v2/credentials v1.1.2 9 | github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue v1.0.3 10 | github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression v1.0.3 11 | github.com/aws/aws-sdk-go-v2/service/dynamodb v1.1.2 12 | github.com/awslabs/goformation v1.4.1 13 | github.com/davecgh/go-spew v1.1.1 14 | github.com/google/uuid v1.1.2 15 | github.com/stretchr/testify v1.5.1 16 | golang.org/x/net v0.0.0-20200202094626-16171245cfb2 // indirect 17 | ) 18 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/aws/aws-sdk-go-v2 v1.2.1 h1:055XAi+MtmhyYX161p+jWRibkCb9YpI2ymXZiW1dwVY= 2 | github.com/aws/aws-sdk-go-v2 v1.2.1/go.mod h1:hTQc/9pYq5bfFACIUY9tc/2SYWd9Vnmw+testmuQeRY= 3 | github.com/aws/aws-sdk-go-v2/config v1.1.2 h1:H2r6cwMvvINFpEC55Y7jcNaR/oc7zYIChrG2497wmBI= 4 | github.com/aws/aws-sdk-go-v2/config v1.1.2/go.mod h1:77yIk+qmCS/94JlxbwV1d+YEyu6Z8FBlCGcSz3TdM6A= 5 | github.com/aws/aws-sdk-go-v2/credentials v1.1.2 h1:YoNqfhxAJGZI+lStIbqgx30UcCqQ86fr7FjTLUvrFOc= 6 | github.com/aws/aws-sdk-go-v2/credentials v1.1.2/go.mod h1:hofjw//lM0XLplgvzPPMA7oD0doQU1QpaIK1nweEEWg= 7 | github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue v1.0.3 h1:xJpS5hGycvWiKDto1ujoeJWzGmRydwbTaEoKUCdJJ28= 8 | github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue v1.0.3/go.mod h1:jdqFlNhMlIITa/4ZYl+GP+qrETsHIuDgnmXC1a76Dp0= 9 | github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression v1.0.3 h1:j3VbwezEL3n183erSP5feZQGnSkkpUKDzgPkK/C3bhg= 10 | github.com/aws/aws-sdk-go-v2/feature/dynamodb/expression v1.0.3/go.mod h1:Us+BB6iLRemUSwD898Jtdq40rP3sF+WfRAb73oWKjls= 11 | github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.0.3 h1:d3bKAGy4XdJyK8hz3Nx3WJJ4TCmYp2498G4mFY5wly0= 12 | github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.0.3/go.mod h1:Zr1Mj+KUMGVQ+WJvTT68EZJxqhjiie2PWSPGEUPaNY0= 13 | github.com/aws/aws-sdk-go-v2/service/dynamodb v1.1.2 h1:Ow9GseCWcUqFGI0t67OqNT+8hB45Cf3M9zRQkMYEnjU= 14 | github.com/aws/aws-sdk-go-v2/service/dynamodb v1.1.2/go.mod h1:xlIykhO1SzJtiDIbIWxQAYRZFROcJUiuqfU+F3c1YoY= 15 | github.com/aws/aws-sdk-go-v2/service/dynamodbstreams v1.1.2 h1:zHusZHxl/6/spzqY+oKshMUhzorOCzb0pE/1iez+j/Y= 16 | github.com/aws/aws-sdk-go-v2/service/dynamodbstreams v1.1.2/go.mod h1:Q0dLW7aL6QT/Iuc53pASEdEJatYJO0u7vqaj8G9dRB8= 17 | github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.0.2 h1:GO0pL4QvQmA0fXJe3MHVO+emtg31MYq5/8sebSWgE6A= 18 | github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.0.2/go.mod h1:bYl7lGFQQdHia3uMQH4p6ImnuOeDNeUoydoXM5x8Yzw= 19 | github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.0.3 h1:dST4y8pZKZdTPs4uwXmGCJmpycz1SHKmCSIhf3GqHEo= 20 | github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.0.3/go.mod h1:C50Z41fJaJ7WgaeeCulOGAU3q4+4se4B3uOPFdhBi2I= 21 | github.com/aws/aws-sdk-go-v2/service/sso v1.1.2 h1:9BnjX/ALn5uLo2DbgkwMpUkPL1VLQVBXcjZxqJBhf44= 22 | github.com/aws/aws-sdk-go-v2/service/sso v1.1.2/go.mod h1:5yU1oE3+CVYYLUsaHt2AVU3CJJZ6ER4pwsrRD1L2KSc= 23 | github.com/aws/aws-sdk-go-v2/service/sts v1.1.2 h1:7Kxqov7uQeP8WUEO0iHz3j9Bh0E1rJrn6cf/OGfcDds= 24 | github.com/aws/aws-sdk-go-v2/service/sts v1.1.2/go.mod h1:zu7rotIY9P4Aoc6ytqLP9jeYrECDHUODB5Gbp+BSHl8= 25 | github.com/aws/smithy-go v1.2.0 h1:0PoGBWXkXDIyVdPaZW9gMhaGzj3UOAgTdiVoHuuZAFA= 26 | github.com/aws/smithy-go v1.2.0/go.mod h1:SObp3lf9smib00L/v3U2eAKG8FyQ7iLrJnQiAmR5n+E= 27 | github.com/awslabs/goformation v1.4.1 h1:jws9kTrcI53Hq2COJAy50uAhgxB5N/Enb9Gmclr/MP4= 28 | github.com/awslabs/goformation v1.4.1/go.mod h1:HezUyH08DSwwGn3GioVXWZYUhkdvC+oGJ7ya7vBRm7k= 29 | github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8= 30 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 31 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 32 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 33 | github.com/golang/protobuf v1.2.0 h1:P3YflyNX/ehuJFLhxviNdFxQPkGK5cDcApsge1SqnvM= 34 | github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 35 | github.com/google/go-cmp v0.5.4 h1:L8R9j+yAqZuZjsqh/z+F1NCffTKKLShY6zXTItVIZ8M= 36 | github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 37 | github.com/google/uuid v1.1.2 h1:EVhdT+1Kseyi1/pUmXKaFxYsDNy9RQYkMWRH68J/W7Y= 38 | github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= 39 | github.com/imdario/mergo v0.3.6 h1:xTNEAn+kxVO7dTZGu0CegyqKZmoWFI0rF8UxjlB2d28= 40 | github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA= 41 | github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg= 42 | github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo= 43 | github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8= 44 | github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U= 45 | github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI= 46 | github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= 47 | github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= 48 | github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE= 49 | github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= 50 | github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE= 51 | github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= 52 | github.com/onsi/ginkgo v1.5.0 h1:uZr+v/TFDdYkdA+j02sPO1kA5owrfjBGCJAogfIyThE= 53 | github.com/onsi/ginkgo v1.5.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 54 | github.com/onsi/gomega v1.2.0 h1:tQjc4uvqBp0z424R9V/S2L18penoUiwZftoY0t48IZ4= 55 | github.com/onsi/gomega v1.2.0/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= 56 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 57 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 58 | github.com/sanathkr/go-yaml v0.0.0-20170819195128-ed9d249f429b h1:jUK33OXuZP/l6babJtnLo1qsGvq6G9so9KMflGAm4YA= 59 | github.com/sanathkr/go-yaml v0.0.0-20170819195128-ed9d249f429b/go.mod h1:8458kAagoME2+LN5//WxE71ysZ3B7r22fdgb7qVmXSY= 60 | github.com/sanathkr/yaml v0.0.0-20170819201035-0056894fa522 h1:fOCp11H0yuyAt2wqlbJtbyPzSgaxHTv8uN1pMpkG1t8= 61 | github.com/sanathkr/yaml v0.0.0-20170819201035-0056894fa522/go.mod h1:tQTYKOQgxoH3v6dEmdHiz4JG+nbxWwM5fgPQUpSZqVQ= 62 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 63 | github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= 64 | github.com/stretchr/testify v1.5.1 h1:nOGnQDM7FYENwehXlg/kFVnos3rEvtKTjRvOWSzb6H4= 65 | github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA= 66 | github.com/xeipuuv/gojsonpointer v0.0.0-20170225233418-6fe8760cad35 h1:0TnXeVP6mx+A4CBf8cQVkQfkhyGBQCmJcT4g6zKzm7M= 67 | github.com/xeipuuv/gojsonpointer v0.0.0-20170225233418-6fe8760cad35/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU= 68 | github.com/xeipuuv/gojsonreference v0.0.0-20150808065054-e02fc20de94c h1:XZWnr3bsDQWAZg4Ne+cPoXRPILrNlPNQfxBuwLl43is= 69 | github.com/xeipuuv/gojsonreference v0.0.0-20150808065054-e02fc20de94c/go.mod h1:GwrjFmJcFw6At/Gs6z4yjiIwzuJ1/+UwLxMQDVQXShQ= 70 | github.com/xeipuuv/gojsonschema v0.0.0-20181112162635-ac52e6811b56 h1:yhqBHs09SmmUoNOHc9jgK4a60T3XFRtPAkYxVnqgY50= 71 | github.com/xeipuuv/gojsonschema v0.0.0-20181112162635-ac52e6811b56/go.mod h1:5yf86TLmAcydyeJq5YvxkGPE2fm/u4myDekKRoLuqhs= 72 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 73 | golang.org/x/net v0.0.0-20170809000501-1c05540f6879/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 74 | golang.org/x/net v0.0.0-20200202094626-16171245cfb2 h1:CCH4IOTTfewWjGOlSp+zGcjutRKlBEZQ6wTn8ozI/nI= 75 | golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 76 | golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f h1:wMNYb4v58l5UBM7MYRLPG6ZhfOqbKu7X5eyFl8ZhKvA= 77 | golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 78 | golang.org/x/sys v0.0.0-20170814044513-c84c1ab9fd18/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 79 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a h1:1BGLXjeY4akVXGgbC9HugT3Jv3hCI0z56oJR5vAMgBU= 80 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 81 | golang.org/x/text v0.0.0-20170814122439-e56139fd9c5b/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 82 | golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= 83 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 84 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4= 85 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 86 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= 87 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 88 | gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY= 89 | gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 90 | gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= 91 | gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 92 | gopkg.in/yaml.v2 v2.2.8 h1:obN1ZagJSUGI0Ek/LBmuj4SNLPfIny3KsKFopxRdj10= 93 | gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 94 | -------------------------------------------------------------------------------- /pkg/dynamo/setup.go: -------------------------------------------------------------------------------- 1 | package dynamo 2 | 3 | import ( 4 | "context" 5 | "testing" 6 | "time" 7 | 8 | "github.com/aws/aws-sdk-go-v2/aws" 9 | "github.com/aws/aws-sdk-go-v2/config" 10 | "github.com/aws/aws-sdk-go-v2/credentials" 11 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 12 | "github.com/awslabs/goformation" 13 | ) 14 | 15 | type EndpointResolver struct{} 16 | 17 | func (e EndpointResolver) ResolveEndpoint(region string, options dynamodb.EndpointResolverOptions) (aws.Endpoint, error) { 18 | return aws.Endpoint{URL: "http://localhost:8000"}, nil 19 | } 20 | 21 | func localDynamoDB(t *testing.T) *dynamodb.Client { 22 | cfg, err := config.LoadDefaultConfig(context.TODO(), 23 | config.WithRegion("local"), 24 | config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("local", "local", "local")), 25 | ) 26 | if err != nil { 27 | t.Fatal("could not setup db connection") 28 | } 29 | 30 | db := dynamodb.NewFromConfig(cfg, dynamodb.WithEndpointResolver(EndpointResolver{})) 31 | 32 | ctx, cancel := context.WithTimeout(context.Background(), 1500*time.Millisecond) 33 | defer cancel() 34 | _, err = db.ListTables(ctx, nil) 35 | if err != nil { 36 | t.Fatal("make sure DynamoDB local runs on port :8000", err) 37 | } 38 | return db 39 | } 40 | 41 | // SetupTable creates table defined in the CloudFormation template file under `path`. 42 | // It returns connection to the DynamoDB and cleanup function, that needs to be run after tests. 43 | func SetupTable(t *testing.T, ctx context.Context, tableName, path string) (*dynamodb.Client, func()) { 44 | db := localDynamoDB(t) 45 | tmpl, err := goformation.Open(path) 46 | if err != nil { 47 | t.Fatal(err) 48 | } 49 | 50 | table, err := tmpl.GetAWSDynamoDBTableWithName(tableName) 51 | if err != nil { 52 | t.Fatal(err) 53 | } 54 | input := FromCloudFormationToCreateInput(*table) 55 | _, err = db.CreateTable(ctx, &input) 56 | if err != nil { 57 | t.Fatal(err) 58 | } 59 | return db, func() { 60 | db.DeleteTable(ctx, &dynamodb.DeleteTableInput{TableName: aws.String(tableName)}) 61 | } 62 | } 63 | -------------------------------------------------------------------------------- /pkg/dynamo/setup_test.go: -------------------------------------------------------------------------------- 1 | package dynamo_test 2 | 3 | import ( 4 | "context" 5 | "dynamodb-with-go/pkg/dynamo" 6 | "errors" 7 | "testing" 8 | 9 | "github.com/aws/aws-sdk-go-v2/aws" 10 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 11 | "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" 12 | "github.com/stretchr/testify/assert" 13 | ) 14 | 15 | func TestSetupTable(t *testing.T) { 16 | ctx := context.Background() 17 | db, cleanup := dynamo.SetupTable(t, ctx, "PartitionKeyTable", "./testdata/template.yml") 18 | 19 | out, err := db.DescribeTable(ctx, &dynamodb.DescribeTableInput{ 20 | TableName: aws.String("PartitionKeyTable"), 21 | }) 22 | assert.NoError(t, err) 23 | assert.Equal(t, "PartitionKeyTable", *out.Table.TableName) 24 | assert.Equal(t, "pk", *out.Table.AttributeDefinitions[0].AttributeName) 25 | 26 | cleanup() 27 | _, err = db.DescribeTable(ctx, &dynamodb.DescribeTableInput{ 28 | TableName: aws.String("PartitionKeyTable"), 29 | }) 30 | 31 | var notfound *types.ResourceNotFoundException 32 | assert.True(t, errors.As(err, ¬found)) 33 | 34 | } 35 | -------------------------------------------------------------------------------- /pkg/dynamo/table.go: -------------------------------------------------------------------------------- 1 | package dynamo 2 | 3 | import ( 4 | "github.com/aws/aws-sdk-go-v2/aws" 5 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 6 | "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" 7 | "github.com/awslabs/goformation/cloudformation" 8 | ) 9 | 10 | // FromCloudFormationToCreateInput transforms DynamoDB table from CloudFormation template 11 | // into CreateTableInput struct, that can be used with aws-sdk-go to create the table. 12 | func FromCloudFormationToCreateInput(t cloudformation.AWSDynamoDBTable) dynamodb.CreateTableInput { 13 | var input dynamodb.CreateTableInput 14 | for _, attrs := range t.AttributeDefinitions { 15 | input.AttributeDefinitions = append(input.AttributeDefinitions, types.AttributeDefinition{ 16 | AttributeName: aws.String(attrs.AttributeName), 17 | AttributeType: types.ScalarAttributeType(attrs.AttributeType), 18 | }) 19 | } 20 | for _, key := range t.KeySchema { 21 | input.KeySchema = append(input.KeySchema, types.KeySchemaElement{ 22 | AttributeName: aws.String(key.AttributeName), 23 | KeyType: types.KeyType(key.KeyType), 24 | }) 25 | } 26 | for _, idx := range t.LocalSecondaryIndexes { 27 | if idx.Projection.ProjectionType != "ALL" { 28 | panic("not implemented") 29 | } 30 | indexKeySchema := []types.KeySchemaElement{} 31 | for _, key := range idx.KeySchema { 32 | indexKeySchema = append(indexKeySchema, types.KeySchemaElement{ 33 | AttributeName: aws.String(key.AttributeName), 34 | KeyType: types.KeyType(key.KeyType), 35 | }) 36 | } 37 | input.LocalSecondaryIndexes = append(input.LocalSecondaryIndexes, types.LocalSecondaryIndex{ 38 | IndexName: aws.String(idx.IndexName), 39 | KeySchema: indexKeySchema, 40 | Projection: &types.Projection{ProjectionType: types.ProjectionType(idx.Projection.ProjectionType)}, 41 | }) 42 | } 43 | for _, idx := range t.GlobalSecondaryIndexes { 44 | if idx.Projection.ProjectionType != "ALL" { 45 | panic("not implemented") 46 | } 47 | indexKeySchema := []types.KeySchemaElement{} 48 | for _, key := range idx.KeySchema { 49 | indexKeySchema = append(indexKeySchema, types.KeySchemaElement{ 50 | AttributeName: aws.String(key.AttributeName), 51 | KeyType: types.KeyType(key.KeyType), 52 | }) 53 | } 54 | input.GlobalSecondaryIndexes = append(input.GlobalSecondaryIndexes, types.GlobalSecondaryIndex{ 55 | IndexName: aws.String(idx.IndexName), 56 | KeySchema: indexKeySchema, 57 | Projection: &types.Projection{ProjectionType: types.ProjectionType(idx.Projection.ProjectionType)}, 58 | }) 59 | } 60 | 61 | input.TableName = aws.String(t.TableName) 62 | input.BillingMode = types.BillingMode(t.BillingMode) 63 | return input 64 | } 65 | -------------------------------------------------------------------------------- /pkg/dynamo/table_test.go: -------------------------------------------------------------------------------- 1 | package dynamo_test 2 | 3 | import ( 4 | "dynamodb-with-go/pkg/dynamo" 5 | "testing" 6 | 7 | "github.com/aws/aws-sdk-go-v2/aws" 8 | "github.com/aws/aws-sdk-go-v2/service/dynamodb" 9 | "github.com/aws/aws-sdk-go-v2/service/dynamodb/types" 10 | "github.com/awslabs/goformation" 11 | "github.com/stretchr/testify/assert" 12 | ) 13 | 14 | func TestTable(t *testing.T) { 15 | t.Run("table with partition key only", func(t *testing.T) { 16 | tmpl, err := goformation.Open("./testdata/template.yml") 17 | assert.NoError(t, err) 18 | 19 | table, err := tmpl.GetAWSDynamoDBTableWithName("PartitionKeyTable") 20 | assert.NoError(t, err) 21 | 22 | input := dynamo.FromCloudFormationToCreateInput(*table) 23 | assert.Equal(t, dynamodb.CreateTableInput{ 24 | AttributeDefinitions: []types.AttributeDefinition{ 25 | { 26 | AttributeName: aws.String("pk"), 27 | AttributeType: types.ScalarAttributeTypeS, 28 | }, 29 | }, 30 | BillingMode: types.BillingModePayPerRequest, 31 | KeySchema: []types.KeySchemaElement{ 32 | { 33 | AttributeName: aws.String("pk"), 34 | KeyType: types.KeyTypeHash, 35 | }, 36 | }, 37 | TableName: aws.String("PartitionKeyTable"), 38 | }, input) 39 | }) 40 | 41 | t.Run("table with composite primary key", func(t *testing.T) { 42 | tmpl, err := goformation.Open("./testdata/template.yml") 43 | assert.NoError(t, err) 44 | 45 | table, err := tmpl.GetAWSDynamoDBTableWithName("CompositePrimaryKeyTable") 46 | assert.NoError(t, err) 47 | 48 | input := dynamo.FromCloudFormationToCreateInput(*table) 49 | assert.Equal(t, dynamodb.CreateTableInput{ 50 | AttributeDefinitions: []types.AttributeDefinition{ 51 | { 52 | AttributeName: aws.String("pk"), 53 | AttributeType: types.ScalarAttributeTypeS, 54 | }, 55 | { 56 | AttributeName: aws.String("sk"), 57 | AttributeType: types.ScalarAttributeTypeS, 58 | }, 59 | }, 60 | BillingMode: types.BillingModePayPerRequest, 61 | KeySchema: []types.KeySchemaElement{ 62 | { 63 | AttributeName: aws.String("pk"), 64 | KeyType: types.KeyTypeHash, 65 | }, 66 | { 67 | AttributeName: aws.String("sk"), 68 | KeyType: types.KeyTypeRange, 69 | }, 70 | }, 71 | TableName: aws.String("CompositePrimaryKeyTable"), 72 | }, input) 73 | }) 74 | 75 | t.Run("table with single local secondary indexes", func(t *testing.T) { 76 | tmpl, err := goformation.Open("./testdata/template.yml") 77 | assert.NoError(t, err) 78 | 79 | table, err := tmpl.GetAWSDynamoDBTableWithName("CompositePrimaryKeyAndLocalIndexTable") 80 | assert.NoError(t, err) 81 | 82 | input := dynamo.FromCloudFormationToCreateInput(*table) 83 | assert.Equal(t, dynamodb.CreateTableInput{ 84 | AttributeDefinitions: []types.AttributeDefinition{ 85 | { 86 | AttributeName: aws.String("pk"), 87 | AttributeType: types.ScalarAttributeTypeS, 88 | }, 89 | { 90 | AttributeName: aws.String("sk"), 91 | AttributeType: types.ScalarAttributeTypeS, 92 | }, 93 | { 94 | AttributeName: aws.String("lsi_sk"), 95 | AttributeType: types.ScalarAttributeTypeS, 96 | }, 97 | }, 98 | BillingMode: types.BillingModePayPerRequest, 99 | KeySchema: []types.KeySchemaElement{ 100 | { 101 | AttributeName: aws.String("pk"), 102 | KeyType: types.KeyTypeHash, 103 | }, 104 | { 105 | AttributeName: aws.String("sk"), 106 | KeyType: types.KeyTypeRange, 107 | }, 108 | }, 109 | LocalSecondaryIndexes: []types.LocalSecondaryIndex{ 110 | { 111 | IndexName: aws.String("MyIndex"), 112 | KeySchema: []types.KeySchemaElement{ 113 | { 114 | AttributeName: aws.String("pk"), 115 | KeyType: types.KeyTypeHash, 116 | }, 117 | { 118 | AttributeName: aws.String("lsi_sk"), 119 | KeyType: types.KeyTypeRange, 120 | }, 121 | }, 122 | Projection: &types.Projection{ 123 | ProjectionType: types.ProjectionTypeAll, 124 | }, 125 | }, 126 | }, 127 | TableName: aws.String("CompositePrimaryKeyAndLocalIndexTable"), 128 | }, input) 129 | }) 130 | 131 | t.Run("table with many local secondary indexes", func(t *testing.T) { 132 | tmpl, err := goformation.Open("./testdata/template.yml") 133 | assert.NoError(t, err) 134 | 135 | table, err := tmpl.GetAWSDynamoDBTableWithName("CompositePrimaryKeyAndManyLocalIndexesTable") 136 | assert.NoError(t, err) 137 | 138 | input := dynamo.FromCloudFormationToCreateInput(*table) 139 | assert.Equal(t, dynamodb.CreateTableInput{ 140 | AttributeDefinitions: []types.AttributeDefinition{ 141 | { 142 | AttributeName: aws.String("pk"), 143 | AttributeType: types.ScalarAttributeTypeS, 144 | }, 145 | { 146 | AttributeName: aws.String("sk"), 147 | AttributeType: types.ScalarAttributeTypeS, 148 | }, 149 | { 150 | AttributeName: aws.String("lsi1_sk"), 151 | AttributeType: types.ScalarAttributeTypeS, 152 | }, 153 | { 154 | AttributeName: aws.String("lsi2_sk"), 155 | AttributeType: types.ScalarAttributeTypeS, 156 | }, 157 | }, 158 | BillingMode: types.BillingModePayPerRequest, 159 | KeySchema: []types.KeySchemaElement{ 160 | { 161 | AttributeName: aws.String("pk"), 162 | KeyType: types.KeyTypeHash, 163 | }, 164 | { 165 | AttributeName: aws.String("sk"), 166 | KeyType: types.KeyTypeRange, 167 | }, 168 | }, 169 | LocalSecondaryIndexes: []types.LocalSecondaryIndex{ 170 | { 171 | IndexName: aws.String("MyIndex1"), 172 | KeySchema: []types.KeySchemaElement{ 173 | { 174 | AttributeName: aws.String("pk"), 175 | KeyType: types.KeyTypeHash, 176 | }, 177 | { 178 | AttributeName: aws.String("lsi1_sk"), 179 | KeyType: types.KeyTypeRange, 180 | }, 181 | }, 182 | Projection: &types.Projection{ 183 | ProjectionType: types.ProjectionTypeAll, 184 | }, 185 | }, 186 | { 187 | IndexName: aws.String("MyIndex2"), 188 | KeySchema: []types.KeySchemaElement{ 189 | { 190 | AttributeName: aws.String("pk"), 191 | KeyType: types.KeyTypeHash, 192 | }, 193 | { 194 | AttributeName: aws.String("lsi2_sk"), 195 | KeyType: types.KeyTypeRange, 196 | }, 197 | }, 198 | Projection: &types.Projection{ 199 | ProjectionType: types.ProjectionTypeAll, 200 | }, 201 | }, 202 | }, 203 | TableName: aws.String("CompositePrimaryKeyAndManyLocalIndexesTable"), 204 | }, input) 205 | }) 206 | 207 | t.Run("table with single global secondary index", func(t *testing.T) { 208 | tmpl, err := goformation.Open("./testdata/template.yml") 209 | assert.NoError(t, err) 210 | 211 | table, err := tmpl.GetAWSDynamoDBTableWithName("CompositePrimaryKeyAndSingleGlobalIndexTable") 212 | assert.NoError(t, err) 213 | 214 | input := dynamo.FromCloudFormationToCreateInput(*table) 215 | assert.Equal(t, dynamodb.CreateTableInput{ 216 | AttributeDefinitions: []types.AttributeDefinition{ 217 | { 218 | AttributeName: aws.String("pk"), 219 | AttributeType: types.ScalarAttributeTypeS, 220 | }, 221 | { 222 | AttributeName: aws.String("sk"), 223 | AttributeType: types.ScalarAttributeTypeS, 224 | }, 225 | { 226 | AttributeName: aws.String("gsi1_pk"), 227 | AttributeType: types.ScalarAttributeTypeS, 228 | }, 229 | { 230 | AttributeName: aws.String("gsi1_sk"), 231 | AttributeType: types.ScalarAttributeTypeS, 232 | }, 233 | }, 234 | BillingMode: types.BillingModePayPerRequest, 235 | KeySchema: []types.KeySchemaElement{ 236 | { 237 | AttributeName: aws.String("pk"), 238 | KeyType: types.KeyTypeHash, 239 | }, 240 | { 241 | AttributeName: aws.String("sk"), 242 | KeyType: types.KeyTypeRange, 243 | }, 244 | }, 245 | GlobalSecondaryIndexes: []types.GlobalSecondaryIndex{ 246 | { 247 | IndexName: aws.String("GlobalSecondaryIndex1"), 248 | KeySchema: []types.KeySchemaElement{ 249 | { 250 | AttributeName: aws.String("gsi1_pk"), 251 | KeyType: types.KeyTypeHash, 252 | }, 253 | { 254 | AttributeName: aws.String("gsi1_sk"), 255 | KeyType: types.KeyTypeRange, 256 | }, 257 | }, 258 | Projection: &types.Projection{ 259 | ProjectionType: types.ProjectionTypeAll, 260 | }, 261 | }, 262 | }, 263 | TableName: aws.String("CompositePrimaryKeyAndSingleGlobalIndexTable"), 264 | }, input) 265 | }) 266 | 267 | t.Run("table with many global secondary indexes", func(t *testing.T) { 268 | tmpl, err := goformation.Open("./testdata/template.yml") 269 | assert.NoError(t, err) 270 | 271 | table, err := tmpl.GetAWSDynamoDBTableWithName("CompositePrimaryKeyAndManyGlobalIndexTable") 272 | assert.NoError(t, err) 273 | 274 | input := dynamo.FromCloudFormationToCreateInput(*table) 275 | assert.Equal(t, dynamodb.CreateTableInput{ 276 | AttributeDefinitions: []types.AttributeDefinition{ 277 | { 278 | AttributeName: aws.String("pk"), 279 | AttributeType: types.ScalarAttributeTypeS, 280 | }, 281 | { 282 | AttributeName: aws.String("sk"), 283 | AttributeType: types.ScalarAttributeTypeS, 284 | }, 285 | { 286 | AttributeName: aws.String("gsi1_pk"), 287 | AttributeType: types.ScalarAttributeTypeS, 288 | }, 289 | { 290 | AttributeName: aws.String("gsi1_sk"), 291 | AttributeType: types.ScalarAttributeTypeS, 292 | }, 293 | { 294 | AttributeName: aws.String("gsi2_pk"), 295 | AttributeType: types.ScalarAttributeTypeS, 296 | }, 297 | { 298 | AttributeName: aws.String("gsi2_sk"), 299 | AttributeType: types.ScalarAttributeTypeS, 300 | }, 301 | }, 302 | BillingMode: types.BillingModePayPerRequest, 303 | KeySchema: []types.KeySchemaElement{ 304 | { 305 | AttributeName: aws.String("pk"), 306 | KeyType: types.KeyTypeHash, 307 | }, 308 | { 309 | AttributeName: aws.String("sk"), 310 | KeyType: types.KeyTypeRange, 311 | }, 312 | }, 313 | GlobalSecondaryIndexes: []types.GlobalSecondaryIndex{ 314 | { 315 | IndexName: aws.String("GlobalSecondaryIndex1"), 316 | KeySchema: []types.KeySchemaElement{ 317 | { 318 | AttributeName: aws.String("gsi1_pk"), 319 | KeyType: types.KeyTypeHash, 320 | }, 321 | { 322 | AttributeName: aws.String("gsi1_sk"), 323 | KeyType: types.KeyTypeRange, 324 | }, 325 | }, 326 | Projection: &types.Projection{ 327 | ProjectionType: types.ProjectionTypeAll, 328 | }, 329 | }, 330 | { 331 | IndexName: aws.String("GlobalSecondaryIndex2"), 332 | KeySchema: []types.KeySchemaElement{ 333 | { 334 | AttributeName: aws.String("gsi2_pk"), 335 | KeyType: types.KeyTypeHash, 336 | }, 337 | { 338 | AttributeName: aws.String("gsi2_sk"), 339 | KeyType: types.KeyTypeRange, 340 | }, 341 | }, 342 | Projection: &types.Projection{ 343 | ProjectionType: types.ProjectionTypeAll, 344 | }, 345 | }, 346 | }, 347 | TableName: aws.String("CompositePrimaryKeyAndManyGlobalIndexTable"), 348 | }, input) 349 | }) 350 | } 351 | -------------------------------------------------------------------------------- /pkg/dynamo/testdata/template.yml: -------------------------------------------------------------------------------- 1 | AWSTemplateFormatVersion: "2010-09-09" 2 | Resources: 3 | PartitionKeyTable: 4 | Type: AWS::DynamoDB::Table 5 | Properties: 6 | AttributeDefinitions: 7 | - AttributeName: pk 8 | AttributeType: S 9 | KeySchema: 10 | - AttributeName: pk 11 | KeyType: HASH 12 | BillingMode: PAY_PER_REQUEST 13 | TableName: PartitionKeyTable 14 | 15 | CompositePrimaryKeyTable: 16 | Type: AWS::DynamoDB::Table 17 | Properties: 18 | AttributeDefinitions: 19 | - AttributeName: pk 20 | AttributeType: S 21 | - AttributeName: sk 22 | AttributeType: S 23 | KeySchema: 24 | - AttributeName: pk 25 | KeyType: HASH 26 | - AttributeName: sk 27 | KeyType: RANGE 28 | BillingMode: PAY_PER_REQUEST 29 | TableName: CompositePrimaryKeyTable 30 | 31 | CompositePrimaryKeyAndLocalIndexTable: 32 | Type: AWS::DynamoDB::Table 33 | Properties: 34 | AttributeDefinitions: 35 | - AttributeName: pk 36 | AttributeType: S 37 | - AttributeName: sk 38 | AttributeType: S 39 | - AttributeName: lsi_sk 40 | AttributeType: S 41 | KeySchema: 42 | - AttributeName: pk 43 | KeyType: HASH 44 | - AttributeName: sk 45 | KeyType: RANGE 46 | LocalSecondaryIndexes: 47 | - IndexName: MyIndex 48 | KeySchema: 49 | - AttributeName: pk 50 | KeyType: HASH 51 | - AttributeName: lsi_sk 52 | KeyType: RANGE 53 | Projection: 54 | ProjectionType: ALL 55 | BillingMode: PAY_PER_REQUEST 56 | TableName: CompositePrimaryKeyAndLocalIndexTable 57 | 58 | CompositePrimaryKeyAndManyLocalIndexesTable: 59 | Type: AWS::DynamoDB::Table 60 | Properties: 61 | AttributeDefinitions: 62 | - AttributeName: pk 63 | AttributeType: S 64 | - AttributeName: sk 65 | AttributeType: S 66 | - AttributeName: lsi1_sk 67 | AttributeType: S 68 | - AttributeName: lsi2_sk 69 | AttributeType: S 70 | KeySchema: 71 | - AttributeName: pk 72 | KeyType: HASH 73 | - AttributeName: sk 74 | KeyType: RANGE 75 | LocalSecondaryIndexes: 76 | - IndexName: MyIndex1 77 | KeySchema: 78 | - AttributeName: pk 79 | KeyType: HASH 80 | - AttributeName: lsi1_sk 81 | KeyType: RANGE 82 | Projection: 83 | ProjectionType: ALL 84 | - IndexName: MyIndex2 85 | KeySchema: 86 | - AttributeName: pk 87 | KeyType: HASH 88 | - AttributeName: lsi2_sk 89 | KeyType: RANGE 90 | Projection: 91 | ProjectionType: ALL 92 | BillingMode: PAY_PER_REQUEST 93 | TableName: CompositePrimaryKeyAndManyLocalIndexesTable 94 | 95 | CompositePrimaryKeyAndSingleGlobalIndexTable: 96 | Type: AWS::DynamoDB::Table 97 | Properties: 98 | AttributeDefinitions: 99 | - AttributeName: pk 100 | AttributeType: S 101 | - AttributeName: sk 102 | AttributeType: S 103 | - AttributeName: gsi1_pk 104 | AttributeType: S 105 | - AttributeName: gsi1_sk 106 | AttributeType: S 107 | KeySchema: 108 | - AttributeName: pk 109 | KeyType: HASH 110 | - AttributeName: sk 111 | KeyType: RANGE 112 | GlobalSecondaryIndexes: 113 | - IndexName: GlobalSecondaryIndex1 114 | KeySchema: 115 | - AttributeName: gsi1_pk 116 | KeyType: HASH 117 | - AttributeName: gsi1_sk 118 | KeyType: RANGE 119 | Projection: 120 | ProjectionType: ALL 121 | BillingMode: PAY_PER_REQUEST 122 | TableName: CompositePrimaryKeyAndSingleGlobalIndexTable 123 | 124 | CompositePrimaryKeyAndManyGlobalIndexTable: 125 | Type: AWS::DynamoDB::Table 126 | Properties: 127 | AttributeDefinitions: 128 | - AttributeName: pk 129 | AttributeType: S 130 | - AttributeName: sk 131 | AttributeType: S 132 | - AttributeName: gsi1_pk 133 | AttributeType: S 134 | - AttributeName: gsi1_sk 135 | AttributeType: S 136 | - AttributeName: gsi2_pk 137 | AttributeType: S 138 | - AttributeName: gsi2_sk 139 | AttributeType: S 140 | KeySchema: 141 | - AttributeName: pk 142 | KeyType: HASH 143 | - AttributeName: sk 144 | KeyType: RANGE 145 | GlobalSecondaryIndexes: 146 | - IndexName: GlobalSecondaryIndex1 147 | KeySchema: 148 | - AttributeName: gsi1_pk 149 | KeyType: HASH 150 | - AttributeName: gsi1_sk 151 | KeyType: RANGE 152 | Projection: 153 | ProjectionType: ALL 154 | - IndexName: GlobalSecondaryIndex2 155 | KeySchema: 156 | - AttributeName: gsi2_pk 157 | KeyType: HASH 158 | - AttributeName: gsi2_sk 159 | KeyType: RANGE 160 | Projection: 161 | ProjectionType: ALL 162 | BillingMode: PAY_PER_REQUEST 163 | TableName: CompositePrimaryKeyAndManyGlobalIndexTable 164 | --------------------------------------------------------------------------------