├── Makefile ├── README.md ├── go.mod ├── go.sum ├── internal ├── app │ ├── handlers.go │ ├── metadata.go │ └── router.go ├── ratings │ ├── lambda_client.go │ ├── repo.go │ └── types.go ├── streams │ └── broker.go └── talks │ ├── repo.go │ └── types.go ├── main.go ├── step-1-getting-started.md ├── step-10-e2e-tests-with-real-dependencies.md ├── step-11-integration-tests-for-the-lambda.md ├── step-12-exploring-the-running-app.md ├── step-2-exploring-the-app.md ├── step-3-running-the-app-locally.md ├── step-4-dev-mode-with-testcontainers.md ├── step-5-adding-redis.md ├── step-6-adding-redpanda.md ├── step-7-adding-localstack.md ├── step-8-adding-integration-tests.md └── step-9-integration-tests-for-api.md /Makefile: -------------------------------------------------------------------------------- 1 | dev: 2 | go run ./... 3 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # workshop-go 2 | 3 | # Introduction 4 | 5 | Are you ready to streamline your application development and testing processes while building robust Go applications? Use this workshop to unlock the potential of Testcontainers for Go ([https://golang.testcontainers.org](https://golang.testcontainers.org)) and embark on a journey to build a talk rating microservice from the ground up. 6 | 7 | In this workshop, we'll guide you through the creation of a feature-rich microservice for rating conference talks, using a variety of technologies such as databases, caches, event brokers and AWS lambdas. 8 | 9 | We will put the emphasis in two crucial aspects: 10 | 11 | - Containerized Local Development: Learn how to harness the power of Testcontainers for Go to simplify your Go application development workflow. 12 | - Effortless Integration Testing: Explore how Testcontainers for Go empowers you to write integration tests with ease. See how containerized development naturally translates into seamless integration testing, ensuring your microservice functions as expected in real-world scenarios. 13 | 14 | By the end of this workshop, you will: 15 | 16 | - Gain mastery in containerized local development, allowing you to confidently manage your application's infrastructure directly in your code. 17 | - Acquire the skills to write integration tests effortlessly, providing the assurance that your microservice performs flawlessly in diverse environments. 18 | 19 | Containerized local development and integration testing can become your greatest allies in building robust and reliable applications. With Testcontainers for Go, they are much easier to achieve. 20 | 21 | ## Table of contents 22 | 23 | * [Introduction](README.md) 24 | * [Step 1: Getting Started](step-1-getting-started.md) 25 | * [Step 2: Exploring the app](step-2-exploring-the-app.md) 26 | * [Step 3: Running the app locally](step-3-running-the-app-locally.md) 27 | * [Step 4: Dev Mode with Testcontainers for Go](step-4-dev-mode-with-testcontainers.md) 28 | * [Step 5: Adding Redis](step-5-adding-redis.md) 29 | * [Step 6: Adding Redpanda](step-6-adding-repanda.md) 30 | * [Step 7: Adding LocalStack](step-7-adding-localstack.md) 31 | * [Step 8: Adding Integration Tests](step-8-adding-integration-tests.md) 32 | * [Step 9: Adding Integration Tests for the API](step-9-integration-tests-for-api.md) 33 | * [Step 10: End-To-End Tests with real dependencies](step-10-e2e-tests-with-real-dependencies.md) 34 | * [Step 11: Integration tests for the Go Lambda](step-11-integration-tests-for-the-lambda.md) 35 | * [Step 12: Exploring the running app](step-12-exploring-the-running-app.md) 36 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/testcontainers/workshop-go 2 | 3 | go 1.24 4 | 5 | require ( 6 | github.com/go-redis/redis/v8 v8.11.5 7 | github.com/gofiber/fiber/v2 v2.52.6 8 | github.com/jackc/pgx/v5 v5.5.3 9 | github.com/twmb/franz-go v1.16.1 10 | ) 11 | 12 | require ( 13 | github.com/andybalholm/brotli v1.1.0 // indirect 14 | github.com/cespare/xxhash/v2 v2.2.0 // indirect 15 | github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect 16 | github.com/fsnotify/fsnotify v1.7.0 // indirect 17 | github.com/google/uuid v1.6.0 // indirect 18 | github.com/jackc/pgpassfile v1.0.0 // indirect 19 | github.com/jackc/pgservicefile v0.0.0-20231201235250-de7065d80cb9 // indirect 20 | github.com/klauspost/compress v1.17.9 // indirect 21 | github.com/mattn/go-colorable v0.1.13 // indirect 22 | github.com/mattn/go-isatty v0.0.20 // indirect 23 | github.com/mattn/go-runewidth v0.0.16 // indirect 24 | github.com/pierrec/lz4/v4 v4.1.21 // indirect 25 | github.com/rivo/uniseg v0.2.0 // indirect 26 | github.com/stretchr/testify v1.10.0 // indirect 27 | github.com/twmb/franz-go/pkg/kmsg v1.7.0 // indirect 28 | github.com/valyala/bytebufferpool v1.0.0 // indirect 29 | github.com/valyala/fasthttp v1.51.0 // indirect 30 | github.com/valyala/tcplisten v1.0.0 // indirect 31 | golang.org/x/crypto v0.37.0 // indirect 32 | golang.org/x/sys v0.32.0 // indirect 33 | golang.org/x/text v0.24.0 // indirect 34 | ) 35 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/andybalholm/brotli v1.1.0 h1:eLKJA0d02Lf0mVpIDgYnqXcUn0GqVmEFny3VuID1U3M= 2 | github.com/andybalholm/brotli v1.1.0/go.mod h1:sms7XGricyQI9K10gOSf56VKKWS4oLer58Q+mhRPtnY= 3 | github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= 4 | github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= 5 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 6 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 7 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 8 | github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78= 9 | github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc= 10 | github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA= 11 | github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM= 12 | github.com/go-redis/redis/v8 v8.11.5 h1:AcZZR7igkdvfVmQTPnu9WE37LRrO/YrBH5zWyjDC0oI= 13 | github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq8Jd4h5lpwo= 14 | github.com/gofiber/fiber/v2 v2.52.6 h1:Rfp+ILPiYSvvVuIPvxrBns+HJp8qGLDnLJawAu27XVI= 15 | github.com/gofiber/fiber/v2 v2.52.6/go.mod h1:YEcBbO/FB+5M1IZNBP9FO3J9281zgPAreiI1oqg8nDw= 16 | github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= 17 | github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= 18 | github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= 19 | github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= 20 | github.com/jackc/pgservicefile v0.0.0-20231201235250-de7065d80cb9 h1:L0QtFUgDarD7Fpv9jeVMgy/+Ec0mtnmYuImjTz6dtDA= 21 | github.com/jackc/pgservicefile v0.0.0-20231201235250-de7065d80cb9/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM= 22 | github.com/jackc/pgx/v5 v5.5.3 h1:Ces6/M3wbDXYpM8JyyPD57ivTtJACFZJd885pdIaV2s= 23 | github.com/jackc/pgx/v5 v5.5.3/go.mod h1:ez9gk+OAat140fv9ErkZDYFWmXLfV+++K0uAOiwgm1A= 24 | github.com/jackc/puddle/v2 v2.2.1 h1:RhxXJtFG022u4ibrCSMSiu5aOq1i77R3OHKNJj77OAk= 25 | github.com/jackc/puddle/v2 v2.2.1/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4= 26 | github.com/klauspost/compress v1.17.9 h1:6KIumPrER1LHsvBVuDa0r5xaG0Es51mhhB9BQB2qeMA= 27 | github.com/klauspost/compress v1.17.9/go.mod h1:Di0epgTjJY877eYKx5yC51cX2A2Vl2ibi7bDH9ttBbw= 28 | github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= 29 | github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg= 30 | github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= 31 | github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= 32 | github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= 33 | github.com/mattn/go-runewidth v0.0.16 h1:E5ScNMtiwvlvB5paMFdw9p4kSQzbXFikJ5SQO6TULQc= 34 | github.com/mattn/go-runewidth v0.0.16/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w= 35 | github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE= 36 | github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU= 37 | github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE= 38 | github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU= 39 | github.com/onsi/gomega v1.18.1 h1:M1GfJqGRrBrrGGsbxzV5dqM2U2ApXefZCQpkukxYRLE= 40 | github.com/onsi/gomega v1.18.1/go.mod h1:0q+aL8jAiMXy9hbwj2mr5GziHiwhAIQpFmmtT5hitRs= 41 | github.com/pierrec/lz4/v4 v4.1.21 h1:yOVMLb6qSIDP67pl/5F7RepeKYu/VmTyEXvuMI5d9mQ= 42 | github.com/pierrec/lz4/v4 v4.1.21/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4= 43 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 44 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 45 | github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY= 46 | github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc= 47 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 48 | github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= 49 | github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 50 | github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA= 51 | github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= 52 | github.com/twmb/franz-go v1.16.1 h1:rpWc7fB9jd7TgmCyfxzenBI+QbgS8ZfJOUQE+tzPtbE= 53 | github.com/twmb/franz-go v1.16.1/go.mod h1:/pER254UPPGp/4WfGqRi+SIRGE50RSQzVubQp6+N4FA= 54 | github.com/twmb/franz-go/pkg/kmsg v1.7.0 h1:a457IbvezYfA5UkiBvyV3zj0Is3y1i8EJgqjJYoij2E= 55 | github.com/twmb/franz-go/pkg/kmsg v1.7.0/go.mod h1:se9Mjdt0Nwzc9lnjJ0HyDtLyBnaBDAd7pCje47OhSyw= 56 | github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw= 57 | github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc= 58 | github.com/valyala/fasthttp v1.51.0 h1:8b30A5JlZ6C7AS81RsWjYMQmrZG6feChmgAolCl1SqA= 59 | github.com/valyala/fasthttp v1.51.0/go.mod h1:oI2XroL+lI7vdXyYoQk03bXBThfFl2cVdIA3Xl7cH8g= 60 | github.com/valyala/tcplisten v1.0.0 h1:rBHj/Xf+E1tRGZyWIWwJDiRY0zc1Js+CV5DqwacVSA8= 61 | github.com/valyala/tcplisten v1.0.0/go.mod h1:T0xQ8SeCZGxckz9qRXTfG43PvQ/mcWh7FwZEA7Ioqkc= 62 | golang.org/x/crypto v0.37.0 h1:kJNSjF/Xp7kU0iB2Z+9viTPMW4EqqsrywMXLJOOsXSE= 63 | golang.org/x/crypto v0.37.0/go.mod h1:vg+k43peMZ0pUMhYmVAWysMK35e6ioLh3wB8ZCAfbVc= 64 | golang.org/x/net v0.21.0 h1:AQyQV4dYCvJ7vGmJyKki9+PBdyvhkSd8EIx/qb0AYv4= 65 | golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44= 66 | golang.org/x/sync v0.13.0 h1:AauUjRAJ9OSnvULf/ARrrVywoJDy0YS2AwQ98I37610= 67 | golang.org/x/sync v0.13.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA= 68 | golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 69 | golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 70 | golang.org/x/sys v0.32.0 h1:s77OFDvIQeibCmezSnk/q6iAfkdiQaJi4VzroCFrN20= 71 | golang.org/x/sys v0.32.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= 72 | golang.org/x/text v0.24.0 h1:dd5Bzh4yt5KYA8f9CJHCP4FB4D51c2c6JvN37xJJkJ0= 73 | golang.org/x/text v0.24.0/go.mod h1:L8rBsPeo2pSS+xqN0d5u2ikmjtmoJbDBT1b7nHvFCdU= 74 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 75 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= 76 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= 77 | gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY= 78 | gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ= 79 | gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 80 | gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= 81 | gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 82 | -------------------------------------------------------------------------------- /internal/app/handlers.go: -------------------------------------------------------------------------------- 1 | package app 2 | 3 | import ( 4 | "encoding/json" 5 | "errors" 6 | "fmt" 7 | "log" 8 | "net/http" 9 | 10 | "github.com/gofiber/fiber/v2" 11 | "github.com/testcontainers/workshop-go/internal/ratings" 12 | "github.com/testcontainers/workshop-go/internal/streams" 13 | "github.com/testcontainers/workshop-go/internal/talks" 14 | ) 15 | 16 | func Root(c *fiber.Ctx) error { 17 | return c.JSON(fiber.Map{ 18 | "metadata": Connections, 19 | }) 20 | } 21 | 22 | // ratingForPost is the struct that will be used to read the JSON payload 23 | // from the POST request when a new rating is added. 24 | type ratingForPost struct { 25 | UUID string `json:"talkId" form:"talkId" binding:"required"` 26 | Rating int64 `json:"value" form:"value" binding:"required"` 27 | } 28 | 29 | // AddRating is the handler for the `POST /ratings` endpoint. 30 | // It will add a new rating to the store, where the rating is read from the JSON payload 31 | // using the following format: 32 | // 33 | // { 34 | // "talkId": "123", 35 | // "value": 5 36 | // } 37 | // 38 | // If the talk with the given UUID exists in the Talks repository, it will send the rating 39 | // to the Streams repository, which will send it to the broker. If the talk does not exist, 40 | // or any of the repositories cannot be created, it will return an error. 41 | func AddRating(c *fiber.Ctx) error { 42 | var r ratingForPost 43 | 44 | if err := c.BodyParser(&r); err != nil { 45 | return handleError(c, http.StatusInternalServerError, err) 46 | } 47 | 48 | talksRepo, err := talks.NewRepository(c.Context(), Connections.Talks) 49 | if err != nil { 50 | return handleError(c, http.StatusInternalServerError, err) 51 | } 52 | 53 | if !talksRepo.Exists(c.Context(), r.UUID) { 54 | return handleError(c, http.StatusNotFound, fmt.Errorf("talk with UUID %s does not exist", r.UUID)) 55 | } 56 | 57 | streamsRepo, err := streams.NewStream(c.Context(), Connections.Streams) 58 | if err != nil { 59 | return handleError(c, http.StatusInternalServerError, err) 60 | } 61 | 62 | ratingsRepo, err := ratings.NewRepository(c.Context(), Connections.Ratings) 63 | if err != nil { 64 | return handleError(c, http.StatusInternalServerError, err) 65 | } 66 | 67 | rating := ratings.Rating{ 68 | TalkUuid: r.UUID, 69 | Value: r.Rating, 70 | } 71 | 72 | ratingsCallback := func() error { 73 | _, err := ratingsRepo.Add(c.Context(), rating) 74 | return err 75 | } 76 | 77 | err = streamsRepo.SendRating(c.Context(), rating, ratingsCallback) 78 | if err != nil { 79 | return handleError(c, http.StatusInternalServerError, err) 80 | } 81 | 82 | return c.Status(http.StatusOK).JSON(fiber.Map{ 83 | "rating": rating, 84 | }) 85 | } 86 | 87 | // talkForRatings is the struct that will be used to get a talk UUID from the query string 88 | // of the GET request when the ratings for a talk are requested. 89 | type talkForRatings struct { 90 | UUID string `json:"talkId" form:"talkId" binding:"required"` 91 | } 92 | 93 | type statsResponse struct { 94 | Avg float64 `json:"avg"` 95 | TotalCount int64 `json:"totalCount"` 96 | } 97 | 98 | // Ratings is the handler for the `GET /ratings?talkId=xxx` endpoint. It will require a talkId parameter 99 | // in the query string and will return all the ratings for the given talk UUID. 100 | func Ratings(c *fiber.Ctx) error { 101 | talkID := c.Query("talkId", "") 102 | if talkID == "" { 103 | return handleError(c, http.StatusInternalServerError, errors.New("talkId is required")) 104 | } 105 | 106 | talk := talkForRatings{UUID: talkID} 107 | 108 | talksRepo, err := talks.NewRepository(c.Context(), Connections.Talks) 109 | if err != nil { 110 | return handleError(c, http.StatusInternalServerError, err) 111 | } 112 | 113 | if !talksRepo.Exists(c.Context(), talk.UUID) { 114 | return handleError(c, http.StatusNotFound, fmt.Errorf("talk with UUID %s does not exist", talk.UUID)) 115 | } 116 | 117 | ratingsRepo, err := ratings.NewRepository(c.Context(), Connections.Ratings) 118 | if err != nil { 119 | return handleError(c, http.StatusInternalServerError, err) 120 | } 121 | 122 | histogram := ratingsRepo.FindAllByUUID(c.Context(), talk.UUID) 123 | 124 | // call the lambda function to get the stats 125 | lambdaClient := ratings.NewLambdaClient(Connections.Lambda) 126 | stats, err := lambdaClient.GetStats(histogram) 127 | if err != nil { 128 | // do not fail if the lambda function is not available, simply do not aggregate the stats 129 | log.Printf("error calling lambda function: %s", err.Error()) 130 | return c.Status(http.StatusOK).JSON(fiber.Map{ 131 | "ratings": histogram, 132 | }) 133 | } 134 | 135 | statsResp := &statsResponse{} 136 | err = json.Unmarshal(stats, statsResp) 137 | if err != nil { 138 | // do not fail if the lambda function is not available, simply do not aggregate the stats 139 | log.Printf("error unmarshalling lambda response: %s", err.Error()) 140 | return c.Status(http.StatusOK).JSON(fiber.Map{ 141 | "ratings": histogram, 142 | }) 143 | } 144 | 145 | return c.Status(http.StatusOK).JSON(fiber.Map{ 146 | "ratings": histogram, 147 | "stats": statsResp, 148 | }) 149 | } 150 | 151 | func handleError(c *fiber.Ctx, code int, err error) error { 152 | return c.Status(code).JSON(fiber.Map{ 153 | "message": err.Error(), 154 | }) 155 | } 156 | -------------------------------------------------------------------------------- /internal/app/metadata.go: -------------------------------------------------------------------------------- 1 | package app 2 | 3 | import "os" 4 | 5 | // The connection string for each of the services needed by the application. 6 | // The application will need them to connect to services, reading it from 7 | // the right environment variable in production, or from the container in development. 8 | type Metadata struct { 9 | Lambda string `json:"ratings_lambda"` // Read from the RATINGS_LAMBDA_URL environment variable 10 | Ratings string `json:"ratings"` // Read from the RATINGS_CONNECTION environment variable 11 | Streams string `json:"streams"` // Read from the STREAMS_CONNECTION environment variable 12 | Talks string `json:"talks"` // Read from the TALKS_CONNECTION environment variable 13 | } 14 | 15 | var Connections *Metadata = &Metadata{ 16 | Lambda: os.Getenv("RATINGS_LAMBDA_URL"), 17 | Ratings: os.Getenv("RATINGS_CONNECTION"), 18 | Streams: os.Getenv("STREAMS_CONNECTION"), 19 | Talks: os.Getenv("TALKS_CONNECTION"), 20 | } 21 | -------------------------------------------------------------------------------- /internal/app/router.go: -------------------------------------------------------------------------------- 1 | package app 2 | 3 | import ( 4 | "github.com/gofiber/fiber/v2" 5 | "github.com/gofiber/fiber/v2/middleware/logger" 6 | ) 7 | 8 | func SetupApp() *fiber.App { 9 | app := fiber.New() 10 | 11 | app.Use(logger.New()) 12 | 13 | app.Get("/", Root) 14 | app.Get("/ratings", Ratings) 15 | app.Post("/ratings", AddRating) 16 | 17 | return app 18 | } 19 | -------------------------------------------------------------------------------- /internal/ratings/lambda_client.go: -------------------------------------------------------------------------------- 1 | package ratings 2 | 3 | import ( 4 | "bytes" 5 | "io" 6 | "net/http" 7 | "time" 8 | ) 9 | 10 | // Repository is the interface that wraps the basic operations with the Redis store. 11 | type LambdaClient struct { 12 | client *http.Client 13 | url string 14 | } 15 | 16 | // NewLambdaClient creates a new client from the Lambda URL. 17 | func NewLambdaClient(lambdaURL string) *LambdaClient { 18 | httpClient := http.Client{ 19 | Timeout: 5 * time.Second, 20 | } 21 | 22 | return &LambdaClient{ 23 | client: &httpClient, 24 | url: lambdaURL, 25 | } 26 | } 27 | 28 | // GetStats returns the stats for the given talk, obtained from a call to the Lambda function. 29 | // The payload is a JSON object with the following structure: 30 | // 31 | // { 32 | // "ratings": { 33 | // "0": 10, 34 | // "1": 20, 35 | // "2": 30, 36 | // "3": 40, 37 | // "4": 50, 38 | // "5": 60 39 | // } 40 | // } 41 | // 42 | // The response from the Lambda function is a JSON object with the following structure: 43 | // 44 | // { 45 | // "avg": 3.5, 46 | // "totalCount": 210, 47 | // } 48 | func (c *LambdaClient) GetStats(histogram map[string]string) ([]byte, error) { 49 | payload := `{"ratings": {` 50 | for rating, count := range histogram { 51 | // we are passing the count as an integer, so we don't need to quote it 52 | payload += `"` + rating + `": ` + count + `,` 53 | } 54 | 55 | if len(histogram) > 0 { 56 | // remove the last comma onl for non-empty histograms 57 | payload = payload[:len(payload)-1] 58 | } 59 | payload += "}}" 60 | 61 | resp, err := c.client.Post(c.url, "application/json", bytes.NewBufferString(payload)) 62 | if err != nil { 63 | return nil, err 64 | } 65 | 66 | return io.ReadAll(resp.Body) 67 | } 68 | -------------------------------------------------------------------------------- /internal/ratings/repo.go: -------------------------------------------------------------------------------- 1 | package ratings 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "os" 7 | 8 | "github.com/go-redis/redis/v8" 9 | ) 10 | 11 | // Repository is the interface that wraps the basic operations with the Redis store. 12 | type Repository struct { 13 | client *redis.Client 14 | } 15 | 16 | // NewRepository creates a new repository. It will receive a context and the Redis connection string. 17 | func NewRepository(ctx context.Context, connStr string) (*Repository, error) { 18 | options, err := redis.ParseURL(connStr) 19 | if err != nil { 20 | _, _ = fmt.Fprintf(os.Stderr, "Unable to connect to Redis: %v\n", err) 21 | return nil, err 22 | } 23 | 24 | cli := redis.NewClient(options) 25 | 26 | pong, err := cli.Ping(ctx).Result() 27 | if err != nil { 28 | // You probably want to retry here 29 | return nil, err 30 | } 31 | 32 | if pong != "PONG" { 33 | // You probably want to retry here 34 | return nil, err 35 | } 36 | 37 | return &Repository{client: cli}, nil 38 | } 39 | 40 | // Add increments in one the counter for the given rating value and talk UUID. 41 | func (r *Repository) Add(ctx context.Context, rating Rating) (int64, error) { 42 | return r.client.HIncrBy(ctx, toKey(rating.TalkUuid), fmt.Sprintf("%d", rating.Value), 1).Result() 43 | } 44 | 45 | // FindAllByUUID returns all the ratings and their counters for the given talk UUID. 46 | func (r *Repository) FindAllByUUID(ctx context.Context, uid string) map[string]string { 47 | return r.client.HGetAll(ctx, toKey(uid)).Val() 48 | } 49 | 50 | // toKey is a helper function that returns the uuid prefixed with "ratings/". 51 | func toKey(uuid string) string { 52 | return "ratings/" + uuid 53 | } 54 | -------------------------------------------------------------------------------- /internal/ratings/types.go: -------------------------------------------------------------------------------- 1 | package ratings 2 | 3 | type Rating struct { 4 | TalkUuid string `json:"talk_uuid"` 5 | Value int64 `json:"value"` 6 | } 7 | -------------------------------------------------------------------------------- /internal/streams/broker.go: -------------------------------------------------------------------------------- 1 | package streams 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "os" 7 | 8 | "github.com/testcontainers/workshop-go/internal/ratings" 9 | "github.com/twmb/franz-go/pkg/kgo" 10 | ) 11 | 12 | const RatingsTopic = "ratings" 13 | 14 | // Repository is the interface that wraps the basic operations with the broker store. 15 | type Repository struct { 16 | client *kgo.Client 17 | } 18 | 19 | // NewStream creates a new repository. It will receive a context and the connection string for the broker. 20 | func NewStream(ctx context.Context, connStr string) (*Repository, error) { 21 | cli, err := kgo.NewClient( 22 | kgo.SeedBrokers(connStr), 23 | kgo.ConsumeTopics(RatingsTopic), 24 | kgo.AllowAutoTopicCreation(), 25 | ) 26 | if err != nil { 27 | _, _ = fmt.Fprintf(os.Stderr, "Unable to connect to the streams: %v\n", err) 28 | // You probably want to retry here 29 | return nil, err 30 | } 31 | 32 | err = cli.Ping(ctx) 33 | if err != nil { 34 | _, _ = fmt.Fprintf(os.Stderr, "Unable to ping the streams: %v\n", err) 35 | return nil, err 36 | } 37 | 38 | return &Repository{client: cli}, nil 39 | } 40 | 41 | // SendRating sends a rating to the broker in an asynchronous way, executing a callback 42 | // when the record is produced. It will notifiy the caller if the operation errored or 43 | // if the context was cancelled. 44 | func (r *Repository) SendRating(ctx context.Context, rating ratings.Rating, produceCallback func() error) error { 45 | record := &kgo.Record{Topic: RatingsTopic, Value: []byte("test")} 46 | 47 | errChan := make(chan error, 1) 48 | 49 | r.client.Produce(ctx, record, func(producedRecord *kgo.Record, err error) { 50 | if err != nil { 51 | errChan <- err 52 | return 53 | } 54 | 55 | err = produceCallback() 56 | if err != nil { 57 | errChan <- err 58 | return 59 | } 60 | 61 | errChan <- nil 62 | }) 63 | 64 | // we are actively waiting for an error to be returned or for the context to be cancelled, 65 | // because we want to notify the caller in those cases 66 | select { 67 | case <-ctx.Done(): 68 | return ctx.Err() 69 | case err := <-errChan: 70 | return err 71 | } 72 | } 73 | -------------------------------------------------------------------------------- /internal/talks/repo.go: -------------------------------------------------------------------------------- 1 | package talks 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "os" 7 | 8 | "github.com/jackc/pgx/v5" 9 | ) 10 | 11 | type Repository struct { 12 | conn *pgx.Conn 13 | } 14 | 15 | // NewRepository creates a new repository. It will receive a context and the PostgreSQL connection string. 16 | func NewRepository(ctx context.Context, connStr string) (*Repository, error) { 17 | conn, err := pgx.Connect(ctx, connStr) 18 | if err != nil { 19 | _, _ = fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err) 20 | return nil, err 21 | } 22 | 23 | return &Repository{ 24 | conn: conn, 25 | }, nil 26 | } 27 | 28 | // Create creates a new talk in the database. 29 | // It uses value semantics at the method receiver to avoid mutating the original repository. 30 | // It uses pointer semantics at the talk parameter to avoid copying the struct, modifying it and returning it. 31 | func (r Repository) Create(ctx context.Context, talk *Talk) error { 32 | query := "INSERT INTO talks (uuid, title) VALUES ($1, $2) RETURNING id" 33 | 34 | return r.conn.QueryRow(ctx, query, talk.UUID, talk.Title).Scan(&talk.ID) 35 | } 36 | 37 | // Exists retrieves a talk from the database by its UUID. 38 | func (r Repository) Exists(ctx context.Context, uid string) bool { 39 | _, err := r.GetByUUID(ctx, uid) 40 | 41 | return err == nil 42 | } 43 | 44 | // GetByUUID retrieves a talk from the database by its UUID. 45 | func (r Repository) GetByUUID(ctx context.Context, uid string) (Talk, error) { 46 | query := "SELECT id, uuid, title FROM talks WHERE uuid = $1" 47 | 48 | var talk Talk 49 | err := r.conn.QueryRow(ctx, query, uid).Scan(&talk.ID, &talk.UUID, &talk.Title) 50 | if err != nil { 51 | return Talk{}, err 52 | } 53 | 54 | return talk, nil 55 | } 56 | -------------------------------------------------------------------------------- /internal/talks/types.go: -------------------------------------------------------------------------------- 1 | package talks 2 | 3 | // Talk is a struct that represents a talk. 4 | type Talk struct { 5 | ID int `json:"id"` 6 | UUID string `json:"uuid"` 7 | Title string `json:"title"` 8 | } 9 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "github.com/testcontainers/workshop-go/internal/app" 5 | ) 6 | 7 | func main() { 8 | app := app.SetupApp() 9 | 10 | app.Listen(":8080") 11 | } 12 | -------------------------------------------------------------------------------- /step-1-getting-started.md: -------------------------------------------------------------------------------- 1 | # Step 1: Getting Started 2 | 3 | ## Check Make 4 | 5 | We'll need Make to run the workshop. 6 | 7 | ```shell 8 | $ make --version 9 | GNU Make 3.81 10 | ``` 11 | 12 | ## Check Go 13 | 14 | We'll need Go 1.24 or newer for this workshop. 15 | 16 | For installing Go, please follow the instructions at [https://golang.org/doc/install](https://golang.org/doc/install), or use your favorite package manager, like [`gvm`](https://github.com/andrewkroh/gvm). 17 | 18 | This workshop uses a [GoFiber](https://gofiber.io/) application. 19 | 20 | ## Check Docker 21 | 22 | Make sure we have a Docker environment available on your machine. 23 | 24 | The recommended Docker environment is [Testcontainers Desktop](https://testcontainers.com/desktop), the free companion app that is the perfect for running Testcontainers on your machine. Please download and install it, and create a free account if you don't have one yet. 25 | 26 | With Testcontainers Desktop, we can simply choose the container runtimes we want to use, and Testcontainers Desktop will take care of the rest. At the same time, we can choose running the container in an embedded runtime, which is a lightweight and performant Docker runtime that is bundled with Testcontainers Desktop (_only available for Mac at the moment_), or using [Testcontainers Cloud](https://testcontainers.com/cloud) as a remote runtime (recommended to avoid straining conference networks by pulling heavy Docker images). 27 | 28 | If you already have a local Docker runtime (on Linux, For Mac, or For Windows), this workshop works perfectly fine with that as well. 29 | 30 | We can check our container runtime by simply running: 31 | 32 | ```shell 33 | $ docker version 34 | Client: 35 | Version: 27.2.1-rd 36 | API version: 1.43 (downgraded from 1.47) 37 | Go version: go1.22.7 38 | Git commit: cc0ee3e 39 | Built: Tue Sep 10 15:41:09 2024 40 | OS/Arch: darwin/arm64 41 | Context: tcd 42 | 43 | Server: Docker Engine - Community 44 | Engine: 45 | Version: 27.5.0 46 | API version: 1.47 (minimum version 1.24) 47 | Go version: go1.22.10 48 | Git commit: 38b84dc 49 | Built: Thu Jan 16 09:42:44 2025 50 | OS/Arch: linux/amd64 51 | Experimental: false 52 | containerd: 53 | Version: 1.7.24 54 | GitCommit: 55 | runc: 56 | Version: 1.1.12-0ubuntu2~22.04.1 57 | GitCommit: 58 | docker-init: 59 | Version: 0.19.0 60 | GitCommit: de40ad0 61 | ``` 62 | 63 | ## Download the project 64 | 65 | Clone the following project from GitHub to your computer: 66 | [https://github.com/testcontainers/workshop-go](https://github.com/testcontainers/workshop-go) 67 | 68 | ## Download the dependencies 69 | 70 | ```shell 71 | go get github.com/go-redis/redis/v8 72 | go get github.com/gofiber/fiber/v2 73 | go get github.com/google/uuid 74 | go get github.com/jackc/pgx/v5 75 | go get github.com/stretchr/testify 76 | go get github.com/testcontainers/testcontainers-go 77 | go get github.com/testcontainers/testcontainers-go/modules/localstack 78 | go get github.com/testcontainers/testcontainers-go/modules/postgres 79 | go get github.com/testcontainers/testcontainers-go/modules/redis 80 | go get github.com/testcontainers/testcontainers-go/modules/redpanda 81 | go get github.com/twmb/franz-go 82 | ``` 83 | 84 | ## \(optionally\) Pull the required images before doing the workshop 85 | 86 | This might be helpful if the internet connection at the workshop venue is somewhat slow. 87 | 88 | ```text 89 | docker pull postgres:15.3-alpine 90 | docker pull redis:6-alpine 91 | docker pull docker.redpanda.com/redpandadata/redpanda:v24.3.7 92 | docker pull localstack/localstack:latest 93 | ``` 94 | 95 | ### 96 | [Next: Exploring the app](step-2-exploring-the-app.md) 97 | -------------------------------------------------------------------------------- /step-10-e2e-tests-with-real-dependencies.md: -------------------------------------------------------------------------------- 1 | # Step 10: End-To-End tests with real dependencies 2 | 3 | In the previous step we added integration tests for the API, and for that we used the [`net/httptest`](https://pkg.go.dev/net/http/httptest) package from the standard library. But the HTTP handlers in the application are consuming other services as runtime dependencies, and if they do not exist, those handlers will return an error (see `internal/app/handlers.go`). 4 | 5 | The tests that we added in the previous step are using the `httptest` package to test the handlers, but they are not testing the dependencies, they are simply checking that the handlers return an error. In this step, we are going to reuse what we did for the `local dev mode` and start the dependencies using `Testcontainers`. 6 | 7 | The tests we are going to add in this step are called `End-To-End` tests (also known as `E2E`), because they are going to test the application with all its dependencies, as the HTTP handlers need them to work. 8 | 9 | ## Reusing the `local dev mode` code 10 | 11 | In the step 4 we added the `internal/app/dev_dependencies.go` file to start the dependencies when running the application in `local dev mode`. It used a Go build tag to include the code only when the `dev` build tag is present. Let's add a build tag to also execute that code for the E2E tests of the handlers. 12 | 13 | Please replace the build tags from to the `internal/app/dev_dependencies.go` file: 14 | 15 | ```diff 16 | - //go:build dev 17 | - // +build dev 18 | + //go:build dev || e2e 19 | + // +build dev e2e 20 | ``` 21 | 22 | The code in this file will be executed if and only if the build tags used in the Go toolchain match `dev` or `e2e`. 23 | 24 | Now copy the `testdata` directory from the root directory of the project to the `internal/app` directory. This step is **mandatory** because the relative paths to access the files to initialize the services (SQL file, lambda scripts) are different when running the tests from the root directory of the project or from the `internal/app` directory. Therefore, we need a `dev-db.sql` file in that package to be used for testing. This will allow having different data for the tests and for the application in `local dev mode`. 25 | 26 | ## Adding Make goals for running the tests 27 | 28 | In order to simplify the experience of running the integration and the E2E tests, let's update the Makefile in the root of the project with two new targets. Please replace the content of the Makefile with the following: 29 | 30 | ```makefile 31 | build-lambda: 32 | $(MAKE) -C lambda-go zip-lambda 33 | 34 | dev: build-lambda 35 | go run -tags dev -v ./... 36 | 37 | test-integration: 38 | go test -v -count=1 ./... 39 | 40 | test-e2e: 41 | go test -v -count=1 -tags e2e ./internal/app 42 | ``` 43 | 44 | The `test-integration` will run the integration tests, and the `test-e2e` will run the E2E tests. 45 | 46 | At this moment the E2E tests live in the `internal/app` directory, only. Therefore the Make goal will specify that directory when running the E2E tests. 47 | 48 | ## E2E Testing the HTTP endpoints 49 | 50 | Let's replace the entire content of the `router_test.go` file in the `internal/app` directory with the following content: 51 | 52 | ```go 53 | //go:build e2e 54 | // +build e2e 55 | 56 | package app_test 57 | 58 | import ( 59 | "bytes" 60 | "net/http" 61 | "testing" 62 | 63 | "github.com/stretchr/testify/assert" 64 | "github.com/stretchr/testify/require" 65 | "github.com/testcontainers/workshop-go/internal/app" 66 | ) 67 | 68 | func TestRoutesWithDependencies(t *testing.T) { 69 | app := app.SetupApp() 70 | 71 | t.Run("GET /ratings", func(t *testing.T) { 72 | req, err := http.NewRequest("GET", "/ratings?talkId=testcontainers-integration-testing", nil) 73 | require.NoError(t, err) 74 | res, err := app.Test(req, -1) 75 | require.NoError(t, err) 76 | 77 | // we are receiving a 200 because the ratings repository is started 78 | assert.Equal(t, http.StatusOK, res.StatusCode) 79 | }) 80 | 81 | t.Run("POST /ratings", func(t *testing.T) { 82 | body := []byte(`{"talkId":"testcontainers-integration-testing","value":5}`) 83 | 84 | req, err := http.NewRequest("POST", "/ratings", bytes.NewReader(body)) 85 | require.NoError(t, err) 86 | 87 | // we need to set the content type header because we are sending a body 88 | req.Header.Add("Content-Type", "application/json") 89 | 90 | res, err := app.Test(req, -1) 91 | require.NoError(t, err) 92 | 93 | // we are receiving a 200 because the ratings repository is started 94 | assert.Equal(t, http.StatusOK, res.StatusCode) 95 | }) 96 | } 97 | 98 | ``` 99 | 100 | - It uses the `e2e` build tag to include the code only when the `e2e` build tag is present. 101 | - It's an exact copy of the `routes_test.go` file, which checked for the errors, but updating the test names to not indicate that the tests are failing. 102 | - It also updates the assertions to demonstrate that the endpoints are returning a `200` instead of a `500` because the dependencies are started. 103 | 104 | If we run the test in this file, the test panics because the SQL file for the Postgres database is not found. 105 | 106 | ```bash 107 | panic: generic container: create container: created hook: can't copy testdata/dev-db.sql to container: open testdata/dev-db.sql: no such file or directory 108 | 109 | goroutine 1 [running]: 110 | github.com/testcontainers/workshop-go/internal/app.init.0() 111 | /Users/mdelapenya/sourcecode/src/github.com/testcontainers/workshop-go/internal/app/dev_dependencies.go:45 +0x94 112 | ``` 113 | 114 | Let's fix that by adding the SQL file to the `testdata` directory in the `internal/app` directory. From the root directory of the project, run the following command: 115 | 116 | ```bash 117 | cp -R ./testdata internal/app/ 118 | ``` 119 | 120 | Now, if we run the tests again with `make test-e2e`, we are going to see that it passes because the dependencies are indeed started, therefore no error should be thrown: 121 | 122 | ```bash 123 | make test-e2e 124 | go test -v -count=1 -tags e2e ./internal/app 125 | # github.com/testcontainers/workshop-go/internal/app.test 126 | 2025/05/07 13:26:12 github.com/testcontainers/testcontainers-go - Connected to docker: 127 | Server Version: 27.5.0 128 | API Version: 1.47 129 | Operating System: Ubuntu 22.04.5 LTS 130 | Total Memory: 15368 MB 131 | Labels: 132 | cloud.docker.run.version=259.c712f5fd 133 | cloud.docker.run.plugin.version=0.2.20 134 | com.docker.desktop.address=unix:///Users/mdelapenya/Library/Containers/com.docker.docker/Data/docker-cli.sock 135 | Testcontainers for Go Version: v0.37.0 136 | Resolved Docker Host: unix:///var/run/docker.sock 137 | Resolved Docker Socket Path: /var/run/docker.sock 138 | Test SessionID: 8b58086044ecb57abf4e109ce45216352370ea529b9b2fc364680b7147d3e754 139 | Test ProcessID: 8932d66c-ae7c-42ff-ac11-339cc58fc906 140 | 2025/05/07 13:26:12 🐳 Creating container for image postgres:15.3-alpine 141 | 2025/05/07 13:26:13 🐳 Creating container for image testcontainers/ryuk:0.11.0 142 | 2025/05/07 13:26:13 ✅ Container created: e103b4e3c91f 143 | 2025/05/07 13:26:13 🐳 Starting container: e103b4e3c91f 144 | 2025/05/07 13:26:13 ✅ Container started: e103b4e3c91f 145 | 2025/05/07 13:26:13 ⏳ Waiting for container id e103b4e3c91f image: testcontainers/ryuk:0.11.0. Waiting for: &{Port:8080/tcp timeout: PollInterval:100ms skipInternalCheck:false} 146 | 2025/05/07 13:26:14 🔔 Container is ready: e103b4e3c91f 147 | 2025/05/07 13:26:14 ✅ Container created: 120a41a9d627 148 | 2025/05/07 13:26:15 🐳 Starting container: 120a41a9d627 149 | 2025/05/07 13:26:15 ✅ Container started: 120a41a9d627 150 | 2025/05/07 13:26:15 ⏳ Waiting for container id 120a41a9d627 image: postgres:15.3-alpine. Waiting for: &{timeout: deadline:0x14000120dc0 Strategies:[0x14000118120]} 151 | 2025/05/07 13:26:16 🔔 Container is ready: 120a41a9d627 152 | 2025/05/07 13:26:17 🐳 Creating container for image redis:6-alpine 153 | 2025/05/07 13:26:17 ✅ Container created: a46d56c7b406 154 | 2025/05/07 13:26:17 🐳 Starting container: a46d56c7b406 155 | 2025/05/07 13:26:17 ✅ Container started: a46d56c7b406 156 | 2025/05/07 13:26:18 ⏳ Waiting for container id a46d56c7b406 image: redis:6-alpine. Waiting for: &{timeout: deadline:0x14000299348 Strategies:[0x14000380e70 0x14000118840]} 157 | 2025/05/07 13:26:18 🔔 Container is ready: a46d56c7b406 158 | 2025/05/07 13:26:19 🐳 Creating container for image docker.redpanda.com/redpandadata/redpanda:v24.3.7 159 | 2025/05/07 13:26:19 ✅ Container created: d69622d55af7 160 | 2025/05/07 13:26:19 🐳 Starting container: d69622d55af7 161 | 2025/05/07 13:26:20 ✅ Container started: d69622d55af7 162 | 2025/05/07 13:26:20 ⏳ Waiting for container id d69622d55af7 image: docker.redpanda.com/redpandadata/redpanda:v24.3.7. Waiting for: &{timeout: deadline: Strategies:[0x140004c1590 0x140004c15c0 0x140004c15f0]} 163 | 2025/05/07 13:26:21 🔔 Container is ready: d69622d55af7 164 | 2025/05/07 13:26:25 Setting LOCALSTACK_HOST to localhost (to match host-routable address for container) 165 | 2025/05/07 13:26:25 🐳 Creating container for image localstack/localstack:latest 166 | 2025/05/07 13:26:25 ✅ Container created: 32f69766f770 167 | 2025/05/07 13:26:28 🐳 Starting container: 32f69766f770 168 | 2025/05/07 13:26:37 ✅ Container started: 32f69766f770 169 | 2025/05/07 13:26:37 ⏳ Waiting for container id 32f69766f770 image: localstack/localstack:latest. Waiting for: &{timeout:0x14000298528 Port:4566/tcp Path:/_localstack/health StatusCodeMatcher:0x100f58740 ResponseMatcher:0x100ff7750 UseTLS:false AllowInsecure:false TLSConfig: Method:GET Body: Headers:map[] ResponseHeadersMatcher:0x100ff7760 PollInterval:100ms UserInfo: ForceIPv4LocalHost:false} 170 | 2025/05/07 13:26:37 🔔 Container is ready: 32f69766f770 171 | === RUN TestRoutesWithDependencies 172 | === RUN TestRoutesWithDependencies/GET_/ratings 173 | 13:26:38 | 200 | 2.132803083s | 0.0.0.0 | GET | /ratings | - 174 | === RUN TestRoutesWithDependencies/POST_/ratings 175 | 13:26:40 | 200 | 2.559743666s | 0.0.0.0 | POST | /ratings | - 176 | --- PASS: TestRoutesWithDependencies (4.69s) 177 | --- PASS: TestRoutesWithDependencies/GET_/ratings (2.13s) 178 | --- PASS: TestRoutesWithDependencies/POST_/ratings (2.56s) 179 | PASS 180 | ok github.com/testcontainers/workshop-go/internal/app 32.315s 181 | ``` 182 | 183 | Please take a look at these things: 184 | 185 | 1. the `e2e` build tag is passed to the Go toolchain (e.g. `-tags e2e`) in the Makefile goal, so the code in the `internal/app/dev_dependencies.go` file is added to this test execution. 186 | 2. both tests for the endpoints (`GET /ratings` and `POST /ratings`) are now passing because the endpoints are returning a `200` instead of a `500`: the dependencies are started, and the endpoints are not returning an error. 187 | 3. the containers for the dependencies are removed after the tests are executed, thanks to [Ryuk](https://github.com/testcontainers/moby-ryuk), the resource reaper for Testcontainers. 188 | 189 | ### Adding a test for the `GET /` endpoint 190 | 191 | When running in production, the `GET /` endpoint returns metadata with the connections for the dependencies. Let's add a test for that endpoint. 192 | 193 | First make sure the imports are properly updated into the `internal/app/router_test.go` file to include the `encoding/json`, `fmt`, and `strings` packages: 194 | 195 | ```go 196 | import ( 197 | "bytes" 198 | "encoding/json" 199 | "fmt" 200 | "io" 201 | "net/http" 202 | "regexp" 203 | "testing" 204 | 205 | "github.com/stretchr/testify/assert" 206 | "github.com/stretchr/testify/require" 207 | "github.com/testcontainers/workshop-go/internal/app" 208 | ) 209 | ``` 210 | 211 | Then please add the following test function into the `internal/app/router_test.go` file: 212 | 213 | ```go 214 | 215 | // the "GET /" endpoint returns a JSON with metadata including 216 | // the connection strings for the dependencies 217 | type responseType struct { 218 | Connections app.Metadata `json:"metadata"` 219 | } 220 | 221 | func TestRootRouteWithDependencies(t *testing.T) { 222 | app := app.SetupApp() 223 | 224 | req, _ := http.NewRequest("GET", "/", nil) 225 | res, err := app.Test(req, -1) 226 | require.NoError(t, err) 227 | 228 | assert.Equal(t, http.StatusOK, res.StatusCode) 229 | 230 | body, err := io.ReadAll(res.Body) 231 | require.NoError(t, err) 232 | 233 | var response responseType 234 | err = json.Unmarshal(body, &response) 235 | require.NoError(t, err) 236 | 237 | // assert that the different connection strings are set 238 | matches(t, response.Connections.Ratings, `redis://(.*):`) 239 | matches(t, response.Connections.Streams, `(.*):`) 240 | matches(t, response.Connections.Talks, `postgres://postgres:postgres@(.*):`) 241 | matches(t, response.Connections.Lambda, `lambda-url.us-east-1.localhost.localstack.cloud:`) 242 | } 243 | 244 | func matches(t *testing.T, actual string, re string) { 245 | matched, err := regexp.MatchString(re, actual) 246 | require.NoError(t, err) 247 | 248 | assert.True(t, matched, fmt.Sprintf("expected %s to be an URL: %s", actual, re)) 249 | } 250 | ``` 251 | 252 | - It uses the `Metadata` struct from the `internal/app/metadata.go` file to unmarshal the response into a response struct. 253 | - It asserts that the different connection strings are set. Because the ports in which each dependency is started are random, we are using a regular expression to check if the connection string is an URL with the expected format. 254 | 255 | Running the tests again with `make test-e2e` shows that the new test is also passing: 256 | 257 | ```bash 258 | === RUN TestRootRouteWithDependencies 259 | 13:24:34 | 200 | 75.166µs | 0.0.0.0 | GET | / | - 260 | --- PASS: TestRootRouteWithDependencies (0.00s) 261 | === RUN TestRoutesWithDependencies 262 | === RUN TestRoutesWithDependencies/GET_/ratings 263 | 13:24:34 | 200 | 1.882394541s | 0.0.0.0 | GET | /ratings | - 264 | === RUN TestRoutesWithDependencies/POST_/ratings 265 | 13:24:35 | 200 | 2.551489917s | 0.0.0.0 | POST | /ratings | - 266 | --- PASS: TestRoutesWithDependencies (4.43s) 267 | ``` 268 | 269 | ### 270 | [Next: Integration tests for the lambda](step-11-integration-tests-for-the-lambda.md) -------------------------------------------------------------------------------- /step-11-integration-tests-for-the-lambda.md: -------------------------------------------------------------------------------- 1 | # Integration tests for the Go Lambda 2 | 3 | Up to this point, we have worked in the ratings application, which consumes a Go lambda. In this step, we will improve the experience on working on the Go lambda as a separate project. We will add integration tests for the lambda, and we will use Testcontainers to run the lambda on LocalStack. 4 | 5 | ## Adding integration tests for the lambda 6 | 7 | We have a "working" lambda, but we don't have any tests for it. Let's add an integration test for it. It will deploy the lambda into LocalStack and invoke it. 8 | 9 | Let's create a `main_test.go` file in the `lambda-go` folder: 10 | 11 | ```go 12 | package main_test 13 | 14 | import ( 15 | "bytes" 16 | "context" 17 | "encoding/json" 18 | "fmt" 19 | "io" 20 | "net/http" 21 | osexec "os/exec" 22 | "path/filepath" 23 | "strings" 24 | "testing" 25 | "time" 26 | 27 | "github.com/stretchr/testify/require" 28 | "github.com/testcontainers/testcontainers-go" 29 | "github.com/testcontainers/testcontainers-go/exec" 30 | "github.com/testcontainers/testcontainers-go/modules/localstack" 31 | ) 32 | 33 | // buildLambda return the path to the ZIP file used to deploy the lambda function. 34 | func buildLambda(t *testing.T) string { 35 | t.Helper() 36 | 37 | makeCmd := osexec.Command("make", "zip-lambda") 38 | makeCmd.Dir = "." 39 | 40 | err := makeCmd.Run() 41 | require.NoError(t, err) 42 | 43 | return filepath.Join("function.zip") 44 | } 45 | 46 | func TestDeployLambda(t *testing.T) { 47 | ctx := context.Background() 48 | 49 | flagsFn := func() string { 50 | labels := testcontainers.GenericLabels() 51 | flags := "" 52 | for k, v := range labels { 53 | flags = fmt.Sprintf("%s -l %s=%s", flags, k, v) 54 | } 55 | return flags 56 | } 57 | 58 | // get the path to the function.zip file, which lives in the lambda-go folder of the project 59 | zipFile := buildLambda(t) 60 | 61 | var functionURL string 62 | 63 | c, err := localstack.Run(ctx, 64 | "localstack/localstack:latest", 65 | testcontainers.WithEnv(map[string]string{ 66 | "SERVICES": "lambda", 67 | "LAMBDA_DOCKER_FLAGS": flagsFn(), 68 | }), 69 | testcontainers.WithFiles(testcontainers.ContainerFile{ 70 | HostFilePath: zipFile, 71 | ContainerFilePath: "/tmp/function.zip", 72 | }), 73 | testcontainers.CustomizeRequest(testcontainers.GenericContainerRequest{ 74 | ContainerRequest: testcontainers.ContainerRequest{ 75 | LifecycleHooks: []testcontainers.ContainerLifecycleHooks{ 76 | { 77 | PostStarts: []testcontainers.ContainerHook{ 78 | func(ctx context.Context, c testcontainers.Container) error { 79 | lambdaName := "localstack-lambda-url-example" 80 | 81 | // the three commands below are doing the following: 82 | // 1. create a lambda function 83 | // 2. create the URL function configuration for the lambda function 84 | // 3. wait for the lambda function to be active 85 | lambdaCommands := [][]string{ 86 | { 87 | "awslocal", "lambda", 88 | "create-function", "--function-name", lambdaName, 89 | "--runtime", "provided.al2", 90 | "--handler", "bootstrap", 91 | "--role", "arn:aws:iam::111122223333:role/lambda-ex", 92 | "--zip-file", "fileb:///tmp/function.zip", 93 | }, 94 | {"awslocal", "lambda", "create-function-url-config", "--function-name", lambdaName, "--auth-type", "NONE"}, 95 | {"awslocal", "lambda", "wait", "function-active-v2", "--function-name", lambdaName}, 96 | } 97 | for _, cmd := range lambdaCommands { 98 | _, _, err := c.Exec(ctx, cmd) 99 | if err != nil { 100 | t.Fatalf("failed to execute command %s: %s", cmd, err) 101 | } 102 | } 103 | 104 | // 4. get the URL for the lambda function 105 | cmd := []string{ 106 | "awslocal", "lambda", "list-function-url-configs", "--function-name", lambdaName, 107 | } 108 | _, reader, err := c.Exec(ctx, cmd, exec.Multiplexed()) 109 | if err != nil { 110 | t.Fatalf("failed to execute command %s: %s", cmd, err) 111 | } 112 | 113 | buf := new(bytes.Buffer) 114 | _, err = buf.ReadFrom(reader) 115 | if err != nil { 116 | t.Fatalf("failed to read from reader: %s", err) 117 | } 118 | 119 | content := buf.Bytes() 120 | 121 | type FunctionURLConfig struct { 122 | FunctionURLConfigs []struct { 123 | FunctionURL string `json:"FunctionUrl"` 124 | FunctionArn string `json:"FunctionArn"` 125 | CreationTime string `json:"CreationTime"` 126 | LastModifiedTime string `json:"LastModifiedTime"` 127 | AuthType string `json:"AuthType"` 128 | } `json:"FunctionUrlConfigs"` 129 | } 130 | 131 | v := &FunctionURLConfig{} 132 | err = json.Unmarshal(content, v) 133 | if err != nil { 134 | t.Fatalf("failed to unmarshal content: %s", err) 135 | } 136 | 137 | functionURL = v.FunctionURLConfigs[0].FunctionURL 138 | 139 | return nil 140 | }, 141 | }, 142 | }, 143 | }, 144 | }, 145 | }), 146 | ) 147 | testcontainers.CleanupContainer(t, c) 148 | require.NoError(t, err) 149 | 150 | // replace the port with the one exposed by the container 151 | mappedPort, err := c.MappedPort(ctx, "4566/tcp") 152 | require.NoError(t, err) 153 | 154 | url := strings.ReplaceAll(functionURL, "4566", mappedPort.Port()) 155 | 156 | // The latest version of localstack does not add ".localstack.cloud" by default, 157 | // that's why we need to add it to the URL. 158 | url = strings.ReplaceAll(url, ".localhost", ".localhost.localstack.cloud") 159 | 160 | // now we can test the lambda function 161 | 162 | histogram := map[string]string{ 163 | "0": "10", 164 | "1": "20", 165 | "2": "30", 166 | "3": "40", 167 | "4": "50", 168 | "5": "60", 169 | } 170 | 171 | payload := `{"ratings": {` 172 | for rating, count := range histogram { 173 | // we are passing the count as an integer, so we don't need to quote it 174 | payload += `"` + rating + `": ` + count + `,` 175 | } 176 | 177 | if len(histogram) > 0 { 178 | // remove the last comma onl for non-empty histograms 179 | payload = payload[:len(payload)-1] 180 | } 181 | payload += "}}" 182 | 183 | httpClient := http.Client{ 184 | Timeout: 15 * time.Second, 185 | } 186 | 187 | resp, err := httpClient.Post(url, "application/json", bytes.NewBufferString(payload)) 188 | require.NoError(t, err) 189 | 190 | stats, err := io.ReadAll(resp.Body) 191 | require.NoError(t, err) 192 | 193 | expected := `{"avg":3.3333333333333335,"totalCount":210}` 194 | require.Equal(t, expected, string(stats)) 195 | } 196 | ``` 197 | 198 | The test above is doing the following: 199 | 200 | 1. It starts LocalStack with the `lambda` service enabled. 201 | 2. It creates a lambda function and a URL configuration for the lambda function. 202 | 3. It gets the URL for the lambda function. 203 | 4. It invokes the lambda function with a payload containing a histogram of ratings. 204 | 205 | The test is using the `testcontainers-go` library to start LocalStack and to execute commands inside the container. It is also using the `awslocal` command to interact with the LocalStack container. 206 | 207 | Let's replace the contents of the `Makefile` for the lambda-go project. We are adding a new target for running the integration tests: 208 | 209 | ```makefile 210 | mod-tidy: 211 | go mod tidy 212 | 213 | build-lambda: mod-tidy 214 | # If you are using Testcontainers Cloud, please add 'GOARCH=amd64' in order to get the localstack's lambdas using the right architecture 215 | GOOS=linux go build -tags lambda.norpc -o bootstrap main.go 216 | 217 | test: mod-tidy 218 | go test -v -count=1 ./... 219 | 220 | zip-lambda: build-lambda 221 | zip -j function.zip bootstrap 222 | ``` 223 | 224 | Now run the integration tests with your IDE or from a terminal, in the lambda directory, but first update the Go dependencies with the `make mod-tidy` command: 225 | 226 | ```shell 227 | $ cd lambda-go 228 | $ make test 229 | go mod tidy 230 | go test -v -count=1 ./... 231 | === RUN TestDeployLambda 232 | 2025/05/07 13:27:48 github.com/testcontainers/testcontainers-go - Connected to docker: 233 | Server Version: 27.5.0 234 | API Version: 1.47 235 | Operating System: Ubuntu 22.04.5 LTS 236 | Total Memory: 15368 MB 237 | Labels: 238 | cloud.docker.run.version=259.c712f5fd 239 | cloud.docker.run.plugin.version=0.2.20 240 | com.docker.desktop.address=unix:///Users/mdelapenya/Library/Containers/com.docker.docker/Data/docker-cli.sock 241 | Testcontainers for Go Version: v0.37.0 242 | Resolved Docker Host: unix:///var/run/docker.sock 243 | Resolved Docker Socket Path: /var/run/docker.sock 244 | Test SessionID: e2a7d32bf743b96698083a73e6d8e091f30cd208028037421e783e8a3840fd43 245 | Test ProcessID: 7fe35795-7665-491d-9523-f1a6118fb8a9 246 | 2025/05/07 13:27:48 Setting LOCALSTACK_HOST to localhost (to match host-routable address for container) 247 | 2025/05/07 13:27:48 🐳 Creating container for image localstack/localstack:latest 248 | 2025/05/07 13:27:48 🐳 Creating container for image testcontainers/ryuk:0.11.0 249 | 2025/05/07 13:27:49 ✅ Container created: 1af05cd523b9 250 | 2025/05/07 13:27:49 🐳 Starting container: 1af05cd523b9 251 | 2025/05/07 13:27:49 ✅ Container started: 1af05cd523b9 252 | 2025/05/07 13:27:49 ⏳ Waiting for container id 1af05cd523b9 image: testcontainers/ryuk:0.11.0. Waiting for: &{Port:8080/tcp timeout: PollInterval:100ms skipInternalCheck:false} 253 | 2025/05/07 13:27:50 🔔 Container is ready: 1af05cd523b9 254 | 2025/05/07 13:27:50 ✅ Container created: 4df47531ebad 255 | 2025/05/07 13:27:51 🐳 Starting container: 4df47531ebad 256 | 2025/05/07 13:27:59 ✅ Container started: 4df47531ebad 257 | 2025/05/07 13:27:59 ⏳ Waiting for container id 4df47531ebad image: localstack/localstack:latest. Waiting for: &{timeout:0x140001265b0 Port:4566/tcp Path:/_localstack/health StatusCodeMatcher:0x1011f3930 ResponseMatcher:0x10125b620 UseTLS:false AllowInsecure:false TLSConfig: Method:GET Body: Headers:map[] ResponseHeadersMatcher:0x10125b630 PollInterval:100ms UserInfo: ForceIPv4LocalHost:false} 258 | 2025/05/07 13:28:00 🔔 Container is ready: 4df47531ebad 259 | 2025/05/07 13:28:01 🐳 Stopping container: 4df47531ebad 260 | 2025/05/07 13:28:04 ✅ Container stopped: 4df47531ebad 261 | 2025/05/07 13:28:04 🐳 Terminating container: 4df47531ebad 262 | 2025/05/07 13:28:04 🚫 Container terminated: 4df47531ebad 263 | --- PASS: TestDeployLambda (17.93s) 264 | PASS 265 | ok github.com/testcontainers/workshop-go/lambda-go 18.756s 266 | ``` 267 | 268 | You'll probably see the `go.mod` and `go.sum` file to change, adding the `testcontainers-go` library and its Go dependencies. 269 | 270 | ## Making the tests to fail 271 | 272 | Let's introduce a bug in the lambda function and see how the tests will fail. In the `main.go` file, let's change how the average of the ratings is calculated: 273 | 274 | 275 | ```diff 276 | var avg float64 277 | if totalCount > 0 { 278 | - avg = float64(sum) / float64(totalCount) 279 | + avg = float64(sum) * float64(totalCount) 280 | } 281 | ``` 282 | 283 | Now run the tests, with your IDE or from a terminal: 284 | 285 | ```shell 286 | $ make test 287 | go test -v -count=1 ./... 288 | === RUN TestDeployLambda 289 | 2025/05/07 13:30:26 github.com/testcontainers/testcontainers-go - Connected to docker: 290 | Server Version: 27.5.0 291 | API Version: 1.47 292 | Operating System: Ubuntu 22.04.5 LTS 293 | Total Memory: 15368 MB 294 | Labels: 295 | cloud.docker.run.version=259.c712f5fd 296 | cloud.docker.run.plugin.version=0.2.20 297 | com.docker.desktop.address=unix:///Users/mdelapenya/Library/Containers/com.docker.docker/Data/docker-cli.sock 298 | Testcontainers for Go Version: v0.37.0 299 | Resolved Docker Host: unix:///var/run/docker.sock 300 | Resolved Docker Socket Path: /var/run/docker.sock 301 | Test SessionID: 7ed8e0d5c1fb58d25e1c52f105288d105b645d89f000355c46de9acd9d622ec5 302 | Test ProcessID: e35bd5f4-db46-4a61-afce-4279c957cb82 303 | 2025/05/07 13:30:26 Setting LOCALSTACK_HOST to localhost (to match host-routable address for container) 304 | 2025/05/07 13:30:26 🐳 Creating container for image localstack/localstack:latest 305 | 2025/05/07 13:30:27 🐳 Creating container for image testcontainers/ryuk:0.11.0 306 | 2025/05/07 13:30:27 ✅ Container created: 0a30b25b9bf9 307 | 2025/05/07 13:30:27 🐳 Starting container: 0a30b25b9bf9 308 | 2025/05/07 13:30:27 ✅ Container started: 0a30b25b9bf9 309 | 2025/05/07 13:30:27 ⏳ Waiting for container id 0a30b25b9bf9 image: testcontainers/ryuk:0.11.0. Waiting for: &{Port:8080/tcp timeout: PollInterval:100ms skipInternalCheck:false} 310 | 2025/05/07 13:30:28 🔔 Container is ready: 0a30b25b9bf9 311 | 2025/05/07 13:30:28 ✅ Container created: 30af32569a44 312 | 2025/05/07 13:30:29 🐳 Starting container: 30af32569a44 313 | 2025/05/07 13:30:38 ✅ Container started: 30af32569a44 314 | 2025/05/07 13:30:38 ⏳ Waiting for container id 30af32569a44 image: localstack/localstack:latest. Waiting for: &{timeout:0x1400048ea90 Port:4566/tcp Path:/_localstack/health StatusCodeMatcher:0x105437930 ResponseMatcher:0x10549f620 UseTLS:false AllowInsecure:false TLSConfig: Method:GET Body: Headers:map[] ResponseHeadersMatcher:0x10549f630 PollInterval:100ms UserInfo: ForceIPv4LocalHost:false} 315 | 2025/05/07 13:30:39 🔔 Container is ready: 30af32569a44 316 | main_test.go:183: 317 | Error Trace: /Users/mdelapenya/sourcecode/src/github.com/testcontainers/workshop-go/lambda-go/main_test.go:183 318 | Error: Not equal: 319 | expected: "{\"avg\":3.3333333333333335,\"totalCount\":210}" 320 | actual : "{\"avg\":147000,\"totalCount\":210}" 321 | 322 | Diff: 323 | --- Expected 324 | +++ Actual 325 | @@ -1 +1 @@ 326 | -{"avg":3.3333333333333335,"totalCount":210} 327 | +{"avg":147000,"totalCount":210} 328 | Test: TestDeployLambda 329 | 2025/05/07 13:30:39 🐳 Stopping container: 30af32569a44 330 | 2025/05/07 13:30:44 ✅ Container stopped: 30af32569a44 331 | 2025/05/07 13:30:44 🐳 Terminating container: 30af32569a44 332 | 2025/05/07 13:30:44 🚫 Container terminated: 30af32569a44 333 | --- FAIL: TestDeployLambda (20.30s) 334 | FAIL 335 | FAIL github.com/testcontainers/workshop-go/lambda-go 21.239s 336 | FAIL 337 | make: *** [test] Error 1 338 | ``` 339 | 340 | As expected, the test failed because the lambda function is returning an incorrect average: 341 | 342 | ```text 343 | main_test.go:183: 344 | Error Trace: /Users/mdelapenya/sourcecode/src/github.com/testcontainers/workshop-go/lambda-go/main_test.go:183 345 | Error: Not equal: 346 | expected: "{\"avg\":3.3333333333333335,\"totalCount\":210}" 347 | actual : "{\"avg\":147000,\"totalCount\":210}" 348 | ``` 349 | 350 | Rollback the change in the `main.go` file, and run the tests again, they will pass again. 351 | 352 | ### 353 | [Next: exploring the running app](step-12-exploring-the-running-app.md) -------------------------------------------------------------------------------- /step-12-exploring-the-running-app.md: -------------------------------------------------------------------------------- 1 | # Exploring the running app 2 | 3 | Once the application is running, we might want to connect to the running containers to inspect the data in the Postgres database or the elements in the Redis store. 4 | 5 | With [Testcontainers Desktop](https://www.testcontainers.com/desktop), it's easy to do that. 6 | 7 | To access the different services, please use your favorite client to connect to them and inspect the data. For the workshop, which was built with VSCode, we are using a [database client extension](https://doc.database-client.com/#/), as it allows connecting to different technologies. 8 | 9 | ## Connecting to the Database 10 | 11 | From the root directory of the workshop, let's first start the application with `make dev`. 12 | 13 | Now create a connection to the Postgres database. When we set `postgres` as user and password, and `talks-db` as database, the connection will fail with the following error: 14 | 15 | > Connection error!connect ECONNREFUSED 127.0.0.1:5432 16 | 17 | The well-known port for Postgres is 5432, but the connection is refused. Why? 18 | 19 | By default, Testcontainers starts the containers and maps the ports to a random available port on the host. So, we need to find out the mapped port to connect to the database: put a break point, inspect the container, or check the logs, among other things. 20 | 21 | Instead, we can use Testcontainers Desktop fixed ports support to connect to the database. 22 | 23 | Open Testcontainers Desktop, and select the `Services` -> `Open config location`. 24 | It will open a directory with the example configuration files for commonly used services. 25 | 26 | Copy the `postgres.toml.example` to `postgres.toml`, and update it's content to the following: 27 | 28 | ```toml 29 | ports = [ 30 | {local-port = 5432, container-port = 5432}, 31 | ] 32 | 33 | selector.image-names = ["postgres"] 34 | ``` 35 | 36 | This configuration will map Postgres container port 5432 to the host port 5432. If we take a look at the Desktop UI, we will see that the Postgres service now appears in the services list, including additional sub-entries for tailing container logs and getting a shell into the container. 37 | 38 | Now, let's try to create the connection again. This time, it will work. 39 | 40 | And it will also work for the Redis store and the Redpanda streaming queue as well. Simply copy the `redis.toml.example` to `redis.toml`, and update it's content to the following: 41 | 42 | ```toml 43 | ports = [ 44 | { local-port = 6379, container-port = 6379 } 45 | ] 46 | 47 | selector.image-names = ["redis"] 48 | ``` 49 | 50 | And finally add the `redpanda.toml`, and update it's content to the following: 51 | 52 | ```toml 53 | ports = [ 54 | {local-port = 9092, container-port = 9093}, 55 | ] 56 | 57 | selector.image-names = ["docker.redpanda.com/redpandadata/redpanda"] 58 | ``` 59 | -------------------------------------------------------------------------------- /step-2-exploring-the-app.md: -------------------------------------------------------------------------------- 1 | # Step 2: Exploring the app 2 | 3 | The app is a simple microservice for rating conference talks. It provides a web API to track the ratings of the talks in real time, storing the results in a PostgreSQL database and a Redis cache, and using Redpanda as a broker for the event stream. Finally, it will use an AWS lambda function to calculate some statistics about the ratings of a talk. 4 | 5 | ## Storage layer 6 | 7 | ### SQL database with the talks 8 | 9 | When a rating is submitted, we must verify that the talk for the given ID is present in the database. 10 | 11 | The database of choice is PostgreSQL, accessed with [jackc/pgx](https://github.com/jackc/pgx) PostgreSQL Driver. 12 | 13 | Check `internal/talks/repo.go`. 14 | 15 | ### Redis 16 | 17 | The application stores the ratings in a Redis store with [redis/go-redis](https://github.com/redis/go-redis) Redis client. 18 | 19 | Check `internal/ratings/repo.go`. 20 | 21 | ### Redpanda 22 | 23 | The application uses ES/CQRS to materialize the events into the state. Redpanda acts as a broker and the [twmb/franz-go](https://github.com/twmb/franz-go) Kafa client. 24 | 25 | Check `internal/streams/broker.go`. 26 | 27 | ## Cloud layer 28 | 29 | ### AWS Lambdas 30 | 31 | The application uses AWS lambda functions to calculate some rating statistics of a talk. The lambda functions are invoked by the application when the ratings for a talk are requested, using HTTP calls to the function URL of the lambda. 32 | 33 | Check `internal/ratings/lambda_client.go`. 34 | 35 | ### Go Lambda 36 | 37 | The Go lambda application will live in the `lambda-go` directory. It is a simple lambda function that receives the talk ID and returns some statistics of the talk. 38 | 39 | We'll start adding the code for the lambda in the `Step 7`. 40 | 41 | ## Web application 42 | 43 | ### API 44 | 45 | The API is a [GoFiber](https://gofiber.io/) REST controller and exposes three endpoints: 46 | 47 | * `POST /ratings { "talkId": ?, "value": 1-5 }` to add a rating for a talk 48 | * `GET /ratings?talkId=?` to get the histogram of ratings of the given talk 49 | * `GET /` returns metadata about the application, including the connection strings to all the backend services (PostgreSQL, Redis, Redpanda). 50 | 51 | Check `internal/app/handlers.go`. 52 | 53 | ### 54 | [Next: Running the app locally](step-3-running-the-app-locally.md) 55 | -------------------------------------------------------------------------------- /step-3-running-the-app-locally.md: -------------------------------------------------------------------------------- 1 | # Step 3: Running the application locally 2 | 3 | Go applications are usually started with `go run` while in development mode. 4 | 5 | In order to simplify the experience of running the application locally, there is a Makefile in the root of the project with the `dev` target. This target starts the application in `local dev mode`, and it will basically be the entrypoint to start the application. 6 | 7 | ## Start the application 8 | 9 | In a terminal, run the following command: 10 | 11 | ```bash 12 | make dev 13 | ``` 14 | 15 | A similar output log will appear: 16 | 17 | ```text 18 | go run ./... 19 | 20 | ┌───────────────────────────────────────────────────┐ 21 | │ Fiber v2.52.6 │ 22 | │ http://127.0.0.1:8080 │ 23 | │ (bound on host 0.0.0.0 and port 8080) │ 24 | │ │ 25 | │ Handlers ............. 5 Processes ........... 1 │ 26 | │ Prefork ....... Disabled PID ............. 17433 │ 27 | └───────────────────────────────────────────────────┘ 28 | 29 | 30 | ``` 31 | 32 | If we open the browser in the URL http://localhost:8080, we will see the metadata of the application, but all the values are empty: 33 | 34 | ```json 35 | {"metadata":{"ratings_lambda":"","ratings":"","streams":"","talks":""}} 36 | ``` 37 | 38 | On the contrary, if we open the ratings endpoint from the API (http://localhost:8080/ratings?talkId=testcontainers-integration-testing), we will get a 500 error and a similar message: 39 | 40 | ```text 41 | {"message":"failed to connect to `host=/private/tmp user=mdelapenya database=`: dial error (dial unix /private/tmp/.s.PGSQL.5432: connect: no such file or directory)"} 42 | ``` 43 | 44 | The logs will show the following: 45 | 46 | ```text 47 | Unable to connect to database: failed to connect to `host=/private/tmp user=mdelapenya database=`: dial error (dial unix /private/tmp/.s.PGSQL.5432: connect: no such file or directory) 48 | 13:03:27 | 500 | 4.20625ms | 127.0.0.1 | GET | /ratings | - 49 | ``` 50 | 51 | It seems the application is not able to connect to the database. Let's fix it. 52 | 53 | ### 54 | [Next: Dev Mode with Testcontainers](step-4-dev-mode-with-testcontainers.md) -------------------------------------------------------------------------------- /step-4-dev-mode-with-testcontainers.md: -------------------------------------------------------------------------------- 1 | # Step 4: Dev mode with Testcontainers 2 | 3 | Remember the Makefile in the root of the project with the `dev` target, the one that starts the application in `local dev mode`? We are going to learn in this workshop how to leverage Go's build tags and init functions to selectively execute code when a `dev` tag is passed to the Go toolchain, only while developing our application. So when the application is started, it will start the runtime dependencies as Docker containers, leveraging Testcontainers for Go. 4 | 5 | To understand how the `local dev mode` with Testcontainers for Go works, please read the following blog post: https://www.docker.com/blog/local-development-of-go-applications-with-testcontainers/ 6 | 7 | ## Adding Talks store 8 | 9 | When the application started, it failed because we need to connect to a Postgres database including some data before we can do anything useful with the talks. 10 | 11 | Let's add a `testdata/dev-db.sql` file with the following content: 12 | 13 | ```sql 14 | CREATE TABLE IF NOT EXISTS talks ( 15 | id serial, 16 | uuid varchar(255), 17 | title varchar(255) 18 | ); 19 | 20 | INSERT 21 | INTO talks (uuid, title) 22 | VALUES ('testcontainers-integration-testing', 'Modern Integration Testing with Testcontainers') 23 | ON CONFLICT do nothing; 24 | 25 | INSERT 26 | INTO talks (uuid, title) 27 | VALUES ('flight-of-the-gopher', 'A look at Go scheduler') 28 | ON CONFLICT do nothing; 29 | 30 | ``` 31 | 32 | Also create an `internal/app/dev_dependencies.go` file with the following content: 33 | 34 | ```go 35 | //go:build dev 36 | // +build dev 37 | 38 | package app 39 | 40 | import ( 41 | "context" 42 | "path/filepath" 43 | "time" 44 | 45 | "github.com/testcontainers/testcontainers-go" 46 | "github.com/testcontainers/testcontainers-go/modules/postgres" 47 | "github.com/testcontainers/testcontainers-go/wait" 48 | ) 49 | 50 | // init will be used to start up the containers for development mode. It will use 51 | // testcontainers-go to start up the following containers: 52 | // - Redis: store for ratings 53 | // All the containers will contribute their connection strings to the Connections struct. 54 | // Please read this blog post for more information: https://www.atomicjar.com/2023/08/local-development-of-go-applications-with-testcontainers/ 55 | func init() { 56 | startupDependenciesFns := []func() (testcontainers.Container, error){ 57 | startTalksStore, 58 | } 59 | 60 | for _, fn := range startupDependenciesFns { 61 | _, err := fn() 62 | if err != nil { 63 | panic(err) 64 | } 65 | } 66 | } 67 | 68 | 69 | func startTalksStore() (testcontainers.Container, error) { 70 | ctx := context.Background() 71 | c, err := postgres.Run(ctx, 72 | "postgres:15.3-alpine", 73 | postgres.WithInitScripts(filepath.Join(".", "testdata", "dev-db.sql")), 74 | postgres.WithDatabase("talks-db"), 75 | postgres.WithUsername("postgres"), 76 | postgres.WithPassword("postgres"), 77 | testcontainers.WithWaitStrategy( 78 | wait.ForLog("database system is ready to accept connections"). 79 | WithOccurrence(2).WithStartupTimeout(15*time.Second)), 80 | ) 81 | if err != nil { 82 | return nil, err 83 | } 84 | 85 | talksConn, err := c.ConnectionString(ctx) 86 | if err != nil { 87 | return nil, err 88 | } 89 | 90 | Connections.Talks = talksConn 91 | return c, nil 92 | } 93 | 94 | ``` 95 | 96 | Let's understand what we have done here: 97 | 98 | - The first two lines include the build tag `dev` and the build constraint `+build dev`. This means that the code in this file will only be compiled when the `dev` tag is passed to the Go toolchain. 99 | - The `init` function will be executed when the application starts. It will start the runtime dependencies as Docker containers, leveraging Testcontainers for Go. 100 | - The `init` function contains a `startupDependenciesFns` slice with the functions that will start the containers. In this case, we only have one function, `startTalksStore`. 101 | - The `startTalksStore` function will start a Postgres database with the `testdata/dev-db.sql` file as initialization script. 102 | - The `Connections.Talks` variable receives the connection string used to connect to the database. The code is overriding the default connection string for the database, which is read from an environment variable (see `internal/app/metadata.go`). 103 | 104 | Now run `go mod tidy` from the root of the project to download the Go dependencies. 105 | 106 | ## Update the make dev target 107 | 108 | The `make dev` target in the Makefile is using the `go run` command to start the application. We need to pass the `dev` build tag to the Go toolchain, so the `init` function in `internal/app/dev_dependencies.go` is executed. 109 | 110 | Update the `make dev` target in the Makefile to pass the `dev` build tag: 111 | 112 | ```makefile 113 | dev: 114 | go run -tags dev -v ./... 115 | ``` 116 | 117 | Finally, stop the application with Ctrl+C and run the application again with `make dev`. This time, the application will start the Postgres database and the application will be able to connect to it. 118 | 119 | ```text 120 | go run -tags dev -v ./... 121 | 122 | ┌───────────────────────────────────────────────────┐ 123 | │ Fiber v2.52.6 │ 124 | │ http://127.0.0.1:8080 │ 125 | │ (bound on host 0.0.0.0 and port 8080) │ 126 | │ │ 127 | │ Handlers ............. 6 Processes ........... 1 │ 128 | │ Prefork ....... Disabled PID ............. 20390 │ 129 | └───────────────────────────────────────────────────┘ 130 | 131 | ``` 132 | 133 | If we open a second terminal and check the containers, we will see the Postgres database running: 134 | 135 | ```text 136 | $ docker ps 137 | CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 138 | 2d5155cb8e58 postgres:15.3-alpine "docker-entrypoint.s…" 36 seconds ago Up 35 seconds 0.0.0.0:32771->5432/tcp, :::32771->5432/tcp gifted_villani 139 | ``` 140 | 141 | On the contrary, if we open again the ratings endpoint from the API (http://localhost:8080/ratings?talkId=testcontainers-integration-testing), we'll still get a 500 error, but with a different message: 142 | 143 | ```text 144 | {"message":"redis: invalid URL scheme: "} 145 | ``` 146 | 147 | The logs will show the following: 148 | 149 | ```text 150 | Unable to connect to Redis: redis: invalid URL scheme: 151 | 13:07:51 | 500 | 903.018542ms | 127.0.0.1 | GET | /ratings | - 152 | ``` 153 | 154 | Now it seems the application is able to connect to the database, but not to Redis. Let's fix it, but first stop the application with Ctrl+C, so the application and the dependencies are terminated. 155 | 156 | Let's add Redis as a dependency in development mode. 157 | 158 | ### 159 | [Next: Adding Redis](step-5-adding-redis.md) 160 | -------------------------------------------------------------------------------- /step-5-adding-redis.md: -------------------------------------------------------------------------------- 1 | # Step 5: Adding Redis 2 | 3 | When the application started, and the ratings for a talk were requested, it failed because we need to connect to a Redis database before we can do anything useful with the ratings. 4 | 5 | Let's add a Redis instance using Testcontainers for Go. 6 | 7 | 1. In the `internal/app/dev_dependencies.go` file, add the following imports: 8 | 9 | ```go 10 | import ( 11 | "context" 12 | "path/filepath" 13 | "time" 14 | 15 | "github.com/testcontainers/testcontainers-go" 16 | "github.com/testcontainers/testcontainers-go/modules/postgres" 17 | "github.com/testcontainers/testcontainers-go/modules/redis" 18 | "github.com/testcontainers/testcontainers-go/wait" 19 | ) 20 | ``` 21 | 22 | 2. Add this function to the file: 23 | 24 | ```go 25 | func startRatingsStore() (testcontainers.Container, error) { 26 | ctx := context.Background() 27 | 28 | c, err := redis.Run(ctx, "redis:6-alpine") 29 | if err != nil { 30 | return nil, err 31 | } 32 | 33 | ratingsConn, err := c.ConnectionString(ctx) 34 | if err != nil { 35 | return nil, err 36 | } 37 | 38 | Connections.Ratings = ratingsConn 39 | return c, nil 40 | } 41 | ``` 42 | 43 | 3. Update the comments for the init function `startupDependenciesFn` slice to include the Redis store: 44 | 45 | ```go 46 | // init will be used to start up the containers for development mode. It will use 47 | // testcontainers-go to start up the following containers: 48 | // - Postgres: store for talks 49 | // - Redis: store for ratings 50 | // All the containers will contribute their connection strings to the Connections struct. 51 | // Please read this blog post for more information: https://www.atomicjar.com/2023/08/local-development-of-go-applications-with-testcontainers/ 52 | func init() { 53 | ``` 54 | 55 | 4. Update the `startupDependenciesFn` slice to include the function that starts the ratings store: 56 | 57 | ```go 58 | startupDependenciesFns := []func() (testcontainers.Container, error){ 59 | startTalksStore, 60 | startRatingsStore, 61 | } 62 | ``` 63 | 64 | The complete file should look like this: 65 | 66 | ```go 67 | //go:build dev 68 | // +build dev 69 | 70 | package app 71 | 72 | import ( 73 | "context" 74 | "path/filepath" 75 | "time" 76 | 77 | "github.com/testcontainers/testcontainers-go" 78 | "github.com/testcontainers/testcontainers-go/modules/postgres" 79 | "github.com/testcontainers/testcontainers-go/modules/redis" 80 | "github.com/testcontainers/testcontainers-go/wait" 81 | ) 82 | 83 | // init will be used to start up the containers for development mode. It will use 84 | // testcontainers-go to start up the following containers: 85 | // - Postgres: store for talks 86 | // - Redis: store for ratings 87 | // All the containers will contribute their connection strings to the Connections struct. 88 | // Please read this blog post for more information: https://www.atomicjar.com/2023/08/local-development-of-go-applications-with-testcontainers/ 89 | func init() { 90 | startupDependenciesFns := []func() (testcontainers.Container, error){ 91 | startTalksStore, 92 | startRatingsStore, 93 | } 94 | 95 | for _, fn := range startupDependenciesFns { 96 | _, err := fn() 97 | if err != nil { 98 | panic(err) 99 | } 100 | } 101 | } 102 | 103 | func startRatingsStore() (testcontainers.Container, error) { 104 | ctx := context.Background() 105 | 106 | c, err := redis.Run(ctx, "redis:6-alpine") 107 | if err != nil { 108 | return nil, err 109 | } 110 | 111 | ratingsConn, err := c.ConnectionString(ctx) 112 | if err != nil { 113 | return nil, err 114 | } 115 | 116 | Connections.Ratings = ratingsConn 117 | return c, nil 118 | } 119 | 120 | func startTalksStore() (testcontainers.Container, error) { 121 | ctx := context.Background() 122 | c, err := postgres.Run(ctx, 123 | "postgres:15.3-alpine", 124 | postgres.WithInitScripts(filepath.Join(".", "testdata", "dev-db.sql")), 125 | postgres.WithDatabase("talks-db"), 126 | postgres.WithUsername("postgres"), 127 | postgres.WithPassword("postgres"), 128 | testcontainers.WithWaitStrategy( 129 | wait.ForLog("database system is ready to accept connections"). 130 | WithOccurrence(2).WithStartupTimeout(15*time.Second)), 131 | ) 132 | if err != nil { 133 | return nil, err 134 | } 135 | 136 | talksConn, err := c.ConnectionString(ctx) 137 | if err != nil { 138 | return nil, err 139 | } 140 | 141 | Connections.Talks = talksConn 142 | return c, nil 143 | } 144 | 145 | ``` 146 | 147 | Now run `go mod tidy` from the root of the project to download the Go dependencies, only the Testcontainers for Go's Redis module. 148 | 149 | Finally, stop the application with Ctrl+C and run the application again with `make dev`. This time, the application will start the Redis store and the application will be able to connect to it. 150 | 151 | ```text 152 | go run -tags dev -v ./... 153 | 154 | ┌───────────────────────────────────────────────────┐ 155 | │ Fiber v2.52.6 │ 156 | │ http://127.0.0.1:8080 │ 157 | │ (bound on host 0.0.0.0 and port 8080) │ 158 | │ │ 159 | │ Handlers ............. 6 Processes ........... 1 │ 160 | │ Prefork ....... Disabled PID ............. 22626 │ 161 | └───────────────────────────────────────────────────┘ 162 | 163 | ``` 164 | 165 | In the second terminal, check the containers, we will see the Redis store running alongside the Postgres database: 166 | 167 | ```text 168 | $ docker ps 169 | CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 170 | 4ef6b38b1baa redis:6-alpine "docker-entrypoint.s…" 2 seconds ago Up 1 second 0.0.0.0:32776->6379/tcp, :::32776->6379/tcp epic_haslett 171 | 0fe7e41a8954 postgres:15.3-alpine "docker-entrypoint.s…" 14 seconds ago Up 13 seconds 0.0.0.0:32775->5432/tcp, :::32775->5432/tcp affectionate_cori 172 | ``` 173 | 174 | If we now open the ratings endpoint from the API (http://localhost:8080/ratings?talkId=testcontainers-integration-testing), then a 200 OK response code is returned, but there are no ratings for the given talk: 175 | 176 | ```text 177 | {"ratings":{}} 178 | ``` 179 | 180 | With `curl`: 181 | 182 | ```shell 183 | curl -X GET http://localhost:8080/ratings\?talkId\=testcontainers-integration-testing 184 | {"ratings":{}}% 185 | ``` 186 | 187 | If we check the logs, we'll notice an error regarding the connection to the AWS lambda function that is used to calculate some statistics for a given rating. By design, if the AWS lambda is not available, the application will not add the statistics to the response, so it's expected to see this error but a valid HTTP response: 188 | 189 | ```text 190 | 2025/05/07 13:10:15 error calling lambda function: Post "": unsupported protocol scheme "" 191 | 13:10:13 | 200 | 1.253877458s | 127.0.0.1 | GET | /ratings | - 192 | ``` 193 | 194 | We are going to fix that in the next steps, adding a way to reproduce the AWS lambda but in a local environment, using LocalStack and Testcontainers for Go. 195 | 196 | Nevertheless, now it seems the application is able to connect to the database, and to Redis. Let's try to send a POST request adding a rating for the talk, using the JSON payload format accepted by the API endopint: 197 | 198 | ```json 199 | { 200 | "talkId": "testcontainers-integration-testing", 201 | "value": 5 202 | } 203 | ``` 204 | 205 | In a terminal, let's send a POST request with `curl`: 206 | 207 | ```shell 208 | curl -X POST -H "Content-Type: application/json" http://localhost:8080/ratings -d '{"talkId":"testcontainers-integration-testing", "value":5}' 209 | ``` 210 | 211 | This time, the response is a 500 error, but different: 212 | 213 | ```json 214 | {"message":"unable to dial: dial tcp :9092: connect: connection refused"}% 215 | ``` 216 | 217 | And in the logs, we'll see the following error: 218 | 219 | ```text 220 | Unable to ping the streams: unable to dial: dial tcp :9092: connect: connection refused 221 | 13:11:05 | 500 | 967.728458ms | 127.0.0.1 | POST | /ratings | - 222 | ``` 223 | 224 | If we recall correctly, the application was using a message queue to send the ratings before storing them in Redis (see `internal/app/handlers.go`), so we need to add a message queue for that. Let's fix it, but first stop the application with Ctrl+C and the application and the dependencies will be terminated. 225 | 226 | ### 227 | [Next: Adding the streaming queue](step-6-adding-redpanda.md) -------------------------------------------------------------------------------- /step-6-adding-redpanda.md: -------------------------------------------------------------------------------- 1 | # Step 6: Adding Redpanda 2 | 3 | When the application started, it failed because we need to connect to a message queue before we can add the ratings for a talk. 4 | 5 | Let's add a Redpanda instance using Testcontainers for Go. 6 | 7 | 1. In the `internal/app/dev_dependencies.go` file, add the following imports: 8 | 9 | ```go 10 | import ( 11 | "context" 12 | "path/filepath" 13 | "time" 14 | 15 | "github.com/testcontainers/testcontainers-go" 16 | "github.com/testcontainers/testcontainers-go/modules/postgres" 17 | "github.com/testcontainers/testcontainers-go/modules/redis" 18 | "github.com/testcontainers/testcontainers-go/modules/redpanda" 19 | "github.com/testcontainers/testcontainers-go/wait" 20 | ) 21 | ``` 22 | 23 | 2. Add this function to the file: 24 | 25 | ```go 26 | func startStreamingQueue() (testcontainers.Container, error) { 27 | ctx := context.Background() 28 | 29 | c, err := redpanda.Run( 30 | ctx, 31 | "docker.redpanda.com/redpandadata/redpanda:v24.3.7", 32 | redpanda.WithAutoCreateTopics(), 33 | ) 34 | 35 | seedBroker, err := c.KafkaSeedBroker(ctx) 36 | if err != nil { 37 | return nil, err 38 | } 39 | 40 | Connections.Streams = seedBroker 41 | return c, nil 42 | } 43 | ``` 44 | 45 | 3. Update the comments for the init function `startupDependenciesFn` slice to include the Redpanda queue: 46 | 47 | ```go 48 | // init will be used to start up the containers for development mode. It will use 49 | // testcontainers-go to start up the following containers: 50 | // - Postgres: store for talks 51 | // - Redis: store for ratings 52 | // - Redpanda: message queue for the ratings 53 | // All the containers will contribute their connection strings to the Connections struct. 54 | // Please read this blog post for more information: https://www.atomicjar.com/2023/08/local-development-of-go-applications-with-testcontainers/ 55 | func init() { 56 | ``` 57 | 58 | 4. Update the `startupDependenciesFn` slice to include the function that starts the streaming queue: 59 | 60 | ```go 61 | startupDependenciesFns := []func() (testcontainers.Container, error){ 62 | startTalksStore, 63 | startRatingsStore, 64 | startStreamingQueue, 65 | } 66 | ``` 67 | 68 | The complete file should look like this: 69 | 70 | ```go 71 | //go:build dev 72 | // +build dev 73 | 74 | package app 75 | 76 | import ( 77 | "context" 78 | "path/filepath" 79 | "time" 80 | 81 | "github.com/testcontainers/testcontainers-go" 82 | "github.com/testcontainers/testcontainers-go/modules/postgres" 83 | "github.com/testcontainers/testcontainers-go/modules/redis" 84 | "github.com/testcontainers/testcontainers-go/modules/redpanda" 85 | "github.com/testcontainers/testcontainers-go/wait" 86 | ) 87 | 88 | // init will be used to start up the containers for development mode. It will use 89 | // testcontainers-go to start up the following containers: 90 | // - Postgres: store for talks 91 | // - Redis: store for ratings 92 | // - Redpanda: message queue for the ratings 93 | // All the containers will contribute their connection strings to the Connections struct. 94 | // Please read this blog post for more information: https://www.atomicjar.com/2023/08/local-development-of-go-applications-with-testcontainers/ 95 | func init() { 96 | startupDependenciesFns := []func() (testcontainers.Container, error){ 97 | startTalksStore, 98 | startRatingsStore, 99 | startStreamingQueue, 100 | } 101 | 102 | for _, fn := range startupDependenciesFns { 103 | _, err := fn() 104 | if err != nil { 105 | panic(err) 106 | } 107 | } 108 | } 109 | 110 | func startRatingsStore() (testcontainers.Container, error) { 111 | ctx := context.Background() 112 | 113 | c, err := redis.Run(ctx, "redis:6-alpine") 114 | if err != nil { 115 | return nil, err 116 | } 117 | 118 | ratingsConn, err := c.ConnectionString(ctx) 119 | if err != nil { 120 | return nil, err 121 | } 122 | 123 | Connections.Ratings = ratingsConn 124 | return c, nil 125 | } 126 | 127 | func startStreamingQueue() (testcontainers.Container, error) { 128 | ctx := context.Background() 129 | 130 | c, err := redpanda.Run( 131 | ctx, 132 | "docker.redpanda.com/redpandadata/redpanda:v24.3.7", 133 | redpanda.WithAutoCreateTopics(), 134 | ) 135 | 136 | seedBroker, err := c.KafkaSeedBroker(ctx) 137 | if err != nil { 138 | return nil, err 139 | } 140 | 141 | Connections.Streams = seedBroker 142 | return c, nil 143 | } 144 | 145 | func startTalksStore() (testcontainers.Container, error) { 146 | ctx := context.Background() 147 | c, err := postgres.Run(ctx, 148 | "postgres:15.3-alpine", 149 | postgres.WithInitScripts(filepath.Join(".", "testdata", "dev-db.sql")), 150 | postgres.WithDatabase("talks-db"), 151 | postgres.WithUsername("postgres"), 152 | postgres.WithPassword("postgres"), 153 | testcontainers.WithWaitStrategy( 154 | wait.ForLog("database system is ready to accept connections"). 155 | WithOccurrence(2).WithStartupTimeout(15*time.Second)), 156 | ) 157 | if err != nil { 158 | return nil, err 159 | } 160 | 161 | talksConn, err := c.ConnectionString(ctx) 162 | if err != nil { 163 | return nil, err 164 | } 165 | 166 | Connections.Talks = talksConn 167 | return c, nil 168 | } 169 | 170 | ``` 171 | 172 | Now run `go mod tidy` from the root of the project to download the Go dependencies, only the Testcontainers for Go's Redpanda module. 173 | 174 | Finally, stop the application with Ctrl+C and run the application again with `make dev`. This time, the application will start the Redis store and the application will be able to connect to it. 175 | 176 | ```text 177 | go run -tags dev -v ./... 178 | 179 | ┌───────────────────────────────────────────────────┐ 180 | │ Fiber v2.52.6 │ 181 | │ http://127.0.0.1:8080 │ 182 | │ (bound on host 0.0.0.0 and port 8080) │ 183 | │ │ 184 | │ Handlers ............. 6 Processes ........... 1 │ 185 | │ Prefork ....... Disabled PID ............. 24313 │ 186 | └───────────────────────────────────────────────────┘ 187 | 188 | ``` 189 | 190 | In the second terminal, check the containers, we will see the Redpanda streaming queue is running alongside the Postgres database and the Redis store: 191 | 192 | ```text 193 | $ docker ps 194 | CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 195 | 1811a3de1f8f docker.redpanda.com/redpandadata/redpanda:v24.3.7 "/entrypoint-tc.sh r…" 3 minutes ago Up 3 minutes 8082/tcp, 0.0.0.0:32781->8081/tcp, :::32781->8081/tcp, 0.0.0.0:32780->9092/tcp, :::32780->9092/tcp, 0.0.0.0:32779->9644/tcp, :::32779->9644/tcp elegant_goldberg 196 | 373f523c83ac redis:6-alpine "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 0.0.0.0:32778->6379/tcp, :::32778->6379/tcp stupefied_franklin 197 | 00bca83e66ca postgres:15.3-alpine "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 0.0.0.0:32777->5432/tcp, :::32777->5432/tcp wizardly_snyder 198 | ``` 199 | 200 | Now the application should be able to connect to the database, to Redis and to the Redpanda streaming queue. Let's try to send a POST request adding a rating for the talk. If we remember, the API accepted a JSON payload with the following format: 201 | 202 | ```json 203 | { 204 | "talkId": "testcontainers-integration-testing", 205 | "value": 5 206 | } 207 | ``` 208 | 209 | In a terminal, let's send a POST request with `curl`: 210 | 211 | ```shell 212 | curl -X POST -H "Content-Type: application/json" http://localhost:8080/ratings -d '{"talkId":"testcontainers-integration-testing", "value":5}' 213 | ``` 214 | 215 | The response should be a 200 OK: 216 | 217 | ```json 218 | {"rating":{"talk_uuid":"testcontainers-integration-testing","value":5}}% 219 | ``` 220 | 221 | The log entry for the POST request: 222 | 223 | ```text 224 | 13:13:10 | 200 | 2.578671833s | 127.0.0.1 | POST | /ratings | - 225 | ``` 226 | 227 | If we open now the ratings endpoint from the API (http://localhost:8080/ratings?talkId=testcontainers-integration-testing), then a 200 OK response code is returned, and the first ratings for the given talk is there. It was a five! ⭐️⭐️⭐️⭐️⭐️ 228 | 229 | ```text 230 | {"ratings":{"5":"1"}} 231 | ``` 232 | 233 | With `curl`: 234 | 235 | ```shell 236 | curl -X GET http://localhost:8080/ratings\?talkId\=testcontainers-integration-testing 237 | {"ratings":{"5":"1"}}% 238 | ``` 239 | 240 | Play around sending multiple POST requests for the two talks we created in the SQL script, and check the histogram that is created for the different rating values. 241 | 242 | In any GET request we'll still see the log entry for the AWS lambda failing to be called. 243 | 244 | ```text 245 | 2025/05/07 13:13:41 error calling lambda function: Post "": unsupported protocol scheme "" 246 | 13:13:39 | 200 | 1.298763167s | 127.0.0.1 | GET | /ratings | - 247 | ``` 248 | 249 | It's time now to fix it, adding a cloud emulator for the AWS Lambda function. 250 | 251 | ### 252 | [Next: Adding Localstack](step-7-adding-localstack.md) -------------------------------------------------------------------------------- /step-7-adding-localstack.md: -------------------------------------------------------------------------------- 1 | # Step 7: Adding LocalStack 2 | 3 | The application is using an AWS lambda function to calculate some statistics (average and total count) for the ratings of a talk. The lambda function is invoked by the application any time the ratings for a talk are requested, using HTTP calls to the function URL of the lambda. 4 | 5 | To enhance the developer experience of consuming this lambda function while developing the application, we will use LocalStack to emulate the AWS cloud environment locally. 6 | 7 | LocalStack is a cloud service emulator that runs in a single container on your laptop or in your CI environment. With LocalStack, we can run your AWS applications or Lambdas entirely on your local machine without connecting to a remote cloud provider! 8 | 9 | ## Creating the lambda function 10 | 11 | The lambda function is a simple Go function that calculates the average rating of a talk. The function is defined in the `main.go` file under a `lambda-go` directory: 12 | 13 | ```go 14 | package main 15 | 16 | import ( 17 | "encoding/json" 18 | "fmt" 19 | "strconv" 20 | 21 | "github.com/aws/aws-lambda-go/events" 22 | "github.com/aws/aws-lambda-go/lambda" 23 | ) 24 | 25 | type RatingsEvent struct { 26 | Ratings map[string]int `json:"ratings"` 27 | } 28 | 29 | type Response struct { 30 | Avg float64 `json:"avg"` 31 | TotalCount int `json:"totalCount"` 32 | } 33 | 34 | var emptyResponse = Response{ 35 | Avg: 0, 36 | TotalCount: 0, 37 | } 38 | 39 | // HandleStats returns the stats for the given talk, obtained from a call to the Lambda function. 40 | // The payload is a JSON object with the following structure: 41 | // 42 | // { 43 | // "ratings": { 44 | // "0": 10, 45 | // "1": 20, 46 | // "2": 30, 47 | // "3": 40, 48 | // "4": 50, 49 | // "5": 60 50 | // } 51 | // } 52 | // 53 | // The response from the Lambda function is a JSON object with the following structure: 54 | // 55 | // { 56 | // "avg": 3.5, 57 | // "totalCount": 210, 58 | // } 59 | func HandleStats(event events.APIGatewayProxyRequest) (Response, error) { 60 | ratingsEvent := RatingsEvent{} 61 | err := json.Unmarshal([]byte(event.Body), &ratingsEvent) 62 | if err != nil { 63 | return emptyResponse, fmt.Errorf("failed to unmarshal ratings event: %s", err) 64 | } 65 | 66 | var totalCount int 67 | var sum int 68 | for rating, count := range ratingsEvent.Ratings { 69 | totalCount += count 70 | 71 | r, err := strconv.Atoi(rating) 72 | if err != nil { 73 | return emptyResponse, fmt.Errorf("failed to convert rating %s to int: %s", rating, err) 74 | } 75 | 76 | sum += count * r 77 | } 78 | 79 | var avg float64 80 | if totalCount > 0 { 81 | avg = float64(sum) / float64(totalCount) 82 | } 83 | 84 | resp := Response{ 85 | Avg: avg, 86 | TotalCount: totalCount, 87 | } 88 | 89 | return resp, nil 90 | } 91 | 92 | func main() { 93 | lambda.Start(HandleStats) 94 | } 95 | 96 | ``` 97 | 98 | Now, in the `lambda-go` directory, create the `go.mod` file for the lambda function: 99 | 100 | ```go 101 | module github.com/testcontainers/workshop-go/lambda-go 102 | 103 | go 1.24 104 | 105 | require github.com/aws/aws-lambda-go v1.48.0 106 | 107 | ``` 108 | 109 | Now, create a Makefile in the `lambda-go` directory. It will simplify how the Go lambda is compiled and packaged as a ZIP file for being deployed to LocalStack. Please add the following content: 110 | 111 | ```Makefile 112 | mod-tidy: 113 | go mod tidy 114 | 115 | build-lambda: mod-tidy 116 | # If you are using Testcontainers Cloud, please add 'GOARCH=amd64' in order to get the localstack's lambdas using the right architecture 117 | GOOS=linux go build -tags lambda.norpc -o bootstrap main.go 118 | 119 | zip-lambda: build-lambda 120 | zip -j function.zip bootstrap 121 | 122 | ``` 123 | 124 | At this point of the workshop, we are treating the lambda as a dependency of our ratings application. In the following steps, we will see how to add integration tests for the lambda function. 125 | 126 | Finally, to integrate the package of the lambda into the local development mode of the application, please replace the contents of the Makefile in the root of the project with the following: 127 | 128 | ```Makefile 129 | build-lambda: 130 | $(MAKE) -C lambda-go zip-lambda 131 | 132 | dev: build-lambda 133 | go run -tags dev -v ./... 134 | 135 | ``` 136 | 137 | We are adding a `build-lambda` goal that will build the lambda function and package it as a ZIP file. The `dev` goal will build the lambda function and start the application in development mode. The rest of the goals are the same as before. 138 | 139 | ## Adding the LocalStack instance 140 | 141 | Let's add a LocalStack instance using Testcontainers for Go. 142 | 143 | 1. In the `internal/app/dev_dependencies.go` file, add the following imports: 144 | 145 | ```go 146 | import ( 147 | "bytes" 148 | "context" 149 | "encoding/json" 150 | "fmt" 151 | osexec "os/exec" 152 | "path/filepath" 153 | "runtime" 154 | "strings" 155 | "time" 156 | 157 | "github.com/testcontainers/testcontainers-go" 158 | "github.com/testcontainers/testcontainers-go/exec" 159 | "github.com/testcontainers/testcontainers-go/modules/localstack" 160 | "github.com/testcontainers/testcontainers-go/modules/postgres" 161 | "github.com/testcontainers/testcontainers-go/modules/redis" 162 | "github.com/testcontainers/testcontainers-go/modules/redpanda" 163 | "github.com/testcontainers/testcontainers-go/wait" 164 | ) 165 | ``` 166 | 167 | 2. Add these two functions to the file: 168 | 169 | ```go 170 | // buildLambda return the path to the ZIP file used to deploy the lambda function. 171 | func buildLambda() string { 172 | _, b, _, _ := runtime.Caller(0) 173 | basepath := filepath.Dir(b) 174 | 175 | lambdaPath := filepath.Join(basepath, "..", "..", "lambda-go") 176 | 177 | makeCmd := osexec.Command("make", "zip-lambda") 178 | makeCmd.Dir = lambdaPath 179 | 180 | err := makeCmd.Run() 181 | if err != nil { 182 | panic(fmt.Errorf("failed to zip lambda: %w", err)) 183 | } 184 | 185 | return filepath.Join(lambdaPath, "function.zip") 186 | } 187 | 188 | func startRatingsLambda() (testcontainers.Container, error) { 189 | ctx := context.Background() 190 | 191 | flagsFn := func() string { 192 | labels := testcontainers.GenericLabels() 193 | flags := "" 194 | for k, v := range labels { 195 | flags = fmt.Sprintf("%s -l %s=%s", flags, k, v) 196 | } 197 | return flags 198 | } 199 | 200 | var functionURL string 201 | 202 | c, err := localstack.Run(ctx, 203 | "localstack/localstack:latest", 204 | testcontainers.WithEnv(map[string]string{ 205 | "SERVICES": "lambda", 206 | "LAMBDA_DOCKER_FLAGS": flagsFn(), 207 | }), 208 | testcontainers.WithFiles(testcontainers.ContainerFile{ 209 | HostFilePath: buildLambda(), 210 | ContainerFilePath: "/tmp/function.zip", 211 | }), 212 | testcontainers.CustomizeRequest(testcontainers.GenericContainerRequest{ 213 | ContainerRequest: testcontainers.ContainerRequest{ 214 | LifecycleHooks: []testcontainers.ContainerLifecycleHooks{ 215 | { 216 | PostStarts: []testcontainers.ContainerHook{ 217 | func(ctx context.Context, c testcontainers.Container) error { 218 | lambdaName := "localstack-lambda-url-example" 219 | 220 | // the three commands below are doing the following: 221 | // 1. create a lambda function 222 | // 2. create the URL function configuration for the lambda function 223 | // 3. wait for the lambda function to be active 224 | lambdaCommands := [][]string{ 225 | { 226 | "awslocal", "lambda", 227 | "create-function", "--function-name", lambdaName, 228 | "--runtime", "provided.al2", 229 | "--handler", "bootstrap", 230 | "--role", "arn:aws:iam::111122223333:role/lambda-ex", 231 | "--zip-file", "fileb:///tmp/function.zip", 232 | }, 233 | {"awslocal", "lambda", "create-function-url-config", "--function-name", lambdaName, "--auth-type", "NONE"}, 234 | {"awslocal", "lambda", "wait", "function-active-v2", "--function-name", lambdaName}, 235 | } 236 | for _, cmd := range lambdaCommands { 237 | _, _, err := c.Exec(ctx, cmd) 238 | if err != nil { 239 | return err 240 | } 241 | } 242 | 243 | // 4. get the URL for the lambda function 244 | cmd := []string{ 245 | "awslocal", "lambda", "list-function-url-configs", "--function-name", lambdaName, 246 | } 247 | _, reader, err := c.Exec(ctx, cmd, exec.Multiplexed()) 248 | if err != nil { 249 | return err 250 | } 251 | 252 | buf := new(bytes.Buffer) 253 | _, err = buf.ReadFrom(reader) 254 | if err != nil { 255 | return err 256 | } 257 | 258 | content := buf.Bytes() 259 | 260 | type FunctionURLConfig struct { 261 | FunctionURLConfigs []struct { 262 | FunctionURL string `json:"FunctionUrl"` 263 | FunctionArn string `json:"FunctionArn"` 264 | CreationTime string `json:"CreationTime"` 265 | LastModifiedTime string `json:"LastModifiedTime"` 266 | AuthType string `json:"AuthType"` 267 | } `json:"FunctionUrlConfigs"` 268 | } 269 | 270 | v := &FunctionURLConfig{} 271 | err = json.Unmarshal(content, v) 272 | if err != nil { 273 | return err 274 | } 275 | 276 | // 5. finally, set the function URL from the response 277 | functionURL = v.FunctionURLConfigs[0].FunctionURL 278 | 279 | return nil 280 | }, 281 | }, 282 | }, 283 | }, 284 | }, 285 | }), 286 | ) 287 | if err != nil { 288 | return nil, err 289 | } 290 | 291 | // replace the port with the one exposed by the container 292 | mappedPort, err := c.MappedPort(ctx, "4566/tcp") 293 | if err != nil { 294 | return nil, err 295 | } 296 | 297 | functionURL = strings.ReplaceAll(functionURL, "4566", mappedPort.Port()) 298 | 299 | // The latest version of localstack does not add ".localstack.cloud" by default, 300 | // that's why need to add it to the URL. 301 | functionURL = strings.ReplaceAll(functionURL, ".localhost", ".localhost.localstack.cloud") 302 | 303 | Connections.Lambda = functionURL 304 | 305 | return c, nil 306 | } 307 | ``` 308 | 309 | The first function will perform a `make zip-lambda` to build the lambda function and package it as a ZIP file, which is convenient to integrate the `Make` build system into the local development experience. 310 | 311 | The second function will: 312 | - start a LocalStack instance, copying the zip file into the container before it starts. See the `Files` attribute in the container request. 313 | - leverate the container lifecycle hooks to execute commands in the container right after it has started. We are going to execute `awslocal lambda` commands inside the LocalStack container to: 314 | - create the lambda from the zip file 315 | - create the URL function configuration for the lambda function 316 | - wait for the lambda function to be active 317 | - read the response of executing an `awslocal lambda` command to get the URL of the lambda function, parsing the JSON response to get the URL of the lambda function. 318 | - finally store the URL of the lambda function in a variable 319 | - update the `Connections` struct with the lambda function URL. 320 | 321 | 3. Update the comments for the init function `startupDependenciesFn` slice to include the LocalStack store: 322 | 323 | ```go 324 | // init will be used to start up the containers for development mode. It will use 325 | // testcontainers-go to start up the following containers: 326 | // - Postgres: store for talks 327 | // - Redis: store for ratings 328 | // - Redpanda: message queue for the ratings 329 | // - LocalStack: cloud emulator for AWS Lambdas 330 | // All the containers will contribute their connection strings to the Connections struct. 331 | // Please read this blog post for more information: https://www.atomicjar.com/2023/08/local-development-of-go-applications-with-testcontainers/ 332 | func init() { 333 | ``` 334 | 335 | 4. Update the `startupDependenciesFn` slice to include the function that starts the ratings store: 336 | 337 | ```go 338 | startupDependenciesFns := []func() (testcontainers.Container, error){ 339 | startTalksStore, 340 | startRatingsStore, 341 | startStreamingQueue, 342 | startRatingsLambda, 343 | } 344 | ``` 345 | 346 | The complete file should look like this: 347 | 348 | ```go 349 | //go:build dev 350 | // +build dev 351 | 352 | package app 353 | 354 | import ( 355 | "bytes" 356 | "context" 357 | "encoding/json" 358 | "fmt" 359 | osexec "os/exec" 360 | "path/filepath" 361 | "runtime" 362 | "strings" 363 | "time" 364 | 365 | "github.com/testcontainers/testcontainers-go" 366 | "github.com/testcontainers/testcontainers-go/exec" 367 | "github.com/testcontainers/testcontainers-go/modules/localstack" 368 | "github.com/testcontainers/testcontainers-go/modules/postgres" 369 | "github.com/testcontainers/testcontainers-go/modules/redis" 370 | "github.com/testcontainers/testcontainers-go/modules/redpanda" 371 | "github.com/testcontainers/testcontainers-go/wait" 372 | ) 373 | 374 | // init will be used to start up the containers for development mode. It will use 375 | // testcontainers-go to start up the following containers: 376 | // - Postgres: store for talks 377 | // - Redis: store for ratings 378 | // - Redpanda: message queue for the ratings 379 | // - LocalStack: cloud emulator for AWS Lambdas 380 | // All the containers will contribute their connection strings to the Connections struct. 381 | // Please read this blog post for more information: https://www.atomicjar.com/2023/08/local-development-of-go-applications-with-testcontainers/ 382 | func init() { 383 | startupDependenciesFns := []func() (testcontainers.Container, error){ 384 | startTalksStore, 385 | startRatingsStore, 386 | startStreamingQueue, 387 | startRatingsLambda, 388 | } 389 | 390 | for _, fn := range startupDependenciesFns { 391 | _, err := fn() 392 | if err != nil { 393 | panic(err) 394 | } 395 | } 396 | } 397 | 398 | // buildLambda return the path to the ZIP file used to deploy the lambda function. 399 | func buildLambda() string { 400 | _, b, _, _ := runtime.Caller(0) 401 | basepath := filepath.Dir(b) 402 | 403 | lambdaPath := filepath.Join(basepath, "..", "..", "lambda-go") 404 | 405 | makeCmd := osexec.Command("make", "zip-lambda") 406 | makeCmd.Dir = lambdaPath 407 | 408 | err := makeCmd.Run() 409 | if err != nil { 410 | panic(fmt.Errorf("failed to zip lambda: %w", err)) 411 | } 412 | 413 | return filepath.Join(lambdaPath, "function.zip") 414 | } 415 | 416 | func startRatingsLambda() (testcontainers.Container, error) { 417 | ctx := context.Background() 418 | 419 | flagsFn := func() string { 420 | labels := testcontainers.GenericLabels() 421 | flags := "" 422 | for k, v := range labels { 423 | flags = fmt.Sprintf("%s -l %s=%s", flags, k, v) 424 | } 425 | return flags 426 | } 427 | 428 | var functionURL string 429 | 430 | c, err := localstack.Run(ctx, 431 | "localstack/localstack:latest", 432 | testcontainers.WithEnv(map[string]string{ 433 | "SERVICES": "lambda", 434 | "LAMBDA_DOCKER_FLAGS": flagsFn(), 435 | }), 436 | testcontainers.WithFiles(testcontainers.ContainerFile{ 437 | HostFilePath: buildLambda(), 438 | ContainerFilePath: "/tmp/function.zip", 439 | }), 440 | testcontainers.CustomizeRequest(testcontainers.GenericContainerRequest{ 441 | ContainerRequest: testcontainers.ContainerRequest{ 442 | LifecycleHooks: []testcontainers.ContainerLifecycleHooks{ 443 | { 444 | PostStarts: []testcontainers.ContainerHook{ 445 | func(ctx context.Context, c testcontainers.Container) error { 446 | lambdaName := "localstack-lambda-url-example" 447 | 448 | // the three commands below are doing the following: 449 | // 1. create a lambda function 450 | // 2. create the URL function configuration for the lambda function 451 | // 3. wait for the lambda function to be active 452 | lambdaCommands := [][]string{ 453 | { 454 | "awslocal", "lambda", 455 | "create-function", "--function-name", lambdaName, 456 | "--runtime", "provided.al2", 457 | "--handler", "bootstrap", 458 | "--role", "arn:aws:iam::111122223333:role/lambda-ex", 459 | "--zip-file", "fileb:///tmp/function.zip", 460 | }, 461 | {"awslocal", "lambda", "create-function-url-config", "--function-name", lambdaName, "--auth-type", "NONE"}, 462 | {"awslocal", "lambda", "wait", "function-active-v2", "--function-name", lambdaName}, 463 | } 464 | for _, cmd := range lambdaCommands { 465 | _, _, err := c.Exec(ctx, cmd) 466 | if err != nil { 467 | return err 468 | } 469 | } 470 | 471 | // 4. get the URL for the lambda function 472 | cmd := []string{ 473 | "awslocal", "lambda", "list-function-url-configs", "--function-name", lambdaName, 474 | } 475 | _, reader, err := c.Exec(ctx, cmd, exec.Multiplexed()) 476 | if err != nil { 477 | return err 478 | } 479 | 480 | buf := new(bytes.Buffer) 481 | _, err = buf.ReadFrom(reader) 482 | if err != nil { 483 | return err 484 | } 485 | 486 | content := buf.Bytes() 487 | 488 | type FunctionURLConfig struct { 489 | FunctionURLConfigs []struct { 490 | FunctionURL string `json:"FunctionUrl"` 491 | FunctionArn string `json:"FunctionArn"` 492 | CreationTime string `json:"CreationTime"` 493 | LastModifiedTime string `json:"LastModifiedTime"` 494 | AuthType string `json:"AuthType"` 495 | } `json:"FunctionUrlConfigs"` 496 | } 497 | 498 | v := &FunctionURLConfig{} 499 | err = json.Unmarshal(content, v) 500 | if err != nil { 501 | return err 502 | } 503 | 504 | // 5. finally, set the function URL from the response 505 | functionURL = v.FunctionURLConfigs[0].FunctionURL 506 | 507 | return nil 508 | }, 509 | }, 510 | }, 511 | }, 512 | }, 513 | }), 514 | ) 515 | if err != nil { 516 | return nil, err 517 | } 518 | 519 | // replace the port with the one exposed by the container 520 | mappedPort, err := c.MappedPort(ctx, "4566/tcp") 521 | if err != nil { 522 | return nil, err 523 | } 524 | 525 | functionURL = strings.ReplaceAll(functionURL, "4566", mappedPort.Port()) 526 | 527 | // The latest version of localstack does not add ".localstack.cloud" by default, 528 | // that's why we need to add it to the URL. 529 | functionURL = strings.ReplaceAll(functionURL, ".localhost", ".localhost.localstack.cloud") 530 | 531 | Connections.Lambda = functionURL 532 | 533 | return c, nil 534 | } 535 | 536 | func startRatingsStore() (testcontainers.Container, error) { 537 | ctx := context.Background() 538 | 539 | c, err := redis.Run(ctx, "redis:6-alpine") 540 | if err != nil { 541 | return nil, err 542 | } 543 | 544 | ratingsConn, err := c.ConnectionString(ctx) 545 | if err != nil { 546 | return nil, err 547 | } 548 | 549 | Connections.Ratings = ratingsConn 550 | return c, nil 551 | } 552 | 553 | func startStreamingQueue() (testcontainers.Container, error) { 554 | ctx := context.Background() 555 | 556 | c, err := redpanda.Run( 557 | ctx, 558 | "docker.redpanda.com/redpandadata/redpanda:v24.3.7", 559 | redpanda.WithAutoCreateTopics(), 560 | ) 561 | 562 | seedBroker, err := c.KafkaSeedBroker(ctx) 563 | if err != nil { 564 | return nil, err 565 | } 566 | 567 | Connections.Streams = seedBroker 568 | return c, nil 569 | } 570 | 571 | func startTalksStore() (testcontainers.Container, error) { 572 | ctx := context.Background() 573 | c, err := postgres.Run(ctx, 574 | "postgres:15.3-alpine", 575 | postgres.WithInitScripts(filepath.Join(".", "testdata", "dev-db.sql")), 576 | postgres.WithDatabase("talks-db"), 577 | postgres.WithUsername("postgres"), 578 | postgres.WithPassword("postgres"), 579 | testcontainers.WithWaitStrategy( 580 | wait.ForLog("database system is ready to accept connections"). 581 | WithOccurrence(2).WithStartupTimeout(15*time.Second)), 582 | ) 583 | if err != nil { 584 | return nil, err 585 | } 586 | 587 | talksConn, err := c.ConnectionString(ctx) 588 | if err != nil { 589 | return nil, err 590 | } 591 | 592 | Connections.Talks = talksConn 593 | return c, nil 594 | } 595 | 596 | ``` 597 | 598 | Now run `go mod tidy` from the root of the project to download the Go dependencies, this time only the Testcontainers for Go's LocalStack module. 599 | 600 | Also run `go mod tidy` from the `lambda-go` directory to download the Go dependencies for the lambda function. 601 | 602 | Finally, stop the application with Ctrl+C and run the application again with `make dev`. This time, the application will build the lambda, will start all the services, and the application will be able to connect to it. 603 | 604 | ```text 605 | go mod tidy 606 | GOOS=linux go build -tags lambda.norpc -o bootstrap main.go 607 | zip -j function.zip bootstrap 608 | adding: bootstrap (deflated 45%) 609 | go run -tags dev -v ./... 610 | 611 | ┌───────────────────────────────────────────────────┐ 612 | │ Fiber v2.52.6 │ 613 | │ http://127.0.0.1:8080 │ 614 | │ (bound on host 0.0.0.0 and port 8080) │ 615 | │ │ 616 | │ Handlers ............. 6 Processes ........... 1 │ 617 | │ Prefork ....... Disabled PID ............. 26322 │ 618 | └───────────────────────────────────────────────────┘ 619 | 620 | ``` 621 | 622 | In the second terminal, check the containers, we will see the LocalStack instance is running alongside the Postgres database, the Redis store and the Redpanda streaming queue: 623 | 624 | ```text 625 | $ docker ps 626 | CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 627 | c514896580c1 localstack/localstack:latest "docker-entrypoint.sh" 2 minutes ago Up 2 minutes (healthy) 4510-4559/tcp, 5678/tcp, 0.0.0.0:32792->4566/tcp, :::32792->4566/tcp priceless_antonelli 628 | 07fb1e908b1e docker.redpanda.com/redpandadata/redpanda:v24.3.7 "/entrypoint-tc.sh r…" 3 minutes ago Up 3 minutes 8082/tcp, 0.0.0.0:32791->8081/tcp, :::32791->8081/tcp, 0.0.0.0:32790->9092/tcp, :::32790->9092/tcp, 0.0.0.0:32789->9644/tcp, :::32789->9644/tcp loving_murdock 629 | bf4fcb4cd74c redis:6-alpine "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 0.0.0.0:32788->6379/tcp, :::32788->6379/tcp angry_shirley 630 | d5ec7cecb562 postgres:15.3-alpine "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 0.0.0.0:32787->5432/tcp, :::32787->5432/tcp laughing_kare 631 | ``` 632 | 633 | The LocalStack instance is now running, and a lambda function is deployed in it. We can verify the lambda function is running by sending a request to the function URL. But we first need to obtain the URL of the lambda. Please do a GET request to the `/` endpoint of the API, where we'll get the metadata of the application. Something similar to this: 634 | 635 | ```bash 636 | $ curl -X GET http://localhost:8080/ 637 | ``` 638 | 639 | The JSON response: 640 | 641 | ```json 642 | {"metadata":{"ratings_lambda":"http://5xnih5q8vrjmzsmis1ic4740eszvzbao.lambda-url.us-east-1.localhost.localstack.cloud:32829/","ratings":"redis://localhost:32825","streams":"localhost:32827","talks":"postgres://postgres:postgres@localhost:32824/talks-db?"}} 643 | ``` 644 | 645 | In your terminal, copy the `ratings_lambda` URL from the response and send a POST request to it with `curl` (please remember to replace the URL with the one we got from the response): 646 | 647 | ```bash 648 | curl -X POST http://5xnih5q8vrjmzsmis1ic4740eszvzbao.lambda-url.us-east-1.localhost.localstack.cloud:32829/ -d '{"ratings":{"2":1,"4":3,"5":1}}' -H "Content-Type: application/json" 649 | ``` 650 | 651 | The JSON response: 652 | 653 | ```json 654 | {"avg": 3.8, "totalCount": 5}% 655 | ``` 656 | 657 | Great! the response contains the average rating of the talk, and the total number of ratings, calculated in the lambda function. 658 | 659 | ### 660 | [Next: Adding integration tests](step-8-adding-integration-tests.md) -------------------------------------------------------------------------------- /step-8-adding-integration-tests.md: -------------------------------------------------------------------------------- 1 | # Step 8: Adding Integration Tests 2 | 3 | Ok, we have a working application, but we don't have any tests. Let's add some integration tests to verify that the application works as expected. 4 | 5 | ## Integration tests for the Ratings store 6 | 7 | Let's add a new file `internal/ratings/repo_test.go` with the following content: 8 | 9 | ```go 10 | package ratings_test 11 | 12 | import ( 13 | "context" 14 | "fmt" 15 | "testing" 16 | 17 | "github.com/stretchr/testify/assert" 18 | "github.com/stretchr/testify/require" 19 | "github.com/testcontainers/testcontainers-go" 20 | tcRedis "github.com/testcontainers/testcontainers-go/modules/redis" 21 | "github.com/testcontainers/workshop-go/internal/ratings" 22 | ) 23 | 24 | func TestNewRepository(t *testing.T) { 25 | ctx := context.Background() 26 | 27 | redisContainer, err := tcRedis.Run(ctx, "redis:6-alpine") 28 | testcontainers.CleanupContainer(t, redisContainer) 29 | require.NoError(t, err) 30 | 31 | connStr, err := redisContainer.ConnectionString(ctx) 32 | require.NoError(t, err) 33 | 34 | repo, err := ratings.NewRepository(ctx, connStr) 35 | require.NoError(t, err) 36 | assert.NotNil(t, repo) 37 | 38 | t.Run("Add rating", func(t *testing.T) { 39 | rating := ratings.Rating{ 40 | TalkUuid: "uuid12345", 41 | Value: 5, 42 | } 43 | 44 | result, err := repo.Add(ctx, rating) 45 | assert.NoError(t, err) 46 | assert.Equal(t, int64(1), result) 47 | }) 48 | 49 | t.Run("Add multiple ratings", func(t *testing.T) { 50 | takUUID := "uuid67890" 51 | max := 100 52 | distribution := 5 53 | 54 | for i := 0; i < max; i++ { 55 | rating := ratings.Rating{ 56 | TalkUuid: takUUID, 57 | Value: int64(i % distribution), // creates a distribution of ratings, 20 of each 58 | } 59 | 60 | // don't care about the result 61 | _, _ = repo.Add(ctx, rating) 62 | } 63 | 64 | values := repo.FindAllByUUID(ctx, takUUID) 65 | assert.Len(t, values, distribution) 66 | 67 | for i := 0; i < distribution; i++ { 68 | assert.Equal(t, fmt.Sprintf("%d", (max/distribution)), values[fmt.Sprintf("%d", i)]) 69 | } 70 | }) 71 | } 72 | 73 | ``` 74 | 75 | This test will start a Redis container, and it will define two tests: 76 | 77 | * `Add rating`: it will add a rating to the store and verify that the result is the same as the one provided 78 | * `Add multiple ratings`: it will add 100 ratings to the store and verify that the distribution of ratings is correct 79 | 80 | The package has been named with the `_test` suffix to indicate that it contains tests. This is a convention in Go and forces us to consume your code as a package, which is a good practice. 81 | 82 | Now run `go mod tidy` from the root of the project to download the Go dependencies, as the workshop is using [testify](https://github.com/stretchr/testify) as the assertions library. 83 | 84 | Finally, run your tests with `go test -v -count=1 ./internal/ratings -run TestNewRepository` from the root of the project. We should see the following output: 85 | 86 | ```text 87 | === RUN TestNewRepository 88 | 2025/03/25 13:28:43 github.com/testcontainers/testcontainers-go - Connected to docker: 89 | Server Version: 27.5.0 (via Testcontainers Desktop 1.19.0) 90 | API Version: 1.46 91 | Operating System: Ubuntu 22.04.5 LTS 92 | Total Memory: 15368 MB 93 | Labels: 94 | cloud.docker.run.version=259.c712f5fd 95 | Testcontainers for Go Version: v0.35.0 96 | Resolved Docker Host: tcp://127.0.0.1:49982 97 | Resolved Docker Socket Path: /var/run/docker.sock 98 | Test SessionID: 108e56b58c673b34136ef7aff4cc8629b6101a9737009f275fed7592aa75d3af 99 | Test ProcessID: f3796459-23e9-4320-a1db-328094645da2 100 | 2025/03/25 13:28:43 🐳 Creating container for image redis:6-alpine 101 | 2025/03/25 13:28:44 🐳 Creating container for image testcontainers/ryuk:0.11.0 102 | 2025/03/25 13:28:44 ✅ Container created: 609a132f79e0 103 | 2025/03/25 13:28:44 🐳 Starting container: 609a132f79e0 104 | 2025/03/25 13:28:44 ✅ Container started: 609a132f79e0 105 | 2025/03/25 13:28:44 ⏳ Waiting for container id 609a132f79e0 image: testcontainers/ryuk:0.11.0. Waiting for: &{Port:8080/tcp timeout: PollInterval:100ms skipInternalCheck:false} 106 | 2025/03/25 13:28:44 🔔 Container is ready: 609a132f79e0 107 | 2025/03/25 13:28:44 ✅ Container created: dac0babc7b42 108 | 2025/03/25 13:28:44 🐳 Starting container: dac0babc7b42 109 | 2025/03/25 13:28:45 ✅ Container started: dac0babc7b42 110 | 2025/03/25 13:28:45 ⏳ Waiting for container id dac0babc7b42 image: redis:6-alpine. Waiting for: &{timeout: Log:* Ready to accept connections IsRegexp:false Occurrence:1 PollInterval:100ms check: submatchCallback: re: log:[]} 111 | 2025/03/25 13:28:45 🔔 Container is ready: dac0babc7b42 112 | === RUN TestNewRepository/Add_rating 113 | === RUN TestNewRepository/Add_multiple_ratings 114 | 2025/03/25 13:28:48 🐳 Stopping container: dac0babc7b42 115 | 2025/03/25 13:28:48 ✅ Container stopped: dac0babc7b42 116 | 2025/03/25 13:28:48 🐳 Terminating container: dac0babc7b42 117 | 2025/03/25 13:28:48 🚫 Container terminated: dac0babc7b42 118 | --- PASS: TestNewRepository (5.18s) 119 | --- PASS: TestNewRepository/Add_rating (0.03s) 120 | --- PASS: TestNewRepository/Add_multiple_ratings (3.35s) 121 | PASS 122 | ok github.com/testcontainers/workshop-go/internal/ratings 5.492s 123 | ``` 124 | 125 | _NOTE: if we experiment longer test execution times it could be caused by the need of pulling the images from the registry._ 126 | 127 | ## Integration tests for the Streaming queue 128 | 129 | Let's add a new file `internal/streams/broker_test.go` with the following content: 130 | 131 | ```go 132 | package streams_test 133 | 134 | import ( 135 | "context" 136 | "errors" 137 | "testing" 138 | 139 | "github.com/stretchr/testify/require" 140 | "github.com/testcontainers/testcontainers-go" 141 | "github.com/testcontainers/testcontainers-go/modules/redpanda" 142 | "github.com/testcontainers/workshop-go/internal/ratings" 143 | "github.com/testcontainers/workshop-go/internal/streams" 144 | ) 145 | 146 | func TestBroker(t *testing.T) { 147 | ctx := context.Background() 148 | 149 | redpandaC, err := redpanda.Run( 150 | ctx, 151 | "docker.redpanda.com/redpandadata/redpanda:v24.3.7", 152 | redpanda.WithAutoCreateTopics(), 153 | ) 154 | testcontainers.CleanupContainer(t, redpandaC) 155 | require.NoError(t, err) 156 | 157 | seedBroker, err := redpandaC.KafkaSeedBroker(ctx) 158 | require.NoError(t, err) 159 | 160 | repo, err := streams.NewStream(ctx, seedBroker) 161 | require.NoError(t, err) 162 | 163 | t.Run("Send Rating without callback", func(t *testing.T) { 164 | noopFn := func() error { return nil } 165 | 166 | err = repo.SendRating(ctx, ratings.Rating{ 167 | TalkUuid: "uuid12345", 168 | Value: 5, 169 | }, noopFn) 170 | require.NoError(t, err) 171 | }) 172 | 173 | t.Run("Send Rating with error in callback", func(t *testing.T) { 174 | var ErrInCallback error = errors.New("error in callback") 175 | 176 | errorFn := func() error { return ErrInCallback } 177 | 178 | err = repo.SendRating(ctx, ratings.Rating{ 179 | TalkUuid: "uuid12345", 180 | Value: 5, 181 | }, errorFn) 182 | require.ErrorIs(t, ErrInCallback, err) 183 | }) 184 | } 185 | 186 | ``` 187 | 188 | This test will start a Redpanda container, and it will define two tests: 189 | 190 | * `Send Rating without callback`: it will send a rating to the broker and verify that the result does not return an error after the callback is executed. 191 | * `Send Rating with error in callback`: it will send a rating to the broker and verify that the result returns an error after the callback is executed. 192 | 193 | Please notice that the package has been named with the `_test` suffix for the same reasons describe above. 194 | 195 | There is no need to run `go mod tidy` again, as we have already downloaded the Go dependencies. 196 | 197 | Finally, run your tests with `go test -v -count=1 ./internal/streams -run TestBroker` from the root of the project. We should see the following output: 198 | 199 | ```text 200 | === RUN TestBroker 201 | 2025/03/25 13:27:43 github.com/testcontainers/testcontainers-go - Connected to docker: 202 | Server Version: 27.5.0 (via Testcontainers Desktop 1.19.0) 203 | API Version: 1.46 204 | Operating System: Ubuntu 22.04.5 LTS 205 | Total Memory: 15368 MB 206 | Labels: 207 | cloud.docker.run.version=259.c712f5fd 208 | Testcontainers for Go Version: v0.35.0 209 | Resolved Docker Host: tcp://127.0.0.1:49982 210 | Resolved Docker Socket Path: /var/run/docker.sock 211 | Test SessionID: f3efe8aa74049e456c1d8711ec74a7ac666105ea4996c5f5166099592f93160c 212 | Test ProcessID: 803f586a-f2aa-41aa-adb5-e5bb2a8ce85e 213 | 2025/03/25 13:27:43 🐳 Creating container for image docker.redpanda.com/redpandadata/redpanda:v24.3.7 214 | 2025/03/25 13:27:43 🐳 Creating container for image testcontainers/ryuk:0.11.0 215 | 2025/03/25 13:27:43 ✅ Container created: 220b54e84226 216 | 2025/03/25 13:27:43 🐳 Starting container: 220b54e84226 217 | 2025/03/25 13:27:43 ✅ Container started: 220b54e84226 218 | 2025/03/25 13:27:43 ⏳ Waiting for container id 220b54e84226 image: testcontainers/ryuk:0.11.0. Waiting for: &{Port:8080/tcp timeout: PollInterval:100ms skipInternalCheck:false} 219 | 2025/03/25 13:27:44 🔔 Container is ready: 220b54e84226 220 | 2025/03/25 13:27:44 ✅ Container created: 801391cb30bf 221 | 2025/03/25 13:27:44 🐳 Starting container: 801391cb30bf 222 | 2025/03/25 13:27:44 ✅ Container started: 801391cb30bf 223 | 2025/03/25 13:27:44 ⏳ Waiting for container id 801391cb30bf image: docker.redpanda.com/redpandadata/redpanda:v24.3.7. Waiting for: &{timeout: deadline: Strategies:[0x1400041cae0 0x1400041cb10 0x1400041cb40]} 224 | 2025/03/25 13:27:44 🔔 Container is ready: 801391cb30bf 225 | === RUN TestBroker/Send_Rating_without_callback 226 | === RUN TestBroker/Send_Rating_with_error_in_callback 227 | 2025/03/25 13:27:47 🐳 Stopping container: 801391cb30bf 228 | 2025/03/25 13:27:47 ✅ Container stopped: 801391cb30bf 229 | 2025/03/25 13:27:47 🐳 Terminating container: 801391cb30bf 230 | 2025/03/25 13:27:47 🚫 Container terminated: 801391cb30bf 231 | --- PASS: TestBroker (4.59s) 232 | --- PASS: TestBroker/Send_Rating_without_callback (0.72s) 233 | --- PASS: TestBroker/Send_Rating_with_error_in_callback (0.03s) 234 | PASS 235 | ok github.com/testcontainers/workshop-go/internal/streams 4.938s 236 | ``` 237 | 238 | _NOTE: if we experiment longer test execution times it could be caused by the need of pulling the images from the registry._ 239 | 240 | ## Integration tests for the Talks store 241 | 242 | Let's add a new file `internal/talks/repo_test.go` with the following content: 243 | 244 | ```go 245 | package talks_test 246 | 247 | import ( 248 | "context" 249 | "path/filepath" 250 | "testing" 251 | "time" 252 | 253 | "github.com/google/uuid" 254 | "github.com/stretchr/testify/assert" 255 | "github.com/testcontainers/testcontainers-go" 256 | "github.com/testcontainers/testcontainers-go/modules/postgres" 257 | "github.com/testcontainers/testcontainers-go/wait" 258 | "github.com/testcontainers/workshop-go/internal/talks" 259 | ) 260 | 261 | func TestNewRepository(t *testing.T) { 262 | ctx := context.Background() 263 | 264 | pgContainer, err := postgres.Run(ctx, 265 | "postgres:15.3-alpine", 266 | postgres.WithInitScripts(filepath.Join("..", "..", "testdata", "dev-db.sql")), // path to the root of the project 267 | postgres.WithDatabase("talks-db"), 268 | postgres.WithUsername("postgres"), 269 | postgres.WithPassword("postgres"), 270 | testcontainers.WithWaitStrategy( 271 | wait.ForLog("database system is ready to accept connections"). 272 | WithOccurrence(2).WithStartupTimeout(5*time.Second)), 273 | ) 274 | testcontainers.CleanupContainer(t, pgContainer) 275 | assert.NoError(t, err) 276 | 277 | connStr, err := pgContainer.ConnectionString(ctx, "sslmode=disable") 278 | assert.NoError(t, err) 279 | 280 | talksRepo, err := talks.NewRepository(ctx, connStr) 281 | assert.NoError(t, err) 282 | 283 | t.Run("Create a talk and retrieve it by UUID", func(t *testing.T) { 284 | uid := uuid.NewString() 285 | title := "Delightful Integration Tests with Testcontainers for Go" 286 | 287 | talk := talks.Talk{ 288 | UUID: uid, 289 | Title: title, 290 | } 291 | 292 | err = talksRepo.Create(ctx, &talk) 293 | assert.NoError(t, err) 294 | assert.Equal(t, talk.ID, 3) // the third, as there are two talks in the testdata/dev-db.sql file 295 | 296 | dbTalk, err := talksRepo.GetByUUID(ctx, uid) 297 | assert.NoError(t, err) 298 | assert.NotNil(t, dbTalk) 299 | assert.Equal(t, 3, talk.ID) 300 | assert.Equal(t, uid, talk.UUID) 301 | assert.Equal(t, title, talk.Title) 302 | }) 303 | 304 | t.Run("Exists by UUID", func(t *testing.T) { 305 | uid := uuid.NewString() 306 | title := "Delightful Integration Tests with Testcontainers for Go" 307 | 308 | talk := talks.Talk{ 309 | UUID: uid, 310 | Title: title, 311 | } 312 | 313 | err = talksRepo.Create(ctx, &talk) 314 | assert.NoError(t, err) 315 | 316 | found := talksRepo.Exists(ctx, uid) 317 | assert.True(t, found) 318 | }) 319 | 320 | t.Run("Does not exist by UUID", func(t *testing.T) { 321 | uid := uuid.NewString() 322 | 323 | found := talksRepo.Exists(ctx, uid) 324 | assert.False(t, found) 325 | }) 326 | } 327 | 328 | ``` 329 | 330 | This test will start a Postgres container, and it will define three tests: 331 | 332 | * `Create a talk and retrieve it by UUID`: it will create a talk in the store and verify that the result is the same as the one provided. 333 | * `Exists by UUID`: it will create a talk in the store and verify that the talk exists. 334 | * `Does not exist by UUID`: it will verify that a talk does not exist in the store. 335 | 336 | Please notice that the package has been named with the `_test` suffix for the same reasons describe above. 337 | 338 | There is no need to run `go mod tidy` again, as we have already downloaded the Go dependencies. 339 | 340 | Finally, run your tests with `go test -v -count=1 ./internal/talks -run TestNewRepository` from the root of the project. We should see the following output: 341 | 342 | ```text 343 | === RUN TestNewRepository 344 | 2025/03/25 13:31:18 github.com/testcontainers/testcontainers-go - Connected to docker: 345 | Server Version: 27.5.0 (via Testcontainers Desktop 1.19.0) 346 | API Version: 1.46 347 | Operating System: Ubuntu 22.04.5 LTS 348 | Total Memory: 15368 MB 349 | Labels: 350 | cloud.docker.run.version=259.c712f5fd 351 | Testcontainers for Go Version: v0.35.0 352 | Resolved Docker Host: tcp://127.0.0.1:49982 353 | Resolved Docker Socket Path: /var/run/docker.sock 354 | Test SessionID: f2ef0f015b36b470b519d04f7a37ceed9394461a3e34adc604278fcbb1a4d0b3 355 | Test ProcessID: b107a0b2-5185-46ca-b78a-1527cc6c54ce 356 | 2025/03/25 13:31:18 🐳 Creating container for image postgres:15.3-alpine 357 | 2025/03/25 13:31:18 🐳 Creating container for image testcontainers/ryuk:0.11.0 358 | 2025/03/25 13:31:18 ✅ Container created: 6dd266218b3f 359 | 2025/03/25 13:31:18 🐳 Starting container: 6dd266218b3f 360 | 2025/03/25 13:31:18 ✅ Container started: 6dd266218b3f 361 | 2025/03/25 13:31:18 ⏳ Waiting for container id 6dd266218b3f image: testcontainers/ryuk:0.11.0. Waiting for: &{Port:8080/tcp timeout: PollInterval:100ms skipInternalCheck:false} 362 | 2025/03/25 13:31:19 🔔 Container is ready: 6dd266218b3f 363 | 2025/03/25 13:31:19 ✅ Container created: 73c4474e064e 364 | 2025/03/25 13:31:19 🐳 Starting container: 73c4474e064e 365 | 2025/03/25 13:31:19 ✅ Container started: 73c4474e064e 366 | 2025/03/25 13:31:19 ⏳ Waiting for container id 73c4474e064e image: postgres:15.3-alpine. Waiting for: &{timeout: deadline:0x140003d5f88 Strategies:[0x14000116840]} 367 | 2025/03/25 13:31:20 🔔 Container is ready: 73c4474e064e 368 | === RUN TestNewRepository/Create_a_talk_and_retrieve_it_by_UUID 369 | === RUN TestNewRepository/Exists_by_UUID 370 | === RUN TestNewRepository/Does_not_exist_by_UUID 371 | 2025/03/25 13:31:21 🐳 Stopping container: 73c4474e064e 372 | 2025/03/25 13:31:21 ✅ Container stopped: 73c4474e064e 373 | 2025/03/25 13:31:21 🐳 Terminating container: 73c4474e064e 374 | 2025/03/25 13:31:21 🚫 Container terminated: 73c4474e064e 375 | --- PASS: TestNewRepository (3.70s) 376 | --- PASS: TestNewRepository/Create_a_talk_and_retrieve_it_by_UUID (0.18s) 377 | --- PASS: TestNewRepository/Exists_by_UUID (0.07s) 378 | --- PASS: TestNewRepository/Does_not_exist_by_UUID (0.04s) 379 | PASS 380 | ok github.com/testcontainers/workshop-go/internal/talks 4.433s 381 | ``` 382 | 383 | _NOTE: if we experiment longer test execution times it could be caused by the need of pulling the images from the registry._ 384 | 385 | ## Integration tests for the Ratings Lambda 386 | 387 | Let's add a new file `internal/ratings/lambda_client_test.go` with the following content: 388 | 389 | ```go 390 | package ratings_test 391 | 392 | import ( 393 | "bytes" 394 | "context" 395 | "encoding/json" 396 | "fmt" 397 | osexec "os/exec" 398 | "path/filepath" 399 | "runtime" 400 | "strings" 401 | "testing" 402 | 403 | "github.com/stretchr/testify/require" 404 | "github.com/testcontainers/testcontainers-go" 405 | "github.com/testcontainers/testcontainers-go/exec" 406 | "github.com/testcontainers/testcontainers-go/modules/localstack" 407 | "github.com/testcontainers/workshop-go/internal/ratings" 408 | ) 409 | 410 | // buildLambda return the path to the ZIP file used to deploy the lambda function. 411 | func buildLambda(t *testing.T) string { 412 | t.Helper() 413 | 414 | _, b, _, _ := runtime.Caller(0) 415 | basepath := filepath.Dir(b) 416 | 417 | lambdaPath := filepath.Join(basepath, "..", "..", "lambda-go") 418 | 419 | makeCmd := osexec.Command("make", "zip-lambda") 420 | makeCmd.Dir = lambdaPath 421 | 422 | err := makeCmd.Run() 423 | require.NoError(t, err) 424 | 425 | return filepath.Join(lambdaPath, "function.zip") 426 | } 427 | 428 | func TestGetStats(t *testing.T) { 429 | ctx := context.Background() 430 | 431 | flagsFn := func() string { 432 | labels := testcontainers.GenericLabels() 433 | flags := "" 434 | for k, v := range labels { 435 | flags = fmt.Sprintf("%s -l %s=%s", flags, k, v) 436 | } 437 | return flags 438 | } 439 | 440 | // get the path to the function.zip file, which lives in the lambda-go folder of the project 441 | zipFile := buildLambda(t) 442 | 443 | var functionURL string 444 | 445 | c, err := localstack.Run(ctx, 446 | "localstack/localstack:latest", 447 | testcontainers.WithEnv(map[string]string{ 448 | "SERVICES": "lambda", 449 | "LAMBDA_DOCKER_FLAGS": flagsFn(), 450 | }), 451 | testcontainers.WithFiles(testcontainers.ContainerFile{ 452 | HostFilePath: zipFile, 453 | ContainerFilePath: "/tmp/function.zip", 454 | }), 455 | testcontainers.CustomizeRequest(testcontainers.GenericContainerRequest{ 456 | ContainerRequest: testcontainers.ContainerRequest{ 457 | LifecycleHooks: []testcontainers.ContainerLifecycleHooks{ 458 | { 459 | PostStarts: []testcontainers.ContainerHook{ 460 | func(ctx context.Context, c testcontainers.Container) error { 461 | lambdaName := "localstack-lambda-url-example" 462 | 463 | // the three commands below are doing the following: 464 | // 1. create a lambda function 465 | // 2. create the URL function configuration for the lambda function 466 | // 3. wait for the lambda function to be active 467 | lambdaCommands := [][]string{ 468 | { 469 | "awslocal", "lambda", 470 | "create-function", "--function-name", lambdaName, 471 | "--runtime", "provided.al2", 472 | "--handler", "bootstrap", 473 | "--role", "arn:aws:iam::111122223333:role/lambda-ex", 474 | "--zip-file", "fileb:///tmp/function.zip", 475 | }, 476 | {"awslocal", "lambda", "create-function-url-config", "--function-name", lambdaName, "--auth-type", "NONE"}, 477 | {"awslocal", "lambda", "wait", "function-active-v2", "--function-name", lambdaName}, 478 | } 479 | for _, cmd := range lambdaCommands { 480 | _, _, err := c.Exec(ctx, cmd) 481 | if err != nil { 482 | return err 483 | } 484 | } 485 | 486 | // 4. get the URL for the lambda function 487 | cmd := []string{ 488 | "awslocal", "lambda", "list-function-url-configs", "--function-name", lambdaName, 489 | } 490 | _, reader, err := c.Exec(ctx, cmd, exec.Multiplexed()) 491 | if err != nil { 492 | return err 493 | } 494 | 495 | buf := new(bytes.Buffer) 496 | _, err = buf.ReadFrom(reader) 497 | if err != nil { 498 | return err 499 | } 500 | 501 | content := buf.Bytes() 502 | 503 | type FunctionURLConfig struct { 504 | FunctionURLConfigs []struct { 505 | FunctionURL string `json:"FunctionUrl"` 506 | FunctionArn string `json:"FunctionArn"` 507 | CreationTime string `json:"CreationTime"` 508 | LastModifiedTime string `json:"LastModifiedTime"` 509 | AuthType string `json:"AuthType"` 510 | } `json:"FunctionUrlConfigs"` 511 | } 512 | 513 | v := &FunctionURLConfig{} 514 | err = json.Unmarshal(content, v) 515 | if err != nil { 516 | return err 517 | } 518 | 519 | // 5. finally, set the function URL from the response 520 | functionURL = v.FunctionURLConfigs[0].FunctionURL 521 | 522 | return nil 523 | }, 524 | }, 525 | }, 526 | }, 527 | }, 528 | }), 529 | ) 530 | testcontainers.CleanupContainer(t, c) 531 | require.NoError(t, err) 532 | 533 | // replace the port with the one exposed by the container 534 | mappedPort, err := c.MappedPort(ctx, "4566/tcp") 535 | require.NoError(t, err) 536 | 537 | url := strings.ReplaceAll(functionURL, "4566", mappedPort.Port()) 538 | 539 | // The latest version of localstack does not add ".localstack.cloud" by default, 540 | // that's why we need to add it to the URL. 541 | url = strings.ReplaceAll(url, ".localhost", ".localhost.localstack.cloud") 542 | 543 | // now we can test the lambda function 544 | lambdaClient := ratings.NewLambdaClient(url) 545 | 546 | histogram := map[string]string{ 547 | "0": "10", 548 | "1": "20", 549 | "2": "30", 550 | "3": "40", 551 | "4": "50", 552 | "5": "60", 553 | } 554 | 555 | stats, err := lambdaClient.GetStats(histogram) 556 | require.NoError(t, err) 557 | 558 | expected := `{"avg":3.3333333333333335,"totalCount":210}` 559 | require.Equal(t, expected, string(stats)) 560 | } 561 | 562 | ``` 563 | 564 | This test will start a LocalStack container, previously building the ZIP file representing the lambda, and it will define one test to verify that the lambda function returns the stats for a given histogram of ratings: 565 | 566 | * `Retrieve the stats for a given histogram of ratings`: it will call the lambda deployed in the LocalStack instance, using a map of ratings as the histogram, and it will verify that the result includes the calculated average and the total count of ratings. 567 | 568 | Please notice that the package has been named with the `_test` suffix for the same reasons describe above. 569 | 570 | There is no need to run `go mod tidy` again, as we have already downloaded the Go dependencies. 571 | 572 | Finally, run your tests with `go test -v -count=1 ./internal/ratings -run TestGetStats` from the root of the project. We should see the following output: 573 | 574 | ```text 575 | === RUN TestGetStats 576 | 2025/03/25 13:35:14 github.com/testcontainers/testcontainers-go - Connected to docker: 577 | Server Version: 27.5.0 (via Testcontainers Desktop 1.19.0) 578 | API Version: 1.46 579 | Operating System: Ubuntu 22.04.5 LTS 580 | Total Memory: 15368 MB 581 | Labels: 582 | cloud.docker.run.version=259.c712f5fd 583 | Testcontainers for Go Version: v0.35.0 584 | Resolved Docker Host: tcp://127.0.0.1:49982 585 | Resolved Docker Socket Path: /var/run/docker.sock 586 | Test SessionID: 4537b6af9f46af836f202c95ef2e5dadf3ba8c33ef605e0191ae857cb20e2ae3 587 | Test ProcessID: 975e388b-e4ee-4f73-8d5f-f16b26a07464 588 | 2025/03/25 13:35:14 Setting LOCALSTACK_HOST to 127.0.0.1 (to match host-routable address for container) 589 | 2025/03/25 13:35:14 🐳 Creating container for image localstack/localstack:latest 590 | 2025/03/25 13:35:15 🐳 Creating container for image testcontainers/ryuk:0.11.0 591 | 2025/03/25 13:35:15 ✅ Container created: 0cfa2462825f 592 | 2025/03/25 13:35:15 🐳 Starting container: 0cfa2462825f 593 | 2025/03/25 13:35:15 ✅ Container started: 0cfa2462825f 594 | 2025/03/25 13:35:15 ⏳ Waiting for container id 0cfa2462825f image: testcontainers/ryuk:0.11.0. Waiting for: &{Port:8080/tcp timeout: PollInterval:100ms skipInternalCheck:false} 595 | 2025/03/25 13:35:15 🔔 Container is ready: 0cfa2462825f 596 | 2025/03/25 13:35:15 ✅ Container created: 7bbf96d6bcca 597 | 2025/03/25 13:35:16 🐳 Starting container: 7bbf96d6bcca 598 | 2025/03/25 13:35:25 ✅ Container started: 7bbf96d6bcca 599 | 2025/03/25 13:35:25 ⏳ Waiting for container id 7bbf96d6bcca image: localstack/localstack:latest. Waiting for: &{timeout:0x140003b7b40 Port:4566/tcp Path:/_localstack/health StatusCodeMatcher:0x1009efae0 ResponseMatcher:0x100a435e0 UseTLS:false AllowInsecure:false TLSConfig: Method:GET Body: Headers:map[] ResponseHeadersMatcher:0x100a435f0 PollInterval:100ms UserInfo: ForceIPv4LocalHost:false} 600 | 2025/03/25 13:35:25 🔔 Container is ready: 7bbf96d6bcca 601 | 2025/03/25 13:35:25 🐳 Stopping container: 7bbf96d6bcca 602 | 2025/03/25 13:35:31 ✅ Container stopped: 7bbf96d6bcca 603 | 2025/03/25 13:35:31 🐳 Terminating container: 7bbf96d6bcca 604 | 2025/03/25 13:35:31 🚫 Container terminated: 7bbf96d6bcca 605 | --- PASS: TestGetStats (16.97s) 606 | PASS 607 | ok github.com/testcontainers/workshop-go/internal/ratings 17.966s 608 | ``` 609 | 610 | _NOTE: if we experiment longer test execution times it could be caused by the need of pulling the images from the registry._ 611 | 612 | We have now added integration tests for the three stores of our application, and our AWS lambda. Let's add some integration tests for the API. 613 | 614 | ### 615 | [Next: Adding integration tests for the APIs](step-9-integration-tests-for-api.md) -------------------------------------------------------------------------------- /step-9-integration-tests-for-api.md: -------------------------------------------------------------------------------- 1 | # Step 9: Integration tests for the API 2 | 3 | In this step we will add integration tests for the API, and for that we are going to use the [`net/httptest`](https://pkg.go.dev/net/http/httptest) package from the standard library. 4 | 5 | ## The `net/httptest` package 6 | 7 | The `net/httptest` package provides a set of utilities for HTTP testing. It includes a test server that implements the `http.Handler` interface, a global `Client` to make requests to test servers, and various functions to parse HTTP responses. 8 | 9 | For the specific use case of `GoFiber`, we are going to follow its [official documentation](https://docs.gofiber.io/recipes/unit-test/). 10 | 11 | ## Testing the HTTP endpoints 12 | 13 | For that, we are going to create a new file called `router_test.go` inside the `internal/app` package. Please add the following content: 14 | 15 | ```go 16 | package app_test 17 | 18 | import ( 19 | "bytes" 20 | "net/http" 21 | "testing" 22 | 23 | "github.com/stretchr/testify/assert" 24 | "github.com/stretchr/testify/require" 25 | "github.com/testcontainers/workshop-go/internal/app" 26 | ) 27 | 28 | func TestRoutesFailBecauseDependenciesAreNotStarted(t *testing.T) { 29 | app := app.SetupApp() 30 | 31 | t.Run("GET /ratings fails", func(t *testing.T) { 32 | req, err := http.NewRequest("GET", "/ratings?talkId=testcontainers-integration-testing", nil) 33 | require.NoError(t, err) 34 | res, err := app.Test(req, -1) 35 | require.NoError(t, err) 36 | 37 | // we are receiving a 500 because the ratings repository is not started 38 | assert.Equal(t, http.StatusInternalServerError, res.StatusCode) 39 | }) 40 | 41 | t.Run("POST /ratings fails", func(t *testing.T) { 42 | body := []byte(`{"talkId":"testcontainers-integration-testing","value":5}`) 43 | 44 | req, err := http.NewRequest("POST", "/ratings", bytes.NewReader(body)) 45 | require.NoError(t, err) 46 | 47 | // we need to set the content type header because we are sending a body 48 | req.Header.Add("Content-Type", "application/json") 49 | 50 | res, err := app.Test(req, -1) 51 | require.NoError(t, err) 52 | 53 | // we are receiving a 500 because the ratings repository is not started 54 | assert.Equal(t, http.StatusInternalServerError, res.StatusCode) 55 | }) 56 | } 57 | 58 | ``` 59 | 60 | Let's check what we are doing here: 61 | 62 | - We are setting up the Gin's router, with the `app.SetupApp` function. 63 | - each subtest defines a new `http.Request`, with the right method and path. 64 | - the `app.Test` method from GoFiber is called with the `http.Request`. 65 | - the `TestRoutesFailBecauseDependenciesAreNotStarted` test method is verifying that the routes that depend on the repositories are failing with a `500 Internal Server error` response code. That's because the runtime dependencies are not started for this test. 66 | 67 | Let's run the test: 68 | 69 | ```bash 70 | go test -v -count=1 ./internal/app -run TestRoutesFailBecauseDependenciesAreNotStarted 71 | === RUN TestRoutesFailBecauseDependenciesAreNotStarted 72 | === RUN TestRoutesFailBecauseDependenciesAreNotStarted/GET_/ratings_fails 73 | Unable to connect to database: failed to connect to `host=/private/tmp user=mdelapenya database=`: dial error (dial unix /private/tmp/.s.PGSQL.5432: connect: no such file or directory) 74 | === RUN TestRoutesFailBecauseDependenciesAreNotStarted/POST_/ratings_fails 75 | Unable to connect to database: failed to connect to `host=/private/tmp user=mdelapenya database=`: dial error (dial unix /private/tmp/.s.PGSQL.5432: connect: no such file or directory) 76 | --- PASS: TestRoutesFailBecauseDependenciesAreNotStarted (0.00s) 77 | --- PASS: TestRoutesFailBecauseDependenciesAreNotStarted/GET_/ratings_fails (0.00s) 78 | --- PASS: TestRoutesFailBecauseDependenciesAreNotStarted/POST_/ratings_fails (0.00s) 79 | PASS 80 | ok github.com/testcontainers/workshop-go/internal/app 1.092s 81 | ``` 82 | 83 | This unit test is not very useful, but it is a good starting point to understand how to test the HTTP endpoints. 84 | 85 | ### 86 | [Next: E2E tests with real dependencies](step-10-e2e-tests-with-real-dependencies.md) --------------------------------------------------------------------------------