├── .github └── workflows │ └── test.yml ├── .gitignore ├── .golangci.yml ├── DESIGN.md ├── Dockerfile ├── LICENSE ├── README.md ├── cmd ├── enqueue.go ├── root.go └── worker.go ├── collector ├── collector.go ├── collector_suite_test.go ├── filesystem │ └── file.go ├── thumbnails │ ├── thumbnails.go │ └── thumbnails_suite_test.go └── watcher │ ├── inotify.go │ ├── kqueue.go │ ├── watcher.go │ └── watcher_suite_test.go ├── config └── config.go ├── docker-compose.yml ├── extractor ├── cmd.go ├── extractor.go ├── extractor_suite_test.go └── inventory │ └── inventory.go ├── go.mod ├── go.sum ├── justfile ├── main.go ├── store └── redis.go ├── tasks ├── enqueuer.go ├── middleware.go ├── task.go └── worker.go └── testvideo ├── Dockerfile ├── OpenSans-Bold.ttf ├── stream.sh └── streams.json /.github/workflows/test.yml: -------------------------------------------------------------------------------- 1 | # This workflow will build a golang project 2 | # For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go 3 | 4 | name: build and test 5 | 6 | on: 7 | push: 8 | branches: [ "main" ] 9 | pull_request: 10 | branches: [ "main" ] 11 | 12 | jobs: 13 | 14 | build: 15 | runs-on: ubuntu-latest 16 | steps: 17 | - uses: actions/checkout@v3 18 | 19 | - name: Set up Go 20 | uses: actions/setup-go@v3 21 | with: 22 | go-version: 1.20.1 23 | 24 | - name: Build 25 | run: go build -v ./... 26 | 27 | - name: Test 28 | run: go test -v ./... -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Binaries for programs and plugins 2 | *.exe 3 | *.exe~ 4 | *.dll 5 | *.so 6 | *.dylib 7 | 8 | # Test binary, build with `go test -c` 9 | *.test 10 | 11 | # Coverage files 12 | coverage.txt 13 | 14 | # Output of the go coverage tool, specifically when used with LiteIDE 15 | *.out 16 | 17 | # Testing data 18 | testvideo/*.m3u8 19 | testvideo/*.ts 20 | testvideo/**/*.jpg 21 | -------------------------------------------------------------------------------- /.golangci.yml: -------------------------------------------------------------------------------- 1 | run: 2 | timeout: 2m 3 | tests: false 4 | skip-dirs: 5 | - "(^|/)tests($|/)" 6 | output: 7 | format: colored-line-number 8 | print-issued-lines: true 9 | print-linter-name: true 10 | uniq-by-line: true 11 | 12 | linters: 13 | enable: 14 | - depguard 15 | - errorlint 16 | - goconst 17 | - goimports 18 | - gosec 19 | - unconvert 20 | - wrapcheck 21 | - gomnd 22 | - lll 23 | - goerr113 24 | - bodyclose 25 | -------------------------------------------------------------------------------- /DESIGN.md: -------------------------------------------------------------------------------- 1 | # Design 2 | 3 | This document describes the rationale behind the Video Samples project. 4 | 5 | ```mermaid 6 | --- 7 | title: Video Samples flowchart 8 | --- 9 | graph LR 10 | A[Extractor] --> |Extract resources| B[(Filesystem)] 11 | C[Collector] --> |Collect resources| B 12 | C --> |Store resources| D(Redis) 13 | E[API] --> |Query resources| D[(Redis)] 14 | ``` 15 | 16 | ## What it is? 17 | 18 | Video Samples is a software used to extract resources from video. 19 | 20 | ## How does it work? 21 | 22 | As technology users, we are not comfortable with the idea of installing multiple frameworks, libs and tools to perform a single task. We like to use the same set of primitives to make something usable. This is the idea behind Video Samples: it is a software that can **extract** video resources, **store** them somewhere and **retrieve** them using a nice API. 23 | 24 | Let's jump right into the design of these components. 25 | 26 | ### Extractor 27 | 28 | An extractor is responsible for generating assets from video and save it in the filesystem. Since we are using ffmpeg, we are restricted to some limitations like the assets that are written to the disk, forcing the program to create a mechanism to know when a new file has been created. 29 | 30 | Extracting resources from video can be done in a multiple ways when using a programming language, but for this project we choose to rely on OS processes. We can build a ffmpeg command line, run it in background and maintain a list of video titles being processed. Extracting thumbnails from live video, for example, is a long running process - we need to apply some kind of healthcheck mechanism to ensure the resources are being extracted. 31 | 32 | We don't want multiples ffmpeg processes running for the same feature and video. To avoid this behavior, we generate a unique ID composed by the video name and the feature. 33 | 34 | ### Collector 35 | 36 | A collector is responsible for collecting the files generated from ffmpeg. When writing the file to disk, 37 | ffmpeg writes the content to the file and then closes it. For event monitors like _inotify_, it is the `CLOSE_WRITE` event. 38 | 39 | A monitor can be started to watch events of a given path and stores it everytime a new file shows up. 40 | 41 | *Thumbnails* for live videos don't need to be kept forever in a database because live videos have the DVR (digital video recording) concept. DVR allows user to rewind the broadcast, going back in time. 42 | 43 | The diagram below describes how we can store and query thumbnails: 44 | 45 | ```mermaid 46 | --- 47 | title: Store and query thumbnails 48 | --- 49 | 50 | graph LR 51 | collector([collector])-. Save thumbnail info
into Redis set .->redisset[(Redis)] 52 | redisset --> success{Success?} 53 | success -->|Yes, save blob| redisblob[(Redis)] 54 | success -->|No| discard[Discard thumb] 55 | ``` 56 | 57 | To save a thumbnail, we can add them to a [sorted set](https://redis.io/commands/zadd/) and set the score with the timestamp value of the moment the thumbnail was inserted. 58 | 59 | After collecting the thumbnail we can insert a member in Redis using unique identifier - we can use the very same identifier to upload the file in the storage. 60 | 61 | Since we are dealing with sorted sets, we can use `ZADD`. Suppose we have a live streaming named _bunny_: 62 | 63 | ``` 64 | ZADD thumbnails/bunny 1677779184 5fdbadc9-b300-4514-9171-0297caba44bb 65 | ``` 66 | 67 | A routine can be fired to remove keys older than the thumbnail TTL (score), using [this Redis operation](https://redis.io/commands/zremrangebyscore/) -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM golang:1.20 2 | 3 | RUN apt-get update && apt-get install -y ffmpeg 4 | 5 | WORKDIR /app 6 | 7 | COPY go.mod go.sum ./ 8 | RUN go mod download 9 | 10 | COPY . . 11 | 12 | RUN go build -o /video_samples 13 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 Mauricio Antunes 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Video Samples - extract resources from video 2 | 3 | **🎥 Have you ever wanted to extract thumbnails or short videos from a real video? This project may be the answer you're looking for.** 4 | 5 | See [the design docs](./DESIGN.md) if you want to know how it works. 6 | 7 | ## 🚀 Install 8 | 9 | Coming soon 10 | 11 | ## 💡 Usage 12 | 13 | Coming soon 14 | 15 | ## Development 16 | 17 | ``` 18 | docker compose up 19 | ``` 20 | 21 | This command will get up and running the following components: 22 | 23 | * **Enqueuer** - enqueue jobs to extract the extractors 24 | * **Worker** - workers to extract resources from video (using _ffmpeg_) 25 | * **Redis** - used by _enqueuer_ and as a datastore to the extracted resources 26 | * **Video API** - a dummy API that servers a JSON endpoint with video streamings URLs 27 | * **Stream** - a live streaming generated with _ffmpeg_ 28 | 29 | ## Tests 30 | 31 | First, make sure you have the [ginkgo](http://onsi.github.io/ginkgo/) test runner installed. 32 | 33 | Then run the test suite: 34 | 35 | ``` 36 | ginkgo -p -v ./... 37 | ``` 38 | -------------------------------------------------------------------------------- /cmd/enqueue.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "github.com/mauricioabreu/video_samples/collector" 5 | "github.com/mauricioabreu/video_samples/config" 6 | "github.com/mauricioabreu/video_samples/extractor/inventory" 7 | "github.com/mauricioabreu/video_samples/tasks" 8 | "github.com/rs/zerolog/log" 9 | "github.com/spf13/cobra" 10 | ) 11 | 12 | func EnqueueCmd() *cobra.Command { 13 | return &cobra.Command{ 14 | Use: "enqueue", 15 | Short: "Enqueue tasks to extract, collect and store video samples", 16 | Run: func(cmd *cobra.Command, args []string) { 17 | cfg, err := config.GetConfig() 18 | if err != nil { 19 | log.Fatal().Err(err).Msg("Failed to get config") 20 | } 21 | getStreams := func(url string) func() ([]inventory.Streaming, error) { 22 | return func() ([]inventory.Streaming, error) { 23 | return inventory.GetStreams(url) 24 | } 25 | } 26 | tasks.Enqueue(getStreams(cfg.InventoryAddress), cfg.RedisAddr) 27 | collector.Collect("testvideo/thumbs/colors") 28 | }, 29 | } 30 | } 31 | -------------------------------------------------------------------------------- /cmd/root.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "fmt" 5 | "os" 6 | 7 | "github.com/spf13/cobra" 8 | ) 9 | 10 | func NewRootCmd() *cobra.Command { 11 | rootCmd := &cobra.Command{ 12 | Use: "video_samples", 13 | Short: "Extract resources from video", 14 | SilenceUsage: true, 15 | SilenceErrors: true, 16 | } 17 | 18 | rootCmd.AddCommand(EnqueueCmd()) 19 | rootCmd.AddCommand(Work()) 20 | 21 | return rootCmd 22 | } 23 | 24 | func Execute() { 25 | if err := NewRootCmd().Execute(); err != nil { 26 | fmt.Println(err) 27 | os.Exit(1) 28 | } 29 | } 30 | -------------------------------------------------------------------------------- /cmd/worker.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "github.com/mauricioabreu/video_samples/config" 5 | "github.com/mauricioabreu/video_samples/tasks" 6 | "github.com/rs/zerolog/log" 7 | "github.com/spf13/cobra" 8 | ) 9 | 10 | func Work() *cobra.Command { 11 | return &cobra.Command{ 12 | Use: "work", 13 | Short: "Start workers to extract and process video samples", 14 | Run: func(cmd *cobra.Command, args []string) { 15 | cfg, err := config.GetConfig() 16 | if err != nil { 17 | log.Fatal().Err(err).Msg("Failed to get config") 18 | } 19 | tasks.StartWorker(&cfg) 20 | }, 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /collector/collector.go: -------------------------------------------------------------------------------- 1 | package collector 2 | 3 | import ( 4 | "github.com/google/uuid" 5 | 6 | "github.com/mauricioabreu/video_samples/collector/filesystem" 7 | "github.com/mauricioabreu/video_samples/collector/thumbnails" 8 | "github.com/mauricioabreu/video_samples/collector/watcher" 9 | "github.com/mauricioabreu/video_samples/config" 10 | "github.com/mauricioabreu/video_samples/store" 11 | "github.com/rs/zerolog/log" 12 | ) 13 | 14 | func Collect(path string) { 15 | cfg, err := config.GetConfig() 16 | if err != nil { 17 | log.Fatal().Err(err).Msg("Failed to get config") 18 | } 19 | rc := store.NewRedis(cfg.RedisAddr) 20 | files, err := watcher.Watch(path) 21 | if err != nil { 22 | log.Fatal().Err(err).Msg("Failed to initialize collector") 23 | } 24 | 25 | for file := range files { 26 | log.Debug().Msgf("File found: %s", file) 27 | thumbnail, err := filesystem.NewFile(file) 28 | if err != nil { 29 | log.Error().Err(err).Msgf("Failed to read file: %s", file) 30 | } 31 | thumbnails.Insert(thumbnail, 60, uuid.NewString, rc) 32 | } 33 | } 34 | -------------------------------------------------------------------------------- /collector/collector_suite_test.go: -------------------------------------------------------------------------------- 1 | package collector_test 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/mauricioabreu/video_samples/collector/watcher" 7 | . "github.com/onsi/ginkgo/v2" 8 | . "github.com/onsi/gomega" 9 | ) 10 | 11 | func TestCollector(t *testing.T) { 12 | RegisterFailHandler(Fail) 13 | RunSpecs(t, "Collector Suite") 14 | } 15 | 16 | var _ = Describe("Collector", func() { 17 | Describe("Match file extension", func() { 18 | When("patterns list contains the extension", func() { 19 | It("matches", func() { 20 | patterns := []string{"jpg", "jpeg", "png"} 21 | Expect(watcher.MatchExt("/thumbnails/bunny/0001.jpg", patterns)).To(BeTrue()) 22 | }) 23 | }) 24 | When("patterns list does not contain the extension", func() { 25 | It("does not match", func() { 26 | patterns := []string{"png", "bmp"} 27 | Expect(watcher.MatchExt("/thumbnails/bunny/0001.jpg", patterns)).To(BeFalse()) 28 | }) 29 | }) 30 | When("path does not have extension", func() { 31 | It("does not match", func() { 32 | patterns := []string{"jpg", "jpeg", "png"} 33 | Expect(watcher.MatchExt("/thumbnails/bunny/0001", patterns)).To(BeFalse()) 34 | }) 35 | }) 36 | }) 37 | }) 38 | -------------------------------------------------------------------------------- /collector/filesystem/file.go: -------------------------------------------------------------------------------- 1 | package filesystem 2 | 3 | import ( 4 | "fmt" 5 | "os" 6 | "path/filepath" 7 | ) 8 | 9 | type File struct { 10 | Path string 11 | Dir string 12 | Data []byte 13 | ModTime int64 14 | } 15 | 16 | func NewFile(path string) (*File, error) { 17 | fileInfo, err := os.Stat(path) 18 | if err != nil { 19 | return nil, fmt.Errorf("failed to read file info: %w", err) 20 | } 21 | data, err := os.ReadFile(path) 22 | if err != nil { 23 | return nil, fmt.Errorf("failed to read file content: %w", err) 24 | } 25 | return &File{ 26 | Path: path, 27 | Dir: filepath.Dir(path), 28 | Data: data, 29 | ModTime: fileInfo.ModTime().Unix(), 30 | }, nil 31 | } 32 | -------------------------------------------------------------------------------- /collector/thumbnails/thumbnails.go: -------------------------------------------------------------------------------- 1 | package thumbnails 2 | 3 | import ( 4 | "context" 5 | "fmt" 6 | "time" 7 | 8 | "github.com/mauricioabreu/video_samples/collector/filesystem" 9 | "github.com/redis/go-redis/v9" 10 | ) 11 | 12 | func Insert(file *filesystem.File, expiryAfter int, uuid func() string, rc *redis.Client) error { 13 | thumbId := fmt.Sprintf("blob/%s", uuid()) 14 | thumbsKey := fmt.Sprintf("thumbnails/%s", file.Dir) 15 | 16 | err := rc.ZAdd(context.TODO(), thumbsKey, redis.Z{Score: float64(file.ModTime), Member: thumbId}).Err() 17 | if err != nil { 18 | return fmt.Errorf("failed to insert into redis: %w", err) 19 | } 20 | 21 | expiration := time.Duration(expiryAfter) * time.Second 22 | if err := rc.Set(context.TODO(), thumbId, file.Data, expiration).Err(); err != nil { 23 | return fmt.Errorf("failed to insert into redis: %w", err) 24 | } 25 | 26 | return nil 27 | } 28 | -------------------------------------------------------------------------------- /collector/thumbnails/thumbnails_suite_test.go: -------------------------------------------------------------------------------- 1 | package thumbnails_test 2 | 3 | import ( 4 | "errors" 5 | "testing" 6 | "time" 7 | 8 | "github.com/go-redis/redismock/v9" 9 | "github.com/mauricioabreu/video_samples/collector/filesystem" 10 | "github.com/mauricioabreu/video_samples/collector/thumbnails" 11 | . "github.com/onsi/ginkgo/v2" 12 | . "github.com/onsi/gomega" 13 | "github.com/redis/go-redis/v9" 14 | ) 15 | 16 | func TestThumbnails(t *testing.T) { 17 | RegisterFailHandler(Fail) 18 | RunSpecs(t, "Thumbnails Suite") 19 | } 20 | 21 | var _ = Describe("Thumbnails insert", func() { 22 | When("Adding to redis succeeds", func() { 23 | It("inserts the thumbnail", func() { 24 | file := &filesystem.File{ 25 | Path: "/thumbnails/bunny/0001.jpg", 26 | Dir: "bunny", 27 | Data: []byte("test_data"), 28 | ModTime: 1678103906, 29 | } 30 | uuid := func() string { return "1" } 31 | redisClient, redisMock := redismock.NewClientMock() 32 | redisMock. 33 | ExpectZAdd("thumbnails/bunny", redis.Z{Score: float64(file.ModTime), Member: "blob/1"}). 34 | SetVal(0) 35 | redisMock. 36 | ExpectSet("blob/1", []byte("test_data"), time.Duration(30)*time.Second).SetVal("OK") 37 | 38 | err := thumbnails.Insert(file, 30, uuid, redisClient) 39 | 40 | Expect(redisMock.ExpectationsWereMet()).To(Not(HaveOccurred())) 41 | Expect(err).To(Not(HaveOccurred())) 42 | }) 43 | }) 44 | When("Adding to set fails", func() { 45 | It("does not insert the thumbnail", func() { 46 | file := &filesystem.File{ 47 | Path: "/thumbnails/bunny/0001.jpg", 48 | Dir: "bunny", 49 | Data: []byte("test_data"), 50 | ModTime: 1678103906, 51 | } 52 | uuid := func() string { return "1" } 53 | redisClient, redisMock := redismock.NewClientMock() 54 | redisMock. 55 | ExpectZAdd("thumbnails/bunny", redis.Z{Score: float64(file.ModTime), Member: uuid()}). 56 | SetErr(errors.New("failed to execute zadd cmd")) 57 | 58 | err := thumbnails.Insert(file, 30, uuid, redisClient) 59 | Expect(err).To(HaveOccurred()) 60 | }) 61 | }) 62 | }) 63 | -------------------------------------------------------------------------------- /collector/watcher/inotify.go: -------------------------------------------------------------------------------- 1 | //go:build linux 2 | // +build linux 3 | 4 | package watcher 5 | 6 | import ( 7 | "github.com/rjeczalik/notify" 8 | ) 9 | 10 | const ( 11 | WriteEvent = notify.Event(notify.InCloseWrite) 12 | ) 13 | -------------------------------------------------------------------------------- /collector/watcher/kqueue.go: -------------------------------------------------------------------------------- 1 | //go:build darwin && !kqueue && cgo 2 | // +build darwin,!kqueue,cgo 3 | 4 | package watcher 5 | 6 | import ( 7 | "github.com/rjeczalik/notify" 8 | ) 9 | 10 | const ( 11 | WriteEvent = notify.Event(notify.FSEventsModified) 12 | ) 13 | -------------------------------------------------------------------------------- /collector/watcher/watcher.go: -------------------------------------------------------------------------------- 1 | package watcher 2 | 3 | import ( 4 | "fmt" 5 | "path/filepath" 6 | 7 | "github.com/rjeczalik/notify" 8 | "github.com/samber/lo" 9 | ) 10 | 11 | type File struct { 12 | Path string 13 | Dir string 14 | ModTime int64 15 | } 16 | 17 | // Magic number. notify demands for a buffered channel because it does not block sending 18 | // to the channel 19 | var filesBuffer = 200 20 | 21 | // Watch files in a given path and sends events to channels to be 22 | // processed later 23 | func Watch(path string) (<-chan string, error) { 24 | files := make(chan string, filesBuffer) 25 | 26 | c := make(chan notify.EventInfo, 1) 27 | if err := notify.Watch(path, c, WriteEvent); err != nil { 28 | return nil, fmt.Errorf("failed to watch %s: %w", path, err) 29 | } 30 | 31 | go func() { 32 | for { 33 | e := <-c 34 | path := e.Path() 35 | if MatchImage(path) { 36 | files <- path 37 | } 38 | } 39 | }() 40 | 41 | return files, nil 42 | } 43 | 44 | // MatchExt checks if a given path matches a list of patterns 45 | func MatchExt(path string, patterns []string) bool { 46 | ext := filepath.Ext(path) 47 | if ext == "" { 48 | return false 49 | } 50 | return lo.Contains(patterns, ext[1:]) 51 | } 52 | 53 | func MatchImage(path string) bool { 54 | return MatchExt(path, []string{"jpg", "png"}) 55 | } 56 | -------------------------------------------------------------------------------- /collector/watcher/watcher_suite_test.go: -------------------------------------------------------------------------------- 1 | package watcher_test 2 | 3 | import ( 4 | "testing" 5 | 6 | . "github.com/onsi/ginkgo/v2" 7 | . "github.com/onsi/gomega" 8 | ) 9 | 10 | func TestWatcher(t *testing.T) { 11 | RegisterFailHandler(Fail) 12 | RunSpecs(t, "Watcher Suite") 13 | } 14 | -------------------------------------------------------------------------------- /config/config.go: -------------------------------------------------------------------------------- 1 | package config 2 | 3 | import ( 4 | "fmt" 5 | 6 | "github.com/caarlos0/env/v8" 7 | ) 8 | 9 | type Config struct { 10 | RedisAddr string `env:"REDIS_ADDR,notEmpty"` 11 | EnqueueConcurrency int `env:"WORKERS_CONCURRENCY" envDefault:"10"` 12 | InventoryAddress string `env:"INVENTORY_ADDRESS,notEmpty"` 13 | } 14 | 15 | func GetConfig() (Config, error) { 16 | cfg := Config{} 17 | if err := env.Parse(&cfg); err != nil { 18 | return cfg, fmt.Errorf("failed to load configs: %w", err) 19 | } 20 | return cfg, nil 21 | } 22 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | services: 2 | worker: 3 | build: . 4 | command: go run main.go work 5 | depends_on: 6 | - redis 7 | environment: 8 | - REDIS_ADDR=redis:6379 9 | - INVENTORY_ADDRESS=http://video-api:8080/streams.json 10 | enqueuer: 11 | build: . 12 | command: go run main.go enqueue 13 | depends_on: 14 | - redis 15 | environment: 16 | - REDIS_ADDR=redis:6379 17 | - INVENTORY_ADDRESS=http://video-api:8080/streams.json 18 | redis: 19 | image: redis:7-alpine 20 | ports: 21 | - "6379:6379" 22 | video-api: 23 | image: python:3.9-alpine 24 | command: python3 -m http.server -d /testvideo 8080 25 | volumes: 26 | - ./testvideo:/testvideo 27 | ports: 28 | - "8080:8080" 29 | stream: 30 | build: testvideo 31 | command: sh -c /testvideo/stream.sh 32 | ports: 33 | - "9090:9090" 34 | -------------------------------------------------------------------------------- /extractor/cmd.go: -------------------------------------------------------------------------------- 1 | package extractor 2 | 3 | import ( 4 | "bufio" 5 | "fmt" 6 | "io" 7 | "os/exec" 8 | 9 | "github.com/rs/zerolog/log" 10 | ) 11 | 12 | type Command struct { 13 | executable string 14 | args []string 15 | } 16 | 17 | func RunCmd(c Command) error { 18 | cmd := exec.Command(c.executable, c.args...) //#nosec G204 19 | stderr, err := cmd.StderrPipe() 20 | if err != nil { 21 | return fmt.Errorf("failed to create a pipe from stderr: %w", err) 22 | } 23 | 24 | if err := cmd.Start(); err != nil { 25 | return fmt.Errorf("failed to start cmd: %w", err) 26 | } 27 | 28 | logCmd(stderr) 29 | 30 | if err := cmd.Wait(); err != nil { 31 | return fmt.Errorf("failed to wait cmd: %w", err) 32 | } 33 | 34 | return nil 35 | } 36 | 37 | // logCmd capture the buffer and log each line of it 38 | func logCmd(rc io.ReadCloser) { 39 | scanner := bufio.NewScanner(rc) 40 | scanner.Split(bufio.ScanLines) 41 | for scanner.Scan() { 42 | log.Info().Msg(scanner.Text()) 43 | } 44 | } 45 | -------------------------------------------------------------------------------- /extractor/extractor.go: -------------------------------------------------------------------------------- 1 | package extractor 2 | 3 | import ( 4 | "fmt" 5 | ) 6 | 7 | type ThumbOptions struct { 8 | Input string 9 | Output string 10 | Scale string 11 | Quality uint 12 | } 13 | 14 | func ExtractThumbs(title string, opts ThumbOptions, runner func(Command) error) error { 15 | args := []string{ 16 | "-hide_banner", 17 | "-loglevel", "warning", 18 | "-live_start_index", "-1", 19 | "-f", "hls", 20 | "-i", opts.Input, 21 | "-vf", fmt.Sprintf("fps=1,scale=%s", opts.Scale), 22 | "-vsync", "vfr", 23 | "-q:v", fmt.Sprintf("%d", opts.Quality), 24 | "-threads", "1", 25 | fmt.Sprintf("%s/%s/%%09d.jpg", opts.Output, title), 26 | } 27 | return runner(Command{executable: "ffmpeg", args: args}) 28 | } 29 | -------------------------------------------------------------------------------- /extractor/extractor_suite_test.go: -------------------------------------------------------------------------------- 1 | package extractor_test 2 | 3 | import ( 4 | "errors" 5 | "testing" 6 | 7 | "github.com/mauricioabreu/video_samples/extractor" 8 | . "github.com/onsi/ginkgo/v2" 9 | . "github.com/onsi/gomega" 10 | ) 11 | 12 | func TestExtractor(t *testing.T) { 13 | RegisterFailHandler(Fail) 14 | RunSpecs(t, "Extractor Suite") 15 | } 16 | 17 | var _ = Describe("Extract resources from video", func() { 18 | Describe("Extract thumbs", func() { 19 | When("Running it works", func() { 20 | It("Start to extract thumbs", func() { 21 | opts := extractor.ThumbOptions{ 22 | Input: "http://localhost:8080/colors/playlist.m3u8", 23 | Scale: "-1:360", 24 | Quality: 5, 25 | } 26 | runner := func(extractor.Command) error { 27 | return nil 28 | } 29 | err := extractor.ExtractThumbs("colors", opts, runner) 30 | Expect(err).To(Not(HaveOccurred())) 31 | }) 32 | }) 33 | When("Running it fails", func() { 34 | It("Start to extract thumbs", func() { 35 | opts := extractor.ThumbOptions{ 36 | Input: "http://localhost:8080/colors/playlist.m3u8", 37 | Scale: "-1:360", 38 | Quality: 5, 39 | } 40 | runner := func(extractor.Command) error { 41 | return errors.New("failed to run") 42 | } 43 | err := extractor.ExtractThumbs("colors", opts, runner) 44 | Expect(err).To(HaveOccurred()) 45 | }) 46 | }) 47 | }) 48 | }) 49 | -------------------------------------------------------------------------------- /extractor/inventory/inventory.go: -------------------------------------------------------------------------------- 1 | package inventory 2 | 3 | import ( 4 | "encoding/json" 5 | "io" 6 | "net/http" 7 | ) 8 | 9 | type Streaming struct { 10 | Name string `json:"name"` 11 | Playlist string `json:"playlist"` 12 | } 13 | 14 | // GetStreams makes HTTP request to an endpoint in order to get 15 | // the inventory of availables videos to extract resources. 16 | func GetStreams(url string) ([]Streaming, error) { 17 | var streamings []Streaming 18 | resp, err := http.Get(url) //nolint 19 | if err != nil { 20 | return streamings, err 21 | } 22 | body, err := io.ReadAll(resp.Body) 23 | if err != nil { 24 | return streamings, err 25 | } 26 | defer resp.Body.Close() 27 | 28 | if err := json.Unmarshal(body, &streamings); err != nil { 29 | return streamings, err 30 | } 31 | 32 | return streamings, nil 33 | } 34 | -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/mauricioabreu/video_samples 2 | 3 | go 1.20 4 | 5 | require ( 6 | github.com/caarlos0/env/v8 v8.0.0 7 | github.com/go-redis/redismock/v9 v9.0.1 8 | github.com/google/uuid v1.3.0 9 | github.com/hibiken/asynq v0.24.0 10 | github.com/onsi/ginkgo/v2 v2.10.0 11 | github.com/onsi/gomega v1.27.7 12 | github.com/redis/go-redis/v9 v9.0.2 13 | github.com/rjeczalik/notify v0.9.3 14 | github.com/rs/zerolog v1.29.0 15 | github.com/samber/lo v1.37.0 16 | github.com/spf13/cobra v1.6.1 17 | ) 18 | 19 | require ( 20 | github.com/cespare/xxhash/v2 v2.2.0 // indirect 21 | github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect 22 | github.com/fsnotify/fsnotify v1.6.0 // indirect 23 | github.com/go-logr/logr v1.2.4 // indirect 24 | github.com/go-redis/redis/v8 v8.11.5 // indirect 25 | github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 // indirect 26 | github.com/golang/protobuf v1.5.3 // indirect 27 | github.com/google/go-cmp v0.5.9 // indirect 28 | github.com/google/pprof v0.0.0-20230228050547-1710fef4ab10 // indirect 29 | github.com/inconshreveable/mousetrap v1.1.0 // indirect 30 | github.com/mattn/go-colorable v0.1.13 // indirect 31 | github.com/mattn/go-isatty v0.0.17 // indirect 32 | github.com/robfig/cron/v3 v3.0.1 // indirect 33 | github.com/spf13/cast v1.5.0 // indirect 34 | github.com/spf13/pflag v1.0.5 // indirect 35 | golang.org/x/exp v0.0.0-20230304125523-9ff063c70017 // indirect 36 | golang.org/x/net v0.10.0 // indirect 37 | golang.org/x/sys v0.8.0 // indirect 38 | golang.org/x/text v0.9.0 // indirect 39 | golang.org/x/time v0.3.0 // indirect 40 | golang.org/x/tools v0.9.3 // indirect 41 | google.golang.org/protobuf v1.28.1 // indirect 42 | gopkg.in/yaml.v3 v3.0.1 // indirect 43 | ) 44 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= 2 | github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= 3 | github.com/bsm/ginkgo/v2 v2.5.0 h1:aOAnND1T40wEdAtkGSkvSICWeQ8L3UASX7YVCqQx+eQ= 4 | github.com/bsm/gomega v1.20.0 h1:JhAwLmtRzXFTx2AkALSLa8ijZafntmhSoU63Ok18Uq8= 5 | github.com/caarlos0/env/v8 v8.0.0 h1:POhxHhSpuxrLMIdvTGARuZqR4Jjm8AYmoi/JKlcScs0= 6 | github.com/caarlos0/env/v8 v8.0.0/go.mod h1:7K4wMY9bH0esiXSSHlfHLX5xKGQMnkH5Fk4TDSSSzfo= 7 | github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= 8 | github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= 9 | github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44= 10 | github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= 11 | github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= 12 | github.com/coreos/go-systemd/v22 v22.3.3-0.20220203105225-a9a7ef127534/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc= 13 | github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= 14 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 15 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 16 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 17 | github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78= 18 | github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc= 19 | github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= 20 | github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= 21 | github.com/frankban/quicktest v1.14.3 h1:FJKSZTDHjyhriyC81FLQ0LY93eSai0ZyR/ZIkd3ZUKE= 22 | github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo= 23 | github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ= 24 | github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY= 25 | github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw= 26 | github.com/go-logr/logr v1.2.4 h1:g01GSCwiDw2xSZfjJ2/T9M+S6pFdcNtFYsp+Y43HYDQ= 27 | github.com/go-logr/logr v1.2.4/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= 28 | github.com/go-redis/redis/v8 v8.11.2/go.mod h1:DLomh7y2e3ggQXQLd1YgmvIfecPJoFl7WU5SOQ/r06M= 29 | github.com/go-redis/redis/v8 v8.11.5 h1:AcZZR7igkdvfVmQTPnu9WE37LRrO/YrBH5zWyjDC0oI= 30 | github.com/go-redis/redis/v8 v8.11.5/go.mod h1:gREzHqY1hg6oD9ngVRbLStwAWKhA0FEgq8Jd4h5lpwo= 31 | github.com/go-redis/redismock/v9 v9.0.1 h1:457RlAT6wDrZx9sMh0xRWuZ3hzPU5k2aWLqgj3KfBQw= 32 | github.com/go-redis/redismock/v9 v9.0.1/go.mod h1:Ojrqw2Kut8BB8HZlXwNgfwhp5xvtVQTjgbIdIMi980g= 33 | github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572 h1:tfuBGBXKqDEevZMzYi5KSi8KkcZtzBcTgAUUtapy0OI= 34 | github.com/go-task/slim-sprig v0.0.0-20230315185526-52ccab3ef572/go.mod h1:9Pwr4B2jHnOSGXyyzV8ROjYa2ojvAY6HCGYYfMoC3Ls= 35 | github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA= 36 | github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= 37 | github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= 38 | github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 39 | github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= 40 | github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= 41 | github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= 42 | github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= 43 | github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= 44 | github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= 45 | github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= 46 | github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= 47 | github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= 48 | github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= 49 | github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= 50 | github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= 51 | github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 52 | github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= 53 | github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 54 | github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 55 | github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 56 | github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 57 | github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= 58 | github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= 59 | github.com/google/pprof v0.0.0-20230228050547-1710fef4ab10 h1:CqYfpuYIjnlNxM3msdyPRKabhXZWbKjf3Q8BWROFBso= 60 | github.com/google/pprof v0.0.0-20230228050547-1710fef4ab10/go.mod h1:79YE0hCXdHag9sBkw2o+N/YnZtTkXi0UT9Nnixa5eYk= 61 | github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= 62 | github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I= 63 | github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= 64 | github.com/hibiken/asynq v0.24.0 h1:r1CiSVYCy1vGq9REKGI/wdB2D5n/QmtzihYHHXOuBUs= 65 | github.com/hibiken/asynq v0.24.0/go.mod h1:FVnRfUTm6gcoDkM/EjF4OIh5/06ergCPUO6pS2B2y+w= 66 | github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU= 67 | github.com/inconshreveable/mousetrap v1.0.1/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= 68 | github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= 69 | github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= 70 | github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= 71 | github.com/kr/pretty v0.3.0 h1:WgNl7dwNpEZ6jJ9k1snq4pZsg7DOEN8hP9Xw0Tsjwk0= 72 | github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= 73 | github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= 74 | github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY= 75 | github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= 76 | github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA= 77 | github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg= 78 | github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= 79 | github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= 80 | github.com/mattn/go-isatty v0.0.17 h1:BTarxUcIeDqL27Mc+vyvdWYSL28zpIhv3RoTdsLMPng= 81 | github.com/mattn/go-isatty v0.0.17/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= 82 | github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A= 83 | github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE= 84 | github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE= 85 | github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk= 86 | github.com/onsi/ginkgo v1.15.0/go.mod h1:hF8qUzuuC8DJGygJH3726JnCZX4MYbRB8yFfISqnKUg= 87 | github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE= 88 | github.com/onsi/ginkgo/v2 v2.10.0 h1:sfUl4qgLdvkChZrWCYndY2EAu9BRIw1YphNAzy1VNWs= 89 | github.com/onsi/ginkgo/v2 v2.10.0/go.mod h1:UDQOh5wbQUlMnkLfVaIUMtQ1Vus92oM+P2JX1aulgcE= 90 | github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY= 91 | github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo= 92 | github.com/onsi/gomega v1.10.5/go.mod h1:gza4q3jKQJijlu05nKWRCW/GavJumGt8aNRxWg7mt48= 93 | github.com/onsi/gomega v1.27.7 h1:fVih9JD6ogIiHUN6ePK7HJidyEDpWGVB5mzM7cWNXoU= 94 | github.com/onsi/gomega v1.27.7/go.mod h1:1p8OOlwo2iUUDsHnOrjE5UKYJ+e3W8eQ3qSlRahPmr4= 95 | github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 96 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 97 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 98 | github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= 99 | github.com/redis/go-redis/v9 v9.0.2 h1:BA426Zqe/7r56kCcvxYLWe1mkaz71LKF77GwgFzSxfE= 100 | github.com/redis/go-redis/v9 v9.0.2/go.mod h1:/xDTe9EF1LM61hek62Poq2nzQSGj0xSrEtEHbBQevps= 101 | github.com/rjeczalik/notify v0.9.3 h1:6rJAzHTGKXGj76sbRgDiDcYj/HniypXmSJo1SWakZeY= 102 | github.com/rjeczalik/notify v0.9.3/go.mod h1:gF3zSOrafR9DQEWSE8TjfI9NkooDxbyT4UgRGKZA0lc= 103 | github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs= 104 | github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro= 105 | github.com/rogpeppe/go-internal v1.6.1 h1:/FiVV8dS/e+YqF2JvO3yXRFbBLTIuSDkuC7aBOAvL+k= 106 | github.com/rs/xid v1.4.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg= 107 | github.com/rs/zerolog v1.29.0 h1:Zes4hju04hjbvkVkOhdl2HpZa+0PmVwigmo8XoORE5w= 108 | github.com/rs/zerolog v1.29.0/go.mod h1:NILgTygv/Uej1ra5XxGf82ZFSLk58MFGAUS2o6usyD0= 109 | github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= 110 | github.com/samber/lo v1.37.0 h1:XjVcB8g6tgUp8rsPsJ2CvhClfImrpL04YpQHXeHPhRw= 111 | github.com/samber/lo v1.37.0/go.mod h1:9vaz2O4o8oOnK23pd2TrXufcbdbJIa3b6cstBWKpopA= 112 | github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE= 113 | github.com/spf13/cast v1.5.0 h1:rj3WzYc11XZaIZMPKmwP96zkFEnnAmV8s6XbB2aY32w= 114 | github.com/spf13/cast v1.5.0/go.mod h1:SpXXQ5YoyJw6s3/6cMTQuxvgRl3PCJiyaX9p6b155UU= 115 | github.com/spf13/cobra v1.6.1 h1:o94oiPyS4KD1mPy2fmcYYHHfCxLqYjJOhGsCHFZtEzA= 116 | github.com/spf13/cobra v1.6.1/go.mod h1:IOw/AERYS7UzyrGinqmz6HLUo219MORXGxhbaJUqzrY= 117 | github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= 118 | github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= 119 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 120 | github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= 121 | github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 122 | github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 123 | github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk= 124 | github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= 125 | github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k= 126 | go.uber.org/goleak v1.1.12 h1:gZAh5/EyT/HQwlpkCy6wTpqfH9H8Lz8zbm3dZh+OyzA= 127 | go.uber.org/goleak v1.1.12/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ= 128 | golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= 129 | golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= 130 | golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= 131 | golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= 132 | golang.org/x/exp v0.0.0-20230304125523-9ff063c70017 h1:3Ea9SZLCB0aRIhSEjM+iaGIlzzeDJdpi579El/YIhEE= 133 | golang.org/x/exp v0.0.0-20230304125523-9ff063c70017/go.mod h1:CxIveKay+FTh1D0yPZemJVgC/95VzuuOLq5Qi4xnoYc= 134 | golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= 135 | golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= 136 | golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 137 | golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= 138 | golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 139 | golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= 140 | golang.org/x/mod v0.10.0 h1:lFO9qtOdlre5W1jxS3r/4szv2/6iXxScdzjoBMXNhYk= 141 | golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 142 | golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 143 | golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 144 | golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= 145 | golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 146 | golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= 147 | golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= 148 | golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A= 149 | golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= 150 | golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= 151 | golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM= 152 | golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M= 153 | golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= 154 | golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= 155 | golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 156 | golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 157 | golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 158 | golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 159 | golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= 160 | golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 161 | golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 162 | golang.org/x/sys v0.0.0-20180926160741-c2ed4eda69e7/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 163 | golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= 164 | golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 165 | golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 166 | golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 167 | golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 168 | golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 169 | golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 170 | golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 171 | golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 172 | golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= 173 | golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 174 | golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 175 | golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 176 | golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 177 | golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 178 | golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU= 179 | golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 180 | golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= 181 | golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= 182 | golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= 183 | golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= 184 | golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= 185 | golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= 186 | golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4= 187 | golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= 188 | golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 189 | golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= 190 | golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= 191 | golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= 192 | golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= 193 | golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= 194 | golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA= 195 | golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk= 196 | golang.org/x/tools v0.9.3 h1:Gn1I8+64MsuTb/HpH+LmQtNas23LhUVr3rYZ0eKuaMM= 197 | golang.org/x/tools v0.9.3/go.mod h1:owI94Op576fPu3cIGQeHs3joujW/2Oc6MtlxbF5dfNc= 198 | golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 199 | golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 200 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 201 | golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 202 | google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= 203 | google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= 204 | google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= 205 | google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= 206 | google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= 207 | google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= 208 | google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= 209 | google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= 210 | google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= 211 | google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= 212 | google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= 213 | google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= 214 | google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= 215 | google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 216 | google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 217 | google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= 218 | google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= 219 | google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= 220 | google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= 221 | google.golang.org/protobuf v1.28.1 h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175w= 222 | google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= 223 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 224 | gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 225 | gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo= 226 | gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 227 | gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys= 228 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ= 229 | gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw= 230 | gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 231 | gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 232 | gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 233 | gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= 234 | gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 235 | honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 236 | honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= 237 | -------------------------------------------------------------------------------- /justfile: -------------------------------------------------------------------------------- 1 | # Run project 2 | run: 3 | docker compose up 4 | 5 | # Stop project 6 | stop: 7 | docker compose down 8 | 9 | # Run tests 10 | test: 11 | ginkgo -p -v ./... 12 | 13 | # Purge generated video assets 14 | clean-video: 15 | rm -f testvideo/*.m3u8 16 | rm -f testvideo/*.ts 17 | 18 | # Purge generated thumbs 19 | clean-thumbs: 20 | rm -rf testvideo/thumbs/**/*.jpg 21 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import "github.com/mauricioabreu/video_samples/cmd" 4 | 5 | func main() { 6 | cmd.Execute() 7 | } 8 | -------------------------------------------------------------------------------- /store/redis.go: -------------------------------------------------------------------------------- 1 | package store 2 | 3 | import "github.com/redis/go-redis/v9" 4 | 5 | func NewRedis(addr string) *redis.Client { 6 | rc := redis.NewClient(&redis.Options{ 7 | Addr: addr, 8 | }) 9 | return rc 10 | } 11 | -------------------------------------------------------------------------------- /tasks/enqueuer.go: -------------------------------------------------------------------------------- 1 | package tasks 2 | 3 | import ( 4 | "fmt" 5 | "time" 6 | 7 | "github.com/hibiken/asynq" 8 | "github.com/mauricioabreu/video_samples/extractor" 9 | "github.com/mauricioabreu/video_samples/extractor/inventory" 10 | "github.com/rs/zerolog/log" 11 | ) 12 | 13 | const runEvery = 30 * time.Second 14 | 15 | func Enqueue(getStreamings func() ([]inventory.Streaming, error), redisAddr string) { 16 | client := asynq.NewClient(asynq.RedisClientOpt{Addr: redisAddr}) 17 | defer client.Close() 18 | 19 | enqueueTasks(getStreamings, client) 20 | timer := time.NewTicker(runEvery) 21 | for range timer.C { 22 | enqueueTasks(getStreamings, client) 23 | } 24 | } 25 | 26 | func enqueueTasks(getStreamings func() ([]inventory.Streaming, error), client *asynq.Client) { 27 | streamings, err := getStreamings() 28 | if err != nil { 29 | log.Error().Err(err).Msg("Failed to get streamings") 30 | return 31 | } 32 | for _, stream := range streamings { 33 | task, err := NewExtractThumbsTask(extractor.ThumbOptions{ 34 | Input: stream.Playlist, 35 | Output: "testvideo/thumbs/", 36 | Scale: "-1:360", 37 | Quality: 5, 38 | }) 39 | if err != nil { 40 | log.Error().Err(err).Msgf("Could not create task: %v", err) 41 | return 42 | } 43 | info, err := client.Enqueue(task, asynq.TaskID(generateID("thumb", stream.Name))) 44 | if err != nil { 45 | log.Error().Err(err).Msgf("Could not enqueue task: %v", err) 46 | return 47 | } 48 | log.Info().Msgf("Enqueued task: id=%s queue=%s", info.ID, info.Queue) 49 | } 50 | } 51 | 52 | func generateID(feature, name string) string { 53 | return fmt.Sprintf("%s_%s", feature, name) 54 | } 55 | -------------------------------------------------------------------------------- /tasks/middleware.go: -------------------------------------------------------------------------------- 1 | package tasks 2 | 3 | import ( 4 | "context" 5 | "time" 6 | 7 | "github.com/hibiken/asynq" 8 | "github.com/rs/zerolog/log" 9 | ) 10 | 11 | func loggingMiddleware(h asynq.Handler) asynq.Handler { 12 | return asynq.HandlerFunc(func(ctx context.Context, t *asynq.Task) error { 13 | start := time.Now() 14 | log.Debug().Msgf("Started processing %s", t.Type()) 15 | if err := h.ProcessTask(ctx, t); err != nil { 16 | log.Error().Err(err).Msgf("Failed to run task: %q", t.Type()) 17 | return err 18 | } 19 | log.Debug().Msgf("Finished processing %s: Elapsed time %v", t.Type(), time.Since(start)) 20 | return nil 21 | }) 22 | } 23 | -------------------------------------------------------------------------------- /tasks/task.go: -------------------------------------------------------------------------------- 1 | package tasks 2 | 3 | import ( 4 | "context" 5 | "encoding/json" 6 | "fmt" 7 | 8 | "github.com/hibiken/asynq" 9 | "github.com/mauricioabreu/video_samples/extractor" 10 | "github.com/rs/zerolog/log" 11 | ) 12 | 13 | const ( 14 | ThumbExtract = "thumbs:extract" 15 | ) 16 | 17 | func NewExtractThumbsTask(to extractor.ThumbOptions) (*asynq.Task, error) { 18 | payload, err := json.Marshal(to) 19 | if err != nil { 20 | return nil, err 21 | } 22 | return asynq.NewTask(ThumbExtract, payload), nil 23 | } 24 | 25 | func HandleThumbsExtractTask(ctx context.Context, t *asynq.Task) error { 26 | var opts extractor.ThumbOptions 27 | if err := json.Unmarshal(t.Payload(), &opts); err != nil { 28 | return fmt.Errorf("failed to process payload: %v %w", err, asynq.SkipRetry) 29 | } 30 | if err := extractor.ExtractThumbs("colors", opts, extractor.RunCmd); err != nil { 31 | return fmt.Errorf("failed to run the extractor: %v", err) 32 | } 33 | log.Info().Msgf("Extracting thumbs from video URL: %s", opts.Input) 34 | return nil 35 | } 36 | -------------------------------------------------------------------------------- /tasks/worker.go: -------------------------------------------------------------------------------- 1 | package tasks 2 | 3 | import ( 4 | "github.com/hibiken/asynq" 5 | "github.com/mauricioabreu/video_samples/config" 6 | "github.com/rs/zerolog/log" 7 | ) 8 | 9 | const maxConcurrency = 10 10 | 11 | func StartWorker(c *config.Config) { 12 | srv := asynq.NewServer( 13 | asynq.RedisClientOpt{Addr: c.RedisAddr}, 14 | asynq.Config{Concurrency: maxConcurrency}, 15 | ) 16 | 17 | mux := asynq.NewServeMux() 18 | mux.HandleFunc(ThumbExtract, HandleThumbsExtractTask) 19 | mux.Use(loggingMiddleware) 20 | 21 | if err := srv.Run(mux); err != nil { 22 | log.Fatal().Err(err).Msgf("Failed to start workers: %s", err) 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /testvideo/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3.9-alpine 2 | 3 | WORKDIR /testvideo 4 | 5 | RUN apk upgrade -U && apk add ffmpeg 6 | 7 | COPY stream.sh OpenSans-Bold.ttf /testvideo/ 8 | 9 | RUN chmod +x /testvideo/stream.sh 10 | -------------------------------------------------------------------------------- /testvideo/OpenSans-Bold.ttf: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mauricioabreu/video_samples/726d631f9f76eded0f694b71d41f1072e5267eea/testvideo/OpenSans-Bold.ttf -------------------------------------------------------------------------------- /testvideo/stream.sh: -------------------------------------------------------------------------------- 1 | ffmpeg -loglevel error -re -f lavfi -i "testsrc=size=1280x720:rate=30" \ 2 | -pix_fmt yuv420p \ 3 | -c:v libx264 -x264opts keyint=30:min-keyint=30:scenecut=-1 \ 4 | -tune zerolatency -profile:v high -preset veryfast -bf 0 -refs 3 \ 5 | -b:v 1400k -bufsize 1400k \ 6 | -vf "drawtext=fontfile='/testvideo/OpenSans-Bold.ttf':text='%{localtime}:box=1:fontcolor=black:boxcolor=white:fontsize=100':x=40:y=400'" \ 7 | -hls_time 5 -hls_list_size 10 -hls_flags delete_segments -f hls /testvideo/playlist.m3u8 & 8 | 9 | python3 -m http.server -d /testvideo 9090 10 | -------------------------------------------------------------------------------- /testvideo/streams.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "name": "colors", 4 | "playlist": "http://stream:9090/playlist.m3u8" 5 | } 6 | ] 7 | --------------------------------------------------------------------------------