├── .gitignore ├── AppDockerfile ├── LICENSE ├── README.md ├── ServerDockerfile ├── app ├── app └── main.go ├── crawler ├── common.go ├── crawler.go ├── parser.go ├── task.go ├── trending_developer.go ├── trending_repo.go ├── work.go ├── work_pool.go └── work_result.go ├── database └── Redis.go ├── docker-compose.yml ├── images ├── 1.png ├── 2.png ├── 3.png ├── 4.png └── 5.png ├── server ├── main.go └── server └── vendor ├── github.com ├── PuerkitoBio │ └── goquery │ │ ├── LICENSE │ │ ├── README.md │ │ ├── array.go │ │ ├── doc.go │ │ ├── expand.go │ │ ├── filter.go │ │ ├── iteration.go │ │ ├── manipulation.go │ │ ├── property.go │ │ ├── query.go │ │ ├── traversal.go │ │ ├── type.go │ │ └── utilities.go ├── andybalholm │ └── cascadia │ │ ├── LICENSE │ │ ├── README.md │ │ ├── go.mod │ │ ├── parser.go │ │ └── selector.go └── go-redis │ └── redis │ ├── LICENSE │ ├── Makefile │ ├── README.md │ ├── cluster.go │ ├── cluster_commands.go │ ├── command.go │ ├── commands.go │ ├── doc.go │ ├── internal │ ├── consistenthash │ │ └── consistenthash.go │ ├── error.go │ ├── hashtag │ │ └── hashtag.go │ ├── internal.go │ ├── log.go │ ├── once.go │ ├── pool │ │ ├── conn.go │ │ ├── pool.go │ │ ├── pool_single.go │ │ └── pool_sticky.go │ ├── proto │ │ ├── reader.go │ │ ├── scan.go │ │ └── write_buffer.go │ ├── singleflight │ │ └── singleflight.go │ ├── util.go │ └── util │ │ ├── safe.go │ │ ├── strconv.go │ │ └── unsafe.go │ ├── iterator.go │ ├── options.go │ ├── parser.go │ ├── pipeline.go │ ├── pubsub.go │ ├── redis.go │ ├── result.go │ ├── ring.go │ ├── script.go │ ├── sentinel.go │ ├── tx.go │ └── universal.go ├── golang.org └── x │ └── net │ ├── LICENSE │ ├── PATENTS │ └── html │ ├── atom │ ├── atom.go │ └── table.go │ ├── const.go │ ├── doc.go │ ├── doctype.go │ ├── entity.go │ ├── escape.go │ ├── foreign.go │ ├── node.go │ ├── parse.go │ ├── render.go │ └── token.go └── vendor.json /.gitignore: -------------------------------------------------------------------------------- 1 | # Binaries for programs and plugins 2 | *.exe 3 | *.dll 4 | *.so 5 | *.dylib 6 | 7 | # Test binary, build with `go test -c` 8 | *.test 9 | 10 | # Output of the go coverage tool, specifically when used with LiteIDE 11 | *.out 12 | 13 | # Project-local glide cache, RE: https://github.com/Masterminds/glide/issues/736 14 | .glide/ 15 | -------------------------------------------------------------------------------- /AppDockerfile: -------------------------------------------------------------------------------- 1 | FROM golang:1.10-alpine 2 | MAINTAINER king129@vip.163.com 3 | 4 | WORKDIR /go/src 5 | 6 | COPY ./crawler ./github-crawler/crawler 7 | COPY ./database ./github-crawler/database 8 | COPY ./app ./github-crawler/app 9 | COPY ./vendor ./github-crawler/vendor 10 | 11 | RUN cd /go/src/github-crawler/app && \ 12 | go install . && \ 13 | which app 14 | 15 | CMD ["app"] -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 king 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # github-crawler 2 | 3 | * 爬取`Github` `Trending`数据,数据存储到`Redis`,在运行之前,请先设置 环境变量 `RedisHost` `RedisPassword`,如果不进行设置,则默认连接到 `127.0.0.1:6379`,`Redis`数据过期时间为2小时的基础上在加1小时的随机数,避免同一时刻大部分数据缓存过期 4 | - 如何运行 5 | * 首先下载本仓库,如果有安装`Go` 环境,则存放 到`GOPATH/src`目录下 6 | * 源码运行 7 | 8 | ```shell 9 | git clone https://github.com/king129/github-crawler.git $GOPATH/src/github-crawler 10 | cd $GOPATH/src/github-crawler 11 | go run app/main.go 12 | ``` 13 | 14 | * 可执行文件运行 (注意: 可执行文件只能在 macOS 平台运行) 15 | 16 | ```shell 17 | // 每五分钟分钟执行60次任务 任务为 每个语言的 repo/developer daily weekly monthly 的数据 18 | cd xxx/github-crawler/app 19 | chmod +x app 20 | ./app 21 | 22 | cd xxx/github-crawler/server 23 | chmod +x server 24 | ./server 25 | ``` 26 | 27 | * Docker 运行 需要安装 docker-compose 28 | 29 | ```shell 30 | cd xxx/github-crawler 31 | // 构建镜像 32 | docker-compose build 33 | // 启动相关镜像 可以添加 -d 参数 后台运行 34 | docker-compose up 35 | ``` 36 | 37 | 38 | * 接口列表 39 | * `/language` 获取所有语言数据 40 | * `/repo` 获取 `Trending Repo`数据 参数`lan` 语言,不传默认为所有语言,`since`不传默认为`daily` 41 | * `/repo` 获取 `Trending Developer`数据 参数`lan` 语言,不传默认为所有语言,`since`不传默认为`daily` 42 | * `lan`参数值需要将语言`/language`接口返回语言将空格替换为`-`,并转为小写 43 | * `since` 参数值有`daily`, `weekly`, `monthly` 44 | 45 | ![](https://github.com/king129/github-crawler/blob/master/images/1.png) 46 | ![](https://github.com/king129/github-crawler/blob/master/images/2.png) 47 | ![](https://github.com/king129/github-crawler/blob/master/images/3.png) 48 | ![](https://github.com/king129/github-crawler/blob/master/images/4.png) 49 | ![](https://github.com/king129/github-crawler/blob/master/images/5.png) 50 | 51 | -------------------------------------------------------------------------------- /ServerDockerfile: -------------------------------------------------------------------------------- 1 | FROM golang:1.10-alpine 2 | MAINTAINER king129@vip.163.com 3 | 4 | WORKDIR /go/src 5 | 6 | COPY ./crawler ./github-crawler/crawler 7 | COPY ./database ./github-crawler/database 8 | COPY ./server ./github-crawler/server 9 | COPY ./vendor ./github-crawler/vendor 10 | 11 | RUN cd /go/src/github-crawler/server && \ 12 | go install . && \ 13 | which server 14 | 15 | CMD ["server"] -------------------------------------------------------------------------------- /app/app: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/0x1306a94/github-crawler/cb4bbaf5d68fb066976ac2d069b509c618d5b23e/app/app -------------------------------------------------------------------------------- /app/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "github-crawler/crawler" 5 | "github-crawler/database" 6 | "fmt" 7 | "encoding/json" 8 | "strings" 9 | "time" 10 | "os" 11 | "math/rand" 12 | ) 13 | 14 | var redisClient *database.Redis 15 | 16 | 17 | func main() { 18 | 19 | redisHost := os.Getenv("RedisHost") 20 | redisPassword := os.Getenv("RedisPassword") 21 | if redisHost == "" { 22 | redisHost = "127.0.0.1:6379" 23 | } 24 | redisClient = database.NewRedis(redisHost, redisPassword, 0) 25 | 26 | if err := redisClient.Connection(); err != nil { 27 | fmt.Println("Redis connect error: ", err) 28 | return 29 | } 30 | 31 | defer redisClient.Disconnect() 32 | 33 | cacheLanguages := redisClient.GetCacheLanguages() 34 | if len(cacheLanguages) > 0 { 35 | fmt.Println("cacheLanguages: \n", cacheLanguages) 36 | } 37 | app := crawler.NewCrawler(cacheLanguages) 38 | if err := app.Start(); err != nil { 39 | fmt.Println("crawler start error: ", err) 40 | return 41 | } 42 | go redisClient.SaveLanguages(app.AllLanguages()) 43 | for result := range app.Result() { 44 | if result.Result != nil && result.ResultType != crawler.ResultTypeLanguage { 45 | go SaveRedis(result) 46 | } 47 | } 48 | } 49 | 50 | 51 | func SaveRedis(result crawler.WorkResult) { 52 | 53 | lan := strings.ToLower(strings.Replace(result.Language, " ", "-", -1)) 54 | key := fmt.Sprintf("trending-%v-%v-%v", result.ResultType,lan, result.Since) 55 | data, err := json.Marshal(result.Result) 56 | if err != nil { 57 | return 58 | } 59 | if len(data) == 0 { 60 | return 61 | } 62 | expiration := time.Hour * 2 63 | random := rand.Intn(3600) 64 | err = redisClient.Save(key, string(data), expiration + time.Duration(random)) 65 | if err == nil { 66 | fmt.Printf("%v save to redis success\n", key) 67 | } else { 68 | fmt.Printf("%v save to redis failure: %v\n", key, err.Error()) 69 | } 70 | } 71 | 72 | -------------------------------------------------------------------------------- /crawler/common.go: -------------------------------------------------------------------------------- 1 | package crawler 2 | 3 | const ( 4 | Host = "https://github.com" 5 | TrendingRepoPath = "https://github.com/trending" 6 | TrendingDevelopersPath = "https://github.com/trending/developers" 7 | ) 8 | 9 | const ( 10 | AllLanguage = "All Language" 11 | ) 12 | 13 | type TaskType string 14 | const ( 15 | TaskTypeLanguage = "language" 16 | TaskTypeRepo = "repo" 17 | TaskTypeDeveloper = "developer" 18 | ) 19 | 20 | type ResultType string 21 | const ( 22 | ResultTypeLanguage = "language" 23 | ResultTypeRepo = "repo" 24 | ResultTypeDeveloper = "developer" 25 | ) 26 | 27 | const ( 28 | SinceToDay = "daily" 29 | SinceWeek = "weekly" 30 | SinceMonth = "monthly" 31 | ) -------------------------------------------------------------------------------- /crawler/crawler.go: -------------------------------------------------------------------------------- 1 | package crawler 2 | 3 | import ( 4 | "errors" 5 | "time" 6 | "fmt" 7 | ) 8 | 9 | type Crawler struct { 10 | languages []string 11 | pool *workPool 12 | ticker *time.Ticker 13 | } 14 | 15 | func NewCrawler(languages []string) *Crawler { 16 | return &Crawler{ 17 | languages: languages, 18 | } 19 | } 20 | 21 | func (c *Crawler) Result() <-chan WorkResult { 22 | if c.pool == nil { 23 | return nil 24 | } 25 | return c.pool.resultQueue 26 | } 27 | 28 | func (c *Crawler) AllLanguages() []string { 29 | return c.languages 30 | } 31 | 32 | func (c *Crawler) Start() (error) { 33 | if c.languages == nil || len(c.languages) == 0 { 34 | languages, err := c.fetchLanguage() 35 | if err != nil { 36 | return err 37 | } 38 | c.languages = languages 39 | c.languages = []string{AllLanguage} 40 | c.languages = append(c.languages, languages...) 41 | } 42 | 43 | c.pool = newWorkPool(10) 44 | fmt.Println("start crawler ....") 45 | c.ticker = time.NewTicker(time.Minute * 5) 46 | go c.pool.run() 47 | go func(crawler *Crawler) { 48 | startOffset := 0 49 | sectionSize := 10 50 | for range crawler.ticker.C { 51 | chunckLans := make([]string, 0) 52 | if startOffset + sectionSize >= len(crawler.languages) { 53 | chunckLans = crawler.languages[startOffset:] 54 | startOffset = 0 55 | } else { 56 | chunckLans = crawler.languages[startOffset:(startOffset + sectionSize)] 57 | startOffset = startOffset + sectionSize 58 | } 59 | for _, lan := range chunckLans { 60 | for _, since := range []string{SinceToDay, SinceWeek, SinceMonth} { 61 | reposT := &repoTask{} 62 | reposT.language = lan 63 | reposT.since = since 64 | reposT.retry = 3 65 | 66 | developerT := &developerTask{} 67 | developerT.language = lan 68 | developerT.since = since 69 | developerT.retry = 3 70 | 71 | crawler.pool.addTask(reposT) 72 | crawler.pool.addTask(developerT) 73 | } 74 | } 75 | } 76 | 77 | }(c) 78 | 79 | return nil 80 | } 81 | func (c *Crawler) Stop() { 82 | c.ticker.Stop() 83 | c.pool.stop() 84 | } 85 | 86 | func (c *Crawler) fetchLanguage() ([]string, error) { 87 | w := NewWork() 88 | go func(w *work) { 89 | w.start() 90 | }(w) 91 | 92 | go func(w *work) { 93 | w.jobQueue <- &languageTask{ 94 | retry: 3, 95 | } 96 | }(w) 97 | 98 | var result []string = nil 99 | for r := range w.resultQueue { 100 | if v, ok := r.Result.([]string); ok { 101 | result = v 102 | } 103 | go func(w *work) { 104 | w.stop() 105 | }(w) 106 | } 107 | if result == nil || len(result) == 0 { 108 | return nil, errors.New("fetchLanguage error") 109 | } 110 | return result, nil 111 | } -------------------------------------------------------------------------------- /crawler/parser.go: -------------------------------------------------------------------------------- 1 | package crawler 2 | 3 | import ( 4 | "github.com/PuerkitoBio/goquery" 5 | "net/http" 6 | "fmt" 7 | "bufio" 8 | "strings" 9 | "errors" 10 | ) 11 | 12 | func fetchResponseFrom(url string) (*http.Response, error) { 13 | resp, err := http.Get(url) 14 | fmt.Println("fetch url: ", url) 15 | if err != nil { 16 | return nil, err 17 | } 18 | if resp.StatusCode != http.StatusOK { 19 | return nil, errors.New("response statusCode not equal to 200 OK") 20 | } 21 | return resp, nil 22 | } 23 | 24 | func parserFromAllLanguage() []string { 25 | resp, err := fetchResponseFrom("https://github.com/trending") 26 | if err != nil { 27 | fmt.Println("fetchResponse error: ", err) 28 | return nil 29 | } 30 | defer resp.Body.Close() 31 | doc, err := goquery.NewDocumentFromReader(bufio.NewReader(resp.Body)) 32 | if err != nil { 33 | fmt.Println("NewDocumentFromReader error: ", err) 34 | return nil 35 | } 36 | 37 | languageSelection := doc. 38 | Find("div[class=select-menu-list]"). 39 | Find("a[role=menuitem]") 40 | 41 | if languageSelection == nil || len(languageSelection.Nodes) == 0 { 42 | return nil 43 | } 44 | languages := languageSelection.Map(func(i int, selection *goquery.Selection) string { 45 | return strings.TrimSpace(selection.Text()) 46 | }) 47 | 48 | if languages == nil || len(languages) == 0 { 49 | return nil 50 | } 51 | return languages 52 | } 53 | 54 | func parserFromRepos(language, since string) *TrendingRepoResult { 55 | 56 | url := TrendingRepoPath 57 | if language != AllLanguage { 58 | url = url + "/" + strings.ToLower(strings.Replace(language, " ", "-", -1)) 59 | } else { 60 | url = url + "/all" 61 | } 62 | 63 | if since != "" { 64 | url = url + "?since=" + since 65 | } 66 | resp, err := fetchResponseFrom(url) 67 | if err != nil { 68 | fmt.Println("fetchResponse error: ", err) 69 | return nil 70 | } 71 | defer resp.Body.Close() 72 | doc, err := goquery.NewDocumentFromReader(bufio.NewReader(resp.Body)) 73 | if err != nil { 74 | fmt.Println("NewDocumentFromReader error: ", err) 75 | return nil 76 | } 77 | listSelection := doc.Find("ol[class=repo-list]").First().Children() 78 | if listSelection == nil || len(listSelection.Nodes) == 0 { 79 | return nil 80 | } 81 | mapObjectFunc := func(i int, selection *goquery.Selection) interface{} { 82 | repo := new(TrendingRepo) 83 | fullName := selection.Find("h3").First().Text() 84 | description := selection.Find("div[class=py-1]").Find(".col-9.d-inline-block.text-gray.m-0.pr-4").First().Text() 85 | language := selection.Find("span[itemprop=programmingLanguage]").First().Text() 86 | stars := selection.Find(".f6.text-gray.mt-2").Find(".muted-link.d-inline-block.mr-3").First().Text() 87 | forkers := selection.Find(".f6.text-gray.mt-2").Find(".muted-link.d-inline-block.mr-3").Eq(1).Text() 88 | gains := selection.Find(".f6.text-gray.mt-2").Find(".d-inline-block.float-sm-right").First().Text() 89 | 90 | repo.FullName = strings.TrimSpace(strings.Replace(fullName, " ", "", -1)) 91 | repo.Description = strings.TrimSpace(description) 92 | repo.Language = strings.TrimSpace(language) 93 | repo.Stars = strings.TrimSpace(strings.Replace(stars, ",", "", -1)) 94 | repo.Forkers = strings.TrimSpace(strings.Replace(forkers, ",", "", -1)) 95 | repo.Gains = strings.TrimSpace(strings.Replace(gains, ",", "", -1)) 96 | return repo 97 | } 98 | result := new(TrendingRepoResult) 99 | Ranking := 1 100 | listSelection.Each(func(i int, selection *goquery.Selection) { 101 | if obj := mapObjectFunc(i, selection); obj != nil { 102 | obj.(*TrendingRepo).Ranking = Ranking 103 | result.Repos = append(result.Repos, obj.(*TrendingRepo)) 104 | Ranking++ 105 | } 106 | }) 107 | if len(result.Repos) == 0 { 108 | return nil 109 | } 110 | result.Language = language 111 | result.Since = since 112 | return result 113 | } 114 | 115 | 116 | func parserFromDevelopers(language, since string) *TrendingDeveloperResult { 117 | 118 | url := TrendingDevelopersPath 119 | if language != AllLanguage { 120 | url = url + "/" + strings.ToLower(strings.Replace(language, " ", "-", -1)) 121 | } else { 122 | url = url + "/all" 123 | } 124 | 125 | if since != "" { 126 | url = url + "?since=" + since 127 | } 128 | resp, err := fetchResponseFrom(url) 129 | if err != nil { 130 | fmt.Println("fetchResponse error: ", err) 131 | return nil 132 | } 133 | defer resp.Body.Close() 134 | doc, err := goquery.NewDocumentFromReader(bufio.NewReader(resp.Body)) 135 | if err != nil { 136 | fmt.Println("NewDocumentFromReader error: ", err) 137 | return nil 138 | } 139 | listSelection := doc.Find("ol[class=list-style-none]").First().Children() 140 | if listSelection == nil || len(listSelection.Nodes) == 0 { 141 | return nil 142 | } 143 | mapObjectFunc := func(i int, selection *goquery.Selection) interface{} { 144 | developer := new(TrendingDeveloper) 145 | name := selection.Find("div[class=mx-2]").Find(".f3.text-normal").Text() 146 | name = strings.TrimSpace(name) 147 | name = strings.Replace(name, " ", "", -1) 148 | name = strings.Replace(name, "\n", "", -1) 149 | name = strings.Replace(name, "(", ",", -1) 150 | name = strings.Replace(name, ")", "", -1) 151 | names := strings.Split(name, ",") 152 | if len(names) == 2 { 153 | developer.Login = names[0] 154 | developer.NickName = names[1] 155 | } else { 156 | return nil 157 | } 158 | 159 | if avatar, ok := selection.Find("div[class=mx-2]").Find("img[class=rounded-1]").Attr("src"); ok { 160 | developer.Avatar = avatar 161 | } 162 | 163 | repoName := selection.Find("div[class=mx-2]").Find("a").Find("span[class=repo]").Text() 164 | description := selection.Find("div[class=mx-2]").Find("a").Find(".repo-snipit-description.css-truncate-target").Text() 165 | 166 | developer.RepoName = strings.TrimSpace(repoName) 167 | developer.RepoDescription = strings.TrimSpace(description) 168 | return developer 169 | } 170 | result := new(TrendingDeveloperResult) 171 | Ranking := 1 172 | listSelection.Each(func(i int, selection *goquery.Selection) { 173 | if obj := mapObjectFunc(i, selection); obj != nil { 174 | obj.(*TrendingDeveloper).Ranking = Ranking 175 | result.Developers = append(result.Developers, obj.(*TrendingDeveloper)) 176 | Ranking++ 177 | } 178 | }) 179 | if len(result.Developers) == 0 { 180 | return nil 181 | } 182 | result.Language = language 183 | result.Since = since 184 | return result 185 | } 186 | -------------------------------------------------------------------------------- /crawler/task.go: -------------------------------------------------------------------------------- 1 | package crawler 2 | 3 | type task interface { 4 | taskType() TaskType 5 | call() interface{} 6 | retryCount() int 7 | sinceDesc() string 8 | languageDesc() string 9 | } 10 | 11 | 12 | type commonTask struct { 13 | language string 14 | since string 15 | retry int 16 | } 17 | 18 | type repoTask struct { 19 | commonTask 20 | } 21 | 22 | type developerTask struct { 23 | commonTask 24 | } 25 | 26 | type languageTask struct { 27 | retry int 28 | } 29 | 30 | func (r *repoTask) sinceDesc() string { 31 | return r.since 32 | } 33 | 34 | func (r *repoTask) languageDesc() string { 35 | return r.language 36 | } 37 | 38 | func (_ *repoTask) taskType() TaskType { 39 | return TaskTypeRepo 40 | } 41 | 42 | func (r *repoTask) call() (interface{}) { 43 | return parserFromRepos(r.language, r.since) 44 | } 45 | 46 | func (r *repoTask) retryCount() int { 47 | return r.retry 48 | } 49 | 50 | func (d *developerTask) sinceDesc() string { 51 | return d.since 52 | } 53 | 54 | func (d *developerTask) languageDesc() string { 55 | return d.language 56 | } 57 | 58 | func (_ *developerTask) taskType() TaskType { 59 | return TaskTypeDeveloper 60 | } 61 | 62 | func (r *developerTask) call() (interface{}) { 63 | return parserFromDevelopers(r.language, r.since) 64 | } 65 | 66 | func (r *developerTask) retryCount() int { 67 | return r.retry 68 | } 69 | 70 | func (_ *languageTask) sinceDesc() string { 71 | return "" 72 | } 73 | 74 | func (_ *languageTask) languageDesc() string { 75 | return "" 76 | } 77 | 78 | func (_ *languageTask) taskType() TaskType { 79 | return TaskTypeLanguage 80 | } 81 | 82 | func (_ *languageTask) call() (interface{}) { 83 | return parserFromAllLanguage() 84 | } 85 | 86 | func (l *languageTask) retryCount() int { 87 | return l.retry 88 | } -------------------------------------------------------------------------------- /crawler/trending_developer.go: -------------------------------------------------------------------------------- 1 | package crawler 2 | 3 | import "fmt" 4 | 5 | type TrendingDeveloperResult struct { 6 | Language string `json:"language"` 7 | Since string `json:"since"` 8 | Developers []*TrendingDeveloper `json:"developers"` 9 | } 10 | 11 | type TrendingDeveloper struct { 12 | Ranking int `json:"ranking"` 13 | Avatar string `json:"avatar"` 14 | Login string `json:"login"` 15 | NickName string `json:"nickName"` 16 | RepoName string `json:"repoName"` 17 | RepoDescription string `json:"repoDescription"` 18 | } 19 | 20 | func (developer *TrendingDeveloper) String() string { 21 | return fmt.Sprintf("\nRanking: %d\nLogin: %v\nNickName: %v\nAvatar: %v\nRepoName: %v\nRepoDescription: %v\n", 22 | developer.Ranking,developer.Login, developer.NickName, developer.Avatar, developer.RepoName, developer.RepoDescription) 23 | } -------------------------------------------------------------------------------- /crawler/trending_repo.go: -------------------------------------------------------------------------------- 1 | package crawler 2 | 3 | import "fmt" 4 | 5 | type TrendingRepoResult struct { 6 | Language string `json:"language"` 7 | Since string `json:"since"` 8 | Repos []*TrendingRepo `json:"repos"` 9 | } 10 | 11 | type TrendingRepo struct{ 12 | Ranking int `json:"ranking"` 13 | FullName string `json:"fullName"` 14 | Language string `json:"language"` 15 | Description string `json:"repoDescription"` 16 | Stars string `json:"stars"` 17 | Forkers string `json:"forkers"` 18 | Gains string `json:"gains"` 19 | } 20 | 21 | func (repo *TrendingRepo) String() string { 22 | return fmt.Sprintf("\nRanking: %d\nFullName: %v\nLanguage: %v\nDescription: %v\nStars: %v\nForkers: %v\nGains: %v\n", 23 | repo.Ranking,repo.FullName, repo.Language, repo.Description, repo.Stars, repo.Forkers, repo.Gains) 24 | } -------------------------------------------------------------------------------- /crawler/work.go: -------------------------------------------------------------------------------- 1 | package crawler 2 | 3 | import "fmt" 4 | 5 | type work struct { 6 | id int 7 | jobQueue chan task 8 | resultQueue chan WorkResult 9 | } 10 | 11 | func NewWork() *work { 12 | return &work{ 13 | jobQueue: make(chan task, 50), 14 | resultQueue: make(chan WorkResult, 100), 15 | } 16 | } 17 | 18 | func (w *work) start() { 19 | fmt.Printf("work id: %d start\n", w.id) 20 | for task := range w.jobQueue { 21 | retryCount := task.retryCount() 22 | des := "" 23 | var resultType ResultType 24 | switch task.taskType() { 25 | case TaskTypeLanguage: 26 | des = "fetch language" 27 | resultType = ResultTypeLanguage 28 | case TaskTypeRepo: 29 | des = "fetch repo" 30 | resultType = ResultTypeRepo 31 | default: 32 | des = "fetch developer" 33 | resultType = ResultTypeDeveloper 34 | } 35 | 36 | if retryCount <= 0 { 37 | retryCount = 1 38 | } 39 | var result interface{} = nil 40 | for i := 0; i < retryCount; i++ { 41 | fmt.Printf("work id: %d exec: %v\n", w.id, des) 42 | r := task.call() 43 | switch task.taskType() { 44 | case TaskTypeLanguage: 45 | if r != nil && len(r.([]string)) > 0 { 46 | result = r 47 | } 48 | default: 49 | if r != nil { 50 | result = r 51 | } 52 | } 53 | if result != nil { 54 | break 55 | } 56 | } 57 | r := WorkResult{ 58 | ResultType: resultType, 59 | Since: task.sinceDesc(), 60 | Language: task.languageDesc(), 61 | Result: result, 62 | } 63 | w.resultQueue <- r 64 | } 65 | fmt.Printf("work id: %d stop\n",w.id) 66 | } 67 | 68 | func (w *work) stop() { 69 | close(w.jobQueue) 70 | close(w.resultQueue) 71 | } -------------------------------------------------------------------------------- /crawler/work_pool.go: -------------------------------------------------------------------------------- 1 | package crawler 2 | 3 | import "math/rand" 4 | 5 | type workPool struct { 6 | pool []*work 7 | resultQueue chan WorkResult 8 | } 9 | 10 | func newWorkPool(poolSize int) *workPool { 11 | p := &workPool{ 12 | pool: make([]*work, 0), 13 | resultQueue: make(chan WorkResult, 100), 14 | } 15 | for i := 0; i < poolSize; i++ { 16 | w := NewWork() 17 | w.id = i + 1 18 | p.pool = append(p.pool, w) 19 | } 20 | return p 21 | } 22 | 23 | func (p *workPool) poolSzie() int { 24 | return len(p.pool) 25 | } 26 | 27 | func (p *workPool) addTask(t task) { 28 | idx := rand.Intn(p.poolSzie()) 29 | work := p.pool[idx] 30 | work.jobQueue <- t 31 | } 32 | 33 | func (p *workPool) run() *workPool { 34 | for _, w := range p.pool { 35 | go w.start() 36 | go func(wp *workPool, w *work) { 37 | for result := range w.resultQueue { 38 | p.resultQueue <- result 39 | } 40 | }(p, w) 41 | } 42 | return p 43 | } 44 | 45 | func (p *workPool) stop() *workPool { 46 | for _, w := range p.pool { 47 | go w.stop() 48 | } 49 | close(p.resultQueue) 50 | return p 51 | } -------------------------------------------------------------------------------- /crawler/work_result.go: -------------------------------------------------------------------------------- 1 | package crawler 2 | 3 | type WorkResult struct { 4 | ResultType ResultType 5 | Since string 6 | Language string 7 | Result interface{} 8 | } 9 | -------------------------------------------------------------------------------- /database/Redis.go: -------------------------------------------------------------------------------- 1 | package database 2 | 3 | import ( 4 | "github.com/go-redis/redis" 5 | "time" 6 | "errors" 7 | "fmt" 8 | "encoding/json" 9 | ) 10 | 11 | type Redis struct { 12 | client *redis.Client 13 | Addr string 14 | Passwprd string 15 | DB int 16 | } 17 | 18 | func NewRedis(addr, password string, db int) *Redis { 19 | return &Redis{ 20 | Addr: addr, 21 | Passwprd: password, 22 | DB: db, 23 | } 24 | } 25 | 26 | func (r *Redis) Connection() (err error) { 27 | r.client = redis.NewClient(&redis.Options{ 28 | Addr: r.Addr, 29 | Password: r.Passwprd, 30 | DB: r.DB, 31 | }) 32 | 33 | retryCount := 10 34 | count := 1 35 | fmt.Println("Redis start Ping") 36 | for { 37 | _, err = r.client.Ping().Result() 38 | if err != nil && count > retryCount { 39 | r.client.Close() 40 | return 41 | } else if err != nil { 42 | fmt.Println("Redis Ping error: ", err) 43 | } else { 44 | break 45 | } 46 | count++ 47 | time.Sleep(time.Second * 5) 48 | } 49 | return 50 | } 51 | func (r *Redis) Disconnect() { 52 | if r.client != nil { 53 | r.client.Close() 54 | } 55 | } 56 | func (r *Redis) Save(key string, obj string, expiration time.Duration) (err error) { 57 | 58 | err = r.client.Set(key, obj, expiration).Err() 59 | if err != nil { 60 | return 61 | } 62 | return 63 | } 64 | 65 | func (r *Redis) Get(key string) (result string, err error) { 66 | 67 | result, err = r.client.Get(key).Result() 68 | if err == redis.Nil { 69 | err = errors.New(fmt.Sprintf("key %v does not exists", key)) 70 | return 71 | } else if err != nil { 72 | return 73 | } 74 | return 75 | } 76 | 77 | func (r *Redis) GetCacheLanguages() []string { 78 | 79 | result, err := r.Get("all-language") 80 | if err != nil { 81 | return nil 82 | } 83 | 84 | var lans []string 85 | err = json.Unmarshal([]byte(result), &lans) 86 | if err != nil { 87 | return nil 88 | } 89 | return lans 90 | } 91 | 92 | func (r *Redis) SaveLanguages(languages []string) { 93 | if len(languages) == 0 { 94 | return 95 | } 96 | data, err := json.Marshal(languages) 97 | if err != nil { 98 | return 99 | } 100 | if len(data) == 0 { 101 | return 102 | } 103 | err = r.Save("all-language", string(data), 0) 104 | if err == nil { 105 | fmt.Println("all-language save to redis success") 106 | } else { 107 | fmt.Println("all-language save to redis failure: ", err) 108 | } 109 | } -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3.0' 2 | 3 | services: 4 | github-trendig-crawler: 5 | build: 6 | context: ./ 7 | dockerfile: AppDockerfile 8 | container_name: github-trendig-crawler 9 | restart: "always" 10 | environment: 11 | RedisHost: "redis:6379" 12 | depends_on: 13 | - redis 14 | 15 | github-trendig-server: 16 | build: 17 | context: ./ 18 | dockerfile: ServerDockerfile 19 | container_name: github-trendig-server 20 | restart: "always" 21 | environment: 22 | RedisHost: "redis:6379" 23 | depends_on: 24 | - redis 25 | ports: 26 | - "8080:8080" 27 | 28 | redis: 29 | image: "redis:alpine" 30 | container_name: github-trendig-redis 31 | restart: "always" 32 | ports: 33 | - "6379:6379" 34 | -------------------------------------------------------------------------------- /images/1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/0x1306a94/github-crawler/cb4bbaf5d68fb066976ac2d069b509c618d5b23e/images/1.png -------------------------------------------------------------------------------- /images/2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/0x1306a94/github-crawler/cb4bbaf5d68fb066976ac2d069b509c618d5b23e/images/2.png -------------------------------------------------------------------------------- /images/3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/0x1306a94/github-crawler/cb4bbaf5d68fb066976ac2d069b509c618d5b23e/images/3.png -------------------------------------------------------------------------------- /images/4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/0x1306a94/github-crawler/cb4bbaf5d68fb066976ac2d069b509c618d5b23e/images/4.png -------------------------------------------------------------------------------- /images/5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/0x1306a94/github-crawler/cb4bbaf5d68fb066976ac2d069b509c618d5b23e/images/5.png -------------------------------------------------------------------------------- /server/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "github-crawler/database" 5 | "fmt" 6 | "net/http" 7 | "encoding/json" 8 | "os" 9 | ) 10 | 11 | var redisClient *database.Redis 12 | 13 | type Response struct { 14 | State int `json:"state"` 15 | ErrorMsg string `json:"error_msg"` 16 | Result interface{} `json:"result"` 17 | } 18 | 19 | type Server struct { 20 | 21 | } 22 | 23 | func main() { 24 | 25 | redisHost := os.Getenv("RedisHost") 26 | redisPassword := os.Getenv("RedisPassword") 27 | if redisHost == "" { 28 | redisHost = "127.0.0.1:6379" 29 | } 30 | redisClient = database.NewRedis(redisHost, redisPassword, 0) 31 | 32 | if err := redisClient.Connection(); err != nil { 33 | fmt.Println("Redis connect error: ", err) 34 | return 35 | } 36 | 37 | defer redisClient.Disconnect() 38 | 39 | fmt.Println("start server http://localhost:8080 ") 40 | err := http.ListenAndServe(":8080", &Server{}) 41 | if err != nil { 42 | fmt.Println("start server error: ", err) 43 | } 44 | } 45 | 46 | func (_ *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) { 47 | if r.Method != http.MethodGet { 48 | w.WriteHeader(http.StatusMethodNotAllowed) 49 | return 50 | } 51 | r.ParseForm() 52 | switch r.URL.Path { 53 | case "/language", "/language/": 54 | AllLanguageHandler(w, r) 55 | case "/repo", "/repo/": 56 | TrendingRepoHandler(w, r) 57 | case "/developer", "/developer/": 58 | TrendingDeveloperHandler(w, r) 59 | case "/": 60 | response := Response{ 61 | State: 1, 62 | Result: map[string]string { 63 | "language" : "/language", 64 | "repo" : "/repo", 65 | "developer" : "/developer", 66 | }, 67 | } 68 | WriteResponse(w, response) 69 | default: 70 | w.WriteHeader(http.StatusNotFound) 71 | } 72 | } 73 | 74 | func AllLanguageHandler(w http.ResponseWriter, r *http.Request) { 75 | cacheLanguages := redisClient.GetCacheLanguages() 76 | response := Response{ 77 | State: 1, 78 | } 79 | if len(cacheLanguages) >= 0 { 80 | response.Result = cacheLanguages 81 | } 82 | WriteResponse(w, response) 83 | } 84 | 85 | func TrendingRepoHandler(w http.ResponseWriter, r *http.Request) { 86 | lan := r.Form.Get("lan") 87 | since := r.Form.Get("since") 88 | if lan == "all-language" || lan == "" { 89 | lan = "all-language" 90 | } 91 | if since == "" { 92 | since = "daily" 93 | } 94 | 95 | key := fmt.Sprintf("trending-repo-%v-%v", lan, since) 96 | str, err := redisClient.Get(key) 97 | response := Response{ 98 | State: 1, 99 | } 100 | if err != nil { 101 | fmt.Println("repo error: ", err) 102 | response.ErrorMsg = err.Error() 103 | } else { 104 | var val interface{} 105 | err = json.Unmarshal([]byte(str), &val) 106 | if err != nil { 107 | response.ErrorMsg = err.Error() 108 | } else { 109 | response.Result = val 110 | } 111 | } 112 | 113 | WriteResponse(w, response) 114 | } 115 | 116 | func TrendingDeveloperHandler(w http.ResponseWriter, r *http.Request) { 117 | lan := r.Form.Get("lan") 118 | since := r.Form.Get("since") 119 | if lan == "all-language" || lan == "" { 120 | lan = "all-language" 121 | } 122 | if since == "" { 123 | since = "daily" 124 | } 125 | 126 | key := fmt.Sprintf("trending-developer-%v-%v", lan, since) 127 | str, err := redisClient.Get(key) 128 | response := Response{ 129 | State: 1, 130 | } 131 | if err != nil { 132 | fmt.Println("repo error: ", err) 133 | response.ErrorMsg = err.Error() 134 | } else { 135 | var val interface{} 136 | err = json.Unmarshal([]byte(str), &val) 137 | if err != nil { 138 | response.ErrorMsg = err.Error() 139 | } else { 140 | response.Result = val 141 | } 142 | } 143 | 144 | WriteResponse(w, response) 145 | } 146 | 147 | func WriteResponse(w http.ResponseWriter, response Response) { 148 | w.Header().Add("Content-Type", "application/json; charset=UTF-8") 149 | w.WriteHeader(http.StatusOK) 150 | data, err := json.Marshal(response) 151 | if err != nil { 152 | w.Write([]byte(`{"state" : 1, "error_msg" : "", "result" : null}`)) 153 | } else { 154 | w.Write(data) 155 | } 156 | 157 | } 158 | 159 | -------------------------------------------------------------------------------- /server/server: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/0x1306a94/github-crawler/cb4bbaf5d68fb066976ac2d069b509c618d5b23e/server/server -------------------------------------------------------------------------------- /vendor/github.com/PuerkitoBio/goquery/LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012-2016, Martin Angers & Contributors 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 5 | 6 | * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 7 | 8 | * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 9 | 10 | * Neither the name of the author nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. 11 | 12 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 13 | -------------------------------------------------------------------------------- /vendor/github.com/PuerkitoBio/goquery/array.go: -------------------------------------------------------------------------------- 1 | package goquery 2 | 3 | import ( 4 | "golang.org/x/net/html" 5 | ) 6 | 7 | const ( 8 | maxUint = ^uint(0) 9 | maxInt = int(maxUint >> 1) 10 | 11 | // ToEnd is a special index value that can be used as end index in a call 12 | // to Slice so that all elements are selected until the end of the Selection. 13 | // It is equivalent to passing (*Selection).Length(). 14 | ToEnd = maxInt 15 | ) 16 | 17 | // First reduces the set of matched elements to the first in the set. 18 | // It returns a new Selection object, and an empty Selection object if the 19 | // the selection is empty. 20 | func (s *Selection) First() *Selection { 21 | return s.Eq(0) 22 | } 23 | 24 | // Last reduces the set of matched elements to the last in the set. 25 | // It returns a new Selection object, and an empty Selection object if 26 | // the selection is empty. 27 | func (s *Selection) Last() *Selection { 28 | return s.Eq(-1) 29 | } 30 | 31 | // Eq reduces the set of matched elements to the one at the specified index. 32 | // If a negative index is given, it counts backwards starting at the end of the 33 | // set. It returns a new Selection object, and an empty Selection object if the 34 | // index is invalid. 35 | func (s *Selection) Eq(index int) *Selection { 36 | if index < 0 { 37 | index += len(s.Nodes) 38 | } 39 | 40 | if index >= len(s.Nodes) || index < 0 { 41 | return newEmptySelection(s.document) 42 | } 43 | 44 | return s.Slice(index, index+1) 45 | } 46 | 47 | // Slice reduces the set of matched elements to a subset specified by a range 48 | // of indices. The start index is 0-based and indicates the index of the first 49 | // element to select. The end index is 0-based and indicates the index at which 50 | // the elements stop being selected (the end index is not selected). 51 | // 52 | // The indices may be negative, in which case they represent an offset from the 53 | // end of the selection. 54 | // 55 | // The special value ToEnd may be specified as end index, in which case all elements 56 | // until the end are selected. This works both for a positive and negative start 57 | // index. 58 | func (s *Selection) Slice(start, end int) *Selection { 59 | if start < 0 { 60 | start += len(s.Nodes) 61 | } 62 | if end == ToEnd { 63 | end = len(s.Nodes) 64 | } else if end < 0 { 65 | end += len(s.Nodes) 66 | } 67 | return pushStack(s, s.Nodes[start:end]) 68 | } 69 | 70 | // Get retrieves the underlying node at the specified index. 71 | // Get without parameter is not implemented, since the node array is available 72 | // on the Selection object. 73 | func (s *Selection) Get(index int) *html.Node { 74 | if index < 0 { 75 | index += len(s.Nodes) // Negative index gets from the end 76 | } 77 | return s.Nodes[index] 78 | } 79 | 80 | // Index returns the position of the first element within the Selection object 81 | // relative to its sibling elements. 82 | func (s *Selection) Index() int { 83 | if len(s.Nodes) > 0 { 84 | return newSingleSelection(s.Nodes[0], s.document).PrevAll().Length() 85 | } 86 | return -1 87 | } 88 | 89 | // IndexSelector returns the position of the first element within the 90 | // Selection object relative to the elements matched by the selector, or -1 if 91 | // not found. 92 | func (s *Selection) IndexSelector(selector string) int { 93 | if len(s.Nodes) > 0 { 94 | sel := s.document.Find(selector) 95 | return indexInSlice(sel.Nodes, s.Nodes[0]) 96 | } 97 | return -1 98 | } 99 | 100 | // IndexMatcher returns the position of the first element within the 101 | // Selection object relative to the elements matched by the matcher, or -1 if 102 | // not found. 103 | func (s *Selection) IndexMatcher(m Matcher) int { 104 | if len(s.Nodes) > 0 { 105 | sel := s.document.FindMatcher(m) 106 | return indexInSlice(sel.Nodes, s.Nodes[0]) 107 | } 108 | return -1 109 | } 110 | 111 | // IndexOfNode returns the position of the specified node within the Selection 112 | // object, or -1 if not found. 113 | func (s *Selection) IndexOfNode(node *html.Node) int { 114 | return indexInSlice(s.Nodes, node) 115 | } 116 | 117 | // IndexOfSelection returns the position of the first node in the specified 118 | // Selection object within this Selection object, or -1 if not found. 119 | func (s *Selection) IndexOfSelection(sel *Selection) int { 120 | if sel != nil && len(sel.Nodes) > 0 { 121 | return indexInSlice(s.Nodes, sel.Nodes[0]) 122 | } 123 | return -1 124 | } 125 | -------------------------------------------------------------------------------- /vendor/github.com/PuerkitoBio/goquery/doc.go: -------------------------------------------------------------------------------- 1 | // Copyright (c) 2012-2016, Martin Angers & Contributors 2 | // All rights reserved. 3 | // 4 | // Redistribution and use in source and binary forms, with or without modification, 5 | // are permitted provided that the following conditions are met: 6 | // 7 | // * Redistributions of source code must retain the above copyright notice, 8 | // this list of conditions and the following disclaimer. 9 | // * Redistributions in binary form must reproduce the above copyright notice, 10 | // this list of conditions and the following disclaimer in the documentation and/or 11 | // other materials provided with the distribution. 12 | // * Neither the name of the author nor the names of its contributors may be used to 13 | // endorse or promote products derived from this software without specific prior written permission. 14 | // 15 | // THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS 16 | // OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY 17 | // AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR 18 | // CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 19 | // DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 20 | // DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, 21 | // WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY 22 | // WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 23 | 24 | /* 25 | Package goquery implements features similar to jQuery, including the chainable 26 | syntax, to manipulate and query an HTML document. 27 | 28 | It brings a syntax and a set of features similar to jQuery to the Go language. 29 | It is based on Go's net/html package and the CSS Selector library cascadia. 30 | Since the net/html parser returns nodes, and not a full-featured DOM 31 | tree, jQuery's stateful manipulation functions (like height(), css(), detach()) 32 | have been left off. 33 | 34 | Also, because the net/html parser requires UTF-8 encoding, so does goquery: it is 35 | the caller's responsibility to ensure that the source document provides UTF-8 encoded HTML. 36 | See the repository's wiki for various options on how to do this. 37 | 38 | Syntax-wise, it is as close as possible to jQuery, with the same method names when 39 | possible, and that warm and fuzzy chainable interface. jQuery being the 40 | ultra-popular library that it is, writing a similar HTML-manipulating 41 | library was better to follow its API than to start anew (in the same spirit as 42 | Go's fmt package), even though some of its methods are less than intuitive (looking 43 | at you, index()...). 44 | 45 | It is hosted on GitHub, along with additional documentation in the README.md 46 | file: https://github.com/puerkitobio/goquery 47 | 48 | Please note that because of the net/html dependency, goquery requires Go1.1+. 49 | 50 | The various methods are split into files based on the category of behavior. 51 | The three dots (...) indicate that various "overloads" are available. 52 | 53 | * array.go : array-like positional manipulation of the selection. 54 | - Eq() 55 | - First() 56 | - Get() 57 | - Index...() 58 | - Last() 59 | - Slice() 60 | 61 | * expand.go : methods that expand or augment the selection's set. 62 | - Add...() 63 | - AndSelf() 64 | - Union(), which is an alias for AddSelection() 65 | 66 | * filter.go : filtering methods, that reduce the selection's set. 67 | - End() 68 | - Filter...() 69 | - Has...() 70 | - Intersection(), which is an alias of FilterSelection() 71 | - Not...() 72 | 73 | * iteration.go : methods to loop over the selection's nodes. 74 | - Each() 75 | - EachWithBreak() 76 | - Map() 77 | 78 | * manipulation.go : methods for modifying the document 79 | - After...() 80 | - Append...() 81 | - Before...() 82 | - Clone() 83 | - Empty() 84 | - Prepend...() 85 | - Remove...() 86 | - ReplaceWith...() 87 | - Unwrap() 88 | - Wrap...() 89 | - WrapAll...() 90 | - WrapInner...() 91 | 92 | * property.go : methods that inspect and get the node's properties values. 93 | - Attr*(), RemoveAttr(), SetAttr() 94 | - AddClass(), HasClass(), RemoveClass(), ToggleClass() 95 | - Html() 96 | - Length() 97 | - Size(), which is an alias for Length() 98 | - Text() 99 | 100 | * query.go : methods that query, or reflect, a node's identity. 101 | - Contains() 102 | - Is...() 103 | 104 | * traversal.go : methods to traverse the HTML document tree. 105 | - Children...() 106 | - Contents() 107 | - Find...() 108 | - Next...() 109 | - Parent[s]...() 110 | - Prev...() 111 | - Siblings...() 112 | 113 | * type.go : definition of the types exposed by goquery. 114 | - Document 115 | - Selection 116 | - Matcher 117 | 118 | * utilities.go : definition of helper functions (and not methods on a *Selection) 119 | that are not part of jQuery, but are useful to goquery. 120 | - NodeName 121 | - OuterHtml 122 | */ 123 | package goquery 124 | -------------------------------------------------------------------------------- /vendor/github.com/PuerkitoBio/goquery/expand.go: -------------------------------------------------------------------------------- 1 | package goquery 2 | 3 | import "golang.org/x/net/html" 4 | 5 | // Add adds the selector string's matching nodes to those in the current 6 | // selection and returns a new Selection object. 7 | // The selector string is run in the context of the document of the current 8 | // Selection object. 9 | func (s *Selection) Add(selector string) *Selection { 10 | return s.AddNodes(findWithMatcher([]*html.Node{s.document.rootNode}, compileMatcher(selector))...) 11 | } 12 | 13 | // AddMatcher adds the matcher's matching nodes to those in the current 14 | // selection and returns a new Selection object. 15 | // The matcher is run in the context of the document of the current 16 | // Selection object. 17 | func (s *Selection) AddMatcher(m Matcher) *Selection { 18 | return s.AddNodes(findWithMatcher([]*html.Node{s.document.rootNode}, m)...) 19 | } 20 | 21 | // AddSelection adds the specified Selection object's nodes to those in the 22 | // current selection and returns a new Selection object. 23 | func (s *Selection) AddSelection(sel *Selection) *Selection { 24 | if sel == nil { 25 | return s.AddNodes() 26 | } 27 | return s.AddNodes(sel.Nodes...) 28 | } 29 | 30 | // Union is an alias for AddSelection. 31 | func (s *Selection) Union(sel *Selection) *Selection { 32 | return s.AddSelection(sel) 33 | } 34 | 35 | // AddNodes adds the specified nodes to those in the 36 | // current selection and returns a new Selection object. 37 | func (s *Selection) AddNodes(nodes ...*html.Node) *Selection { 38 | return pushStack(s, appendWithoutDuplicates(s.Nodes, nodes, nil)) 39 | } 40 | 41 | // AndSelf adds the previous set of elements on the stack to the current set. 42 | // It returns a new Selection object containing the current Selection combined 43 | // with the previous one. 44 | // Deprecated: This function has been deprecated and is now an alias for AddBack(). 45 | func (s *Selection) AndSelf() *Selection { 46 | return s.AddBack() 47 | } 48 | 49 | // AddBack adds the previous set of elements on the stack to the current set. 50 | // It returns a new Selection object containing the current Selection combined 51 | // with the previous one. 52 | func (s *Selection) AddBack() *Selection { 53 | return s.AddSelection(s.prevSel) 54 | } 55 | 56 | // AddBackFiltered reduces the previous set of elements on the stack to those that 57 | // match the selector string, and adds them to the current set. 58 | // It returns a new Selection object containing the current Selection combined 59 | // with the filtered previous one 60 | func (s *Selection) AddBackFiltered(selector string) *Selection { 61 | return s.AddSelection(s.prevSel.Filter(selector)) 62 | } 63 | 64 | // AddBackMatcher reduces the previous set of elements on the stack to those that match 65 | // the mateher, and adds them to the curernt set. 66 | // It returns a new Selection object containing the current Selection combined 67 | // with the filtered previous one 68 | func (s *Selection) AddBackMatcher(m Matcher) *Selection { 69 | return s.AddSelection(s.prevSel.FilterMatcher(m)) 70 | } 71 | -------------------------------------------------------------------------------- /vendor/github.com/PuerkitoBio/goquery/filter.go: -------------------------------------------------------------------------------- 1 | package goquery 2 | 3 | import "golang.org/x/net/html" 4 | 5 | // Filter reduces the set of matched elements to those that match the selector string. 6 | // It returns a new Selection object for this subset of matching elements. 7 | func (s *Selection) Filter(selector string) *Selection { 8 | return s.FilterMatcher(compileMatcher(selector)) 9 | } 10 | 11 | // FilterMatcher reduces the set of matched elements to those that match 12 | // the given matcher. It returns a new Selection object for this subset 13 | // of matching elements. 14 | func (s *Selection) FilterMatcher(m Matcher) *Selection { 15 | return pushStack(s, winnow(s, m, true)) 16 | } 17 | 18 | // Not removes elements from the Selection that match the selector string. 19 | // It returns a new Selection object with the matching elements removed. 20 | func (s *Selection) Not(selector string) *Selection { 21 | return s.NotMatcher(compileMatcher(selector)) 22 | } 23 | 24 | // NotMatcher removes elements from the Selection that match the given matcher. 25 | // It returns a new Selection object with the matching elements removed. 26 | func (s *Selection) NotMatcher(m Matcher) *Selection { 27 | return pushStack(s, winnow(s, m, false)) 28 | } 29 | 30 | // FilterFunction reduces the set of matched elements to those that pass the function's test. 31 | // It returns a new Selection object for this subset of elements. 32 | func (s *Selection) FilterFunction(f func(int, *Selection) bool) *Selection { 33 | return pushStack(s, winnowFunction(s, f, true)) 34 | } 35 | 36 | // NotFunction removes elements from the Selection that pass the function's test. 37 | // It returns a new Selection object with the matching elements removed. 38 | func (s *Selection) NotFunction(f func(int, *Selection) bool) *Selection { 39 | return pushStack(s, winnowFunction(s, f, false)) 40 | } 41 | 42 | // FilterNodes reduces the set of matched elements to those that match the specified nodes. 43 | // It returns a new Selection object for this subset of elements. 44 | func (s *Selection) FilterNodes(nodes ...*html.Node) *Selection { 45 | return pushStack(s, winnowNodes(s, nodes, true)) 46 | } 47 | 48 | // NotNodes removes elements from the Selection that match the specified nodes. 49 | // It returns a new Selection object with the matching elements removed. 50 | func (s *Selection) NotNodes(nodes ...*html.Node) *Selection { 51 | return pushStack(s, winnowNodes(s, nodes, false)) 52 | } 53 | 54 | // FilterSelection reduces the set of matched elements to those that match a 55 | // node in the specified Selection object. 56 | // It returns a new Selection object for this subset of elements. 57 | func (s *Selection) FilterSelection(sel *Selection) *Selection { 58 | if sel == nil { 59 | return pushStack(s, winnowNodes(s, nil, true)) 60 | } 61 | return pushStack(s, winnowNodes(s, sel.Nodes, true)) 62 | } 63 | 64 | // NotSelection removes elements from the Selection that match a node in the specified 65 | // Selection object. It returns a new Selection object with the matching elements removed. 66 | func (s *Selection) NotSelection(sel *Selection) *Selection { 67 | if sel == nil { 68 | return pushStack(s, winnowNodes(s, nil, false)) 69 | } 70 | return pushStack(s, winnowNodes(s, sel.Nodes, false)) 71 | } 72 | 73 | // Intersection is an alias for FilterSelection. 74 | func (s *Selection) Intersection(sel *Selection) *Selection { 75 | return s.FilterSelection(sel) 76 | } 77 | 78 | // Has reduces the set of matched elements to those that have a descendant 79 | // that matches the selector. 80 | // It returns a new Selection object with the matching elements. 81 | func (s *Selection) Has(selector string) *Selection { 82 | return s.HasSelection(s.document.Find(selector)) 83 | } 84 | 85 | // HasMatcher reduces the set of matched elements to those that have a descendant 86 | // that matches the matcher. 87 | // It returns a new Selection object with the matching elements. 88 | func (s *Selection) HasMatcher(m Matcher) *Selection { 89 | return s.HasSelection(s.document.FindMatcher(m)) 90 | } 91 | 92 | // HasNodes reduces the set of matched elements to those that have a 93 | // descendant that matches one of the nodes. 94 | // It returns a new Selection object with the matching elements. 95 | func (s *Selection) HasNodes(nodes ...*html.Node) *Selection { 96 | return s.FilterFunction(func(_ int, sel *Selection) bool { 97 | // Add all nodes that contain one of the specified nodes 98 | for _, n := range nodes { 99 | if sel.Contains(n) { 100 | return true 101 | } 102 | } 103 | return false 104 | }) 105 | } 106 | 107 | // HasSelection reduces the set of matched elements to those that have a 108 | // descendant that matches one of the nodes of the specified Selection object. 109 | // It returns a new Selection object with the matching elements. 110 | func (s *Selection) HasSelection(sel *Selection) *Selection { 111 | if sel == nil { 112 | return s.HasNodes() 113 | } 114 | return s.HasNodes(sel.Nodes...) 115 | } 116 | 117 | // End ends the most recent filtering operation in the current chain and 118 | // returns the set of matched elements to its previous state. 119 | func (s *Selection) End() *Selection { 120 | if s.prevSel != nil { 121 | return s.prevSel 122 | } 123 | return newEmptySelection(s.document) 124 | } 125 | 126 | // Filter based on the matcher, and the indicator to keep (Filter) or 127 | // to get rid of (Not) the matching elements. 128 | func winnow(sel *Selection, m Matcher, keep bool) []*html.Node { 129 | // Optimize if keep is requested 130 | if keep { 131 | return m.Filter(sel.Nodes) 132 | } 133 | // Use grep 134 | return grep(sel, func(i int, s *Selection) bool { 135 | return !m.Match(s.Get(0)) 136 | }) 137 | } 138 | 139 | // Filter based on an array of nodes, and the indicator to keep (Filter) or 140 | // to get rid of (Not) the matching elements. 141 | func winnowNodes(sel *Selection, nodes []*html.Node, keep bool) []*html.Node { 142 | if len(nodes)+len(sel.Nodes) < minNodesForSet { 143 | return grep(sel, func(i int, s *Selection) bool { 144 | return isInSlice(nodes, s.Get(0)) == keep 145 | }) 146 | } 147 | 148 | set := make(map[*html.Node]bool) 149 | for _, n := range nodes { 150 | set[n] = true 151 | } 152 | return grep(sel, func(i int, s *Selection) bool { 153 | return set[s.Get(0)] == keep 154 | }) 155 | } 156 | 157 | // Filter based on a function test, and the indicator to keep (Filter) or 158 | // to get rid of (Not) the matching elements. 159 | func winnowFunction(sel *Selection, f func(int, *Selection) bool, keep bool) []*html.Node { 160 | return grep(sel, func(i int, s *Selection) bool { 161 | return f(i, s) == keep 162 | }) 163 | } 164 | -------------------------------------------------------------------------------- /vendor/github.com/PuerkitoBio/goquery/iteration.go: -------------------------------------------------------------------------------- 1 | package goquery 2 | 3 | // Each iterates over a Selection object, executing a function for each 4 | // matched element. It returns the current Selection object. The function 5 | // f is called for each element in the selection with the index of the 6 | // element in that selection starting at 0, and a *Selection that contains 7 | // only that element. 8 | func (s *Selection) Each(f func(int, *Selection)) *Selection { 9 | for i, n := range s.Nodes { 10 | f(i, newSingleSelection(n, s.document)) 11 | } 12 | return s 13 | } 14 | 15 | // EachWithBreak iterates over a Selection object, executing a function for each 16 | // matched element. It is identical to Each except that it is possible to break 17 | // out of the loop by returning false in the callback function. It returns the 18 | // current Selection object. 19 | func (s *Selection) EachWithBreak(f func(int, *Selection) bool) *Selection { 20 | for i, n := range s.Nodes { 21 | if !f(i, newSingleSelection(n, s.document)) { 22 | return s 23 | } 24 | } 25 | return s 26 | } 27 | 28 | // Map passes each element in the current matched set through a function, 29 | // producing a slice of string holding the returned values. The function 30 | // f is called for each element in the selection with the index of the 31 | // element in that selection starting at 0, and a *Selection that contains 32 | // only that element. 33 | func (s *Selection) Map(f func(int, *Selection) string) (result []string) { 34 | for i, n := range s.Nodes { 35 | result = append(result, f(i, newSingleSelection(n, s.document))) 36 | } 37 | 38 | return result 39 | } 40 | -------------------------------------------------------------------------------- /vendor/github.com/PuerkitoBio/goquery/property.go: -------------------------------------------------------------------------------- 1 | package goquery 2 | 3 | import ( 4 | "bytes" 5 | "regexp" 6 | "strings" 7 | 8 | "golang.org/x/net/html" 9 | ) 10 | 11 | var rxClassTrim = regexp.MustCompile("[\t\r\n]") 12 | 13 | // Attr gets the specified attribute's value for the first element in the 14 | // Selection. To get the value for each element individually, use a looping 15 | // construct such as Each or Map method. 16 | func (s *Selection) Attr(attrName string) (val string, exists bool) { 17 | if len(s.Nodes) == 0 { 18 | return 19 | } 20 | return getAttributeValue(attrName, s.Nodes[0]) 21 | } 22 | 23 | // AttrOr works like Attr but returns default value if attribute is not present. 24 | func (s *Selection) AttrOr(attrName, defaultValue string) string { 25 | if len(s.Nodes) == 0 { 26 | return defaultValue 27 | } 28 | 29 | val, exists := getAttributeValue(attrName, s.Nodes[0]) 30 | if !exists { 31 | return defaultValue 32 | } 33 | 34 | return val 35 | } 36 | 37 | // RemoveAttr removes the named attribute from each element in the set of matched elements. 38 | func (s *Selection) RemoveAttr(attrName string) *Selection { 39 | for _, n := range s.Nodes { 40 | removeAttr(n, attrName) 41 | } 42 | 43 | return s 44 | } 45 | 46 | // SetAttr sets the given attribute on each element in the set of matched elements. 47 | func (s *Selection) SetAttr(attrName, val string) *Selection { 48 | for _, n := range s.Nodes { 49 | attr := getAttributePtr(attrName, n) 50 | if attr == nil { 51 | n.Attr = append(n.Attr, html.Attribute{Key: attrName, Val: val}) 52 | } else { 53 | attr.Val = val 54 | } 55 | } 56 | 57 | return s 58 | } 59 | 60 | // Text gets the combined text contents of each element in the set of matched 61 | // elements, including their descendants. 62 | func (s *Selection) Text() string { 63 | var buf bytes.Buffer 64 | 65 | // Slightly optimized vs calling Each: no single selection object created 66 | var f func(*html.Node) 67 | f = func(n *html.Node) { 68 | if n.Type == html.TextNode { 69 | // Keep newlines and spaces, like jQuery 70 | buf.WriteString(n.Data) 71 | } 72 | if n.FirstChild != nil { 73 | for c := n.FirstChild; c != nil; c = c.NextSibling { 74 | f(c) 75 | } 76 | } 77 | } 78 | for _, n := range s.Nodes { 79 | f(n) 80 | } 81 | 82 | return buf.String() 83 | } 84 | 85 | // Size is an alias for Length. 86 | func (s *Selection) Size() int { 87 | return s.Length() 88 | } 89 | 90 | // Length returns the number of elements in the Selection object. 91 | func (s *Selection) Length() int { 92 | return len(s.Nodes) 93 | } 94 | 95 | // Html gets the HTML contents of the first element in the set of matched 96 | // elements. It includes text and comment nodes. 97 | func (s *Selection) Html() (ret string, e error) { 98 | // Since there is no .innerHtml, the HTML content must be re-created from 99 | // the nodes using html.Render. 100 | var buf bytes.Buffer 101 | 102 | if len(s.Nodes) > 0 { 103 | for c := s.Nodes[0].FirstChild; c != nil; c = c.NextSibling { 104 | e = html.Render(&buf, c) 105 | if e != nil { 106 | return 107 | } 108 | } 109 | ret = buf.String() 110 | } 111 | 112 | return 113 | } 114 | 115 | // AddClass adds the given class(es) to each element in the set of matched elements. 116 | // Multiple class names can be specified, separated by a space or via multiple arguments. 117 | func (s *Selection) AddClass(class ...string) *Selection { 118 | classStr := strings.TrimSpace(strings.Join(class, " ")) 119 | 120 | if classStr == "" { 121 | return s 122 | } 123 | 124 | tcls := getClassesSlice(classStr) 125 | for _, n := range s.Nodes { 126 | curClasses, attr := getClassesAndAttr(n, true) 127 | for _, newClass := range tcls { 128 | if !strings.Contains(curClasses, " "+newClass+" ") { 129 | curClasses += newClass + " " 130 | } 131 | } 132 | 133 | setClasses(n, attr, curClasses) 134 | } 135 | 136 | return s 137 | } 138 | 139 | // HasClass determines whether any of the matched elements are assigned the 140 | // given class. 141 | func (s *Selection) HasClass(class string) bool { 142 | class = " " + class + " " 143 | for _, n := range s.Nodes { 144 | classes, _ := getClassesAndAttr(n, false) 145 | if strings.Contains(classes, class) { 146 | return true 147 | } 148 | } 149 | return false 150 | } 151 | 152 | // RemoveClass removes the given class(es) from each element in the set of matched elements. 153 | // Multiple class names can be specified, separated by a space or via multiple arguments. 154 | // If no class name is provided, all classes are removed. 155 | func (s *Selection) RemoveClass(class ...string) *Selection { 156 | var rclasses []string 157 | 158 | classStr := strings.TrimSpace(strings.Join(class, " ")) 159 | remove := classStr == "" 160 | 161 | if !remove { 162 | rclasses = getClassesSlice(classStr) 163 | } 164 | 165 | for _, n := range s.Nodes { 166 | if remove { 167 | removeAttr(n, "class") 168 | } else { 169 | classes, attr := getClassesAndAttr(n, true) 170 | for _, rcl := range rclasses { 171 | classes = strings.Replace(classes, " "+rcl+" ", " ", -1) 172 | } 173 | 174 | setClasses(n, attr, classes) 175 | } 176 | } 177 | 178 | return s 179 | } 180 | 181 | // ToggleClass adds or removes the given class(es) for each element in the set of matched elements. 182 | // Multiple class names can be specified, separated by a space or via multiple arguments. 183 | func (s *Selection) ToggleClass(class ...string) *Selection { 184 | classStr := strings.TrimSpace(strings.Join(class, " ")) 185 | 186 | if classStr == "" { 187 | return s 188 | } 189 | 190 | tcls := getClassesSlice(classStr) 191 | 192 | for _, n := range s.Nodes { 193 | classes, attr := getClassesAndAttr(n, true) 194 | for _, tcl := range tcls { 195 | if strings.Contains(classes, " "+tcl+" ") { 196 | classes = strings.Replace(classes, " "+tcl+" ", " ", -1) 197 | } else { 198 | classes += tcl + " " 199 | } 200 | } 201 | 202 | setClasses(n, attr, classes) 203 | } 204 | 205 | return s 206 | } 207 | 208 | func getAttributePtr(attrName string, n *html.Node) *html.Attribute { 209 | if n == nil { 210 | return nil 211 | } 212 | 213 | for i, a := range n.Attr { 214 | if a.Key == attrName { 215 | return &n.Attr[i] 216 | } 217 | } 218 | return nil 219 | } 220 | 221 | // Private function to get the specified attribute's value from a node. 222 | func getAttributeValue(attrName string, n *html.Node) (val string, exists bool) { 223 | if a := getAttributePtr(attrName, n); a != nil { 224 | val = a.Val 225 | exists = true 226 | } 227 | return 228 | } 229 | 230 | // Get and normalize the "class" attribute from the node. 231 | func getClassesAndAttr(n *html.Node, create bool) (classes string, attr *html.Attribute) { 232 | // Applies only to element nodes 233 | if n.Type == html.ElementNode { 234 | attr = getAttributePtr("class", n) 235 | if attr == nil && create { 236 | n.Attr = append(n.Attr, html.Attribute{ 237 | Key: "class", 238 | Val: "", 239 | }) 240 | attr = &n.Attr[len(n.Attr)-1] 241 | } 242 | } 243 | 244 | if attr == nil { 245 | classes = " " 246 | } else { 247 | classes = rxClassTrim.ReplaceAllString(" "+attr.Val+" ", " ") 248 | } 249 | 250 | return 251 | } 252 | 253 | func getClassesSlice(classes string) []string { 254 | return strings.Split(rxClassTrim.ReplaceAllString(" "+classes+" ", " "), " ") 255 | } 256 | 257 | func removeAttr(n *html.Node, attrName string) { 258 | for i, a := range n.Attr { 259 | if a.Key == attrName { 260 | n.Attr[i], n.Attr[len(n.Attr)-1], n.Attr = 261 | n.Attr[len(n.Attr)-1], html.Attribute{}, n.Attr[:len(n.Attr)-1] 262 | return 263 | } 264 | } 265 | } 266 | 267 | func setClasses(n *html.Node, attr *html.Attribute, classes string) { 268 | classes = strings.TrimSpace(classes) 269 | if classes == "" { 270 | removeAttr(n, "class") 271 | return 272 | } 273 | 274 | attr.Val = classes 275 | } 276 | -------------------------------------------------------------------------------- /vendor/github.com/PuerkitoBio/goquery/query.go: -------------------------------------------------------------------------------- 1 | package goquery 2 | 3 | import "golang.org/x/net/html" 4 | 5 | // Is checks the current matched set of elements against a selector and 6 | // returns true if at least one of these elements matches. 7 | func (s *Selection) Is(selector string) bool { 8 | return s.IsMatcher(compileMatcher(selector)) 9 | } 10 | 11 | // IsMatcher checks the current matched set of elements against a matcher and 12 | // returns true if at least one of these elements matches. 13 | func (s *Selection) IsMatcher(m Matcher) bool { 14 | if len(s.Nodes) > 0 { 15 | if len(s.Nodes) == 1 { 16 | return m.Match(s.Nodes[0]) 17 | } 18 | return len(m.Filter(s.Nodes)) > 0 19 | } 20 | 21 | return false 22 | } 23 | 24 | // IsFunction checks the current matched set of elements against a predicate and 25 | // returns true if at least one of these elements matches. 26 | func (s *Selection) IsFunction(f func(int, *Selection) bool) bool { 27 | return s.FilterFunction(f).Length() > 0 28 | } 29 | 30 | // IsSelection checks the current matched set of elements against a Selection object 31 | // and returns true if at least one of these elements matches. 32 | func (s *Selection) IsSelection(sel *Selection) bool { 33 | return s.FilterSelection(sel).Length() > 0 34 | } 35 | 36 | // IsNodes checks the current matched set of elements against the specified nodes 37 | // and returns true if at least one of these elements matches. 38 | func (s *Selection) IsNodes(nodes ...*html.Node) bool { 39 | return s.FilterNodes(nodes...).Length() > 0 40 | } 41 | 42 | // Contains returns true if the specified Node is within, 43 | // at any depth, one of the nodes in the Selection object. 44 | // It is NOT inclusive, to behave like jQuery's implementation, and 45 | // unlike Javascript's .contains, so if the contained 46 | // node is itself in the selection, it returns false. 47 | func (s *Selection) Contains(n *html.Node) bool { 48 | return sliceContains(s.Nodes, n) 49 | } 50 | -------------------------------------------------------------------------------- /vendor/github.com/PuerkitoBio/goquery/type.go: -------------------------------------------------------------------------------- 1 | package goquery 2 | 3 | import ( 4 | "errors" 5 | "io" 6 | "net/http" 7 | "net/url" 8 | 9 | "github.com/andybalholm/cascadia" 10 | 11 | "golang.org/x/net/html" 12 | ) 13 | 14 | // Document represents an HTML document to be manipulated. Unlike jQuery, which 15 | // is loaded as part of a DOM document, and thus acts upon its containing 16 | // document, GoQuery doesn't know which HTML document to act upon. So it needs 17 | // to be told, and that's what the Document class is for. It holds the root 18 | // document node to manipulate, and can make selections on this document. 19 | type Document struct { 20 | *Selection 21 | Url *url.URL 22 | rootNode *html.Node 23 | } 24 | 25 | // NewDocumentFromNode is a Document constructor that takes a root html Node 26 | // as argument. 27 | func NewDocumentFromNode(root *html.Node) *Document { 28 | return newDocument(root, nil) 29 | } 30 | 31 | // NewDocument is a Document constructor that takes a string URL as argument. 32 | // It loads the specified document, parses it, and stores the root Document 33 | // node, ready to be manipulated. 34 | // 35 | // Deprecated: Use the net/http standard library package to make the request 36 | // and validate the response before calling goquery.NewDocumentFromReader 37 | // with the response's body. 38 | func NewDocument(url string) (*Document, error) { 39 | // Load the URL 40 | res, e := http.Get(url) 41 | if e != nil { 42 | return nil, e 43 | } 44 | return NewDocumentFromResponse(res) 45 | } 46 | 47 | // NewDocumentFromReader returns a Document from an io.Reader. 48 | // It returns an error as second value if the reader's data cannot be parsed 49 | // as html. It does not check if the reader is also an io.Closer, the 50 | // provided reader is never closed by this call. It is the responsibility 51 | // of the caller to close it if required. 52 | func NewDocumentFromReader(r io.Reader) (*Document, error) { 53 | root, e := html.Parse(r) 54 | if e != nil { 55 | return nil, e 56 | } 57 | return newDocument(root, nil), nil 58 | } 59 | 60 | // NewDocumentFromResponse is another Document constructor that takes an http response as argument. 61 | // It loads the specified response's document, parses it, and stores the root Document 62 | // node, ready to be manipulated. The response's body is closed on return. 63 | // 64 | // Deprecated: Use goquery.NewDocumentFromReader with the response's body. 65 | func NewDocumentFromResponse(res *http.Response) (*Document, error) { 66 | if res == nil { 67 | return nil, errors.New("Response is nil") 68 | } 69 | defer res.Body.Close() 70 | if res.Request == nil { 71 | return nil, errors.New("Response.Request is nil") 72 | } 73 | 74 | // Parse the HTML into nodes 75 | root, e := html.Parse(res.Body) 76 | if e != nil { 77 | return nil, e 78 | } 79 | 80 | // Create and fill the document 81 | return newDocument(root, res.Request.URL), nil 82 | } 83 | 84 | // CloneDocument creates a deep-clone of a document. 85 | func CloneDocument(doc *Document) *Document { 86 | return newDocument(cloneNode(doc.rootNode), doc.Url) 87 | } 88 | 89 | // Private constructor, make sure all fields are correctly filled. 90 | func newDocument(root *html.Node, url *url.URL) *Document { 91 | // Create and fill the document 92 | d := &Document{nil, url, root} 93 | d.Selection = newSingleSelection(root, d) 94 | return d 95 | } 96 | 97 | // Selection represents a collection of nodes matching some criteria. The 98 | // initial Selection can be created by using Document.Find, and then 99 | // manipulated using the jQuery-like chainable syntax and methods. 100 | type Selection struct { 101 | Nodes []*html.Node 102 | document *Document 103 | prevSel *Selection 104 | } 105 | 106 | // Helper constructor to create an empty selection 107 | func newEmptySelection(doc *Document) *Selection { 108 | return &Selection{nil, doc, nil} 109 | } 110 | 111 | // Helper constructor to create a selection of only one node 112 | func newSingleSelection(node *html.Node, doc *Document) *Selection { 113 | return &Selection{[]*html.Node{node}, doc, nil} 114 | } 115 | 116 | // Matcher is an interface that defines the methods to match 117 | // HTML nodes against a compiled selector string. Cascadia's 118 | // Selector implements this interface. 119 | type Matcher interface { 120 | Match(*html.Node) bool 121 | MatchAll(*html.Node) []*html.Node 122 | Filter([]*html.Node) []*html.Node 123 | } 124 | 125 | // compileMatcher compiles the selector string s and returns 126 | // the corresponding Matcher. If s is an invalid selector string, 127 | // it returns a Matcher that fails all matches. 128 | func compileMatcher(s string) Matcher { 129 | cs, err := cascadia.Compile(s) 130 | if err != nil { 131 | return invalidMatcher{} 132 | } 133 | return cs 134 | } 135 | 136 | // invalidMatcher is a Matcher that always fails to match. 137 | type invalidMatcher struct{} 138 | 139 | func (invalidMatcher) Match(n *html.Node) bool { return false } 140 | func (invalidMatcher) MatchAll(n *html.Node) []*html.Node { return nil } 141 | func (invalidMatcher) Filter(ns []*html.Node) []*html.Node { return nil } 142 | -------------------------------------------------------------------------------- /vendor/github.com/PuerkitoBio/goquery/utilities.go: -------------------------------------------------------------------------------- 1 | package goquery 2 | 3 | import ( 4 | "bytes" 5 | 6 | "golang.org/x/net/html" 7 | ) 8 | 9 | // used to determine if a set (map[*html.Node]bool) should be used 10 | // instead of iterating over a slice. The set uses more memory and 11 | // is slower than slice iteration for small N. 12 | const minNodesForSet = 1000 13 | 14 | var nodeNames = []string{ 15 | html.ErrorNode: "#error", 16 | html.TextNode: "#text", 17 | html.DocumentNode: "#document", 18 | html.CommentNode: "#comment", 19 | } 20 | 21 | // NodeName returns the node name of the first element in the selection. 22 | // It tries to behave in a similar way as the DOM's nodeName property 23 | // (https://developer.mozilla.org/en-US/docs/Web/API/Node/nodeName). 24 | // 25 | // Go's net/html package defines the following node types, listed with 26 | // the corresponding returned value from this function: 27 | // 28 | // ErrorNode : #error 29 | // TextNode : #text 30 | // DocumentNode : #document 31 | // ElementNode : the element's tag name 32 | // CommentNode : #comment 33 | // DoctypeNode : the name of the document type 34 | // 35 | func NodeName(s *Selection) string { 36 | if s.Length() == 0 { 37 | return "" 38 | } 39 | switch n := s.Get(0); n.Type { 40 | case html.ElementNode, html.DoctypeNode: 41 | return n.Data 42 | default: 43 | if n.Type >= 0 && int(n.Type) < len(nodeNames) { 44 | return nodeNames[n.Type] 45 | } 46 | return "" 47 | } 48 | } 49 | 50 | // OuterHtml returns the outer HTML rendering of the first item in 51 | // the selection - that is, the HTML including the first element's 52 | // tag and attributes. 53 | // 54 | // Unlike InnerHtml, this is a function and not a method on the Selection, 55 | // because this is not a jQuery method (in javascript-land, this is 56 | // a property provided by the DOM). 57 | func OuterHtml(s *Selection) (string, error) { 58 | var buf bytes.Buffer 59 | 60 | if s.Length() == 0 { 61 | return "", nil 62 | } 63 | n := s.Get(0) 64 | if err := html.Render(&buf, n); err != nil { 65 | return "", err 66 | } 67 | return buf.String(), nil 68 | } 69 | 70 | // Loop through all container nodes to search for the target node. 71 | func sliceContains(container []*html.Node, contained *html.Node) bool { 72 | for _, n := range container { 73 | if nodeContains(n, contained) { 74 | return true 75 | } 76 | } 77 | 78 | return false 79 | } 80 | 81 | // Checks if the contained node is within the container node. 82 | func nodeContains(container *html.Node, contained *html.Node) bool { 83 | // Check if the parent of the contained node is the container node, traversing 84 | // upward until the top is reached, or the container is found. 85 | for contained = contained.Parent; contained != nil; contained = contained.Parent { 86 | if container == contained { 87 | return true 88 | } 89 | } 90 | return false 91 | } 92 | 93 | // Checks if the target node is in the slice of nodes. 94 | func isInSlice(slice []*html.Node, node *html.Node) bool { 95 | return indexInSlice(slice, node) > -1 96 | } 97 | 98 | // Returns the index of the target node in the slice, or -1. 99 | func indexInSlice(slice []*html.Node, node *html.Node) int { 100 | if node != nil { 101 | for i, n := range slice { 102 | if n == node { 103 | return i 104 | } 105 | } 106 | } 107 | return -1 108 | } 109 | 110 | // Appends the new nodes to the target slice, making sure no duplicate is added. 111 | // There is no check to the original state of the target slice, so it may still 112 | // contain duplicates. The target slice is returned because append() may create 113 | // a new underlying array. If targetSet is nil, a local set is created with the 114 | // target if len(target) + len(nodes) is greater than minNodesForSet. 115 | func appendWithoutDuplicates(target []*html.Node, nodes []*html.Node, targetSet map[*html.Node]bool) []*html.Node { 116 | // if there are not that many nodes, don't use the map, faster to just use nested loops 117 | // (unless a non-nil targetSet is passed, in which case the caller knows better). 118 | if targetSet == nil && len(target)+len(nodes) < minNodesForSet { 119 | for _, n := range nodes { 120 | if !isInSlice(target, n) { 121 | target = append(target, n) 122 | } 123 | } 124 | return target 125 | } 126 | 127 | // if a targetSet is passed, then assume it is reliable, otherwise create one 128 | // and initialize it with the current target contents. 129 | if targetSet == nil { 130 | targetSet = make(map[*html.Node]bool, len(target)) 131 | for _, n := range target { 132 | targetSet[n] = true 133 | } 134 | } 135 | for _, n := range nodes { 136 | if !targetSet[n] { 137 | target = append(target, n) 138 | targetSet[n] = true 139 | } 140 | } 141 | 142 | return target 143 | } 144 | 145 | // Loop through a selection, returning only those nodes that pass the predicate 146 | // function. 147 | func grep(sel *Selection, predicate func(i int, s *Selection) bool) (result []*html.Node) { 148 | for i, n := range sel.Nodes { 149 | if predicate(i, newSingleSelection(n, sel.document)) { 150 | result = append(result, n) 151 | } 152 | } 153 | return result 154 | } 155 | 156 | // Creates a new Selection object based on the specified nodes, and keeps the 157 | // source Selection object on the stack (linked list). 158 | func pushStack(fromSel *Selection, nodes []*html.Node) *Selection { 159 | result := &Selection{nodes, fromSel.document, fromSel} 160 | return result 161 | } 162 | -------------------------------------------------------------------------------- /vendor/github.com/andybalholm/cascadia/LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2011 Andy Balholm. All rights reserved. 2 | 3 | Redistribution and use in source and binary forms, with or without 4 | modification, are permitted provided that the following conditions are 5 | met: 6 | 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above 10 | copyright notice, this list of conditions and the following disclaimer 11 | in the documentation and/or other materials provided with the 12 | distribution. 13 | 14 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 15 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 16 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 17 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 18 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 19 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 20 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 21 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 22 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 23 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 24 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 25 | -------------------------------------------------------------------------------- /vendor/github.com/andybalholm/cascadia/README.md: -------------------------------------------------------------------------------- 1 | # cascadia 2 | 3 | [![](https://travis-ci.org/andybalholm/cascadia.svg)](https://travis-ci.org/andybalholm/cascadia) 4 | 5 | The Cascadia package implements CSS selectors for use with the parse trees produced by the html package. 6 | 7 | To test CSS selectors without writing Go code, check out [cascadia](https://github.com/suntong/cascadia) the command line tool, a thin wrapper around this package. 8 | -------------------------------------------------------------------------------- /vendor/github.com/andybalholm/cascadia/go.mod: -------------------------------------------------------------------------------- 1 | module "github.com/andybalholm/cascadia" 2 | 3 | require "golang.org/x/net" v0.0.0-20180218175443-cbe0f9307d01 4 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2013 The github.com/go-redis/redis Authors. 2 | All rights reserved. 3 | 4 | Redistribution and use in source and binary forms, with or without 5 | modification, are permitted provided that the following conditions are 6 | met: 7 | 8 | * Redistributions of source code must retain the above copyright 9 | notice, this list of conditions and the following disclaimer. 10 | * Redistributions in binary form must reproduce the above 11 | copyright notice, this list of conditions and the following disclaimer 12 | in the documentation and/or other materials provided with the 13 | distribution. 14 | 15 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 16 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 17 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 18 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 19 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 20 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 21 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 22 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 23 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 24 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 25 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/Makefile: -------------------------------------------------------------------------------- 1 | all: testdeps 2 | go test ./... 3 | go test ./... -short -race 4 | env GOOS=linux GOARCH=386 go test ./... 5 | go vet 6 | 7 | testdeps: testdata/redis/src/redis-server 8 | 9 | bench: testdeps 10 | go test ./... -test.run=NONE -test.bench=. -test.benchmem 11 | 12 | .PHONY: all test testdeps bench 13 | 14 | testdata/redis: 15 | mkdir -p $@ 16 | wget -qO- https://github.com/antirez/redis/archive/unstable.tar.gz | tar xvz --strip-components=1 -C $@ 17 | 18 | testdata/redis/src/redis-server: testdata/redis 19 | sed -i.bak 's/libjemalloc.a/libjemalloc.a -lrt/g' $ 54 | } 55 | 56 | func ExampleClient() { 57 | err := client.Set("key", "value", 0).Err() 58 | if err != nil { 59 | panic(err) 60 | } 61 | 62 | val, err := client.Get("key").Result() 63 | if err != nil { 64 | panic(err) 65 | } 66 | fmt.Println("key", val) 67 | 68 | val2, err := client.Get("key2").Result() 69 | if err == redis.Nil { 70 | fmt.Println("key2 does not exist") 71 | } else if err != nil { 72 | panic(err) 73 | } else { 74 | fmt.Println("key2", val2) 75 | } 76 | // Output: key value 77 | // key2 does not exist 78 | } 79 | ``` 80 | 81 | ## Howto 82 | 83 | Please go through [examples](https://godoc.org/github.com/go-redis/redis#pkg-examples) to get an idea how to use this package. 84 | 85 | ## Look and feel 86 | 87 | Some corner cases: 88 | 89 | SET key value EX 10 NX 90 | set, err := client.SetNX("key", "value", 10*time.Second).Result() 91 | 92 | SORT list LIMIT 0 2 ASC 93 | vals, err := client.Sort("list", redis.Sort{Offset: 0, Count: 2, Order: "ASC"}).Result() 94 | 95 | ZRANGEBYSCORE zset -inf +inf WITHSCORES LIMIT 0 2 96 | vals, err := client.ZRangeByScoreWithScores("zset", redis.ZRangeBy{ 97 | Min: "-inf", 98 | Max: "+inf", 99 | Offset: 0, 100 | Count: 2, 101 | }).Result() 102 | 103 | ZINTERSTORE out 2 zset1 zset2 WEIGHTS 2 3 AGGREGATE SUM 104 | vals, err := client.ZInterStore("out", redis.ZStore{Weights: []int64{2, 3}}, "zset1", "zset2").Result() 105 | 106 | EVAL "return {KEYS[1],ARGV[1]}" 1 "key" "hello" 107 | vals, err := client.Eval("return {KEYS[1],ARGV[1]}", []string{"key"}, "hello").Result() 108 | 109 | ## Benchmark 110 | 111 | go-redis vs redigo: 112 | 113 | ``` 114 | BenchmarkSetGoRedis10Conns64Bytes-4 200000 7621 ns/op 210 B/op 6 allocs/op 115 | BenchmarkSetGoRedis100Conns64Bytes-4 200000 7554 ns/op 210 B/op 6 allocs/op 116 | BenchmarkSetGoRedis10Conns1KB-4 200000 7697 ns/op 210 B/op 6 allocs/op 117 | BenchmarkSetGoRedis100Conns1KB-4 200000 7688 ns/op 210 B/op 6 allocs/op 118 | BenchmarkSetGoRedis10Conns10KB-4 200000 9214 ns/op 210 B/op 6 allocs/op 119 | BenchmarkSetGoRedis100Conns10KB-4 200000 9181 ns/op 210 B/op 6 allocs/op 120 | BenchmarkSetGoRedis10Conns1MB-4 2000 583242 ns/op 2337 B/op 6 allocs/op 121 | BenchmarkSetGoRedis100Conns1MB-4 2000 583089 ns/op 2338 B/op 6 allocs/op 122 | BenchmarkSetRedigo10Conns64Bytes-4 200000 7576 ns/op 208 B/op 7 allocs/op 123 | BenchmarkSetRedigo100Conns64Bytes-4 200000 7782 ns/op 208 B/op 7 allocs/op 124 | BenchmarkSetRedigo10Conns1KB-4 200000 7958 ns/op 208 B/op 7 allocs/op 125 | BenchmarkSetRedigo100Conns1KB-4 200000 7725 ns/op 208 B/op 7 allocs/op 126 | BenchmarkSetRedigo10Conns10KB-4 100000 18442 ns/op 208 B/op 7 allocs/op 127 | BenchmarkSetRedigo100Conns10KB-4 100000 18818 ns/op 208 B/op 7 allocs/op 128 | BenchmarkSetRedigo10Conns1MB-4 2000 668829 ns/op 226 B/op 7 allocs/op 129 | BenchmarkSetRedigo100Conns1MB-4 2000 679542 ns/op 226 B/op 7 allocs/op 130 | ``` 131 | 132 | Redis Cluster: 133 | 134 | ``` 135 | BenchmarkRedisPing-4 200000 6983 ns/op 116 B/op 4 allocs/op 136 | BenchmarkRedisClusterPing-4 100000 11535 ns/op 117 B/op 4 allocs/op 137 | ``` 138 | 139 | ## See also 140 | 141 | - [Golang PostgreSQL ORM](https://github.com/go-pg/pg) 142 | - [Golang msgpack](https://github.com/vmihailenco/msgpack) 143 | - [Golang message task queue](https://github.com/go-msgqueue/msgqueue) 144 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/cluster_commands.go: -------------------------------------------------------------------------------- 1 | package redis 2 | 3 | import "sync/atomic" 4 | 5 | func (c *ClusterClient) DBSize() *IntCmd { 6 | cmd := NewIntCmd("dbsize") 7 | var size int64 8 | err := c.ForEachMaster(func(master *Client) error { 9 | n, err := master.DBSize().Result() 10 | if err != nil { 11 | return err 12 | } 13 | atomic.AddInt64(&size, n) 14 | return nil 15 | }) 16 | if err != nil { 17 | cmd.setErr(err) 18 | return cmd 19 | } 20 | cmd.val = size 21 | return cmd 22 | } 23 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/doc.go: -------------------------------------------------------------------------------- 1 | /* 2 | Package redis implements a Redis client. 3 | */ 4 | package redis 5 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/consistenthash/consistenthash.go: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright 2013 Google Inc. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"); 5 | you may not use this file except in compliance with the License. 6 | You may obtain a copy of the License at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | Unless required by applicable law or agreed to in writing, software 11 | distributed under the License is distributed on an "AS IS" BASIS, 12 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | See the License for the specific language governing permissions and 14 | limitations under the License. 15 | */ 16 | 17 | // Package consistenthash provides an implementation of a ring hash. 18 | package consistenthash 19 | 20 | import ( 21 | "hash/crc32" 22 | "sort" 23 | "strconv" 24 | ) 25 | 26 | type Hash func(data []byte) uint32 27 | 28 | type Map struct { 29 | hash Hash 30 | replicas int 31 | keys []int // Sorted 32 | hashMap map[int]string 33 | } 34 | 35 | func New(replicas int, fn Hash) *Map { 36 | m := &Map{ 37 | replicas: replicas, 38 | hash: fn, 39 | hashMap: make(map[int]string), 40 | } 41 | if m.hash == nil { 42 | m.hash = crc32.ChecksumIEEE 43 | } 44 | return m 45 | } 46 | 47 | // Returns true if there are no items available. 48 | func (m *Map) IsEmpty() bool { 49 | return len(m.keys) == 0 50 | } 51 | 52 | // Adds some keys to the hash. 53 | func (m *Map) Add(keys ...string) { 54 | for _, key := range keys { 55 | for i := 0; i < m.replicas; i++ { 56 | hash := int(m.hash([]byte(strconv.Itoa(i) + key))) 57 | m.keys = append(m.keys, hash) 58 | m.hashMap[hash] = key 59 | } 60 | } 61 | sort.Ints(m.keys) 62 | } 63 | 64 | // Gets the closest item in the hash to the provided key. 65 | func (m *Map) Get(key string) string { 66 | if m.IsEmpty() { 67 | return "" 68 | } 69 | 70 | hash := int(m.hash([]byte(key))) 71 | 72 | // Binary search for appropriate replica. 73 | idx := sort.Search(len(m.keys), func(i int) bool { return m.keys[i] >= hash }) 74 | 75 | // Means we have cycled back to the first replica. 76 | if idx == len(m.keys) { 77 | idx = 0 78 | } 79 | 80 | return m.hashMap[m.keys[idx]] 81 | } 82 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/error.go: -------------------------------------------------------------------------------- 1 | package internal 2 | 3 | import ( 4 | "io" 5 | "net" 6 | "strings" 7 | 8 | "github.com/go-redis/redis/internal/proto" 9 | ) 10 | 11 | func IsRetryableError(err error, retryNetError bool) bool { 12 | if IsNetworkError(err) { 13 | return retryNetError 14 | } 15 | s := err.Error() 16 | if s == "ERR max number of clients reached" { 17 | return true 18 | } 19 | if strings.HasPrefix(s, "LOADING ") { 20 | return true 21 | } 22 | if strings.HasPrefix(s, "CLUSTERDOWN ") { 23 | return true 24 | } 25 | return false 26 | } 27 | 28 | func IsRedisError(err error) bool { 29 | _, ok := err.(proto.RedisError) 30 | return ok 31 | } 32 | 33 | func IsNetworkError(err error) bool { 34 | if err == io.EOF { 35 | return true 36 | } 37 | _, ok := err.(net.Error) 38 | return ok 39 | } 40 | 41 | func IsReadOnlyError(err error) bool { 42 | return strings.HasPrefix(err.Error(), "READONLY ") 43 | } 44 | 45 | func IsBadConn(err error, allowTimeout bool) bool { 46 | if err == nil { 47 | return false 48 | } 49 | if IsRedisError(err) { 50 | return false 51 | } 52 | if allowTimeout { 53 | if netErr, ok := err.(net.Error); ok && netErr.Timeout() { 54 | return false 55 | } 56 | } 57 | return true 58 | } 59 | 60 | func IsMovedError(err error) (moved bool, ask bool, addr string) { 61 | if !IsRedisError(err) { 62 | return 63 | } 64 | 65 | s := err.Error() 66 | if strings.HasPrefix(s, "MOVED ") { 67 | moved = true 68 | } else if strings.HasPrefix(s, "ASK ") { 69 | ask = true 70 | } else { 71 | return 72 | } 73 | 74 | ind := strings.LastIndex(s, " ") 75 | if ind == -1 { 76 | return false, false, "" 77 | } 78 | addr = s[ind+1:] 79 | return 80 | } 81 | 82 | func IsLoadingError(err error) bool { 83 | return strings.HasPrefix(err.Error(), "LOADING ") 84 | } 85 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/hashtag/hashtag.go: -------------------------------------------------------------------------------- 1 | package hashtag 2 | 3 | import ( 4 | "math/rand" 5 | "strings" 6 | ) 7 | 8 | const SlotNumber = 16384 9 | 10 | // CRC16 implementation according to CCITT standards. 11 | // Copyright 2001-2010 Georges Menie (www.menie.org) 12 | // Copyright 2013 The Go Authors. All rights reserved. 13 | // http://redis.io/topics/cluster-spec#appendix-a-crc16-reference-implementation-in-ansi-c 14 | var crc16tab = [256]uint16{ 15 | 0x0000, 0x1021, 0x2042, 0x3063, 0x4084, 0x50a5, 0x60c6, 0x70e7, 16 | 0x8108, 0x9129, 0xa14a, 0xb16b, 0xc18c, 0xd1ad, 0xe1ce, 0xf1ef, 17 | 0x1231, 0x0210, 0x3273, 0x2252, 0x52b5, 0x4294, 0x72f7, 0x62d6, 18 | 0x9339, 0x8318, 0xb37b, 0xa35a, 0xd3bd, 0xc39c, 0xf3ff, 0xe3de, 19 | 0x2462, 0x3443, 0x0420, 0x1401, 0x64e6, 0x74c7, 0x44a4, 0x5485, 20 | 0xa56a, 0xb54b, 0x8528, 0x9509, 0xe5ee, 0xf5cf, 0xc5ac, 0xd58d, 21 | 0x3653, 0x2672, 0x1611, 0x0630, 0x76d7, 0x66f6, 0x5695, 0x46b4, 22 | 0xb75b, 0xa77a, 0x9719, 0x8738, 0xf7df, 0xe7fe, 0xd79d, 0xc7bc, 23 | 0x48c4, 0x58e5, 0x6886, 0x78a7, 0x0840, 0x1861, 0x2802, 0x3823, 24 | 0xc9cc, 0xd9ed, 0xe98e, 0xf9af, 0x8948, 0x9969, 0xa90a, 0xb92b, 25 | 0x5af5, 0x4ad4, 0x7ab7, 0x6a96, 0x1a71, 0x0a50, 0x3a33, 0x2a12, 26 | 0xdbfd, 0xcbdc, 0xfbbf, 0xeb9e, 0x9b79, 0x8b58, 0xbb3b, 0xab1a, 27 | 0x6ca6, 0x7c87, 0x4ce4, 0x5cc5, 0x2c22, 0x3c03, 0x0c60, 0x1c41, 28 | 0xedae, 0xfd8f, 0xcdec, 0xddcd, 0xad2a, 0xbd0b, 0x8d68, 0x9d49, 29 | 0x7e97, 0x6eb6, 0x5ed5, 0x4ef4, 0x3e13, 0x2e32, 0x1e51, 0x0e70, 30 | 0xff9f, 0xefbe, 0xdfdd, 0xcffc, 0xbf1b, 0xaf3a, 0x9f59, 0x8f78, 31 | 0x9188, 0x81a9, 0xb1ca, 0xa1eb, 0xd10c, 0xc12d, 0xf14e, 0xe16f, 32 | 0x1080, 0x00a1, 0x30c2, 0x20e3, 0x5004, 0x4025, 0x7046, 0x6067, 33 | 0x83b9, 0x9398, 0xa3fb, 0xb3da, 0xc33d, 0xd31c, 0xe37f, 0xf35e, 34 | 0x02b1, 0x1290, 0x22f3, 0x32d2, 0x4235, 0x5214, 0x6277, 0x7256, 35 | 0xb5ea, 0xa5cb, 0x95a8, 0x8589, 0xf56e, 0xe54f, 0xd52c, 0xc50d, 36 | 0x34e2, 0x24c3, 0x14a0, 0x0481, 0x7466, 0x6447, 0x5424, 0x4405, 37 | 0xa7db, 0xb7fa, 0x8799, 0x97b8, 0xe75f, 0xf77e, 0xc71d, 0xd73c, 38 | 0x26d3, 0x36f2, 0x0691, 0x16b0, 0x6657, 0x7676, 0x4615, 0x5634, 39 | 0xd94c, 0xc96d, 0xf90e, 0xe92f, 0x99c8, 0x89e9, 0xb98a, 0xa9ab, 40 | 0x5844, 0x4865, 0x7806, 0x6827, 0x18c0, 0x08e1, 0x3882, 0x28a3, 41 | 0xcb7d, 0xdb5c, 0xeb3f, 0xfb1e, 0x8bf9, 0x9bd8, 0xabbb, 0xbb9a, 42 | 0x4a75, 0x5a54, 0x6a37, 0x7a16, 0x0af1, 0x1ad0, 0x2ab3, 0x3a92, 43 | 0xfd2e, 0xed0f, 0xdd6c, 0xcd4d, 0xbdaa, 0xad8b, 0x9de8, 0x8dc9, 44 | 0x7c26, 0x6c07, 0x5c64, 0x4c45, 0x3ca2, 0x2c83, 0x1ce0, 0x0cc1, 45 | 0xef1f, 0xff3e, 0xcf5d, 0xdf7c, 0xaf9b, 0xbfba, 0x8fd9, 0x9ff8, 46 | 0x6e17, 0x7e36, 0x4e55, 0x5e74, 0x2e93, 0x3eb2, 0x0ed1, 0x1ef0, 47 | } 48 | 49 | func Key(key string) string { 50 | if s := strings.IndexByte(key, '{'); s > -1 { 51 | if e := strings.IndexByte(key[s+1:], '}'); e > 0 { 52 | return key[s+1 : s+e+1] 53 | } 54 | } 55 | return key 56 | } 57 | 58 | func RandomSlot() int { 59 | return rand.Intn(SlotNumber) 60 | } 61 | 62 | // hashSlot returns a consistent slot number between 0 and 16383 63 | // for any given string key. 64 | func Slot(key string) int { 65 | if key == "" { 66 | return RandomSlot() 67 | } 68 | key = Key(key) 69 | return int(crc16sum(key)) % SlotNumber 70 | } 71 | 72 | func crc16sum(key string) (crc uint16) { 73 | for i := 0; i < len(key); i++ { 74 | crc = (crc << 8) ^ crc16tab[(byte(crc>>8)^key[i])&0x00ff] 75 | } 76 | return 77 | } 78 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/internal.go: -------------------------------------------------------------------------------- 1 | package internal 2 | 3 | import ( 4 | "math/rand" 5 | "time" 6 | ) 7 | 8 | // Retry backoff with jitter sleep to prevent overloaded conditions during intervals 9 | // https://www.awsarchitectureblog.com/2015/03/backoff.html 10 | func RetryBackoff(retry int, minBackoff, maxBackoff time.Duration) time.Duration { 11 | if retry < 0 { 12 | retry = 0 13 | } 14 | 15 | backoff := minBackoff << uint(retry) 16 | if backoff > maxBackoff || backoff < minBackoff { 17 | backoff = maxBackoff 18 | } 19 | 20 | if backoff == 0 { 21 | return 0 22 | } 23 | return time.Duration(rand.Int63n(int64(backoff))) 24 | } 25 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/log.go: -------------------------------------------------------------------------------- 1 | package internal 2 | 3 | import ( 4 | "fmt" 5 | "log" 6 | ) 7 | 8 | var Logger *log.Logger 9 | 10 | func Logf(s string, args ...interface{}) { 11 | if Logger == nil { 12 | return 13 | } 14 | Logger.Output(2, fmt.Sprintf(s, args...)) 15 | } 16 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/once.go: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright 2014 The Camlistore Authors 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"); 5 | you may not use this file except in compliance with the License. 6 | You may obtain a copy of the License at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | Unless required by applicable law or agreed to in writing, software 11 | distributed under the License is distributed on an "AS IS" BASIS, 12 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | See the License for the specific language governing permissions and 14 | limitations under the License. 15 | */ 16 | 17 | package internal 18 | 19 | import ( 20 | "sync" 21 | "sync/atomic" 22 | ) 23 | 24 | // A Once will perform a successful action exactly once. 25 | // 26 | // Unlike a sync.Once, this Once's func returns an error 27 | // and is re-armed on failure. 28 | type Once struct { 29 | m sync.Mutex 30 | done uint32 31 | } 32 | 33 | // Do calls the function f if and only if Do has not been invoked 34 | // without error for this instance of Once. In other words, given 35 | // var once Once 36 | // if once.Do(f) is called multiple times, only the first call will 37 | // invoke f, even if f has a different value in each invocation unless 38 | // f returns an error. A new instance of Once is required for each 39 | // function to execute. 40 | // 41 | // Do is intended for initialization that must be run exactly once. Since f 42 | // is niladic, it may be necessary to use a function literal to capture the 43 | // arguments to a function to be invoked by Do: 44 | // err := config.once.Do(func() error { return config.init(filename) }) 45 | func (o *Once) Do(f func() error) error { 46 | if atomic.LoadUint32(&o.done) == 1 { 47 | return nil 48 | } 49 | // Slow-path. 50 | o.m.Lock() 51 | defer o.m.Unlock() 52 | var err error 53 | if o.done == 0 { 54 | err = f() 55 | if err == nil { 56 | atomic.StoreUint32(&o.done, 1) 57 | } 58 | } 59 | return err 60 | } 61 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/pool/conn.go: -------------------------------------------------------------------------------- 1 | package pool 2 | 3 | import ( 4 | "net" 5 | "sync/atomic" 6 | "time" 7 | 8 | "github.com/go-redis/redis/internal/proto" 9 | ) 10 | 11 | var noDeadline = time.Time{} 12 | 13 | type Conn struct { 14 | netConn net.Conn 15 | 16 | Rd *proto.Reader 17 | Wb *proto.WriteBuffer 18 | 19 | Inited bool 20 | usedAt atomic.Value 21 | } 22 | 23 | func NewConn(netConn net.Conn) *Conn { 24 | cn := &Conn{ 25 | netConn: netConn, 26 | Wb: proto.NewWriteBuffer(), 27 | } 28 | cn.Rd = proto.NewReader(cn.netConn) 29 | cn.SetUsedAt(time.Now()) 30 | return cn 31 | } 32 | 33 | func (cn *Conn) UsedAt() time.Time { 34 | return cn.usedAt.Load().(time.Time) 35 | } 36 | 37 | func (cn *Conn) SetUsedAt(tm time.Time) { 38 | cn.usedAt.Store(tm) 39 | } 40 | 41 | func (cn *Conn) SetNetConn(netConn net.Conn) { 42 | cn.netConn = netConn 43 | cn.Rd.Reset(netConn) 44 | } 45 | 46 | func (cn *Conn) IsStale(timeout time.Duration) bool { 47 | return timeout > 0 && time.Since(cn.UsedAt()) > timeout 48 | } 49 | 50 | func (cn *Conn) SetReadTimeout(timeout time.Duration) error { 51 | now := time.Now() 52 | cn.SetUsedAt(now) 53 | if timeout > 0 { 54 | return cn.netConn.SetReadDeadline(now.Add(timeout)) 55 | } 56 | return cn.netConn.SetReadDeadline(noDeadline) 57 | } 58 | 59 | func (cn *Conn) SetWriteTimeout(timeout time.Duration) error { 60 | now := time.Now() 61 | cn.SetUsedAt(now) 62 | if timeout > 0 { 63 | return cn.netConn.SetWriteDeadline(now.Add(timeout)) 64 | } 65 | return cn.netConn.SetWriteDeadline(noDeadline) 66 | } 67 | 68 | func (cn *Conn) Write(b []byte) (int, error) { 69 | return cn.netConn.Write(b) 70 | } 71 | 72 | func (cn *Conn) RemoteAddr() net.Addr { 73 | return cn.netConn.RemoteAddr() 74 | } 75 | 76 | func (cn *Conn) Close() error { 77 | return cn.netConn.Close() 78 | } 79 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/pool/pool.go: -------------------------------------------------------------------------------- 1 | package pool 2 | 3 | import ( 4 | "errors" 5 | "net" 6 | "sync" 7 | "sync/atomic" 8 | "time" 9 | 10 | "github.com/go-redis/redis/internal" 11 | ) 12 | 13 | var ErrClosed = errors.New("redis: client is closed") 14 | var ErrPoolTimeout = errors.New("redis: connection pool timeout") 15 | 16 | var timers = sync.Pool{ 17 | New: func() interface{} { 18 | t := time.NewTimer(time.Hour) 19 | t.Stop() 20 | return t 21 | }, 22 | } 23 | 24 | // Stats contains pool state information and accumulated stats. 25 | type Stats struct { 26 | Hits uint32 // number of times free connection was found in the pool 27 | Misses uint32 // number of times free connection was NOT found in the pool 28 | Timeouts uint32 // number of times a wait timeout occurred 29 | 30 | TotalConns uint32 // number of total connections in the pool 31 | FreeConns uint32 // number of free connections in the pool 32 | StaleConns uint32 // number of stale connections removed from the pool 33 | } 34 | 35 | type Pooler interface { 36 | NewConn() (*Conn, error) 37 | CloseConn(*Conn) error 38 | 39 | Get() (*Conn, bool, error) 40 | Put(*Conn) error 41 | Remove(*Conn) error 42 | 43 | Len() int 44 | FreeLen() int 45 | Stats() *Stats 46 | 47 | Close() error 48 | } 49 | 50 | type Options struct { 51 | Dialer func() (net.Conn, error) 52 | OnClose func(*Conn) error 53 | 54 | PoolSize int 55 | PoolTimeout time.Duration 56 | IdleTimeout time.Duration 57 | IdleCheckFrequency time.Duration 58 | } 59 | 60 | type ConnPool struct { 61 | opt *Options 62 | 63 | dialErrorsNum uint32 // atomic 64 | 65 | lastDialError error 66 | lastDialErrorMu sync.RWMutex 67 | 68 | queue chan struct{} 69 | 70 | connsMu sync.Mutex 71 | conns []*Conn 72 | 73 | freeConnsMu sync.Mutex 74 | freeConns []*Conn 75 | 76 | stats Stats 77 | 78 | _closed uint32 // atomic 79 | } 80 | 81 | var _ Pooler = (*ConnPool)(nil) 82 | 83 | func NewConnPool(opt *Options) *ConnPool { 84 | p := &ConnPool{ 85 | opt: opt, 86 | 87 | queue: make(chan struct{}, opt.PoolSize), 88 | conns: make([]*Conn, 0, opt.PoolSize), 89 | freeConns: make([]*Conn, 0, opt.PoolSize), 90 | } 91 | if opt.IdleTimeout > 0 && opt.IdleCheckFrequency > 0 { 92 | go p.reaper(opt.IdleCheckFrequency) 93 | } 94 | return p 95 | } 96 | 97 | func (p *ConnPool) NewConn() (*Conn, error) { 98 | if p.closed() { 99 | return nil, ErrClosed 100 | } 101 | 102 | if atomic.LoadUint32(&p.dialErrorsNum) >= uint32(p.opt.PoolSize) { 103 | return nil, p.getLastDialError() 104 | } 105 | 106 | netConn, err := p.opt.Dialer() 107 | if err != nil { 108 | p.setLastDialError(err) 109 | if atomic.AddUint32(&p.dialErrorsNum, 1) == uint32(p.opt.PoolSize) { 110 | go p.tryDial() 111 | } 112 | return nil, err 113 | } 114 | 115 | cn := NewConn(netConn) 116 | p.connsMu.Lock() 117 | p.conns = append(p.conns, cn) 118 | p.connsMu.Unlock() 119 | 120 | return cn, nil 121 | } 122 | 123 | func (p *ConnPool) tryDial() { 124 | for { 125 | if p.closed() { 126 | return 127 | } 128 | 129 | conn, err := p.opt.Dialer() 130 | if err != nil { 131 | p.setLastDialError(err) 132 | time.Sleep(time.Second) 133 | continue 134 | } 135 | 136 | atomic.StoreUint32(&p.dialErrorsNum, 0) 137 | _ = conn.Close() 138 | return 139 | } 140 | } 141 | 142 | func (p *ConnPool) setLastDialError(err error) { 143 | p.lastDialErrorMu.Lock() 144 | p.lastDialError = err 145 | p.lastDialErrorMu.Unlock() 146 | } 147 | 148 | func (p *ConnPool) getLastDialError() error { 149 | p.lastDialErrorMu.RLock() 150 | err := p.lastDialError 151 | p.lastDialErrorMu.RUnlock() 152 | return err 153 | } 154 | 155 | // Get returns existed connection from the pool or creates a new one. 156 | func (p *ConnPool) Get() (*Conn, bool, error) { 157 | if p.closed() { 158 | return nil, false, ErrClosed 159 | } 160 | 161 | select { 162 | case p.queue <- struct{}{}: 163 | default: 164 | timer := timers.Get().(*time.Timer) 165 | timer.Reset(p.opt.PoolTimeout) 166 | 167 | select { 168 | case p.queue <- struct{}{}: 169 | if !timer.Stop() { 170 | <-timer.C 171 | } 172 | timers.Put(timer) 173 | case <-timer.C: 174 | timers.Put(timer) 175 | atomic.AddUint32(&p.stats.Timeouts, 1) 176 | return nil, false, ErrPoolTimeout 177 | } 178 | } 179 | 180 | for { 181 | p.freeConnsMu.Lock() 182 | cn := p.popFree() 183 | p.freeConnsMu.Unlock() 184 | 185 | if cn == nil { 186 | break 187 | } 188 | 189 | if cn.IsStale(p.opt.IdleTimeout) { 190 | p.CloseConn(cn) 191 | continue 192 | } 193 | 194 | atomic.AddUint32(&p.stats.Hits, 1) 195 | return cn, false, nil 196 | } 197 | 198 | atomic.AddUint32(&p.stats.Misses, 1) 199 | 200 | newcn, err := p.NewConn() 201 | if err != nil { 202 | <-p.queue 203 | return nil, false, err 204 | } 205 | 206 | return newcn, true, nil 207 | } 208 | 209 | func (p *ConnPool) popFree() *Conn { 210 | if len(p.freeConns) == 0 { 211 | return nil 212 | } 213 | 214 | idx := len(p.freeConns) - 1 215 | cn := p.freeConns[idx] 216 | p.freeConns = p.freeConns[:idx] 217 | return cn 218 | } 219 | 220 | func (p *ConnPool) Put(cn *Conn) error { 221 | if data := cn.Rd.PeekBuffered(); data != nil { 222 | internal.Logf("connection has unread data: %q", data) 223 | return p.Remove(cn) 224 | } 225 | p.freeConnsMu.Lock() 226 | p.freeConns = append(p.freeConns, cn) 227 | p.freeConnsMu.Unlock() 228 | <-p.queue 229 | return nil 230 | } 231 | 232 | func (p *ConnPool) Remove(cn *Conn) error { 233 | _ = p.CloseConn(cn) 234 | <-p.queue 235 | return nil 236 | } 237 | 238 | func (p *ConnPool) CloseConn(cn *Conn) error { 239 | p.connsMu.Lock() 240 | for i, c := range p.conns { 241 | if c == cn { 242 | p.conns = append(p.conns[:i], p.conns[i+1:]...) 243 | break 244 | } 245 | } 246 | p.connsMu.Unlock() 247 | 248 | return p.closeConn(cn) 249 | } 250 | 251 | func (p *ConnPool) closeConn(cn *Conn) error { 252 | if p.opt.OnClose != nil { 253 | _ = p.opt.OnClose(cn) 254 | } 255 | return cn.Close() 256 | } 257 | 258 | // Len returns total number of connections. 259 | func (p *ConnPool) Len() int { 260 | p.connsMu.Lock() 261 | l := len(p.conns) 262 | p.connsMu.Unlock() 263 | return l 264 | } 265 | 266 | // FreeLen returns number of free connections. 267 | func (p *ConnPool) FreeLen() int { 268 | p.freeConnsMu.Lock() 269 | l := len(p.freeConns) 270 | p.freeConnsMu.Unlock() 271 | return l 272 | } 273 | 274 | func (p *ConnPool) Stats() *Stats { 275 | return &Stats{ 276 | Hits: atomic.LoadUint32(&p.stats.Hits), 277 | Misses: atomic.LoadUint32(&p.stats.Misses), 278 | Timeouts: atomic.LoadUint32(&p.stats.Timeouts), 279 | 280 | TotalConns: uint32(p.Len()), 281 | FreeConns: uint32(p.FreeLen()), 282 | StaleConns: atomic.LoadUint32(&p.stats.StaleConns), 283 | } 284 | } 285 | 286 | func (p *ConnPool) closed() bool { 287 | return atomic.LoadUint32(&p._closed) == 1 288 | } 289 | 290 | func (p *ConnPool) Filter(fn func(*Conn) bool) error { 291 | var firstErr error 292 | p.connsMu.Lock() 293 | for _, cn := range p.conns { 294 | if fn(cn) { 295 | if err := p.closeConn(cn); err != nil && firstErr == nil { 296 | firstErr = err 297 | } 298 | } 299 | } 300 | p.connsMu.Unlock() 301 | return firstErr 302 | } 303 | 304 | func (p *ConnPool) Close() error { 305 | if !atomic.CompareAndSwapUint32(&p._closed, 0, 1) { 306 | return ErrClosed 307 | } 308 | 309 | var firstErr error 310 | p.connsMu.Lock() 311 | for _, cn := range p.conns { 312 | if err := p.closeConn(cn); err != nil && firstErr == nil { 313 | firstErr = err 314 | } 315 | } 316 | p.conns = nil 317 | p.connsMu.Unlock() 318 | 319 | p.freeConnsMu.Lock() 320 | p.freeConns = nil 321 | p.freeConnsMu.Unlock() 322 | 323 | return firstErr 324 | } 325 | 326 | func (p *ConnPool) reapStaleConn() bool { 327 | if len(p.freeConns) == 0 { 328 | return false 329 | } 330 | 331 | cn := p.freeConns[0] 332 | if !cn.IsStale(p.opt.IdleTimeout) { 333 | return false 334 | } 335 | 336 | p.CloseConn(cn) 337 | p.freeConns = append(p.freeConns[:0], p.freeConns[1:]...) 338 | 339 | return true 340 | } 341 | 342 | func (p *ConnPool) ReapStaleConns() (int, error) { 343 | var n int 344 | for { 345 | p.queue <- struct{}{} 346 | p.freeConnsMu.Lock() 347 | 348 | reaped := p.reapStaleConn() 349 | 350 | p.freeConnsMu.Unlock() 351 | <-p.queue 352 | 353 | if reaped { 354 | n++ 355 | } else { 356 | break 357 | } 358 | } 359 | return n, nil 360 | } 361 | 362 | func (p *ConnPool) reaper(frequency time.Duration) { 363 | ticker := time.NewTicker(frequency) 364 | defer ticker.Stop() 365 | 366 | for range ticker.C { 367 | if p.closed() { 368 | break 369 | } 370 | n, err := p.ReapStaleConns() 371 | if err != nil { 372 | internal.Logf("ReapStaleConns failed: %s", err) 373 | continue 374 | } 375 | atomic.AddUint32(&p.stats.StaleConns, uint32(n)) 376 | } 377 | } 378 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/pool/pool_single.go: -------------------------------------------------------------------------------- 1 | package pool 2 | 3 | type SingleConnPool struct { 4 | cn *Conn 5 | } 6 | 7 | var _ Pooler = (*SingleConnPool)(nil) 8 | 9 | func NewSingleConnPool(cn *Conn) *SingleConnPool { 10 | return &SingleConnPool{ 11 | cn: cn, 12 | } 13 | } 14 | 15 | func (p *SingleConnPool) NewConn() (*Conn, error) { 16 | panic("not implemented") 17 | } 18 | 19 | func (p *SingleConnPool) CloseConn(*Conn) error { 20 | panic("not implemented") 21 | } 22 | 23 | func (p *SingleConnPool) Get() (*Conn, bool, error) { 24 | return p.cn, false, nil 25 | } 26 | 27 | func (p *SingleConnPool) Put(cn *Conn) error { 28 | if p.cn != cn { 29 | panic("p.cn != cn") 30 | } 31 | return nil 32 | } 33 | 34 | func (p *SingleConnPool) Remove(cn *Conn) error { 35 | if p.cn != cn { 36 | panic("p.cn != cn") 37 | } 38 | return nil 39 | } 40 | 41 | func (p *SingleConnPool) Len() int { 42 | return 1 43 | } 44 | 45 | func (p *SingleConnPool) FreeLen() int { 46 | return 0 47 | } 48 | 49 | func (p *SingleConnPool) Stats() *Stats { 50 | return nil 51 | } 52 | 53 | func (p *SingleConnPool) Close() error { 54 | return nil 55 | } 56 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/pool/pool_sticky.go: -------------------------------------------------------------------------------- 1 | package pool 2 | 3 | import "sync" 4 | 5 | type StickyConnPool struct { 6 | pool *ConnPool 7 | reusable bool 8 | 9 | cn *Conn 10 | closed bool 11 | mu sync.Mutex 12 | } 13 | 14 | var _ Pooler = (*StickyConnPool)(nil) 15 | 16 | func NewStickyConnPool(pool *ConnPool, reusable bool) *StickyConnPool { 17 | return &StickyConnPool{ 18 | pool: pool, 19 | reusable: reusable, 20 | } 21 | } 22 | 23 | func (p *StickyConnPool) NewConn() (*Conn, error) { 24 | panic("not implemented") 25 | } 26 | 27 | func (p *StickyConnPool) CloseConn(*Conn) error { 28 | panic("not implemented") 29 | } 30 | 31 | func (p *StickyConnPool) Get() (*Conn, bool, error) { 32 | p.mu.Lock() 33 | defer p.mu.Unlock() 34 | 35 | if p.closed { 36 | return nil, false, ErrClosed 37 | } 38 | if p.cn != nil { 39 | return p.cn, false, nil 40 | } 41 | 42 | cn, _, err := p.pool.Get() 43 | if err != nil { 44 | return nil, false, err 45 | } 46 | p.cn = cn 47 | return cn, true, nil 48 | } 49 | 50 | func (p *StickyConnPool) putUpstream() (err error) { 51 | err = p.pool.Put(p.cn) 52 | p.cn = nil 53 | return err 54 | } 55 | 56 | func (p *StickyConnPool) Put(cn *Conn) error { 57 | p.mu.Lock() 58 | defer p.mu.Unlock() 59 | 60 | if p.closed { 61 | return ErrClosed 62 | } 63 | return nil 64 | } 65 | 66 | func (p *StickyConnPool) removeUpstream() error { 67 | err := p.pool.Remove(p.cn) 68 | p.cn = nil 69 | return err 70 | } 71 | 72 | func (p *StickyConnPool) Remove(cn *Conn) error { 73 | p.mu.Lock() 74 | defer p.mu.Unlock() 75 | 76 | if p.closed { 77 | return nil 78 | } 79 | return p.removeUpstream() 80 | } 81 | 82 | func (p *StickyConnPool) Len() int { 83 | p.mu.Lock() 84 | defer p.mu.Unlock() 85 | 86 | if p.cn == nil { 87 | return 0 88 | } 89 | return 1 90 | } 91 | 92 | func (p *StickyConnPool) FreeLen() int { 93 | p.mu.Lock() 94 | defer p.mu.Unlock() 95 | 96 | if p.cn == nil { 97 | return 1 98 | } 99 | return 0 100 | } 101 | 102 | func (p *StickyConnPool) Stats() *Stats { 103 | return nil 104 | } 105 | 106 | func (p *StickyConnPool) Close() error { 107 | p.mu.Lock() 108 | defer p.mu.Unlock() 109 | 110 | if p.closed { 111 | return ErrClosed 112 | } 113 | p.closed = true 114 | var err error 115 | if p.cn != nil { 116 | if p.reusable { 117 | err = p.putUpstream() 118 | } else { 119 | err = p.removeUpstream() 120 | } 121 | } 122 | return err 123 | } 124 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/proto/reader.go: -------------------------------------------------------------------------------- 1 | package proto 2 | 3 | import ( 4 | "bufio" 5 | "fmt" 6 | "io" 7 | "strconv" 8 | 9 | "github.com/go-redis/redis/internal/util" 10 | ) 11 | 12 | const bytesAllocLimit = 1024 * 1024 // 1mb 13 | 14 | const ( 15 | ErrorReply = '-' 16 | StatusReply = '+' 17 | IntReply = ':' 18 | StringReply = '$' 19 | ArrayReply = '*' 20 | ) 21 | 22 | //------------------------------------------------------------------------------ 23 | 24 | const Nil = RedisError("redis: nil") 25 | 26 | type RedisError string 27 | 28 | func (e RedisError) Error() string { return string(e) } 29 | 30 | //------------------------------------------------------------------------------ 31 | 32 | type MultiBulkParse func(*Reader, int64) (interface{}, error) 33 | 34 | type Reader struct { 35 | src *bufio.Reader 36 | buf []byte 37 | } 38 | 39 | func NewReader(rd io.Reader) *Reader { 40 | return &Reader{ 41 | src: bufio.NewReader(rd), 42 | buf: make([]byte, 4096), 43 | } 44 | } 45 | 46 | func (r *Reader) Reset(rd io.Reader) { 47 | r.src.Reset(rd) 48 | } 49 | 50 | func (r *Reader) PeekBuffered() []byte { 51 | if n := r.src.Buffered(); n != 0 { 52 | b, _ := r.src.Peek(n) 53 | return b 54 | } 55 | return nil 56 | } 57 | 58 | func (r *Reader) ReadN(n int) ([]byte, error) { 59 | b, err := readN(r.src, r.buf, n) 60 | if err != nil { 61 | return nil, err 62 | } 63 | r.buf = b 64 | return b, nil 65 | } 66 | 67 | func (r *Reader) ReadLine() ([]byte, error) { 68 | line, isPrefix, err := r.src.ReadLine() 69 | if err != nil { 70 | return nil, err 71 | } 72 | if isPrefix { 73 | return nil, bufio.ErrBufferFull 74 | } 75 | if len(line) == 0 { 76 | return nil, fmt.Errorf("redis: reply is empty") 77 | } 78 | if isNilReply(line) { 79 | return nil, Nil 80 | } 81 | return line, nil 82 | } 83 | 84 | func (r *Reader) ReadReply(m MultiBulkParse) (interface{}, error) { 85 | line, err := r.ReadLine() 86 | if err != nil { 87 | return nil, err 88 | } 89 | 90 | switch line[0] { 91 | case ErrorReply: 92 | return nil, ParseErrorReply(line) 93 | case StatusReply: 94 | return parseStatusValue(line), nil 95 | case IntReply: 96 | return util.ParseInt(line[1:], 10, 64) 97 | case StringReply: 98 | return r.readTmpBytesValue(line) 99 | case ArrayReply: 100 | n, err := parseArrayLen(line) 101 | if err != nil { 102 | return nil, err 103 | } 104 | return m(r, n) 105 | } 106 | return nil, fmt.Errorf("redis: can't parse %.100q", line) 107 | } 108 | 109 | func (r *Reader) ReadIntReply() (int64, error) { 110 | line, err := r.ReadLine() 111 | if err != nil { 112 | return 0, err 113 | } 114 | switch line[0] { 115 | case ErrorReply: 116 | return 0, ParseErrorReply(line) 117 | case IntReply: 118 | return util.ParseInt(line[1:], 10, 64) 119 | default: 120 | return 0, fmt.Errorf("redis: can't parse int reply: %.100q", line) 121 | } 122 | } 123 | 124 | func (r *Reader) ReadTmpBytesReply() ([]byte, error) { 125 | line, err := r.ReadLine() 126 | if err != nil { 127 | return nil, err 128 | } 129 | switch line[0] { 130 | case ErrorReply: 131 | return nil, ParseErrorReply(line) 132 | case StringReply: 133 | return r.readTmpBytesValue(line) 134 | case StatusReply: 135 | return parseStatusValue(line), nil 136 | default: 137 | return nil, fmt.Errorf("redis: can't parse string reply: %.100q", line) 138 | } 139 | } 140 | 141 | func (r *Reader) ReadBytesReply() ([]byte, error) { 142 | b, err := r.ReadTmpBytesReply() 143 | if err != nil { 144 | return nil, err 145 | } 146 | cp := make([]byte, len(b)) 147 | copy(cp, b) 148 | return cp, nil 149 | } 150 | 151 | func (r *Reader) ReadStringReply() (string, error) { 152 | b, err := r.ReadTmpBytesReply() 153 | if err != nil { 154 | return "", err 155 | } 156 | return string(b), nil 157 | } 158 | 159 | func (r *Reader) ReadFloatReply() (float64, error) { 160 | b, err := r.ReadTmpBytesReply() 161 | if err != nil { 162 | return 0, err 163 | } 164 | return util.ParseFloat(b, 64) 165 | } 166 | 167 | func (r *Reader) ReadArrayReply(m MultiBulkParse) (interface{}, error) { 168 | line, err := r.ReadLine() 169 | if err != nil { 170 | return nil, err 171 | } 172 | switch line[0] { 173 | case ErrorReply: 174 | return nil, ParseErrorReply(line) 175 | case ArrayReply: 176 | n, err := parseArrayLen(line) 177 | if err != nil { 178 | return nil, err 179 | } 180 | return m(r, n) 181 | default: 182 | return nil, fmt.Errorf("redis: can't parse array reply: %.100q", line) 183 | } 184 | } 185 | 186 | func (r *Reader) ReadArrayLen() (int64, error) { 187 | line, err := r.ReadLine() 188 | if err != nil { 189 | return 0, err 190 | } 191 | switch line[0] { 192 | case ErrorReply: 193 | return 0, ParseErrorReply(line) 194 | case ArrayReply: 195 | return parseArrayLen(line) 196 | default: 197 | return 0, fmt.Errorf("redis: can't parse array reply: %.100q", line) 198 | } 199 | } 200 | 201 | func (r *Reader) ReadScanReply() ([]string, uint64, error) { 202 | n, err := r.ReadArrayLen() 203 | if err != nil { 204 | return nil, 0, err 205 | } 206 | if n != 2 { 207 | return nil, 0, fmt.Errorf("redis: got %d elements in scan reply, expected 2", n) 208 | } 209 | 210 | cursor, err := r.ReadUint() 211 | if err != nil { 212 | return nil, 0, err 213 | } 214 | 215 | n, err = r.ReadArrayLen() 216 | if err != nil { 217 | return nil, 0, err 218 | } 219 | 220 | keys := make([]string, n) 221 | for i := int64(0); i < n; i++ { 222 | key, err := r.ReadStringReply() 223 | if err != nil { 224 | return nil, 0, err 225 | } 226 | keys[i] = key 227 | } 228 | 229 | return keys, cursor, err 230 | } 231 | 232 | func (r *Reader) readTmpBytesValue(line []byte) ([]byte, error) { 233 | if isNilReply(line) { 234 | return nil, Nil 235 | } 236 | 237 | replyLen, err := strconv.Atoi(string(line[1:])) 238 | if err != nil { 239 | return nil, err 240 | } 241 | 242 | b, err := r.ReadN(replyLen + 2) 243 | if err != nil { 244 | return nil, err 245 | } 246 | return b[:replyLen], nil 247 | } 248 | 249 | func (r *Reader) ReadInt() (int64, error) { 250 | b, err := r.ReadTmpBytesReply() 251 | if err != nil { 252 | return 0, err 253 | } 254 | return util.ParseInt(b, 10, 64) 255 | } 256 | 257 | func (r *Reader) ReadUint() (uint64, error) { 258 | b, err := r.ReadTmpBytesReply() 259 | if err != nil { 260 | return 0, err 261 | } 262 | return util.ParseUint(b, 10, 64) 263 | } 264 | 265 | // -------------------------------------------------------------------- 266 | 267 | func readN(r io.Reader, b []byte, n int) ([]byte, error) { 268 | if n == 0 && b == nil { 269 | return make([]byte, 0), nil 270 | } 271 | 272 | if cap(b) >= n { 273 | b = b[:n] 274 | _, err := io.ReadFull(r, b) 275 | return b, err 276 | } 277 | b = b[:cap(b)] 278 | 279 | pos := 0 280 | for pos < n { 281 | diff := n - len(b) 282 | if diff > bytesAllocLimit { 283 | diff = bytesAllocLimit 284 | } 285 | b = append(b, make([]byte, diff)...) 286 | 287 | nn, err := io.ReadFull(r, b[pos:]) 288 | if err != nil { 289 | return nil, err 290 | } 291 | pos += nn 292 | } 293 | 294 | return b, nil 295 | } 296 | 297 | func formatInt(n int64) string { 298 | return strconv.FormatInt(n, 10) 299 | } 300 | 301 | func formatUint(u uint64) string { 302 | return strconv.FormatUint(u, 10) 303 | } 304 | 305 | func formatFloat(f float64) string { 306 | return strconv.FormatFloat(f, 'f', -1, 64) 307 | } 308 | 309 | func isNilReply(b []byte) bool { 310 | return len(b) == 3 && 311 | (b[0] == StringReply || b[0] == ArrayReply) && 312 | b[1] == '-' && b[2] == '1' 313 | } 314 | 315 | func ParseErrorReply(line []byte) error { 316 | return RedisError(string(line[1:])) 317 | } 318 | 319 | func parseStatusValue(line []byte) []byte { 320 | return line[1:] 321 | } 322 | 323 | func parseArrayLen(line []byte) (int64, error) { 324 | if isNilReply(line) { 325 | return 0, Nil 326 | } 327 | return util.ParseInt(line[1:], 10, 64) 328 | } 329 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/proto/scan.go: -------------------------------------------------------------------------------- 1 | package proto 2 | 3 | import ( 4 | "encoding" 5 | "fmt" 6 | "reflect" 7 | 8 | "github.com/go-redis/redis/internal/util" 9 | ) 10 | 11 | func Scan(b []byte, v interface{}) error { 12 | switch v := v.(type) { 13 | case nil: 14 | return fmt.Errorf("redis: Scan(nil)") 15 | case *string: 16 | *v = util.BytesToString(b) 17 | return nil 18 | case *[]byte: 19 | *v = b 20 | return nil 21 | case *int: 22 | var err error 23 | *v, err = util.Atoi(b) 24 | return err 25 | case *int8: 26 | n, err := util.ParseInt(b, 10, 8) 27 | if err != nil { 28 | return err 29 | } 30 | *v = int8(n) 31 | return nil 32 | case *int16: 33 | n, err := util.ParseInt(b, 10, 16) 34 | if err != nil { 35 | return err 36 | } 37 | *v = int16(n) 38 | return nil 39 | case *int32: 40 | n, err := util.ParseInt(b, 10, 32) 41 | if err != nil { 42 | return err 43 | } 44 | *v = int32(n) 45 | return nil 46 | case *int64: 47 | n, err := util.ParseInt(b, 10, 64) 48 | if err != nil { 49 | return err 50 | } 51 | *v = n 52 | return nil 53 | case *uint: 54 | n, err := util.ParseUint(b, 10, 64) 55 | if err != nil { 56 | return err 57 | } 58 | *v = uint(n) 59 | return nil 60 | case *uint8: 61 | n, err := util.ParseUint(b, 10, 8) 62 | if err != nil { 63 | return err 64 | } 65 | *v = uint8(n) 66 | return nil 67 | case *uint16: 68 | n, err := util.ParseUint(b, 10, 16) 69 | if err != nil { 70 | return err 71 | } 72 | *v = uint16(n) 73 | return nil 74 | case *uint32: 75 | n, err := util.ParseUint(b, 10, 32) 76 | if err != nil { 77 | return err 78 | } 79 | *v = uint32(n) 80 | return nil 81 | case *uint64: 82 | n, err := util.ParseUint(b, 10, 64) 83 | if err != nil { 84 | return err 85 | } 86 | *v = n 87 | return nil 88 | case *float32: 89 | n, err := util.ParseFloat(b, 32) 90 | if err != nil { 91 | return err 92 | } 93 | *v = float32(n) 94 | return err 95 | case *float64: 96 | var err error 97 | *v, err = util.ParseFloat(b, 64) 98 | return err 99 | case *bool: 100 | *v = len(b) == 1 && b[0] == '1' 101 | return nil 102 | case encoding.BinaryUnmarshaler: 103 | return v.UnmarshalBinary(b) 104 | default: 105 | return fmt.Errorf( 106 | "redis: can't unmarshal %T (consider implementing BinaryUnmarshaler)", v) 107 | } 108 | } 109 | 110 | func ScanSlice(data []string, slice interface{}) error { 111 | v := reflect.ValueOf(slice) 112 | if !v.IsValid() { 113 | return fmt.Errorf("redis: ScanSlice(nil)") 114 | } 115 | if v.Kind() != reflect.Ptr { 116 | return fmt.Errorf("redis: ScanSlice(non-pointer %T)", slice) 117 | } 118 | v = v.Elem() 119 | if v.Kind() != reflect.Slice { 120 | return fmt.Errorf("redis: ScanSlice(non-slice %T)", slice) 121 | } 122 | 123 | next := makeSliceNextElemFunc(v) 124 | for i, s := range data { 125 | elem := next() 126 | if err := Scan([]byte(s), elem.Addr().Interface()); err != nil { 127 | err = fmt.Errorf("redis: ScanSlice index=%d value=%q failed: %s", i, s, err) 128 | return err 129 | } 130 | } 131 | 132 | return nil 133 | } 134 | 135 | func makeSliceNextElemFunc(v reflect.Value) func() reflect.Value { 136 | elemType := v.Type().Elem() 137 | 138 | if elemType.Kind() == reflect.Ptr { 139 | elemType = elemType.Elem() 140 | return func() reflect.Value { 141 | if v.Len() < v.Cap() { 142 | v.Set(v.Slice(0, v.Len()+1)) 143 | elem := v.Index(v.Len() - 1) 144 | if elem.IsNil() { 145 | elem.Set(reflect.New(elemType)) 146 | } 147 | return elem.Elem() 148 | } 149 | 150 | elem := reflect.New(elemType) 151 | v.Set(reflect.Append(v, elem)) 152 | return elem.Elem() 153 | } 154 | } 155 | 156 | zero := reflect.Zero(elemType) 157 | return func() reflect.Value { 158 | if v.Len() < v.Cap() { 159 | v.Set(v.Slice(0, v.Len()+1)) 160 | return v.Index(v.Len() - 1) 161 | } 162 | 163 | v.Set(reflect.Append(v, zero)) 164 | return v.Index(v.Len() - 1) 165 | } 166 | } 167 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/proto/write_buffer.go: -------------------------------------------------------------------------------- 1 | package proto 2 | 3 | import ( 4 | "encoding" 5 | "fmt" 6 | "strconv" 7 | ) 8 | 9 | type WriteBuffer struct { 10 | b []byte 11 | } 12 | 13 | func NewWriteBuffer() *WriteBuffer { 14 | return &WriteBuffer{ 15 | b: make([]byte, 0, 4096), 16 | } 17 | } 18 | 19 | func (w *WriteBuffer) Len() int { return len(w.b) } 20 | func (w *WriteBuffer) Bytes() []byte { return w.b } 21 | func (w *WriteBuffer) Reset() { w.b = w.b[:0] } 22 | 23 | func (w *WriteBuffer) Append(args []interface{}) error { 24 | w.b = append(w.b, ArrayReply) 25 | w.b = strconv.AppendUint(w.b, uint64(len(args)), 10) 26 | w.b = append(w.b, '\r', '\n') 27 | 28 | for _, arg := range args { 29 | if err := w.append(arg); err != nil { 30 | return err 31 | } 32 | } 33 | return nil 34 | } 35 | 36 | func (w *WriteBuffer) append(val interface{}) error { 37 | switch v := val.(type) { 38 | case nil: 39 | w.AppendString("") 40 | case string: 41 | w.AppendString(v) 42 | case []byte: 43 | w.AppendBytes(v) 44 | case int: 45 | w.AppendString(formatInt(int64(v))) 46 | case int8: 47 | w.AppendString(formatInt(int64(v))) 48 | case int16: 49 | w.AppendString(formatInt(int64(v))) 50 | case int32: 51 | w.AppendString(formatInt(int64(v))) 52 | case int64: 53 | w.AppendString(formatInt(v)) 54 | case uint: 55 | w.AppendString(formatUint(uint64(v))) 56 | case uint8: 57 | w.AppendString(formatUint(uint64(v))) 58 | case uint16: 59 | w.AppendString(formatUint(uint64(v))) 60 | case uint32: 61 | w.AppendString(formatUint(uint64(v))) 62 | case uint64: 63 | w.AppendString(formatUint(v)) 64 | case float32: 65 | w.AppendString(formatFloat(float64(v))) 66 | case float64: 67 | w.AppendString(formatFloat(v)) 68 | case bool: 69 | if v { 70 | w.AppendString("1") 71 | } else { 72 | w.AppendString("0") 73 | } 74 | case encoding.BinaryMarshaler: 75 | b, err := v.MarshalBinary() 76 | if err != nil { 77 | return err 78 | } 79 | w.AppendBytes(b) 80 | default: 81 | return fmt.Errorf( 82 | "redis: can't marshal %T (consider implementing encoding.BinaryMarshaler)", val) 83 | } 84 | return nil 85 | } 86 | 87 | func (w *WriteBuffer) AppendString(s string) { 88 | w.b = append(w.b, StringReply) 89 | w.b = strconv.AppendUint(w.b, uint64(len(s)), 10) 90 | w.b = append(w.b, '\r', '\n') 91 | w.b = append(w.b, s...) 92 | w.b = append(w.b, '\r', '\n') 93 | } 94 | 95 | func (w *WriteBuffer) AppendBytes(p []byte) { 96 | w.b = append(w.b, StringReply) 97 | w.b = strconv.AppendUint(w.b, uint64(len(p)), 10) 98 | w.b = append(w.b, '\r', '\n') 99 | w.b = append(w.b, p...) 100 | w.b = append(w.b, '\r', '\n') 101 | } 102 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/singleflight/singleflight.go: -------------------------------------------------------------------------------- 1 | /* 2 | Copyright 2013 Google Inc. 3 | 4 | Licensed under the Apache License, Version 2.0 (the "License"); 5 | you may not use this file except in compliance with the License. 6 | You may obtain a copy of the License at 7 | 8 | http://www.apache.org/licenses/LICENSE-2.0 9 | 10 | Unless required by applicable law or agreed to in writing, software 11 | distributed under the License is distributed on an "AS IS" BASIS, 12 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 13 | See the License for the specific language governing permissions and 14 | limitations under the License. 15 | */ 16 | 17 | // Package singleflight provides a duplicate function call suppression 18 | // mechanism. 19 | package singleflight 20 | 21 | import "sync" 22 | 23 | // call is an in-flight or completed Do call 24 | type call struct { 25 | wg sync.WaitGroup 26 | val interface{} 27 | err error 28 | } 29 | 30 | // Group represents a class of work and forms a namespace in which 31 | // units of work can be executed with duplicate suppression. 32 | type Group struct { 33 | mu sync.Mutex // protects m 34 | m map[string]*call // lazily initialized 35 | } 36 | 37 | // Do executes and returns the results of the given function, making 38 | // sure that only one execution is in-flight for a given key at a 39 | // time. If a duplicate comes in, the duplicate caller waits for the 40 | // original to complete and receives the same results. 41 | func (g *Group) Do(key string, fn func() (interface{}, error)) (interface{}, error) { 42 | g.mu.Lock() 43 | if g.m == nil { 44 | g.m = make(map[string]*call) 45 | } 46 | if c, ok := g.m[key]; ok { 47 | g.mu.Unlock() 48 | c.wg.Wait() 49 | return c.val, c.err 50 | } 51 | c := new(call) 52 | c.wg.Add(1) 53 | g.m[key] = c 54 | g.mu.Unlock() 55 | 56 | c.val, c.err = fn() 57 | c.wg.Done() 58 | 59 | g.mu.Lock() 60 | delete(g.m, key) 61 | g.mu.Unlock() 62 | 63 | return c.val, c.err 64 | } 65 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/util.go: -------------------------------------------------------------------------------- 1 | package internal 2 | 3 | import "github.com/go-redis/redis/internal/util" 4 | 5 | func ToLower(s string) string { 6 | if isLower(s) { 7 | return s 8 | } 9 | 10 | b := make([]byte, len(s)) 11 | for i := range b { 12 | c := s[i] 13 | if c >= 'A' && c <= 'Z' { 14 | c += 'a' - 'A' 15 | } 16 | b[i] = c 17 | } 18 | return util.BytesToString(b) 19 | } 20 | 21 | func isLower(s string) bool { 22 | for i := 0; i < len(s); i++ { 23 | c := s[i] 24 | if c >= 'A' && c <= 'Z' { 25 | return false 26 | } 27 | } 28 | return true 29 | } 30 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/util/safe.go: -------------------------------------------------------------------------------- 1 | // +build appengine 2 | 3 | package util 4 | 5 | func BytesToString(b []byte) string { 6 | return string(b) 7 | } 8 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/util/strconv.go: -------------------------------------------------------------------------------- 1 | package util 2 | 3 | import "strconv" 4 | 5 | func Atoi(b []byte) (int, error) { 6 | return strconv.Atoi(BytesToString(b)) 7 | } 8 | 9 | func ParseInt(b []byte, base int, bitSize int) (int64, error) { 10 | return strconv.ParseInt(BytesToString(b), base, bitSize) 11 | } 12 | 13 | func ParseUint(b []byte, base int, bitSize int) (uint64, error) { 14 | return strconv.ParseUint(BytesToString(b), base, bitSize) 15 | } 16 | 17 | func ParseFloat(b []byte, bitSize int) (float64, error) { 18 | return strconv.ParseFloat(BytesToString(b), bitSize) 19 | } 20 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/internal/util/unsafe.go: -------------------------------------------------------------------------------- 1 | // +build !appengine 2 | 3 | package util 4 | 5 | import ( 6 | "unsafe" 7 | ) 8 | 9 | // BytesToString converts byte slice to string. 10 | func BytesToString(b []byte) string { 11 | return *(*string)(unsafe.Pointer(&b)) 12 | } 13 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/iterator.go: -------------------------------------------------------------------------------- 1 | package redis 2 | 3 | import "sync" 4 | 5 | // ScanIterator is used to incrementally iterate over a collection of elements. 6 | // It's safe for concurrent use by multiple goroutines. 7 | type ScanIterator struct { 8 | mu sync.Mutex // protects Scanner and pos 9 | cmd *ScanCmd 10 | pos int 11 | } 12 | 13 | // Err returns the last iterator error, if any. 14 | func (it *ScanIterator) Err() error { 15 | it.mu.Lock() 16 | err := it.cmd.Err() 17 | it.mu.Unlock() 18 | return err 19 | } 20 | 21 | // Next advances the cursor and returns true if more values can be read. 22 | func (it *ScanIterator) Next() bool { 23 | it.mu.Lock() 24 | defer it.mu.Unlock() 25 | 26 | // Instantly return on errors. 27 | if it.cmd.Err() != nil { 28 | return false 29 | } 30 | 31 | // Advance cursor, check if we are still within range. 32 | if it.pos < len(it.cmd.page) { 33 | it.pos++ 34 | return true 35 | } 36 | 37 | for { 38 | // Return if there is no more data to fetch. 39 | if it.cmd.cursor == 0 { 40 | return false 41 | } 42 | 43 | // Fetch next page. 44 | if it.cmd._args[0] == "scan" { 45 | it.cmd._args[1] = it.cmd.cursor 46 | } else { 47 | it.cmd._args[2] = it.cmd.cursor 48 | } 49 | 50 | err := it.cmd.process(it.cmd) 51 | if err != nil { 52 | return false 53 | } 54 | 55 | it.pos = 1 56 | 57 | // Redis can occasionally return empty page. 58 | if len(it.cmd.page) > 0 { 59 | return true 60 | } 61 | } 62 | } 63 | 64 | // Val returns the key/field at the current cursor position. 65 | func (it *ScanIterator) Val() string { 66 | var v string 67 | it.mu.Lock() 68 | if it.cmd.Err() == nil && it.pos > 0 && it.pos <= len(it.cmd.page) { 69 | v = it.cmd.page[it.pos-1] 70 | } 71 | it.mu.Unlock() 72 | return v 73 | } 74 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/options.go: -------------------------------------------------------------------------------- 1 | package redis 2 | 3 | import ( 4 | "crypto/tls" 5 | "errors" 6 | "fmt" 7 | "net" 8 | "net/url" 9 | "runtime" 10 | "strconv" 11 | "strings" 12 | "time" 13 | 14 | "github.com/go-redis/redis/internal/pool" 15 | ) 16 | 17 | type Options struct { 18 | // The network type, either tcp or unix. 19 | // Default is tcp. 20 | Network string 21 | // host:port address. 22 | Addr string 23 | 24 | // Dialer creates new network connection and has priority over 25 | // Network and Addr options. 26 | Dialer func() (net.Conn, error) 27 | 28 | // Hook that is called when new connection is established. 29 | OnConnect func(*Conn) error 30 | 31 | // Optional password. Must match the password specified in the 32 | // requirepass server configuration option. 33 | Password string 34 | // Database to be selected after connecting to the server. 35 | DB int 36 | 37 | // Maximum number of retries before giving up. 38 | // Default is to not retry failed commands. 39 | MaxRetries int 40 | // Minimum backoff between each retry. 41 | // Default is 8 milliseconds; -1 disables backoff. 42 | MinRetryBackoff time.Duration 43 | // Maximum backoff between each retry. 44 | // Default is 512 milliseconds; -1 disables backoff. 45 | MaxRetryBackoff time.Duration 46 | 47 | // Dial timeout for establishing new connections. 48 | // Default is 5 seconds. 49 | DialTimeout time.Duration 50 | // Timeout for socket reads. If reached, commands will fail 51 | // with a timeout instead of blocking. 52 | // Default is 3 seconds. 53 | ReadTimeout time.Duration 54 | // Timeout for socket writes. If reached, commands will fail 55 | // with a timeout instead of blocking. 56 | // Default is ReadTimeout. 57 | WriteTimeout time.Duration 58 | 59 | // Maximum number of socket connections. 60 | // Default is 10 connections per every CPU as reported by runtime.NumCPU. 61 | PoolSize int 62 | // Amount of time client waits for connection if all connections 63 | // are busy before returning an error. 64 | // Default is ReadTimeout + 1 second. 65 | PoolTimeout time.Duration 66 | // Amount of time after which client closes idle connections. 67 | // Should be less than server's timeout. 68 | // Default is 5 minutes. 69 | IdleTimeout time.Duration 70 | // Frequency of idle checks. 71 | // Default is 1 minute. 72 | // When minus value is set, then idle check is disabled. 73 | IdleCheckFrequency time.Duration 74 | 75 | // Enables read only queries on slave nodes. 76 | readOnly bool 77 | 78 | // TLS Config to use. When set TLS will be negotiated. 79 | TLSConfig *tls.Config 80 | } 81 | 82 | func (opt *Options) init() { 83 | if opt.Network == "" { 84 | opt.Network = "tcp" 85 | } 86 | if opt.Dialer == nil { 87 | opt.Dialer = func() (net.Conn, error) { 88 | conn, err := net.DialTimeout(opt.Network, opt.Addr, opt.DialTimeout) 89 | if opt.TLSConfig == nil || err != nil { 90 | return conn, err 91 | } 92 | t := tls.Client(conn, opt.TLSConfig) 93 | return t, t.Handshake() 94 | } 95 | } 96 | if opt.PoolSize == 0 { 97 | opt.PoolSize = 10 * runtime.NumCPU() 98 | } 99 | if opt.DialTimeout == 0 { 100 | opt.DialTimeout = 5 * time.Second 101 | } 102 | switch opt.ReadTimeout { 103 | case -1: 104 | opt.ReadTimeout = 0 105 | case 0: 106 | opt.ReadTimeout = 3 * time.Second 107 | } 108 | switch opt.WriteTimeout { 109 | case -1: 110 | opt.WriteTimeout = 0 111 | case 0: 112 | opt.WriteTimeout = opt.ReadTimeout 113 | } 114 | if opt.PoolTimeout == 0 { 115 | opt.PoolTimeout = opt.ReadTimeout + time.Second 116 | } 117 | if opt.IdleTimeout == 0 { 118 | opt.IdleTimeout = 5 * time.Minute 119 | } 120 | if opt.IdleCheckFrequency == 0 { 121 | opt.IdleCheckFrequency = time.Minute 122 | } 123 | 124 | switch opt.MinRetryBackoff { 125 | case -1: 126 | opt.MinRetryBackoff = 0 127 | case 0: 128 | opt.MinRetryBackoff = 8 * time.Millisecond 129 | } 130 | switch opt.MaxRetryBackoff { 131 | case -1: 132 | opt.MaxRetryBackoff = 0 133 | case 0: 134 | opt.MaxRetryBackoff = 512 * time.Millisecond 135 | } 136 | } 137 | 138 | // ParseURL parses an URL into Options that can be used to connect to Redis. 139 | func ParseURL(redisURL string) (*Options, error) { 140 | o := &Options{Network: "tcp"} 141 | u, err := url.Parse(redisURL) 142 | if err != nil { 143 | return nil, err 144 | } 145 | 146 | if u.Scheme != "redis" && u.Scheme != "rediss" { 147 | return nil, errors.New("invalid redis URL scheme: " + u.Scheme) 148 | } 149 | 150 | if u.User != nil { 151 | if p, ok := u.User.Password(); ok { 152 | o.Password = p 153 | } 154 | } 155 | 156 | if len(u.Query()) > 0 { 157 | return nil, errors.New("no options supported") 158 | } 159 | 160 | h, p, err := net.SplitHostPort(u.Host) 161 | if err != nil { 162 | h = u.Host 163 | } 164 | if h == "" { 165 | h = "localhost" 166 | } 167 | if p == "" { 168 | p = "6379" 169 | } 170 | o.Addr = net.JoinHostPort(h, p) 171 | 172 | f := strings.FieldsFunc(u.Path, func(r rune) bool { 173 | return r == '/' 174 | }) 175 | switch len(f) { 176 | case 0: 177 | o.DB = 0 178 | case 1: 179 | if o.DB, err = strconv.Atoi(f[0]); err != nil { 180 | return nil, fmt.Errorf("invalid redis database number: %q", f[0]) 181 | } 182 | default: 183 | return nil, errors.New("invalid redis URL path: " + u.Path) 184 | } 185 | 186 | if u.Scheme == "rediss" { 187 | o.TLSConfig = &tls.Config{ServerName: h} 188 | } 189 | return o, nil 190 | } 191 | 192 | func newConnPool(opt *Options) *pool.ConnPool { 193 | return pool.NewConnPool(&pool.Options{ 194 | Dialer: opt.Dialer, 195 | PoolSize: opt.PoolSize, 196 | PoolTimeout: opt.PoolTimeout, 197 | IdleTimeout: opt.IdleTimeout, 198 | IdleCheckFrequency: opt.IdleCheckFrequency, 199 | }) 200 | } 201 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/parser.go: -------------------------------------------------------------------------------- 1 | package redis 2 | 3 | import ( 4 | "fmt" 5 | "net" 6 | "strconv" 7 | "time" 8 | 9 | "github.com/go-redis/redis/internal/proto" 10 | ) 11 | 12 | // Implements proto.MultiBulkParse 13 | func sliceParser(rd *proto.Reader, n int64) (interface{}, error) { 14 | vals := make([]interface{}, 0, n) 15 | for i := int64(0); i < n; i++ { 16 | v, err := rd.ReadReply(sliceParser) 17 | if err != nil { 18 | if err == Nil { 19 | vals = append(vals, nil) 20 | continue 21 | } 22 | if err, ok := err.(proto.RedisError); ok { 23 | vals = append(vals, err) 24 | continue 25 | } 26 | return nil, err 27 | } 28 | 29 | switch v := v.(type) { 30 | case []byte: 31 | vals = append(vals, string(v)) 32 | default: 33 | vals = append(vals, v) 34 | } 35 | } 36 | return vals, nil 37 | } 38 | 39 | // Implements proto.MultiBulkParse 40 | func boolSliceParser(rd *proto.Reader, n int64) (interface{}, error) { 41 | bools := make([]bool, 0, n) 42 | for i := int64(0); i < n; i++ { 43 | n, err := rd.ReadIntReply() 44 | if err != nil { 45 | return nil, err 46 | } 47 | bools = append(bools, n == 1) 48 | } 49 | return bools, nil 50 | } 51 | 52 | // Implements proto.MultiBulkParse 53 | func stringSliceParser(rd *proto.Reader, n int64) (interface{}, error) { 54 | ss := make([]string, 0, n) 55 | for i := int64(0); i < n; i++ { 56 | s, err := rd.ReadStringReply() 57 | if err == Nil { 58 | ss = append(ss, "") 59 | } else if err != nil { 60 | return nil, err 61 | } else { 62 | ss = append(ss, s) 63 | } 64 | } 65 | return ss, nil 66 | } 67 | 68 | // Implements proto.MultiBulkParse 69 | func stringStringMapParser(rd *proto.Reader, n int64) (interface{}, error) { 70 | m := make(map[string]string, n/2) 71 | for i := int64(0); i < n; i += 2 { 72 | key, err := rd.ReadStringReply() 73 | if err != nil { 74 | return nil, err 75 | } 76 | 77 | value, err := rd.ReadStringReply() 78 | if err != nil { 79 | return nil, err 80 | } 81 | 82 | m[key] = value 83 | } 84 | return m, nil 85 | } 86 | 87 | // Implements proto.MultiBulkParse 88 | func stringIntMapParser(rd *proto.Reader, n int64) (interface{}, error) { 89 | m := make(map[string]int64, n/2) 90 | for i := int64(0); i < n; i += 2 { 91 | key, err := rd.ReadStringReply() 92 | if err != nil { 93 | return nil, err 94 | } 95 | 96 | n, err := rd.ReadIntReply() 97 | if err != nil { 98 | return nil, err 99 | } 100 | 101 | m[key] = n 102 | } 103 | return m, nil 104 | } 105 | 106 | // Implements proto.MultiBulkParse 107 | func stringStructMapParser(rd *proto.Reader, n int64) (interface{}, error) { 108 | m := make(map[string]struct{}, n) 109 | for i := int64(0); i < n; i++ { 110 | key, err := rd.ReadStringReply() 111 | if err != nil { 112 | return nil, err 113 | } 114 | 115 | m[key] = struct{}{} 116 | } 117 | return m, nil 118 | } 119 | 120 | // Implements proto.MultiBulkParse 121 | func zSliceParser(rd *proto.Reader, n int64) (interface{}, error) { 122 | zz := make([]Z, n/2) 123 | for i := int64(0); i < n; i += 2 { 124 | var err error 125 | 126 | z := &zz[i/2] 127 | 128 | z.Member, err = rd.ReadStringReply() 129 | if err != nil { 130 | return nil, err 131 | } 132 | 133 | z.Score, err = rd.ReadFloatReply() 134 | if err != nil { 135 | return nil, err 136 | } 137 | } 138 | return zz, nil 139 | } 140 | 141 | // Implements proto.MultiBulkParse 142 | func clusterSlotsParser(rd *proto.Reader, n int64) (interface{}, error) { 143 | slots := make([]ClusterSlot, n) 144 | for i := 0; i < len(slots); i++ { 145 | n, err := rd.ReadArrayLen() 146 | if err != nil { 147 | return nil, err 148 | } 149 | if n < 2 { 150 | err := fmt.Errorf("redis: got %d elements in cluster info, expected at least 2", n) 151 | return nil, err 152 | } 153 | 154 | start, err := rd.ReadIntReply() 155 | if err != nil { 156 | return nil, err 157 | } 158 | 159 | end, err := rd.ReadIntReply() 160 | if err != nil { 161 | return nil, err 162 | } 163 | 164 | nodes := make([]ClusterNode, n-2) 165 | for j := 0; j < len(nodes); j++ { 166 | n, err := rd.ReadArrayLen() 167 | if err != nil { 168 | return nil, err 169 | } 170 | if n != 2 && n != 3 { 171 | err := fmt.Errorf("got %d elements in cluster info address, expected 2 or 3", n) 172 | return nil, err 173 | } 174 | 175 | ip, err := rd.ReadStringReply() 176 | if err != nil { 177 | return nil, err 178 | } 179 | 180 | port, err := rd.ReadIntReply() 181 | if err != nil { 182 | return nil, err 183 | } 184 | nodes[j].Addr = net.JoinHostPort(ip, strconv.FormatInt(port, 10)) 185 | 186 | if n == 3 { 187 | id, err := rd.ReadStringReply() 188 | if err != nil { 189 | return nil, err 190 | } 191 | nodes[j].Id = id 192 | } 193 | } 194 | 195 | slots[i] = ClusterSlot{ 196 | Start: int(start), 197 | End: int(end), 198 | Nodes: nodes, 199 | } 200 | } 201 | return slots, nil 202 | } 203 | 204 | func newGeoLocationParser(q *GeoRadiusQuery) proto.MultiBulkParse { 205 | return func(rd *proto.Reader, n int64) (interface{}, error) { 206 | var loc GeoLocation 207 | var err error 208 | 209 | loc.Name, err = rd.ReadStringReply() 210 | if err != nil { 211 | return nil, err 212 | } 213 | if q.WithDist { 214 | loc.Dist, err = rd.ReadFloatReply() 215 | if err != nil { 216 | return nil, err 217 | } 218 | } 219 | if q.WithGeoHash { 220 | loc.GeoHash, err = rd.ReadIntReply() 221 | if err != nil { 222 | return nil, err 223 | } 224 | } 225 | if q.WithCoord { 226 | n, err := rd.ReadArrayLen() 227 | if err != nil { 228 | return nil, err 229 | } 230 | if n != 2 { 231 | return nil, fmt.Errorf("got %d coordinates, expected 2", n) 232 | } 233 | 234 | loc.Longitude, err = rd.ReadFloatReply() 235 | if err != nil { 236 | return nil, err 237 | } 238 | loc.Latitude, err = rd.ReadFloatReply() 239 | if err != nil { 240 | return nil, err 241 | } 242 | } 243 | 244 | return &loc, nil 245 | } 246 | } 247 | 248 | func newGeoLocationSliceParser(q *GeoRadiusQuery) proto.MultiBulkParse { 249 | return func(rd *proto.Reader, n int64) (interface{}, error) { 250 | locs := make([]GeoLocation, 0, n) 251 | for i := int64(0); i < n; i++ { 252 | v, err := rd.ReadReply(newGeoLocationParser(q)) 253 | if err != nil { 254 | return nil, err 255 | } 256 | switch vv := v.(type) { 257 | case []byte: 258 | locs = append(locs, GeoLocation{ 259 | Name: string(vv), 260 | }) 261 | case *GeoLocation: 262 | locs = append(locs, *vv) 263 | default: 264 | return nil, fmt.Errorf("got %T, expected string or *GeoLocation", v) 265 | } 266 | } 267 | return locs, nil 268 | } 269 | } 270 | 271 | func geoPosParser(rd *proto.Reader, n int64) (interface{}, error) { 272 | var pos GeoPos 273 | var err error 274 | 275 | pos.Longitude, err = rd.ReadFloatReply() 276 | if err != nil { 277 | return nil, err 278 | } 279 | 280 | pos.Latitude, err = rd.ReadFloatReply() 281 | if err != nil { 282 | return nil, err 283 | } 284 | 285 | return &pos, nil 286 | } 287 | 288 | func geoPosSliceParser(rd *proto.Reader, n int64) (interface{}, error) { 289 | positions := make([]*GeoPos, 0, n) 290 | for i := int64(0); i < n; i++ { 291 | v, err := rd.ReadReply(geoPosParser) 292 | if err != nil { 293 | if err == Nil { 294 | positions = append(positions, nil) 295 | continue 296 | } 297 | return nil, err 298 | } 299 | switch v := v.(type) { 300 | case *GeoPos: 301 | positions = append(positions, v) 302 | default: 303 | return nil, fmt.Errorf("got %T, expected *GeoPos", v) 304 | } 305 | } 306 | return positions, nil 307 | } 308 | 309 | func commandInfoParser(rd *proto.Reader, n int64) (interface{}, error) { 310 | var cmd CommandInfo 311 | var err error 312 | 313 | if n != 6 { 314 | return nil, fmt.Errorf("redis: got %d elements in COMMAND reply, wanted 6", n) 315 | } 316 | 317 | cmd.Name, err = rd.ReadStringReply() 318 | if err != nil { 319 | return nil, err 320 | } 321 | 322 | arity, err := rd.ReadIntReply() 323 | if err != nil { 324 | return nil, err 325 | } 326 | cmd.Arity = int8(arity) 327 | 328 | flags, err := rd.ReadReply(stringSliceParser) 329 | if err != nil { 330 | return nil, err 331 | } 332 | cmd.Flags = flags.([]string) 333 | 334 | firstKeyPos, err := rd.ReadIntReply() 335 | if err != nil { 336 | return nil, err 337 | } 338 | cmd.FirstKeyPos = int8(firstKeyPos) 339 | 340 | lastKeyPos, err := rd.ReadIntReply() 341 | if err != nil { 342 | return nil, err 343 | } 344 | cmd.LastKeyPos = int8(lastKeyPos) 345 | 346 | stepCount, err := rd.ReadIntReply() 347 | if err != nil { 348 | return nil, err 349 | } 350 | cmd.StepCount = int8(stepCount) 351 | 352 | for _, flag := range cmd.Flags { 353 | if flag == "readonly" { 354 | cmd.ReadOnly = true 355 | break 356 | } 357 | } 358 | 359 | return &cmd, nil 360 | } 361 | 362 | // Implements proto.MultiBulkParse 363 | func commandInfoSliceParser(rd *proto.Reader, n int64) (interface{}, error) { 364 | m := make(map[string]*CommandInfo, n) 365 | for i := int64(0); i < n; i++ { 366 | v, err := rd.ReadReply(commandInfoParser) 367 | if err != nil { 368 | return nil, err 369 | } 370 | vv := v.(*CommandInfo) 371 | m[vv.Name] = vv 372 | 373 | } 374 | return m, nil 375 | } 376 | 377 | // Implements proto.MultiBulkParse 378 | func timeParser(rd *proto.Reader, n int64) (interface{}, error) { 379 | if n != 2 { 380 | return nil, fmt.Errorf("got %d elements, expected 2", n) 381 | } 382 | 383 | sec, err := rd.ReadInt() 384 | if err != nil { 385 | return nil, err 386 | } 387 | 388 | microsec, err := rd.ReadInt() 389 | if err != nil { 390 | return nil, err 391 | } 392 | 393 | return time.Unix(sec, microsec*1000), nil 394 | } 395 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/pipeline.go: -------------------------------------------------------------------------------- 1 | package redis 2 | 3 | import ( 4 | "sync" 5 | 6 | "github.com/go-redis/redis/internal/pool" 7 | ) 8 | 9 | type pipelineExecer func([]Cmder) error 10 | 11 | type Pipeliner interface { 12 | StatefulCmdable 13 | Process(cmd Cmder) error 14 | Close() error 15 | Discard() error 16 | Exec() ([]Cmder, error) 17 | } 18 | 19 | var _ Pipeliner = (*Pipeline)(nil) 20 | 21 | // Pipeline implements pipelining as described in 22 | // http://redis.io/topics/pipelining. It's safe for concurrent use 23 | // by multiple goroutines. 24 | type Pipeline struct { 25 | statefulCmdable 26 | 27 | exec pipelineExecer 28 | 29 | mu sync.Mutex 30 | cmds []Cmder 31 | closed bool 32 | } 33 | 34 | func (c *Pipeline) Process(cmd Cmder) error { 35 | c.mu.Lock() 36 | c.cmds = append(c.cmds, cmd) 37 | c.mu.Unlock() 38 | return nil 39 | } 40 | 41 | // Close closes the pipeline, releasing any open resources. 42 | func (c *Pipeline) Close() error { 43 | c.mu.Lock() 44 | c.discard() 45 | c.closed = true 46 | c.mu.Unlock() 47 | return nil 48 | } 49 | 50 | // Discard resets the pipeline and discards queued commands. 51 | func (c *Pipeline) Discard() error { 52 | c.mu.Lock() 53 | err := c.discard() 54 | c.mu.Unlock() 55 | return err 56 | } 57 | 58 | func (c *Pipeline) discard() error { 59 | if c.closed { 60 | return pool.ErrClosed 61 | } 62 | c.cmds = c.cmds[:0] 63 | return nil 64 | } 65 | 66 | // Exec executes all previously queued commands using one 67 | // client-server roundtrip. 68 | // 69 | // Exec always returns list of commands and error of the first failed 70 | // command if any. 71 | func (c *Pipeline) Exec() ([]Cmder, error) { 72 | c.mu.Lock() 73 | defer c.mu.Unlock() 74 | 75 | if c.closed { 76 | return nil, pool.ErrClosed 77 | } 78 | 79 | if len(c.cmds) == 0 { 80 | return nil, nil 81 | } 82 | 83 | cmds := c.cmds 84 | c.cmds = nil 85 | 86 | return cmds, c.exec(cmds) 87 | } 88 | 89 | func (c *Pipeline) pipelined(fn func(Pipeliner) error) ([]Cmder, error) { 90 | if err := fn(c); err != nil { 91 | return nil, err 92 | } 93 | cmds, err := c.Exec() 94 | _ = c.Close() 95 | return cmds, err 96 | } 97 | 98 | func (c *Pipeline) Pipelined(fn func(Pipeliner) error) ([]Cmder, error) { 99 | return c.pipelined(fn) 100 | } 101 | 102 | func (c *Pipeline) Pipeline() Pipeliner { 103 | return c 104 | } 105 | 106 | func (c *Pipeline) TxPipelined(fn func(Pipeliner) error) ([]Cmder, error) { 107 | return c.pipelined(fn) 108 | } 109 | 110 | func (c *Pipeline) TxPipeline() Pipeliner { 111 | return c 112 | } 113 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/result.go: -------------------------------------------------------------------------------- 1 | package redis 2 | 3 | import "time" 4 | 5 | // NewCmdResult returns a Cmd initalised with val and err for testing 6 | func NewCmdResult(val interface{}, err error) *Cmd { 7 | var cmd Cmd 8 | cmd.val = val 9 | cmd.setErr(err) 10 | return &cmd 11 | } 12 | 13 | // NewSliceResult returns a SliceCmd initalised with val and err for testing 14 | func NewSliceResult(val []interface{}, err error) *SliceCmd { 15 | var cmd SliceCmd 16 | cmd.val = val 17 | cmd.setErr(err) 18 | return &cmd 19 | } 20 | 21 | // NewStatusResult returns a StatusCmd initalised with val and err for testing 22 | func NewStatusResult(val string, err error) *StatusCmd { 23 | var cmd StatusCmd 24 | cmd.val = val 25 | cmd.setErr(err) 26 | return &cmd 27 | } 28 | 29 | // NewIntResult returns an IntCmd initalised with val and err for testing 30 | func NewIntResult(val int64, err error) *IntCmd { 31 | var cmd IntCmd 32 | cmd.val = val 33 | cmd.setErr(err) 34 | return &cmd 35 | } 36 | 37 | // NewDurationResult returns a DurationCmd initalised with val and err for testing 38 | func NewDurationResult(val time.Duration, err error) *DurationCmd { 39 | var cmd DurationCmd 40 | cmd.val = val 41 | cmd.setErr(err) 42 | return &cmd 43 | } 44 | 45 | // NewBoolResult returns a BoolCmd initalised with val and err for testing 46 | func NewBoolResult(val bool, err error) *BoolCmd { 47 | var cmd BoolCmd 48 | cmd.val = val 49 | cmd.setErr(err) 50 | return &cmd 51 | } 52 | 53 | // NewStringResult returns a StringCmd initalised with val and err for testing 54 | func NewStringResult(val string, err error) *StringCmd { 55 | var cmd StringCmd 56 | cmd.val = []byte(val) 57 | cmd.setErr(err) 58 | return &cmd 59 | } 60 | 61 | // NewFloatResult returns a FloatCmd initalised with val and err for testing 62 | func NewFloatResult(val float64, err error) *FloatCmd { 63 | var cmd FloatCmd 64 | cmd.val = val 65 | cmd.setErr(err) 66 | return &cmd 67 | } 68 | 69 | // NewStringSliceResult returns a StringSliceCmd initalised with val and err for testing 70 | func NewStringSliceResult(val []string, err error) *StringSliceCmd { 71 | var cmd StringSliceCmd 72 | cmd.val = val 73 | cmd.setErr(err) 74 | return &cmd 75 | } 76 | 77 | // NewBoolSliceResult returns a BoolSliceCmd initalised with val and err for testing 78 | func NewBoolSliceResult(val []bool, err error) *BoolSliceCmd { 79 | var cmd BoolSliceCmd 80 | cmd.val = val 81 | cmd.setErr(err) 82 | return &cmd 83 | } 84 | 85 | // NewStringStringMapResult returns a StringStringMapCmd initalised with val and err for testing 86 | func NewStringStringMapResult(val map[string]string, err error) *StringStringMapCmd { 87 | var cmd StringStringMapCmd 88 | cmd.val = val 89 | cmd.setErr(err) 90 | return &cmd 91 | } 92 | 93 | // NewStringIntMapCmdResult returns a StringIntMapCmd initalised with val and err for testing 94 | func NewStringIntMapCmdResult(val map[string]int64, err error) *StringIntMapCmd { 95 | var cmd StringIntMapCmd 96 | cmd.val = val 97 | cmd.setErr(err) 98 | return &cmd 99 | } 100 | 101 | // NewZSliceCmdResult returns a ZSliceCmd initalised with val and err for testing 102 | func NewZSliceCmdResult(val []Z, err error) *ZSliceCmd { 103 | var cmd ZSliceCmd 104 | cmd.val = val 105 | cmd.setErr(err) 106 | return &cmd 107 | } 108 | 109 | // NewScanCmdResult returns a ScanCmd initalised with val and err for testing 110 | func NewScanCmdResult(keys []string, cursor uint64, err error) *ScanCmd { 111 | var cmd ScanCmd 112 | cmd.page = keys 113 | cmd.cursor = cursor 114 | cmd.setErr(err) 115 | return &cmd 116 | } 117 | 118 | // NewClusterSlotsCmdResult returns a ClusterSlotsCmd initalised with val and err for testing 119 | func NewClusterSlotsCmdResult(val []ClusterSlot, err error) *ClusterSlotsCmd { 120 | var cmd ClusterSlotsCmd 121 | cmd.val = val 122 | cmd.setErr(err) 123 | return &cmd 124 | } 125 | 126 | // NewGeoLocationCmdResult returns a GeoLocationCmd initalised with val and err for testing 127 | func NewGeoLocationCmdResult(val []GeoLocation, err error) *GeoLocationCmd { 128 | var cmd GeoLocationCmd 129 | cmd.locations = val 130 | cmd.setErr(err) 131 | return &cmd 132 | } 133 | 134 | // NewCommandsInfoCmdResult returns a CommandsInfoCmd initalised with val and err for testing 135 | func NewCommandsInfoCmdResult(val map[string]*CommandInfo, err error) *CommandsInfoCmd { 136 | var cmd CommandsInfoCmd 137 | cmd.val = val 138 | cmd.setErr(err) 139 | return &cmd 140 | } 141 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/script.go: -------------------------------------------------------------------------------- 1 | package redis 2 | 3 | import ( 4 | "crypto/sha1" 5 | "encoding/hex" 6 | "io" 7 | "strings" 8 | ) 9 | 10 | type scripter interface { 11 | Eval(script string, keys []string, args ...interface{}) *Cmd 12 | EvalSha(sha1 string, keys []string, args ...interface{}) *Cmd 13 | ScriptExists(hashes ...string) *BoolSliceCmd 14 | ScriptLoad(script string) *StringCmd 15 | } 16 | 17 | var _ scripter = (*Client)(nil) 18 | var _ scripter = (*Ring)(nil) 19 | var _ scripter = (*ClusterClient)(nil) 20 | 21 | type Script struct { 22 | src, hash string 23 | } 24 | 25 | func NewScript(src string) *Script { 26 | h := sha1.New() 27 | io.WriteString(h, src) 28 | return &Script{ 29 | src: src, 30 | hash: hex.EncodeToString(h.Sum(nil)), 31 | } 32 | } 33 | 34 | func (s *Script) Hash() string { 35 | return s.hash 36 | } 37 | 38 | func (s *Script) Load(c scripter) *StringCmd { 39 | return c.ScriptLoad(s.src) 40 | } 41 | 42 | func (s *Script) Exists(c scripter) *BoolSliceCmd { 43 | return c.ScriptExists(s.hash) 44 | } 45 | 46 | func (s *Script) Eval(c scripter, keys []string, args ...interface{}) *Cmd { 47 | return c.Eval(s.src, keys, args...) 48 | } 49 | 50 | func (s *Script) EvalSha(c scripter, keys []string, args ...interface{}) *Cmd { 51 | return c.EvalSha(s.hash, keys, args...) 52 | } 53 | 54 | // Run optimistically uses EVALSHA to run the script. If script does not exist 55 | // it is retried using EVAL. 56 | func (s *Script) Run(c scripter, keys []string, args ...interface{}) *Cmd { 57 | r := s.EvalSha(c, keys, args...) 58 | if err := r.Err(); err != nil && strings.HasPrefix(err.Error(), "NOSCRIPT ") { 59 | return s.Eval(c, keys, args...) 60 | } 61 | return r 62 | } 63 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/sentinel.go: -------------------------------------------------------------------------------- 1 | package redis 2 | 3 | import ( 4 | "errors" 5 | "net" 6 | "strings" 7 | "sync" 8 | "time" 9 | 10 | "github.com/go-redis/redis/internal" 11 | "github.com/go-redis/redis/internal/pool" 12 | ) 13 | 14 | //------------------------------------------------------------------------------ 15 | 16 | // FailoverOptions are used to configure a failover client and should 17 | // be passed to NewFailoverClient. 18 | type FailoverOptions struct { 19 | // The master name. 20 | MasterName string 21 | // A seed list of host:port addresses of sentinel nodes. 22 | SentinelAddrs []string 23 | 24 | // Following options are copied from Options struct. 25 | 26 | OnConnect func(*Conn) error 27 | 28 | Password string 29 | DB int 30 | 31 | MaxRetries int 32 | 33 | DialTimeout time.Duration 34 | ReadTimeout time.Duration 35 | WriteTimeout time.Duration 36 | 37 | PoolSize int 38 | PoolTimeout time.Duration 39 | IdleTimeout time.Duration 40 | IdleCheckFrequency time.Duration 41 | } 42 | 43 | func (opt *FailoverOptions) options() *Options { 44 | return &Options{ 45 | Addr: "FailoverClient", 46 | 47 | OnConnect: opt.OnConnect, 48 | 49 | DB: opt.DB, 50 | Password: opt.Password, 51 | 52 | MaxRetries: opt.MaxRetries, 53 | 54 | DialTimeout: opt.DialTimeout, 55 | ReadTimeout: opt.ReadTimeout, 56 | WriteTimeout: opt.WriteTimeout, 57 | 58 | PoolSize: opt.PoolSize, 59 | PoolTimeout: opt.PoolTimeout, 60 | IdleTimeout: opt.IdleTimeout, 61 | IdleCheckFrequency: opt.IdleCheckFrequency, 62 | } 63 | } 64 | 65 | // NewFailoverClient returns a Redis client that uses Redis Sentinel 66 | // for automatic failover. It's safe for concurrent use by multiple 67 | // goroutines. 68 | func NewFailoverClient(failoverOpt *FailoverOptions) *Client { 69 | opt := failoverOpt.options() 70 | opt.init() 71 | 72 | failover := &sentinelFailover{ 73 | masterName: failoverOpt.MasterName, 74 | sentinelAddrs: failoverOpt.SentinelAddrs, 75 | 76 | opt: opt, 77 | } 78 | 79 | c := Client{ 80 | baseClient: baseClient{ 81 | opt: opt, 82 | connPool: failover.Pool(), 83 | 84 | onClose: func() error { 85 | return failover.Close() 86 | }, 87 | }, 88 | } 89 | c.baseClient.init() 90 | c.setProcessor(c.Process) 91 | 92 | return &c 93 | } 94 | 95 | //------------------------------------------------------------------------------ 96 | 97 | type sentinelClient struct { 98 | cmdable 99 | baseClient 100 | } 101 | 102 | func newSentinel(opt *Options) *sentinelClient { 103 | opt.init() 104 | c := sentinelClient{ 105 | baseClient: baseClient{ 106 | opt: opt, 107 | connPool: newConnPool(opt), 108 | }, 109 | } 110 | c.baseClient.init() 111 | c.cmdable.setProcessor(c.Process) 112 | return &c 113 | } 114 | 115 | func (c *sentinelClient) PubSub() *PubSub { 116 | return &PubSub{ 117 | opt: c.opt, 118 | 119 | newConn: func(channels []string) (*pool.Conn, error) { 120 | return c.newConn() 121 | }, 122 | closeConn: c.connPool.CloseConn, 123 | } 124 | } 125 | 126 | func (c *sentinelClient) GetMasterAddrByName(name string) *StringSliceCmd { 127 | cmd := NewStringSliceCmd("SENTINEL", "get-master-addr-by-name", name) 128 | c.Process(cmd) 129 | return cmd 130 | } 131 | 132 | func (c *sentinelClient) Sentinels(name string) *SliceCmd { 133 | cmd := NewSliceCmd("SENTINEL", "sentinels", name) 134 | c.Process(cmd) 135 | return cmd 136 | } 137 | 138 | type sentinelFailover struct { 139 | sentinelAddrs []string 140 | 141 | opt *Options 142 | 143 | pool *pool.ConnPool 144 | poolOnce sync.Once 145 | 146 | mu sync.RWMutex 147 | masterName string 148 | _masterAddr string 149 | sentinel *sentinelClient 150 | } 151 | 152 | func (d *sentinelFailover) Close() error { 153 | return d.resetSentinel() 154 | } 155 | 156 | func (d *sentinelFailover) Pool() *pool.ConnPool { 157 | d.poolOnce.Do(func() { 158 | d.opt.Dialer = d.dial 159 | d.pool = newConnPool(d.opt) 160 | }) 161 | return d.pool 162 | } 163 | 164 | func (d *sentinelFailover) dial() (net.Conn, error) { 165 | addr, err := d.MasterAddr() 166 | if err != nil { 167 | return nil, err 168 | } 169 | return net.DialTimeout("tcp", addr, d.opt.DialTimeout) 170 | } 171 | 172 | func (d *sentinelFailover) MasterAddr() (string, error) { 173 | d.mu.Lock() 174 | defer d.mu.Unlock() 175 | 176 | addr, err := d.masterAddr() 177 | if err != nil { 178 | return "", err 179 | } 180 | 181 | if d._masterAddr != addr { 182 | d.switchMaster(addr) 183 | } 184 | 185 | return addr, nil 186 | } 187 | 188 | func (d *sentinelFailover) masterAddr() (string, error) { 189 | // Try last working sentinel. 190 | if d.sentinel != nil { 191 | addr, err := d.sentinel.GetMasterAddrByName(d.masterName).Result() 192 | if err == nil { 193 | addr := net.JoinHostPort(addr[0], addr[1]) 194 | internal.Logf("sentinel: master=%q addr=%q", d.masterName, addr) 195 | return addr, nil 196 | } 197 | 198 | internal.Logf("sentinel: GetMasterAddrByName name=%q failed: %s", d.masterName, err) 199 | d._resetSentinel() 200 | } 201 | 202 | for i, sentinelAddr := range d.sentinelAddrs { 203 | sentinel := newSentinel(&Options{ 204 | Addr: sentinelAddr, 205 | 206 | DialTimeout: d.opt.DialTimeout, 207 | ReadTimeout: d.opt.ReadTimeout, 208 | WriteTimeout: d.opt.WriteTimeout, 209 | 210 | PoolSize: d.opt.PoolSize, 211 | PoolTimeout: d.opt.PoolTimeout, 212 | IdleTimeout: d.opt.IdleTimeout, 213 | }) 214 | 215 | masterAddr, err := sentinel.GetMasterAddrByName(d.masterName).Result() 216 | if err != nil { 217 | internal.Logf("sentinel: GetMasterAddrByName master=%q failed: %s", d.masterName, err) 218 | sentinel.Close() 219 | continue 220 | } 221 | 222 | // Push working sentinel to the top. 223 | d.sentinelAddrs[0], d.sentinelAddrs[i] = d.sentinelAddrs[i], d.sentinelAddrs[0] 224 | d.setSentinel(sentinel) 225 | 226 | addr := net.JoinHostPort(masterAddr[0], masterAddr[1]) 227 | return addr, nil 228 | } 229 | 230 | return "", errors.New("redis: all sentinels are unreachable") 231 | } 232 | 233 | func (d *sentinelFailover) switchMaster(masterAddr string) { 234 | internal.Logf( 235 | "sentinel: new master=%q addr=%q", 236 | d.masterName, masterAddr, 237 | ) 238 | _ = d.Pool().Filter(func(cn *pool.Conn) bool { 239 | return cn.RemoteAddr().String() != masterAddr 240 | }) 241 | d._masterAddr = masterAddr 242 | } 243 | 244 | func (d *sentinelFailover) setSentinel(sentinel *sentinelClient) { 245 | d.discoverSentinels(sentinel) 246 | d.sentinel = sentinel 247 | go d.listen(sentinel) 248 | } 249 | 250 | func (d *sentinelFailover) resetSentinel() error { 251 | var err error 252 | d.mu.Lock() 253 | if d.sentinel != nil { 254 | err = d._resetSentinel() 255 | } 256 | d.mu.Unlock() 257 | return err 258 | } 259 | 260 | func (d *sentinelFailover) _resetSentinel() error { 261 | err := d.sentinel.Close() 262 | d.sentinel = nil 263 | return err 264 | } 265 | 266 | func (d *sentinelFailover) discoverSentinels(sentinel *sentinelClient) { 267 | sentinels, err := sentinel.Sentinels(d.masterName).Result() 268 | if err != nil { 269 | internal.Logf("sentinel: Sentinels master=%q failed: %s", d.masterName, err) 270 | return 271 | } 272 | for _, sentinel := range sentinels { 273 | vals := sentinel.([]interface{}) 274 | for i := 0; i < len(vals); i += 2 { 275 | key := vals[i].(string) 276 | if key == "name" { 277 | sentinelAddr := vals[i+1].(string) 278 | if !contains(d.sentinelAddrs, sentinelAddr) { 279 | internal.Logf( 280 | "sentinel: discovered new sentinel=%q for master=%q", 281 | sentinelAddr, d.masterName, 282 | ) 283 | d.sentinelAddrs = append(d.sentinelAddrs, sentinelAddr) 284 | } 285 | } 286 | } 287 | } 288 | } 289 | 290 | func (d *sentinelFailover) listen(sentinel *sentinelClient) { 291 | var pubsub *PubSub 292 | for { 293 | if pubsub == nil { 294 | pubsub = sentinel.PubSub() 295 | 296 | if err := pubsub.Subscribe("+switch-master"); err != nil { 297 | internal.Logf("sentinel: Subscribe failed: %s", err) 298 | pubsub.Close() 299 | d.resetSentinel() 300 | return 301 | } 302 | } 303 | 304 | msg, err := pubsub.ReceiveMessage() 305 | if err != nil { 306 | if err != pool.ErrClosed { 307 | internal.Logf("sentinel: ReceiveMessage failed: %s", err) 308 | pubsub.Close() 309 | } 310 | d.resetSentinel() 311 | return 312 | } 313 | 314 | switch msg.Channel { 315 | case "+switch-master": 316 | parts := strings.Split(msg.Payload, " ") 317 | if parts[0] != d.masterName { 318 | internal.Logf("sentinel: ignore addr for master=%q", parts[0]) 319 | continue 320 | } 321 | addr := net.JoinHostPort(parts[3], parts[4]) 322 | 323 | d.mu.Lock() 324 | if d._masterAddr != addr { 325 | d.switchMaster(addr) 326 | } 327 | d.mu.Unlock() 328 | } 329 | } 330 | } 331 | 332 | func contains(slice []string, str string) bool { 333 | for _, s := range slice { 334 | if s == str { 335 | return true 336 | } 337 | } 338 | return false 339 | } 340 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/tx.go: -------------------------------------------------------------------------------- 1 | package redis 2 | 3 | import ( 4 | "github.com/go-redis/redis/internal/pool" 5 | "github.com/go-redis/redis/internal/proto" 6 | ) 7 | 8 | // TxFailedErr transaction redis failed. 9 | const TxFailedErr = proto.RedisError("redis: transaction failed") 10 | 11 | // Tx implements Redis transactions as described in 12 | // http://redis.io/topics/transactions. It's NOT safe for concurrent use 13 | // by multiple goroutines, because Exec resets list of watched keys. 14 | // If you don't need WATCH it is better to use Pipeline. 15 | type Tx struct { 16 | statefulCmdable 17 | baseClient 18 | } 19 | 20 | func (c *Client) newTx() *Tx { 21 | tx := Tx{ 22 | baseClient: baseClient{ 23 | opt: c.opt, 24 | connPool: pool.NewStickyConnPool(c.connPool.(*pool.ConnPool), true), 25 | }, 26 | } 27 | tx.baseClient.init() 28 | tx.statefulCmdable.setProcessor(tx.Process) 29 | return &tx 30 | } 31 | 32 | func (c *Client) Watch(fn func(*Tx) error, keys ...string) error { 33 | tx := c.newTx() 34 | if len(keys) > 0 { 35 | if err := tx.Watch(keys...).Err(); err != nil { 36 | _ = tx.Close() 37 | return err 38 | } 39 | } 40 | 41 | err := fn(tx) 42 | _ = tx.Close() 43 | return err 44 | } 45 | 46 | // Close closes the transaction, releasing any open resources. 47 | func (c *Tx) Close() error { 48 | _ = c.Unwatch().Err() 49 | return c.baseClient.Close() 50 | } 51 | 52 | // Watch marks the keys to be watched for conditional execution 53 | // of a transaction. 54 | func (c *Tx) Watch(keys ...string) *StatusCmd { 55 | args := make([]interface{}, 1+len(keys)) 56 | args[0] = "watch" 57 | for i, key := range keys { 58 | args[1+i] = key 59 | } 60 | cmd := NewStatusCmd(args...) 61 | c.Process(cmd) 62 | return cmd 63 | } 64 | 65 | // Unwatch flushes all the previously watched keys for a transaction. 66 | func (c *Tx) Unwatch(keys ...string) *StatusCmd { 67 | args := make([]interface{}, 1+len(keys)) 68 | args[0] = "unwatch" 69 | for i, key := range keys { 70 | args[1+i] = key 71 | } 72 | cmd := NewStatusCmd(args...) 73 | c.Process(cmd) 74 | return cmd 75 | } 76 | 77 | func (c *Tx) Pipeline() Pipeliner { 78 | pipe := Pipeline{ 79 | exec: c.processTxPipeline, 80 | } 81 | pipe.statefulCmdable.setProcessor(pipe.Process) 82 | return &pipe 83 | } 84 | 85 | // Pipelined executes commands queued in the fn in a transaction 86 | // and restores the connection state to normal. 87 | // 88 | // When using WATCH, EXEC will execute commands only if the watched keys 89 | // were not modified, allowing for a check-and-set mechanism. 90 | // 91 | // Exec always returns list of commands. If transaction fails 92 | // TxFailedErr is returned. Otherwise Exec returns error of the first 93 | // failed command or nil. 94 | func (c *Tx) Pipelined(fn func(Pipeliner) error) ([]Cmder, error) { 95 | return c.Pipeline().Pipelined(fn) 96 | } 97 | 98 | func (c *Tx) TxPipelined(fn func(Pipeliner) error) ([]Cmder, error) { 99 | return c.Pipelined(fn) 100 | } 101 | 102 | func (c *Tx) TxPipeline() Pipeliner { 103 | return c.Pipeline() 104 | } 105 | -------------------------------------------------------------------------------- /vendor/github.com/go-redis/redis/universal.go: -------------------------------------------------------------------------------- 1 | package redis 2 | 3 | import "time" 4 | 5 | // UniversalOptions information is required by UniversalClient to establish 6 | // connections. 7 | type UniversalOptions struct { 8 | // Either a single address or a seed list of host:port addresses 9 | // of cluster/sentinel nodes. 10 | Addrs []string 11 | 12 | // The sentinel master name. 13 | // Only failover clients. 14 | MasterName string 15 | 16 | // Database to be selected after connecting to the server. 17 | // Only single-node and failover clients. 18 | DB int 19 | 20 | // Only cluster clients. 21 | 22 | // Enables read only queries on slave nodes. 23 | ReadOnly bool 24 | 25 | MaxRedirects int 26 | RouteByLatency bool 27 | 28 | // Common options 29 | 30 | MaxRetries int 31 | Password string 32 | DialTimeout time.Duration 33 | ReadTimeout time.Duration 34 | WriteTimeout time.Duration 35 | PoolSize int 36 | PoolTimeout time.Duration 37 | IdleTimeout time.Duration 38 | IdleCheckFrequency time.Duration 39 | } 40 | 41 | func (o *UniversalOptions) cluster() *ClusterOptions { 42 | if len(o.Addrs) == 0 { 43 | o.Addrs = []string{"127.0.0.1:6379"} 44 | } 45 | 46 | return &ClusterOptions{ 47 | Addrs: o.Addrs, 48 | MaxRedirects: o.MaxRedirects, 49 | RouteByLatency: o.RouteByLatency, 50 | ReadOnly: o.ReadOnly, 51 | 52 | MaxRetries: o.MaxRetries, 53 | Password: o.Password, 54 | DialTimeout: o.DialTimeout, 55 | ReadTimeout: o.ReadTimeout, 56 | WriteTimeout: o.WriteTimeout, 57 | PoolSize: o.PoolSize, 58 | PoolTimeout: o.PoolTimeout, 59 | IdleTimeout: o.IdleTimeout, 60 | IdleCheckFrequency: o.IdleCheckFrequency, 61 | } 62 | } 63 | 64 | func (o *UniversalOptions) failover() *FailoverOptions { 65 | if len(o.Addrs) == 0 { 66 | o.Addrs = []string{"127.0.0.1:26379"} 67 | } 68 | 69 | return &FailoverOptions{ 70 | SentinelAddrs: o.Addrs, 71 | MasterName: o.MasterName, 72 | DB: o.DB, 73 | 74 | MaxRetries: o.MaxRetries, 75 | Password: o.Password, 76 | DialTimeout: o.DialTimeout, 77 | ReadTimeout: o.ReadTimeout, 78 | WriteTimeout: o.WriteTimeout, 79 | PoolSize: o.PoolSize, 80 | PoolTimeout: o.PoolTimeout, 81 | IdleTimeout: o.IdleTimeout, 82 | IdleCheckFrequency: o.IdleCheckFrequency, 83 | } 84 | } 85 | 86 | func (o *UniversalOptions) simple() *Options { 87 | addr := "127.0.0.1:6379" 88 | if len(o.Addrs) > 0 { 89 | addr = o.Addrs[0] 90 | } 91 | 92 | return &Options{ 93 | Addr: addr, 94 | DB: o.DB, 95 | 96 | MaxRetries: o.MaxRetries, 97 | Password: o.Password, 98 | DialTimeout: o.DialTimeout, 99 | ReadTimeout: o.ReadTimeout, 100 | WriteTimeout: o.WriteTimeout, 101 | PoolSize: o.PoolSize, 102 | PoolTimeout: o.PoolTimeout, 103 | IdleTimeout: o.IdleTimeout, 104 | IdleCheckFrequency: o.IdleCheckFrequency, 105 | } 106 | } 107 | 108 | // -------------------------------------------------------------------- 109 | 110 | // UniversalClient is an abstract client which - based on the provided options - 111 | // can connect to either clusters, or sentinel-backed failover instances or simple 112 | // single-instance servers. This can be useful for testing cluster-specific 113 | // applications locally. 114 | type UniversalClient interface { 115 | Cmdable 116 | Process(cmd Cmder) error 117 | WrapProcess(fn func(oldProcess func(cmd Cmder) error) func(cmd Cmder) error) 118 | Subscribe(channels ...string) *PubSub 119 | PSubscribe(channels ...string) *PubSub 120 | Close() error 121 | } 122 | 123 | var _ UniversalClient = (*Client)(nil) 124 | var _ UniversalClient = (*ClusterClient)(nil) 125 | 126 | // NewUniversalClient returns a new multi client. The type of client returned depends 127 | // on the following three conditions: 128 | // 129 | // 1. if a MasterName is passed a sentinel-backed FailoverClient will be returned 130 | // 2. if the number of Addrs is two or more, a ClusterClient will be returned 131 | // 3. otherwise, a single-node redis Client will be returned. 132 | func NewUniversalClient(opts *UniversalOptions) UniversalClient { 133 | if opts.MasterName != "" { 134 | return NewFailoverClient(opts.failover()) 135 | } else if len(opts.Addrs) > 1 { 136 | return NewClusterClient(opts.cluster()) 137 | } 138 | return NewClient(opts.simple()) 139 | } 140 | -------------------------------------------------------------------------------- /vendor/golang.org/x/net/LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2009 The Go Authors. All rights reserved. 2 | 3 | Redistribution and use in source and binary forms, with or without 4 | modification, are permitted provided that the following conditions are 5 | met: 6 | 7 | * Redistributions of source code must retain the above copyright 8 | notice, this list of conditions and the following disclaimer. 9 | * Redistributions in binary form must reproduce the above 10 | copyright notice, this list of conditions and the following disclaimer 11 | in the documentation and/or other materials provided with the 12 | distribution. 13 | * Neither the name of Google Inc. nor the names of its 14 | contributors may be used to endorse or promote products derived from 15 | this software without specific prior written permission. 16 | 17 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 18 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 19 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 20 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 21 | OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 22 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT 23 | LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 24 | DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 25 | THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 26 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 27 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 28 | -------------------------------------------------------------------------------- /vendor/golang.org/x/net/PATENTS: -------------------------------------------------------------------------------- 1 | Additional IP Rights Grant (Patents) 2 | 3 | "This implementation" means the copyrightable works distributed by 4 | Google as part of the Go project. 5 | 6 | Google hereby grants to You a perpetual, worldwide, non-exclusive, 7 | no-charge, royalty-free, irrevocable (except as stated in this section) 8 | patent license to make, have made, use, offer to sell, sell, import, 9 | transfer and otherwise run, modify and propagate the contents of this 10 | implementation of Go, where such license applies only to those patent 11 | claims, both currently owned or controlled by Google and acquired in 12 | the future, licensable by Google that are necessarily infringed by this 13 | implementation of Go. This grant does not include claims that would be 14 | infringed only as a consequence of further modification of this 15 | implementation. If you or your agent or exclusive licensee institute or 16 | order or agree to the institution of patent litigation against any 17 | entity (including a cross-claim or counterclaim in a lawsuit) alleging 18 | that this implementation of Go or any code incorporated within this 19 | implementation of Go constitutes direct or contributory patent 20 | infringement, or inducement of patent infringement, then any patent 21 | rights granted to you under this License for this implementation of Go 22 | shall terminate as of the date such litigation is filed. 23 | -------------------------------------------------------------------------------- /vendor/golang.org/x/net/html/atom/atom.go: -------------------------------------------------------------------------------- 1 | // Copyright 2012 The Go Authors. All rights reserved. 2 | // Use of this source code is governed by a BSD-style 3 | // license that can be found in the LICENSE file. 4 | 5 | // Package atom provides integer codes (also known as atoms) for a fixed set of 6 | // frequently occurring HTML strings: tag names and attribute keys such as "p" 7 | // and "id". 8 | // 9 | // Sharing an atom's name between all elements with the same tag can result in 10 | // fewer string allocations when tokenizing and parsing HTML. Integer 11 | // comparisons are also generally faster than string comparisons. 12 | // 13 | // The value of an atom's particular code is not guaranteed to stay the same 14 | // between versions of this package. Neither is any ordering guaranteed: 15 | // whether atom.H1 < atom.H2 may also change. The codes are not guaranteed to 16 | // be dense. The only guarantees are that e.g. looking up "div" will yield 17 | // atom.Div, calling atom.Div.String will return "div", and atom.Div != 0. 18 | package atom // import "golang.org/x/net/html/atom" 19 | 20 | // Atom is an integer code for a string. The zero value maps to "". 21 | type Atom uint32 22 | 23 | // String returns the atom's name. 24 | func (a Atom) String() string { 25 | start := uint32(a >> 8) 26 | n := uint32(a & 0xff) 27 | if start+n > uint32(len(atomText)) { 28 | return "" 29 | } 30 | return atomText[start : start+n] 31 | } 32 | 33 | func (a Atom) string() string { 34 | return atomText[a>>8 : a>>8+a&0xff] 35 | } 36 | 37 | // fnv computes the FNV hash with an arbitrary starting value h. 38 | func fnv(h uint32, s []byte) uint32 { 39 | for i := range s { 40 | h ^= uint32(s[i]) 41 | h *= 16777619 42 | } 43 | return h 44 | } 45 | 46 | func match(s string, t []byte) bool { 47 | for i, c := range t { 48 | if s[i] != c { 49 | return false 50 | } 51 | } 52 | return true 53 | } 54 | 55 | // Lookup returns the atom whose name is s. It returns zero if there is no 56 | // such atom. The lookup is case sensitive. 57 | func Lookup(s []byte) Atom { 58 | if len(s) == 0 || len(s) > maxAtomLen { 59 | return 0 60 | } 61 | h := fnv(hash0, s) 62 | if a := table[h&uint32(len(table)-1)]; int(a&0xff) == len(s) && match(a.string(), s) { 63 | return a 64 | } 65 | if a := table[(h>>16)&uint32(len(table)-1)]; int(a&0xff) == len(s) && match(a.string(), s) { 66 | return a 67 | } 68 | return 0 69 | } 70 | 71 | // String returns a string whose contents are equal to s. In that sense, it is 72 | // equivalent to string(s) but may be more efficient. 73 | func String(s []byte) string { 74 | if a := Lookup(s); a != 0 { 75 | return a.String() 76 | } 77 | return string(s) 78 | } 79 | -------------------------------------------------------------------------------- /vendor/golang.org/x/net/html/const.go: -------------------------------------------------------------------------------- 1 | // Copyright 2011 The Go Authors. All rights reserved. 2 | // Use of this source code is governed by a BSD-style 3 | // license that can be found in the LICENSE file. 4 | 5 | package html 6 | 7 | // Section 12.2.4.2 of the HTML5 specification says "The following elements 8 | // have varying levels of special parsing rules". 9 | // https://html.spec.whatwg.org/multipage/syntax.html#the-stack-of-open-elements 10 | var isSpecialElementMap = map[string]bool{ 11 | "address": true, 12 | "applet": true, 13 | "area": true, 14 | "article": true, 15 | "aside": true, 16 | "base": true, 17 | "basefont": true, 18 | "bgsound": true, 19 | "blockquote": true, 20 | "body": true, 21 | "br": true, 22 | "button": true, 23 | "caption": true, 24 | "center": true, 25 | "col": true, 26 | "colgroup": true, 27 | "dd": true, 28 | "details": true, 29 | "dir": true, 30 | "div": true, 31 | "dl": true, 32 | "dt": true, 33 | "embed": true, 34 | "fieldset": true, 35 | "figcaption": true, 36 | "figure": true, 37 | "footer": true, 38 | "form": true, 39 | "frame": true, 40 | "frameset": true, 41 | "h1": true, 42 | "h2": true, 43 | "h3": true, 44 | "h4": true, 45 | "h5": true, 46 | "h6": true, 47 | "head": true, 48 | "header": true, 49 | "hgroup": true, 50 | "hr": true, 51 | "html": true, 52 | "iframe": true, 53 | "img": true, 54 | "input": true, 55 | "isindex": true, // The 'isindex' element has been removed, but keep it for backwards compatibility. 56 | "keygen": true, 57 | "li": true, 58 | "link": true, 59 | "listing": true, 60 | "main": true, 61 | "marquee": true, 62 | "menu": true, 63 | "meta": true, 64 | "nav": true, 65 | "noembed": true, 66 | "noframes": true, 67 | "noscript": true, 68 | "object": true, 69 | "ol": true, 70 | "p": true, 71 | "param": true, 72 | "plaintext": true, 73 | "pre": true, 74 | "script": true, 75 | "section": true, 76 | "select": true, 77 | "source": true, 78 | "style": true, 79 | "summary": true, 80 | "table": true, 81 | "tbody": true, 82 | "td": true, 83 | "template": true, 84 | "textarea": true, 85 | "tfoot": true, 86 | "th": true, 87 | "thead": true, 88 | "title": true, 89 | "tr": true, 90 | "track": true, 91 | "ul": true, 92 | "wbr": true, 93 | "xmp": true, 94 | } 95 | 96 | func isSpecialElement(element *Node) bool { 97 | switch element.Namespace { 98 | case "", "html": 99 | return isSpecialElementMap[element.Data] 100 | case "svg": 101 | return element.Data == "foreignObject" 102 | } 103 | return false 104 | } 105 | -------------------------------------------------------------------------------- /vendor/golang.org/x/net/html/doc.go: -------------------------------------------------------------------------------- 1 | // Copyright 2010 The Go Authors. All rights reserved. 2 | // Use of this source code is governed by a BSD-style 3 | // license that can be found in the LICENSE file. 4 | 5 | /* 6 | Package html implements an HTML5-compliant tokenizer and parser. 7 | 8 | Tokenization is done by creating a Tokenizer for an io.Reader r. It is the 9 | caller's responsibility to ensure that r provides UTF-8 encoded HTML. 10 | 11 | z := html.NewTokenizer(r) 12 | 13 | Given a Tokenizer z, the HTML is tokenized by repeatedly calling z.Next(), 14 | which parses the next token and returns its type, or an error: 15 | 16 | for { 17 | tt := z.Next() 18 | if tt == html.ErrorToken { 19 | // ... 20 | return ... 21 | } 22 | // Process the current token. 23 | } 24 | 25 | There are two APIs for retrieving the current token. The high-level API is to 26 | call Token; the low-level API is to call Text or TagName / TagAttr. Both APIs 27 | allow optionally calling Raw after Next but before Token, Text, TagName, or 28 | TagAttr. In EBNF notation, the valid call sequence per token is: 29 | 30 | Next {Raw} [ Token | Text | TagName {TagAttr} ] 31 | 32 | Token returns an independent data structure that completely describes a token. 33 | Entities (such as "<") are unescaped, tag names and attribute keys are 34 | lower-cased, and attributes are collected into a []Attribute. For example: 35 | 36 | for { 37 | if z.Next() == html.ErrorToken { 38 | // Returning io.EOF indicates success. 39 | return z.Err() 40 | } 41 | emitToken(z.Token()) 42 | } 43 | 44 | The low-level API performs fewer allocations and copies, but the contents of 45 | the []byte values returned by Text, TagName and TagAttr may change on the next 46 | call to Next. For example, to extract an HTML page's anchor text: 47 | 48 | depth := 0 49 | for { 50 | tt := z.Next() 51 | switch tt { 52 | case html.ErrorToken: 53 | return z.Err() 54 | case html.TextToken: 55 | if depth > 0 { 56 | // emitBytes should copy the []byte it receives, 57 | // if it doesn't process it immediately. 58 | emitBytes(z.Text()) 59 | } 60 | case html.StartTagToken, html.EndTagToken: 61 | tn, _ := z.TagName() 62 | if len(tn) == 1 && tn[0] == 'a' { 63 | if tt == html.StartTagToken { 64 | depth++ 65 | } else { 66 | depth-- 67 | } 68 | } 69 | } 70 | } 71 | 72 | Parsing is done by calling Parse with an io.Reader, which returns the root of 73 | the parse tree (the document element) as a *Node. It is the caller's 74 | responsibility to ensure that the Reader provides UTF-8 encoded HTML. For 75 | example, to process each anchor node in depth-first order: 76 | 77 | doc, err := html.Parse(r) 78 | if err != nil { 79 | // ... 80 | } 81 | var f func(*html.Node) 82 | f = func(n *html.Node) { 83 | if n.Type == html.ElementNode && n.Data == "a" { 84 | // Do something with n... 85 | } 86 | for c := n.FirstChild; c != nil; c = c.NextSibling { 87 | f(c) 88 | } 89 | } 90 | f(doc) 91 | 92 | The relevant specifications include: 93 | https://html.spec.whatwg.org/multipage/syntax.html and 94 | https://html.spec.whatwg.org/multipage/syntax.html#tokenization 95 | */ 96 | package html // import "golang.org/x/net/html" 97 | 98 | // The tokenization algorithm implemented by this package is not a line-by-line 99 | // transliteration of the relatively verbose state-machine in the WHATWG 100 | // specification. A more direct approach is used instead, where the program 101 | // counter implies the state, such as whether it is tokenizing a tag or a text 102 | // node. Specification compliance is verified by checking expected and actual 103 | // outputs over a test suite rather than aiming for algorithmic fidelity. 104 | 105 | // TODO(nigeltao): Does a DOM API belong in this package or a separate one? 106 | // TODO(nigeltao): How does parsing interact with a JavaScript engine? 107 | -------------------------------------------------------------------------------- /vendor/golang.org/x/net/html/doctype.go: -------------------------------------------------------------------------------- 1 | // Copyright 2011 The Go Authors. All rights reserved. 2 | // Use of this source code is governed by a BSD-style 3 | // license that can be found in the LICENSE file. 4 | 5 | package html 6 | 7 | import ( 8 | "strings" 9 | ) 10 | 11 | // parseDoctype parses the data from a DoctypeToken into a name, 12 | // public identifier, and system identifier. It returns a Node whose Type 13 | // is DoctypeNode, whose Data is the name, and which has attributes 14 | // named "system" and "public" for the two identifiers if they were present. 15 | // quirks is whether the document should be parsed in "quirks mode". 16 | func parseDoctype(s string) (n *Node, quirks bool) { 17 | n = &Node{Type: DoctypeNode} 18 | 19 | // Find the name. 20 | space := strings.IndexAny(s, whitespace) 21 | if space == -1 { 22 | space = len(s) 23 | } 24 | n.Data = s[:space] 25 | // The comparison to "html" is case-sensitive. 26 | if n.Data != "html" { 27 | quirks = true 28 | } 29 | n.Data = strings.ToLower(n.Data) 30 | s = strings.TrimLeft(s[space:], whitespace) 31 | 32 | if len(s) < 6 { 33 | // It can't start with "PUBLIC" or "SYSTEM". 34 | // Ignore the rest of the string. 35 | return n, quirks || s != "" 36 | } 37 | 38 | key := strings.ToLower(s[:6]) 39 | s = s[6:] 40 | for key == "public" || key == "system" { 41 | s = strings.TrimLeft(s, whitespace) 42 | if s == "" { 43 | break 44 | } 45 | quote := s[0] 46 | if quote != '"' && quote != '\'' { 47 | break 48 | } 49 | s = s[1:] 50 | q := strings.IndexRune(s, rune(quote)) 51 | var id string 52 | if q == -1 { 53 | id = s 54 | s = "" 55 | } else { 56 | id = s[:q] 57 | s = s[q+1:] 58 | } 59 | n.Attr = append(n.Attr, Attribute{Key: key, Val: id}) 60 | if key == "public" { 61 | key = "system" 62 | } else { 63 | key = "" 64 | } 65 | } 66 | 67 | if key != "" || s != "" { 68 | quirks = true 69 | } else if len(n.Attr) > 0 { 70 | if n.Attr[0].Key == "public" { 71 | public := strings.ToLower(n.Attr[0].Val) 72 | switch public { 73 | case "-//w3o//dtd w3 html strict 3.0//en//", "-/w3d/dtd html 4.0 transitional/en", "html": 74 | quirks = true 75 | default: 76 | for _, q := range quirkyIDs { 77 | if strings.HasPrefix(public, q) { 78 | quirks = true 79 | break 80 | } 81 | } 82 | } 83 | // The following two public IDs only cause quirks mode if there is no system ID. 84 | if len(n.Attr) == 1 && (strings.HasPrefix(public, "-//w3c//dtd html 4.01 frameset//") || 85 | strings.HasPrefix(public, "-//w3c//dtd html 4.01 transitional//")) { 86 | quirks = true 87 | } 88 | } 89 | if lastAttr := n.Attr[len(n.Attr)-1]; lastAttr.Key == "system" && 90 | strings.ToLower(lastAttr.Val) == "http://www.ibm.com/data/dtd/v11/ibmxhtml1-transitional.dtd" { 91 | quirks = true 92 | } 93 | } 94 | 95 | return n, quirks 96 | } 97 | 98 | // quirkyIDs is a list of public doctype identifiers that cause a document 99 | // to be interpreted in quirks mode. The identifiers should be in lower case. 100 | var quirkyIDs = []string{ 101 | "+//silmaril//dtd html pro v0r11 19970101//", 102 | "-//advasoft ltd//dtd html 3.0 aswedit + extensions//", 103 | "-//as//dtd html 3.0 aswedit + extensions//", 104 | "-//ietf//dtd html 2.0 level 1//", 105 | "-//ietf//dtd html 2.0 level 2//", 106 | "-//ietf//dtd html 2.0 strict level 1//", 107 | "-//ietf//dtd html 2.0 strict level 2//", 108 | "-//ietf//dtd html 2.0 strict//", 109 | "-//ietf//dtd html 2.0//", 110 | "-//ietf//dtd html 2.1e//", 111 | "-//ietf//dtd html 3.0//", 112 | "-//ietf//dtd html 3.2 final//", 113 | "-//ietf//dtd html 3.2//", 114 | "-//ietf//dtd html 3//", 115 | "-//ietf//dtd html level 0//", 116 | "-//ietf//dtd html level 1//", 117 | "-//ietf//dtd html level 2//", 118 | "-//ietf//dtd html level 3//", 119 | "-//ietf//dtd html strict level 0//", 120 | "-//ietf//dtd html strict level 1//", 121 | "-//ietf//dtd html strict level 2//", 122 | "-//ietf//dtd html strict level 3//", 123 | "-//ietf//dtd html strict//", 124 | "-//ietf//dtd html//", 125 | "-//metrius//dtd metrius presentational//", 126 | "-//microsoft//dtd internet explorer 2.0 html strict//", 127 | "-//microsoft//dtd internet explorer 2.0 html//", 128 | "-//microsoft//dtd internet explorer 2.0 tables//", 129 | "-//microsoft//dtd internet explorer 3.0 html strict//", 130 | "-//microsoft//dtd internet explorer 3.0 html//", 131 | "-//microsoft//dtd internet explorer 3.0 tables//", 132 | "-//netscape comm. corp.//dtd html//", 133 | "-//netscape comm. corp.//dtd strict html//", 134 | "-//o'reilly and associates//dtd html 2.0//", 135 | "-//o'reilly and associates//dtd html extended 1.0//", 136 | "-//o'reilly and associates//dtd html extended relaxed 1.0//", 137 | "-//softquad software//dtd hotmetal pro 6.0::19990601::extensions to html 4.0//", 138 | "-//softquad//dtd hotmetal pro 4.0::19971010::extensions to html 4.0//", 139 | "-//spyglass//dtd html 2.0 extended//", 140 | "-//sq//dtd html 2.0 hotmetal + extensions//", 141 | "-//sun microsystems corp.//dtd hotjava html//", 142 | "-//sun microsystems corp.//dtd hotjava strict html//", 143 | "-//w3c//dtd html 3 1995-03-24//", 144 | "-//w3c//dtd html 3.2 draft//", 145 | "-//w3c//dtd html 3.2 final//", 146 | "-//w3c//dtd html 3.2//", 147 | "-//w3c//dtd html 3.2s draft//", 148 | "-//w3c//dtd html 4.0 frameset//", 149 | "-//w3c//dtd html 4.0 transitional//", 150 | "-//w3c//dtd html experimental 19960712//", 151 | "-//w3c//dtd html experimental 970421//", 152 | "-//w3c//dtd w3 html//", 153 | "-//w3o//dtd w3 html 3.0//", 154 | "-//webtechs//dtd mozilla html 2.0//", 155 | "-//webtechs//dtd mozilla html//", 156 | } 157 | -------------------------------------------------------------------------------- /vendor/golang.org/x/net/html/escape.go: -------------------------------------------------------------------------------- 1 | // Copyright 2010 The Go Authors. All rights reserved. 2 | // Use of this source code is governed by a BSD-style 3 | // license that can be found in the LICENSE file. 4 | 5 | package html 6 | 7 | import ( 8 | "bytes" 9 | "strings" 10 | "unicode/utf8" 11 | ) 12 | 13 | // These replacements permit compatibility with old numeric entities that 14 | // assumed Windows-1252 encoding. 15 | // https://html.spec.whatwg.org/multipage/syntax.html#consume-a-character-reference 16 | var replacementTable = [...]rune{ 17 | '\u20AC', // First entry is what 0x80 should be replaced with. 18 | '\u0081', 19 | '\u201A', 20 | '\u0192', 21 | '\u201E', 22 | '\u2026', 23 | '\u2020', 24 | '\u2021', 25 | '\u02C6', 26 | '\u2030', 27 | '\u0160', 28 | '\u2039', 29 | '\u0152', 30 | '\u008D', 31 | '\u017D', 32 | '\u008F', 33 | '\u0090', 34 | '\u2018', 35 | '\u2019', 36 | '\u201C', 37 | '\u201D', 38 | '\u2022', 39 | '\u2013', 40 | '\u2014', 41 | '\u02DC', 42 | '\u2122', 43 | '\u0161', 44 | '\u203A', 45 | '\u0153', 46 | '\u009D', 47 | '\u017E', 48 | '\u0178', // Last entry is 0x9F. 49 | // 0x00->'\uFFFD' is handled programmatically. 50 | // 0x0D->'\u000D' is a no-op. 51 | } 52 | 53 | // unescapeEntity reads an entity like "<" from b[src:] and writes the 54 | // corresponding "<" to b[dst:], returning the incremented dst and src cursors. 55 | // Precondition: b[src] == '&' && dst <= src. 56 | // attribute should be true if parsing an attribute value. 57 | func unescapeEntity(b []byte, dst, src int, attribute bool) (dst1, src1 int) { 58 | // https://html.spec.whatwg.org/multipage/syntax.html#consume-a-character-reference 59 | 60 | // i starts at 1 because we already know that s[0] == '&'. 61 | i, s := 1, b[src:] 62 | 63 | if len(s) <= 1 { 64 | b[dst] = b[src] 65 | return dst + 1, src + 1 66 | } 67 | 68 | if s[i] == '#' { 69 | if len(s) <= 3 { // We need to have at least "&#.". 70 | b[dst] = b[src] 71 | return dst + 1, src + 1 72 | } 73 | i++ 74 | c := s[i] 75 | hex := false 76 | if c == 'x' || c == 'X' { 77 | hex = true 78 | i++ 79 | } 80 | 81 | x := '\x00' 82 | for i < len(s) { 83 | c = s[i] 84 | i++ 85 | if hex { 86 | if '0' <= c && c <= '9' { 87 | x = 16*x + rune(c) - '0' 88 | continue 89 | } else if 'a' <= c && c <= 'f' { 90 | x = 16*x + rune(c) - 'a' + 10 91 | continue 92 | } else if 'A' <= c && c <= 'F' { 93 | x = 16*x + rune(c) - 'A' + 10 94 | continue 95 | } 96 | } else if '0' <= c && c <= '9' { 97 | x = 10*x + rune(c) - '0' 98 | continue 99 | } 100 | if c != ';' { 101 | i-- 102 | } 103 | break 104 | } 105 | 106 | if i <= 3 { // No characters matched. 107 | b[dst] = b[src] 108 | return dst + 1, src + 1 109 | } 110 | 111 | if 0x80 <= x && x <= 0x9F { 112 | // Replace characters from Windows-1252 with UTF-8 equivalents. 113 | x = replacementTable[x-0x80] 114 | } else if x == 0 || (0xD800 <= x && x <= 0xDFFF) || x > 0x10FFFF { 115 | // Replace invalid characters with the replacement character. 116 | x = '\uFFFD' 117 | } 118 | 119 | return dst + utf8.EncodeRune(b[dst:], x), src + i 120 | } 121 | 122 | // Consume the maximum number of characters possible, with the 123 | // consumed characters matching one of the named references. 124 | 125 | for i < len(s) { 126 | c := s[i] 127 | i++ 128 | // Lower-cased characters are more common in entities, so we check for them first. 129 | if 'a' <= c && c <= 'z' || 'A' <= c && c <= 'Z' || '0' <= c && c <= '9' { 130 | continue 131 | } 132 | if c != ';' { 133 | i-- 134 | } 135 | break 136 | } 137 | 138 | entityName := string(s[1:i]) 139 | if entityName == "" { 140 | // No-op. 141 | } else if attribute && entityName[len(entityName)-1] != ';' && len(s) > i && s[i] == '=' { 142 | // No-op. 143 | } else if x := entity[entityName]; x != 0 { 144 | return dst + utf8.EncodeRune(b[dst:], x), src + i 145 | } else if x := entity2[entityName]; x[0] != 0 { 146 | dst1 := dst + utf8.EncodeRune(b[dst:], x[0]) 147 | return dst1 + utf8.EncodeRune(b[dst1:], x[1]), src + i 148 | } else if !attribute { 149 | maxLen := len(entityName) - 1 150 | if maxLen > longestEntityWithoutSemicolon { 151 | maxLen = longestEntityWithoutSemicolon 152 | } 153 | for j := maxLen; j > 1; j-- { 154 | if x := entity[entityName[:j]]; x != 0 { 155 | return dst + utf8.EncodeRune(b[dst:], x), src + j + 1 156 | } 157 | } 158 | } 159 | 160 | dst1, src1 = dst+i, src+i 161 | copy(b[dst:dst1], b[src:src1]) 162 | return dst1, src1 163 | } 164 | 165 | // unescape unescapes b's entities in-place, so that "a<b" becomes "a': 214 | esc = ">" 215 | case '"': 216 | // """ is shorter than """. 217 | esc = """ 218 | case '\r': 219 | esc = " " 220 | default: 221 | panic("unrecognized escape character") 222 | } 223 | s = s[i+1:] 224 | if _, err := w.WriteString(esc); err != nil { 225 | return err 226 | } 227 | i = strings.IndexAny(s, escapedChars) 228 | } 229 | _, err := w.WriteString(s) 230 | return err 231 | } 232 | 233 | // EscapeString escapes special characters like "<" to become "<". It 234 | // escapes only five such characters: <, >, &, ' and ". 235 | // UnescapeString(EscapeString(s)) == s always holds, but the converse isn't 236 | // always true. 237 | func EscapeString(s string) string { 238 | if strings.IndexAny(s, escapedChars) == -1 { 239 | return s 240 | } 241 | var buf bytes.Buffer 242 | escape(&buf, s) 243 | return buf.String() 244 | } 245 | 246 | // UnescapeString unescapes entities like "<" to become "<". It unescapes a 247 | // larger range of entities than EscapeString escapes. For example, "á" 248 | // unescapes to "á", as does "á" and "&xE1;". 249 | // UnescapeString(EscapeString(s)) == s always holds, but the converse isn't 250 | // always true. 251 | func UnescapeString(s string) string { 252 | for _, c := range s { 253 | if c == '&' { 254 | return string(unescape([]byte(s), false)) 255 | } 256 | } 257 | return s 258 | } 259 | -------------------------------------------------------------------------------- /vendor/golang.org/x/net/html/foreign.go: -------------------------------------------------------------------------------- 1 | // Copyright 2011 The Go Authors. All rights reserved. 2 | // Use of this source code is governed by a BSD-style 3 | // license that can be found in the LICENSE file. 4 | 5 | package html 6 | 7 | import ( 8 | "strings" 9 | ) 10 | 11 | func adjustAttributeNames(aa []Attribute, nameMap map[string]string) { 12 | for i := range aa { 13 | if newName, ok := nameMap[aa[i].Key]; ok { 14 | aa[i].Key = newName 15 | } 16 | } 17 | } 18 | 19 | func adjustForeignAttributes(aa []Attribute) { 20 | for i, a := range aa { 21 | if a.Key == "" || a.Key[0] != 'x' { 22 | continue 23 | } 24 | switch a.Key { 25 | case "xlink:actuate", "xlink:arcrole", "xlink:href", "xlink:role", "xlink:show", 26 | "xlink:title", "xlink:type", "xml:base", "xml:lang", "xml:space", "xmlns:xlink": 27 | j := strings.Index(a.Key, ":") 28 | aa[i].Namespace = a.Key[:j] 29 | aa[i].Key = a.Key[j+1:] 30 | } 31 | } 32 | } 33 | 34 | func htmlIntegrationPoint(n *Node) bool { 35 | if n.Type != ElementNode { 36 | return false 37 | } 38 | switch n.Namespace { 39 | case "math": 40 | if n.Data == "annotation-xml" { 41 | for _, a := range n.Attr { 42 | if a.Key == "encoding" { 43 | val := strings.ToLower(a.Val) 44 | if val == "text/html" || val == "application/xhtml+xml" { 45 | return true 46 | } 47 | } 48 | } 49 | } 50 | case "svg": 51 | switch n.Data { 52 | case "desc", "foreignObject", "title": 53 | return true 54 | } 55 | } 56 | return false 57 | } 58 | 59 | func mathMLTextIntegrationPoint(n *Node) bool { 60 | if n.Namespace != "math" { 61 | return false 62 | } 63 | switch n.Data { 64 | case "mi", "mo", "mn", "ms", "mtext": 65 | return true 66 | } 67 | return false 68 | } 69 | 70 | // Section 12.2.6.5. 71 | var breakout = map[string]bool{ 72 | "b": true, 73 | "big": true, 74 | "blockquote": true, 75 | "body": true, 76 | "br": true, 77 | "center": true, 78 | "code": true, 79 | "dd": true, 80 | "div": true, 81 | "dl": true, 82 | "dt": true, 83 | "em": true, 84 | "embed": true, 85 | "h1": true, 86 | "h2": true, 87 | "h3": true, 88 | "h4": true, 89 | "h5": true, 90 | "h6": true, 91 | "head": true, 92 | "hr": true, 93 | "i": true, 94 | "img": true, 95 | "li": true, 96 | "listing": true, 97 | "menu": true, 98 | "meta": true, 99 | "nobr": true, 100 | "ol": true, 101 | "p": true, 102 | "pre": true, 103 | "ruby": true, 104 | "s": true, 105 | "small": true, 106 | "span": true, 107 | "strong": true, 108 | "strike": true, 109 | "sub": true, 110 | "sup": true, 111 | "table": true, 112 | "tt": true, 113 | "u": true, 114 | "ul": true, 115 | "var": true, 116 | } 117 | 118 | // Section 12.2.6.5. 119 | var svgTagNameAdjustments = map[string]string{ 120 | "altglyph": "altGlyph", 121 | "altglyphdef": "altGlyphDef", 122 | "altglyphitem": "altGlyphItem", 123 | "animatecolor": "animateColor", 124 | "animatemotion": "animateMotion", 125 | "animatetransform": "animateTransform", 126 | "clippath": "clipPath", 127 | "feblend": "feBlend", 128 | "fecolormatrix": "feColorMatrix", 129 | "fecomponenttransfer": "feComponentTransfer", 130 | "fecomposite": "feComposite", 131 | "feconvolvematrix": "feConvolveMatrix", 132 | "fediffuselighting": "feDiffuseLighting", 133 | "fedisplacementmap": "feDisplacementMap", 134 | "fedistantlight": "feDistantLight", 135 | "feflood": "feFlood", 136 | "fefunca": "feFuncA", 137 | "fefuncb": "feFuncB", 138 | "fefuncg": "feFuncG", 139 | "fefuncr": "feFuncR", 140 | "fegaussianblur": "feGaussianBlur", 141 | "feimage": "feImage", 142 | "femerge": "feMerge", 143 | "femergenode": "feMergeNode", 144 | "femorphology": "feMorphology", 145 | "feoffset": "feOffset", 146 | "fepointlight": "fePointLight", 147 | "fespecularlighting": "feSpecularLighting", 148 | "fespotlight": "feSpotLight", 149 | "fetile": "feTile", 150 | "feturbulence": "feTurbulence", 151 | "foreignobject": "foreignObject", 152 | "glyphref": "glyphRef", 153 | "lineargradient": "linearGradient", 154 | "radialgradient": "radialGradient", 155 | "textpath": "textPath", 156 | } 157 | 158 | // Section 12.2.6.1 159 | var mathMLAttributeAdjustments = map[string]string{ 160 | "definitionurl": "definitionURL", 161 | } 162 | 163 | var svgAttributeAdjustments = map[string]string{ 164 | "attributename": "attributeName", 165 | "attributetype": "attributeType", 166 | "basefrequency": "baseFrequency", 167 | "baseprofile": "baseProfile", 168 | "calcmode": "calcMode", 169 | "clippathunits": "clipPathUnits", 170 | "contentscripttype": "contentScriptType", 171 | "contentstyletype": "contentStyleType", 172 | "diffuseconstant": "diffuseConstant", 173 | "edgemode": "edgeMode", 174 | "externalresourcesrequired": "externalResourcesRequired", 175 | "filterres": "filterRes", 176 | "filterunits": "filterUnits", 177 | "glyphref": "glyphRef", 178 | "gradienttransform": "gradientTransform", 179 | "gradientunits": "gradientUnits", 180 | "kernelmatrix": "kernelMatrix", 181 | "kernelunitlength": "kernelUnitLength", 182 | "keypoints": "keyPoints", 183 | "keysplines": "keySplines", 184 | "keytimes": "keyTimes", 185 | "lengthadjust": "lengthAdjust", 186 | "limitingconeangle": "limitingConeAngle", 187 | "markerheight": "markerHeight", 188 | "markerunits": "markerUnits", 189 | "markerwidth": "markerWidth", 190 | "maskcontentunits": "maskContentUnits", 191 | "maskunits": "maskUnits", 192 | "numoctaves": "numOctaves", 193 | "pathlength": "pathLength", 194 | "patterncontentunits": "patternContentUnits", 195 | "patterntransform": "patternTransform", 196 | "patternunits": "patternUnits", 197 | "pointsatx": "pointsAtX", 198 | "pointsaty": "pointsAtY", 199 | "pointsatz": "pointsAtZ", 200 | "preservealpha": "preserveAlpha", 201 | "preserveaspectratio": "preserveAspectRatio", 202 | "primitiveunits": "primitiveUnits", 203 | "refx": "refX", 204 | "refy": "refY", 205 | "repeatcount": "repeatCount", 206 | "repeatdur": "repeatDur", 207 | "requiredextensions": "requiredExtensions", 208 | "requiredfeatures": "requiredFeatures", 209 | "specularconstant": "specularConstant", 210 | "specularexponent": "specularExponent", 211 | "spreadmethod": "spreadMethod", 212 | "startoffset": "startOffset", 213 | "stddeviation": "stdDeviation", 214 | "stitchtiles": "stitchTiles", 215 | "surfacescale": "surfaceScale", 216 | "systemlanguage": "systemLanguage", 217 | "tablevalues": "tableValues", 218 | "targetx": "targetX", 219 | "targety": "targetY", 220 | "textlength": "textLength", 221 | "viewbox": "viewBox", 222 | "viewtarget": "viewTarget", 223 | "xchannelselector": "xChannelSelector", 224 | "ychannelselector": "yChannelSelector", 225 | "zoomandpan": "zoomAndPan", 226 | } 227 | -------------------------------------------------------------------------------- /vendor/golang.org/x/net/html/node.go: -------------------------------------------------------------------------------- 1 | // Copyright 2011 The Go Authors. All rights reserved. 2 | // Use of this source code is governed by a BSD-style 3 | // license that can be found in the LICENSE file. 4 | 5 | package html 6 | 7 | import ( 8 | "golang.org/x/net/html/atom" 9 | ) 10 | 11 | // A NodeType is the type of a Node. 12 | type NodeType uint32 13 | 14 | const ( 15 | ErrorNode NodeType = iota 16 | TextNode 17 | DocumentNode 18 | ElementNode 19 | CommentNode 20 | DoctypeNode 21 | scopeMarkerNode 22 | ) 23 | 24 | // Section 12.2.4.3 says "The markers are inserted when entering applet, 25 | // object, marquee, template, td, th, and caption elements, and are used 26 | // to prevent formatting from "leaking" into applet, object, marquee, 27 | // template, td, th, and caption elements". 28 | var scopeMarker = Node{Type: scopeMarkerNode} 29 | 30 | // A Node consists of a NodeType and some Data (tag name for element nodes, 31 | // content for text) and are part of a tree of Nodes. Element nodes may also 32 | // have a Namespace and contain a slice of Attributes. Data is unescaped, so 33 | // that it looks like "a 0 { 161 | return (*s)[i-1] 162 | } 163 | return nil 164 | } 165 | 166 | // index returns the index of the top-most occurrence of n in the stack, or -1 167 | // if n is not present. 168 | func (s *nodeStack) index(n *Node) int { 169 | for i := len(*s) - 1; i >= 0; i-- { 170 | if (*s)[i] == n { 171 | return i 172 | } 173 | } 174 | return -1 175 | } 176 | 177 | // insert inserts a node at the given index. 178 | func (s *nodeStack) insert(i int, n *Node) { 179 | (*s) = append(*s, nil) 180 | copy((*s)[i+1:], (*s)[i:]) 181 | (*s)[i] = n 182 | } 183 | 184 | // remove removes a node from the stack. It is a no-op if n is not present. 185 | func (s *nodeStack) remove(n *Node) { 186 | i := s.index(n) 187 | if i == -1 { 188 | return 189 | } 190 | copy((*s)[i:], (*s)[i+1:]) 191 | j := len(*s) - 1 192 | (*s)[j] = nil 193 | *s = (*s)[:j] 194 | } 195 | -------------------------------------------------------------------------------- /vendor/golang.org/x/net/html/render.go: -------------------------------------------------------------------------------- 1 | // Copyright 2011 The Go Authors. All rights reserved. 2 | // Use of this source code is governed by a BSD-style 3 | // license that can be found in the LICENSE file. 4 | 5 | package html 6 | 7 | import ( 8 | "bufio" 9 | "errors" 10 | "fmt" 11 | "io" 12 | "strings" 13 | ) 14 | 15 | type writer interface { 16 | io.Writer 17 | io.ByteWriter 18 | WriteString(string) (int, error) 19 | } 20 | 21 | // Render renders the parse tree n to the given writer. 22 | // 23 | // Rendering is done on a 'best effort' basis: calling Parse on the output of 24 | // Render will always result in something similar to the original tree, but it 25 | // is not necessarily an exact clone unless the original tree was 'well-formed'. 26 | // 'Well-formed' is not easily specified; the HTML5 specification is 27 | // complicated. 28 | // 29 | // Calling Parse on arbitrary input typically results in a 'well-formed' parse 30 | // tree. However, it is possible for Parse to yield a 'badly-formed' parse tree. 31 | // For example, in a 'well-formed' parse tree, no element is a child of 32 | // another element: parsing "" results in two sibling elements. 33 | // Similarly, in a 'well-formed' parse tree, no element is a child of a 34 | // element: parsing "

" results in a

with two sibling 35 | // children; the is reparented to the

's parent. However, calling 36 | // Parse on "
" does not return an error, but the result has an 37 | // element with an child, and is therefore not 'well-formed'. 38 | // 39 | // Programmatically constructed trees are typically also 'well-formed', but it 40 | // is possible to construct a tree that looks innocuous but, when rendered and 41 | // re-parsed, results in a different tree. A simple example is that a solitary 42 | // text node would become a tree containing , and elements. 43 | // Another example is that the programmatic equivalent of "abc" 44 | // becomes "abc". 45 | func Render(w io.Writer, n *Node) error { 46 | if x, ok := w.(writer); ok { 47 | return render(x, n) 48 | } 49 | buf := bufio.NewWriter(w) 50 | if err := render(buf, n); err != nil { 51 | return err 52 | } 53 | return buf.Flush() 54 | } 55 | 56 | // plaintextAbort is returned from render1 when a element 57 | // has been rendered. No more end tags should be rendered after that. 58 | var plaintextAbort = errors.New("html: internal error (plaintext abort)") 59 | 60 | func render(w writer, n *Node) error { 61 | err := render1(w, n) 62 | if err == plaintextAbort { 63 | err = nil 64 | } 65 | return err 66 | } 67 | 68 | func render1(w writer, n *Node) error { 69 | // Render non-element nodes; these are the easy cases. 70 | switch n.Type { 71 | case ErrorNode: 72 | return errors.New("html: cannot render an ErrorNode node") 73 | case TextNode: 74 | return escape(w, n.Data) 75 | case DocumentNode: 76 | for c := n.FirstChild; c != nil; c = c.NextSibling { 77 | if err := render1(w, c); err != nil { 78 | return err 79 | } 80 | } 81 | return nil 82 | case ElementNode: 83 | // No-op. 84 | case CommentNode: 85 | if _, err := w.WriteString("<!--"); err != nil { 86 | return err 87 | } 88 | if _, err := w.WriteString(n.Data); err != nil { 89 | return err 90 | } 91 | if _, err := w.WriteString("-->"); err != nil { 92 | return err 93 | } 94 | return nil 95 | case DoctypeNode: 96 | if _, err := w.WriteString("<!DOCTYPE "); err != nil { 97 | return err 98 | } 99 | if _, err := w.WriteString(n.Data); err != nil { 100 | return err 101 | } 102 | if n.Attr != nil { 103 | var p, s string 104 | for _, a := range n.Attr { 105 | switch a.Key { 106 | case "public": 107 | p = a.Val 108 | case "system": 109 | s = a.Val 110 | } 111 | } 112 | if p != "" { 113 | if _, err := w.WriteString(" PUBLIC "); err != nil { 114 | return err 115 | } 116 | if err := writeQuoted(w, p); err != nil { 117 | return err 118 | } 119 | if s != "" { 120 | if err := w.WriteByte(' '); err != nil { 121 | return err 122 | } 123 | if err := writeQuoted(w, s); err != nil { 124 | return err 125 | } 126 | } 127 | } else if s != "" { 128 | if _, err := w.WriteString(" SYSTEM "); err != nil { 129 | return err 130 | } 131 | if err := writeQuoted(w, s); err != nil { 132 | return err 133 | } 134 | } 135 | } 136 | return w.WriteByte('>') 137 | default: 138 | return errors.New("html: unknown node type") 139 | } 140 | 141 | // Render the <xxx> opening tag. 142 | if err := w.WriteByte('<'); err != nil { 143 | return err 144 | } 145 | if _, err := w.WriteString(n.Data); err != nil { 146 | return err 147 | } 148 | for _, a := range n.Attr { 149 | if err := w.WriteByte(' '); err != nil { 150 | return err 151 | } 152 | if a.Namespace != "" { 153 | if _, err := w.WriteString(a.Namespace); err != nil { 154 | return err 155 | } 156 | if err := w.WriteByte(':'); err != nil { 157 | return err 158 | } 159 | } 160 | if _, err := w.WriteString(a.Key); err != nil { 161 | return err 162 | } 163 | if _, err := w.WriteString(`="`); err != nil { 164 | return err 165 | } 166 | if err := escape(w, a.Val); err != nil { 167 | return err 168 | } 169 | if err := w.WriteByte('"'); err != nil { 170 | return err 171 | } 172 | } 173 | if voidElements[n.Data] { 174 | if n.FirstChild != nil { 175 | return fmt.Errorf("html: void element <%s> has child nodes", n.Data) 176 | } 177 | _, err := w.WriteString("/>") 178 | return err 179 | } 180 | if err := w.WriteByte('>'); err != nil { 181 | return err 182 | } 183 | 184 | // Add initial newline where there is danger of a newline beging ignored. 185 | if c := n.FirstChild; c != nil && c.Type == TextNode && strings.HasPrefix(c.Data, "\n") { 186 | switch n.Data { 187 | case "pre", "listing", "textarea": 188 | if err := w.WriteByte('\n'); err != nil { 189 | return err 190 | } 191 | } 192 | } 193 | 194 | // Render any child nodes. 195 | switch n.Data { 196 | case "iframe", "noembed", "noframes", "noscript", "plaintext", "script", "style", "xmp": 197 | for c := n.FirstChild; c != nil; c = c.NextSibling { 198 | if c.Type == TextNode { 199 | if _, err := w.WriteString(c.Data); err != nil { 200 | return err 201 | } 202 | } else { 203 | if err := render1(w, c); err != nil { 204 | return err 205 | } 206 | } 207 | } 208 | if n.Data == "plaintext" { 209 | // Don't render anything else. <plaintext> must be the 210 | // last element in the file, with no closing tag. 211 | return plaintextAbort 212 | } 213 | default: 214 | for c := n.FirstChild; c != nil; c = c.NextSibling { 215 | if err := render1(w, c); err != nil { 216 | return err 217 | } 218 | } 219 | } 220 | 221 | // Render the </xxx> closing tag. 222 | if _, err := w.WriteString("</"); err != nil { 223 | return err 224 | } 225 | if _, err := w.WriteString(n.Data); err != nil { 226 | return err 227 | } 228 | return w.WriteByte('>') 229 | } 230 | 231 | // writeQuoted writes s to w surrounded by quotes. Normally it will use double 232 | // quotes, but if s contains a double quote, it will use single quotes. 233 | // It is used for writing the identifiers in a doctype declaration. 234 | // In valid HTML, they can't contain both types of quotes. 235 | func writeQuoted(w writer, s string) error { 236 | var q byte = '"' 237 | if strings.Contains(s, `"`) { 238 | q = '\'' 239 | } 240 | if err := w.WriteByte(q); err != nil { 241 | return err 242 | } 243 | if _, err := w.WriteString(s); err != nil { 244 | return err 245 | } 246 | if err := w.WriteByte(q); err != nil { 247 | return err 248 | } 249 | return nil 250 | } 251 | 252 | // Section 12.1.2, "Elements", gives this list of void elements. Void elements 253 | // are those that can't have any contents. 254 | var voidElements = map[string]bool{ 255 | "area": true, 256 | "base": true, 257 | "br": true, 258 | "col": true, 259 | "command": true, 260 | "embed": true, 261 | "hr": true, 262 | "img": true, 263 | "input": true, 264 | "keygen": true, 265 | "link": true, 266 | "meta": true, 267 | "param": true, 268 | "source": true, 269 | "track": true, 270 | "wbr": true, 271 | } 272 | -------------------------------------------------------------------------------- /vendor/vendor.json: -------------------------------------------------------------------------------- 1 | { 2 | "comment": "", 3 | "ignore": "test", 4 | "package": [ 5 | { 6 | "checksumSHA1": "uvutHOxcwOHqLNLbHcTz5cI3fEA=", 7 | "path": "github.com/PuerkitoBio/goquery", 8 | "revision": "ea1bc64a63088ac2bbf1a2edcae38413794a7710", 9 | "revisionTime": "2018-03-24T16:22:12Z" 10 | }, 11 | { 12 | "checksumSHA1": "Q/2QpI7E35SsNYfaxLsWHFry9k4=", 13 | "path": "github.com/andybalholm/cascadia", 14 | "revision": "901648c87902174f774fac311d7f176f8647bdaa", 15 | "revisionTime": "2018-02-20T18:43:36Z" 16 | }, 17 | { 18 | "checksumSHA1": "GE9y3KX8Fe7oXWPnUJ70BEmX3cg=", 19 | "path": "github.com/go-redis/redis", 20 | "revision": "877867d2845fbaf86798befe410b6ceb6f5c29a3", 21 | "revisionTime": "2018-03-14T10:42:51Z" 22 | }, 23 | { 24 | "checksumSHA1": "L/n8TwOiRwhhL2TvFNafxB6rAwY=", 25 | "path": "github.com/go-redis/redis/internal", 26 | "revision": "877867d2845fbaf86798befe410b6ceb6f5c29a3", 27 | "revisionTime": "2018-03-14T10:42:51Z" 28 | }, 29 | { 30 | "checksumSHA1": "GQZsUVg/+6UpQAYpc4luMvMutSI=", 31 | "path": "github.com/go-redis/redis/internal/consistenthash", 32 | "revision": "877867d2845fbaf86798befe410b6ceb6f5c29a3", 33 | "revisionTime": "2018-03-14T10:42:51Z" 34 | }, 35 | { 36 | "checksumSHA1": "GE9tg94JwCzMbBwn3Q2LaFs5Rx0=", 37 | "path": "github.com/go-redis/redis/internal/hashtag", 38 | "revision": "877867d2845fbaf86798befe410b6ceb6f5c29a3", 39 | "revisionTime": "2018-03-14T10:42:51Z" 40 | }, 41 | { 42 | "checksumSHA1": "AtcctlDri3zLh6U5sDnT4NULEtw=", 43 | "path": "github.com/go-redis/redis/internal/pool", 44 | "revision": "877867d2845fbaf86798befe410b6ceb6f5c29a3", 45 | "revisionTime": "2018-03-14T10:42:51Z" 46 | }, 47 | { 48 | "checksumSHA1": "2DZs/18lBWzZk2rxWLVtPUzNXhs=", 49 | "path": "github.com/go-redis/redis/internal/proto", 50 | "revision": "877867d2845fbaf86798befe410b6ceb6f5c29a3", 51 | "revisionTime": "2018-03-14T10:42:51Z" 52 | }, 53 | { 54 | "checksumSHA1": "sjbjSJrTdvyDwNjrWjTNeXgUhaU=", 55 | "path": "github.com/go-redis/redis/internal/singleflight", 56 | "revision": "877867d2845fbaf86798befe410b6ceb6f5c29a3", 57 | "revisionTime": "2018-03-14T10:42:51Z" 58 | }, 59 | { 60 | "checksumSHA1": "SvyqZrzHTVxCIBoawql2ucvfWLE=", 61 | "path": "github.com/go-redis/redis/internal/util", 62 | "revision": "877867d2845fbaf86798befe410b6ceb6f5c29a3", 63 | "revisionTime": "2018-03-14T10:42:51Z" 64 | }, 65 | { 66 | "checksumSHA1": "L68uqaYKEOxkq2xwnNV2bRP7yO4=", 67 | "path": "golang.org/x/net/html", 68 | "revision": "6078986fec03a1dcc236c34816c71b0e05018fda", 69 | "revisionTime": "2017-09-09T04:35:08Z" 70 | }, 71 | { 72 | "checksumSHA1": "NvlH++AI8+25Z2Vs8WKxLVCNrxo=", 73 | "path": "golang.org/x/net/html/atom", 74 | "revision": "6078986fec03a1dcc236c34816c71b0e05018fda", 75 | "revisionTime": "2017-09-09T04:35:08Z" 76 | } 77 | ], 78 | "rootPath": "github-crawler" 79 | } 80 | --------------------------------------------------------------------------------