├── LICENSE ├── README_zh.md ├── ai ├── client.go └── message.go ├── cmd ├── import.go ├── import_test.go ├── root.go ├── search.go └── search_test.go ├── example └── data.json ├── go.mod ├── go.sum ├── main.go ├── options └── config_flags.go ├── qdrant └── client.go ├── readme.md └── util ├── proxy.go └── proxy_test.go /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2023 song 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README_zh.md: -------------------------------------------------------------------------------- 1 | Translations: [English](readme.md) | [简体中文](README_zh.md) 2 | ## kbai 3 | 使用 golang 基于 openai chatgpt embedding + qdrant 实现知识库的导入和问答 4 | ``` 5 | ❯ kbai -h 6 | a local knowledge base, based on chatgpt and qdrant 7 | 8 | usage: 9 | kbai [flags] 10 | kbai [command] 11 | 12 | available commands: 13 | completion generate the autocompletion script for the specified shell 14 | help help about any command 15 | import import data to vector database 16 | search ask the knowledge base example: kbai ask --msg 'first, the chicken or the egg' 17 | 18 | flags: 19 | --apikey string openai apikey:default from env apikey 20 | --collection string qdrant collection name default: kubernetes (default "kubernetes") 21 | -h, --help help for kbai 22 | --proxy string http client proxy default:socks5://127.0.0.1:1080 (default "socks5://127.0.0.1:1080") 23 | --qdrant string qdrant address default: 127.0.0.1:6334 (default "127.0.0.1:6334") 24 | --vectorsize uint qdrant vector size default: 1536 (default 1536) 25 | 26 | use "kbai [command] --help" for more information about a command. 27 | ``` 28 | ### 流程 29 | ![](https://img-blog.csdnimg.cn/img_convert/ef425236d64bca26fb73bf1d01614b50.png) 30 | ## 安装 31 | go build 安装 32 | ``` 33 | sudo go build -o kbai github.com/webws/embedding-knowledge-base && sudo mv ./kbai /usr/local/bin 34 | ``` 35 | 或 golang 源码使用 36 | ``` 37 | git clone https://github.com/webws/embedding-knowledge-base.git && cd ./embedding-knowledge-base 38 | 39 | ``` 40 | ## 使用示例 41 | 必须先启动qdrant qdrant 是一个开源的向量搜索引擎,支持多种向量距离计算方式 42 | 43 | docker 运行 qdrant 44 | ``` 45 | docker run --rm -p 6334:6334 qdrant/qdrant 46 | ``` 47 | 48 | 设置 openai apikey 49 | ``` 50 | export apikey=xxx 51 | ``` 52 | ### 导入 53 | 54 | 这里使用的测试数据是k8s相关的知识库,真实数据需自己准备 55 | 56 | 导入知识库 57 | ``` 58 | kbai import --datafile ./example/data.json 59 | ``` 60 | data.json 数据格式如下,为 真实数据需自己准备 61 | 62 | ``` 63 | [ 64 | { 65 | "questions": "这是问题", 66 | "answers": "这是答案" 67 | }, 68 | ] 69 | ``` 70 | 说明: 71 | ```text 72 | 默认的 代理 是 "socks5://127.0.0.1:1080" 自定义 可使用 --proxy 指定 73 | ``` 74 | ### kbai 搜索数据 75 | 搜索问题(源码执行) 76 | ``` 77 | kbai search --msg "网关是什么" 78 | ``` 79 | 回答 80 | ```text 81 | the answer to the knowledge base: 82 | 在kubernetes中，网关通常指的是ingress（入 口）资源对象。ingress是一种kubernetes api对象，用于配置和管理集群中的http和https流量入口。它充当了从集群外部访问集群内部服务的入口点 83 | 84 | results of chatgpt answers with reference answers: 85 | ，同时提供负载均衡、ssl/tls终止和基于域名的路由等功能。ingress资源对象定义了一组规则，这些规则指定了通过特定http路径或主机名将请求路由到后端服务的方式。可以使用不同的ingress控制器实现这些规则，如nginx、traefik等。这样就可以在集群中创建多个ingress资源对象来管理不同的流量入口。 86 | 87 | only chatgpt answers: 88 | 网关是一种网络设备，用于连接两个或多个不同类型的网络，以便实现数据以不同协议进行传递和转换。网关起到了连接不同网络之间的桥梁作用，将两个或多个网络互相连接起来，并负责数据的路由和转发。网关可以是硬件设备，如路由器，也可以是软件程序，如互联网网关。网关通常用于连接本地网络与互联网，使得局域网中的计算机能够访问互联网上的资源。除了连接不同网络的功能，网关还可以实现安全性、负载均衡、数据过滤等功能。 89 | ``` 90 | 1. 第一个是知识库的回答(the answer to the knowledge base): 91 | 2. 第二个 是结合知识库 chatgpt 的回答(results of chatgpt answers with reference answers) 92 | 3. 第三个 仅chatgpt 回答 93 | 94 | 可以看出 直接问chatgpt,得到的答案可能跟k8s无关,结合k8s本地知识库,可以让回答偏向 数据集设定的主题 95 | 96 | 如果直接搜索 与知识库无关或违规问题,将搜索不到任务数据 97 | 98 | ``` 99 | go run ./ search --msg "苹果不洗能吃吗" 100 | ``` 101 | 回答 102 | ``` 103 | rearch term violation or exceeding category 104 | ``` 105 | ## 参考地址 106 | * [https://github.com/spf13/cobra](https://github.com/spf13/cobra) 107 | * [https://github.com/kubernetes/kubernetes](https://github.com/kubernetes/kubernetes) 108 | * [https://github.com/gohugoio/hugo](https://github.com/gohugoio/hugo) 109 | * [https://github.com/qdrant/qdrant](https://github.com/qdrant/qdrant) 110 | 111 | 112 | -------------------------------------------------------------------------------- /ai/client.go: -------------------------------------------------------------------------------- 1 | package ai 2 | 3 | import ( 4 | "context" 5 | "errors" 6 | "log" 7 | "net/http" 8 | "net/url" 9 | 10 | openai "github.com/sashabaranov/go-openai" 11 | "github.com/webws/embedding-knowledge-base/util" 12 | "github.com/webws/go-moda/logger" 13 | ) 14 | 15 | type AIClient struct { 16 | openai *openai.Client 17 | proxy string 18 | apiKey string 19 | } 20 | 21 | func NewAiClient(proxy, apiKey string, needCheckProxy bool) (*AIClient, error) { 22 | c := &AIClient{proxy: proxy, apiKey: apiKey} 23 | config := openai.DefaultConfig(c.apiKey) 24 | config.HTTPClient, _ = newHttpClientWithProxy(c.proxy) 25 | pass := util.ProxyCheck(config.HTTPClient, c.proxy, "https://www.google.com", 5) 26 | if needCheckProxy && !pass { 27 | config.HTTPClient, _ = newHttpClientWithProxy("") 28 | } 29 | c.openai = openai.NewClientWithConfig(config) 30 | return c, nil 31 | } 32 | 33 | func newHttpClientWithProxy(proxy string) (*http.Client, error) { 34 | client := &http.Client{} 35 | if proxy != "" { 36 | uri, err := url.Parse(proxy) 37 | if err != nil { 38 | log.Fatalln(err) 39 | logger.Errorw("NewAiClient", "err", err) 40 | return nil, err 41 | 42 | } 43 | client.Transport = &http.Transport{ 44 | Proxy: http.ProxyURL(uri), 45 | } 46 | } 47 | return client, nil 48 | } 49 | 50 | // SimpleGetVec Get vector from prompt 51 | func (c *AIClient) SimpleGetVec(prompt string) ([]float32, error) { 52 | req := openai.EmbeddingRequest{ 53 | Input: []string{prompt}, 54 | Model: openai.AdaEmbeddingV2, 55 | } 56 | rsp, err := c.openai.CreateEmbeddings(context.TODO(), req) 57 | if err != nil { 58 | return nil, err 59 | } 60 | if len(rsp.Data) > 0 { 61 | return rsp.Data[0].Embedding, nil 62 | } 63 | return nil, errors.New("embedding length is 0") 64 | } 65 | 66 | func (c *AIClient) Chat(prompt string) (string, error) { 67 | messageStore := InitChatMessages() 68 | messageStore.AddForUser(prompt) 69 | defer messageStore.Clear() 70 | rsp, err := c.openai.CreateChatCompletion(context.TODO(), openai.ChatCompletionRequest{ 71 | Model: openai.GPT3Dot5Turbo, 72 | Messages: messageStore.ToMessage(), 73 | }) 74 | if err != nil { 75 | return "", err 76 | } 77 | 78 | messageStore.AddForAssistant(rsp.Choices[0].Message.Content) 79 | return messageStore.GetLast(), nil 80 | } 81 | -------------------------------------------------------------------------------- /ai/message.go: -------------------------------------------------------------------------------- 1 | package ai 2 | 3 | import ( 4 | openai "github.com/sashabaranov/go-openai" 5 | ) 6 | 7 | type ( 8 | ChatMessages []*ChatMessage 9 | ChatMessage struct { 10 | Msg openai.ChatCompletionMessage 11 | } 12 | ) 13 | 14 | const ( 15 | RoleUser = "user" 16 | RoleAssistant = "assistant" 17 | RoleSystem = "system" 18 | ) 19 | 20 | func (cm *ChatMessages) Clear() { 21 | *cm = make([]*ChatMessage, 0) 22 | } 23 | 24 | func InitChatMessages() ChatMessages { 25 | cm := make(ChatMessages, 0) 26 | // cm.AddForSystem("You are a helpful K8S assistant. Use the provided text to form your answer. Keep your answer within 10 sentences. Accurate, helpful, concise and to the point") 27 | // cm.AddForSystem("You are a helpful K8S assistant. I will provide you with text, including the question title or keywords, question description, and reference answer. If I provide a reference answer, please try to use it as much as possible.Try to answer in Chinese as much as possible, Keep your answer within 10 sentences. Accurate, useful, concise and to the point") 28 | return cm 29 | } 30 | 31 | func (cm *ChatMessages) AddFor(msg string, role string) { 32 | *cm = append(*cm, &ChatMessage{ 33 | Msg: openai.ChatCompletionMessage{ 34 | Role: role, 35 | Content: msg, 36 | }, 37 | }) 38 | } 39 | 40 | func (cm *ChatMessages) AddForAssistant(msg string) { 41 | cm.AddFor(msg, RoleAssistant) 42 | } 43 | 44 | func (cm *ChatMessages) AddForSystem(msg string) { 45 | cm.AddFor(msg, RoleSystem) 46 | } 47 | 48 | func (cm *ChatMessages) AddForUser(msg string) { 49 | cm.AddFor(msg, RoleUser) 50 | } 51 | 52 | func (cm *ChatMessages) ToMessage() []openai.ChatCompletionMessage { 53 | ret := make([]openai.ChatCompletionMessage, len(*cm)) 54 | for index, c := range *cm { 55 | ret[index] = c.Msg 56 | } 57 | return ret 58 | } 59 | 60 | func (cm *ChatMessages) GetLast() string { 61 | if len(*cm) == 0 { 62 | return "nothing" 63 | } 64 | 65 | return (*cm)[len(*cm)-1].Msg.Content 66 | } 67 | -------------------------------------------------------------------------------- /cmd/import.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "crypto/md5" 5 | "encoding/hex" 6 | "encoding/json" 7 | "io" 8 | "os" 9 | 10 | "github.com/spf13/cobra" 11 | "github.com/webws/embedding-knowledge-base/ai" 12 | "github.com/webws/embedding-knowledge-base/options" 13 | "github.com/webws/embedding-knowledge-base/qdrant" 14 | "github.com/webws/go-moda/logger" 15 | 16 | pb "github.com/qdrant/go-client/qdrant" 17 | ) 18 | 19 | var FlagdataFile = "dataFile" // cmd import flag 20 | type QA struct { 21 | Questions string `json:"questions" bson:"questions"` 22 | Answers string `json:"answers" bson:"answers"` 23 | } 24 | 25 | func NewImportCmd(configFlags options.ConfigFlags) *cobra.Command { 26 | var dataFile string 27 | importCmd := &cobra.Command{ 28 | Use: "import", 29 | Short: "import data to vector database", 30 | Long: "import data to vector database", 31 | RunE: func(cmd *cobra.Command, args []string) error { 32 | qdrantClient := qdrant.NewQdrantClient(configFlags.Qdrant, configFlags.Collection, configFlags.VectorSize) 33 | defer qdrantClient.Close() 34 | aiClient, err := ai.NewAiClient(configFlags.Proxy, configFlags.ApiKey, configFlags.Proxy == options.DefaultSocksProxy) 35 | if err != nil { 36 | return err 37 | } 38 | if err = qdrantClient.CreateCollection(configFlags.Collection, configFlags.VectorSize); err != nil { 39 | return err 40 | } 41 | qas, err := convertToQAs(dataFile) 42 | if err != nil { 43 | return err 44 | } 45 | points := []*pb.PointStruct{} 46 | logger.Infow("import", "data", qas) 47 | qpsLenth := len(qas) 48 | for i, qa := range qas { 49 | embedding, err := aiClient.SimpleGetVec(qa.Questions) 50 | if err != nil { 51 | logger.Errorw("SimpleGetVec", "err", err, "question", qa.Questions, "index", i, "total", qpsLenth) 52 | return err 53 | } 54 | point := buildPoint(qa.Questions, qa.Answers, embedding) 55 | points = append(points, point) 56 | } 57 | return qdrantClient.CreatePoints(points) 58 | }, 59 | } 60 | importCmd.Flags().StringVarP(&dataFile, FlagdataFile, "p", "", "import dataFile") 61 | importCmd.MarkFlagRequired(FlagdataFile) 62 | 63 | return importCmd 64 | } 65 | 66 | func buildPoint(question string, answers string, embedding []float32) *pb.PointStruct { 67 | point := &pb.PointStruct{} 68 | // point id 69 | // uuid := fmt.Sprintf("%s%d", md5str(question), time.Now().UnixNano()) 70 | point.Id = &pb.PointId{ 71 | PointIdOptions: &pb.PointId_Uuid{ 72 | Uuid: md5str(question), 73 | }, 74 | } 75 | 76 | // vector 77 | point.Vectors = &pb.Vectors{ 78 | VectorsOptions: &pb.Vectors_Vector{ 79 | Vector: &pb.Vector{ 80 | Data: embedding, 81 | }, 82 | }, 83 | } 84 | 85 | // payload 86 | ret := make(map[string]*pb.Value) 87 | ret["question"] = &pb.Value{Kind: &pb.Value_StringValue{StringValue: question}} 88 | ret["answers"] = &pb.Value{Kind: &pb.Value_StringValue{StringValue: answers}} 89 | point.Payload = ret 90 | return point 91 | } 92 | 93 | func convertToQAs(dataFilePath string) ([]*QA, error) { 94 | f, err := os.Open(dataFilePath) 95 | if err != nil { 96 | return nil, err 97 | } 98 | // TODO :big file handle 99 | defer f.Close() 100 | content, err := io.ReadAll(f) 101 | if err != nil { 102 | return nil, err 103 | } 104 | var qas []*QA 105 | err = json.Unmarshal(content, &qas) 106 | if err != nil { 107 | return nil, err 108 | } 109 | return qas, nil 110 | } 111 | 112 | func md5str(s string) string { 113 | h := md5.New() 114 | h.Write([]byte(s)) 115 | return hex.EncodeToString(h.Sum(nil)) 116 | } 117 | -------------------------------------------------------------------------------- /cmd/import_test.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/assert" 7 | "github.com/webws/embedding-knowledge-base/options" 8 | ) 9 | 10 | func TestNewImportCmd(t *testing.T) { 11 | // first, you must start the vector database qdrant 12 | // docker run --rm -p 6334:6334 qdrant/qdrant 13 | c := options.NewConfigFlags() 14 | c.ApiKey = "you apiKey" // or env set apiKey 15 | cmd := NewImportCmd(*c) 16 | cmd.Flags().Set(FlagdataFile, "../example/data.json") 17 | err := cmd.RunE(cmd, []string{}) 18 | assert.NoError(t, err) 19 | } 20 | -------------------------------------------------------------------------------- /cmd/root.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "os" 5 | 6 | "github.com/spf13/cobra" 7 | "github.com/webws/embedding-knowledge-base/options" 8 | ) 9 | 10 | func Execute() { 11 | rootCmd := &cobra.Command{ 12 | Use: "kbai", 13 | Short: "a local knowledge base, based on chatgpt and qdrant", 14 | Long: "a local knowledge base, based on chatgpt and qdrant", 15 | // Long: `Built a local smart search knowledge repository using Golang, OpenAI Embedding, and the qdrant vector database`, 16 | RunE: func(cmd *cobra.Command, args []string) error { 17 | if err := cobra.MinimumNArgs(1)(cmd, args); err != nil { 18 | return cmd.Help() 19 | } 20 | return nil 21 | }, 22 | } 23 | configFlags := options.NewConfigFlags() 24 | configFlags.AddFlags(rootCmd.PersistentFlags()) 25 | 26 | rootCmd.AddCommand(NewImportCmd(*configFlags)) 27 | rootCmd.AddCommand(NewSearchCmd(*configFlags)) 28 | err := rootCmd.Execute() 29 | if err != nil { 30 | os.Exit(1) 31 | } 32 | } 33 | -------------------------------------------------------------------------------- /cmd/search.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "fmt" 5 | 6 | "github.com/spf13/cobra" 7 | "github.com/webws/embedding-knowledge-base/ai" 8 | "github.com/webws/embedding-knowledge-base/options" 9 | "github.com/webws/embedding-knowledge-base/qdrant" 10 | "github.com/webws/go-moda/logger" 11 | ) 12 | 13 | var flagmsg = "msg" // cmd ask flag question 14 | 15 | func NewSearchCmd(configFlags options.ConfigFlags) *cobra.Command { 16 | var msg string 17 | searchCmd := &cobra.Command{ 18 | Use: "search", 19 | Short: "ask the knowledge base example: kbai ask --msg 'First, the chicken or the egg'", 20 | Long: "ask the knowledge base example: kbai ask --msg 'First, the chicken or the egg'", 21 | RunE: func(cmd *cobra.Command, args []string) error { 22 | // logger.Debugw("search", "config", configFlags) 23 | qdrantClient := qdrant.NewQdrantClient(configFlags.Qdrant, configFlags.Collection, configFlags.VectorSize) 24 | defer qdrantClient.Close() 25 | 26 | aiClient, err := ai.NewAiClient(configFlags.Proxy, configFlags.ApiKey, configFlags.Proxy == options.DefaultSocksProxy) 27 | if err != nil { 28 | return err 29 | } 30 | vector, err := aiClient.SimpleGetVec(msg) 31 | if err != nil { 32 | return err 33 | } 34 | points, err := qdrantClient.Search(vector) 35 | if err != nil { 36 | logger.Errorw("qdrant search fail", "err", err) 37 | return err 38 | } 39 | if len(points) == 0 { 40 | fmt.Println("rearch term violation or exceeding category") 41 | return nil 42 | // return errors.New("rearch term violation or exceeding category") 43 | } 44 | // Score less than 0.8, rearch term violation or exceeding category 45 | if points[0].Score < 0.8 { 46 | fmt.Println("rearch term violation or exceeding category") 47 | return nil 48 | // return errors.New("rearch term violation or exceeding category") 49 | } 50 | 51 | answer := points[0].Payload["answers"].GetStringValue() 52 | fmt.Printf("The answer to the knowledge base:\n%s\n", answer) 53 | 54 | tmpl := "question:%s" + "reference answer: %s" 55 | finalPrompt := fmt.Sprintf(tmpl, points[0].Payload["question"].GetStringValue(), answer) 56 | 57 | chatgptAnswer, err := aiClient.Chat(finalPrompt) 58 | if err != nil { 59 | return err 60 | } 61 | fmt.Printf("Results of chatgpt answers with reference answers:\n%s\n", chatgptAnswer) 62 | chatgptAnswer, err = aiClient.Chat(msg) 63 | if err != nil { 64 | return err 65 | } 66 | fmt.Printf("only chatgpt answers:\n%s\n", chatgptAnswer) 67 | 68 | return nil 69 | }, 70 | } 71 | searchCmd.Flags().StringVar(&msg, flagmsg, "", "example: kbai search --msg 'First, the chicken or the egg'") 72 | searchCmd.MarkFlagRequired(flagmsg) 73 | return searchCmd 74 | } 75 | -------------------------------------------------------------------------------- /cmd/search_test.go: -------------------------------------------------------------------------------- 1 | package cmd 2 | 3 | import ( 4 | "testing" 5 | 6 | "github.com/stretchr/testify/assert" 7 | "github.com/webws/embedding-knowledge-base/options" 8 | ) 9 | 10 | func TestNewSearchCmd(t *testing.T) { 11 | // first, you must start the vector database qdrant 12 | // docker run --rm -p 6334:6334 qdrant/qdrant 13 | c := options.NewConfigFlags() 14 | c.ApiKey = "you apiKey" // or env set apiKey 15 | scmd := NewSearchCmd(*c) 16 | scmd.Flags().Set(flagmsg, "k8s") 17 | err := scmd.RunE(scmd, []string{}) 18 | assert.NoError(t, err) 19 | } 20 | -------------------------------------------------------------------------------- /example/data.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "questions": "什么是Kubernetes中的Deployment？", 4 | "answers": "Deployment是Kubernetes中用于管理应用程序副本的资源对象。它提供了副本的声明性定义，可以实现应用程序的部署、扩展和更新。" 5 | }, 6 | { 7 | "questions": "Kubernetes中的Service有什么作用？", 8 | "answers": "Service用于定义一组Pod的访问方式和网络策略。它为Pod提供了一个稳定的网络地址，并可以实现负载均衡、服务发现和内部通信。" 9 | }, 10 | { 11 | "questions": "如何扩展Kubernetes中的Pod副本数量？", 12 | "answers": "可以通过更新Deployment的副本数量来扩展Kubernetes中的Pod。可以通过kubectl命令或修改Deployment的YAML文件来指定所需的副本数量。" 13 | }, 14 | { 15 | "questions": "什么是Kubernetes中的命名空间（Namespace）？", 16 | "answers": "命名空间是Kubernetes中用于隔离和组织资源的一种机制。它可以将不同的资源划分到不同的命名空间中，实现资源的逻辑隔离和管理。" 17 | }, 18 | { 19 | "questions": "Kubernetes中的ConfigMap和Secret 有什么区别？", 20 | "answers": "ConfigMap用于存储应用程序的配置数据，而Secret用于存储敏感的密钥和凭证信息。它们的区别在于Secret的数据会被加密存储，并且可以安全地用于敏感信息的传递。" 21 | }, 22 | { 23 | "questions": "如何在Kubernetes中进行水平扩展（Horizontal Pod Autoscaling）？", 24 | "answers": "可以使用Horizontal Pod Autoscaler（HPA）来实现在Kubernetes中的水平扩展。HPA可以根据Pod的CPU利用率或自定义指标自动调整副本数量。" 25 | }, 26 | { 27 | "questions": "Kubernetes中的Ingress是什么？", 28 | "answers": "Ingress是Kubernetes中用于暴露HTTP和HTTPS服务的一种资源对象。它可以实现负载均衡、路由和TLS终止等功能。" 29 | }, 30 | { 31 | "questions": "如何在Kubernetes中进行滚动更新（Rolling Update）？", 32 | "answers": "可以通过更新Deployment的版本或修改Deployment的YAML文件来实现在Kubernetes中的滚动更新。滚动更新可以确保应用程序在更新过程中保持可用性。" 33 | }, 34 | { 35 | "questions": "Kubernetes 中的PersistentVolume和PersistentVolumeClaim有什么关系？", 36 | "answers": "PersistentVolume（PV）和PersistentVolumeClaim（PVC）是Kubernetes中用于持久化存储的两个关键概念。" 37 | }, 38 | { 39 | "questions": "什么是Kubernetes中的DaemonSet？", 40 | "answers": "DaemonSet是Kubernetes中一种特殊的控制器，用于在集群中的节点上运行一个Pod副本。它确保每个节点上都有一个副本在运行，用于执行特定的任务或服务" 41 | }, 42 | { 43 | "questions": "什么是Kubernetes中的Nginx Ingress？", 44 | "answers": "Kubernetes中的Nginx Ingress是一种用于暴露HTTP和HTTPS服务的Ingress控制器。它基于Nginx反向代理实现负载均衡、路由和TLS终止等功能。Nginx Ingress可以根据请求的域名、路径和其他规则将流量转发到相应的后端服务。通过使用Nginx Ingress，可以实现灵活的流量管理和应用程序的访问控制。" 45 | }, 46 | { 47 | "questions": "Kubernetes（K8s）中的网关是什么？", 48 | "answers": "在Kubernetes中，网关通常指的是Ingress（入 口）资源对象。Ingress是一种Kubernetes API对象，用于配置和管理集群中的HTTP和HTTPS流量入口。它充当了从集群外部访问集群内部服务的入口点" 49 | } 50 | ] -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/webws/embedding-knowledge-base 2 | 3 | go 1.21 4 | 5 | toolchain go1.21.0 6 | 7 | require ( 8 | github.com/qdrant/go-client v1.1.1 9 | github.com/sashabaranov/go-openai v1.8.0 10 | ) 11 | 12 | require ( 13 | github.com/spf13/pflag v1.0.5 14 | google.golang.org/grpc v1.55.0 15 | ) 16 | 17 | require ( 18 | github.com/alecthomas/colour v0.1.0 // indirect 19 | github.com/alecthomas/repr v0.0.0-20210801044451-80ca428c5142 // indirect 20 | github.com/davecgh/go-spew v1.1.1 // indirect 21 | github.com/inconshreveable/mousetrap v1.1.0 // indirect 22 | github.com/mattn/go-isatty v0.0.17 // indirect 23 | github.com/pmezard/go-difflib v1.0.0 // indirect 24 | github.com/sergi/go-diff v1.2.0 // indirect 25 | go.uber.org/atomic v1.9.0 // indirect 26 | go.uber.org/multierr v1.8.0 // indirect 27 | go.uber.org/zap v1.24.0 // indirect 28 | gopkg.in/yaml.v3 v3.0.1 // indirect 29 | ) 30 | 31 | require ( 32 | github.com/alecthomas/assert v1.0.0 33 | github.com/golang/protobuf v1.5.3 // indirect 34 | github.com/spf13/cobra v1.7.0 35 | github.com/stretchr/testify v1.8.4 36 | github.com/webws/go-moda v0.0.0-20230916221114-19e0fc168096 37 | golang.org/x/net v0.10.0 // indirect 38 | golang.org/x/sys v0.8.0 // indirect 39 | golang.org/x/text v0.9.0 // indirect 40 | google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 // indirect 41 | google.golang.org/protobuf v1.30.0 // indirect 42 | ) 43 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/alecthomas/assert v1.0.0 h1:3XmGh/PSuLzDbK3W2gUbRXwgW5lqPkuqvRgeQ30FI5o= 2 | github.com/alecthomas/assert v1.0.0/go.mod h1:va/d2JC+M7F6s+80kl/R3G7FUiW6JzUO+hPhLyJ36ZY= 3 | github.com/alecthomas/colour v0.1.0 h1:nOE9rJm6dsZ66RGWYSFrXw461ZIt9A6+nHgL7FRrDUk= 4 | github.com/alecthomas/colour v0.1.0/go.mod h1:QO9JBoKquHd+jz9nshCh40fOfO+JzsoXy8qTHF68zU0= 5 | github.com/alecthomas/repr v0.0.0-20210801044451-80ca428c5142 h1:8Uy0oSf5co/NZXje7U1z8Mpep++QJOldL2hs/sBQf48= 6 | github.com/alecthomas/repr v0.0.0-20210801044451-80ca428c5142/go.mod h1:2kn6fqh/zIyPLmm3ugklbEi5hg5wS435eygvNfaDQL8= 7 | github.com/benbjohnson/clock v1.1.0 h1:Q92kusRqC1XV2MjkWETPvjJVqKetz1OzxZB7mHJLju8= 8 | github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA= 9 | github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= 10 | github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 11 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 12 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 13 | github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk= 14 | github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg= 15 | github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY= 16 | github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= 17 | github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38= 18 | github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= 19 | github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= 20 | github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= 21 | github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo= 22 | github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ= 23 | github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI= 24 | github.com/mattn/go-isatty v0.0.17 h1:BTarxUcIeDqL27Mc+vyvdWYSL28zpIhv3RoTdsLMPng= 25 | github.com/mattn/go-isatty v0.0.17/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= 26 | github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I= 27 | github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= 28 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 29 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 30 | github.com/qdrant/go-client v1.1.1 h1:K9ViznsQhuWaIrdNQGJF6SsMq3Rgw0tkt8XgmDkiNW0= 31 | github.com/qdrant/go-client v1.1.1/go.mod h1:680gkxNAsVtre0Z8hAQmtPzJtz1xFAyCu2TUxULtnoE= 32 | github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= 33 | github.com/sashabaranov/go-openai v1.8.0 h1:IZrNK/gGqxtp0j19F4NLGbmfoOkyDpM3oC9i/tv9bBM= 34 | github.com/sashabaranov/go-openai v1.8.0/go.mod h1:lj5b/K+zjTSFxVLijLSTDZuP7adOgerWeFyZLUhAKRg= 35 | github.com/sergi/go-diff v1.2.0 h1:XU+rvMAioB0UC3q1MFrIQy4Vo5/4VsRDQQXHsEya6xQ= 36 | github.com/sergi/go-diff v1.2.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM= 37 | github.com/spf13/cobra v1.7.0 h1:hyqWnYt1ZQShIddO5kBpj3vu05/++x6tJ6dg8EC572I= 38 | github.com/spf13/cobra v1.7.0/go.mod h1:uLxZILRyS/50WlhOIKD7W6V5bgeIt+4sICxh6uRMrb0= 39 | github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= 40 | github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= 41 | github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= 42 | github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= 43 | github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= 44 | github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= 45 | github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= 46 | github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= 47 | github.com/webws/go-moda v0.0.0-20230916221114-19e0fc168096 h1:iWPqkZHSKIafpgzRFfeocTzg3gghxzLY7HCzmn/7qhk= 48 | github.com/webws/go-moda v0.0.0-20230916221114-19e0fc168096/go.mod h1:+DKpHKOWS0lJKo4QgIa6F4vnGe/I2f9j/N0zLQNVDYM= 49 | go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc= 50 | go.uber.org/atomic v1.9.0 h1:ECmE8Bn/WFTYwEW/bpKD3M8VtR/zQVbavAoalC1PYyE= 51 | go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc= 52 | go.uber.org/goleak v1.1.11 h1:wy28qYRKZgnJTxGxvye5/wgWr1EKjmUDGYox5mGlRlI= 53 | go.uber.org/goleak v1.1.11/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ= 54 | go.uber.org/multierr v1.8.0 h1:dg6GjLku4EH+249NNmoIciG9N/jURbDG+pFlTkhzIC8= 55 | go.uber.org/multierr v1.8.0/go.mod h1:7EAYxJLBy9rStEaz58O2t4Uvip6FSURkq8/ppBp95ak= 56 | go.uber.org/zap v1.24.0 h1:FiJd5l1UOLj0wCgbSE0rwwXHzEdAZS6hiiSnxJN/D60= 57 | go.uber.org/zap v1.24.0/go.mod h1:2kMP+WWQ8aoFoedH3T2sq6iJ2yDWpHbP0f6MQbS9Gkg= 58 | golang.org/x/net v0.10.0 h1:X2//UzNDwYmtCLn7To6G58Wr6f5ahEAQgKNzv9Y951M= 59 | golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg= 60 | golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 61 | golang.org/x/sys v0.8.0 h1:EBmGv8NaZBZTWvrbjNoL6HVt+IVy3QDQpJs7VRIw3tU= 62 | golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= 63 | golang.org/x/text v0.9.0 h1:2sjJmO8cDvYveuX97RDLsxlyUxLl+GHoLxBiRdHllBE= 64 | golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8= 65 | golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= 66 | google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1 h1:KpwkzHKEF7B9Zxg18WzOa7djJ+Ha5DzthMyZYQfEn2A= 67 | google.golang.org/genproto v0.0.0-20230410155749-daa745c078e1/go.mod h1:nKE/iIaLqn2bQwXBg8f1g2Ylh6r5MN5CmZvuzZCgsCU= 68 | google.golang.org/grpc v1.55.0 h1:3Oj82/tFSCeUrRTg/5E/7d/W5A1tj6Ky1ABAuZuv5ag= 69 | google.golang.org/grpc v1.55.0/go.mod h1:iYEXKGkEBhg1PjZQvoYEVPTDkHo1/bjTnfwTeGONTY8= 70 | google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= 71 | google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= 72 | google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng= 73 | google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I= 74 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= 75 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 76 | gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 77 | gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 78 | gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= 79 | gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 80 | gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 81 | gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= 82 | gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 83 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "github.com/webws/embedding-knowledge-base/cmd" 5 | ) 6 | 7 | func main() { 8 | cmd.Execute() 9 | } 10 | -------------------------------------------------------------------------------- /options/config_flags.go: -------------------------------------------------------------------------------- 1 | package options 2 | 3 | import ( 4 | "fmt" 5 | "os" 6 | 7 | "github.com/spf13/pflag" 8 | ) 9 | 10 | const ( 11 | DefaultQdrantAddr = "127.0.0.1:6334" 12 | DefaultSocksProxy = "socks5://127.0.0.1:1080" 13 | ) 14 | 15 | var ( 16 | flagQdrant = "qdrant" // qdrant address 17 | // flagDataFile = "data_file" // cmd import flag 18 | // msgFlag = "msg" // cmd ask flag question 19 | flagApiKey = "apiKey" // open apikey 20 | flagProxy = "proxy" // openai http proxy 21 | flagCollection = "collection" 22 | flagVectorSize = "vectorSize" 23 | ) 24 | 25 | type ConfigFlags struct { 26 | Qdrant string 27 | ApiKey string 28 | Proxy string 29 | Collection string 30 | VectorSize uint64 31 | } 32 | 33 | // NewConfigFlags 34 | func NewConfigFlags() *ConfigFlags { 35 | return &ConfigFlags{ 36 | Qdrant: DefaultQdrantAddr, 37 | ApiKey: "", 38 | Proxy: DefaultSocksProxy, 39 | Collection: "kubernetes", 40 | VectorSize: 1536, 41 | } 42 | } 43 | 44 | // AddFlags binds client configuration flags to a given flagset 45 | func (cf *ConfigFlags) AddFlags(fs *pflag.FlagSet) { 46 | fs.StringVar(&cf.Qdrant, flagQdrant, DefaultQdrantAddr, fmt.Sprintf("qdrant address default: %s", DefaultQdrantAddr)) 47 | fs.StringVar(&cf.Proxy, flagProxy, DefaultSocksProxy, fmt.Sprintf("http client proxy default:%s ", DefaultSocksProxy)) 48 | apiKey := os.Getenv(flagApiKey) 49 | fs.StringVar(&cf.ApiKey, flagApiKey, apiKey, "openai apikey:default from env "+flagApiKey) 50 | fs.StringVar(&cf.Collection, flagCollection, cf.Collection, "qdrant collection name default: "+cf.Collection) 51 | fs.Uint64Var(&cf.VectorSize, flagVectorSize, cf.VectorSize, "qdrant vector size default: "+fmt.Sprintf("%d", cf.VectorSize)) 52 | } 53 | -------------------------------------------------------------------------------- /qdrant/client.go: -------------------------------------------------------------------------------- 1 | package qdrant 2 | 3 | import ( 4 | "context" 5 | "strings" 6 | 7 | pb "github.com/qdrant/go-client/qdrant" 8 | "github.com/webws/go-moda/logger" 9 | "google.golang.org/grpc" 10 | "google.golang.org/grpc/credentials/insecure" 11 | ) 12 | 13 | const ( 14 | ErrNotFound = "Not found" 15 | ErrAlreadyExists = "already exists" 16 | ) 17 | 18 | type QdrantClient struct { 19 | grpcConn *grpc.ClientConn 20 | collection string 21 | size uint64 22 | } 23 | 24 | func (qc *QdrantClient) Close() { 25 | qc.grpcConn.Close() 26 | } 27 | 28 | func (qc *QdrantClient) Collection() pb.CollectionsClient { 29 | return pb.NewCollectionsClient(qc.grpcConn) 30 | } 31 | 32 | func NewQdrantClient(qdrantAddr, collection string, size uint64) *QdrantClient { 33 | conn, err := grpc.Dial(qdrantAddr, 34 | grpc.WithTransportCredentials(insecure.NewCredentials())) 35 | if err != nil { 36 | logger.Fatalw("did not connect", "err", err) 37 | } 38 | return &QdrantClient{grpcConn: conn, collection: collection, size: size} 39 | } 40 | 41 | func toPayload(payload map[string]string) map[string]*pb.Value { 42 | ret := make(map[string]*pb.Value) 43 | for k, v := range payload { 44 | ret[k] = &pb.Value{Kind: &pb.Value_StringValue{StringValue: v}} 45 | } 46 | return ret 47 | } 48 | 49 | func (qc *QdrantClient) DeleteCollection(name string) error { 50 | cc := pb.NewCollectionsClient(qc.grpcConn) 51 | _, err := cc.Delete(context.TODO(), &pb.DeleteCollection{ 52 | CollectionName: name, 53 | }) 54 | return err 55 | } 56 | 57 | func (qc *QdrantClient) CreateCollection(name string, size uint64) error { 58 | cc := pb.NewCollectionsClient(qc.grpcConn) 59 | req := &pb.CreateCollection{ 60 | CollectionName: name, 61 | VectorsConfig: &pb.VectorsConfig{ 62 | Config: &pb.VectorsConfig_Params{ 63 | Params: &pb.VectorParams{ 64 | Size: size, 65 | Distance: pb.Distance_Cosine, 66 | }, 67 | }, 68 | }, 69 | } 70 | _, err := cc.Create(context.Background(), req) 71 | if err != nil && strings.Contains(err.Error(), ErrAlreadyExists) { 72 | return nil 73 | } 74 | if err != nil { 75 | logger.Errorw("CreateCollection", "err", err) 76 | return err 77 | } 78 | return nil 79 | } 80 | 81 | func (qc *QdrantClient) CreatePoints(points []*pb.PointStruct) error { 82 | pc := pb.NewPointsClient(qc.grpcConn) 83 | 84 | wait := true 85 | pointsReq := pb.UpsertPoints{ 86 | CollectionName: qc.collection, 87 | Points: points, 88 | Wait: &wait, 89 | } 90 | 91 | _, err := pc.Upsert(context.TODO(), &pointsReq) 92 | if err != nil { 93 | logger.Errorw("CreatePoints fail", "err", err) 94 | return err 95 | } 96 | return nil 97 | } 98 | 99 | func (qc *QdrantClient) CreatePoint(uuid string, collection string, vector []float32, payload map[string]string) error { 100 | point := &pb.PointStruct{} 101 | point.Id = &pb.PointId{ 102 | PointIdOptions: &pb.PointId_Uuid{ 103 | Uuid: uuid, 104 | }, 105 | } 106 | point.Vectors = &pb.Vectors{ 107 | VectorsOptions: &pb.Vectors_Vector{ 108 | Vector: &pb.Vector{ 109 | Data: vector, 110 | }, 111 | }, 112 | } 113 | point.Payload = toPayload(payload) 114 | 115 | pc := pb.NewPointsClient(qc.grpcConn) 116 | 117 | wait := true 118 | points := pb.UpsertPoints{ 119 | CollectionName: collection, 120 | Points: []*pb.PointStruct{point}, 121 | Wait: &wait, 122 | } 123 | 124 | _, err := pc.Upsert(context.TODO(), &points) 125 | if err != nil { 126 | return err 127 | } 128 | return nil 129 | } 130 | 131 | func (qc *QdrantClient) Search(vector []float32) ([]*pb.ScoredPoint, error) { 132 | sc := pb.NewPointsClient(qc.grpcConn) 133 | rsp, err := sc.Search(context.Background(), &pb.SearchPoints{ 134 | CollectionName: qc.collection, 135 | Vector: vector, 136 | Limit: 3, // only take three 137 | WithPayload: &pb.WithPayloadSelector{ 138 | SelectorOptions: &pb.WithPayloadSelector_Include{ 139 | Include: &pb.PayloadIncludeSelector{ 140 | Fields: []string{"question", "answers"}, 141 | }, 142 | }, 143 | }, 144 | }) 145 | if err != nil && strings.Contains(err.Error(), ErrNotFound) { 146 | if err := qc.CreateCollection(qc.collection, qc.size); err != nil { 147 | logger.Errorw("Search CreateCollection fail", "err", err) 148 | return nil, err 149 | } 150 | return qc.Search(vector) 151 | } 152 | 153 | if err != nil { 154 | return nil, err 155 | } 156 | 157 | return rsp.Result, nil 158 | } 159 | -------------------------------------------------------------------------------- /readme.md: -------------------------------------------------------------------------------- 1 | Translations: [English](readme.md) | [简体中文](README_zh.md) 2 | ## kbai 3 | a local knowledge base, based on chatgpt and qdrant 4 | ### usage 5 | local knowledge base based on chatgpt and qdrant, supporting data import and Q&A 6 | ``` 7 | ❯ kbai -h 8 | a local knowledge base, based on chatgpt and qdrant 9 | 10 | usage: 11 | kbai [flags] 12 | kbai [command] 13 | 14 | available commands: 15 | completion generate the autocompletion script for the specified shell 16 | help help about any command 17 | import import data to vector database 18 | search ask the knowledge base example: kbai ask --msg 'first, the chicken or the egg' 19 | 20 | flags: 21 | --apikey string openai apikey:default from env apikey 22 | --collection string qdrant collection name default: kubernetes (default "kubernetes") 23 | -h, --help help for kbai 24 | --proxy string http client proxy default:socks5://127.0.0.1:1080 (default "socks5://127.0.0.1:1080") 25 | --qdrant string qdrant address default: 127.0.0.1:6334 (default "127.0.0.1:6334") 26 | --vectorsize uint qdrant vector size default: 1536 (default 1536) 27 | 28 | use "kbai [command] --help" for more information about a command. 29 | ``` 30 | ## install 31 | go build install rename and move to the $PATH 32 | ``` 33 | sudo go build -o kbai github.com/webws/embedding-knowledge-base && sudo mv ./kbai /usr/local/bin 34 | ``` 35 | Or use golang to execute source code 36 | ``` 37 | git clone https://github.com/webws/embedding-knowledge-base.git && cd ./embedding-knowledge-base 38 | 39 | ``` 40 | or download binary file from release……(todo) 41 | ## use example 42 | first, you must start the vector database qdrant 43 | ``` 44 | docker run --rm -p 6334:6334 qdrant/qdrant 45 | ``` 46 | 47 | set openai apikey 48 | ``` 49 | export apikey=xxx 50 | ``` 51 | ### import 52 | import the prepared JSON data into qdrant 53 | ``` 54 | kbai import --datafile ./example/data.json 55 | ``` 56 | example format of file data.json 57 | ``` 58 | [ 59 | { 60 | "questions": "question", 61 | "answers": "answer" 62 | }, 63 | ] 64 | ``` 65 | note:the imported sample data content is: k8s related knowledge, and it is in Chinese 66 | 67 | ### search 68 | 69 | ask knowledge related to the knowledge base 70 | ``` 71 | kbai search --msg "What is a gateway" 72 | 73 | The answer to the knowledge base: 74 | 在Kubernetes中，网关通常指的是Ingress（入 口）资源对象。Ingress是一种Kubernetes API对象，用于配置和管理集群中的HTTP和HTTPS流量入口。它充当了从集群外部访问集群内部服务的入口点 75 | 76 | Results of chatgpt answers with reference answers: 77 | Ingress acts as a gateway in Kubernetes, allowing external traffic to access the internal services within the cluster. It provides a configuration layer for managing HTTP and HTTPS traffic routing rules, load balancing, and SSL termination. Ingress resources define the rules for how the traffic should be directed to the appropriate backend services based on the requested host or URL path. Ingress controllers, such as Nginx or Traefik, are responsible for implementing these rules and routing the traffic accordingly. 78 | 79 | only chatgpt answers: 80 | A gateway is a device or software that acts as an entry point or interface between two different networks, systems, or protocols. It serves as a connecting link that allows data to flow between different networks, translating between different protocols or formats if necessary. Gateways are commonly used in computer networks to connect local area networks (LANs) to wide area networks (WANs) or to bridge different types of networks. They can also be used in telecommunications to connect different types of networks, such as telephone networks and internet networks. 81 | ``` 82 | ps: 83 | >It can be seen that directly asking chatgpt may result in an answer unrelated to k8s. Combining with the local knowledge base of k8s can bias the answer towards the theme set in the dataset 84 | 85 | 86 | ask questions unrelated to the knowledge base,There won't be any results 87 | 88 | ``` 89 | kbai search --msg "Can apples be eaten without washing?" 90 | rearch term violation or exceeding category 91 | ``` 92 | ### reference project 93 | * [https://github.com/spf13/cobra](https://github.com/spf13/cobra) 94 | * [https://github.com/kubernetes/kubernetes](https://github.com/kubernetes/kubernetes) 95 | * [https://github.com/gohugoio/hugo](https://github.com/gohugoio/hugo) 96 | * [https://github.com/qdrant/qdrant](https://github.com/qdrant/qdrant) 97 | -------------------------------------------------------------------------------- /util/proxy.go: -------------------------------------------------------------------------------- 1 | package util 2 | 3 | import ( 4 | "context" 5 | "net/http" 6 | "net/url" 7 | "time" 8 | ) 9 | 10 | func ProxyCheck(client *http.Client, proxy, urlTarget string, timeout int64) bool { 11 | ctx, cncl := context.WithTimeout(context.Background(), time.Second*time.Duration(timeout)) 12 | defer cncl() 13 | if client == nil { 14 | client = &http.Client{} 15 | } 16 | if proxy != "" { 17 | proxyURL, _ := url.Parse(proxy) 18 | client.Transport = &http.Transport{ 19 | Proxy: http.ProxyURL(proxyURL), 20 | } 21 | } 22 | req, _ := http.NewRequestWithContext(ctx, "GET", urlTarget, nil) 23 | resp, err := client.Do(req) 24 | 25 | if resp != nil { 26 | defer resp.Body.Close() 27 | } 28 | if err != nil { 29 | return false 30 | } 31 | 32 | if resp.StatusCode != http.StatusOK { 33 | return false 34 | } 35 | return true 36 | } 37 | -------------------------------------------------------------------------------- /util/proxy_test.go: -------------------------------------------------------------------------------- 1 | package util 2 | 3 | import ( 4 | "fmt" 5 | "testing" 6 | 7 | "github.com/stretchr/testify/assert" 8 | ) 9 | 10 | func TestProxyTest(t *testing.T) { 11 | b := ProxyCheck(nil, "socks5://127.0.0.1:1080", "https://google.com", 10) 12 | fmt.Println(b) 13 | assert.True(t, b) 14 | // assert.True(t, ProxyCheck(nil, "socks5://127.0.0.1:1080", "https://google.com", 5)) 15 | } 16 | --------------------------------------------------------------------------------