├── .gitignore ├── README.md ├── bin └── .gitignore ├── pkg └── .gitignore └── src ├── _rebuild_vendor.py ├── cmds ├── service_rpc_lb.go └── service_rpc_proxy.go ├── doc └── rpc_architecture.jpg ├── glide.lock ├── glide.yaml ├── proxy ├── backend_conn_lb.go ├── backend_conn_proxy.go ├── backend_conn_proxy_test.go ├── backend_service_lb.go ├── backend_service_proxy.go ├── backend_service_proxy_test.go ├── buffered_framed_transport.go ├── config.go ├── consts.go ├── endpoint.go ├── endpoint_test.go ├── heartbeat.go ├── heartbeat_test.go ├── memory_alloc.go ├── memory_alloc_test.go ├── memory_buffer.go ├── memory_buffer_test.go ├── request.go ├── request_ordered_map.go ├── request_ordered_map_test.go ├── request_test.go ├── router_proxy.go ├── rpc_command_line.go ├── server_general_rpc.go ├── server_lb.go ├── server_proxy.go ├── session_nonblock.go ├── session_nonblock_test.go ├── session_proxy.go ├── session_proxy_test.go ├── stats.go ├── thrift_exception.go ├── thrift_exception_test.go ├── topology.go ├── topology_test.go ├── utils.go ├── utils_color.go ├── utils_config_checker.go ├── utils_log.go └── zk.go ├── rebuild_vendor.sh ├── scripts ├── README.md ├── RpcThrift.Services.thrift ├── build_lb.sh ├── build_proxy.sh ├── config.local.ini ├── config.test.ini ├── control_lb.sh ├── control_proxy.sh ├── control_proxy_profile.sh ├── deploy_proxy.sh ├── idl_update.sh ├── idl_update_python.sh ├── lb-config.online.ini ├── proxy-config.online.ini ├── rpc_proxy_online.service ├── rpc_proxy_test.service ├── scp_lb.sh └── scp_proxy.sh ├── start_env.sh └── sync.sh /.gitignore: -------------------------------------------------------------------------------- 1 | /log/ 2 | /lib/ 3 | .idea 4 | *.pyc 5 | /gen-py 6 | vendor 7 | *.log 8 | /src/service_* 9 | /src/config.ini -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # RPC Proxy 2 | ## 开发环境 3 | * 工作目录: 约定使用 `~/goprojects/` 4 | * mkdir -p rpc_proxy/src/git.chunyu.me/infra/ 5 | * cd `rpc_proxy/src/git.chunyu.me/infra/` 6 | * git clone git@git.chunyu.me:infra/rpc_proxy.git 7 | * IDE(`Pycharm + Go plugin`) 8 | * https://github.com/go-lang-plugin-org/go-lang-idea-plugin 9 | * 在最新的Pycharm中安装: 10 | * https://plugins.jetbrains.com/plugin/5047 (版本: 0.9.748) 11 | * 配置: 12 | * 在Pycharm的 Preferences 中选择: 13 | * Languages & Frameworks 14 | * 选择go 15 | * 设置Go SDK(设置GORoot) 16 | * 选择Go Libraries, 选择Global Libraries(忽略), 使用Project libraries) 17 | * 输入`source start_env.sh`脚本中输出的地址，例如: 18 | * /Users/feiwang/goprojects/rpc_proxy 19 | 20 | # 基于Thrift的RPC Proxy 21 | 相关的子项目: 22 | * [RPC Proxy Java SDK](https://git.chunyu.me/infra/rpc_proxy_java) 23 | * [RPC Proxy Python SDK](https://git.chunyu.me/infra/rpc_proxy_python) 24 | 25 | ## RPC的分层 26 | Rpc Proxy分为4层，从前端往后端依次标记为L1, L2, L3, L4 27 | ### L1层(应用层) 28 | * 最前端的RPC Client, 主要由thrift中间语言生成代码 29 | * 上面思路: 我们修改了transport&protocol两个模块，通过简单的修改得到了python版本的zerothrift 30 | * Java, Go等也能很方面地实现自己的RPC Client 31 | * 其他语言只要支持Thrift和ZeroMq, 都可以方便地实现自己的Client 32 | 33 | ```python 34 | # Transport层是所有的RPC服务都共用的(除非在同一个请求内部实现了并发) 35 | _ = get_base_protocol(settings.RPC_LOCAL_PROXY) 36 | protocol = get_service_protocol("typo") 37 | 38 | from cy_typo.services.TypoService import Client 39 | _typo_client = Client(protocol) 40 | 41 | 42 | # 函数调用 43 | rpc_result = _typo_client.correct_typo(content) 44 | content = rpc_result.fixed 45 | 46 | # 或者使用Pool 47 | pool = ConnectionPool(10, "127.0.0.1:5550") 48 | with pool.base_protocol() as base_protocol: 49 | protocol = get_service_protocol("typo", base_protocol) 50 | client = Client(protocol) 51 | ``` 52 | * 相应的Python RPC Client&Server的实现: https://github.com/wfxiang08/zerothrift 53 | 54 | ### L2层(Proxy层) 55 | * 由于Python的进程功能太弱，不变在内部实现连接池等；如果让它们直接直连后端的Server, 则整个逻辑(connections)会非常乱，端口管理也会非常麻烦 56 | * 服务的发现和负载均衡放在Client端赖做也会极大地增加了Client端的开发难度和负担 57 | * Local Proxy层以产品为单位，负责发现所有的服务，并和L1层交互，让L1层不用关心服务部署的路径 58 | * Proxy层也有简单的负载均衡的处理 59 | 60 | ```bash 61 | # Go中的deamon似乎不太容易实现，借助: nohup &可以实现类似的效果(Codis也如此) 62 | # 默认的proxy(绑定: 127.0.0.0:5550) 63 | nohup rpc_proxy -c config.ini -L proxy.log >/dev/null 2>&1 & 64 | # 在测试服务器上部署，给大家开发测试使用的proxy 65 | nohup rpc_proxy -c config_test.ini -L proxy_test.log >/dev/null 2>&1 & 66 | ``` 67 | 68 | ### L3层(负载均衡) 69 | * Java/Go等天然支持多线程等，基本上不需要负载均衡, 因此这一层主要面向python 70 | * 负责管理后端的Server, 自动进行负载均衡，以及处理后端服务的关闭，重启等异常情况 71 | * 负责服务的注册 72 | * 如果服务的正常关闭，会提前通知L2层，让L2层控制流量不再进入L3层的当前节点 73 | * 如果服务异常关闭，则5s左右, L2层就会感知，并且下线对应的节点 74 | 75 | ```bash 76 | ## 配置文件 77 | config.ini 78 | zk=rd1:2181,rd2:2181,mysql2:2181 79 | 80 | # 线上的product一律以: online开头 81 | # 而测试的product禁止使用 online的服务 82 | product=test 83 | verbose=0 84 | 85 | zk_session_timeout=30 86 | 87 | ## Load Balance 88 | service=typo 89 | front_host= 90 | front_port=5555 91 | back_address=127.0.0.1:5556 92 | 93 | # 使用网络的IP, 如果没有指定front_host, 则使用使用当前机器的内网的Ip来注册 94 | ip_prefix=10. 95 | 96 | ## Server 97 | worker_pool_size=2 98 | 99 | ## Client/Proxy 100 | proxy_address=127.0.0.1:5550 101 | 102 | 103 | # Go中的deamon似乎不太容易实现，借助: nohup &可以实现类似的效果(Codis也如此) 104 | nohup rpc_lb -c config.ini -L lb.log >/dev/null 2>&1 & 105 | ``` 106 | 107 | ### L3层也可以直接就是Rpc服务层，例如: 108 | * 对于go, java直接在这一层提供服务 109 | 110 | ```go 111 | func main() { 112 | 113 | proxy.RpcMain(BINARY_NAME, SERVICE_DESC, 114 | // 默认的ThriftServer的配置checker 115 | proxy.ConfigCheckThriftService, 116 | 117 | // 可以根据配置config来创建processor 118 | func(config *utils.Config) proxy.Server { 119 | // 可以根据配置config来创建processor 120 | alg.InitGeoFix(FILE_GPS_2_SOGOU) 121 | processor := gs.NewGeoLocationServiceProcessor(&alg.GeoLocationService{}) 122 | return proxy.NewThriftRpcServer(config, processor) 123 | }) 124 | } 125 | ``` 126 | 127 | ### L4层 128 | * 对于Python, rpc_proxy python框架即可 129 | * 通过thrift idl生成接口, Processor等 130 | * 实现Processor的接口 131 | * 合理地配置参数 132 | * 例如: worker_pool_size：如果是Django DB App则worker_pool_size=1(不支持异步数据库，多并发没有意义，即便支持异步数据库，Django的数据库逻辑也不支持高并发); 但是如果不使用DB, 那么Django还是可以支持多并发的 133 | * 注意在iptables中开启相关的端口 134 | * thrift只支持utf8格式的字符串，因此在使用过程中注意 135 | * 字符串如果是unicode, 一方面len可能和utf8的 len不一样，另一方面，数据在序列化时可能报编码错误 136 | 137 | ```python 138 | class TypoProcessor(object): 139 | def correct_typo(self, query): 140 | # print "Query: ", query 141 | result = TypoResult() 142 | new_query, mappings = typo_manager.correct_typo(query) 143 | result.fixed = new_query 144 | result.fixes = [] 145 | for old, new in mappings: 146 | result.fixes.append(FixedTerm(old, new)) 147 | return result 148 | 149 | config_path = "config.ini" 150 | config = parse_config(config_path) 151 | endpoint = config["back_address"] 152 | service = config["service"] 153 | worker_pool_size = int(config["worker_pool_size"]) 154 | 155 | processor = Processor(TypoProcessor()) 156 | s = RpcWorker(processor, endpoint, pool_size=worker_pool_size, service=service) 157 | s.connect(endpoint) 158 | s.run() 159 | ``` 160 | ## 系统架构 161 | ![architecture](doc/rpc_architecture.jpg) 162 | ## 运维部署 163 | * 编译: 164 | * cd ~/goprojects/rpc_proxy/src/git.chunyu.me/infra/rpc_proxy 165 | * 运行编译脚本: 166 | * `bash scripts/build_lb.sh` 167 | * go build cmds/rpc_lb.go 168 | * 编译脚本和直接编译相比，将当前代码的版本，编译时间等打包到最终的文件中 169 | * `bash scripts/build_proxy.sh` 170 | * 运维 171 | * scp rpc_* node:/usr/local/rpc_proxy/bin/ 172 | * sudo cp rpc_* /usr/local/rpc_proxy/bin/ 173 | * 然后control_proxy.sh脚本和control_lb.sh脚本也需要部署在合适的地方 174 | * 注意 scripts目录下的脚本 -------------------------------------------------------------------------------- /bin/.gitignore: -------------------------------------------------------------------------------- 1 | * 2 | !.gitignore -------------------------------------------------------------------------------- /pkg/.gitignore: -------------------------------------------------------------------------------- 1 | * 2 | !.gitignore -------------------------------------------------------------------------------- /src/_rebuild_vendor.py: -------------------------------------------------------------------------------- 1 | # -*- coding:utf-8 -*- 2 | import os 3 | import glob 4 | import subprocess 5 | import sys 6 | 7 | GOPATH = "" 8 | PATH = "" 9 | GO_EXE_PATH="go" 10 | def rebuild_vendor_packages(root_dir): 11 | for dir in os.listdir(root_dir): 12 | full_dir = os.path.join(root_dir, dir) 13 | if os.path.isdir(full_dir): 14 | # 如果是碰到测试，example目录，则直接跳过 15 | if full_dir.find("example") != -1 or full_dir.find("test") != -1: 16 | continue 17 | 18 | files = glob.glob("%s/*.go" % full_dir) 19 | if len(files) > 0: 20 | print "Install golang package: %s" % full_dir 21 | subprocess.Popen([GO_EXE_PATH, "install", full_dir], env={'GOPATH': GOPATH, 'PATH':PATH}).wait() 22 | else: 23 | # 如果碰到空目录，则继续遍历 24 | rebuild_vendor_packages(full_dir) 25 | 26 | 27 | if __name__ == "__main__": 28 | if len(sys.argv) < 2: 29 | print "No go path is sepecified" 30 | exit(0) 31 | GO_EXE_PATH = sys.argv[1] 32 | 33 | GOPATH = os.environ.get("GOPATH", "") 34 | if not GOPATH: 35 | print "GOPATH is not set" 36 | exit(0) 37 | PATH = os.environ.get("PATH", "") 38 | rebuild_vendor_packages("vendor") 39 | -------------------------------------------------------------------------------- /src/cmds/service_rpc_lb.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "proxy" 5 | ) 6 | 7 | const ( 8 | BINARY_NAME = "rpc_lb" 9 | SERVICE_DESC = "Thrift RPC Load Balance Service" 10 | ) 11 | 12 | var ( 13 | buildDate string 14 | gitVersion string 15 | ) 16 | 17 | func main() { 18 | proxy.RpcMain(BINARY_NAME, SERVICE_DESC, 19 | // 验证LB的配置 20 | proxy.ConfigCheckRpcLB, 21 | // 根据配置创建一个Server 22 | func(config *proxy.ServiceConfig) proxy.Server { 23 | return proxy.NewThriftLoadBalanceServer(config) 24 | }, buildDate, gitVersion) 25 | 26 | } 27 | -------------------------------------------------------------------------------- /src/cmds/service_rpc_proxy.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "proxy" 5 | ) 6 | 7 | const ( 8 | BINARY_NAME = "rpc_proxy" 9 | SERVICE_DESC = "Thrift RPC Local Proxy v0.1" 10 | ) 11 | 12 | var ( 13 | gitVersion string 14 | buildDate string 15 | ) 16 | 17 | func main() { 18 | proxy.RpcMainProxy(BINARY_NAME, SERVICE_DESC, 19 | // 验证LB的配置 20 | proxy.ConfigCheckRpcProxy, 21 | // 根据配置创建一个Server 22 | func(config *proxy.ProxyConfig) proxy.Server { 23 | // 正式的服务 24 | return proxy.NewProxyServer(config) 25 | }, buildDate, gitVersion) 26 | 27 | } 28 | -------------------------------------------------------------------------------- /src/doc/rpc_architecture.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/wfxiang08/rpc_proxy/2f31f548a765f65599f292259356f8ab8fcc399a/src/doc/rpc_architecture.jpg -------------------------------------------------------------------------------- /src/glide.lock: -------------------------------------------------------------------------------- 1 | hash: 547eec29f366ff270f84c572d263c3548eb9adc795f4a75c3d2e6c63732c1dda 2 | updated: 2017-10-09T11:31:39.505567117+08:00 3 | imports: 4 | - name: github.com/c4pt0r/cfg 5 | version: 429e6985f0b07bceaa33b57ee612fbca4797a9a6 6 | - name: github.com/coreos/etcd 7 | version: 1408b337b67d4cb8355b3666b7eb8aaefbbfc167 8 | subpackages: 9 | - error 10 | - name: github.com/coreos/go-etcd 11 | version: 003851be7bb0694fe3cc457a49529a19388ee7cf 12 | subpackages: 13 | - etcd 14 | - name: github.com/fatih/color 15 | version: 62e9147c64a1ed519147b62a56a14e83e2be02c1 16 | - name: github.com/mattn/go-colorable 17 | version: 5411d3eea5978e6cdc258b30de592b60df6aba96 18 | repo: https://github.com/mattn/go-colorable 19 | - name: github.com/mattn/go-isatty 20 | version: 57fdcb988a5c543893cc61bce354a6e24ab70022 21 | repo: https://github.com/mattn/go-isatty 22 | - name: github.com/ngaut/log 23 | version: d2af3a61f64d093457fb23b25d20f4ce3cd551ce 24 | - name: github.com/ngaut/logging 25 | version: f98f5f4cd523647678af6f1726c5b5a14c9193d0 26 | - name: github.com/ngaut/pools 27 | version: 6352e005618615ffaf1cb1c6622b8e91435751fe 28 | - name: github.com/ngaut/sync2 29 | version: 7a24ed77b2efb460c1468b7dc917821c66e80e55 30 | - name: github.com/ugorji/go 31 | version: ded73eae5db7e7a0ef6f55aace87a2873c5d2b74 32 | subpackages: 33 | - codec 34 | - name: github.com/wfxiang08/cyutils 35 | version: dfb8bc2b5402cb534e3f53cbd9fc3897820f7b07 36 | subpackages: 37 | - utils 38 | - utils/atomic2 39 | - utils/errors 40 | - utils/rolling_log 41 | - utils/trace 42 | - name: github.com/wfxiang08/go-zookeeper 43 | version: 184aa014b12119755d0eea6ffad2756ca6f73614 44 | subpackages: 45 | - zk 46 | - name: github.com/wfxiang08/go_thrift 47 | version: 96149a376920afc9acf82e69177d06b7d66f88d6 48 | subpackages: 49 | - thrift 50 | - name: github.com/wfxiang08/thrift_rpc_base 51 | version: d0a731c06159d6573328569480469d668d3eef2d 52 | subpackages: 53 | - rpc_utils 54 | - rpcthrift/services 55 | - zkhelper 56 | - name: golang.org/x/net 57 | version: b400c2eff1badec7022a8c8f5bea058b6315eed7 58 | subpackages: 59 | - context 60 | - name: golang.org/x/sys 61 | version: e24f485414aeafb646f6fca458b0bf869c0880a1 62 | repo: https://go.googlesource.com/sys 63 | subpackages: 64 | - unix 65 | testImports: 66 | - name: github.com/davecgh/go-spew 67 | version: 04cdfd42973bb9c8589fd6a731800cf222fde1a9 68 | subpackages: 69 | - spew 70 | - name: github.com/pmezard/go-difflib 71 | version: d8ed2627bdf02c080bf22230dbb337003b7aba2d 72 | subpackages: 73 | - difflib 74 | - name: github.com/stretchr/testify 75 | version: f6abca593680b2315d2075e0f5e2a9751e3f431a 76 | subpackages: 77 | - assert 78 | -------------------------------------------------------------------------------- /src/glide.yaml: -------------------------------------------------------------------------------- 1 | package: . 2 | import: 3 | - package: github.com/wfxiang08/cyutils 4 | subpackages: 5 | - utils 6 | - utils/atomic2 7 | - utils/errors 8 | - utils/rolling_log 9 | - package: github.com/c4pt0r/cfg 10 | - package: github.com/fatih/color 11 | - package: github.com/ngaut/logging 12 | - package: github.com/wfxiang08/go_thrift 13 | - package: github.com/wfxiang08/thrift_rpc_base 14 | - package: github.com/wfxiang08/go-zookeeper 15 | subpackages: 16 | - zk 17 | -------------------------------------------------------------------------------- /src/proxy/backend_conn_lb.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "fmt" 7 | "time" 8 | 9 | "github.com/wfxiang08/cyutils/utils/atomic2" 10 | "github.com/wfxiang08/cyutils/utils/errors" 11 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 12 | "github.com/wfxiang08/go_thrift/thrift" 13 | ) 14 | 15 | type BackendConnLBStateChanged interface { 16 | StateChanged(conn *BackendConnLB) 17 | } 18 | 19 | type BackendConnLB struct { 20 | transport thrift.TTransport 21 | address string 22 | serviceName string 23 | input chan *Request // 输入的请求, 有: 1024个Buffer 24 | 25 | seqNumRequestMap *RequestMap 26 | currentSeqId int32 // 范围: 1 ~ 100000 27 | Index int 28 | delegate BackendConnLBStateChanged 29 | verbose bool 30 | IsConnActive atomic2.Bool // 是否处于Active状态呢 31 | 32 | hbLastTime atomic2.Int64 33 | hbTicker *time.Ticker 34 | } 35 | 36 | // 37 | // LB(Load Balancer)启动一个Server和后端的服务(Backend)通信， 后端服务负责主动连接LB； 38 | // 1. LB负责定期地和Backend进行ping/pang; 39 | // 如果LB发现Backend长时间没有反应，或者出错，则端口和Backend之间的连接 40 | // 2. Backend根据config.ini主动注册LB, 按照一定的策略重连 41 | // 42 | // BackendConnLB 43 | // 1. 为Backend主动向LB注册之后，和LB之间建立的一条Connection 44 | // 2. 底层的conn在LB BackendService中accepts时就已经建立好，因此BackendConnLB 45 | // 就是建立在transport之上的控制逻辑 46 | // 47 | func NewBackendConnLB(transport thrift.TTransport, serviceName string, 48 | address string, delegate BackendConnLBStateChanged, 49 | verbose bool) *BackendConnLB { 50 | requestMap, _ := NewRequestMap(4096) 51 | bc := &BackendConnLB{ 52 | transport: transport, 53 | address: address, 54 | serviceName: serviceName, 55 | input: make(chan *Request, 1024), 56 | 57 | seqNumRequestMap: requestMap, 58 | currentSeqId: BACKEND_CONN_MIN_SEQ_ID, 59 | 60 | Index: INVALID_ARRAY_INDEX, // 用于记录: BackendConnLB在数组中的位置 61 | 62 | delegate: delegate, 63 | verbose: verbose, 64 | } 65 | bc.IsConnActive.Set(true) 66 | go bc.Run() 67 | return bc 68 | } 69 | 70 | func (bc *BackendConnLB) MarkConnActiveFalse() { 71 | // 从Active切换到非正常状态 72 | if bc.IsConnActive.CompareAndSwap(true, false) && bc.delegate != nil { 73 | // bc.IsConnActive.Set(false) 74 | bc.delegate.StateChanged(bc) // 通知其他人状态出现问题 75 | } else { 76 | bc.IsConnActive.Set(false) 77 | } 78 | } 79 | 80 | // run之间 transport刚刚建立，因此服务的可靠性比较高 81 | func (bc *BackendConnLB) Run() { 82 | log.Printf(Green("[%s]Add New BackendConnLB: %s"), bc.serviceName, bc.address) 83 | 84 | // 1. 首先BackendConn将当前 input中的数据写到后端服务中 85 | err := bc.loopWriter() 86 | 87 | // 2. 从Active切换到非正常状态, 同时不再从backend_service_lb接受新的任务 88 | // 可能出现异常，也可能正常退出(反正不干活了) 89 | bc.MarkConnActiveFalse() 90 | 91 | log.Printf(Red("[%s]Remove Faild BackendConnLB: %s"), bc.serviceName, bc.address) 92 | 93 | if err == nil { 94 | // bc.input被关闭了，应该就没有 Request 了 95 | } else { 96 | // 如果出现err, 则将bc.input中现有的数据都flush回去（直接报错) 97 | for i := len(bc.input); i != 0; i-- { 98 | r := <-bc.input 99 | bc.setResponse(r, nil, err) 100 | } 101 | } 102 | 103 | } 104 | 105 | func (bc *BackendConnLB) Address() string { 106 | return bc.address 107 | } 108 | 109 | // 110 | // Request为将要发送到后端进程请求，包括lb层的心跳，或来自前端的正常请求 111 | // 112 | func (bc *BackendConnLB) PushBack(r *Request) { 113 | // 关键路径必须有Log, 高频路径的Log需要受verbose状态的控制 114 | if bc.IsConnActive.Get() { 115 | // log.Printf("Push Request to backend: %s", r.Request.Name) 116 | r.Service = bc.serviceName 117 | r.Wait.Add(1) 118 | bc.input <- r 119 | 120 | } else { 121 | r.Response.Err = errors.New(fmt.Sprintf("[%s] Request Assigned to inactive BackendConnLB", bc.serviceName)) 122 | log.Warn(Magenta("Push Request To Inactive Backend")) 123 | } 124 | 125 | } 126 | 127 | // 128 | // 数据: LB ---> backend services 129 | // 130 | // 如果input关闭，且loopWriter正常处理完毕之后，返回nil 131 | // 其他情况返回error 132 | // 133 | func (bc *BackendConnLB) loopWriter() error { 134 | // 正常情况下, ok总是为True; 除非bc.input的发送者主动关闭了channel, 表示再也没有新的Task过来了 135 | // 参考: https://tour.golang.org/concurrency/4 136 | // 如果input没有关闭，则会block 137 | c := NewTBufferedFramedTransport(bc.transport, 100*time.Microsecond, 20) 138 | 139 | // bc.MarkConnActiveOK() // 准备接受数据 140 | // BackendConnLB 在构造之初就有打开的transport, 并且Active默认为OK 141 | 142 | defer func() { 143 | bc.hbTicker.Stop() 144 | bc.hbTicker = nil 145 | }() 146 | 147 | // 从"RPC Backend" RPC Worker 中读取结果, 然后返回给proxy 148 | bc.loopReader(c) // 异步 149 | 150 | // 建立连接之后，就启动HB 151 | bc.hbTicker = time.NewTicker(time.Second) 152 | bc.hbLastTime.Set(time.Now().Unix()) 153 | 154 | var r *Request 155 | var ok bool 156 | 157 | for true { 158 | // 等待输入的Event, 或者 heartbeatTimeout 159 | select { 160 | case <-bc.hbTicker.C: 161 | // 两种情况下，心跳会超时 162 | // 1. 对方挂了 163 | // 2. 自己快要挂了，然后就不再发送心跳；没有了心跳，就会超时 164 | if time.Now().Unix()-bc.hbLastTime.Get() > HB_TIMEOUT { 165 | // 强制关闭c 166 | c.Close() 167 | return errors.New("Worker HB timeout") 168 | } else { 169 | if bc.IsConnActive.Get() { 170 | // 定时添加Ping的任务 171 | r := NewPingRequest() 172 | bc.PushBack(r) 173 | 174 | // 同时检测当前的异常请求 175 | expired := microseconds() - REQUEST_EXPIRED_TIME_MICRO // 以microsecond为单位 176 | bc.seqNumRequestMap.RemoveExpired(expired) 177 | } 178 | } 179 | 180 | case r, ok = <-bc.input: 181 | if !ok { 182 | return nil 183 | } else { 184 | // 如果暂时没有数据输入，则p策略可能就有问题了 185 | // 只有写入数据，才有可能产生flush; 如果是最后一个数据必须自己flush, 否则就可能无限期等待 186 | // 187 | if r.Request.TypeId == MESSAGE_TYPE_HEART_BEAT { 188 | // 过期的HB信号，直接放弃 189 | if time.Now().Unix()-r.Start > 4 { 190 | log.Warnf(Red("Expired HB Signals")) 191 | } 192 | } else if r.Request.TypeId == MESSAGE_TYPE_STOP_CONFIRM { 193 | // 强制写一个新的Response到Worker，不关心是否写成功；也不关心反馈结果 194 | c.Write(r.Request.Data) 195 | err := c.FlushBuffer(true) 196 | 197 | if err != nil { 198 | log.ErrorErrorf(err, "Stop confirm Error") 199 | } 200 | } else { 201 | // 先不做优化 202 | var flush = true // len(bc.input) == 0 203 | 204 | // 1. 替换新的SeqId(currentSeqId只在当前线程中使用, 不需要同步) 205 | r.ReplaceSeqId(bc.currentSeqId) 206 | bc.IncreaseCurrentSeqId() 207 | 208 | // 2. 主动控制Buffer的flush 209 | // 先记录SeqId <--> Request, 再发送请求 210 | // 否则: 请求从后端返回，记录还没有完成，就容易导致Request丢失 211 | bc.seqNumRequestMap.Add(r.Response.SeqId, r) 212 | c.Write(r.Request.Data) 213 | err := c.FlushBuffer(flush) 214 | 215 | if err != nil { 216 | bc.seqNumRequestMap.Pop(r.Response.SeqId) 217 | log.ErrorErrorf(err, "FlushBuffer Error: %v\n", err) 218 | 219 | // 进入不可用状态(不可用状态下，通过自我心跳进入可用状态) 220 | return bc.setResponse(r, nil, err) 221 | } 222 | } 223 | } 224 | } 225 | 226 | } 227 | return nil 228 | } 229 | 230 | // 231 | // 从"RPC Backend" RPC Worker 中读取结果, ReadFrame读取的是一个thrift message 232 | // 存在两种情况: 233 | // 1. 正常读取thrift message, 然后从frame解码得到seqId, 然后得到request, 结束请求 234 | // 2. 读取错误 235 | // 将现有的requests全部flush回去 236 | // 237 | func (bc *BackendConnLB) loopReader(c *TBufferedFramedTransport) { 238 | go func() { 239 | defer c.Close() 240 | 241 | for true { 242 | // 坚信: EOF只有在连接被关闭的情况下才会发生，其他情况下, Read等操作被会被block住 243 | // EOF有两种情况: 244 | // 1. 连接正常关闭，最后数据等完整读取 --> io.EOF 245 | // 2. 连接异常关闭，数据不完整 --> io.ErrUnexpectedEOF 246 | // 247 | // rpc_server ---> backend_conn 248 | frame, err := c.ReadFrame() // 有可能被堵住 249 | 250 | if err != nil { 251 | // 如果出错，则Flush所有的请求 252 | err1, ok := err.(thrift.TTransportException) 253 | if !ok || err1.TypeId() != thrift.END_OF_FILE { 254 | log.ErrorErrorf(err, Red("ReadFrame From rpc_server with Error: %v\n"), err) 255 | } 256 | // TODO: 可能需要细化，有些错误出现之后，可能需要给其他的请求一些机会 257 | bc.flushRequests(err) 258 | break 259 | } else { 260 | bc.setResponse(nil, frame, err) 261 | } 262 | } 263 | }() 264 | } 265 | 266 | // 处理所有的等待中的请求 267 | func (bc *BackendConnLB) flushRequests(err error) { 268 | // 告诉BackendService, 不再接受新的请求 269 | bc.MarkConnActiveFalse() 270 | 271 | seqRequest := bc.seqNumRequestMap.Purge() 272 | 273 | for _, request := range seqRequest { 274 | if request.Request.TypeId == MESSAGE_TYPE_HEART_BEAT { 275 | // 心跳出错了，则直接直接跳过 276 | } else { 277 | log.Printf(Red("Handle Failed Request: %s.%s"), request.Service, request.Request.Name) 278 | request.Response.Err = err 279 | request.Wait.Done() 280 | } 281 | } 282 | 283 | // 关闭输入 284 | close(bc.input) 285 | 286 | } 287 | 288 | // 配对 Request, resp, err 289 | // PARAM: resp []byte 为一帧完整的thrift数据包 290 | func (bc *BackendConnLB) setResponse(r *Request, data []byte, err error) error { 291 | // log.Printf("#setResponse: data: %v", data) 292 | // 表示出现错误了 293 | if data == nil { 294 | log.Printf("No Data From Server, error: %v\n", err) 295 | r.Response.Err = err 296 | } else { 297 | // 从resp中读取基本的信息 298 | typeId, method, seqId, err := DecodeThriftTypIdSeqId(data) 299 | 300 | // 解码错误，直接报错 301 | if err != nil { 302 | log.ErrorErrorf(err, "Decode SeqId Error: %v", err) 303 | return err 304 | } 305 | 306 | if typeId == MESSAGE_TYPE_STOP { 307 | // 不再接受新的输入 308 | // 直接来自后端的服务(不遵循: Request/Reply模型) 309 | // 或者回传给后端一个确认停止消息 310 | bc.MarkConnActiveFalse() 311 | 312 | // 临时再发送一个请求; 有些语言没有异步，不太方便设置timeout, 那么通过stop confirm告知这是最后一个请求 313 | r := NewStopConfirmRequest() 314 | r.Service = bc.serviceName 315 | r.Wait.Add(1) 316 | bc.input <- r 317 | 318 | return nil 319 | } 320 | 321 | // 找到对应的Request 322 | 323 | req := bc.seqNumRequestMap.Pop(seqId) 324 | 325 | // 如果是心跳，则OK 326 | if typeId == MESSAGE_TYPE_HEART_BEAT { 327 | bc.hbLastTime.Set(time.Now().Unix()) 328 | return nil 329 | } 330 | 331 | if req == nil { 332 | log.Errorf("#setResponse not found, seqId: %d", seqId) 333 | return nil 334 | } else { 335 | 336 | if req.Response.SeqId != seqId { 337 | log.Errorf("Data From Server, SeqId not match, Ex: %d, Ret: %d", req.Request.SeqId, seqId) 338 | } 339 | r = req 340 | r.Response.TypeId = typeId 341 | if req.Request.Name != method { 342 | data = nil 343 | err = req.NewInvalidResponseError(method, "conn_lb") 344 | } 345 | } 346 | } 347 | 348 | r.Response.Data, r.Response.Err = data, err 349 | // 还原SeqId 350 | if data != nil { 351 | r.RestoreSeqId() 352 | } 353 | 354 | 355 | r.Wait.Done() 356 | return err 357 | } 358 | 359 | func (bc *BackendConnLB) IncreaseCurrentSeqId() { 360 | // 备案(只有loopWriter操作，不加锁) 361 | bc.currentSeqId++ 362 | if bc.currentSeqId > BACKEND_CONN_MAX_SEQ_ID { 363 | bc.currentSeqId = BACKEND_CONN_MIN_SEQ_ID 364 | } 365 | } 366 | -------------------------------------------------------------------------------- /src/proxy/backend_conn_proxy.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "fmt" 7 | "strings" 8 | "sync" 9 | "time" 10 | 11 | "github.com/wfxiang08/cyutils/utils/atomic2" 12 | "github.com/wfxiang08/cyutils/utils/errors" 13 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 14 | "github.com/wfxiang08/go_thrift/thrift" 15 | "github.com/wfxiang08/thrift_rpc_base/rpc_utils" 16 | ) 17 | 18 | type BackendConnStateChanged interface { 19 | StateChanged(conn *BackendConn) 20 | } 21 | 22 | var ( 23 | backendConnIndex int32 = 0 24 | backendConnIndexMutex *sync.Mutex = &sync.Mutex{} 25 | ) 26 | 27 | type BackendConn struct { 28 | addr string 29 | service string 30 | input chan *Request // 输入的请求, 有: 1024个Buffer 31 | 32 | // seqNum2Request 读写基本上差不多 33 | seqNumRequestMap *RequestMap 34 | currentSeqId int32 // 范围: 1 ~ 100000 35 | minSeqId int32 36 | maxSeqId int32 37 | 38 | Index int 39 | delegate *BackService 40 | 41 | IsMarkOffline atomic2.Bool // 是否标记下线 42 | IsConnActive atomic2.Bool // 是否处于Active状态呢 43 | verbose bool 44 | 45 | hbLastTime atomic2.Int64 46 | hbTicker *time.Ticker 47 | } 48 | 49 | func NewBackendConn(addr string, delegate *BackService, service string, verbose bool) *BackendConn { 50 | requestMap, _ := NewRequestMap(4096) 51 | 52 | var minSeqId int32 53 | backendConnIndexMutex.Lock() 54 | log.Infof("Create Backend Conn with index: %d", backendConnIndex) 55 | 56 | // 主要是区分不同的: backendConnIndex 57 | minSeqId = backendConnIndex 58 | 59 | backendConnIndex += 1 60 | if backendConnIndex > 100 { 61 | backendConnIndex = 0 62 | } 63 | backendConnIndexMutex.Unlock() 64 | 65 | // BACKEND_CONN_MAX_SEQ_ID = 1000000 66 | minSeqId = minSeqId * BACKEND_CONN_MAX_SEQ_ID 67 | 68 | bc := &BackendConn{ 69 | addr: addr, 70 | service: service, 71 | input: make(chan *Request, 1024), 72 | seqNumRequestMap: requestMap, 73 | 74 | currentSeqId: minSeqId, 75 | minSeqId: minSeqId, 76 | maxSeqId: minSeqId + BACKEND_CONN_MAX_SEQ_ID - 1, 77 | Index: INVALID_ARRAY_INDEX, 78 | 79 | delegate: delegate, 80 | verbose: verbose, 81 | } 82 | go bc.Run() 83 | return bc 84 | } 85 | 86 | // 87 | // MarkOffline发生场景: 88 | // 1. 后端服务即将下线，预先通知 89 | // 2. 后端服务已经挂了，zk检测到 90 | // 91 | // BackendConn 在这里暂时理解关闭conn, 而是从 backend_service_proxy中下线当前的conn, 92 | // 然后conn的关闭根据 心跳&Conn的读写异常来判断; 因此 IsConnActive = false 情况下，心跳不能关闭 93 | // 94 | func (bc *BackendConn) MarkOffline() { 95 | if !bc.IsMarkOffline.Get() { 96 | log.Printf(Magenta("[%s]BackendConn: %s MarkOffline"), bc.service, bc.addr) 97 | bc.IsMarkOffline.Set(true) 98 | 99 | // 不再接受(来自backend_service_proxy的)新的输入 100 | bc.MarkConnActiveFalse() 101 | 102 | close(bc.input) 103 | } 104 | } 105 | 106 | func (bc *BackendConn) MarkConnActiveFalse() { 107 | if bc.IsConnActive.Get() { 108 | // 从Active切换到非正常状态 109 | bc.IsConnActive.Set(false) 110 | 111 | if bc.delegate != nil { 112 | bc.delegate.StateChanged(bc) // 通知其他人状态出现问题 113 | } 114 | 115 | // 日志延后, 控制信息尽快生效 116 | log.Printf(Red("[%s]MarkConnActiveFalse: %s, %p"), bc.service, bc.addr, bc.delegate) 117 | } 118 | } 119 | 120 | // 121 | // 从Active切换到非正常状态 122 | // 123 | func (bc *BackendConn) MarkConnActiveOK() { 124 | // if !bc.IsConnActive { 125 | // log.Printf(Green("MarkConnActiveOK: %s, %p"), bc.addr, bc.delegate) 126 | // } 127 | 128 | bc.IsConnActive.Set(true) 129 | if bc.delegate != nil { 130 | bc.delegate.StateChanged(bc) // 通知其他人状态出现问题 131 | } 132 | 133 | } 134 | 135 | func (bc *BackendConn) Addr() string { 136 | return bc.addr 137 | } 138 | 139 | // 140 | // 目前有两类请求: 141 | // 1. ping request 142 | // 2. 正常的请求 143 | func (bc *BackendConn) PushBack(r *Request) { 144 | if bc.IsConnActive.Get() && !bc.IsMarkOffline.Get() { 145 | // 1. 处于Active状态，并且没有标记下线, 则将 Request 添加到 input 中 146 | r.Wait.Add(1) 147 | bc.input <- r 148 | } else { 149 | // 2. 直接报错（返回) 150 | r.Response.Err = errors.New(fmt.Sprintf("[%s] Request Assigned to inactive BackendConn", bc.service)) 151 | log.Warn(Magenta("Push Request To Inactive Backend")) 152 | } 153 | } 154 | 155 | // 156 | // 确保Socket成功连接到后端服务器 157 | // 158 | func (bc *BackendConn) ensureConn() (transport thrift.TTransport, err error) { 159 | // 1. 创建连接(只要IP没有问题， err一般就是空) 160 | timeout := time.Second * REQUEST_EXPIRED_TIME_SECONDS // 15s 161 | if strings.Contains(bc.addr, ":") { 162 | transport, err = thrift.NewTSocketTimeout(bc.addr, timeout) 163 | } else { 164 | transport, err = rpc_utils.NewTUnixDomainTimeout(bc.addr, timeout) 165 | } 166 | log.Printf(Cyan("[%s]Create Socket To: %s"), bc.service, bc.addr) 167 | 168 | if err != nil { 169 | log.ErrorErrorf(err, "[%s]Create Socket Failed: %v, Addr: %s", err, bc.service, bc.addr) 170 | // 连接不上，失败 171 | return nil, err 172 | } 173 | 174 | // 2. 只要服务存在，一般不会出现err 175 | sleepInterval := 1 176 | err = transport.Open() 177 | for err != nil && !bc.IsMarkOffline.Get() { 178 | log.ErrorErrorf(err, "[%s]Socket Open Failed: %v, Addr: %s", bc.service, err, bc.addr) 179 | 180 | // Sleep: 1, 2, 4这几个间隔 181 | time.Sleep(time.Duration(sleepInterval) * time.Second) 182 | 183 | if sleepInterval < 4 { 184 | sleepInterval *= 2 185 | } 186 | err = transport.Open() 187 | } 188 | return transport, err 189 | } 190 | 191 | // 192 | // 不断建立到后端的逻辑，负责: BackendConn#input到redis的数据的输入和返回 193 | // 194 | func (bc *BackendConn) Run() { 195 | 196 | for k := 0; !bc.IsMarkOffline.Get(); k++ { 197 | 198 | // 1. 首先BackendConn将当前 input中的数据写到后端服务中 199 | transport, err := bc.ensureConn() 200 | if err != nil { 201 | log.ErrorErrorf(err, "[%s]BackendConn#ensureConn error: %v", bc.service, err) 202 | return 203 | } 204 | 205 | connOver := &sync.WaitGroup{} 206 | c := NewTBufferedFramedTransport(transport, 100*time.Microsecond, 20) 207 | 208 | bc.MarkConnActiveOK() // 准备接受数据 209 | connOver.Add(1) 210 | bc.loopReader(c, connOver) // 异步(读取来自后端服务器的返回数据) 211 | // 2. 将 bc.input 中的请求写入 后端的Rpc Server 212 | err = bc.loopWriter(c) // 同步 213 | 214 | // 3. 停止接受Request 215 | bc.MarkConnActiveFalse() 216 | 217 | // 等待Conn正式关闭 218 | connOver.Wait() 219 | 220 | // 4. 将bc.input中剩余的 Request直接出错处理 221 | if err == nil { 222 | log.Printf(Red("[%s]BackendConn#loopWriter normal Exit..."), bc.service) 223 | break 224 | } else { 225 | // 对于尚未处理的Request, 直接报错 226 | for i := len(bc.input); i != 0; i-- { 227 | r := <-bc.input 228 | bc.setResponse(r, nil, err) 229 | } 230 | } 231 | } 232 | } 233 | 234 | // 235 | // 将 bc.input 中的Request写入后端的服务器 236 | // 237 | func (bc *BackendConn) loopWriter(c *TBufferedFramedTransport) error { 238 | 239 | defer func() { 240 | // 关闭心跳的Ticker 241 | bc.hbTicker.Stop() 242 | bc.hbTicker = nil 243 | }() 244 | 245 | var r *Request 246 | var ok bool 247 | 248 | // 准备HB Ticker 249 | bc.hbTicker = time.NewTicker(time.Second) 250 | bc.hbLastTime.Set(time.Now().Unix()) 251 | 252 | for true { 253 | // 等待输入的Event, 或者 heartbeatTimeout 254 | select { 255 | case <-bc.hbTicker.C: 256 | if time.Now().Unix()-bc.hbLastTime.Get() > HB_TIMEOUT { 257 | return errors.New(fmt.Sprintf("[%s]HB timeout", bc.service)) 258 | } else { 259 | // 定时添加Ping的任务; 如果标记下线，则不在心跳 260 | if !bc.IsMarkOffline.Get() { 261 | // 发送心跳信息 262 | r := NewPingRequest() 263 | bc.PushBack(r) 264 | 265 | // 同时检测当前的异常请求 266 | expired := microseconds() - REQUEST_EXPIRED_TIME_MICRO // 以microsecond为单位 267 | // microseconds() - request.Start > REQUEST_EXPIRED_TIME_MICRO 268 | // 超时: microseconds() - REQUEST_EXPIRED_TIME_MICRO > request.Start 269 | bc.seqNumRequestMap.RemoveExpired(expired) 270 | 271 | } 272 | } 273 | 274 | case r, ok = <-bc.input: 275 | if !ok { 276 | return nil 277 | } else { 278 | // 279 | // 如果暂时没有数据输入，则p策略可能就有问题了 280 | // 只有写入数据，才有可能产生flush; 如果是最后一个数据必须自己flush, 否则就可能无限期等待 281 | // 282 | if r.Request.TypeId == MESSAGE_TYPE_HEART_BEAT { 283 | // 过期的HB信号，直接放弃 284 | if time.Now().Unix()-r.Start > 4 { 285 | log.Printf(Magenta("Expired HB Signal")) 286 | } 287 | } 288 | 289 | // 请求正常转发给后端的Rpc Server 290 | var flush = len(bc.input) == 0 291 | 292 | // 1. 替换新的SeqId 293 | r.ReplaceSeqId(bc.currentSeqId) 294 | bc.IncreaseCurrentSeqId() 295 | 296 | // 2. 主动控制Buffer的flush 297 | // 先记录SeqId <--> Request, 再发送请求 298 | // 否则: 请求从后端返回，记录还没有完成，就容易导致Request丢失 299 | bc.seqNumRequestMap.Add(r.Response.SeqId, r) 300 | 301 | // 2. 主动控制Buffer的flush 302 | c.Write(r.Request.Data) 303 | err := c.FlushBuffer(flush) 304 | 305 | if err == nil { 306 | log.Debugf("--> SeqId: %d vs. %d To Backend", r.Request.SeqId, r.Response.SeqId) 307 | 308 | } else { 309 | bc.seqNumRequestMap.Pop(r.Response.SeqId) // 如果写错了，在删除 310 | // 进入不可用状态(不可用状态下，通过自我心跳进入可用状态) 311 | return bc.setResponse(r, nil, err) 312 | } 313 | } 314 | } 315 | } 316 | 317 | return nil 318 | } 319 | 320 | // 321 | // Client <---> Proxy[BackendConn] <---> RPC Server[包含LB] 322 | // BackConn <====> RPC Server 323 | // loopReader从RPC Server读取数据，然后根据返回的结果来设置: Client的Request的状态 324 | // 325 | // 1. bc.flushRequest 326 | // 2. bc.setResponse 327 | // 328 | func (bc *BackendConn) loopReader(c *TBufferedFramedTransport, connOver *sync.WaitGroup) { 329 | go func() { 330 | defer connOver.Done() 331 | defer c.Close() 332 | 333 | lastTime := time.Now().Unix() 334 | // Active状态，或者最近5s有数据返回 335 | // 设计理由：服务在线，则请求正常发送； 336 | // 服务下线后，则期待后端服务的数据继续返回(最多等待5s) 337 | for bc.IsConnActive.Get() || (time.Now().Unix()-lastTime < 5) { 338 | // 读取来自后端服务的数据，通过 setResponse 转交给 前端 339 | // client <---> proxy <-----> backend_conn <---> rpc_server 340 | // ReadFrame需要有一个度? 如果碰到EOF该如何处理呢? 341 | 342 | // io.EOF在两种情况下会出现 343 | // 344 | resp, err := c.ReadFrame() 345 | lastTime = time.Now().Unix() 346 | if err != nil { 347 | err1, ok := err.(thrift.TTransportException) 348 | if !ok || err1.TypeId() != thrift.END_OF_FILE { 349 | log.ErrorErrorf(err, Red("[%s]ReadFrame From Server with Error: %v"), bc.service, err) 350 | } 351 | bc.flushRequests(err) 352 | break 353 | } else { 354 | 355 | bc.setResponse(nil, resp, err) 356 | } 357 | } 358 | 359 | bc.flushRequests(errors.New("BackendConn Timeout")) 360 | }() 361 | } 362 | 363 | // 处理所有的等待中的请求 364 | func (bc *BackendConn) flushRequests(err error) { 365 | // 告诉BackendService, 不再接受新的请求 366 | bc.MarkConnActiveFalse() 367 | 368 | seqRequest := bc.seqNumRequestMap.Purge() 369 | for _, request := range seqRequest { 370 | request.Response.Err = err 371 | request.Wait.Done() 372 | log.Debugf("FlushRequests, SeqId: %d", request.Response.SeqId) 373 | } 374 | 375 | } 376 | 377 | // 配对 Request, resp, err 378 | // PARAM: resp []byte 为一帧完整的thrift数据包 379 | func (bc *BackendConn) setResponse(r *Request, data []byte, err error) error { 380 | // 表示出现错误了 381 | if data == nil { 382 | log.Debugf("[%s] SeqId: %d, No Data From Server, error: %v", r.Service, r.Response.SeqId, err) 383 | r.Response.Err = err 384 | } else { 385 | // 从resp中读取基本的信息 386 | typeId, method, seqId, err := DecodeThriftTypIdSeqId(data) 387 | // if err != nil { 388 | // log.Debugf("SeqId: %d, Decoded, error: %v", seqId, err) 389 | // } else { 390 | // log.Debugf("SeqId: %d, Decoded", seqId) 391 | // } 392 | // 解码错误，直接报错 393 | if err != nil { 394 | log.Debugf("SeqId: %d, Decoded, error: %v", seqId, err) 395 | return err 396 | } 397 | 398 | // 找到对应的Request 399 | req := bc.seqNumRequestMap.Pop(seqId) 400 | 401 | // 如果是心跳，则OK 402 | if typeId == MESSAGE_TYPE_HEART_BEAT { 403 | bc.hbLastTime.Set(time.Now().Unix()) 404 | // if req != nil { 405 | // log.Printf("HB RT: %.3fms", float64(microseconds()-req.Start)*0.001) 406 | // } 407 | return nil 408 | } 409 | 410 | if req == nil { 411 | // return errors.New("Invalid Response") 412 | // 由于是异步返回，因此回来找不到也正常 413 | log.Errorf("#setResponse not found, seqId: %d", seqId) 414 | return nil 415 | } else { 416 | 417 | if req.Response.SeqId != seqId { 418 | log.Errorf("Data From Server, SeqId not match, Ex: %d, Ret: %d", req.Request.SeqId, seqId) 419 | } 420 | r = req 421 | r.Response.TypeId = typeId 422 | if req.Request.Name != method { 423 | data = nil 424 | err = req.NewInvalidResponseError(method, "conn_proxy") 425 | } 426 | } 427 | } 428 | 429 | // 正常返回数据，或者报错 430 | r.Response.Data, r.Response.Err = data, err 431 | 432 | // 还原SeqId 433 | if data != nil { 434 | r.RestoreSeqId() 435 | } 436 | 437 | // 设置几个控制用的channel 438 | r.Wait.Done() 439 | 440 | return err 441 | } 442 | 443 | func (bc *BackendConn) IncreaseCurrentSeqId() { 444 | // 备案(只有loopWriter操作，不加锁) 445 | bc.currentSeqId++ 446 | if bc.currentSeqId > bc.maxSeqId { 447 | bc.currentSeqId = bc.minSeqId 448 | } 449 | } 450 | -------------------------------------------------------------------------------- /src/proxy/backend_conn_proxy_test.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "fmt" 7 | "github.com/stretchr/testify/assert" 8 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 9 | "github.com/wfxiang08/go_thrift/thrift" 10 | "testing" 11 | "time" 12 | ) 13 | 14 | // 15 | // go test proxy -v -run "TestBackend" 16 | // 17 | func TestBackend(t *testing.T) { 18 | 19 | // 作为一个Server 20 | transport, err := thrift.NewTServerSocket("127.0.0.1:0") 21 | assert.NoError(t, err) 22 | err = transport.Open() // 打开Transport 23 | assert.NoError(t, err) 24 | 25 | defer transport.Close() 26 | 27 | err = transport.Listen() // 开始监听 28 | assert.NoError(t, err) 29 | 30 | addr := transport.Addr().String() 31 | 32 | fmt.Println("Addr: ", addr) 33 | 34 | var requestNum int32 = 10 35 | 36 | requests := make([]*Request, 0, requestNum) 37 | 38 | var i int32 39 | for i = 0; i < requestNum; i++ { 40 | buf := make([]byte, 100, 100) 41 | l := fakeData("Hello", thrift.CALL, i+1, buf[0:0]) 42 | buf = buf[0:l] 43 | 44 | req, _ := NewRequest(buf, false) 45 | 46 | req.Wait.Add(1) // 因为go routine可能还没有执行，代码就跑到最后面进行校验了 47 | 48 | assert.Equal(t, i+1, req.Request.SeqId, "Request SeqId是否靠谱") 49 | 50 | requests = append(requests, req) 51 | } 52 | 53 | go func() { 54 | // 客户端代码 55 | bc := NewBackendConn(addr, nil, "test", true) 56 | bc.currentSeqId = 10 57 | 58 | // 准备发送数据 59 | var i int32 60 | for i = 0; i < requestNum; i++ { 61 | fmt.Println("Sending Request to Backend Conn", i) 62 | bc.PushBack(requests[i]) 63 | 64 | requests[i].Wait.Done() 65 | } 66 | 67 | // 需要等待数据返回? 68 | time.Sleep(time.Second * 2) 69 | 70 | }() 71 | 72 | go func() { 73 | // 服务器端代码 74 | tran, err := transport.Accept() 75 | if err != nil { 76 | log.ErrorErrorf(err, "Error: %v\n", err) 77 | } 78 | assert.NoError(t, err) 79 | 80 | bt := NewTBufferedFramedTransport(tran, time.Microsecond*100, 2) 81 | 82 | // 在当前的这个t上读写数据 83 | var i int32 84 | for i = 0; i < requestNum; i++ { 85 | request, err := bt.ReadFrame() 86 | assert.NoError(t, err) 87 | 88 | req, _ := NewRequest(request, false) 89 | assert.Equal(t, req.Request.SeqId, i+10) 90 | fmt.Printf("Server Got Request, and SeqNum OK, Id: %d, Frame Size: %d\n", i, len(request)) 91 | 92 | // 回写数据 93 | bt.Write(request) 94 | bt.FlushBuffer(true) 95 | 96 | } 97 | 98 | tran.Close() 99 | }() 100 | 101 | fmt.Println("Requests Len: ", len(requests)) 102 | for idx, r := range requests { 103 | r.Wait.Wait() 104 | 105 | // r 原始的请求 106 | req, _ := NewRequest(r.Response.Data, false) 107 | 108 | log.Printf(Green("SeqMatch[%d]: Orig: %d, Return: %d\n"), idx, req.Request.SeqId, r.Request.SeqId) 109 | assert.Equal(t, req.Request.SeqId, r.Request.SeqId) 110 | } 111 | log.Println("OK") 112 | } 113 | -------------------------------------------------------------------------------- /src/proxy/backend_service_lb.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 7 | "github.com/wfxiang08/go_thrift/thrift" 8 | "os" 9 | "runtime" 10 | "strings" 11 | "sync" 12 | "time" 13 | 14 | "github.com/wfxiang08/thrift_rpc_base/rpc_utils" 15 | ) 16 | 17 | // 18 | // Proxy中用来和后端服务通信的模块 19 | // 20 | type BackServiceLB struct { 21 | serviceName string 22 | backendAddr string 23 | 24 | // Mutex的使用： 25 | // 避免使用匿名的Mutex, 需要指定一个语义明确的变量，限定它的使用范围(另可多定义几个Mutex, 不能滥用) 26 | // 27 | // 同时保护: activeConns 和 currentConnIndex 28 | activeConnsLock sync.Mutex 29 | activeConns []*BackendConnLB // 每一个BackendConn应该有一定的高可用保障 30 | currentConnIndex int 31 | 32 | verbose bool 33 | exitEvt chan bool 34 | ch chan thrift.TTransport 35 | } 36 | 37 | // 创建一个BackService 38 | func NewBackServiceLB(serviceName string, backendAddr string, verbose bool, 39 | falconClient string, exitEvt chan bool) *BackServiceLB { 40 | 41 | service := &BackServiceLB{ 42 | serviceName: serviceName, 43 | backendAddr: backendAddr, 44 | activeConns: make([]*BackendConnLB, 0, 10), 45 | verbose: verbose, 46 | exitEvt: exitEvt, 47 | currentConnIndex: 0, 48 | ch: make(chan thrift.TTransport, 4096), 49 | } 50 | 51 | service.run() 52 | 53 | StartTicker(falconClient, serviceName) 54 | return service 55 | 56 | } 57 | 58 | // 59 | // 后端如何处理一个Request, 处理完毕之后直接返回，因为Caller已经做好异步处理了 60 | // Dispatch完毕之后，Request中带有完整的结果 61 | // 62 | func (s *BackServiceLB) Dispatch(r *Request) error { 63 | backendConn := s.nextBackendConn() 64 | 65 | r.Service = s.serviceName 66 | 67 | if backendConn == nil { 68 | // 没有后端服务 69 | if s.verbose { 70 | log.Printf(Red("[%s]No BackSocket Found: %s"), 71 | s.serviceName, r.Request.Name) 72 | } 73 | // 从errMsg来构建异常 74 | errMsg := GetWorkerNotFoundData(r, "BackServiceLB") 75 | // log.Printf(Magenta("---->Convert Error Back to Exception:[%d] %s\n"), len(errMsg), string(errMsg)) 76 | r.Response.Data = errMsg 77 | 78 | return nil 79 | } else { 80 | // if s.verbose { 81 | // log.Println("SendMessage With: ", backendConn.Addr4Log(), "For Service: ", s.serviceName) 82 | // } 83 | backendConn.PushBack(r) 84 | r.Wait.Wait() // 等待处理完毕 85 | 86 | return nil 87 | } 88 | } 89 | 90 | func (s *BackServiceLB) run() { 91 | go func() { 92 | // 定时汇报当前的状态 93 | for true { 94 | log.Printf(Green("[Report]: %s --> %d workers, coroutine: %d"), 95 | s.serviceName, s.Active(), runtime.NumGoroutine()) 96 | time.Sleep(time.Second * 10) 97 | } 98 | }() 99 | 100 | var transport thrift.TServerTransport 101 | var err error 102 | 103 | // 3. 读取后端服务的配置 104 | isUnixDomain := false 105 | // 127.0.0.1:9999(以:区分不同的类型) 106 | if !strings.Contains(s.backendAddr, ":") { 107 | if rpc_utils.FileExist(s.backendAddr) { 108 | os.Remove(s.backendAddr) 109 | } 110 | transport, err = rpc_utils.NewTServerUnixDomain(s.backendAddr) 111 | isUnixDomain = true 112 | } else { 113 | transport, err = thrift.NewTServerSocket(s.backendAddr) 114 | } 115 | 116 | if err != nil { 117 | log.ErrorErrorf(err, "[%s]Server Socket Create Failed: %v", s.serviceName, err) 118 | panic("BackendAddr Invalid") 119 | } 120 | 121 | err = transport.Listen() 122 | if err != nil { 123 | log.ErrorErrorf(err, "[%s]Server Socket Open Failed: %v", s.serviceName, err) 124 | panic("Server Socket Open Failed") 125 | } 126 | 127 | // 和transport.open做的事情一样，如果Open没错，则Listen也不会有问题 128 | 129 | log.Printf(Green("[%s]LB Backend Services listens at: %s"), s.serviceName, s.backendAddr) 130 | 131 | s.ch = make(chan thrift.TTransport, 4096) 132 | 133 | // 强制退出? TODO: Graceful退出 134 | go func() { 135 | <-s.exitEvt 136 | log.Info(Red("Receive Exit Signals....")) 137 | transport.Interrupt() 138 | transport.Close() 139 | }() 140 | 141 | go func() { 142 | var backendAddr string 143 | for trans := range s.ch { 144 | // 为每个Connection建立一个Session 145 | socket, ok := trans.(rpc_utils.SocketAddr) 146 | if ok { 147 | if isUnixDomain { 148 | backendAddr = s.backendAddr 149 | } else { 150 | backendAddr = socket.Addr().String() 151 | } 152 | 153 | // 有可能连接刚刚创建，就立马挂了 154 | conn := NewBackendConnLB(trans, s.serviceName, backendAddr, s, s.verbose) 155 | 156 | // 因为连接刚刚建立，可靠性还是挺高的，因此直接加入到列表中 157 | s.activeConnsLock.Lock() 158 | // 可能在这里就出现 IsConnActive 为False的情况，如果出现了，就不再加入activeConns 159 | if conn.IsConnActive.Get() { 160 | conn.Index = len(s.activeConns) 161 | s.activeConns = append(s.activeConns, conn) 162 | } 163 | s.activeConnsLock.Unlock() 164 | 165 | log.Printf(Green("%s --> %d workers"), s.serviceName, conn.Index) 166 | } else { 167 | panic("Invalid Socket Type") 168 | } 169 | 170 | } 171 | }() 172 | 173 | // Accept什么时候出错，出错之后如何处理呢? 174 | go func() { 175 | for { 176 | c, err := transport.Accept() 177 | if err != nil { 178 | return 179 | } else { 180 | s.ch <- c 181 | } 182 | } 183 | }() 184 | } 185 | 186 | func (s *BackServiceLB) Active() int { 187 | s.activeConnsLock.Lock() 188 | defer s.activeConnsLock.Unlock() 189 | return len(s.activeConns) 190 | } 191 | 192 | // 获取下一个active状态的BackendConn 193 | func (s *BackServiceLB) nextBackendConn() *BackendConnLB { 194 | s.activeConnsLock.Lock() 195 | defer s.activeConnsLock.Unlock() 196 | 197 | // TODO: 暂时采用RoundRobin的方法，可以采用其他具有优先级排列的方法 198 | var backSocket *BackendConnLB 199 | 200 | if len(s.activeConns) == 0 { 201 | if s.verbose { 202 | log.Debugf(Cyan("[%s]ActiveConns Len 0"), s.serviceName) 203 | } 204 | backSocket = nil 205 | } else { 206 | if s.currentConnIndex >= len(s.activeConns) { 207 | s.currentConnIndex = 0 208 | } 209 | backSocket = s.activeConns[s.currentConnIndex] 210 | s.currentConnIndex++ 211 | if s.verbose { 212 | log.Debugf(Cyan("[%s]ActiveConns Len %d, CurrentIndex: %d"), s.serviceName, 213 | len(s.activeConns), s.currentConnIndex) 214 | } 215 | } 216 | return backSocket 217 | } 218 | 219 | // 只有在conn出现错误时才会调用 220 | func (s *BackServiceLB) StateChanged(conn *BackendConnLB) { 221 | s.activeConnsLock.Lock() 222 | defer s.activeConnsLock.Unlock() 223 | 224 | log.Printf(Green("[%s]StateChanged: %s, Index: %d, Count: %d"), conn.serviceName, conn.address, conn.Index, len(s.activeConns)) 225 | if conn.IsConnActive.Get() { 226 | // BackServiceLB 只有一个状态转移: Active --> Not Active 227 | log.Printf(Magenta("Unexpected BackendConnLB State")) 228 | if s.verbose { 229 | panic("Unexpected BackendConnLB State") 230 | } 231 | } else { 232 | log.Printf(Red("Remove BackendConn From activeConns: %s, Index: %d, Count: %d"), 233 | conn.Address(), conn.Index, len(s.activeConns)) 234 | 235 | // 从数组中删除一个元素(O(1)的操作) 236 | if conn.Index != INVALID_ARRAY_INDEX { 237 | // 1. 和最后一个元素进行交换 238 | lastIndex := len(s.activeConns) - 1 239 | if lastIndex != conn.Index { 240 | lastConn := s.activeConns[lastIndex] 241 | 242 | // 将最后一个元素和当前的元素交换位置 243 | s.activeConns[conn.Index] = lastConn 244 | lastConn.Index = conn.Index 245 | 246 | // 删除引用 247 | s.activeConns[lastIndex] = nil 248 | conn.Index = INVALID_ARRAY_INDEX 249 | 250 | } 251 | log.Printf(Red("Remove BackendConn From activeConns: %s"), conn.Address()) 252 | 253 | // 2. slice 254 | s.activeConns = s.activeConns[0:lastIndex] 255 | 256 | } 257 | } 258 | } 259 | -------------------------------------------------------------------------------- /src/proxy/backend_service_proxy.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "github.com/wfxiang08/cyutils/utils/atomic2" 7 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 8 | "runtime" 9 | "strings" 10 | "sync" 11 | "time" 12 | ) 13 | 14 | // 15 | // Proxy中用来和后端服务通信的模块 16 | // 17 | type BackService struct { 18 | productName string 19 | serviceName string 20 | topo *Topology 21 | 22 | // 同时保护: activeConns 和 currentConnIndex 23 | activeConnsLock sync.Mutex 24 | activeConns []*BackendConn // 每一个BackendConn应该有一定的高可用保障 25 | currentConnIndex int 26 | 27 | // 用于zk的状态管理(记录当前有效的Conn) 28 | addr2Conn map[string]*BackendConn 29 | verbose bool 30 | stop atomic2.Bool 31 | lastRequestTime atomic2.Int64 32 | evtbus chan interface{} 33 | } 34 | 35 | // 创建一个BackService 36 | func NewBackService(productName string, serviceName string, topo *Topology, verbose bool) *BackService { 37 | 38 | service := &BackService{ 39 | productName: productName, 40 | serviceName: serviceName, 41 | activeConns: make([]*BackendConn, 0, 10), 42 | addr2Conn: make(map[string]*BackendConn), 43 | topo: topo, 44 | verbose: verbose, 45 | } 46 | 47 | service.WatchBackServiceNodes() 48 | 49 | go func() { 50 | for !service.stop.Get() { 51 | log.Printf(Blue("[Report]: %s --> %d backservice, coroutine: %d"), 52 | service.serviceName, service.Active(), runtime.NumGoroutine()) 53 | time.Sleep(time.Second * 10) 54 | } 55 | }() 56 | 57 | return service 58 | 59 | } 60 | 61 | func (s *BackService) Stop() { 62 | // 标志停止 63 | s.stop.Set(true) 64 | // 触发一个事件（之后ServiceNodes也不再监控) 65 | s.evtbus <- true 66 | go func() { 67 | // TODO: 68 | for true { 69 | now := time.Now().Unix() 70 | if now-s.lastRequestTime.Get() > 10 { 71 | break 72 | } else { 73 | time.Sleep(time.Second) 74 | } 75 | } 76 | for len(s.activeConns) > 0 { 77 | s.activeConns[0].MarkOffline() 78 | } 79 | 80 | log.Printf(Red("Mark All Connections Off: %s"), s.serviceName) 81 | 82 | }() 83 | } 84 | 85 | func (s *BackService) Active() int { 86 | return len(s.activeConns) 87 | } 88 | 89 | // 90 | // 如何处理后端服务的变化呢? 91 | // 92 | func (s *BackService) WatchBackServiceNodes() { 93 | s.evtbus = make(chan interface{}, 2) 94 | servicePath := s.topo.ProductServicePath(s.serviceName) 95 | 96 | go func() { 97 | for !s.stop.Get() { 98 | serviceIds, err := s.topo.WatchChildren(servicePath, s.evtbus) 99 | 100 | if err == nil { 101 | // 如何监听endpoints的变化呢? 102 | addressMap := make(map[string]bool, len(serviceIds)) 103 | 104 | for _, serviceId := range serviceIds { 105 | log.Printf(Green("---->Find Endpoint: %s for Service: %s"), serviceId, s.serviceName) 106 | endpointInfo, err := GetServiceEndpoint(s.topo, s.serviceName, serviceId) 107 | 108 | if err != nil { 109 | log.ErrorErrorf(err, "Service Endpoint Read Error: %v\n", err) 110 | } else { 111 | 112 | log.Printf(Green("---->Add endpoint %s To Service %s"), 113 | endpointInfo.Frontend, s.serviceName) 114 | 115 | if strings.Contains(endpointInfo.Frontend, ":") { 116 | addressMap[endpointInfo.Frontend] = true 117 | } else if s.productName == TEST_PRODUCT_NAME { 118 | // unix domain socket只在测试的时候可以使用(因为不能实现跨机器访问） 119 | addressMap[endpointInfo.Frontend] = true 120 | } 121 | } 122 | } 123 | 124 | for addr, _ := range addressMap { 125 | conn, ok := s.addr2Conn[addr] 126 | if ok && !conn.IsMarkOffline.Get() { 127 | continue 128 | } else { 129 | // 创建新的连接（心跳成功之后就自动加入到 s.activeConns 中 130 | s.addr2Conn[addr] = NewBackendConn(addr, s, s.serviceName, s.verbose) 131 | } 132 | } 133 | 134 | for addr, conn := range s.addr2Conn { 135 | _, ok := addressMap[addr] 136 | if !ok { 137 | conn.MarkOffline() 138 | 139 | // 删除: 然后等待Conn自生自灭 140 | delete(s.addr2Conn, addr) 141 | } 142 | } 143 | 144 | // 等待事件 145 | <-s.evtbus 146 | } else { 147 | log.WarnErrorf(err, "zk read failed: %s", servicePath) 148 | // 如果读取失败则，则继续等待5s 149 | time.Sleep(time.Duration(5) * time.Second) 150 | } 151 | 152 | } 153 | }() 154 | } 155 | 156 | // 获取下一个active状态的BackendConn 157 | func (s *BackService) NextBackendConn() *BackendConn { 158 | var backSocket *BackendConn 159 | 160 | s.activeConnsLock.Lock() 161 | defer s.activeConnsLock.Unlock() 162 | 163 | if len(s.activeConns) == 0 { 164 | backSocket = nil 165 | } else { 166 | if s.currentConnIndex >= len(s.activeConns) { 167 | s.currentConnIndex = 0 168 | } 169 | backSocket = s.activeConns[s.currentConnIndex] 170 | s.currentConnIndex++ 171 | } 172 | 173 | return backSocket 174 | } 175 | 176 | // 177 | // 将消息发送到Backend上去 178 | // 179 | func (s *BackService) HandleRequest(req *Request) (err error) { 180 | // 并发度可能很高 181 | backendConn := s.NextBackendConn() 182 | 183 | s.lastRequestTime.Set(time.Now().Unix()) 184 | 185 | if backendConn == nil { 186 | // 没有后端服务 187 | if s.verbose { 188 | log.Println(Red("No BackSocket Found for service:"), s.serviceName) 189 | } 190 | // 从errMsg来构建异常 191 | errMsg := GetWorkerNotFoundData(req, "BackService") 192 | req.Response.Data = errMsg 193 | // XXX: 没有等待，req.Wait.Wait() 直接返回 194 | 195 | return nil 196 | } else { 197 | if s.verbose { 198 | log.Println("SendMessage With: ", backendConn.Addr(), "For Service: ", s.serviceName) 199 | } 200 | backendConn.PushBack(req) 201 | return nil 202 | } 203 | } 204 | 205 | func (s *BackService) StateChanged(conn *BackendConn) { 206 | // log.Printf(Cyan("[%s]StateChanged: %s, Index: %d, Count: %d, IsConnActive: %t"), 207 | // s.serviceName, conn.addr, conn.Index, len(s.activeConns), 208 | // conn.IsConnActive.Get()) 209 | 210 | s.activeConnsLock.Lock() 211 | defer s.activeConnsLock.Unlock() 212 | 213 | if conn.IsConnActive.Get() { 214 | // 上线: BackendConn 215 | log.Printf(Cyan("[%s]MarkConnActiveOK: %s, Index: %d, Count: %d"), 216 | s.serviceName, conn.addr, conn.Index, len(s.activeConns)) 217 | 218 | if conn.Index == INVALID_ARRAY_INDEX { 219 | conn.Index = len(s.activeConns) 220 | s.activeConns = append(s.activeConns, conn) 221 | 222 | log.Printf(Green("[%s]Add BackendConn to activeConns: %s, Total Actives: %d"), 223 | s.serviceName, conn.Addr(), len(s.activeConns)) 224 | } 225 | } else { 226 | // 下线BackendConn(急速执行 227 | connIndex := conn.Index 228 | if conn.Index != INVALID_ARRAY_INDEX { 229 | lastIndex := len(s.activeConns) - 1 230 | 231 | // 将最后一个元素和当前的元素交换位置 232 | if lastIndex != conn.Index { 233 | 234 | lastConn := s.activeConns[lastIndex] 235 | s.activeConns[conn.Index] = lastConn 236 | lastConn.Index = conn.Index 237 | } 238 | 239 | s.activeConns[lastIndex] = nil 240 | conn.Index = INVALID_ARRAY_INDEX 241 | 242 | // slice 243 | s.activeConns = s.activeConns[0:lastIndex] 244 | log.Printf(Red("[%s]Remove BackendConn From activeConns: %s, Remains: %d"), 245 | s.serviceName, conn.Addr(), len(s.activeConns)) 246 | } 247 | log.Printf(Red("[%s]Remove BackendConn From activeConns: %s, Index: %d"), 248 | s.serviceName, conn.Addr(), connIndex) 249 | } 250 | } 251 | -------------------------------------------------------------------------------- /src/proxy/backend_service_proxy_test.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | -------------------------------------------------------------------------------- /src/proxy/buffered_framed_transport.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "bytes" 7 | "encoding/binary" 8 | "fmt" 9 | "github.com/wfxiang08/go_thrift/thrift" 10 | 11 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 12 | "io" 13 | "time" 14 | ) 15 | 16 | const DEFAULT_MAX_LENGTH = 16384000 17 | 18 | // Framed 和 Buffered各做了什么事情呢? 19 | 20 | // TTransport接口的实现: Read/Write/Close/Flush/Open/IsOpen 21 | // Framed就是在数据块的开头加上了数据库的长度，相当于增加了一个Frame的概念 22 | type TBufferedFramedTransport struct { 23 | *thrift.TBufferedTransport 24 | FrameSize int // Current remaining size of the frame. if ==0 read next frame header 25 | LenghR [4]byte // 临时使用(无状态) 26 | LenghW [4]byte // 临时使用(无状态) 27 | 28 | Buffer *bytes.Buffer // 结构体，不需要初始化 29 | 30 | // 安全控制 31 | maxLength int 32 | 33 | MaxBuffered int // 单位: 请求个数 34 | MaxInterval int64 // 单位: nanoseconds 35 | nbuffered int 36 | lastflush int64 // 单位: ns(1e-9s) 37 | } 38 | 39 | func NewTBufferedFramedTransport(transport thrift.TTransport, 40 | maxInterval time.Duration, 41 | maxBuffered int) *TBufferedFramedTransport { 42 | 43 | return &TBufferedFramedTransport{ 44 | TBufferedTransport: thrift.NewTBufferedTransport(transport, 64*1024), 45 | maxLength: DEFAULT_MAX_LENGTH, 46 | Buffer: bytes.NewBuffer(make([]byte, 0, 1024)), 47 | MaxInterval: int64(maxInterval), 48 | MaxBuffered: maxBuffered, 49 | } 50 | } 51 | 52 | // Transport上的Buffer是低频需求，不需要优化 53 | func NewTBufferedFramedTransportMaxLength(transport thrift.TTransport, 54 | maxInterval time.Duration, maxBuffered int, 55 | maxLength int) *TBufferedFramedTransport { 56 | 57 | return &TBufferedFramedTransport{ 58 | TBufferedTransport: thrift.NewTBufferedTransport(transport, 64*1024), 59 | maxLength: maxLength, 60 | Buffer: bytes.NewBuffer(make([]byte, 0, 1024)), 61 | MaxInterval: int64(maxInterval), 62 | MaxBuffered: maxBuffered, 63 | } 64 | } 65 | 66 | // 读取Frame的完整的数据，包含 67 | func (p *TBufferedFramedTransport) ReadFrame() (frame []byte, err error) { 68 | 69 | if p.FrameSize != 0 { 70 | err = thrift.NewTTransportExceptionFromError( 71 | fmt.Errorf("Unexpected frame size: %d", p.FrameSize)) 72 | return nil, err 73 | } 74 | var frameSize int 75 | 76 | // Reader来自transport, 中间被封装了多长 77 | frameSize, err = p.readFrameHeader() 78 | if err != nil { 79 | err1, ok := err.(thrift.TTransportException) 80 | 81 | if ok { 82 | err = thrift.NewTTransportException(err1.TypeId(), 83 | fmt.Sprintf("Frame Header Read Error: %s", err1.Error())) 84 | } else { 85 | err = thrift.NewTTransportExceptionFromError( 86 | fmt.Errorf("Frame Header Read Error: %s", err.Error())) 87 | } 88 | return 89 | } 90 | 91 | bytes := getSlice(frameSize, frameSize) 92 | 93 | // 什么时候会出现? 94 | // 1. 如果tcp package比较大，则容易出现package的一部分先到，另一部分随后再到 95 | // 2. 在同一个机器上的两个进程之间出现的概率低(Time Delay小); 跨机器访问，出现概率高 96 | var l int 97 | l, err = io.ReadFull(p.Reader, bytes) 98 | // l, err = p.Reader.Read(bytes) 99 | if l != frameSize { 100 | log.Warnf(Red("<==== ReadFrame frame size: %d, Got: %d"), frameSize, l) 101 | } 102 | 103 | if err != nil { 104 | err1, ok := err.(thrift.TTransportException) 105 | if ok { 106 | err = thrift.NewTTransportException(err1.TypeId(), 107 | fmt.Sprintf("Frame Data Read Error: %s", err1.Error())) 108 | } else { 109 | err = thrift.NewTTransportExceptionFromError( 110 | fmt.Errorf("Frame Data Read Error: %s", err.Error())) 111 | } 112 | return nil, err 113 | } 114 | 115 | return bytes, nil 116 | 117 | } 118 | 119 | func (p *TBufferedFramedTransport) Read(buf []byte) (l int, err error) { 120 | 121 | // 1. 首先读取Frame Header 122 | if p.FrameSize == 0 { 123 | p.FrameSize, err = p.readFrameHeader() 124 | if err != nil { 125 | return 126 | } 127 | } 128 | 129 | // 2. 异常处理 130 | if p.FrameSize < len(buf) { 131 | frameSize := p.FrameSize 132 | tmp := make([]byte, p.FrameSize) 133 | l, err = p.Read(tmp) 134 | copy(buf, tmp) 135 | if err == nil { 136 | err = thrift.NewTTransportExceptionFromError( 137 | fmt.Errorf("Not enough frame size %d to read %d bytes", 138 | frameSize, len(buf))) 139 | return 140 | } 141 | } 142 | 143 | // 3. 读取剩下的数据 144 | got, err := p.Reader.Read(buf) 145 | p.FrameSize = p.FrameSize - got 146 | //sanity check 147 | if p.FrameSize < 0 { 148 | return 0, thrift.NewTTransportException(thrift.UNKNOWN_TRANSPORT_EXCEPTION, 149 | "Negative frame size") 150 | } 151 | return got, thrift.NewTTransportExceptionFromError(err) 152 | } 153 | 154 | func (p *TBufferedFramedTransport) ReadByte() (c byte, err error) { 155 | if p.FrameSize == 0 { 156 | p.FrameSize, err = p.readFrameHeader() 157 | if err != nil { 158 | return 159 | } 160 | } 161 | if p.FrameSize < 1 { 162 | return 0, thrift.NewTTransportExceptionFromError( 163 | fmt.Errorf("Not enough frame size %d to read %d bytes", p.FrameSize, 1)) 164 | } 165 | c, err = p.Reader.ReadByte() 166 | if err == nil { 167 | p.FrameSize-- 168 | } 169 | return 170 | } 171 | 172 | func (p *TBufferedFramedTransport) Write(buf []byte) (int, error) { 173 | // 直接写入Buffer 174 | n, err := p.Buffer.Write(buf) 175 | if n != len(buf) { 176 | log.Errorf("TBufferedFramedTransport#Write Error") 177 | } 178 | return n, thrift.NewTTransportExceptionFromError(err) 179 | } 180 | 181 | func (p *TBufferedFramedTransport) WriteByte(c byte) error { 182 | return p.Buffer.WriteByte(c) 183 | } 184 | 185 | func (p *TBufferedFramedTransport) WriteString(s string) (n int, err error) { 186 | return p.Buffer.WriteString(s) 187 | } 188 | func (p *TBufferedFramedTransport) Flush() error { 189 | // 兼容默认的策略 190 | return p.FlushBuffer(true) 191 | } 192 | 193 | func (p *TBufferedFramedTransport) flushTransport(force bool) error { 194 | if p.nbuffered > 0 && (force || p.needFlush()) { 195 | // 这个如何控制呢? 196 | if err := p.Writer.Flush(); err != nil { 197 | log.ErrorErrorf(err, "FlushTransport Error, %v", err) 198 | return err 199 | } 200 | p.nbuffered = 0 201 | p.lastflush = time.Now().UnixNano() 202 | } 203 | return nil 204 | } 205 | 206 | // 207 | // 先写入数据，然后再Flush Transport 208 | // 209 | func (p *TBufferedFramedTransport) FlushBuffer(force bool) error { 210 | size := p.Buffer.Len() 211 | 212 | // 没有有效的数据，直接返回 213 | if size == 0 { 214 | return nil 215 | } 216 | // TODO: 待优化 217 | force = true 218 | 219 | // 1. 将p.buf的大小以BigEndian模式写入: buf中 220 | buf := p.LenghW[:4] 221 | binary.BigEndian.PutUint32(buf, uint32(size)) 222 | 223 | // log.Printf("----> Frame Size: %d, %v\n", size, buf) 224 | // 然后transport中先写入: 长度信息 225 | _, err := p.Writer.Write(buf) 226 | if err != nil { 227 | return thrift.NewTTransportExceptionFromError(err) 228 | } 229 | 230 | // log.Printf("Flushing buffer with size: %d to backend", size) 231 | 232 | // 2. 然后继续写入p.buf中的数据 233 | if size > 0 { 234 | var ( 235 | n int64 236 | err error 237 | ) 238 | // 如果 err == io.ErrShortWrite， p.Writer中也有buffer, 因此可以不用考虑异常 239 | if n, err = p.Buffer.WriteTo(p.Writer); err != nil { 240 | log.ErrorErrorf(err, "Error Flushing Expect Write: %d, but %d\n", 241 | size, n) 242 | return thrift.NewTTransportExceptionFromError(err) 243 | } 244 | if n < int64(size) { 245 | log.Printf(Red("Buffer Write Not Finished")) 246 | } 247 | } 248 | 249 | p.nbuffered++ 250 | 251 | // Buffer重新开始处理数据 252 | p.Buffer.Reset() 253 | 254 | // Flush Buffer 255 | return p.flushTransport(force) 256 | } 257 | 258 | func (p *TBufferedFramedTransport) readFrameHeader() (int, error) { 259 | buf := p.LenghR[:4] 260 | 261 | if _, err := io.ReadFull(p.Reader, buf); err != nil { 262 | return 0, err 263 | } 264 | size := int(binary.BigEndian.Uint32(buf)) 265 | if size < 0 || size > p.maxLength { 266 | return 0, thrift.NewTTransportException(thrift.UNKNOWN_TRANSPORT_EXCEPTION, 267 | fmt.Sprintf("Incorrect frame size (%d)", size)) 268 | } 269 | return size, nil 270 | } 271 | 272 | func (p *TBufferedFramedTransport) needFlush() bool { 273 | if p.nbuffered != 0 { 274 | // request个数 buffer到一定数量 275 | if p.nbuffered >= p.MaxBuffered { 276 | return true 277 | } 278 | 279 | // 或者间隔到达一定时间，则flush 280 | if time.Now().UnixNano()-p.lastflush > p.MaxInterval { 281 | return true 282 | } 283 | } 284 | return false 285 | } 286 | -------------------------------------------------------------------------------- /src/proxy/config.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | 4 | package proxy 5 | 6 | import ( 7 | "fmt" 8 | "github.com/c4pt0r/cfg" 9 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 10 | "os" 11 | "path" 12 | "strings" 13 | ) 14 | 15 | type ProductConfig struct { 16 | ProductName string 17 | ZkAddr string 18 | ZkSessionTimeout int 19 | } 20 | type ServiceConfig struct { 21 | ProductConfig 22 | StandAlone bool 23 | Service string 24 | FrontHost string 25 | FrontPort string 26 | FrontSock string 27 | FrontendAddr string 28 | IpPrefix string 29 | 30 | BackAddr string 31 | 32 | Profile bool 33 | Verbose bool 34 | 35 | // 提供给dashboard来查看服务状态 36 | WorkDir string 37 | CodeUrlVersion string 38 | 39 | // 用于监控 40 | FalconClient string 41 | } 42 | 43 | type ProxyConfig struct { 44 | ProductConfig 45 | 46 | ProxyAddr string 47 | Profile bool 48 | Verbose bool 49 | } 50 | 51 | // 52 | // 通过参数依赖，保证getFrontendAddr的调用位置（必须等待Host, IpPrefix, Port读取完毕之后) 53 | // 54 | func (conf *ServiceConfig) getFrontendAddr(frontHost, ipPrefix, frontPort string) string { 55 | if conf.FrontSock != "" { 56 | return conf.FrontSock 57 | } 58 | 59 | var frontendAddr = "" 60 | // 如果没有指定FrontHost, 则自动根据 IpPrefix来进行筛选， 61 | // 例如: IpPrefix: 10., 那么最终内网IP： 10.4.10.2之类的被选中 62 | if frontHost == "" { 63 | log.Println("FrontHost: ", frontHost, ", Prefix: ", ipPrefix) 64 | if ipPrefix != "" { 65 | frontHost = GetIpWithPrefix(ipPrefix) 66 | } 67 | } 68 | if frontPort != "" && frontHost != "" { 69 | frontendAddr = fmt.Sprintf("%s:%s", frontHost, frontPort) 70 | } 71 | return frontendAddr 72 | } 73 | 74 | func LoadConf(configFile string) (*ServiceConfig, error) { 75 | c := cfg.NewCfg(configFile) 76 | if err := c.Load(); err != nil { 77 | log.PanicErrorf(err, "load config '%s' failed", configFile) 78 | } 79 | 80 | conf := &ServiceConfig{} 81 | 82 | // 读取product 83 | conf.ProductName, _ = c.ReadString("product", "test") 84 | if len(conf.ProductName) == 0 { 85 | log.Panicf("invalid config: product entry is missing in %s", configFile) 86 | } 87 | 88 | // 读取zk 89 | conf.ZkAddr, _ = c.ReadString("zk", "") 90 | if len(conf.ZkAddr) == 0 { 91 | log.Panicf("invalid config: need zk entry is missing in %s", configFile) 92 | } 93 | conf.ZkAddr = strings.TrimSpace(conf.ZkAddr) 94 | 95 | loadConfInt := func(entry string, defInt int) int { 96 | v, _ := c.ReadInt(entry, defInt) 97 | if v < 0 { 98 | log.Panicf("invalid config: read %s = %d", entry, v) 99 | } 100 | return v 101 | } 102 | 103 | conf.ZkSessionTimeout = loadConfInt("zk_session_timeout", 30) 104 | conf.Verbose = loadConfInt("verbose", 0) == 1 105 | 106 | // 是否独立于zookeeper独立运行 107 | conf.StandAlone = loadConfInt("stand_alone", 0) == 1 108 | 109 | conf.Service, _ = c.ReadString("service", "") 110 | conf.Service = strings.TrimSpace(conf.Service) 111 | 112 | conf.FrontHost, _ = c.ReadString("front_host", "") 113 | conf.FrontHost = strings.TrimSpace(conf.FrontHost) 114 | 115 | conf.FrontPort, _ = c.ReadString("front_port", "") 116 | conf.FrontPort = strings.TrimSpace(conf.FrontPort) 117 | 118 | conf.FrontSock, _ = c.ReadString("front_sock", "") 119 | conf.FrontSock = strings.TrimSpace(conf.FrontSock) 120 | // 配置文件中使用的是相对路径，在注册到zk时，需要还原成为绝对路径 121 | if len(conf.FrontSock) > 0 && !strings.HasPrefix(conf.FrontSock, "/") { 122 | dir, _ := os.Getwd() 123 | conf.FrontSock = path.Clean(path.Join(dir, conf.FrontSock)) 124 | } 125 | 126 | conf.IpPrefix, _ = c.ReadString("ip_prefix", "") 127 | conf.IpPrefix = strings.TrimSpace(conf.IpPrefix) 128 | 129 | // 注意先后顺序: 130 | // FrontHost, FrontPort, IpPrefix之后才能计算FrontendAddr 131 | conf.FrontendAddr = conf.getFrontendAddr(conf.FrontHost, conf.IpPrefix, conf.FrontPort) 132 | 133 | conf.BackAddr, _ = c.ReadString("back_address", "") 134 | conf.BackAddr = strings.TrimSpace(conf.BackAddr) 135 | 136 | conf.FalconClient, _ = c.ReadString("falcon_client", "") 137 | 138 | profile, _ := c.ReadInt("profile", 0) 139 | conf.Profile = profile == 1 140 | return conf, nil 141 | } 142 | 143 | func LoadProxyConf(configFile string) (*ProxyConfig, error) { 144 | c := cfg.NewCfg(configFile) 145 | if err := c.Load(); err != nil { 146 | log.PanicErrorf(err, "load config '%s' failed", configFile) 147 | } 148 | 149 | conf := &ProxyConfig{} 150 | 151 | // 读取product 152 | conf.ProductName, _ = c.ReadString("product", "test") 153 | if len(conf.ProductName) == 0 { 154 | log.Panicf("invalid config: product entry is missing in %s", configFile) 155 | } 156 | 157 | // 读取zk 158 | conf.ZkAddr, _ = c.ReadString("zk", "") 159 | if len(conf.ZkAddr) == 0 { 160 | log.Panicf("invalid config: need zk entry is missing in %s", configFile) 161 | } 162 | conf.ZkAddr = strings.TrimSpace(conf.ZkAddr) 163 | 164 | loadConfInt := func(entry string, defInt int) int { 165 | v, _ := c.ReadInt(entry, defInt) 166 | if v < 0 { 167 | log.Panicf("invalid config: read %s = %d", entry, v) 168 | } 169 | return v 170 | } 171 | 172 | conf.ZkSessionTimeout = loadConfInt("zk_session_timeout", 30) 173 | conf.Verbose = loadConfInt("verbose", 0) == 1 174 | 175 | conf.ProxyAddr, _ = c.ReadString("proxy_address", "") 176 | conf.ProxyAddr = strings.TrimSpace(conf.ProxyAddr) 177 | 178 | profile, _ := c.ReadInt("profile", 0) 179 | conf.Profile = profile == 1 180 | return conf, nil 181 | } 182 | -------------------------------------------------------------------------------- /src/proxy/consts.go: -------------------------------------------------------------------------------- 1 | package proxy 2 | 3 | const ( 4 | // Thrift协议中 SEQ_ID的访问 5 | BACKEND_CONN_MIN_SEQ_ID = 1 6 | BACKEND_CONN_MAX_SEQ_ID = 1000000 // 100万次请求一个轮回(也挺不容易的) 7 | INVALID_ARRAY_INDEX = -1 // 无效的数组元素下标 8 | HB_TIMEOUT = 6 // 心跳超时时间间隔 9 | REQUEST_EXPIRED_TIME_SECONDS = 15 10 | REQUEST_EXPIRED_TIME_MICRO = REQUEST_EXPIRED_TIME_SECONDS * 1000000 // 15s 11 | TEST_PRODUCT_NAME = "test" 12 | VERSION = "0.1.0-2015121712" // 版本信息 13 | ) 14 | -------------------------------------------------------------------------------- /src/proxy/endpoint.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "encoding/json" 7 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 8 | zookeeper "github.com/wfxiang08/go-zookeeper/zk" 9 | "os" 10 | os_path "path" 11 | "time" 12 | "github.com/wfxiang08/thrift_rpc_base/zkhelper" 13 | ) 14 | 15 | type ServiceEndpoint struct { 16 | Service string `json:"service"` 17 | ServiceId string `json:"service_id"` 18 | Frontend string `json:"frontend"` 19 | DeployPath string `json:"deploy_path"` 20 | CodeUrlVerion string `json:"code_url_version"` 21 | Hostname string `json:"hostname"` 22 | StartTime string `json:"start_time"` 23 | } 24 | 25 | func NewServiceEndpoint(service string, serviceId string, frontend string, 26 | deployPath string, codeUrlVerion string) *ServiceEndpoint { 27 | 28 | hostname, err := os.Hostname() 29 | if err != nil { 30 | hostname = "Unknown" 31 | } 32 | 33 | startTime := FormatYYYYmmDDHHMMSS(time.Now()) 34 | 35 | return &ServiceEndpoint{ 36 | Service: service, 37 | ServiceId: serviceId, 38 | Frontend: frontend, 39 | DeployPath: deployPath, 40 | CodeUrlVerion: codeUrlVerion, 41 | Hostname: hostname, 42 | StartTime: startTime, 43 | } 44 | } 45 | 46 | // 47 | // 删除Service Endpoint 48 | // 49 | func (s *ServiceEndpoint) DeleteServiceEndpoint(top *Topology) { 50 | path := top.ProductServiceEndPointPath(s.Service, s.ServiceId) 51 | if ok, _ := top.Exist(path); ok { 52 | zkhelper.DeleteRecursive(top.ZkConn, path, -1) 53 | log.Println(Red("DeleteServiceEndpoint"), "Path: ", path) 54 | } 55 | } 56 | 57 | // 58 | // 注册一个服务的Endpoints 59 | // 60 | func (s *ServiceEndpoint) AddServiceEndpoint(topo *Topology) error { 61 | path := topo.ProductServiceEndPointPath(s.Service, s.ServiceId) 62 | data, err := json.Marshal(s) 63 | if err != nil { 64 | return err 65 | } 66 | 67 | // 创建Service(XXX: Service本身不包含数据) 68 | CreateRecursive(topo.ZkConn, os_path.Dir(path), "", 0, zkhelper.DefaultDirACLs()) 69 | 70 | // 当前的Session挂了，服务就下线 71 | // topo.FlagEphemeral 72 | 73 | // 参考： https://www.box.com/blog/a-gotcha-when-using-zookeeper-ephemeral-nodes/ 74 | // 如果之前的Session信息还存在，则先删除；然后再添加 75 | topo.ZkConn.Delete(path, -1) 76 | var pathCreated string 77 | pathCreated, err = topo.ZkConn.Create(path, []byte(data), int32(zookeeper.FlagEphemeral), zkhelper.DefaultFileACLs()) 78 | 79 | log.Println(Green("AddServiceEndpoint"), "Path: ", pathCreated, ", Error: ", err) 80 | return err 81 | } 82 | 83 | func GetServiceEndpoint(top *Topology, service string, serviceId string) (endpoint *ServiceEndpoint, err error) { 84 | 85 | path := top.ProductServiceEndPointPath(service, serviceId) 86 | data, _, err := top.ZkConn.Get(path) 87 | if err != nil { 88 | return nil, err 89 | } 90 | endpoint = &ServiceEndpoint{} 91 | err = json.Unmarshal(data, endpoint) 92 | if err != nil { 93 | return nil, err 94 | } else { 95 | return endpoint, nil 96 | } 97 | } 98 | -------------------------------------------------------------------------------- /src/proxy/endpoint_test.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "encoding/json" 7 | "fmt" 8 | "github.com/stretchr/testify/assert" 9 | "testing" 10 | ) 11 | 12 | // 13 | // go test proxy -v -run "TestEndpoint" 14 | // 15 | func TestEndpoint(t *testing.T) { 16 | // {"service":"Service","service_id":"ServiceId","frontend":"Frontend","deploy_path":"DeployPath","hostname":"Hostname","start_time":"StartTime"} 17 | 18 | // 1. 正常的Marshal & Unmarshal 19 | endpoint := &ServiceEndpoint{ 20 | Service: "Service", 21 | ServiceId: "ServiceId", 22 | Frontend: "Frontend", 23 | DeployPath: "DeployPath", 24 | Hostname: "Hostname", 25 | StartTime: "StartTime", 26 | } 27 | 28 | data, _ := json.Marshal(endpoint) 29 | fmt.Println("Endpoint: ", string(data)) 30 | 31 | assert.True(t, true) 32 | 33 | // 2. 缺少字段时的Unmarshal(缺少的字段为空) 34 | data21 := []byte(`{"service":"Service","service_id":"ServiceId","frontend":"Frontend"}`) 35 | 36 | endpoint2 := &ServiceEndpoint{} 37 | err2 := json.Unmarshal(data21, endpoint2) 38 | assert.True(t, err2 == nil) 39 | 40 | fmt.Println("Error2: ", err2) 41 | data22, _ := json.Marshal(endpoint2) 42 | fmt.Println("Endpoint2: ", string(data22)) 43 | 44 | // 3. 字段多的情况下的Unmarshal(多余的字段直接忽略) 45 | data31 := []byte(`{"service":"Service", "serviceA":"AService","service_id":"ServiceId","frontend":"Frontend"}`) 46 | endpoint3 := &ServiceEndpoint{} 47 | err3 := json.Unmarshal(data31, endpoint3) 48 | assert.True(t, err3 == nil) 49 | fmt.Println("Error3: ", err3) 50 | data32, _ := json.Marshal(endpoint3) 51 | fmt.Println("Endpoint3: ", string(data32)) 52 | 53 | } 54 | -------------------------------------------------------------------------------- /src/proxy/heartbeat.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "github.com/wfxiang08/go_thrift/thrift" 7 | "github.com/wfxiang08/thrift_rpc_base/rpcthrift/services" 8 | ) 9 | 10 | // 11 | // 生成Thrift格式的Exception Message 12 | // 13 | func HandlePingRequest(req *Request) { 14 | req.Response.Data = req.Request.Data 15 | } 16 | 17 | func HandleProxyPingRequest(req *Request) { 18 | transport := NewTMemoryBufferLen(30) 19 | protocol := thrift.NewTBinaryProtocolTransport(transport) 20 | protocol.WriteMessageBegin("ping", thrift.REPLY, req.Request.SeqId) 21 | result := services.NewRpcServiceBasePingResult() 22 | result.Write(protocol) 23 | protocol.WriteMessageEnd() 24 | protocol.Flush() 25 | 26 | req.Response.Data = transport.Bytes() 27 | } 28 | 29 | func NewPingRequest() *Request { 30 | // 构建thrift的Transport 31 | 32 | transport := NewTMemoryBufferLen(30) 33 | protocol := thrift.NewTBinaryProtocolTransport(transport) 34 | protocol.WriteMessageBegin("ping", MESSAGE_TYPE_HEART_BEAT, 0) 35 | protocol.WriteMessageEnd() 36 | protocol.Flush() 37 | 38 | r := &Request{} 39 | // 告诉Request, Data中不包含service，在ReplaceSeqId时不需要特别处理 40 | r.ProxyRequest = false 41 | r.Start = microseconds() 42 | r.Request.Data = transport.Bytes() 43 | r.Request.Name = "ping" 44 | r.Request.SeqId = 0 // SeqId在这里无效，因此设置为0 45 | r.Request.TypeId = MESSAGE_TYPE_HEART_BEAT 46 | return r 47 | } 48 | 49 | func NewStopConfirmRequest() *Request { 50 | // 构建thrift的Transport 51 | 52 | transport := NewTMemoryBufferLen(30) 53 | protocol := thrift.NewTBinaryProtocolTransport(transport) 54 | protocol.WriteMessageBegin("stop_confirm", MESSAGE_TYPE_STOP_CONFIRM, 0) 55 | protocol.WriteMessageEnd() 56 | protocol.Flush() 57 | 58 | r := &Request{} 59 | // 告诉Request, Data中不包含service，在ReplaceSeqId时不需要特别处理 60 | r.ProxyRequest = false 61 | r.Start = microseconds() 62 | r.Request.Data = transport.Bytes() 63 | r.Request.Name = "stop_confirm" 64 | r.Request.SeqId = 0 // SeqId在这里无效，因此设置为0 65 | r.Request.TypeId = MESSAGE_TYPE_STOP_CONFIRM 66 | return r 67 | } -------------------------------------------------------------------------------- /src/proxy/heartbeat_test.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "encoding/binary" 7 | "fmt" 8 | "github.com/stretchr/testify/assert" 9 | "testing" 10 | ) 11 | 12 | // 13 | // go test proxy -v -run "TestHeatbeat" 14 | // 15 | func TestHeatbeat(t *testing.T) { 16 | 17 | request := NewPingRequest() 18 | 19 | fmt.Printf("Data Len: %d, Data:[%s]", len(request.Request.Data), request.Request.Data) 20 | 21 | for i := 0; i < len(request.Request.Data); i++ { 22 | fmt.Printf("%d,", request.Request.Data[i]) 23 | } 24 | fmt.Println() 25 | 26 | assert.False(t, request.ProxyRequest) 27 | 28 | request.Service = "测试" 29 | len1 := request.Request.Data 30 | request.ReplaceSeqId(121) 31 | 32 | len2 := request.Request.Data 33 | assert.Equal(t, len1, len2) 34 | 35 | buf := make([]byte, 4, 4) 36 | binary.BigEndian.PutUint32(buf, uint32(16)) 37 | fmt.Println("Frame Len: ") 38 | for i := 0; i < len(buf); i++ { 39 | fmt.Printf("%d,", buf[i]) 40 | } 41 | fmt.Println() 42 | 43 | } 44 | -------------------------------------------------------------------------------- /src/proxy/memory_alloc.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | 4 | package proxy 5 | 6 | import ( 7 | "reflect" 8 | "unsafe" 9 | ) 10 | 11 | const ( 12 | DEFAULT_SLICE_LEN = 1024 13 | ) 14 | 15 | var memoryBuffer1024 chan []byte = make(chan []byte, 1000) // 1M 16 | var memoryBuffer2048 chan []byte = make(chan []byte, 1000) // 2M 17 | 18 | func debugBuffer1024Size() int { 19 | return len(memoryBuffer1024) 20 | } 21 | 22 | func debugBuffer2048Size() int { 23 | return len(memoryBuffer2048) 24 | } 25 | 26 | // 27 | // 自己管理内存： 申请(尽量不要使用，很容易出现错误) 28 | // 29 | // 实现逻辑: make([]byte, size, xxx) 30 | // 31 | func getSlice(initSize int, capacity int) []byte { 32 | if initSize > capacity { 33 | panic("Invalid Slice Size") 34 | } 35 | 36 | var result []byte 37 | if capacity < DEFAULT_SLICE_LEN { 38 | select { 39 | case result = <-memoryBuffer1024: 40 | default: 41 | return make([]byte, initSize, DEFAULT_SLICE_LEN) 42 | } 43 | } else if capacity < 2048 { 44 | select { 45 | case result = <-memoryBuffer2048: 46 | default: 47 | return make([]byte, initSize, 2048) 48 | } 49 | } else { 50 | return make([]byte, initSize, capacity) 51 | } 52 | 53 | // 将所有的Slice状态(length)reset 54 | return initSlice(result, initSize) 55 | } 56 | 57 | // 58 | // slice s 和某个标准(1024/2048)大小的 slice 的起始地址相同, cap相同，但是len不一样 59 | // initSlice将从s出发，创建一个len一样的slice 60 | // 61 | func initSlice(s []byte, initSize int) []byte { 62 | if len(s) == initSize { 63 | return s 64 | } else { 65 | var b []byte 66 | h := (*reflect.SliceHeader)(unsafe.Pointer(&b)) 67 | h.Data = (*reflect.SliceHeader)(unsafe.Pointer(&s)).Data 68 | h.Len = initSize 69 | h.Cap = cap(s) 70 | return b 71 | } 72 | } 73 | 74 | func getSliceId(s []byte) uintptr { 75 | return (*reflect.SliceHeader)(unsafe.Pointer(&s)).Data 76 | } 77 | 78 | // 79 | // 自己管理内存：释放 80 | // 1. 同一个slice不要归还多次 81 | // 2. 经过slice处理之后的slice不要归还，例如: v = v[3:4], 82 | // 83 | func returnSlice(slice []byte) bool { 84 | if cap(slice) == 1024 { 85 | select { 86 | case memoryBuffer1024 <- slice: 87 | return true 88 | default: 89 | // DO NOTHING 90 | } 91 | } else if cap(slice) == 2048 { 92 | select { 93 | case memoryBuffer2048 <- slice: 94 | return true 95 | default: 96 | // DO NOTHING 97 | 98 | } 99 | } 100 | return false 101 | } 102 | -------------------------------------------------------------------------------- /src/proxy/memory_alloc_test.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "fmt" 7 | "github.com/stretchr/testify/assert" 8 | "testing" 9 | ) 10 | 11 | // 12 | // go test proxy -v -run "TestMemoryAlloc" 13 | // 14 | func TestMemoryAlloc(t *testing.T) { 15 | v1 := getSlice(100, 100) 16 | assert.True(t, len(v1) == 100, cap(v1) == 1024) 17 | 18 | v2 := v1[4:5] 19 | 20 | // Slice一旦操作之后，其记录的信息就不完整了, 根据v2无法获取v1的信息 21 | fmt.Printf("V1: %d, %d, V2: %d, %d\n", len(v1), cap(v1), len(v2), cap(v2)) 22 | assert.False(t, returnSlice(v2)) 23 | assert.True(t, returnSlice(v1)) 24 | 25 | // 同一个slice不要归还多次 26 | 27 | } 28 | -------------------------------------------------------------------------------- /src/proxy/memory_buffer.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "bytes" 7 | ) 8 | 9 | // Memory buffer-based implementation of the TTransport interface. 10 | type TMemoryBuffer struct { 11 | // *bytes.Buffer 和 bytes.Buffer不一样， 使用起来一样，但是: 前者似乎可以定制，可以指定buffer的初始大小 12 | *bytes.Buffer 13 | size int 14 | } 15 | 16 | func NewTMemoryBuffer() *TMemoryBuffer { 17 | return &TMemoryBuffer{Buffer: &bytes.Buffer{}, size: 0} 18 | } 19 | 20 | func NewTMemoryBufferLen(size int) *TMemoryBuffer { 21 | buf := make([]byte, 0, size) 22 | return &TMemoryBuffer{Buffer: bytes.NewBuffer(buf), size: size} 23 | } 24 | 25 | // 26 | // 直接利用现成的Buffer, 由于size不能控制，因此只能拷贝一份出来 27 | // 28 | func NewTMemoryBufferWithBuf(buf []byte) *TMemoryBuffer { 29 | // buf作为bytes.Buffer的已有的数据，因此需要特别注意 30 | return &TMemoryBuffer{Buffer: bytes.NewBuffer(buf), size: len(buf)} 31 | } 32 | 33 | func (p *TMemoryBuffer) IsOpen() bool { 34 | return true 35 | } 36 | 37 | func (p *TMemoryBuffer) Open() error { 38 | return nil 39 | } 40 | 41 | func (p *TMemoryBuffer) Close() error { 42 | p.Buffer.Reset() 43 | return nil 44 | } 45 | 46 | // Flushing a memory buffer is a no-op 47 | func (p *TMemoryBuffer) Flush() error { 48 | return nil 49 | } 50 | 51 | 52 | func (p *TMemoryBuffer) RemainingBytes() (num_bytes uint64) { 53 | return uint64(p.Buffer.Len()) 54 | } 55 | -------------------------------------------------------------------------------- /src/proxy/memory_buffer_test.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "bytes" 7 | "github.com/stretchr/testify/assert" 8 | "reflect" 9 | "testing" 10 | "unsafe" 11 | ) 12 | 13 | // 14 | // go test proxy -v -run "TestMemoryBuffer" 15 | // 16 | func TestMemoryBuffer(t *testing.T) { 17 | 18 | var buf *TMemoryBuffer 19 | 20 | // Case 1: 21 | buf = NewTMemoryBuffer() 22 | // 长度 23 | assert.Equal(t, 0, buf.Buffer.Len()) 24 | assert.Equal(t, 0, buf.Len()) 25 | 26 | buf.Write([]byte("hello")) 27 | buf.Write([]byte(" world")) 28 | 29 | // Bytes()的用法 30 | assert.True(t, bytes.Equal(buf.Bytes(), []byte("hello world"))) 31 | 32 | // Case 2: 33 | // 这个size只是给了buf一个默认的cap, 除此之外没有其他的用处 34 | buf = NewTMemoryBufferLen(10) 35 | buf.Write([]byte("1234567890abcdefg")) 36 | assert.Equal(t, len("1234567890abcdefg"), buf.Len()) 37 | assert.Equal(t, len("1234567890abcdefg"), buf.Buffer.Len()) 38 | 39 | // Case 3.1 40 | byteBuf := make([]byte, 10, 10) 41 | buf = NewTMemoryBufferWithBuf(byteBuf[0:0]) // 注意: 0:0的意义 42 | buf.Write([]byte("1234567890")) 43 | 44 | // buf没有resize, 则byteBuf和buf中的buffer一致 45 | assert.True(t, bytes.Equal(byteBuf, buf.Bytes())) 46 | 47 | // 如果slice大小不够，则会重新创建，从而到只: byteBuf和buf中的buffer不一样 48 | buf.Write([]byte("123456789012")) 49 | assert.False(t, bytes.Equal(byteBuf, buf.Bytes())) 50 | 51 | // Case 3.2 52 | byteBuf = make([]byte, 11, 30) 53 | copy(byteBuf, []byte("hello world")) 54 | 55 | buf = NewTMemoryBufferWithBuf(byteBuf) 56 | buf.Write([]byte("Hi")) 57 | 58 | // 因为slice的长度限制了byteBuf 59 | assert.NotEqual(t, "hello worldHi", string(byteBuf)) 60 | // 潜在的slice是相同的 61 | assert.Equal(t, "hello worldHi", string(copySlice(byteBuf, buf.Len()))) 62 | 63 | } 64 | 65 | func copySlice(s []byte, sl int) []byte { 66 | var b []byte 67 | 68 | h := (*reflect.SliceHeader)(unsafe.Pointer(&b)) 69 | h.Data = (*reflect.SliceHeader)(unsafe.Pointer(&s)).Data 70 | h.Len = sl 71 | h.Cap = len(s) 72 | 73 | return b 74 | } 75 | -------------------------------------------------------------------------------- /src/proxy/request.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "encoding/binary" 7 | "errors" 8 | "fmt" 9 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 10 | "github.com/wfxiang08/go_thrift/thrift" 11 | "io" 12 | "strings" 13 | "sync" 14 | ) 15 | 16 | type Dispatcher interface { 17 | Dispatch(r *Request) error 18 | } 19 | 20 | const ( 21 | // 不能使用负数 22 | MESSAGE_TYPE_HEART_BEAT thrift.TMessageType = 20 23 | MESSAGE_TYPE_STOP thrift.TMessageType = 21 24 | MESSAGE_TYPE_STOP_CONFIRM thrift.TMessageType = 22 25 | ) 26 | 27 | type Request struct { 28 | Service string // 服务 29 | ProxyRequest bool // Service是否出现在Request.Data中，默认为true, 但是心跳等信号中没有service 30 | 31 | // 原始的数据(虽然拷贝有点点效率低，但是和zeromq相比也差不多) 32 | Request struct { 33 | Name string 34 | TypeId thrift.TMessageType 35 | SeqId int32 36 | Data []byte 37 | DataOrig []byte 38 | } 39 | 40 | Start int64 41 | 42 | // 返回的数据类型 43 | Response struct { 44 | Data []byte 45 | Err error 46 | SeqId int32 // -1保留，表示没有对应的SeqNum 47 | TypeId thrift.TMessageType 48 | } 49 | 50 | Wait sync.WaitGroup 51 | } 52 | 53 | // 54 | // 给定一个thrift message，构建一个Request对象 55 | // 56 | func NewRequest(data []byte, serviceInReq bool) (*Request, error) { 57 | request := &Request{ 58 | ProxyRequest: serviceInReq, 59 | Start: microseconds(), 60 | } 61 | request.Request.Data = data 62 | err := request.DecodeRequest() 63 | 64 | if err != nil { 65 | return nil, err 66 | } else { 67 | return request, nil 68 | } 69 | 70 | } 71 | 72 | // 73 | // 利用自身信息生成 timeout Error(注意: 其中的SeqId必须为r.Request.SeqId(它不是从后台返回，不会进行SeqId的replace) 74 | // 75 | func (r *Request) NewTimeoutError() error { 76 | return errors.New(fmt.Sprintf("Timeout Exception, %s.%s.%d", 77 | r.Service, r.Request.Name, r.Request.SeqId)) 78 | } 79 | 80 | func (r *Request) NewInvalidResponseError(method string, module string) error { 81 | return errors.New(fmt.Sprintf("[%s]Invalid Response Exception, %s.%s.%d, Method Ret: %s", 82 | module, r.Service, r.Request.Name, r.Request.SeqId, method)) 83 | } 84 | 85 | // 86 | // 从Request.Data中读取出 Request的Name, TypeId, SeqId 87 | // RequestName可能和thrift package中的name不一致，Service部分从Name中剔除 88 | // 89 | func (r *Request) DecodeRequest() error { 90 | var err error 91 | transport := NewTMemoryBufferWithBuf(r.Request.Data) 92 | protocol := thrift.NewTBinaryProtocolTransport(transport) 93 | 94 | // 如果数据异常，则直接报错 95 | r.Request.Name, r.Request.TypeId, r.Request.SeqId, err = protocol.ReadMessageBegin() 96 | if err != nil { 97 | return err 98 | } 99 | 100 | // 参考 ： TMultiplexedProtocol 101 | idx := strings.Index(r.Request.Name, thrift.MULTIPLEXED_SEPARATOR) 102 | if idx == -1 { 103 | r.Service = "" 104 | } else { 105 | r.Service = r.Request.Name[0:idx] 106 | r.Request.Name = r.Request.Name[idx+1 : len(r.Request.Name)] 107 | } 108 | return nil 109 | } 110 | 111 | // 112 | // 将Request中的SeqNum进行替换（修改Request部分的数据) 113 | // 114 | func (r *Request) ReplaceSeqId(newSeq int32) { 115 | if r.Request.Data != nil { 116 | // log.Printf(Green("Replace SeqNum: %d --> %d"), r.Request.SeqId, newSeq) 117 | if r.Response.SeqId != 0 { 118 | log.Errorf("Unexpected Response SedId") 119 | } 120 | r.Response.SeqId = newSeq 121 | 122 | start := 0 123 | 124 | if r.ProxyRequest { 125 | start = len(r.Service) 126 | } 127 | if start > 0 { 128 | start += 1 // ":" 129 | // log.Printf("Service: %s, Name: %s\n", r.Service, r.Request.Name) 130 | } 131 | transport := NewTMemoryBufferWithBuf(r.Request.Data[start:start]) 132 | protocol := thrift.NewTBinaryProtocolTransport(transport) 133 | protocol.WriteMessageBegin(r.Request.Name, r.Request.TypeId, newSeq) 134 | 135 | if start > 0 { 136 | r.Request.DataOrig = r.Request.Data 137 | } 138 | // 将service从name中剥离出去 139 | r.Request.Data = r.Request.Data[start:len(r.Request.Data)] 140 | 141 | } else { 142 | log.Errorf("ReplaceSeqId called on processed Data") 143 | } 144 | } 145 | 146 | func (r *Request) Recycle() { 147 | var sliceId uintptr = 0 148 | // 将其中的Buffer归还(returnSlice) 149 | if r.Request.DataOrig != nil { 150 | returnSlice(r.Request.DataOrig) 151 | sliceId = getSliceId(r.Request.DataOrig) 152 | 153 | r.Request.DataOrig = nil 154 | r.Request.Data = nil 155 | } else if r.Request.Data != nil { 156 | sliceId = getSliceId(r.Request.Data) 157 | returnSlice(r.Request.Data) 158 | r.Request.Data = nil 159 | } 160 | if r.Response.Data != nil { 161 | if sliceId != getSliceId(r.Response.Data) { 162 | returnSlice(r.Response.Data) 163 | } 164 | r.Response.Data = nil 165 | } 166 | } 167 | 168 | func (r *Request) RestoreSeqId() { 169 | if r.Response.Data != nil { 170 | // e = p.WriteI32(seqId) 171 | // i32 + (i32 + len(str)) + i32[SeqId] 172 | // 直接按照TBinaryProtocol协议，修改指定位置的数据: SeqId 173 | startIdx := 4 + 4 + len(r.Request.Name) 174 | v := r.Response.Data[startIdx : startIdx+4] 175 | binary.BigEndian.PutUint32(v, uint32(r.Request.SeqId)) 176 | 177 | // transport := NewTMemoryBufferWithBuf(r.Response.Data[0:0]) 178 | // protocol := thrift.NewTBinaryProtocolTransport(transport) 179 | 180 | // // 切换回原始的SeqId 181 | // // r.Response.TypeId 和 r.Request.TypeId可能不一样，要以Response为准 182 | // protocol.WriteMessageBegin(r.Request.Name, r.Response.TypeId, r.Request.SeqId) 183 | } 184 | } 185 | 186 | // 187 | // 给定thrift Message, 解码出: typeId, seqId 188 | // 189 | func DecodeThriftTypIdSeqId(data []byte) (typeId thrift.TMessageType, method string, seqId int32, err error) { 190 | 191 | // 解码typeId 192 | if len(data) < 4 { 193 | err = thrift.NewTProtocolException(io.ErrUnexpectedEOF) 194 | return 195 | } 196 | size := int32(binary.BigEndian.Uint32(data[0:4])) 197 | typeId = thrift.TMessageType(size & 0x0ff) 198 | 199 | // 解码name的长度，并且跳过name 200 | if len(data) < 8 { 201 | err = thrift.NewTProtocolException(io.ErrUnexpectedEOF) 202 | return 203 | } 204 | size = int32(binary.BigEndian.Uint32(data[4:8])) 205 | if len(data) < 12+int(size) { 206 | err = thrift.NewTProtocolException(io.ErrUnexpectedEOF) 207 | return 208 | } 209 | 210 | method = string(data[8 : 8+size]) 211 | // 解码seqId 212 | seqId = int32(binary.BigEndian.Uint32(data[8+size : 12+size])) 213 | 214 | // transport := NewTMemoryBufferWithBuf(data) 215 | // protocol := thrift.NewTBinaryProtocolTransport(transport) 216 | 217 | // _, typeId, seqId, err = protocol.ReadMessageBegin() 218 | return 219 | } 220 | -------------------------------------------------------------------------------- /src/proxy/request_ordered_map.go: -------------------------------------------------------------------------------- 1 | // This package provides a simple LRU cache. It is based on the 2 | // LRU implementation in groupcache: 3 | // https://github.com/golang/groupcache/tree/master/lru 4 | package proxy 5 | 6 | import ( 7 | "container/list" 8 | "errors" 9 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 10 | "sync" 11 | ) 12 | 13 | // 14 | // 特性: 15 | // 1. 限定大小 16 | // 2. 查看最后添加的元素(以便evict) 17 | // 3. 元素按照添加的时间排序 18 | // 19 | // 代码设计原则: 20 | // 大写开头的函数对外公开，负责加锁等；小写字母开头的函数，只负责完成逻辑，不考虑锁 21 | // 22 | type RequestMap struct { 23 | size int 24 | evictList *list.List 25 | items map[int32]*list.Element 26 | lock sync.RWMutex 27 | } 28 | 29 | // 列表中的元素类型 30 | type Entry struct { 31 | key int32 32 | value *Request 33 | } 34 | 35 | func NewRequestMap(size int) (*RequestMap, error) { 36 | if size <= 0 { 37 | return nil, errors.New("Must provide a positive size") 38 | } 39 | c := &RequestMap{ 40 | size: size, 41 | evictList: list.New(), 42 | items: make(map[int32]*list.Element, size), 43 | } 44 | return c, nil 45 | } 46 | 47 | // 返回所有元素，重新初始化List/Map 48 | func (c *RequestMap) Purge() []*Request { 49 | c.lock.Lock() 50 | 51 | // 拷贝剩余的Requests 52 | results := make([]*Request, 0, len(c.items)) 53 | for _, element := range c.items { 54 | results = append(results, element.Value.(*Entry).value) 55 | } 56 | 57 | // 重新初始化 58 | c.evictList = list.New() 59 | c.items = make(map[int32]*list.Element, c.size) 60 | 61 | c.lock.Unlock() 62 | 63 | return results 64 | } 65 | 66 | // 67 | // 添加新的key, value 68 | // 69 | func (c *RequestMap) Add(key int32, value *Request) bool { 70 | c.lock.Lock() 71 | 72 | // 如果key存在，则覆盖之前的元素；并添加Warning 73 | if ent, ok := c.items[key]; ok { 74 | c.evictList.MoveToFront(ent) 75 | ent.Value.(*Entry).value = value 76 | log.Errorf(Red("Duplicated Key Found in RequestOrderedMap: %d"), key) 77 | 78 | c.lock.Unlock() 79 | return false 80 | } 81 | 82 | // Add new item 83 | ent := &Entry{key, value} 84 | entry := c.evictList.PushFront(ent) 85 | c.items[key] = entry 86 | 87 | // 如果超过指定的大小，则清除元素 88 | evict := c.evictList.Len() > c.size 89 | if evict { 90 | c.removeOldest() 91 | } 92 | c.lock.Unlock() 93 | return evict 94 | } 95 | 96 | // 97 | // 读取Key, 并将它从Map中删除 98 | // 99 | func (c *RequestMap) Pop(key int32) *Request { 100 | c.lock.Lock() 101 | 102 | var result *Request = nil 103 | if ent, ok := c.items[key]; ok { 104 | c.removeElement(ent) 105 | result = ent.Value.(*Entry).value 106 | } 107 | 108 | c.lock.Unlock() 109 | return result 110 | } 111 | 112 | // 113 | // 读取Key, 不调整元素的顺序 114 | // 115 | func (c *RequestMap) Get(key int32) (value *Request, ok bool) { 116 | c.lock.RLock() 117 | 118 | if ent, ok := c.items[key]; ok { 119 | value, ok = ent.Value.(*Entry).value, true 120 | } 121 | c.lock.RUnlock() 122 | return 123 | } 124 | 125 | func (c *RequestMap) Contains(key int32) (ok bool) { 126 | c.lock.RLock() 127 | _, ok = c.items[key] 128 | c.lock.RUnlock() 129 | return ok 130 | } 131 | 132 | // 133 | // 删除指定的Key， 返回是否删除OK 134 | // 135 | func (c *RequestMap) Remove(key int32) bool { 136 | c.lock.Lock() 137 | 138 | result := false 139 | if ent, ok := c.items[key]; ok { 140 | c.removeElement(ent) 141 | result = true 142 | } 143 | c.lock.Unlock() 144 | return result 145 | } 146 | 147 | // 148 | // RemoveOldest removes the oldest item from the cache. 149 | // 150 | func (c *RequestMap) RemoveOldest() { 151 | c.lock.Lock() 152 | c.removeOldest() 153 | c.lock.Unlock() 154 | } 155 | 156 | // 157 | // 按照从旧到新的顺序返回 Keys的列表 158 | // 159 | func (c *RequestMap) Keys() []int32 { 160 | c.lock.RLock() 161 | 162 | keys := make([]int32, len(c.items)) 163 | ent := c.evictList.Back() 164 | i := 0 165 | for ent != nil { 166 | keys[i] = ent.Value.(*Entry).key 167 | ent = ent.Prev() 168 | i++ 169 | } 170 | c.lock.RUnlock() 171 | return keys 172 | } 173 | 174 | // 175 | // 获取当前的元素个数 176 | // 177 | func (c *RequestMap) Len() int { 178 | c.lock.RLock() 179 | result := c.evictList.Len() 180 | c.lock.RUnlock() 181 | return result 182 | } 183 | 184 | // 185 | // 清除过期的Request 186 | // 187 | func (c *RequestMap) RemoveExpired(expiredInMicro int64) { 188 | c.lock.Lock() 189 | 190 | for true { 191 | ent := c.evictList.Back() 192 | 193 | // 如果Map为空，则返回 194 | if ent == nil { 195 | break 196 | } 197 | 198 | // 如果请求还没有过期，则不再返回 199 | entry := ent.Value.(*Entry) 200 | request := entry.value 201 | if request.Start > expiredInMicro { 202 | break 203 | } 204 | 205 | // 1. 准备删除当前的元素 206 | c.removeElement(ent) 207 | 208 | // 2. 如果出问题了，则打印原始的请求的数据 209 | log.Warnf(Red("Remove Expired Request: %s.%s [%d]"), 210 | request.Service, request.Request.Name, request.Response.SeqId) 211 | 212 | // 3. 处理Request 213 | request.Response.Err = request.NewTimeoutError() 214 | request.Wait.Done() 215 | } 216 | 217 | c.lock.Unlock() 218 | } 219 | 220 | // 221 | // 读取最旧的元素 222 | // 223 | func (c *RequestMap) PeekOldest() (key int32, value *Request, ok bool) { 224 | c.lock.RLock() 225 | ent := c.evictList.Back() 226 | 227 | if ent != nil { 228 | entry := ent.Value.(*Entry) 229 | key, value, ok = entry.key, entry.value, true 230 | } else { 231 | key, value, ok = 0, nil, false 232 | } 233 | 234 | c.lock.RUnlock() 235 | return 236 | } 237 | 238 | // 删除最旧的元素 239 | func (c *RequestMap) removeOldest() { 240 | ent := c.evictList.Back() 241 | if ent != nil { 242 | c.removeElement(ent) 243 | } 244 | } 245 | 246 | // 删除指定的元素(参数: list.Element) 247 | func (c *RequestMap) removeElement(e *list.Element) { 248 | 249 | c.evictList.Remove(e) 250 | kv := e.Value.(*Entry) 251 | 252 | // log.Printf("Remove Element: %s, With key: %d", kv.value.Request.Name, kv.key) 253 | delete(c.items, kv.key) 254 | } 255 | -------------------------------------------------------------------------------- /src/proxy/request_ordered_map_test.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "fmt" 7 | "github.com/stretchr/testify/assert" 8 | "testing" 9 | ) 10 | 11 | // 12 | // go test proxy -v -run "TestRequestMap" 13 | // 14 | func TestRequestMap(t *testing.T) { 15 | 16 | requestMap, _ := NewRequestMap(20) 17 | 18 | var i int32 19 | for i = 0; i < 10; i++ { 20 | request := NewPingRequest() 21 | request.Start += int64(1000000 * 5 * (i - 5)) 22 | request.Response.SeqId = i 23 | 24 | requestMap.Add(request.Response.SeqId, request) 25 | } 26 | 27 | assert.Equal(t, 10, requestMap.Len()) 28 | 29 | sid, request, ok := requestMap.PeekOldest() 30 | fmt.Printf("Request: %s\n, Sid: %d\n", request.Request.Name, request.Response.SeqId) 31 | assert.Equal(t, int32(0), int32(sid)) 32 | assert.True(t, ok) 33 | 34 | expired := microseconds() 35 | for true { 36 | sid, request, ok := requestMap.PeekOldest() 37 | if ok && request.Start < expired { 38 | requestMap.Remove(sid) 39 | } else { 40 | break 41 | } 42 | } 43 | 44 | assert.Equal(t, 4, requestMap.Len()) 45 | 46 | requests := requestMap.Purge() 47 | assert.Equal(t, 4, len(requests)) 48 | 49 | for _, request := range requests { 50 | fmt.Printf("Request: %s\n, Sid: %d\n", request.Request.Name, request.Response.SeqId) 51 | } 52 | } 53 | -------------------------------------------------------------------------------- /src/proxy/request_test.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "fmt" 7 | thrift "github.com/wfxiang08/go_thrift/thrift" 8 | "github.com/stretchr/testify/assert" 9 | "testing" 10 | ) 11 | 12 | func fakeData(name string, typeId thrift.TMessageType, seqId int32, buf []byte) int { 13 | transport := NewTMemoryBufferWithBuf(buf) 14 | protocol := thrift.NewTBinaryProtocolTransport(transport) 15 | 16 | // 切换回原始的SeqId 17 | protocol.WriteMessageBegin(name, typeId, seqId) 18 | return transport.Buffer.Len() 19 | 20 | } 21 | 22 | // 23 | // go test proxy -v -run "TestRequest" 24 | // 25 | func TestRequest(t *testing.T) { 26 | 27 | a := make([]byte, 1000) 28 | c := a 29 | d := a 30 | var e []byte 31 | fmt.Println("E: ", getSliceId(e), "A: ", getSliceId(c)) 32 | assert.True(t, getSliceId(c) == getSliceId(d)) 33 | 34 | data := make([]byte, 1000, 1000) 35 | size := fakeData("demo:hello", thrift.CALL, 0, data[0:0]) 36 | data = data[0:size] 37 | 38 | r, _ := NewRequest(data, true) 39 | 40 | assert.Equal(t, "demo", r.Service) 41 | assert.Equal(t, "hello", r.Request.Name) 42 | 43 | // fmt.Printf("Name: %s, SeqId: %d, TypeId: %d\n", r.Request.Name, 44 | // r.Request.SeqId, r.Request.TypeId) 45 | 46 | var newSeqId int32 = 10 47 | r.ReplaceSeqId(newSeqId) 48 | 49 | _, _, seqId1, _ := DecodeThriftTypIdSeqId(r.Request.Data) 50 | assert.Equal(t, newSeqId, seqId1) // r.Request.Data中的id被替换成功 51 | 52 | r.Response.Data = r.Request.Data 53 | r.RestoreSeqId() 54 | 55 | // 恢复正常 56 | _, _, seqId2, _ := DecodeThriftTypIdSeqId(r.Response.Data) 57 | 58 | // fmt.Printf("Reqeust SeqId: %d, %d\n", r.Request.SeqId, seqId2) 59 | assert.Equal(t, 0, int(seqId2)) 60 | 61 | } 62 | -------------------------------------------------------------------------------- /src/proxy/router_proxy.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | 4 | package proxy 5 | 6 | import ( 7 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 8 | "sync" 9 | "time" 10 | ) 11 | 12 | type Router struct { 13 | productName string 14 | 15 | // 只用于保护: services 16 | serviceLock sync.RWMutex 17 | services map[string]*BackService 18 | 19 | topo *Topology 20 | verbose bool 21 | } 22 | 23 | func NewRouter(productName string, topo *Topology, verbose bool) *Router { 24 | r := &Router{ 25 | productName: productName, 26 | services: make(map[string]*BackService), 27 | topo: topo, 28 | verbose: verbose, 29 | } 30 | 31 | // 监控服务的变化 32 | r.WatchServices() 33 | 34 | return r 35 | } 36 | 37 | // 38 | // 后端如何处理一个Request 39 | // 40 | func (s *Router) Dispatch(r *Request) error { 41 | backService := s.GetBackService(r.Service) 42 | if backService == nil { 43 | log.Printf(Cyan("Service Not Found for: %s.%s\n"), r.Service, r.Request.Name) 44 | r.Response.Data = GetServiceNotFoundData(r) 45 | return nil 46 | } else { 47 | return backService.HandleRequest(r) 48 | } 49 | } 50 | 51 | // Router负责监听zk中服务列表的变化 52 | func (bk *Router) WatchServices() { 53 | var evtbus chan interface{} = make(chan interface{}, 2) 54 | 55 | // 1. 保证Service目录存在，否则会报错 56 | servicesPath := bk.topo.ProductServicesPath() 57 | _, e1 := bk.topo.CreateDir(servicesPath) 58 | if e1 != nil { 59 | log.PanicErrorf(e1, "Zk Path Create Failed: %s", servicesPath) 60 | } 61 | 62 | go func() { 63 | for true { 64 | // 无限监听 65 | services, err := bk.topo.WatchChildren(servicesPath, evtbus) 66 | 67 | if err == nil { 68 | bk.serviceLock.Lock() 69 | // 保证数据更新是有效的 70 | oldServices := bk.services 71 | bk.services = make(map[string]*BackService, len(services)) 72 | for _, service := range services { 73 | log.Println("Found Service: ", Magenta(service)) 74 | 75 | back, ok := oldServices[service] 76 | if ok { 77 | bk.services[service] = back 78 | delete(oldServices, service) 79 | } else { 80 | 81 | bk.addBackService(service) 82 | } 83 | } 84 | if len(oldServices) > 0 { 85 | for _, conn := range oldServices { 86 | // 标记下线(现在应该不会有新的请求，最多只会处理一些收尾的工作 87 | conn.Stop() 88 | } 89 | 90 | } 91 | 92 | bk.serviceLock.Unlock() 93 | 94 | // 等待事件 95 | <-evtbus 96 | } else { 97 | log.ErrorErrorf(err, "zk watch error: %s, error: %v\n", 98 | servicesPath, err) 99 | time.Sleep(time.Duration(5) * time.Second) 100 | } 101 | } 102 | }() 103 | 104 | // 读取zk, 等待 105 | log.Println("ProductName: ", Magenta(bk.topo.ProductName)) 106 | } 107 | 108 | // 添加一个后台服务(非线程安全) 109 | func (bk *Router) addBackService(service string) { 110 | 111 | backService, ok := bk.services[service] 112 | if !ok { 113 | backService = NewBackService(bk.productName, service, bk.topo, bk.verbose) 114 | bk.services[service] = backService 115 | } 116 | 117 | } 118 | func (bk *Router) GetBackService(service string) *BackService { 119 | bk.serviceLock.RLock() 120 | backService, ok := bk.services[service] 121 | bk.serviceLock.RUnlock() 122 | 123 | if ok { 124 | return backService 125 | } else { 126 | return nil 127 | } 128 | } 129 | -------------------------------------------------------------------------------- /src/proxy/rpc_command_line.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "flag" 7 | "fmt" 8 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 9 | "net/http" 10 | _ "net/http/pprof" 11 | "os" 12 | ) 13 | 14 | //-flag 15 | //-flag=x 16 | //-flag x 17 | var ( 18 | configFile = flag.String("c", "", "config file") 19 | version = flag.Bool("V", false, "code-url-version") 20 | logFile = flag.String("L", "", "set output log file, default is stdout") 21 | logLevel = flag.String("log_level", "", "work-dir") 22 | profileAddr = flag.String("profile_address", "", "profile address") 23 | workDirFlag = flag.String("work_dir", "", "work-dir") 24 | codeUrlVersion = flag.String("code_url_version", "", "code-url-version") 25 | ) 26 | 27 | func RpcMain(binaryName string, serviceDesc string, configCheck ConfigCheck, 28 | serverFactory ServerFactorory, buildDate string, gitVersion string) { 29 | 30 | flag.Parse() 31 | 32 | // 1. 准备解析参数 33 | if *version { 34 | versionStr := fmt.Sprintf("Version: %s\nBuildDate: %s\nDesc: %s\nAuthor: wfxiang08@gmail.com", gitVersion, buildDate, serviceDesc) 35 | fmt.Println(Green(versionStr)) 36 | os.Exit(1) 37 | } 38 | 39 | // 这就是为什么 Codis 傻乎乎起一个 http server的目的 40 | if len(*profileAddr) > 0 { 41 | go func() { 42 | log.Printf(Red("Profile Address: %s"), *profileAddr) 43 | log.Println(http.ListenAndServe(*profileAddr, nil)) 44 | }() 45 | } 46 | 47 | // 2. 解析Log相关的配置 48 | log.SetLevel(log.LEVEL_INFO) 49 | 50 | var maxKeepDays int = 3 51 | 52 | // set output log file 53 | if len(*logFile) > 0 { 54 | f, err := log.NewRollingFile(*logFile, maxKeepDays) 55 | 56 | if err != nil { 57 | log.PanicErrorf(err, "open rolling log file failed: %s", *logFile) 58 | } else { 59 | defer f.Close() 60 | log.StdLog = log.New(f, "") 61 | } 62 | } 63 | log.SetLevel(log.LEVEL_INFO) 64 | log.SetFlags(log.Flags() | log.Lshortfile) 65 | 66 | // set log level 67 | if len(*logLevel) > 0 { 68 | SetLogLevel(*logLevel) 69 | } 70 | 71 | // 没有就没有 72 | var workDir string 73 | if len(*workDirFlag) == 0 { 74 | workDir, _ = os.Getwd() 75 | } else { 76 | workDir = *workDirFlag 77 | } 78 | 79 | log.Printf("WorkDir: %s, CodeUrl: %s", workDir, codeUrlVersion) 80 | 81 | // 3. 解析Config 82 | if len(*configFile) == 0 { 83 | log.Panicf("Config file not specified") 84 | } 85 | conf, err := LoadConf(*configFile) 86 | if err != nil { 87 | log.PanicErrorf(err, "load config failed") 88 | } 89 | 90 | // 额外的配置信息 91 | conf.WorkDir = workDir 92 | conf.CodeUrlVersion = *codeUrlVersion 93 | 94 | if configCheck != nil { 95 | configCheck(conf) 96 | } else { 97 | log.Panic("No Config Check Given") 98 | } 99 | // 每次启动的时候都打印版本信息 100 | log.Infof(Green("-----------------\n%s\n--------------------------------------------------------------------"), version) 101 | 102 | // 启动服务 103 | server := serverFactory(conf) 104 | server.Run() 105 | } 106 | 107 | func RpcMainProxy(binaryName string, serviceDesc string, configCheck ProxyConfigCheck, 108 | proxyFactory ProxyFactorory, buildDate string, gitVersion string) { 109 | 110 | flag.Parse() 111 | 112 | // 1. 准备解析参数 113 | if *version { 114 | versionStr := fmt.Sprintf("Version: %s\nBuildDate: %s\nDesc: %s\nAuthor: wfxiang08@gmail.com", gitVersion, buildDate, serviceDesc) 115 | fmt.Println(Green(versionStr)) 116 | os.Exit(1) 117 | } 118 | 119 | // 这就是为什么 Codis 傻乎乎起一个 http server的目的 120 | if len(*profileAddr) > 0 { 121 | go func() { 122 | log.Printf(Red("Profile Address: %s"), *profileAddr) 123 | log.Println(http.ListenAndServe(*profileAddr, nil)) 124 | }() 125 | } 126 | 127 | // 2. 解析Log相关的配置 128 | log.SetLevel(log.LEVEL_INFO) 129 | 130 | var maxKeepDays int = 3 131 | 132 | // set output log file 133 | if len(*logFile) > 0 { 134 | f, err := log.NewRollingFile(*logFile, maxKeepDays) 135 | 136 | if err != nil { 137 | log.PanicErrorf(err, "open rolling log file failed: %s", *logFile) 138 | } else { 139 | defer f.Close() 140 | log.StdLog = log.New(f, "") 141 | } 142 | } 143 | log.SetLevel(log.LEVEL_INFO) 144 | log.SetFlags(log.Flags() | log.Lshortfile) 145 | 146 | // set log level 147 | if len(*logLevel) > 0 { 148 | SetLogLevel(*logLevel) 149 | } 150 | 151 | // 没有就没有 152 | var workDir string 153 | if len(*workDirFlag) == 0 { 154 | workDir, _ = os.Getwd() 155 | } else { 156 | workDir = *workDirFlag 157 | } 158 | 159 | log.Printf("WorkDir: %s, CodeUrl: %s", workDir, codeUrlVersion) 160 | 161 | // 3. 解析Config 162 | if len(*configFile) == 0 { 163 | log.Panicf("Config file not specified") 164 | } 165 | conf, err := LoadProxyConf(*configFile) 166 | if err != nil { 167 | log.PanicErrorf(err, "load config failed") 168 | } 169 | 170 | if configCheck != nil { 171 | configCheck(conf) 172 | } else { 173 | log.Panic("No Config Check Given") 174 | } 175 | // 每次启动的时候都打印版本信息 176 | log.Infof(Green("-----------------\n%s\n--------------------------------------------------------------------"), version) 177 | 178 | // 启动服务 179 | server := proxyFactory(conf) 180 | server.Run() 181 | } 182 | -------------------------------------------------------------------------------- /src/proxy/server_general_rpc.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "fmt" 7 | "github.com/wfxiang08/cyutils/utils/atomic2" 8 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 9 | topozk "github.com/wfxiang08/go-zookeeper/zk" 10 | "github.com/wfxiang08/go_thrift/thrift" 11 | "github.com/wfxiang08/thrift_rpc_base/rpc_utils" 12 | "os" 13 | "os/signal" 14 | "runtime/debug" 15 | "strings" 16 | "syscall" 17 | "time" 18 | "context" 19 | ) 20 | 21 | type Server interface { 22 | Run() 23 | } 24 | type ServerFactorory func(config *ServiceConfig) Server 25 | type ProxyFactorory func(config *ProxyConfig) Server 26 | 27 | // 28 | // Thrift Server的参数 29 | // 30 | type ThriftRpcServer struct { 31 | ZkAddr string 32 | ProductName string 33 | ServiceName string 34 | FrontendAddr string 35 | Topo *Topology 36 | Processor thrift.TProcessor 37 | Verbose bool 38 | lastRequestTime atomic2.Int64 39 | config *ServiceConfig 40 | } 41 | 42 | func NewThriftRpcServer(config *ServiceConfig, processor thrift.TProcessor) *ThriftRpcServer { 43 | log.Printf("FrontAddr: %s\n", Magenta(config.FrontendAddr)) 44 | 45 | return &ThriftRpcServer{ 46 | config: config, 47 | ZkAddr: config.ZkAddr, 48 | ProductName: config.ProductName, 49 | ServiceName: config.Service, 50 | FrontendAddr: config.FrontendAddr, 51 | Verbose: config.Verbose, 52 | Processor: processor, 53 | } 54 | 55 | } 56 | 57 | // 58 | // 根据前端的地址生成服务的id 59 | // 例如: 127.0.0.1:5555 --> 127_0_0_1_5555 60 | // "/Users/feiwang/gowork/src/github.com/wfxiang08/rpc_proxyiplocation/aa.sock" 61 | func GetServiceIdentity(frontendAddr string) string { 62 | 63 | fid := strings.Replace(frontendAddr, ".", "_", -1) 64 | fid = strings.Replace(fid, ":", "_", -1) 65 | fid = strings.Replace(fid, "/", "", -1) 66 | if len(fid) > 20 { 67 | // unix domain socket 68 | fid = fid[len(fid) - 20 : len(fid)] 69 | } 70 | return fid 71 | 72 | } 73 | 74 | // 75 | // 去ZK注册当前的Service 76 | // 77 | func RegisterService(serviceName, frontendAddr, serviceId string, topo *Topology, evtExit chan interface{}, 78 | workDir string, codeUrlVerion string, state *atomic2.Bool, stateChan chan bool) *ServiceEndpoint { 79 | 80 | // 1. 准备数据 81 | // 记录Service Endpoint的信息 82 | servicePath := topo.ProductServicePath(serviceName) 83 | 84 | // 确保东西存在 85 | if ok, _ := topo.Exist(servicePath); !ok { 86 | topo.CreateDir(servicePath) 87 | } 88 | 89 | // 用来从zookeeper获取事件 90 | evtbus := make(chan interface{}) 91 | 92 | // 2. 将信息添加到Zk中, 并且监控Zk的状态(如果添加失败会怎么样?) 93 | endpoint := NewServiceEndpoint(serviceName, serviceId, frontendAddr, workDir, codeUrlVerion) 94 | 95 | // deployPath 96 | 97 | // 为了保证Add是干净的，需要先删除，保证自己才是Owner 98 | endpoint.DeleteServiceEndpoint(topo) 99 | 100 | // 如果没有状态，或状态为true, 则上线 101 | if state == nil || state.Get() { 102 | endpoint.AddServiceEndpoint(topo) 103 | } 104 | 105 | go func() { 106 | 107 | for true { 108 | // Watch的目的是监控: 当前的zk session的状态, 如果session出现异常，则重新注册 109 | _, err := topo.WatchNode(servicePath, evtbus) 110 | 111 | if err == nil { 112 | // 如果成功添加Watch, 则等待退出，或者ZK Event 113 | select { 114 | case <-evtExit: 115 | return 116 | case <-stateChan: 117 | // 如何状态变化(则重新注册) 118 | endpoint.DeleteServiceEndpoint(topo) 119 | if state == nil || state.Get() { 120 | endpoint.AddServiceEndpoint(topo) 121 | } 122 | case e := <-evtbus: 123 | event := e.(topozk.Event) 124 | if event.State == topozk.StateExpired || 125 | event.Type == topozk.EventNotWatching { 126 | // Session过期了，则需要删除之前的数据， 127 | // 因为当前的session不是之前的数据的Owner 128 | endpoint.DeleteServiceEndpoint(topo) 129 | 130 | if state == nil || state.Get() { 131 | endpoint.AddServiceEndpoint(topo) 132 | } 133 | 134 | } 135 | } 136 | 137 | } else { 138 | // 如果添加Watch失败，则等待退出，或等待timer接触，进行下一次尝试 139 | timer := time.NewTimer(time.Second * time.Duration(10)) 140 | log.ErrorErrorf(err, "Reg Failed, Wait 10 seconds...., %v", err) 141 | select { 142 | case <-evtExit: 143 | return 144 | case <-timer.C: 145 | // pass 146 | } 147 | } 148 | 149 | } 150 | }() 151 | return endpoint 152 | } 153 | 154 | // 155 | // 后端如何处理一个Request 156 | // 157 | func (p *ThriftRpcServer) Dispatch(r *Request) error { 158 | transport := NewTMemoryBufferWithBuf(r.Request.Data) 159 | ip := thrift.NewTBinaryProtocolTransport(transport) 160 | 161 | slice := getSlice(0, DEFAULT_SLICE_LEN) 162 | transport = NewTMemoryBufferWithBuf(slice) 163 | op := thrift.NewTBinaryProtocolTransport(transport) 164 | 165 | defaultContext := context.Background() 166 | p.Processor.Process(defaultContext, ip, op) 167 | 168 | r.Response.Data = transport.Bytes() 169 | 170 | _, _, seqId, _ := DecodeThriftTypIdSeqId(r.Response.Data) 171 | 172 | log.Debugf("SeqId: %d vs. %d, Dispatch Over", r.Request.SeqId, seqId) 173 | // // 如果transport重新分配了内存，则立即归还slice 174 | // if cap(r.Response.Data) != DEFAULT_SLICE_LEN { 175 | // returnSlice(slice) 176 | // } 177 | return nil 178 | } 179 | 180 | func (p *ThriftRpcServer) Run() { 181 | // // 1. 创建到zk的连接 182 | // 有条件地做服务注册 183 | registerService := !p.config.StandAlone // 不独立运行则注册服务 184 | if registerService { 185 | p.Topo = NewTopology(p.ProductName, p.ZkAddr) 186 | } 187 | 188 | // 127.0.0.1:5555 --> 127_0_0_1:5555 189 | lbServiceName := GetServiceIdentity(p.FrontendAddr) 190 | 191 | exitSignal := make(chan os.Signal, 1) 192 | signal.Notify(exitSignal, syscall.SIGTERM, syscall.SIGINT, syscall.SIGKILL) 193 | 194 | // syscall.SIGKILL 195 | // kill -9 pid 196 | // kill -s SIGKILL pid 还是留给运维吧 197 | 198 | if len(p.config.FalconClient) > 0 { 199 | StartTicker(p.config.FalconClient, p.ServiceName) 200 | } 201 | 202 | // 初始状态为不上线 203 | var state atomic2.Bool 204 | state.Set(false) 205 | stateChan := make(chan bool) 206 | 207 | // 注册服务 208 | evtExit := make(chan interface{}) 209 | 210 | var endpoint *ServiceEndpoint = nil 211 | if registerService { 212 | endpoint = RegisterService(p.ServiceName, p.FrontendAddr, lbServiceName, 213 | p.Topo, evtExit, p.config.WorkDir, p.config.CodeUrlVersion, 214 | &state, stateChan) 215 | } 216 | 217 | // 3. 读取"前端"的配置 218 | var transport thrift.TServerTransport 219 | var err error 220 | 221 | isUnixDomain := false 222 | // 127.0.0.1:9999(以:区分不同的类型) 223 | if !strings.Contains(p.FrontendAddr, ":") { 224 | if rpc_utils.FileExist(p.FrontendAddr) { 225 | os.Remove(p.FrontendAddr) 226 | } 227 | transport, err = rpc_utils.NewTServerUnixDomain(p.FrontendAddr) 228 | isUnixDomain = true 229 | } else { 230 | transport, err = thrift.NewTServerSocket(p.FrontendAddr) 231 | } 232 | 233 | if err != nil { 234 | log.ErrorErrorf(err, Red("Server Socket Create Failed: %v"), err) 235 | panic(fmt.Sprintf("Invalid FrontendAddr: %s", p.FrontendAddr)) 236 | } 237 | 238 | err = transport.Listen() 239 | if err != nil { 240 | log.ErrorErrorf(err, Red("Server Socket Open Failed: %v"), err) 241 | panic(fmt.Sprintf("Server Socket Open Failed: %s", p.FrontendAddr)) 242 | } 243 | 244 | ch := make(chan thrift.TTransport, 4096) 245 | defer close(ch) 246 | 247 | // 强制退出? TODO: Graceful退出 248 | go func() { 249 | <-exitSignal 250 | log.Info(Magenta("Receive Exit Signals....")) 251 | 252 | if registerService { 253 | evtExit <- true 254 | } 255 | if registerService { 256 | endpoint.DeleteServiceEndpoint(p.Topo) 257 | } else { 258 | // 如果没有注册到后端服务器，则直接不再接受新请求 259 | transport.Interrupt() 260 | } 261 | 262 | // 等待 263 | start := time.Now().Unix() 264 | for true { 265 | // 如果5s内没有接受到新的请求了，则退出 266 | now := time.Now().Unix() 267 | if now - p.lastRequestTime.Get() > 5 { 268 | log.Info(Red("Graceful Exit...")) 269 | break 270 | } else { 271 | log.Printf(Cyan("Sleeping %d seconds\n"), now - start) 272 | time.Sleep(time.Second) 273 | } 274 | } 275 | 276 | // 确定处理完毕数据之后，关闭transport 277 | if registerService { 278 | transport.Interrupt() 279 | } 280 | transport.Close() 281 | }() 282 | 283 | go func() { 284 | var address string 285 | for c := range ch { 286 | // 为每个Connection建立一个Session 287 | socket, ok := c.(rpc_utils.SocketAddr) 288 | 289 | if ok { 290 | if isUnixDomain { 291 | address = p.FrontendAddr 292 | } else { 293 | address = socket.Addr().String() 294 | } 295 | } else { 296 | address = "unknow" 297 | } 298 | 299 | // Session独立处理自己的请求 300 | if registerService { 301 | x := NewNonBlockSession(c, address, p.Verbose, &p.lastRequestTime) 302 | go x.Serve(p, 1000) 303 | } else { 304 | go func(c thrift.TTransport) { 305 | // 打印异常信息 306 | defer func() { 307 | c.Close() 308 | if e := recover(); e != nil { 309 | log.Errorf("panic in processor: %v: %s", e, debug.Stack()) 310 | } 311 | }() 312 | 313 | ip := thrift.NewTBinaryProtocolTransport(thrift.NewTFramedTransport(c)) 314 | op := thrift.NewTBinaryProtocolTransport(thrift.NewTFramedTransport(c)) 315 | 316 | for true { 317 | defaultContext := context.Background() 318 | ok, err := p.Processor.Process(defaultContext, ip, op) 319 | 320 | if err != nil { 321 | // 如果链路出现异常（必须结束) 322 | if err, ok := err.(thrift.TTransportException); ok && (err.TypeId() == thrift.END_OF_FILE || 323 | strings.Contains(err.Error(), "use of closed network connection")) { 324 | return 325 | } else if err != nil { 326 | // 其他链路问题，直接报错，退出 327 | log.ErrorErrorf(err, "Error processing request, quit") 328 | return 329 | } 330 | 331 | // 如果是方法未知，则直接报错，然后跳过 332 | if err, ok := err.(thrift.TApplicationException); ok && err.TypeId() == thrift.UNKNOWN_METHOD { 333 | log.ErrorErrorf(err, "Error processing request, continue") 334 | return 335 | } 336 | 337 | log.ErrorErrorf(err, "Error processing request") 338 | // INTERNAL_ERROR --> ok: true 可以继续 339 | // PROTOCOL_ERROR --> ok: false 340 | // 非业务的错误都终止当前的连接 341 | return 342 | } 343 | 344 | // 其他情况下，ok == false，意味io过程中存在读写错误，connection上存在脏数据 345 | // 用户自定义的异常，必须继承自: *services.RpcException, 否则整个流程就容易出现问题 346 | if !ok { 347 | if err == nil { 348 | log.Printf("Process Not OK, stopped") 349 | } 350 | break 351 | } 352 | 353 | } 354 | }(c) 355 | } 356 | } 357 | }() 358 | 359 | // 准备上线服务 360 | state.Set(true) 361 | if registerService { 362 | stateChan <- true 363 | } 364 | 365 | log.Printf("Begin to accept requests...") 366 | // Accept什么时候出错，出错之后如何处理呢? 367 | for { 368 | c, err := transport.Accept() 369 | if err != nil { 370 | break 371 | } else { 372 | ch <- c 373 | } 374 | } 375 | } 376 | -------------------------------------------------------------------------------- /src/proxy/server_lb.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "fmt" 7 | "github.com/wfxiang08/cyutils/utils/atomic2" 8 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 9 | "github.com/wfxiang08/thrift_rpc_base/rpc_utils" 10 | thrift "github.com/wfxiang08/go_thrift/thrift" 11 | "os" 12 | "os/signal" 13 | "strings" 14 | "syscall" 15 | "time" 16 | ) 17 | 18 | // 19 | // Thrift Server的参数 20 | // 21 | type ThriftLoadBalanceServer struct { 22 | productName string 23 | serviceName string 24 | frontendAddr string // 绑定的端口 25 | backendAddr string 26 | lbServiceName string 27 | topo *Topology // ZK相关 28 | zkAddr string 29 | verbose bool 30 | backendService *BackServiceLB 31 | exitEvt chan bool 32 | lastRequestTime atomic2.Int64 33 | config *ServiceConfig 34 | } 35 | 36 | func NewThriftLoadBalanceServer(config *ServiceConfig) *ThriftLoadBalanceServer { 37 | log.Printf("FrontAddr: %s\n", Magenta(config.FrontendAddr)) 38 | 39 | // 前端对接rpc_proxy 40 | p := &ThriftLoadBalanceServer{ 41 | config: config, 42 | zkAddr: config.ZkAddr, 43 | productName: config.ProductName, 44 | serviceName: config.Service, 45 | frontendAddr: config.FrontendAddr, 46 | backendAddr: config.BackAddr, 47 | verbose: config.Verbose, 48 | exitEvt: make(chan bool), 49 | } 50 | 51 | p.topo = NewTopology(p.productName, p.zkAddr) 52 | p.lbServiceName = GetServiceIdentity(p.frontendAddr) 53 | 54 | // 后端对接: 各种python的rpc server 55 | p.backendService = NewBackServiceLB(p.serviceName, p.backendAddr, p.verbose, 56 | p.config.FalconClient, p.exitEvt) 57 | return p 58 | 59 | } 60 | 61 | func (p *ThriftLoadBalanceServer) Run() { 62 | // // 1. 创建到zk的连接 63 | 64 | // 127.0.0.1:5555 --> 127_0_0_1:5555 65 | 66 | exitSignal := make(chan os.Signal, 1) 67 | 68 | signal.Notify(exitSignal, syscall.SIGTERM, syscall.SIGINT, syscall.SIGKILL) 69 | // syscall.SIGKILL 70 | // kill -9 pid 71 | // kill -s SIGKILL pid 还是留给运维吧 72 | // 73 | 74 | // 注册服务 75 | evtExit := make(chan interface{}) 76 | 77 | // 初始状态为不上线 78 | var state atomic2.Bool 79 | state.Set(false) 80 | stateChan := make(chan bool) 81 | 82 | serviceEndpoint := RegisterService(p.serviceName, p.frontendAddr, p.lbServiceName, 83 | p.topo, evtExit, p.config.WorkDir, p.config.CodeUrlVersion, &state, stateChan) 84 | 85 | // var suideTime time.Time 86 | 87 | // isAlive := true 88 | 89 | // 3. 读取后端服务的配置 90 | var transport thrift.TServerTransport 91 | var err error 92 | 93 | isUnixDomain := false 94 | // 127.0.0.1:9999(以:区分不同的类型) 95 | if !strings.Contains(p.frontendAddr, ":") { 96 | if rpc_utils.FileExist(p.frontendAddr) { 97 | os.Remove(p.frontendAddr) 98 | } 99 | transport, err = rpc_utils.NewTServerUnixDomain(p.frontendAddr) 100 | isUnixDomain = true 101 | } else { 102 | transport, err = thrift.NewTServerSocket(p.frontendAddr) 103 | } 104 | 105 | if err != nil { 106 | log.ErrorErrorf(err, "Server Socket Create Failed: %v", err) 107 | panic(fmt.Sprintf("Invalid FrontendAddress: %s", p.frontendAddr)) 108 | } 109 | 110 | err = transport.Listen() 111 | if err != nil { 112 | log.ErrorErrorf(err, "Server Socket Create Failed: %v", err) 113 | panic(fmt.Sprintf("Binding Error FrontendAddress: %s", p.frontendAddr)) 114 | } 115 | 116 | ch := make(chan thrift.TTransport, 4096) 117 | defer close(ch) 118 | 119 | // 等待后端服务起来 120 | waitTicker := time.NewTicker(time.Second) 121 | 122 | // 等待上线采用的策略: 123 | // 1. 检测到有效的Worker注册之后，再等5s即可像zk注册; 避免了Worker没有连接上来，就有请求过来 124 | // 2. 一旦注册之后，就不再使用该策略；避免服务故障时，lb频繁更新zk, 导致proxy等频繁读取zk 125 | START_WAIT: 126 | for true { 127 | select { 128 | case <-waitTicker.C: 129 | if p.backendService.Active() <= 0 { 130 | log.Infof("Sleep Waiting for back Service to Start") 131 | time.Sleep(time.Second) 132 | } else { 133 | break START_WAIT 134 | } 135 | case <-exitSignal: 136 | // 直接退出 137 | transport.Interrupt() 138 | transport.Close() 139 | return 140 | } 141 | } 142 | 143 | log.Infof("Stop Waiting") 144 | // 停止: waitTicker, 再等等就继续了 145 | waitTicker.Stop() 146 | time.Sleep(time.Second * 5) 147 | 148 | log.Infof("Begin to Reg To Zk...") 149 | state.Set(true) 150 | stateChan <- true 151 | 152 | // 强制退出? TODO: Graceful退出 153 | go func() { 154 | <-exitSignal 155 | 156 | // 通知RegisterService终止循环 157 | evtExit <- true 158 | log.Info(Green("Receive Exit Signals....")) 159 | serviceEndpoint.DeleteServiceEndpoint(p.topo) 160 | 161 | start := time.Now().Unix() 162 | for true { 163 | // 如果5s内没有接受到新的请求了，则退出 164 | now := time.Now().Unix() 165 | if now-p.lastRequestTime.Get() > 5 { 166 | log.Printf(Red("[%s]Graceful Exit..."), p.serviceName) 167 | break 168 | } else { 169 | log.Printf(Cyan("[%s]Sleeping %d seconds before Exit...\n"), 170 | p.serviceName, now-start) 171 | time.Sleep(time.Second) 172 | } 173 | } 174 | 175 | transport.Interrupt() 176 | transport.Close() 177 | }() 178 | 179 | go func() { 180 | var address string 181 | for c := range ch { 182 | // 为每个Connection建立一个Session 183 | socket, ok := c.(rpc_utils.SocketAddr) 184 | 185 | if ok { 186 | if isUnixDomain { 187 | address = p.frontendAddr 188 | } else { 189 | address = socket.Addr().String() 190 | } 191 | } else { 192 | address = "unknow" 193 | } 194 | x := NewNonBlockSession(c, address, p.verbose, &p.lastRequestTime) 195 | // Session独立处理自己的请求 196 | go x.Serve(p.backendService, 1000) 197 | } 198 | }() 199 | 200 | // Accept什么时候出错，出错之后如何处理呢? 201 | for { 202 | c, err := transport.Accept() 203 | if err != nil { 204 | close(ch) 205 | break 206 | } else { 207 | ch <- c 208 | } 209 | } 210 | } 211 | -------------------------------------------------------------------------------- /src/proxy/server_proxy.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "fmt" 7 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 8 | "github.com/wfxiang08/go_thrift/thrift" 9 | "github.com/wfxiang08/thrift_rpc_base/rpc_utils" 10 | "os" 11 | "strings" 12 | "time" 13 | ) 14 | 15 | const ( 16 | HEARTBEAT_INTERVAL = 1000 * time.Millisecond 17 | ) 18 | 19 | type ProxyServer struct { 20 | productName string 21 | proxyAddr string 22 | zkAdresses string 23 | topo *Topology 24 | verbose bool 25 | profile bool 26 | router *Router 27 | } 28 | 29 | func NewProxyServer(config *ProxyConfig) *ProxyServer { 30 | p := &ProxyServer{ 31 | productName: config.ProductName, 32 | proxyAddr: config.ProxyAddr, 33 | zkAdresses: config.ZkAddr, 34 | verbose: config.Verbose, 35 | profile: config.Profile, 36 | } 37 | p.topo = NewTopology(p.productName, p.zkAdresses) 38 | p.router = NewRouter(p.productName, p.topo, p.verbose) 39 | return p 40 | } 41 | 42 | // 43 | // 两参数是必须的: ProductName, zkAddress, frontAddr可以用来测试 44 | // 45 | func (p *ProxyServer) Run() { 46 | 47 | var transport thrift.TServerTransport 48 | var err error 49 | 50 | log.Printf(Magenta("Start Proxy at Address: %s"), p.proxyAddr) 51 | // 读取后端服务的配置 52 | isUnixDomain := false 53 | if !strings.Contains(p.proxyAddr, ":") { 54 | if rpc_utils.FileExist(p.proxyAddr) { 55 | os.Remove(p.proxyAddr) 56 | } 57 | transport, err = rpc_utils.NewTServerUnixDomain(p.proxyAddr) 58 | isUnixDomain = true 59 | } else { 60 | transport, err = thrift.NewTServerSocket(p.proxyAddr) 61 | } 62 | if err != nil { 63 | log.ErrorErrorf(err, "Server Socket Create Failed: %v, Front: %s", err, p.proxyAddr) 64 | } 65 | 66 | // 开始监听 67 | // transport.Open() 68 | transport.Listen() 69 | 70 | ch := make(chan thrift.TTransport, 4096) 71 | defer close(ch) 72 | defer func() { 73 | log.Infof(Red("==> Exit rpc_proxy")) 74 | if err := recover(); err != nil { 75 | log.Infof("Error rpc_proxy: %s", err) 76 | } 77 | }() 78 | go func() { 79 | var address string 80 | for c := range ch { 81 | // 为每个Connection建立一个Session 82 | socket, ok := c.(rpc_utils.SocketAddr) 83 | if isUnixDomain { 84 | address = p.proxyAddr 85 | } else if ok { 86 | address = socket.Addr().String() 87 | } else { 88 | address = "unknow" 89 | } 90 | x := NewSession(c, address, p.verbose) 91 | // Session独立处理自己的请求 92 | go x.Serve(p.router, 1000) 93 | } 94 | }() 95 | 96 | // Accept什么时候出错，出错之后如何处理呢? 97 | for { 98 | c, err := transport.Accept() 99 | if err != nil { 100 | log.ErrorErrorf(err, "Accept Error: %v", err) 101 | break 102 | } else { 103 | ch <- c 104 | } 105 | } 106 | } 107 | 108 | func printList(msgs []string) string { 109 | results := make([]string, 0, len(msgs)) 110 | results = append(results, fmt.Sprintf("Msgs Len: %d, ", len(msgs)-1)) 111 | for i := 0; i < len(msgs)-1; i++ { 112 | results = append(results, fmt.Sprintf("[%s]", msgs[i])) 113 | } 114 | return strings.Join(results, ",") 115 | } 116 | -------------------------------------------------------------------------------- /src/proxy/session_nonblock.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "github.com/wfxiang08/cyutils/utils/atomic2" 7 | "github.com/wfxiang08/cyutils/utils/errors" 8 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 9 | "github.com/wfxiang08/go_thrift/thrift" 10 | "sync" 11 | "time" 12 | ) 13 | 14 | // 15 | // 用于rpc proxy或者load balance用来管理Client的 16 | // 17 | type NonBlockSession struct { 18 | *TBufferedFramedTransport 19 | 20 | RemoteAddress string 21 | 22 | closed atomic2.Bool 23 | verbose bool 24 | 25 | // 用于记录整个RPC服务的最后的访问时间，然后用于Graceful Stop 26 | lastRequestTime *atomic2.Int64 27 | 28 | lastSeqId int32 29 | } 30 | 31 | func NewNonBlockSession(c thrift.TTransport, address string, verbose bool, 32 | lastRequestTime *atomic2.Int64) *NonBlockSession { 33 | return NewNonBlockSessionSize(c, address, verbose, lastRequestTime, 1024 * 32, 1800) 34 | } 35 | 36 | func NewNonBlockSessionSize(c thrift.TTransport, address string, verbose bool, 37 | lastRequestTime *atomic2.Int64, bufsize int, timeout int) *NonBlockSession { 38 | s := &NonBlockSession{ 39 | RemoteAddress: address, 40 | lastRequestTime: lastRequestTime, 41 | verbose: verbose, 42 | TBufferedFramedTransport: NewTBufferedFramedTransport(c, time.Microsecond * 100, 20), 43 | } 44 | 45 | // 还是基于c net.Conn进行读写，只是采用Redis协议进行编码解码 46 | // Reader 处理Client发送过来的消息 47 | // Writer 将后端服务的数据返回给Client 48 | log.Printf(Green("Session From Proxy [%s] created"), address) 49 | return s 50 | } 51 | 52 | func (s *NonBlockSession) Close() error { 53 | s.closed.Set(true) 54 | return s.TBufferedFramedTransport.Close() 55 | } 56 | 57 | func (s *NonBlockSession) IsClosed() bool { 58 | return s.closed.Get() 59 | } 60 | 61 | func (s *NonBlockSession) Serve(d Dispatcher, maxPipeline int) { 62 | var errlist errors.ErrorList 63 | 64 | defer func() { 65 | // 只限制第一个Error 66 | if err := errlist.First(); err != nil { 67 | log.Infof("Session [%p] closed, Error = %v", s, err) 68 | } else { 69 | log.Infof("Session [%p] closed, Quit", s) 70 | } 71 | }() 72 | 73 | // 来自connection的各种请求 74 | tasks := make(chan *Request, maxPipeline) 75 | go func() { 76 | defer func() { 77 | // 出现错误了，直接关闭Session 78 | s.Close() 79 | 80 | log.Infof(Red("Session [%p] closed, Abandon %d Tasks"), s, len(tasks)) 81 | 82 | for _ = range tasks { 83 | // close(tasks)关闭for loop 84 | 85 | } 86 | }() 87 | if err := s.loopWriter(tasks); err != nil { 88 | errlist.PushBack(err) 89 | } 90 | }() 91 | 92 | // 用于等待for中的go func执行完毕 93 | var wait sync.WaitGroup 94 | for true { 95 | // Reader不停地解码， 将Request 96 | request, err := s.ReadFrame() 97 | 98 | if err != nil { 99 | errlist.PushBack(err) 100 | break 101 | } 102 | // 来自proxy的请求, request中不带有service 103 | r, err1 := NewRequest(request, false) 104 | if err1 != nil { 105 | errlist.PushBack(err1) 106 | break 107 | } 108 | 109 | wait.Add(1) 110 | go func(r*Request) { 111 | // 异步执行(直接通过goroutine来调度，因为: SessionNonBlock中的不同的Request相互独立) 112 | 113 | _ = s.handleRequest(r, d) 114 | // if r.Request.TypeId != MESSAGE_TYPE_HEART_BEAT { 115 | // log.Debugf(Magenta("[%p] --> SeqId: %d, Goroutine: %d"), s, r.Request.SeqId, runtime.NumGoroutine()) 116 | // } 117 | 118 | // 数据请求完毕之后，将Request交给tasks, 然后再写回Client 119 | 120 | tasks <- r 121 | wait.Done() 122 | // if r.Request.TypeId != MESSAGE_TYPE_HEART_BEAT { 123 | // log.Printf("Time RT: %.3fms", float64(microseconds()-r.Start)*0.001) 124 | // } 125 | 126 | }(r) 127 | } 128 | // 等待go func执行完毕 129 | wait.Wait() 130 | close(tasks) 131 | return 132 | } 133 | 134 | // 135 | // 136 | // NonBlock和Block的区别: 137 | // NonBlock的 Request和Response是不需要配对的， Request和Response是独立的，例如: 138 | // ---> RequestA, RequestB 139 | // <--- RequestB, RequestA 后请求的，可以先返回 140 | // 141 | func (s *NonBlockSession) loopWriter(tasks <-chan *Request) error { 142 | for r := range tasks { 143 | // 1. tasks中的请求是已经请求完毕的，loopWriter负责将它们的数据写回到rpc proxy 144 | s.handleResponse(r) 145 | 146 | // 2. 将结果写回给Client 147 | _, err := s.TBufferedFramedTransport.Write(r.Response.Data) 148 | r.Recycle() 149 | if err != nil { 150 | log.ErrorErrorf(err, "SeqId: %d, Write back Data Error: %v\n", r.Request.SeqId, err) 151 | return err 152 | } 153 | 154 | // typeId, sedId, _ := DecodeThriftTypIdSeqId(r.Response.Data) 155 | 156 | // if typeId != MESSAGE_TYPE_HEART_BEAT { 157 | // if sedId != s.lastSeqId { 158 | // log.Errorf(Red("Invalid SedId for Writer: %d vs. %d"), sedId, s.lastSeqId) 159 | // } 160 | // } 161 | 162 | // log.Printf(Magenta("Task: %d"), r.Response.SeqId) 163 | 164 | // 3. Flush 165 | err = s.TBufferedFramedTransport.FlushBuffer(len(tasks) == 0) // len(tasks) == 0 166 | // if err != nil { 167 | // log.Debugf(Magenta("Write Back to Client/Proxy SeqId: %d, Error: %v"), r.Request.SeqId, err) 168 | // } else { 169 | // log.Debugf(Magenta("Write Back to Client/Proxy SeqId: %d"), r.Request.SeqId) 170 | // } 171 | 172 | if err != nil { 173 | return err 174 | } 175 | } 176 | return nil 177 | } 178 | 179 | // 获取可以直接返回给Client的response 180 | func (s *NonBlockSession) handleResponse(r *Request) { 181 | 182 | // 将Err转换成为Exception 183 | if r.Response.Err != nil { 184 | log.Println("#handleResponse, Error ----> Reponse Data") 185 | r.Response.Data = GetThriftException(r, "nonblock_session") 186 | } 187 | 188 | incrOpStats(r.Request.Name, microseconds() - r.Start) 189 | } 190 | 191 | // 处理来自Client的请求 192 | // 将它的请教交给后端的Dispatcher 193 | // 194 | func (s *NonBlockSession) handleRequest(r *Request, d Dispatcher) error { 195 | // 构建Request 196 | // log.Printf("HandleRequest: %s\n", string(request)) 197 | // 处理心跳 198 | if r.Request.TypeId == MESSAGE_TYPE_HEART_BEAT { 199 | // log.Printf(Magenta("PING/PANG")) 200 | HandlePingRequest(r) 201 | return nil 202 | } 203 | 204 | // if r.Request.SeqId-s.lastSeqId != 1 { 205 | // log.Errorf(Red("Invalid SedId: %d vs. %d"), r.Request.SeqId, 206 | // s.lastSeqId) 207 | // } 208 | // s.lastSeqId = r.Request.SeqId 209 | 210 | // 正常请求 211 | if s.lastRequestTime != nil { 212 | s.lastRequestTime.Set(time.Now().Unix()) 213 | } 214 | 215 | // log.Debugf("Before Dispatch, SeqId: %d", r.Request.SeqId) 216 | 217 | // 交给Dispatch 218 | return d.Dispatch(r) 219 | } 220 | -------------------------------------------------------------------------------- /src/proxy/session_nonblock_test.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "github.com/stretchr/testify/assert" 7 | 8 | "testing" 9 | ) 10 | 11 | // 12 | // go test proxy -v -run "TestSessionNonBlock" 13 | // 14 | func TestSessionNonBlock(t *testing.T) { 15 | 16 | // result = client.correct_typo_simple("HELLO") 17 | typoHello := []byte{128, 1, 0, 1, 0, 0, 0, 19, 99, 111, 114, 114, 101, 99, 18 | 116, 95, 116, 121, 112, 111, 95, 115, 105, 109, 112, 108, 101, 0, 0, 3, 19 | 239, 0, 5, 72, 69, 76, 76, 79, 0} 20 | s := &NonBlockSession{} 21 | d := &testDispatcher1{} 22 | 23 | // 确保SessionNonBlock创建的Request满足要求 24 | r, err1 := NewRequest(typoHello, false) 25 | s.handleRequest(r, d) 26 | assert.False(t, d.Request.ProxyRequest) 27 | } 28 | 29 | type testDispatcher1 struct { 30 | Request *Request 31 | } 32 | 33 | func (d *testDispatcher1) Dispatch(r *Request) error { 34 | d.Request = r 35 | return nil 36 | } 37 | -------------------------------------------------------------------------------- /src/proxy/session_proxy.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "github.com/wfxiang08/go_thrift/thrift" 7 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 8 | "time" 9 | ) 10 | 11 | type Session struct { 12 | *TBufferedFramedTransport 13 | 14 | RemoteAddress string 15 | Ops int64 16 | LastOpUnix int64 17 | CreateUnix int64 18 | verbose bool 19 | } 20 | 21 | // c： client <---> proxy之间的连接 22 | func NewSession(c thrift.TTransport, address string, verbose bool) *Session { 23 | return NewSessionSize(c, address, verbose, 1024 * 32, 5000) 24 | } 25 | 26 | func NewSessionSize(c thrift.TTransport, address string, verbose bool, 27 | bufsize int, timeout int) *Session { 28 | 29 | s := &Session{ 30 | CreateUnix: time.Now().Unix(), 31 | RemoteAddress: address, 32 | verbose: verbose, 33 | TBufferedFramedTransport: NewTBufferedFramedTransport(c, time.Microsecond * 100, 20), 34 | } 35 | 36 | // Reader 处理Client发送过来的消息 37 | // Writer 将后端服务的数据返回给Client 38 | log.Infof(Green("NewSession To: %s"), s.RemoteAddress) 39 | return s 40 | } 41 | 42 | func (s *Session) Close() error { 43 | log.Printf(Red("Close Proxy Session")) 44 | return s.TBufferedFramedTransport.Close() 45 | } 46 | 47 | // Session是同步处理请求，因此没有必要搞多个 48 | func (s *Session) Serve(d *Router, maxPipeline int) { 49 | defer func() { 50 | s.Close() 51 | log.Infof(Red("==> Session Over: %s, Total %d Ops"), s.RemoteAddress, s.Ops) 52 | if err := recover(); err != nil { 53 | log.Infof(Red("Catch session error: %v"), err) 54 | } 55 | }() 56 | 57 | requests := make(chan *Request, 20) 58 | 59 | go func() { 60 | var err error 61 | for r := range requests { 62 | // 3. 等待请求处理完毕 63 | // 先调用的先返回 64 | s.handleResponse(r) 65 | 66 | // 4. 将结果写回给Client 67 | if s.verbose { 68 | log.Debugf("[%s]Session#loopWriter --> client FrameSize: %d", 69 | r.Service, len(r.Response.Data)) 70 | } 71 | 72 | // 5. 将请求返回给Client, r.Response.Data ---> Client 73 | _, err = s.TBufferedFramedTransport.Write(r.Response.Data) 74 | r.Recycle() // 重置: Request 75 | 76 | if err != nil { 77 | // 写数据出错，那只能说明连接坏了，直接断开 78 | log.ErrorErrorf(err, "Write back Data Error: %v", err) 79 | return 80 | } 81 | 82 | // 6. Flush 83 | err = s.TBufferedFramedTransport.FlushBuffer(true) // len(tasks) == 0 84 | if err != nil { 85 | log.ErrorErrorf(err, "Write back Data Error: %v", err) 86 | return 87 | } 88 | } 89 | 90 | }() 91 | 92 | // 读写分离 93 | for true { 94 | // 1. 读取请求 95 | request, err := s.ReadFrame() 96 | s.Ops += 1 97 | 98 | // 读取出错，直接退出 99 | if err != nil { 100 | err1, ok := err.(thrift.TTransportException) 101 | if !ok || err1.TypeId() != thrift.END_OF_FILE { 102 | log.ErrorErrorf(err, Red("ReadFrame Error: %v"), err) 103 | } 104 | // r.Recycle() 105 | close(requests) 106 | return 107 | } 108 | 109 | var r *Request 110 | // 2. 处理请求 111 | r, err = s.handleRequest(request, d) 112 | if err != nil { 113 | // r.Recycle() // 出错之后也要主动返回数据 114 | log.ErrorErrorf(err, Red("handleRequest Error: %v"), err) 115 | r.Response.Err = err 116 | } 117 | 118 | requests <- r 119 | } 120 | } 121 | 122 | // 123 | // 124 | // 等待Request请求的返回: Session最终被Block住 125 | // 126 | func (s *Session) handleResponse(r *Request) { 127 | // 等待结果的出现 128 | r.Wait.Wait() 129 | 130 | // 将Err转换成为Exception 131 | if r.Response.Err != nil { 132 | r.Response.Data = GetThriftException(r, "proxy_session") 133 | log.Infof(Magenta("---->Convert Error Back to Exception, Err: %v"), r.Response.Err) 134 | } 135 | 136 | // 如何处理Data和Err呢? 137 | incrOpStats(r.Request.Name, microseconds() - r.Start) 138 | } 139 | 140 | // 处理来自Client的请求 141 | func (s *Session) handleRequest(request []byte, d *Router) (*Request, error) { 142 | // 构建Request 143 | if s.verbose { 144 | log.Printf("HandleRequest: %s", string(request)) 145 | } 146 | r, err := NewRequest(request, true) 147 | if err != nil { 148 | return r, err 149 | } 150 | 151 | // 增加统计 152 | s.LastOpUnix = time.Now().Unix() 153 | s.Ops++ 154 | if r.Request.TypeId == MESSAGE_TYPE_HEART_BEAT { 155 | HandleProxyPingRequest(r) // 直接返回数据 156 | return r, nil 157 | } 158 | 159 | // 交给Dispatch 160 | // Router 161 | return r, d.Dispatch(r) 162 | } 163 | 164 | func microseconds() int64 { 165 | return time.Now().UnixNano() / int64(time.Microsecond) 166 | } 167 | -------------------------------------------------------------------------------- /src/proxy/session_proxy_test.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "fmt" 7 | thrift "github.com/wfxiang08/go_thrift/thrift" 8 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 9 | "github.com/stretchr/testify/assert" 10 | "testing" 11 | "time" 12 | ) 13 | 14 | func init() { 15 | log.SetLevel(log.LEVEL_INFO) 16 | log.SetFlags(log.Flags() | log.Lshortfile) 17 | } 18 | 19 | type fakeServer struct { 20 | } 21 | 22 | func (p *fakeServer) Dispatch(r *Request) error { 23 | log.Printf("Request SeqId: %d, MethodName: %s\n", r.Request.SeqId, r.Request.Name) 24 | r.Wait.Add(1) 25 | go func() { 26 | time.Sleep(time.Millisecond) 27 | r.Response.Data = []byte(string(r.Request.Data)) 28 | 29 | typeId, _, seqId, _ := DecodeThriftTypIdSeqId(r.Response.Data) 30 | log.Printf(Green("TypeId: %d, SeqId: %d\n"), typeId, seqId) 31 | r.Wait.Done() 32 | }() 33 | // r.RestoreSeqId() 34 | // r.Wait.Done() 35 | return nil 36 | } 37 | 38 | // 39 | // go test proxy -v -run "TestSession" 40 | // 41 | func TestSession(t *testing.T) { 42 | // 作为一个Server 43 | transport, err := thrift.NewTServerSocket("127.0.0.1:0") 44 | err = transport.Open() // 打开Transport 45 | defer transport.Close() 46 | 47 | err = transport.Listen() // 开始监听 48 | assert.NoError(t, err) 49 | 50 | addr := transport.Addr().String() 51 | fmt.Println("Addr: ", addr) 52 | 53 | // 1. Fake Requests 54 | var requestNum int32 = 10 55 | requests := make([]*Request, 0, requestNum) 56 | var i int32 57 | for i = 0; i < requestNum; i++ { 58 | buf := make([]byte, 100, 100) 59 | l := fakeData("Hello", thrift.CALL, i+1, buf[0:0]) 60 | buf = buf[0:l] 61 | 62 | req, _ := NewRequest(buf, true) 63 | 64 | req.Wait.Add(1) // 因为go routine可能还没有执行，代码就跑到最后面进行校验了 65 | 66 | assert.Equal(t, i+1, req.Request.SeqId, "Request SeqId是否靠谱") 67 | 68 | requests = append(requests, req) 69 | } 70 | 71 | // 2. 将请求交给BackendConn 72 | go func() { 73 | // 模拟请求: 74 | // 客户端代码 75 | bc := NewBackendConn(addr, nil, "test", true) 76 | bc.currentSeqId = 10 77 | 78 | // 上线 BackendConn 79 | bc.IsConnActive.Set(true) 80 | 81 | // 准备发送数据 82 | var i int32 83 | for i = 0; i < requestNum; i++ { 84 | fmt.Println("Sending Request to Backend Conn", i) 85 | bc.PushBack(requests[i]) 86 | 87 | requests[i].Wait.Done() 88 | } 89 | 90 | // 需要等待数据返回? 91 | time.Sleep(time.Second * 2) 92 | }() 93 | 94 | server := &fakeServer{} 95 | go func() { 96 | // 服务器端代码 97 | tran, err := transport.Accept() 98 | defer tran.Close() 99 | if err != nil { 100 | log.ErrorErrorf(err, "Error: %v\n", err) 101 | } 102 | assert.NoError(t, err) 103 | 104 | // 建立一个长连接, 同上面的: NewBackendConn通信 105 | session := NewSession(tran, "", true) 106 | session.Serve(server, 6) 107 | 108 | time.Sleep(time.Second * 2) 109 | }() 110 | 111 | for i = 0; i < requestNum; i++ { 112 | fmt.Println("===== Before Wait") 113 | requests[i].Wait.Wait() 114 | fmt.Println("===== Before After Wait") 115 | 116 | log.Printf("Request: %d, .....", i) 117 | assert.Equal(t, len(requests[i].Request.Data), len(requests[i].Response.Data)) 118 | } 119 | } 120 | -------------------------------------------------------------------------------- /src/proxy/stats.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | 4 | package proxy 5 | 6 | import ( 7 | "encoding/json" 8 | "fmt" 9 | "sync" 10 | "time" 11 | 12 | "github.com/wfxiang08/cyutils/utils" 13 | "github.com/wfxiang08/cyutils/utils/atomic2" 14 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 15 | ) 16 | 17 | // 18 | // 单个的统计指标 19 | // 20 | type OpStats struct { 21 | opstr string 22 | calls atomic2.Int64 // 次数 23 | usecs atomic2.Int64 // 总时间(us) 24 | } 25 | 26 | func (s *OpStats) OpStr() string { 27 | return s.opstr 28 | } 29 | 30 | func (s *OpStats) Calls() int64 { 31 | return s.calls.Get() 32 | } 33 | 34 | func (s *OpStats) USecs() int64 { 35 | return s.usecs.Get() 36 | } 37 | 38 | func (s *OpStats) USecsPerCall() int64 { 39 | var perusecs int64 = 0 40 | if s.calls.Get() != 0 { 41 | perusecs = s.usecs.Get() / s.calls.Get() 42 | } 43 | return perusecs 44 | } 45 | 46 | func (s *OpStats) MarshalJSON() ([]byte, error) { 47 | var m = make(map[string]interface{}) 48 | var calls = s.calls.Get() 49 | var usecs = s.usecs.Get() 50 | 51 | var perusecs int64 = 0 52 | if calls != 0 { 53 | perusecs = usecs / calls 54 | } 55 | 56 | m["cmd"] = s.opstr 57 | m["calls"] = calls 58 | m["usecs"] = usecs 59 | m["usecs_percall"] = perusecs 60 | return json.Marshal(m) 61 | } 62 | 63 | // 64 | // 所有的Commands的统计信息 65 | // 66 | var cmdstats struct { 67 | requests atomic2.Int64 68 | 69 | opmap map[string]*OpStats 70 | 71 | histMaps chan *OpStatsInfo 72 | rwlck sync.RWMutex 73 | ticker *time.Ticker 74 | start sync.Once 75 | } 76 | 77 | type OpStatsInfo struct { 78 | opmap map[string]*OpStats 79 | timestamp time.Time 80 | } 81 | 82 | func init() { 83 | cmdstats.opmap = make(map[string]*OpStats) 84 | } 85 | 86 | const ( 87 | EMPTY_STR = "" 88 | ) 89 | 90 | // 主动调用 91 | func StartTicker(falconClient string, service string) { 92 | // 如果没有监控配置，则直接返回 93 | if len(falconClient) == 0 { 94 | return 95 | } 96 | 97 | log.Printf(Green("Log to falconClient: %s"), falconClient) 98 | 99 | cmdstats.histMaps = make(chan *OpStatsInfo, 5) 100 | cmdstats.ticker = time.NewTicker(time.Minute) 101 | 102 | var statsInfo *OpStatsInfo 103 | hostname := utils.Hostname() 104 | go func() { 105 | for true { 106 | statsInfo = <-cmdstats.histMaps 107 | 108 | // 准备发送 109 | // 需要处理timeout 110 | metrics := make([]*utils.MetaData, 0, 3) 111 | t := statsInfo.timestamp.Unix() 112 | 113 | for method, stats := range statsInfo.opmap { 114 | 115 | metricCount := &utils.MetaData{ 116 | Metric: fmt.Sprintf("%s.%s.calls", service, method), 117 | Endpoint: hostname, 118 | Value: stats.Calls(), 119 | CounterType: utils.DATA_TYPE_GAUGE, 120 | Tags: EMPTY_STR, 121 | Timestamp: t, 122 | Step: 60, // 一分钟一次采样 123 | } 124 | 125 | metricAvg := &utils.MetaData{ 126 | Metric: fmt.Sprintf("%s.%s.avgrt", service, method), 127 | Endpoint: hostname, 128 | Value: float64(stats.USecsPerCall()) * 0.001, // 单位: ms 129 | CounterType: utils.DATA_TYPE_GAUGE, 130 | Tags: EMPTY_STR, 131 | Timestamp: t, 132 | Step: 60, // 一分钟一次采样 133 | } 134 | 135 | metrics = append(metrics, metricCount, metricAvg) 136 | } 137 | 138 | // 准备发送数据到Local Agent 139 | // 10s timeout 140 | log.Printf("Send %d Metrics....", len(metrics)) 141 | if len(metrics) > 0 { 142 | utils.SendData(metrics, falconClient, time.Second*10) 143 | } 144 | 145 | } 146 | 147 | }() 148 | 149 | go func() { 150 | // 死循环: 最终进程退出时自动被杀掉 151 | var t time.Time 152 | for t = range cmdstats.ticker.C { 153 | // 到了指定的时间点之后将过去一分钟的统计数据转移到: 154 | cmdstats.rwlck.Lock() 155 | cmdstats.histMaps <- &OpStatsInfo{ 156 | opmap: cmdstats.opmap, 157 | timestamp: t, 158 | } 159 | cmdstats.opmap = make(map[string]*OpStats) 160 | cmdstats.rwlck.Unlock() 161 | } 162 | }() 163 | } 164 | 165 | func OpCounts() int64 { 166 | return cmdstats.requests.Get() 167 | } 168 | 169 | func GetOpStats(methodName string, create bool) *OpStats { 170 | cmdstats.rwlck.RLock() 171 | s := cmdstats.opmap[methodName] 172 | cmdstats.rwlck.RUnlock() 173 | 174 | if s != nil || !create { 175 | return s 176 | } 177 | 178 | cmdstats.rwlck.Lock() 179 | s = cmdstats.opmap[methodName] 180 | if s == nil { 181 | s = &OpStats{opstr: methodName} 182 | cmdstats.opmap[methodName] = s 183 | } 184 | cmdstats.rwlck.Unlock() 185 | return s 186 | } 187 | 188 | func GetAllOpStats() []*OpStats { 189 | var all = make([]*OpStats, 0, 128) 190 | cmdstats.rwlck.RLock() 191 | for _, s := range cmdstats.opmap { 192 | all = append(all, s) 193 | } 194 | cmdstats.rwlck.RUnlock() 195 | return all 196 | } 197 | 198 | func incrOpStats(methodName string, usecs int64) { 199 | s := GetOpStats(methodName, true) 200 | s.calls.Incr() 201 | s.usecs.Add(usecs) 202 | cmdstats.requests.Incr() 203 | } 204 | -------------------------------------------------------------------------------- /src/proxy/thrift_exception.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | "fmt" 7 | thrift "github.com/wfxiang08/go_thrift/thrift" 8 | ) 9 | 10 | // 11 | // 生成Thrift格式的Exception Message 12 | // 13 | func GetServiceNotFoundData(req *Request) []byte { 14 | req.Response.TypeId = thrift.EXCEPTION 15 | 16 | // 构建thrift的Transport 17 | transport := thrift.NewTMemoryBufferLen(100) 18 | protocol := thrift.NewTBinaryProtocolTransport(transport) 19 | 20 | // 构建一个Message, 写入Exception 21 | msg := fmt.Sprintf("Service: %s Not Found", req.Service) 22 | exc := thrift.NewTApplicationException(thrift.UNKNOWN_APPLICATION_EXCEPTION, msg) 23 | 24 | protocol.WriteMessageBegin(req.Request.Name, thrift.EXCEPTION, req.Request.SeqId) 25 | exc.Write(protocol) 26 | protocol.WriteMessageEnd() 27 | protocol.Flush() 28 | 29 | bytes := transport.Bytes() 30 | return bytes 31 | } 32 | 33 | func GetWorkerNotFoundData(req *Request, module string) []byte { 34 | req.Response.TypeId = thrift.EXCEPTION 35 | 36 | // 构建thrift的Transport 37 | transport := thrift.NewTMemoryBufferLen(100) 38 | protocol := thrift.NewTBinaryProtocolTransport(transport) 39 | 40 | // 构建一个Message, 写入Exception 41 | msg := fmt.Sprintf("Worker FOR %s#%s.%s Not Found", module, req.Service, req.Request.Name) 42 | exc := thrift.NewTApplicationException(thrift.INTERNAL_ERROR, msg) 43 | 44 | protocol.WriteMessageBegin(req.Request.Name, thrift.EXCEPTION, req.Request.SeqId) 45 | exc.Write(protocol) 46 | protocol.WriteMessageEnd() 47 | 48 | bytes := transport.Bytes() 49 | return bytes 50 | } 51 | 52 | func GetThriftException(req *Request, module string) []byte { 53 | req.Response.TypeId = thrift.EXCEPTION 54 | 55 | // 构建thrift的Transport 56 | transport := thrift.NewTMemoryBufferLen(256) 57 | protocol := thrift.NewTBinaryProtocolTransport(transport) 58 | 59 | msg := fmt.Sprintf("Module: %s, Service: %s, Method: %s, Error: %v", module, req.Service, req.Request.Name, req.Response.Err) 60 | 61 | // 构建一个Message, 写入Exception 62 | exc := thrift.NewTApplicationException(thrift.INTERNAL_ERROR, msg) 63 | 64 | // 注意消息的格式 65 | protocol.WriteMessageBegin(req.Request.Name, thrift.EXCEPTION, req.Request.SeqId) 66 | exc.Write(protocol) 67 | protocol.WriteMessageEnd() 68 | 69 | bytes := transport.Bytes() 70 | return bytes 71 | } 72 | 73 | // 74 | // 解析Thrift数据的Message Header 75 | // 76 | func ParseThriftMsgBegin(msg []byte) (name string, typeId thrift.TMessageType, seqId int32, err error) { 77 | transport := thrift.NewTMemoryBufferLen(256) 78 | transport.Write(msg) 79 | protocol := thrift.NewTBinaryProtocolTransport(transport) 80 | name, typeId, seqId, err = protocol.ReadMessageBegin() 81 | return 82 | } 83 | -------------------------------------------------------------------------------- /src/proxy/thrift_exception_test.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | // "fmt" 7 | // thrift "github.com/wfxiang08/go_thrift/thrift" 8 | // "utils/assert" 9 | // "strings" 10 | "testing" 11 | ) 12 | 13 | // 14 | // go test proxy -v -run "TestGetThriftException" 15 | // 16 | func TestGetThriftException(t *testing.T) { 17 | 18 | // serviceName := "accounts" 19 | // data := GetServiceNotFoundData(serviceName, 0) 20 | // fmt.Println("Exception Data: ", string(data)) 21 | 22 | // transport := NewTMemoryBufferWithBuf(data) 23 | // exc := thrift.NewTApplicationException(-1, "") 24 | // protocol := thrift.NewTBinaryProtocolTransport(transport) 25 | 26 | // // 注意: Read函数返回的是一个新的对象 27 | // protocol.ReadMessageBegin() 28 | // exc, _ = exc.Read(protocol) 29 | // protocol.ReadMessageEnd() 30 | 31 | // fmt.Println("Exc: ", exc.TypeId(), "Error: ", exc.Error()) 32 | 33 | // var errMsg string = exc.Error() 34 | // assert.Must(strings.Contains(errMsg, serviceName)) 35 | } 36 | -------------------------------------------------------------------------------- /src/proxy/topology.go: -------------------------------------------------------------------------------- 1 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 2 | package proxy 3 | 4 | import ( 5 | "encoding/json" 6 | "fmt" 7 | "github.com/wfxiang08/cyutils/utils/errors" 8 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 9 | topo "github.com/wfxiang08/go-zookeeper/zk" 10 | "github.com/wfxiang08/thrift_rpc_base/zkhelper" 11 | color "github.com/fatih/color" 12 | os_path "path" 13 | "strings" 14 | ) 15 | 16 | var green = color.New(color.FgGreen).SprintFunc() 17 | 18 | // 19 | // 设计方案: 20 | // 1. zk中保存如下的数据 21 | // /chunyu/ProductName/Services/Services1 22 | // /host1:port2 23 | // /host2:port2 24 | // /Services2 25 | // 26 | // 27 | // Topology需要关注: 28 | // /chunyu/ProductName/Services 下面的Children的变化，关注有哪些服务，能自动发现 29 | // 关注每一个服务的变化 30 | // 这个和Codis的区别和联系? 31 | // 32 | type Topology struct { 33 | ProductName string // 例如: 线上服务， 测试服务等等 34 | zkAddr string // zk的地址 35 | ZkConn zkhelper.Conn // zk的连接 36 | basePath string 37 | } 38 | 39 | // 春雨产品服务列表对应的Path 40 | func (top *Topology) productBasePath(productName string) string { 41 | return fmt.Sprintf("/zk/product/%s", productName) 42 | } 43 | func (top *Topology) ProductServicesPath() string { 44 | return fmt.Sprintf("%s/services", top.basePath) 45 | } 46 | 47 | func (top *Topology) ProductServicePath(service string) string { 48 | return fmt.Sprintf("%s/services/%s", top.basePath, service) 49 | } 50 | 51 | // 获取具体的某个EndPoint的Path 52 | func (top *Topology) ProductServiceEndPointPath(service string, endpoint string) string { 53 | return fmt.Sprintf("%s/services/%s/%s", top.basePath, service, endpoint) 54 | } 55 | 56 | func (top *Topology) FullPath(path string) string { 57 | if !strings.HasPrefix(path, top.basePath) { 58 | path = fmt.Sprintf("%s%s", top.basePath, path) 59 | } 60 | return path 61 | } 62 | 63 | // 指定的path是否在zk中存在 64 | func (top *Topology) Exist(path string) (bool, error) { 65 | path = top.FullPath(path) 66 | return zkhelper.NodeExists(top.ZkConn, path) 67 | } 68 | 69 | func NewTopology(ProductName string, zkAddr string) *Topology { 70 | // 创建Topology对象，并且初始化ZkConn 71 | t := &Topology{zkAddr: zkAddr, ProductName: ProductName} 72 | t.basePath = t.productBasePath(ProductName) 73 | t.InitZkConn() 74 | return t 75 | } 76 | 77 | func (top *Topology) InitZkConn() { 78 | var err error 79 | // 连接到zk 80 | // 30s的timeout 81 | top.ZkConn, err = zkhelper.ConnectToZk(top.zkAddr, 30) // 参考: Codis的默认配置 82 | if err != nil { 83 | log.PanicErrorf(err, "init failed") 84 | } 85 | } 86 | 87 | func (top *Topology) IsChildrenChangedEvent(e interface{}) bool { 88 | return e.(topo.Event).Type == topo.EventNodeChildrenChanged 89 | } 90 | 91 | func (top *Topology) DeleteDir(path string) { 92 | dir := top.FullPath(path) 93 | if ok, _ := top.Exist(dir); ok { 94 | zkhelper.DeleteRecursive(top.ZkConn, dir, -1) 95 | } 96 | } 97 | 98 | // 创建指定的Path 99 | func (top *Topology) CreateDir(path string) (string, error) { 100 | dir := top.FullPath(path) 101 | if ok, _ := top.Exist(dir); ok { 102 | log.Println("Path Exists") 103 | return dir, nil 104 | } else { 105 | return zkhelper.CreateRecursive(top.ZkConn, dir, "", 0, zkhelper.DefaultDirACLs()) 106 | } 107 | } 108 | 109 | func (top *Topology) SetPathData(path string, data []byte) { 110 | dir := top.FullPath(path) 111 | top.ZkConn.Set(dir, data, -1) 112 | } 113 | 114 | // 115 | // 设置RPC Proxy的数据: 116 | // 绑定的前端的ip/port, 例如: {"rpc_front": "tcp://127.0.0.1:5550"} 117 | // 118 | func (top *Topology) SetRpcProxyData(proxyInfo map[string]interface{}) error { 119 | path := top.FullPath("/rpc_proxy") 120 | data, err := json.Marshal(proxyInfo) 121 | if err != nil { 122 | return err 123 | } 124 | 125 | // topo.FlagEphemeral 这里的ProxyInfo是手动配置的，需要持久化 126 | path, err = CreateOrUpdate(top.ZkConn, path, string(data), 0, zkhelper.DefaultDirACLs(), true) 127 | log.Println(green("SetRpcProxyData"), "Path: ", path, ", Error: ", err, ", Data: ", string(data)) 128 | return err 129 | } 130 | 131 | // 132 | // 读取RPC Proxy的数据: 133 | // 绑定的前端的ip/port, 例如: {"rpc_front": "tcp://127.0.0.1:5550"} 134 | // 135 | func (top *Topology) GetRpcProxyData() (proxyInfo map[string]interface{}, e error) { 136 | path := top.FullPath("/rpc_proxy") 137 | data, _, err := top.ZkConn.Get(path) 138 | 139 | log.Println("Data: ", data, ", err: ", err) 140 | if err != nil { 141 | return nil, err 142 | } 143 | proxyInfo = make(map[string]interface{}) 144 | err = json.Unmarshal(data, &proxyInfo) 145 | if err != nil { 146 | return nil, err 147 | } else { 148 | return proxyInfo, nil 149 | } 150 | } 151 | 152 | // 153 | // evtch: 154 | // 1. 来自Zookeeper驱动通知 155 | // 2. 该通知最终需要通过 evtbus 传递给其他人 156 | // 157 | func (top *Topology) doWatch(evtch <-chan topo.Event, evtbus chan interface{}) { 158 | e := <-evtch 159 | 160 | // http://wiki.apache.org/hadoop/ZooKeeper/FAQ 161 | // 如何处理? 照理说不会发生的 162 | if e.State == topo.StateExpired || e.Type == topo.EventNotWatching { 163 | log.Warnf("session expired: %+v", e) 164 | evtbus <- e 165 | return 166 | } 167 | 168 | log.Warnf("topo event %+v", e) 169 | 170 | switch e.Type { 171 | //case topo.EventNodeCreated: 172 | //case topo.EventNodeDataChanged: 173 | case topo.EventNodeChildrenChanged: //only care children changed 174 | //todo:get changed node and decode event 175 | default: 176 | 177 | // log.Warnf("%+v", e) 178 | } 179 | 180 | evtbus <- e 181 | } 182 | 183 | func (top *Topology) WatchChildren(path string, evtbus chan interface{}) ([]string, error) { 184 | // 获取Children的信息 185 | content, _, evtch, err := top.ZkConn.ChildrenW(path) 186 | if err != nil { 187 | return nil, errors.Trace(err) 188 | } 189 | 190 | go top.doWatch(evtch, evtbus) 191 | return content, nil 192 | } 193 | 194 | // 读取当前path对应的数据，监听之后的事件 195 | func (top *Topology) WatchNode(path string, evtbus chan interface{}) ([]byte, error) { 196 | content, _, evtch, err := top.ZkConn.GetW(path) 197 | if err != nil { 198 | return nil, errors.Trace(err) 199 | } 200 | 201 | // 从: evtch 读取数据，然后再传递到 evtbus 202 | // evtbus 是外部可控的 channel 203 | if evtbus != nil { 204 | go top.doWatch(evtch, evtbus) 205 | } 206 | return content, nil 207 | } 208 | 209 | // Create a path and any pieces required, think mkdir -p. 210 | // Intermediate znodes are always created empty. 211 | func CreateRecursive(zconn zkhelper.Conn, zkPath, value string, flags int, aclv []topo.ACL) (pathCreated string, err error) { 212 | parts := strings.Split(zkPath, "/") 213 | if parts[1] != zkhelper.MagicPrefix { 214 | return "", fmt.Errorf("zkutil: non /%v path: %v", zkhelper.MagicPrefix, zkPath) 215 | } 216 | 217 | pathCreated, err = zconn.Create(zkPath, []byte(value), int32(flags), aclv) 218 | 219 | if zkhelper.ZkErrorEqual(err, topo.ErrNoNode) { 220 | // Make sure that nodes are either "file" or "directory" to mirror file system 221 | // semantics. 222 | dirAclv := make([]topo.ACL, len(aclv)) 223 | for i, acl := range aclv { 224 | dirAclv[i] = acl 225 | dirAclv[i].Perms = zkhelper.PERM_DIRECTORY 226 | } 227 | _, err = CreateRecursive(zconn, os_path.Dir(zkPath), "", 0, dirAclv) 228 | if err != nil && !zkhelper.ZkErrorEqual(err, topo.ErrNodeExists) { 229 | return "", err 230 | } 231 | pathCreated, err = zconn.Create(zkPath, []byte(value), int32(flags), aclv) 232 | } 233 | return 234 | } 235 | 236 | func CreateOrUpdate(zconn zkhelper.Conn, zkPath, value string, flags int, aclv []topo.ACL, recursive bool) (pathCreated string, err error) { 237 | if recursive { 238 | pathCreated, err = CreateRecursive(zconn, zkPath, value, flags, aclv) 239 | } else { 240 | pathCreated, err = zconn.Create(zkPath, []byte(value), int32(flags), aclv) 241 | } 242 | if err != nil && zkhelper.ZkErrorEqual(err, topo.ErrNodeExists) { 243 | pathCreated = "" 244 | _, err = zconn.Set(zkPath, []byte(value), -1) 245 | } 246 | return 247 | } 248 | -------------------------------------------------------------------------------- /src/proxy/topology_test.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | //import ( 6 | // "fmt" 7 | // "utils/assert" 8 | 9 | // "net" 10 | // "testing" 11 | //) 12 | 13 | //func TestZooKeeper(t *testing.T) { 14 | 15 | // ifaces, _ := net.Interfaces() 16 | // // handle err 17 | // for _, i := range ifaces { 18 | // addrs, _ := i.Addrs() 19 | // // handle err 20 | // for _, addr := range addrs { 21 | // var ip net.IP 22 | // switch v := addr.(type) { 23 | // case *net.IPNet: 24 | // ip = v.IP 25 | // case *net.IPAddr: 26 | // ip = v.IP 27 | // } 28 | // fmt.Println("IP: ", ip) 29 | // // process IP address 30 | // } 31 | // } 32 | 33 | // // 创建一个Topology 34 | // top := NewTopology("online_service", "127.0.0.1:2181") 35 | 36 | // testPath := "/hello" 37 | // testPath = top.FullPath(testPath) 38 | // top.DeleteDir(testPath) 39 | // path, err := top.CreateDir(testPath) 40 | // fmt.Println("Full Path: ", path) 41 | // fmt.Println("Error: ", err) 42 | // assert.Must(true) 43 | 44 | // var proxyInfo map[string]interface{} = make(map[string]interface{}) 45 | 46 | // proxyInfo["rpc_front"] = "tcp://127.0.0.1:5550" 47 | 48 | // fmt.Println("ProxyInfo: ", proxyInfo) 49 | 50 | // top.SetRpcProxyData(proxyInfo) 51 | // data, err := top.GetRpcProxyData() 52 | // fmt.Println("Data: ", data, ", error: ", err) 53 | 54 | // // var endpointInfo map[string]interface{} = make(map[string]interface{}) 55 | 56 | // // endpointInfo["endpoint"] = "tcp://127.0.0.1:5555" 57 | // // top.AddServiceEndPoint("account", "server001", endpointInfo) 58 | // top.DeleteServiceEndPoint("account", "server001") 59 | 60 | // // top.SetPathData(testPath, []byte("hello")) 61 | // // var wait chan int32 = make(chan int32, 4) 62 | 63 | // // go func() { 64 | // // var evtbus chan interface{} = make(chan interface{}) 65 | // // var pathes []string 66 | // // for i := 0; i < 10; i++ { 67 | // // pathes, _ = top.WatchChildren(testPath, evtbus) 68 | // // fmt.Println("pathes: ", pathes) 69 | // // <-evtbus 70 | // // } 71 | 72 | // // wait <- 1 73 | 74 | // // pathes, _ = top.WatchChildren(testPath, evtbus) 75 | // // fmt.Println("pathes: ", pathes) 76 | // // wait <- 2 77 | // // }() 78 | 79 | // // go func() { 80 | // // for i := 0; i < 10; i++ { 81 | // // path := fmt.Sprintf("%s/node_%d", testPath, i) 82 | // // top.CreateDir(path) 83 | // // } 84 | 85 | // // wait <- 100 86 | // // }() 87 | 88 | // // go func() { 89 | // // var evtbus chan interface{} = make(chan interface{}) 90 | // // content, _ := top.WatchNode(testPath, evtbus) 91 | // // fmt.Println("--------Content: ", string(content)) 92 | // // fmt.Println("--------EvtBus: ", <-evtbus) 93 | 94 | // // content, _ = top.WatchNode(testPath, evtbus) 95 | // // fmt.Println("--------Content: ", string(content)) 96 | // // wait <- 1234 97 | // // }() 98 | 99 | // // go func() { 100 | // // top.SetPathData(testPath, []byte("world")) 101 | // // wait <- 5678 102 | 103 | // // time.Sleep(100 * time.Millisecond) 104 | 105 | // // top.SetPathData(testPath, []byte("world")) 106 | // // wait <- 0 107 | 108 | // // }() 109 | 110 | // // for i := 0; i < 3; i++ { 111 | // // fmt.Println("Waiting: ", <-wait) 112 | // // } 113 | //} 114 | -------------------------------------------------------------------------------- /src/proxy/utils.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | 4 | package proxy 5 | 6 | import ( 7 | "fmt" 8 | "net" 9 | "os" 10 | "path/filepath" 11 | "reflect" 12 | "strings" 13 | "time" 14 | "unsafe" 15 | 16 | "github.com/wfxiang08/cyutils/utils/errors" 17 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 18 | "github.com/wfxiang08/thrift_rpc_base/zkhelper" 19 | "github.com/c4pt0r/cfg" 20 | ) 21 | 22 | func InitConfig() (*cfg.Cfg, error) { 23 | configFile := os.Getenv("CODIS_CONF") 24 | if len(configFile) == 0 { 25 | configFile = "config.ini" 26 | } 27 | ret := cfg.NewCfg(configFile) 28 | if err := ret.Load(); err != nil { 29 | return nil, errors.Trace(err) 30 | } else { 31 | return ret, nil 32 | } 33 | } 34 | 35 | func InitConfigFromFile(filename string) (*cfg.Cfg, error) { 36 | ret := cfg.NewCfg(filename) 37 | if err := ret.Load(); err != nil { 38 | return nil, errors.Trace(err) 39 | } 40 | return ret, nil 41 | } 42 | 43 | // 44 | // 获取带有指定Prefix的Ip 45 | // 46 | func GetIpWithPrefix(prefix string) string { 47 | 48 | ifaces, _ := net.Interfaces() 49 | // handle err 50 | for _, i := range ifaces { 51 | addrs, _ := i.Addrs() 52 | // handle err 53 | for _, addr := range addrs { 54 | var ip net.IP 55 | switch v := addr.(type) { 56 | case *net.IPNet: 57 | ip = v.IP 58 | case *net.IPAddr: 59 | ip = v.IP 60 | } 61 | 62 | ipAddr := ip.String() 63 | // fmt.Println("ipAddr: ", ipAddr) 64 | if strings.HasPrefix(ipAddr, prefix) { 65 | return ipAddr 66 | } 67 | 68 | } 69 | } 70 | return "" 71 | } 72 | 73 | func GetZkLock(zkConn zkhelper.Conn, productName string) zkhelper.ZLocker { 74 | zkPath := fmt.Sprintf("/zk/codis/db_%s/LOCK", productName) 75 | return zkhelper.CreateMutex(zkConn, zkPath) 76 | } 77 | 78 | func GetExecutorPath() string { 79 | filedirectory := filepath.Dir(os.Args[0]) 80 | execPath, err := filepath.Abs(filedirectory) 81 | if err != nil { 82 | log.PanicErrorf(err, "get executor path failed") 83 | } 84 | return execPath 85 | } 86 | 87 | type Strings []string 88 | 89 | func (s1 Strings) Eq(s2 []string) bool { 90 | if len(s1) != len(s2) { 91 | return false 92 | } 93 | for i := 0; i < len(s1); i++ { 94 | if s1[i] != s2[i] { 95 | return false 96 | } 97 | } 98 | return true 99 | } 100 | 101 | const ( 102 | EMPTY_MSG = "" 103 | ) 104 | 105 | // 106 | // "", tail... ----> head, tail... 107 | // 将msgs拆分成为两部分, 第一部分为: head(包含路由信息);第二部分为: tails包含信息部分 108 | // 109 | func Unwrap(msgs []string) (head string, tails []string) { 110 | head = msgs[0] 111 | if len(msgs) > 1 && msgs[1] == EMPTY_MSG { 112 | tails = msgs[2:] 113 | } else { 114 | tails = msgs[1:] 115 | } 116 | return 117 | } 118 | 119 | // 将msgs中前面多余的EMPTY_MSG删除，可能是 zeromq的不同的socket的配置不匹配导致的 120 | func TrimLeftEmptyMsg(msgs []string) []string { 121 | for index, msg := range msgs { 122 | if msg != EMPTY_MSG { 123 | return msgs[index:len(msgs)] 124 | } 125 | } 126 | return msgs 127 | } 128 | 129 | // 打印zeromq中的消息，用于Debug 130 | func PrintZeromqMsgs(msgs []string, prefix string) { 131 | 132 | // fmt.Printf("Message Length: %d, Prefix: %s\n", len(msgs), prefix) 133 | // for idx, msg := range msgs { 134 | // fmt.Printf(" idx: %d, msg: %s\n", idx, msg) 135 | // } 136 | } 137 | 138 | func Copy(s string) string { 139 | var b []byte 140 | h := (*reflect.SliceHeader)(unsafe.Pointer(&b)) 141 | h.Data = (*reflect.StringHeader)(unsafe.Pointer(&s)).Data 142 | h.Len = len(s) 143 | h.Cap = len(s) 144 | 145 | return string(b) 146 | } 147 | 148 | // 判断给定的文件是否存在 149 | func FileExist(file string) bool { 150 | var err error 151 | _, err = os.Stat(file) 152 | return !os.IsNotExist(err) 153 | } 154 | 155 | // 获取给定的日期的"开始时刻", 00:00:00 156 | func StartOfDay(t time.Time) time.Time { 157 | year, month, day := t.Date() 158 | return time.Date(year, month, day, 0, 0, 0, 0, t.Location()) 159 | } 160 | 161 | // 第二天的开始时间 162 | func NextStartOfDay(t time.Time) time.Time { 163 | year, month, day := t.Date() 164 | return time.Date(year, month, day, 0, 0, 0, 0, t.Location()).AddDate(0, 0, 1) 165 | } 166 | -------------------------------------------------------------------------------- /src/proxy/utils_color.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | color "github.com/fatih/color" 7 | ) 8 | 9 | // 警告信息采用红色显示 10 | var Red = color.New(color.FgRed).SprintFunc() 11 | 12 | // 新增服务等采用绿色显示 13 | var Green = color.New(color.FgGreen).SprintFunc() 14 | 15 | var Magenta = color.New(color.FgMagenta).SprintFunc() 16 | var Cyan = color.New(color.FgCyan).SprintFunc() 17 | 18 | var Blue = color.New(color.FgBlue).SprintFunc() 19 | -------------------------------------------------------------------------------- /src/proxy/utils_config_checker.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 7 | ) 8 | 9 | type ConfigCheck func(conf *ServiceConfig) 10 | type ProxyConfigCheck func(conf *ProxyConfig) 11 | 12 | // 13 | // 一般的ThriftService的配置检测 14 | // 15 | func ConfigCheckThriftService(conf *ServiceConfig) { 16 | if conf.ProductName == "" { 17 | log.Panic("Invalid ProductName") 18 | } 19 | if conf.FrontendAddr == "" { 20 | log.Panic("Invalid FrontendAddress") 21 | } 22 | 23 | if conf.Service == "" { 24 | log.Panic("Invalid ServiceName") 25 | } 26 | 27 | if conf.ZkAddr == "" { 28 | log.Panic("Invalid zookeeper address") 29 | } 30 | } 31 | 32 | // 33 | // RPC Proxy的Config Checker 34 | // 35 | func ConfigCheckRpcProxy(conf *ProxyConfig) { 36 | if conf.ProductName == "" { 37 | log.Panic("Invalid ProductName") 38 | } 39 | if conf.ZkAddr == "" { 40 | log.Panic("Invalid zookeeper address") 41 | } 42 | if conf.ProxyAddr == "" { 43 | log.Panic("Invalid Proxy address") 44 | } 45 | } 46 | 47 | // 48 | // RPC LB的Config Checker 49 | // 50 | func ConfigCheckRpcLB(conf *ServiceConfig) { 51 | if conf.ProductName == "" { 52 | log.Panic("Invalid ProductName") 53 | } 54 | 55 | if conf.ZkAddr == "" { 56 | log.Panic("Invalid zookeeper address") 57 | } 58 | 59 | if conf.Service == "" { 60 | log.Panic("Invalid ServiceName") 61 | } 62 | 63 | if conf.BackAddr == "" { 64 | log.Panic("Invalid backend address") 65 | } 66 | if conf.FrontendAddr == "" { 67 | log.Panic("Invalid frontend address") 68 | } 69 | } 70 | -------------------------------------------------------------------------------- /src/proxy/utils_log.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | package proxy 4 | 5 | import ( 6 | log "github.com/wfxiang08/cyutils/utils/rolling_log" 7 | "os" 8 | "strings" 9 | "syscall" 10 | "time" 11 | ) 12 | 13 | // 14 | // 设置LogLevel 15 | // 16 | func SetLogLevel(level string) { 17 | var lv = log.LEVEL_INFO 18 | switch strings.ToLower(level) { 19 | case "error": 20 | lv = log.LEVEL_ERROR 21 | case "warn", "warning": 22 | lv = log.LEVEL_WARN 23 | case "debug": 24 | lv = log.LEVEL_DEBUG 25 | case "info": 26 | fallthrough 27 | default: 28 | lv = log.LEVEL_INFO 29 | } 30 | log.SetLevel(lv) 31 | log.Infof("set log level to %s", lv) 32 | } 33 | 34 | // 35 | // 设置CrashLog 36 | // 37 | func SetCrashLog(file string) { 38 | f, err := os.OpenFile(file, os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0666) 39 | if err != nil { 40 | log.InfoErrorf(err, "cannot open crash log file: %s", file) 41 | } else { 42 | syscall.Dup2(int(f.Fd()), 2) 43 | } 44 | } 45 | 46 | func FormatYYYYmmDDHHMMSS(date time.Time) string { 47 | return date.Format("2006-01-02 15:04:05") 48 | } 49 | -------------------------------------------------------------------------------- /src/proxy/zk.go: -------------------------------------------------------------------------------- 1 | //// Copyright 2015 Spring Rain Software Compnay LTD. All Rights Reserved. 2 | //// Licensed under the MIT (MIT-LICENSE.txt) license. 3 | 4 | package proxy 5 | 6 | import ( 7 | "github.com/wfxiang08/go-zookeeper/zk" 8 | "github.com/wfxiang08/thrift_rpc_base/zkhelper" 9 | log "github.com/ngaut/logging" 10 | "sync" 11 | "time" 12 | ) 13 | 14 | const retryMaxOnOps = 10 15 | 16 | type ConnBuilder interface { 17 | // Get a conn that will retry automatically when getting error caused by connection issues. 18 | // If retry can not rebuild the connection, there will be a fetal error 19 | GetSafeConn() zkhelper.Conn 20 | 21 | // Get a conn that will return error caused by connection issues 22 | // It will try to rebuild the connection after return error. 23 | GetUnsafeConn() zkhelper.Conn 24 | } 25 | 26 | type connBuilder struct { 27 | connection zkhelper.Conn 28 | builder func() (zkhelper.Conn, error) 29 | createdOn time.Time 30 | lock sync.RWMutex 31 | safeConnInstance *safeConn 32 | unsafeConnInstance *unsafeConn 33 | } 34 | 35 | func NewConnBuilder(buildFunc func() (zkhelper.Conn, error)) ConnBuilder { 36 | b := &connBuilder{ 37 | builder: buildFunc, 38 | } 39 | b.safeConnInstance = &safeConn{builder: b} 40 | b.unsafeConnInstance = &unsafeConn{builder: b} 41 | b.resetConnection() 42 | return b 43 | } 44 | 45 | func (b *connBuilder) resetConnection() { 46 | b.lock.Lock() 47 | defer b.lock.Unlock() 48 | if b.builder == nil { 49 | log.Fatal("no connection builder") 50 | } 51 | if time.Now().Before(b.createdOn.Add(time.Second)) { 52 | return 53 | } 54 | if b.connection != nil { 55 | b.connection.Close() 56 | } 57 | var err error 58 | b.connection, err = b.builder() // this is asnyc 59 | if err == nil { 60 | b.safeConnInstance.Conn = b.connection 61 | b.unsafeConnInstance.Conn = b.connection 62 | b.createdOn = time.Now() 63 | return 64 | } 65 | log.Fatal("can not build new zk session, exit") 66 | } 67 | 68 | func (b *connBuilder) GetSafeConn() zkhelper.Conn { 69 | return b.safeConnInstance 70 | } 71 | 72 | func (b *connBuilder) GetUnsafeConn() zkhelper.Conn { 73 | return b.unsafeConnInstance 74 | } 75 | 76 | type conn struct { 77 | zkhelper.Conn 78 | builder *connBuilder 79 | } 80 | 81 | type safeConn conn 82 | type unsafeConn conn 83 | 84 | func isConnectionError(e error) bool { 85 | return !zkhelper.ZkErrorEqual(zk.ErrNoNode, e) && !zkhelper.ZkErrorEqual(zk.ErrNodeExists, e) && 86 | !zkhelper.ZkErrorEqual(zk.ErrNodeExists, e) && !zkhelper.ZkErrorEqual(zk.ErrNotEmpty, e) 87 | } 88 | 89 | func (c *safeConn) Get(path string) (data []byte, stat zk.Stat, err error) { 90 | for i := 0; i <= retryMaxOnOps; i++ { 91 | c.builder.lock.RLock() 92 | data, stat, err = c.Conn.Get(path) 93 | c.builder.lock.RUnlock() 94 | if err == nil || !isConnectionError(err) { 95 | return 96 | } 97 | c.builder.resetConnection() 98 | } 99 | log.Warning(err) 100 | log.Fatal("zk error after retries") 101 | return 102 | } 103 | 104 | func (c *safeConn) GetW(path string) (data []byte, stat zk.Stat, watch <-chan zk.Event, err error) { 105 | for i := 0; i <= retryMaxOnOps; i++ { 106 | c.builder.lock.RLock() 107 | data, stat, watch, err = c.Conn.GetW(path) 108 | c.builder.lock.RUnlock() 109 | if err == nil || !isConnectionError(err) { 110 | return 111 | } 112 | c.builder.resetConnection() 113 | } 114 | log.Warning(err) 115 | log.Fatal("zk error after retries") 116 | return 117 | } 118 | 119 | func (c *safeConn) Children(path string) (children []string, stat zk.Stat, err error) { 120 | for i := 0; i <= retryMaxOnOps; i++ { 121 | c.builder.lock.RLock() 122 | children, stat, err = c.Conn.Children(path) 123 | c.builder.lock.RUnlock() 124 | if err == nil || !isConnectionError(err) { 125 | return 126 | } 127 | c.builder.resetConnection() 128 | } 129 | log.Warning(err) 130 | log.Fatal("zk error after retries") 131 | return 132 | } 133 | 134 | func (c *safeConn) ChildrenW(path string) (children []string, stat zk.Stat, watch <-chan zk.Event, err error) { 135 | for i := 0; i <= retryMaxOnOps; i++ { 136 | c.builder.lock.RLock() 137 | children, stat, watch, err = c.Conn.ChildrenW(path) 138 | c.builder.lock.RUnlock() 139 | if err == nil || !isConnectionError(err) { 140 | return 141 | } 142 | c.builder.resetConnection() 143 | } 144 | log.Warning(err) 145 | log.Fatal("zk error after retries") 146 | return 147 | } 148 | 149 | func (c *safeConn) Exists(path string) (exist bool, stat zk.Stat, err error) { 150 | for i := 0; i <= retryMaxOnOps; i++ { 151 | c.builder.lock.RLock() 152 | exist, stat, err = c.Conn.Exists(path) 153 | c.builder.lock.RUnlock() 154 | if err == nil || !isConnectionError(err) { 155 | return 156 | } 157 | c.builder.resetConnection() 158 | } 159 | log.Warning(err) 160 | log.Fatal("zk error after retries") 161 | return 162 | } 163 | 164 | func (c *safeConn) ExistsW(path string) (exist bool, stat zk.Stat, watch <-chan zk.Event, err error) { 165 | for i := 0; i <= retryMaxOnOps; i++ { 166 | c.builder.lock.RLock() 167 | exist, stat, watch, err = c.Conn.ExistsW(path) 168 | c.builder.lock.RUnlock() 169 | if err == nil || !isConnectionError(err) { 170 | return 171 | } 172 | c.builder.resetConnection() 173 | } 174 | log.Warning(err) 175 | log.Fatal("zk error after retries") 176 | return 177 | } 178 | 179 | func (c *safeConn) Create(path string, value []byte, flags int32, aclv []zk.ACL) (pathCreated string, err error) { 180 | for i := 0; i <= retryMaxOnOps; i++ { 181 | c.builder.lock.RLock() 182 | pathCreated, err = c.Conn.Create(path, value, flags, aclv) 183 | c.builder.lock.RUnlock() 184 | if err == nil || !isConnectionError(err) { 185 | return 186 | } 187 | c.builder.resetConnection() 188 | } 189 | log.Warning(err) 190 | log.Fatal("zk error after retries") 191 | return 192 | } 193 | 194 | func (c *safeConn) Set(path string, value []byte, version int32) (stat zk.Stat, err error) { 195 | for i := 0; i <= retryMaxOnOps; i++ { 196 | c.builder.lock.RLock() 197 | stat, err = c.Conn.Set(path, value, version) 198 | c.builder.lock.RUnlock() 199 | if err == nil || !isConnectionError(err) { 200 | return 201 | } 202 | c.builder.resetConnection() 203 | } 204 | log.Warning(err) 205 | log.Fatal("zk error after retries") 206 | return 207 | } 208 | 209 | func (c *safeConn) Delete(path string, version int32) (err error) { 210 | for i := 0; i <= retryMaxOnOps; i++ { 211 | c.builder.lock.RLock() 212 | err = c.Conn.Delete(path, version) 213 | c.builder.lock.RUnlock() 214 | if err == nil || !isConnectionError(err) { 215 | return 216 | } 217 | c.builder.resetConnection() 218 | } 219 | log.Warning(err) 220 | log.Fatal("zk error after retries") 221 | return 222 | } 223 | 224 | func (c *safeConn) Close() { 225 | log.Fatal("do not close zk connection by yourself") 226 | } 227 | 228 | func (c *safeConn) GetACL(path string) (acl []zk.ACL, stat zk.Stat, err error) { 229 | for i := 0; i <= retryMaxOnOps; i++ { 230 | c.builder.lock.RLock() 231 | acl, stat, err = c.Conn.GetACL(path) 232 | c.builder.lock.RUnlock() 233 | if err == nil || !isConnectionError(err) { 234 | return 235 | } 236 | c.builder.resetConnection() 237 | } 238 | log.Warning(err) 239 | log.Fatal("zk error after retries") 240 | return 241 | } 242 | 243 | func (c *safeConn) SetACL(path string, aclv []zk.ACL, version int32) (stat zk.Stat, err error) { 244 | for i := 0; i <= retryMaxOnOps; i++ { 245 | c.builder.lock.RLock() 246 | stat, err = c.Conn.SetACL(path, aclv, version) 247 | c.builder.lock.RUnlock() 248 | if err == nil || !isConnectionError(err) { 249 | return 250 | } 251 | c.builder.resetConnection() 252 | } 253 | log.Warning(err) 254 | log.Fatal("zk error after retries") 255 | return 256 | } 257 | 258 | func (c *safeConn) Seq2Str(seq int64) string { 259 | return c.Conn.Seq2Str(seq) 260 | } 261 | 262 | func (c *unsafeConn) Get(path string) (data []byte, stat zk.Stat, err error) { 263 | c.builder.lock.RLock() 264 | data, stat, err = c.Conn.Get(path) 265 | c.builder.lock.RUnlock() 266 | if err != nil && isConnectionError(err) { 267 | go c.builder.resetConnection() 268 | } 269 | return 270 | } 271 | 272 | func (c *unsafeConn) GetW(path string) (data []byte, stat zk.Stat, watch <-chan zk.Event, err error) { 273 | c.builder.lock.RLock() 274 | data, stat, watch, err = c.Conn.GetW(path) 275 | c.builder.lock.RUnlock() 276 | if err != nil && isConnectionError(err) { 277 | go c.builder.resetConnection() 278 | } 279 | return 280 | } 281 | 282 | func (c *unsafeConn) Children(path string) (children []string, stat zk.Stat, err error) { 283 | c.builder.lock.RLock() 284 | children, stat, err = c.Conn.Children(path) 285 | c.builder.lock.RUnlock() 286 | if err != nil && isConnectionError(err) { 287 | go c.builder.resetConnection() 288 | } 289 | return 290 | } 291 | 292 | func (c *unsafeConn) ChildrenW(path string) (children []string, stat zk.Stat, watch <-chan zk.Event, err error) { 293 | c.builder.lock.RLock() 294 | children, stat, watch, err = c.Conn.ChildrenW(path) 295 | c.builder.lock.RUnlock() 296 | if err != nil && isConnectionError(err) { 297 | go c.builder.resetConnection() 298 | } 299 | return 300 | } 301 | 302 | func (c *unsafeConn) Exists(path string) (exist bool, stat zk.Stat, err error) { 303 | c.builder.lock.RLock() 304 | exist, stat, err = c.Conn.Exists(path) 305 | c.builder.lock.RUnlock() 306 | if err != nil && isConnectionError(err) { 307 | go c.builder.resetConnection() 308 | } 309 | return 310 | } 311 | 312 | func (c *unsafeConn) ExistsW(path string) (exist bool, stat zk.Stat, watch <-chan zk.Event, err error) { 313 | c.builder.lock.RLock() 314 | exist, stat, watch, err = c.Conn.ExistsW(path) 315 | c.builder.lock.RUnlock() 316 | if err != nil && isConnectionError(err) { 317 | go c.builder.resetConnection() 318 | } 319 | return 320 | } 321 | 322 | func (c *unsafeConn) Create(path string, value []byte, flags int32, aclv []zk.ACL) (pathCreated string, err error) { 323 | c.builder.lock.RLock() 324 | pathCreated, err = c.Conn.Create(path, value, flags, aclv) 325 | c.builder.lock.RUnlock() 326 | if err != nil && isConnectionError(err) { 327 | go c.builder.resetConnection() 328 | } 329 | return 330 | } 331 | 332 | func (c *unsafeConn) Set(path string, value []byte, version int32) (stat zk.Stat, err error) { 333 | c.builder.lock.RLock() 334 | stat, err = c.Conn.Set(path, value, version) 335 | c.builder.lock.RUnlock() 336 | if err != nil && isConnectionError(err) { 337 | go c.builder.resetConnection() 338 | } 339 | return 340 | } 341 | 342 | func (c *unsafeConn) Delete(path string, version int32) (err error) { 343 | c.builder.lock.RLock() 344 | err = c.Conn.Delete(path, version) 345 | c.builder.lock.RUnlock() 346 | if err != nil && isConnectionError(err) { 347 | go c.builder.resetConnection() 348 | } 349 | return 350 | } 351 | 352 | func (c *unsafeConn) Close() { 353 | log.Fatal("do not close zk connection by yourself") 354 | } 355 | 356 | func (c *unsafeConn) GetACL(path string) (acl []zk.ACL, stat zk.Stat, err error) { 357 | c.builder.lock.RLock() 358 | acl, stat, err = c.Conn.GetACL(path) 359 | c.builder.lock.RUnlock() 360 | if err != nil && isConnectionError(err) { 361 | go c.builder.resetConnection() 362 | } 363 | return 364 | } 365 | 366 | func (c *unsafeConn) SetACL(path string, aclv []zk.ACL, version int32) (stat zk.Stat, err error) { 367 | c.builder.lock.RLock() 368 | stat, err = c.Conn.SetACL(path, aclv, version) 369 | c.builder.lock.RUnlock() 370 | if err != nil && isConnectionError(err) { 371 | go c.builder.resetConnection() 372 | } 373 | return 374 | } 375 | 376 | func (c *unsafeConn) Seq2Str(seq int64) string { 377 | return c.Conn.Seq2Str(seq) 378 | } 379 | -------------------------------------------------------------------------------- /src/rebuild_vendor.sh: -------------------------------------------------------------------------------- 1 | python _rebuild_vendor.py `which go` 2 | # 查看rebuild的结果 3 | tree ../pkg/ -------------------------------------------------------------------------------- /src/scripts/README.md: -------------------------------------------------------------------------------- 1 | # rpc_proxy的系统脚本 2 | 工作目录: ~/goproject/rpc_proxy/src 3 | 4 | ## 2. centos 7 5 | 6 | * 同时将测试和线上系统服务部署到目标机器: 7 | 8 | * bash scripts/deploy_proxy.sh 9 | * 将rpc_proxy, rpc_lb拷贝到目标机器(包括localhost)的目录下: /usr/local/rpc_proxy/bin/ 10 | * 将配置文件拷贝到: /usr/local/rpc_proxy/目录下 11 | * config.ucloud-online.ini --> config.online.ini 12 | * config.ucloud-test.ini --> config.test.ini 13 | * 拷贝systemctl脚本: 14 | * rpc_proxy_online.service --> /lib/systemd/system/rpc_proxy_online.service 15 | * rpc_proxy_test.service --> /lib/systemd/system/rpc_proxy_test.service 16 | * 默认不启动rpc_proxy服务 17 | * systemctl daemon-reload 18 | * systemctl enable rpc_proxy_online.service 19 | * systemctl enable rpc_proxy_test.service 20 | * enable安装脚本依赖关系，如果enable之后，下次启动时会自动开启 21 | * systemctl disable rpc_proxy_online.service 22 | * systemctl disable rpc_proxy_test.service 23 | * disable安装脚本依赖关系，如果disable之后，下次启动时不会自动开启 24 | * systemctl start|stop|status|restart rpc_proxy_test.service 25 | * 在enable的情况下，开启动服务 26 | 27 | * rpc_proxy启停的消息，或者异常退出的消息，通过: /var/log/messages 来查看 28 | -------------------------------------------------------------------------------- /src/scripts/RpcThrift.Services.thrift: -------------------------------------------------------------------------------- 1 | namespace php RpcThrift.Services 2 | 3 | exception RpcException { 4 | 1: i32 code, 5 | 2: string msg 6 | } 7 | 8 | service RpcServiceBase { 9 | void ping(); 10 | } -------------------------------------------------------------------------------- /src/scripts/build_lb.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | go build -ldflags "-X main.buildDate=`date +%Y%m%d%H%M%S` -X main.gitVersion=`git rev-parse HEAD`" cmds/service_rpc_lb.go 3 | -------------------------------------------------------------------------------- /src/scripts/build_proxy.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | go build -ldflags "-X main.buildDate=`date +%Y%m%d%H%M%S` -X main.gitVersion=`git rev-parse HEAD`" cmds/service_rpc_proxy.go -------------------------------------------------------------------------------- /src/scripts/config.local.ini: -------------------------------------------------------------------------------- 1 | # 使用本机提供的zk来测试RPC服务 2 | zk=127.0.0.1:2181 3 | product=test 4 | verbose=1 5 | 6 | zk_session_timeout=30 7 | rpc_timeout=15 8 | service=account 9 | front_host=127.0.0.1 10 | front_port=5555 11 | # back_address=127.0.0.1:5556 12 | back_address=run/typo_backend.sock 13 | 14 | # 使用网络的IP, 如果没有指定front_host, 则使用使用当前机器的内网的Ip来注册 15 | ip_prefix=10. 16 | 17 | worker_pool_size=2 18 | 19 | # proxy_address=127.0.0.1:5550 20 | proxy_address=/usr/local/rpc_proxy/proxy.sock 21 | 22 | profile=0 23 | falcon_client=http://127.0.0.1:1988/v1/push -------------------------------------------------------------------------------- /src/scripts/config.test.ini: -------------------------------------------------------------------------------- 1 | # 使用本机提供的zk来测试RPC服务 2 | zk=m1:2181 3 | product=test 4 | verbose=1 5 | 6 | zk_session_timeout=30 7 | 8 | rpc_timeout=15 9 | 10 | service= 11 | front_host= 12 | front_port= 13 | # back_address=127.0.0.1:5556 14 | # back_address=run/typo_backend.sock 15 | 16 | # 使用网络的IP, 如果没有指定front_host, 则使用使用当前机器的内网的Ip来注册 17 | ip_prefix=172.20. 18 | 19 | worker_pool_size=2 20 | 21 | # proxy_address=127.0.0.1:5550 22 | proxy_address=/usr/local/rpc_proxy/proxy.sock 23 | 24 | # falcon_client=http://127.0.0.1:1988/v1/push 25 | 26 | profile=0 -------------------------------------------------------------------------------- /src/scripts/control_lb.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # control_lb脚本必须和具体的产品项目放在一起 4 | WORKSPACE=$(cd $(dirname $0)/; pwd) 5 | cd $WORKSPACE 6 | # 确保log目录存在(相对产品放在一起) 7 | mkdir -p log 8 | conf=config.ini 9 | logfile=log/lb.log 10 | pidfile=log/app_lb.pid 11 | stdfile=log/app_lb.log 12 | 13 | 14 | BASE_DIR=/usr/local/rpc_proxy/ 15 | app=${BASE_DIR}bin/service_rpc_lb 16 | 17 | 18 | function check_pid() { 19 | if [ -f $pidfile ];then 20 | pid=`cat $pidfile` 21 | if [ -n $pid ]; then 22 | running=`ps -p $pid|grep -v "PID TTY" |wc -l` 23 | return $running 24 | fi 25 | fi 26 | return 0 27 | } 28 | 29 | function start() { 30 | check_pid 31 | running=$? 32 | if [ $running -gt 0 ];then 33 | echo -n "$app now is running already, pid=" 34 | cat $pidfile 35 | return 1 36 | fi 37 | 38 | if ! [ -f $conf ];then 39 | echo "Config file $conf doesn't exist, creating one." 40 | exit -1 41 | fi 42 | 43 | GIT_URL=`git config --get remote.origin.url` 44 | GIT_VERSION=`git rev-parse --short HEAD` 45 | 46 | # 每次重启之后 stdfile被覆盖 47 | nohup $app -c $conf -L $logfile --work-dir=${WORKSPACE} --code-url-version="${GIT_URL}@${GIT_VERSION}" &> $stdfile & 48 | echo $! > $pidfile 49 | echo "$app started..., pid=$!" 50 | } 51 | 52 | function stop() { 53 | check_pid 54 | running=$? 55 | # 由于lb等需要graceful stop, 因此stop过程需要等待 56 | if [ $running -gt 0 ];then 57 | pid=`cat $pidfile` 58 | kill -15 $pid 59 | status="0" 60 | while [ "$status" == "0" ];do 61 | echo "Waiting for process ${pid} ..." 62 | sleep 1 63 | ps -p$pid 2>&1 > /dev/null 64 | status=$? 65 | done 66 | echo "$app stoped..." 67 | else 68 | echo "$app already stoped..." 69 | fi 70 | } 71 | 72 | function restart() { 73 | stop 74 | sleep 1 75 | start 76 | } 77 | 78 | # 查看当前的进程的状态 79 | function status() { 80 | check_pid 81 | running=$? 82 | if [ $running -gt 0 ];then 83 | echo started 84 | else 85 | echo stoped 86 | fi 87 | } 88 | 89 | # 查看最新的Log文件 90 | function tailf() { 91 | date=`date +"%Y%m%d"` 92 | tail -Fn 200 "${logfile}-${date}" 93 | } 94 | 95 | 96 | function help() { 97 | echo "$0 start|stop|restart|status|tail" 98 | } 99 | 100 | if [ "$1" == "" ]; then 101 | help 102 | elif [ "$1" == "stop" ];then 103 | stop 104 | elif [ "$1" == "start" ];then 105 | start 106 | elif [ "$1" == "restart" ];then 107 | restart 108 | elif [ "$1" == "status" ];then 109 | status 110 | elif [ "$1" == "tail" ];then 111 | tailf 112 | else 113 | help 114 | fi 115 | -------------------------------------------------------------------------------- /src/scripts/control_proxy.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | 4 | # 由于proxy基本上是每个机器上有一个拷贝，并且和项目无关，因此放在一个固定地方 5 | BASE_DIR=/usr/local/rpc_proxy/ 6 | mkdir -p ${BASE_DIR}log 7 | 8 | cd ${BASE_DIR} 9 | 10 | app=${BASE_DIR}bin/service_rpc_proxy 11 | conf=${BASE_DIR}config.ini 12 | 13 | logfile=${BASE_DIR}log/proxy.log 14 | pidfile=${BASE_DIR}log/proxy.pid 15 | stdfile=${BASE_DIR}log/app.log 16 | 17 | # sock_file=/var/log/rpc_proxy/rpc_proxy.sock 18 | 19 | function check_pid() { 20 | if [ -f $pidfile ];then 21 | pid=`cat $pidfile` 22 | if [ -n $pid ]; then 23 | running=`ps -p $pid|grep -v "PID TTY" |wc -l` 24 | return $running 25 | fi 26 | fi 27 | return 0 28 | } 29 | 30 | function start() { 31 | check_pid 32 | running=$? 33 | if [ $running -gt 0 ];then 34 | echo -n "$app now is running already, pid=" 35 | cat $pidfile 36 | return 1 37 | fi 38 | 39 | if ! [ -f $conf ];then 40 | echo "Config file $conf doesn't exist, creating one." 41 | exit -1 42 | fi 43 | 44 | nohup $app -c $conf -L $logfile &> $stdfile & 45 | echo $! > $pidfile 46 | echo "$app started..., pid=$!" 47 | } 48 | 49 | function stop() { 50 | pid=`cat $pidfile` 51 | kill -15 $pid 52 | echo "$app stoped..." 53 | } 54 | 55 | function restart() { 56 | stop 57 | sleep 1 58 | start 59 | } 60 | 61 | function status() { 62 | check_pid 63 | running=$? 64 | if [ $running -gt 0 ];then 65 | echo started 66 | else 67 | echo stoped 68 | fi 69 | } 70 | 71 | function tailf() { 72 | date=`date +"%Y%m%d"` 73 | tail -Fn 200 "${logfile}-${date}" 74 | } 75 | 76 | 77 | function help() { 78 | echo "$0 start|stop|restart|status|tail" 79 | } 80 | 81 | if [ "$1" == "" ]; then 82 | help 83 | elif [ "$1" == "stop" ];then 84 | stop 85 | elif [ "$1" == "start" ];then 86 | start 87 | elif [ "$1" == "restart" ];then 88 | restart 89 | elif [ "$1" == "status" ];then 90 | status 91 | elif [ "$1" == "tail" ];then 92 | tailf 93 | else 94 | help 95 | fi 96 | -------------------------------------------------------------------------------- /src/scripts/control_proxy_profile.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | 4 | # 由于proxy基本上是每个机器上有一个拷贝，并且和项目无关，因此放在一个固定地方 5 | BASE_DIR=/usr/local/rpc_proxy/ 6 | mkdir -p ${BASE_DIR}log 7 | 8 | cd ${BASE_DIR} 9 | 10 | app=${BASE_DIR}bin/service_rpc_proxy 11 | conf=${BASE_DIR}config.ini 12 | 13 | logfile=${BASE_DIR}log/proxy.log 14 | pidfile=${BASE_DIR}log/proxy.pid 15 | stdfile=${BASE_DIR}log/app.log 16 | 17 | # sock_file=/var/log/rpc_proxy/rpc_proxy.sock 18 | 19 | function check_pid() { 20 | if [ -f $pidfile ];then 21 | pid=`cat $pidfile` 22 | if [ -n $pid ]; then 23 | running=`ps -p $pid|grep -v "PID TTY" |wc -l` 24 | return $running 25 | fi 26 | fi 27 | return 0 28 | } 29 | 30 | function start() { 31 | check_pid 32 | running=$? 33 | if [ $running -gt 0 ];then 34 | echo -n "$app now is running already, pid=" 35 | cat $pidfile 36 | return 1 37 | fi 38 | 39 | if ! [ -f $conf ];then 40 | echo "Config file $conf doesn't exist, creating one." 41 | exit -1 42 | fi 43 | 44 | nohup $app -c $conf -L $logfile --profile-addr=60.29.249.103:7171 &> $stdfile & 45 | echo $! > $pidfile 46 | echo "$app started..., pid=$!" 47 | } 48 | 49 | function stop() { 50 | pid=`cat $pidfile` 51 | kill -15 $pid 52 | echo "$app stoped..." 53 | } 54 | 55 | function restart() { 56 | stop 57 | sleep 1 58 | start 59 | } 60 | 61 | function status() { 62 | check_pid 63 | running=$? 64 | if [ $running -gt 0 ];then 65 | echo started 66 | else 67 | echo stoped 68 | fi 69 | } 70 | 71 | function tailf() { 72 | date=`date +"%Y%m%d"` 73 | tail -Fn 200 "${logfile}-${date}" 74 | } 75 | 76 | 77 | function help() { 78 | echo "$0 start|stop|restart|status|tail" 79 | } 80 | 81 | if [ "$1" == "" ]; then 82 | help 83 | elif [ "$1" == "stop" ];then 84 | stop 85 | elif [ "$1" == "start" ];then 86 | start 87 | elif [ "$1" == "restart" ];then 88 | restart 89 | elif [ "$1" == "status" ];then 90 | status 91 | elif [ "$1" == "tail" ];then 92 | tailf 93 | else 94 | help 95 | fi 96 | -------------------------------------------------------------------------------- /src/scripts/deploy_proxy.sh: -------------------------------------------------------------------------------- 1 | if [ "$#" -ne 1 ]; then 2 | echo "Please input hostname" 3 | exit -1 4 | fi 5 | 6 | host_name=$1 7 | 8 | # 创建目录，拷贝rpc_proxy/rpc_lb 9 | ssh root@${host_name} "mkdir -p /usr/local/rpc_proxy/bin/" 10 | ssh root@${host_name} "mkdir -p /usr/local/rpc_proxy/log/" 11 | 12 | # 拷贝: rpc_lb 13 | ssh root@${host_name} "rm -f /usr/local/rpc_proxy/bin/rpc_lb" 14 | scp rpc_lb root@${host_name}:/usr/local/rpc_proxy/bin/rpc_lb 15 | 16 | # 拷贝: rpc_proxy 17 | ssh root@${host_name} "rm -f /usr/local/rpc_proxy/bin/rpc_proxy" 18 | scp rpc_proxy root@${host_name}:/usr/local/rpc_proxy/bin/rpc_proxy 19 | 20 | # 拷贝脚本 21 | scp scripts/control_lb.sh root@${host_name}:/usr/local/rpc_proxy/ 22 | scp scripts/control_proxy.sh root@${host_name}:/usr/local/rpc_proxy/ 23 | 24 | # 同时拷贝测试和线上配置 25 | scp scripts/rpc_proxy_test.service root@${host_name}:/lib/systemd/system/ 26 | scp scripts/rpc_proxy_online.service root@${host_name}:/lib/systemd/system/ 27 | 28 | 29 | echo "请选择启动测试服务rpc或线上服务rpc(最好不要都启动)" 30 | echo "ssh root@${host_name} systemctl daemon-reload" 31 | echo "ssh root@${host_name} systemctl enable rpc_proxy_online.service" 32 | echo "ssh root@${host_name} systemctl start rpc_proxy_online.service" 33 | echo "---" 34 | echo "ssh root@${host_name} systemctl daemon-reload" 35 | echo "ssh root@${host_name} systemctl enable rpc_proxy_test.service" 36 | echo "ssh root@${host_name} systemctl start rpc_proxy_test.service" 37 | 38 | -------------------------------------------------------------------------------- /src/scripts/idl_update.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | thrift -r --gen go:thrift_import="github.com/wfxiang08/go_thrift/thrift" scripts/RpcThrift.Services.thrift -------------------------------------------------------------------------------- /src/scripts/idl_update_python.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | thrift -r --gen py RpcThrift.Services.thrift.thrift 3 | -------------------------------------------------------------------------------- /src/scripts/lb-config.online.ini: -------------------------------------------------------------------------------- 1 | # 三台Master的Ip 2 | zk=172.20.2.139:2181 3 | 4 | # 线上的product一律以: online开头 5 | product=online_api 6 | verbose=1 7 | rpc_timeout=15 8 | zk_session_timeout=30 9 | 10 | service= 11 | front_host= 12 | front_port= 13 | # back_address=127.0.0.1:5556 14 | # back_address=run/typo_backend.sock 15 | 16 | # 使用网络的IP, 如果没有指定front_host, 则使用使用当前机器的内网的Ip来注册 17 | ip_prefix=172.20. 18 | 19 | worker_pool_size=2 20 | 21 | # proxy_address=127.0.0.1:5550 22 | proxy_address=/usr/local/rpc_proxy/online_proxy.sock 23 | 24 | # 监控 25 | falcon_client=http://127.0.0.1:1988/v1/push 26 | profile=0 -------------------------------------------------------------------------------- /src/scripts/proxy-config.online.ini: -------------------------------------------------------------------------------- 1 | # 三台Master的Ip 2 | zk=172.20.2.139:2181 3 | 4 | # 线上的product一律以: online开头 5 | product=online_api 6 | verbose=1 7 | rpc_timeout=15 8 | zk_session_timeout=30 9 | 10 | proxy_address=/usr/local/rpc_proxy/online_proxy.sock 11 | profile=0 -------------------------------------------------------------------------------- /src/scripts/rpc_proxy_online.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=rpc_proxy online version 3 | After=network.target 4 | 5 | [Service] 6 | ExecStart=/usr/local/rpc_proxy/bin/service_rpc_proxy -c /usr/local/rpc_proxy/proxy-config.online.ini -L /usr/local/rpc_proxy/log/proxy-online.log 7 | 8 | User=worker 9 | Group=worker 10 | 11 | [Install] 12 | WantedBy=multi-user.target 13 | -------------------------------------------------------------------------------- /src/scripts/rpc_proxy_test.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=rpc_proxy test version 3 | After=network.target 4 | 5 | [Service] 6 | ExecStart=/usr/local/rpc_proxy/bin/rpc_proxy -c /usr/local/rpc_proxy/config.test.ini -L /usr/local/rpc_proxy/log/proxy-test.log 7 | 8 | User=root 9 | Group=root 10 | 11 | [Install] 12 | WantedBy=multi-user.target 13 | -------------------------------------------------------------------------------- /src/scripts/scp_lb.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | if [ "$#" -ne 1 ]; then 3 | echo "Please input hostname" 4 | exit -1 5 | fi 6 | 7 | host_name=$1 8 | 9 | ssh root@${host_name} "mkdir -p /usr/local/rpc_proxy/bin/" 10 | ssh root@${host_name} "mkdir -p /usr/local/rpc_proxy/log/" 11 | 12 | ssh root@${host_name} "rm -f /usr/local/rpc_proxy/bin/service_rpc_lb" 13 | scp service_rpc_lb root@${host_name}:/usr/local/rpc_proxy/bin/service_rpc_lb 14 | ssh root@${host_name} "chown -R worker.worker /usr/local/rpc_proxy/" 15 | 16 | -------------------------------------------------------------------------------- /src/scripts/scp_proxy.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | if [ "$#" -ne 1 ]; then 3 | echo "Please input hostname" 4 | exit -1 5 | fi 6 | 7 | host_name=$1 8 | 9 | ssh root@${host_name} "mkdir -p /usr/local/rpc_proxy/bin/" 10 | ssh root@${host_name} "mkdir -p /usr/local/rpc_proxy/log/" 11 | 12 | ssh root@${host_name} "rm -f /usr/local/rpc_proxy/bin/service_rpc_proxy" 13 | scp service_rpc_proxy root@${host_name}:/usr/local/rpc_proxy/bin/service_rpc_proxy 14 | scp scripts/proxy-config.online.ini root@${host_name}:/usr/local/rpc_proxy/proxy-config.online.ini 15 | 16 | ssh root@${host_name} "chown -R worker.worker /usr/local/rpc_proxy/" 17 | 18 | 19 | # 拷贝systemctl 20 | scp scripts/rpc_proxy_online.service root@${host_name}:/lib/systemd/system/rpc_proxy_online.service 21 | 22 | # 启动服务 23 | ssh root@${host_name} "systemctl daemon-reload" 24 | # ssh root@${host_name} "if systemctl is-active rpc_proxy_online.service; then systemctl reload rpc_proxy_online; else systemctl start rpc_proxy_online; fi" 25 | ssh root@${host_name} "systemctl restart rpc_proxy_online.service" 26 | -------------------------------------------------------------------------------- /src/start_env.sh: -------------------------------------------------------------------------------- 1 | PWD=`pwd` 2 | export GOPATH=$(cd $PWD/..; pwd) 3 | export GOBIN=$GOPATH/bin 4 | echo "----------------" 5 | echo "\$GOPATH = ${GOPATH}" 6 | echo "----------------" -------------------------------------------------------------------------------- /src/sync.sh: -------------------------------------------------------------------------------- 1 | rsync -avp --exclude=.git --exclude=vendor --exclude=.idea --exclude='/tool_*' --exclude='/task_*' --exclude='/service_*' . media1:/root/workspace/rpc_proxy/src 2 | --------------------------------------------------------------------------------