├── .gitignore ├── Dockerfile ├── README.md ├── glide.lock ├── glide.yaml ├── main.go └── myhelloworld.yaml /.gitignore: -------------------------------------------------------------------------------- 1 | vendor/ 2 | .DS_store 3 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | FROM golang:1.8.3-onbuild 2 | 3 | EXPOSE 8080 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # When Istio Meets Jaeger - An Example of End-to-end Distributed Tracing 2 | 3 | Kubernetes is great! It helps many engineering teams to realize the dream of SOA (Service Oriented Architecture). For the longest time, we build our applications around the concept of monolith mindset, which is essentially having a large computational instance running all services provided in an application. Things like account management, billing, report generation are all running from a shared resource. This worked pretty well until SOA came along and promised us a much brighter future. By breaking down applications to smaller components, and having them to talk to each other using REST or gRPC. We ~~hope~~ expect things will only get better from there but only to realize a new set of challenges awaits. How about cross services communication? How about observability between microservices such as logging or tracing? This post demonstrates how to set up OpenTracing inside a Kubernetes cluster that enables end-to-end tracing between services, and inside the service itself with a right instrumentation. 4 | 5 | # Kubernetes Cluster Setup 6 | First thing first, we will need to have a working Kubernetes cluster to play with. I use [kops](https://github.com/kubernetes/kops) on AWS because it offers an array of automated cluster operations like upgrade, scaling up/down and multiple instance groups. In addition to the handy cluster opertations, the kops team has been tailing latest Kubernetes version closely to provide up-to-date cluster experience. I found that very cool :D 7 | 8 | To get kops up and running there are [some steps](https://github.com/kubernetes/kops/blob/master/docs/aws.md) to follow. 9 | 10 | ## Create Cluster 11 | Creating a Kubernetes cluster with kops on AWS could be as simple as an one [cluster create command](https://github.com/kubernetes/kops/blob/master/docs/cli/kops_create_cluster.md). 12 | 13 | ``` 14 | kops create cluster \ 15 | --name steven.buffer-k8s.com \ 16 | --cloud aws \ 17 | --master-size t2.medium \ 18 | --master-zones=us-east-1b \ 19 | --node-size m5.large \ 20 | --zones=us-east-1a,us-east-1b,us-east-1c,us-east-1d,us-east-1e,us-east-1f \ 21 | --node-count=2 \ 22 | --kubernetes-version=1.8.6 \ 23 | --vpc=vpc-1234567a \ 24 | --network-cidr=10.0.0.0/16 \ 25 | --networking=flannel \ 26 | --authorization=RBAC \ 27 | --ssh-public-key="~/.ssh/kube_aws_rsa.pub" \ 28 | --yes 29 | ``` 30 | 31 | This command tells AWS to create a Kubernetes cluster with 1 master node and 2 minion (worker) nodes in `us-east-1` on VPC `1234567a` using CIDR `10.0.0.0/16`. It will take around 10 minutes for the cluster to spin up and ready to use. In the meantime you could use `watch kubectl get nodes` to monitor the progress. 32 | 33 | Once completed, we should be ready to install [Istio](https://istio.io/) on the new Kubernetes cluster. It's a service mesh that manages traffic between services running on the same cluster. Because of this nature it makes Istio a perfect candidate to trace requests across services. 34 | 35 | # Install Istio 36 | Download Istio from their [GitHub repo](https://github.com/istio/istio/releases/tag/0.4.0). 37 | 38 | From the downloaded Istio directory you could install Istio to the Kubernetes cluster with this command 39 | 40 | `kubectl apply -f install/kubernetes/istio.yaml` 41 | 42 | Now Istio should be up and running on the cluster. It aslo creates a Nginx Ingress Controller that takes external requests. We will include how to set ip up later. 43 | 44 | # Install Jaeger 45 | Jaeger and Instio work side by side to provide tracing across services. You could insall Jaeger with this command 46 | 47 | `kubectl create -n istio-system -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/all-in-one/jaeger-all-in-one-template.yml` 48 | 49 | After completed, you should be able to access the Jaeger UI. Viola! 50 | 51 | ![Jaeger](https://dha4w82d62smt.cloudfront.net/items/3v2P1z1S1R243e2o2y3f/Image%202018-01-27%20at%2012.39.25%20PM.png?X-CloudApp-Visitor-Id=2456693) 52 | 53 | # Instrument Code 54 | After installing Jaeger and Istio you will be able to see cross services traces automajically! This is because Envoy sidecars injected by Istio handle inter-service traffic, while the deployed application only talks to the assigned sidecar. 55 | 56 | You can find my [GitHub repo](https://github.com/stevenc81/jaeger-tracing-example) with a small sample application. 57 | 58 | `main.go` looks like this 59 | 60 | ``` 61 | package main 62 | 63 | import ( 64 | "fmt" 65 | "log" 66 | "net/http" 67 | "runtime" 68 | "time" 69 | 70 | "github.com/opentracing-contrib/go-stdlib/nethttp" 71 | opentracing "github.com/opentracing/opentracing-go" 72 | jaeger "github.com/uber/jaeger-client-go" 73 | "github.com/uber/jaeger-client-go/zipkin" 74 | ) 75 | 76 | func indexHandler(w http.ResponseWriter, r *http.Request) { 77 | fmt.Fprintf(w, "hello world, I'm running on %s with an %s CPU ", runtime.GOOS, runtime.GOARCH) 78 | } 79 | 80 | func getTimeHandler(w http.ResponseWriter, r *http.Request) { 81 | log.Print("Received getTime request") 82 | t := time.Now() 83 | ts := t.Format("Mon Jan _2 15:04:05 2006") 84 | fmt.Fprintf(w, "The time is %s", ts) 85 | } 86 | 87 | func main() { 88 | zipkinPropagator := zipkin.NewZipkinB3HTTPHeaderPropagator() 89 | injector := jaeger.TracerOptions.Injector(opentracing.HTTPHeaders, zipkinPropagator) 90 | extractor := jaeger.TracerOptions.Extractor(opentracing.HTTPHeaders, zipkinPropagator) 91 | 92 | // Zipkin shares span ID between client and server spans; it must be enabled via the following option. 93 | zipkinSharedRPCSpan := jaeger.TracerOptions.ZipkinSharedRPCSpan(true) 94 | 95 | sender, _ := jaeger.NewUDPTransport("jaeger-agent.istio-system:5775", 0) 96 | tracer, closer := jaeger.NewTracer( 97 | "myhelloworld", 98 | jaeger.NewConstSampler(true), 99 | jaeger.NewRemoteReporter( 100 | sender, 101 | jaeger.ReporterOptions.BufferFlushInterval(1*time.Second)), 102 | injector, 103 | extractor, 104 | zipkinSharedRPCSpan, 105 | ) 106 | defer closer.Close() 107 | 108 | http.HandleFunc("/", indexHandler) 109 | http.HandleFunc("/gettime", getTimeHandler) 110 | http.ListenAndServe( 111 | ":8080", 112 | nethttp.Middleware(tracer, http.DefaultServeMux)) 113 | } 114 | 115 | ``` 116 | 117 | From line 28 - 30 we create a Zipkin propagator to tell Jaeger to capture OpenZipkin context from request headers. You might ask how these headers get to a request in the first place? Remember when I said Istio sidecar handles service communication and you application only talks to it? Yeah, you might have guessed it already. In order for Istio to trace a request between services, a set of headers are injected by Istio's Ingress Controller when a request enters the cluster. It then gets prapagated arond Envoy sidecars and each one reports the associated span to Jaeger. This helps connecting the spans to a single trace. Our application code takes advantage of these headers to collapse the inter-service with inner-service spans. 118 | 119 | Here is a list of OpenZipkin headers injected by Istio Ingress Controller 120 | 121 | ``` 122 | x-request-id 123 | x-b3-traceid 124 | x-b3-spanid 125 | x-b3-parentspanid 126 | x-b3-sampled 127 | x-b3-flags 128 | x-ot-span-context 129 | ``` 130 | 131 | To deploy the sample applciation you could use this yaml file 132 | 133 | ``` 134 | apiVersion: extensions/v1beta1 135 | kind: Deployment 136 | metadata: 137 | labels: 138 | app: myhelloworld 139 | name: myhelloworld 140 | spec: 141 | replicas: 1 142 | template: 143 | metadata: 144 | labels: 145 | app: myhelloworld 146 | spec: 147 | containers: 148 | - image: stevenc81/jaeger-tracing-example:0.1 149 | imagePullPolicy: Always 150 | name: myhelloworld 151 | ports: 152 | - containerPort: 8080 153 | restartPolicy: Always 154 | --- 155 | apiVersion: v1 156 | kind: Service 157 | metadata: 158 | name: myhelloworld 159 | labels: 160 | app: myhelloworld 161 | spec: 162 | type: NodePort 163 | ports: 164 | - port: 80 165 | targetPort: 8080 166 | name: http 167 | selector: 168 | app: myhelloworld 169 | --- 170 | apiVersion: extensions/v1beta1 171 | kind: Ingress 172 | metadata: 173 | name: myhelloworld 174 | annotations: 175 | kubernetes.io/ingress.class: "istio" 176 | spec: 177 | rules: 178 | - http: 179 | paths: 180 | - path: / 181 | backend: 182 | serviceName: myhelloworld 183 | servicePort: 80 184 | - path: /gettime 185 | backend: 186 | serviceName: myhelloworld 187 | servicePort: 80 188 | ``` 189 | 190 | # See the Traces 191 | Time to reap profit! When we send a request to the Istio Ingress Controller it will be traced between services as well as inside the application. From the screenshot we could see 3 spans reported from different places 192 | 193 | * Ingress Controller 194 | * Envoy sidecar for application 195 | * Application code 196 | 197 | ![Jaeger](https://dha4w82d62smt.cloudfront.net/items/0c0w373p3J3V1g3Q2B3J/Image%202018-01-27%20at%2012.48.23%20PM.png?X-CloudApp-Visitor-Id=2456693) 198 | 199 | Expand the trace (to show 3 spans) and see the end-to-end tracing we shall :D 200 | ![Trace](https://dha4w82d62smt.cloudfront.net/items/1W03470s3T402V0d3Q3G/Image%202018-01-27%20at%2012.50.15%20PM.png?X-CloudApp-Visitor-Id=2456693) 201 | 202 | 203 | # Closing Words 204 | * SOA brings whole set of new issues, especially around service observability 205 | * Istio + Jager integration solves it on the service to service level 206 | * Using OpenZipkin prapagator with Jaeger allows for true end-to-end tracing 207 | -------------------------------------------------------------------------------- /glide.lock: -------------------------------------------------------------------------------- 1 | hash: 2408713f37775b5977d7eab669a6837294a6b81d04eacd241b040b1aca7617d1 2 | updated: 2018-01-21T12:41:00.483676-08:00 3 | imports: 4 | - name: github.com/apache/thrift 5 | version: b2a4d4ae21c789b689dd162deb819665567f481c 6 | subpackages: 7 | - lib/go/thrift 8 | - name: github.com/codahale/hdrhistogram 9 | version: f8ad88b59a584afeee9d334eff879b104439117b 10 | - name: github.com/jaegertracing/jaeger-client-go 11 | version: 3ac96c6e679cb60a74589b0d0aa7c70a906183f7 12 | subpackages: 13 | - config 14 | - name: github.com/opentracing-contrib/go-stdlib 15 | version: 1de4cc2120e71f745a5810488bf64b29b6d7d9f6 16 | subpackages: 17 | - nethttp 18 | - name: github.com/opentracing/opentracing-go 19 | version: 1949ddbfd147afd4d964a9f00b24eb291e0e7c38 20 | subpackages: 21 | - ext 22 | - log 23 | - name: github.com/uber/jaeger-client-go 24 | version: 3ac96c6e679cb60a74589b0d0aa7c70a906183f7 25 | subpackages: 26 | - internal/baggage 27 | - internal/baggage/remote 28 | - internal/spanlog 29 | - log 30 | - rpcmetrics 31 | - thrift-gen/agent 32 | - thrift-gen/baggage 33 | - thrift-gen/jaeger 34 | - thrift-gen/sampling 35 | - thrift-gen/zipkincore 36 | - utils 37 | - name: github.com/uber/jaeger-lib 38 | version: 7f95f4f7e80028096410abddaae2556e4c61b59f 39 | subpackages: 40 | - metrics 41 | - name: golang.org/x/net 42 | version: a337091b0525af65de94df2eb7e98bd9962dcbe2 43 | subpackages: 44 | - context 45 | testImports: [] 46 | -------------------------------------------------------------------------------- /glide.yaml: -------------------------------------------------------------------------------- 1 | package: main 2 | import: 3 | - package: github.com/uber/jaeger-client-go 4 | version: ^2.11.2 5 | - package: github.com/opentracing-contrib/go-stdlib 6 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "log" 6 | "net/http" 7 | "runtime" 8 | "time" 9 | 10 | "github.com/opentracing-contrib/go-stdlib/nethttp" 11 | opentracing "github.com/opentracing/opentracing-go" 12 | jaeger "github.com/uber/jaeger-client-go" 13 | "github.com/uber/jaeger-client-go/zipkin" 14 | ) 15 | 16 | func indexHandler(w http.ResponseWriter, r *http.Request) { 17 | fmt.Fprintf(w, "hello world, I'm running on %s with an %s CPU ", runtime.GOOS, runtime.GOARCH) 18 | } 19 | 20 | func getTimeHandler(w http.ResponseWriter, r *http.Request) { 21 | log.Print("Received getTime request") 22 | t := time.Now() 23 | ts := t.Format("Mon Jan _2 15:04:05 2006") 24 | fmt.Fprintf(w, "The time is %s", ts) 25 | } 26 | 27 | func main() { 28 | zipkinPropagator := zipkin.NewZipkinB3HTTPHeaderPropagator() 29 | injector := jaeger.TracerOptions.Injector(opentracing.HTTPHeaders, zipkinPropagator) 30 | extractor := jaeger.TracerOptions.Extractor(opentracing.HTTPHeaders, zipkinPropagator) 31 | 32 | // Zipkin shares span ID between client and server spans; it must be enabled via the following option. 33 | zipkinSharedRPCSpan := jaeger.TracerOptions.ZipkinSharedRPCSpan(true) 34 | 35 | sender, _ := jaeger.NewUDPTransport("jaeger-agent.istio-system:5775", 0) 36 | tracer, closer := jaeger.NewTracer( 37 | "myhelloworld", 38 | jaeger.NewConstSampler(true), 39 | jaeger.NewRemoteReporter( 40 | sender, 41 | jaeger.ReporterOptions.BufferFlushInterval(1*time.Second)), 42 | injector, 43 | extractor, 44 | zipkinSharedRPCSpan, 45 | ) 46 | defer closer.Close() 47 | 48 | http.HandleFunc("/", indexHandler) 49 | http.HandleFunc("/gettime", getTimeHandler) 50 | http.ListenAndServe( 51 | ":8080", 52 | nethttp.Middleware(tracer, http.DefaultServeMux)) 53 | } 54 | -------------------------------------------------------------------------------- /myhelloworld.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: extensions/v1beta1 2 | kind: Deployment 3 | metadata: 4 | labels: 5 | app: myhelloworld 6 | name: myhelloworld 7 | spec: 8 | replicas: 1 9 | template: 10 | metadata: 11 | labels: 12 | app: myhelloworld 13 | spec: 14 | containers: 15 | - image: stevenc81/jaeger-tracing-example:0.1 16 | imagePullPolicy: Always 17 | name: myhelloworld 18 | ports: 19 | - containerPort: 8080 20 | restartPolicy: Always 21 | --- 22 | apiVersion: v1 23 | kind: Service 24 | metadata: 25 | name: myhelloworld 26 | labels: 27 | app: myhelloworld 28 | spec: 29 | type: NodePort 30 | ports: 31 | - port: 80 32 | targetPort: 8080 33 | name: http 34 | selector: 35 | app: myhelloworld 36 | --- 37 | apiVersion: extensions/v1beta1 38 | kind: Ingress 39 | metadata: 40 | name: myhelloworld 41 | annotations: 42 | kubernetes.io/ingress.class: "istio" 43 | spec: 44 | rules: 45 | - http: 46 | paths: 47 | - path: / 48 | backend: 49 | serviceName: myhelloworld 50 | servicePort: 80 51 | - path: /gettime 52 | backend: 53 | serviceName: myhelloworld 54 | servicePort: 80 55 | --------------------------------------------------------------------------------