├── README.md
├── anycatch.go
├── anysend.go
├── http.go
├── main.go
└── webapp
├── GeoLite2-City.mmdb.gz
├── app.yaml
├── config.go
├── main.go
└── public
├── index.html
└── world.jpg
/README.md:
--------------------------------------------------------------------------------
1 |
2 | Detecting anycast addresses and more
3 | ===
4 | Anycast networks are a pretty interesting way to fix quite a few issues with networked services that involve needing global spread.
5 | One of the interesting things is that a computer cannot really tell (unless it has a full routing table of many providers in more than one geographic location maybe) that a IP is using anycast.
6 |
7 |
Anycast primer
8 |
9 | Before I start this, I just want to go though what an anycast actually is.
10 | Most programmers who have done networking have heard the comparison between a IP address and a telephone number, if you want to explain the idea of an anycast IP address to someone, I believe that the most simple way is to explain that if your computer is `34.24.77.2` (not my IP, not even routable) and lets say the direct translation to a phone number was `0121 441 526` and an anycast IP was `3.22.51.1`. Then its phone number would be like that of `999` (For Americans `911`, Other EU places `112`)
11 | This kind of number is the "anycast" of telephone systems, because that phone number will be directed to the most *preferable* (hopefully the closest) call center in your region, this is why if I was to dial `999` in London, I would most likely be sent to a call center in London to take my emergency call, rather to one in Scotland.
12 | Networks do the same thing basically.
13 | Here is a small drawing of a network setup:
14 |
15 | 
16 |
17 | Here you see a group of routers, all have the same IP address, Lets pretend that my router does know about all 3 of them, Because how most routing tables are setup, the packets will be sent to the fastest route possible (though, that does mean the fastest route it has, that may not in all cases mean the closest, "fastest" can in some cases also mean "cheapest") this is achieved by the "metric" on the route. Assuming the route metrics have been setup correctly. Packets will flow to the right place.
18 |
19 |
Detecting anycast addresses
20 |
21 | Since there is nothing special about anycast addresses, other than that they are addresses that are "advertised" in the internet routing table in more than one place. There is no way to look at an address (other than what I mentioned above about seeing the global routing in many places) and know its anycasted.
22 | One way for a human to check that an address is anycasted it to just trace route it from more than one location, Here is an example with 8.8.8.8 (an anycasted DNS server)
23 | Server on West Coast USA:
24 |
25 | ```
26 | ben@storm:~$ mtr -rwc 15 8.8.8.8
27 | HOST: storm Loss% Snt Last Avg Best Wrst StDev
28 | 1. 162.244.92.1 0.0% 15 0.6 6.3 0.6 59.1 15.0
29 | 2. 10.1.1.5 0.0% 15 0.6 12.3 0.5 90.0 23.7
30 | 3. any2ix.coresite.com 0.0% 15 8.0 8.4 7.9 12.3 1.1
31 | 4. 209.85.250.99 0.0% 15 8.2 11.5 8.1 42.3 9.1
32 | 5. google-public-dns-a.google.com 0.0% 15 8.3 9.7 7.9 18.7 3.5
33 | ```
34 |
35 | Server in Amsterdam:
36 |
37 | ```
38 | ben@Spitfire:~$ mtr -rwc 15 8.8.8.8
39 | HOST: Spitfire Loss% Snt Last Avg Best Wrst StDev
40 | 1. 95.46.198.1 0.0% 15 1.6 8.3 0.5 68.9 17.2
41 | 2. 80ge.cr0-br2-br3.smartdc.rtd.i3d.net 0.0% 15 1.3 4.4 0.5 13.8 4.5
42 | 3. 30ge.ar0-cr0.nikhef.ams.i3d.net 0.0% 15 1.5 17.9 1.5 56.5 17.8
43 | 4. core1.ams.net.google.com 0.0% 15 2.6 2.9 2.0 4.5 0.8
44 | 5. 209.85.241.237 0.0% 15 3.8 4.1 2.2 14.2 3.1
45 | 6. google-public-dns-a.google.com 0.0% 15 2.4 3.0 2.0 5.7 0.9
46 | ```
47 |
48 | Notice on both of those servers, the round trip time is less than 10ms on each? Now unless routers have invented electron teleportation (the bare minimum time in a single direction to get from those two servers is [42ms](http://www.wolframalpha.com/input/?i=distance+between+LAX+and+AMS)) this is an anycast address
49 |
Discovery of users destination in regards to an anycast IP
50 | This is normally a hard task for providers who use anycast, since you cannot guess where providers are going to route, since you cannot see their routing table or how their routers are configured in terms of what they have their metrics set to. They can only assume, For example “French IP addresses will hit the French announcement of the IP address”.
51 | This is not always true, this is amplified more in regions of bad connectivity that have lesser connectivity than other regions, because they will end up buying off carriers that go direct to regions like Europe and North America where connectivity is better.
52 | Where a user ends up if they try and contact an anycast IP depends on who their ISP is buying their greater connectivity off.
53 | However, The provider can use their own anycast network to test where the user will end up, assuming that the target replies back when sent something (ICMP Echo, TCP SYN) you can listen back for the response. You don’t need to worry about where on the anycast network you send the probe, all you need to do is listen back for the response.
54 | This works fine, as long as the target itself not an anycast address (and you can detect that as I have written above, by sending out from many nodes at once and seeing if the responses land back at different places)
55 |
PoC
56 | I have a set of 3 servers (One on West Coast USA, East Coast USA, and one in Luxembourg, EU) that are setup to be any casted, and have assembled an anycast network out of the three servers.
57 | Here is it detecting an anycast IP:
58 |
59 | 
60 |
61 | and here is it testing against a set of unicast destinations:
62 | University of Sydney:
63 |
64 | 
65 |
66 | StackExchange (They are based in NY)
67 |
68 | 
69 |
70 | and finally the BBC:
71 |
72 | 
73 |
74 | Want to try the tool out for yourself? You can find mine here: https://anycatch.benjojo.co.uk
75 | Or if you want to build your own in the case that you have an anycast network infra of your own, You can find the code here: https://github.com/benjojo/AnyCatch
76 | If you are using it, or find anything interesting with my own tool, Do let me know!
77 |
--------------------------------------------------------------------------------
/anycatch.go:
--------------------------------------------------------------------------------
1 | package main
2 |
3 | import (
4 | "github.com/akrennmair/gopcap"
5 | "log"
6 | "net"
7 | )
8 |
9 | const (
10 | TYPE_IP = 0x0800
11 | TYPE_ARP = 0x0806
12 | TYPE_IP6 = 0x86DD
13 |
14 | IP_ICMP = 1
15 | IP_INIP = 4
16 | IP_TCP = 6
17 | IP_UDP = 17
18 | )
19 |
20 | var lastIPs []string
21 | var ipptr int
22 |
23 | func StartListeningForPings(device, anycastIP string, snaplen int) {
24 | lastIPs = make([]string, 255)
25 | ipptr = 0
26 |
27 | var incomingIP net.IP = net.ParseIP(anycastIP)
28 |
29 | if incomingIP == nil || anycastIP == "1.2.3.4" {
30 | log.Fatal("Incorrect Anycast IP given")
31 | }
32 |
33 | if device == "" {
34 | devs, err := pcap.Findalldevs()
35 | if err != nil {
36 | log.Fatal("tcpdump: couldn't find any devices: %s\n", err)
37 | }
38 | if 0 == len(devs) {
39 | log.Fatal("tcpdump: Device error, RTFM please")
40 | }
41 | device = devs[0].Name
42 | }
43 |
44 | h, err := pcap.Openlive(device, int32(snaplen), true, 0)
45 | if h == nil {
46 | log.Fatal("tcpdump: %s\n", err)
47 | return
48 | }
49 | defer h.Close()
50 |
51 | // if expr != "" {
52 | // ferr := h.Setfilter(expr)
53 | // if ferr != nil {
54 | // log.Fatal("tcpdump: %s\n", ferr)
55 | // out.Flush()
56 | // }
57 | // }
58 |
59 | for pkt := h.Next(); pkt != nil; pkt = h.Next() {
60 | pkt.Decode()
61 | if pkt.IP != nil {
62 | if pkt.IP.Protocol == 1 && pkt.IP.DestAddr() == incomingIP.String() {
63 |
64 | // type Icmphdr struct {
65 | // Type uint8
66 | // Code uint8
67 | // Checksum uint16
68 | // Id uint16
69 | // Seq uint16
70 | // Data []byte
71 | // }
72 |
73 | for level, headerr := range pkt.Headers {
74 | switch header := headerr.(type) {
75 | case *pcap.Icmphdr:
76 | if header.Type == 0 {
77 | log.Printf("What(%d) ICMP! %s %d %d %d %d %d '%x'", level, pkt.IP.SrcAddr(), header.Type, header.Code, header.Checksum, header.Id, header.Seq, pkt.Payload)
78 | LogPing(string(pkt.Payload))
79 | }
80 | case *pcap.Iphdr:
81 | //log.Printf("What(%d) ICMP! %d %d %d %d %d", level, header.Type, header.Code, header.Checksum, header.Id, header.Seq)
82 | default:
83 | log.Printf("Ahem %s ", header)
84 | }
85 | }
86 | }
87 | }
88 | }
89 | }
90 |
91 | func min(a, b int) int {
92 | if a < b {
93 | return a
94 | }
95 | return b
96 | }
97 |
98 | func LogPing(ip string) {
99 | if ipptr+1 > len(lastIPs) {
100 | ipptr = 0
101 | }
102 |
103 | lastIPs[ipptr] = ip
104 | ipptr++
105 | }
106 |
--------------------------------------------------------------------------------
/anysend.go:
--------------------------------------------------------------------------------
1 | package main
2 |
3 | import (
4 | "bytes"
5 | "encoding/binary"
6 | "log"
7 | "net"
8 | )
9 |
10 | func SendPingPacket(Target, AnyIP, Payload string) {
11 | if len(Payload) != 8 {
12 | log.Printf("Bad payload request")
13 | return
14 | }
15 |
16 | raddr := &net.IPAddr{IP: net.ParseIP(Target).To4()}
17 | laddr := &net.IPAddr{IP: net.ParseIP(AnyIP)}
18 |
19 | con, err := net.DialIP("ip4:1", laddr, raddr)
20 | if err != nil {
21 | log.Fatalf("unable to make raw socket to dial out from, err was %s", err.Error())
22 | }
23 |
24 | // Now to hand craft a ICMP echo request packet!
25 |
26 | // pkt := p.Payload
27 | // icmp := new(Icmphdr)
28 | // icmp.Type = pkt[0]
29 | // icmp.Code = pkt[1]
30 | // icmp.Checksum = binary.BigEndian.Uint16(pkt[2:4])
31 | // icmp.Id = binary.BigEndian.Uint16(pkt[4:6])
32 | // icmp.Seq = binary.BigEndian.Uint16(pkt[6:8])
33 | // p.Payload = pkt[8:]
34 | // p.Headers = append(p.Headers, icmp)
35 | // return icmp
36 |
37 | payload := []byte(Payload)
38 | packet := make([]byte, 7+len(payload)+1) // 7 for the packet itself, 8 for the "ANYCATCH" string
39 |
40 | packet[0] = 8 // Type, in this case a echo request
41 | packet[1] = 0 // Code, in this case there is no sub code for this packet type
42 |
43 | packet[2] = 0 // checksum of packet, we will do this later when we are done
44 | packet[3] = 0 // checksum of packet, we will do this later when we are done
45 |
46 | packet[4] = 69 // ICMP ID of the request, in this case I am just filling this in
47 | packet[5] = 69 // ICMP ID of the request
48 |
49 | packet[6] = 69 // ICMP Seq of the request, in this case I am just filling this in
50 | packet[7] = 69 // ICMP Seq of the request
51 |
52 | for i := 0; i < len(payload); i++ {
53 | packet[8+i] = payload[i]
54 | }
55 | csum, _ := getChecksum(packet)
56 |
57 | buf := new(bytes.Buffer)
58 | binary.Write(buf, binary.BigEndian, csum)
59 | packet[2] = buf.Bytes()[0]
60 | packet[3] = buf.Bytes()[1]
61 |
62 | con.Write(packet)
63 | log.Printf("Done.")
64 |
65 | }
66 |
67 | func getChecksum(data []byte) (uint16, error) {
68 | buf := new(bytes.Buffer)
69 | err := binary.Write(buf, binary.BigEndian, data)
70 | if err != nil {
71 | return 0, err
72 | }
73 | arr := data
74 |
75 | var sum uint32
76 | countTo := (len(arr) / 2) * 2
77 |
78 | // Sum as if we were iterating over uint16's
79 | for i := 0; i < countTo; i += 2 {
80 | p1 := (uint32)(arr[i+1]) * 256
81 | p2 := (uint32)(arr[i])
82 | sum += p1 + p2
83 | }
84 |
85 | // Potentially sum the last byte
86 | if countTo < len(arr) {
87 | sum += (uint32)(arr[len(arr)-1])
88 | }
89 |
90 | // Fold into 16 bits.
91 | sum = (sum >> 16) + (sum & 0xFFFF)
92 | sum = sum + (sum >> 16)
93 |
94 | // Take the 1's complement, and swap bytes.
95 | answer := ^((uint16)(sum & 0xFFFF))
96 | answer = (answer >> 8) | ((answer << 8) & 0xFF00)
97 |
98 | return answer, nil
99 | }
100 |
--------------------------------------------------------------------------------
/http.go:
--------------------------------------------------------------------------------
1 | package main
2 |
3 | import (
4 | "encoding/json"
5 | "github.com/go-martini/martini"
6 | "net/http"
7 | )
8 |
9 | func StartServer(password string) {
10 | m := martini.Classic()
11 | m.Get("/Get", LastPings)
12 | m.Get("/Send/:ip/:token", SendPing)
13 |
14 | m.Use(func(res http.ResponseWriter, req *http.Request) {
15 | if req.Header.Get("X-API-KEY") != password {
16 | res.WriteHeader(http.StatusUnauthorized)
17 | }
18 | })
19 |
20 | // m.Run()
21 | m.RunOnAddr(":2374")
22 | }
23 |
24 | func SendPing(rw http.ResponseWriter, req *http.Request, params martini.Params) {
25 | for i := 0; i < 3; i++ {
26 | SendPingPacket(params["ip"], AnycastIP, params["token"])
27 | }
28 | }
29 |
30 | func LastPings(rw http.ResponseWriter, req *http.Request) {
31 | b, err := json.Marshal(lastIPs)
32 | if err != nil {
33 | http.Error(rw, "Issue in making json", http.StatusInternalServerError)
34 | }
35 |
36 | rw.Write(b)
37 | }
38 |
--------------------------------------------------------------------------------
/main.go:
--------------------------------------------------------------------------------
1 | package main
2 |
3 | import (
4 | "flag"
5 | "log"
6 | "os"
7 | )
8 |
9 | var AnycastIP string = ""
10 |
11 | func main() {
12 |
13 | var device *string = flag.String("i", "", "interface")
14 | var snaplen *int = flag.Int("s", 65535, "snaplen")
15 | var anycastIP *string = flag.String("a", "1.2.3.4", "anycastip")
16 | var password *string = flag.String("p", "wow", "password for http interface")
17 |
18 | flag.Usage = func() {
19 | log.Printf("usage: %s [ -i interface ] [ -a anycastip ] [ -s snaplen ] [ -X ] [ expression ]\n", os.Args[0])
20 | os.Exit(1)
21 | }
22 |
23 | flag.Parse()
24 | AnycastIP = *anycastIP
25 |
26 | go StartListeningForPings(*device, *anycastIP, *snaplen)
27 | StartServer(*password)
28 |
29 | }
30 |
--------------------------------------------------------------------------------
/webapp/GeoLite2-City.mmdb.gz:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/benjojo/AnyCatch/f5aafbf216a393436cf67595866393133adc69fc/webapp/GeoLite2-City.mmdb.gz
--------------------------------------------------------------------------------
/webapp/app.yaml:
--------------------------------------------------------------------------------
1 | application: anycatch
2 | version: 1
3 | runtime: go
4 | api_version: go1
5 |
6 | handlers:
7 |
8 | - url: /.*
9 | script: _go_app
--------------------------------------------------------------------------------
/webapp/config.go:
--------------------------------------------------------------------------------
1 | package main
2 |
3 | const WorkerPassword string = ""
4 |
--------------------------------------------------------------------------------
/webapp/main.go:
--------------------------------------------------------------------------------
1 | package main
2 |
3 | import (
4 | "appengine"
5 | "appengine/urlfetch"
6 | "crypto/rand"
7 | "encoding/json"
8 | "fmt"
9 | "github.com/benjojo/maxminddb-golang"
10 | "github.com/codegangsta/martini"
11 | "io/ioutil"
12 | "net"
13 | "net/http"
14 | "time"
15 | )
16 |
17 | func init() {
18 | m := martini.Classic()
19 | m.Get("/discover/:ip", SendOutPings)
20 |
21 | m.Use(func(res http.ResponseWriter, req *http.Request) {
22 | res.Header().Add("Cache-Control", "public")
23 | res.Header().Add("X-Powered-By", "uW0t-m8")
24 | })
25 |
26 | http.Handle("/", m)
27 | }
28 |
29 | type Results struct {
30 | Geoip struct {
31 | Lati float64 `json:"lati"`
32 | Long float64 `json:"long"`
33 | } `json:"geoip"`
34 | Hits []ServerHit `json:"hit"`
35 | Ip string `json:"ip"`
36 | }
37 |
38 | type ServerHit struct {
39 | Geoip struct {
40 | Lati float64 `json:"lati"`
41 | Long float64 `json:"long"`
42 | } `json:"geoip"`
43 | Name string `json:"name"`
44 | }
45 |
46 | type GeoIPCity struct {
47 | Country struct {
48 | GeoNameID uint `maxminddb:"geoname_id"`
49 | IsoCode string `maxminddb:"iso_code"`
50 | Names map[string]string `maxminddb:"names"`
51 | } `maxminddb:"country"`
52 | Location struct {
53 | Latitude float64 `maxminddb:"latitude"`
54 | Longitude float64 `maxminddb:"longitude"`
55 | MetroCode uint `maxminddb:"metro_code"`
56 | TimeZone string `maxminddb:"time_zone"`
57 | } `maxminddb:"location"`
58 | }
59 |
60 | type Worker struct {
61 | Name string
62 | URL string
63 | Latitude float64
64 | Longitude float64
65 | }
66 |
67 | var db *maxminddb.Reader
68 | var dbloaded bool = false
69 |
70 | func SendOutPings(rw http.ResponseWriter, req *http.Request, params martini.Params) string {
71 |
72 | c := appengine.NewContext(req)
73 |
74 | SendBack := Results{}
75 | if !dbloaded {
76 | var err error
77 | db, err = maxminddb.OpenGzip("GeoLite2-City.mmdb.gz")
78 | if err != nil {
79 | http.Error(rw, fmt.Sprintf("Error reading geoip db: %s", err), http.StatusInternalServerError)
80 | }
81 | dbloaded = true
82 | }
83 |
84 | ip := net.ParseIP(params["ip"]).To4()
85 |
86 | if ip == nil {
87 | addr, err := net.LookupIP(params["ip"])
88 | if err != nil {
89 | http.Error(rw, fmt.Sprintf("Not a valid IPv4 or DNS name: %s / %s", params["ip"], err), http.StatusBadRequest)
90 | return ""
91 | }
92 | if len(addr) != 0 {
93 | ip = addr[0].To4()
94 | } else {
95 | http.Error(rw, fmt.Sprintf("No DNS names found for: %s", params["ip"]), http.StatusBadRequest)
96 | return ""
97 | }
98 |
99 | }
100 | GIP := GeoIPCity{}
101 | db.Lookup(ip, &GIP)
102 |
103 | SendBack.Geoip.Lati = GIP.Location.Latitude
104 | SendBack.Geoip.Long = GIP.Location.Longitude
105 | SendBack.Hits = make([]ServerHit, 0)
106 | workers := []Worker{
107 | {Name: "storm", URL: "storm.benjojo.co.uk:2374", Latitude: 33.9425, Longitude: -118.4080},
108 | {Name: "belle", URL: "belle.benjojo.co.uk:2374", Latitude: 40.6397, Longitude: -73.7788},
109 | {Name: "flora", URL: "flora.benjojo.co.uk:2374", Latitude: 49.6233, Longitude: 6.2044},
110 | }
111 | token := RandString(8)
112 |
113 | for _, v := range workers {
114 |
115 | client := urlfetch.Client(c)
116 | req, _ := http.NewRequest("GET", fmt.Sprintf("http://%s/Send/%s/%s", v.URL, params["ip"], token), nil)
117 | req.Header.Add("X-API-KEY", WorkerPassword)
118 | res, err := client.Do(req)
119 | if err != nil {
120 | http.Error(rw, fmt.Sprintf("Cannot contact worker %s", err), http.StatusInternalServerError)
121 | return ""
122 | }
123 | if res.StatusCode != 200 {
124 | continue
125 | }
126 | }
127 |
128 | time.Sleep(time.Second * 3)
129 |
130 | for _, v := range workers {
131 | client := urlfetch.Client(c)
132 | req, err := http.NewRequest("GET", fmt.Sprintf("http://%s/Get", v.URL), nil)
133 | req.Header.Add("Cache-Control", `max-age=0, must-revalidate`)
134 | req.Header.Add("X-API-KEY", WorkerPassword)
135 | re, err := client.Do(req)
136 |
137 | if err != nil {
138 | http.Error(rw, fmt.Sprintf("Cannot contact worker %s", err), http.StatusInternalServerError)
139 | return ""
140 | }
141 | bytes, err := ioutil.ReadAll(re.Body)
142 | if err != nil {
143 | continue
144 | }
145 | tokens := make([]string, 0)
146 | json.Unmarshal(bytes, &tokens)
147 | if Contains(tokens, token) {
148 | SB := ServerHit{}
149 | SB.Name = v.Name
150 | SB.Geoip.Lati = v.Latitude
151 | SB.Geoip.Long = v.Longitude
152 | SendBack.Hits = append(SendBack.Hits, SB)
153 | }
154 | }
155 |
156 | SendBack.Ip = params["ip"]
157 |
158 | output, _ := json.Marshal(SendBack)
159 | return string(output)
160 | }
161 |
162 | func RandString(n int) string {
163 | const alphanum = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"
164 | var bytes = make([]byte, n)
165 | rand.Read(bytes)
166 | for i, b := range bytes {
167 | bytes[i] = alphanum[b%byte(len(alphanum))]
168 | }
169 | return string(bytes)
170 | }
171 |
172 | func Contains(in []string, test string) bool {
173 | for _, v := range in {
174 | if v == test {
175 | return true
176 | }
177 | }
178 | return false
179 | }
180 |
--------------------------------------------------------------------------------
/webapp/public/index.html:
--------------------------------------------------------------------------------
1 |
2 |
3 | Anycatch - Find where IP's hit on a anycast network
4 |
5 |
6 |