├── .gitignore ├── ip_a.png ├── sender-Dockerfile ├── listener-Dockerfile ├── run.sh ├── ethSender.py ├── ethListen.py └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | listener.log 2 | -------------------------------------------------------------------------------- /ip_a.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/brthor/docker-layer2-icc/HEAD/ip_a.png -------------------------------------------------------------------------------- /sender-Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3 2 | 3 | RUN apt-get update && apt-get install -y nano 4 | 5 | COPY ./ethSender.py /ethSender.py 6 | 7 | CMD python -u /ethSender.py -------------------------------------------------------------------------------- /listener-Dockerfile: -------------------------------------------------------------------------------- 1 | FROM python:3 2 | 3 | RUN apt-get update && apt-get install -y nano 4 | 5 | COPY ./ethListen.py /ethListen.py 6 | 7 | CMD python -u /ethListen.py 8 | -------------------------------------------------------------------------------- /run.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | docker network remove noiccnetwork 4 | docker network create -o com.docker.network.bridge.enable_icc=false noiccnetwork 5 | 6 | docker build -t "eth-listener" -f listener-Dockerfile . 7 | docker build -t "eth-sender" -f sender-Dockerfile . 8 | 9 | docker rm -f eth-listener eth-sender 10 | listener_id=$(docker run --network noiccnetwork -d --name eth-listener eth-listener) 11 | echo $listener_id 12 | docker logs -f eth-listener > listener.log & 13 | 14 | tail -f listener.log | docker run --network noiccnetwork -i --name eth-sender eth-sender 15 | 16 | docker logs -f eth-listener -------------------------------------------------------------------------------- /ethSender.py: -------------------------------------------------------------------------------- 1 | """ 2 | Based off: https://gist.github.com/cslarsen/11339448 3 | """ 4 | 5 | from socket import * 6 | import time, sys 7 | 8 | ETH_P_ALL = 3 9 | 10 | def macStrToBytes(macStr): 11 | import binascii 12 | macbytes = binascii.unhexlify(macStr.replace(':', '')) 13 | 14 | return macbytes 15 | 16 | def getMacAddr(): 17 | from uuid import getnode as get_mac 18 | mac = get_mac() 19 | addrStr = ':'.join(("%012X" % mac)[i:i+2] for i in range(0, 12, 2)) 20 | 21 | return macStrToBytes(addrStr) 22 | 23 | def sendeth(src, dst, eth_type, payload, interface = "eth0"): 24 | """Send raw Ethernet packet on interface.""" 25 | 26 | assert(len(src) == len(dst) == 6) # 48-bit ethernet addresses 27 | assert(len(eth_type) == 2) # 16-bit ethernet type 28 | 29 | s = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL)) 30 | 31 | # From the docs: "For raw packet 32 | # sockets the address is a tuple (ifname, proto [,pkttype [,hatype]])" 33 | s.bind((interface, 0)) 34 | s.setblocking(0) 35 | 36 | ret = s.send(dst + src + eth_type + payload) 37 | 38 | return ret 39 | 40 | if __name__ == "__main__": 41 | print("Sent %d-byte Ethernet packet on eth0" % 42 | sendeth(getMacAddr(), 43 | macStrToBytes(input("Enter the Destination Mac Address.")), # Put DEST_ADDR passed in Stdin via pipe from listener container 44 | b"\x06\x00", # Raw Ethernet 2 Frame Type 45 | b"HELLO from the SENDER")) -------------------------------------------------------------------------------- /ethListen.py: -------------------------------------------------------------------------------- 1 | """ 2 | Based off: https://gist.github.com/cslarsen/11339448 3 | """ 4 | 5 | from socket import * 6 | import time 7 | 8 | ETH_P_ALL = 3 9 | 10 | def printMacAddr(): 11 | from uuid import getnode as get_mac 12 | mac = get_mac() 13 | addrStr = ':'.join(("%012X" % mac)[i:i+2] for i in range(0, 12, 2)) 14 | 15 | # addrStr piped to ethSender, important it's printed first 16 | print(addrStr) 17 | print("^ Mac Address ^") 18 | 19 | def printPacket(packet, now, message): 20 | packetMessage = packet[14:].decode('ascii', 'ignore') 21 | 22 | if packetMessage == "hello": 23 | print ("Got message from sender.") 24 | 25 | print(message, "Len:", len(packet), "bytes time:", now, "message:", packetMessage) 26 | 27 | def listeneth(interface = "eth0"): 28 | s = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL)) 29 | 30 | # From the docs: "For raw packet 31 | # sockets the address is a tuple (ifname, proto [,pkttype [,hatype]])" 32 | s.bind((interface, 0)) 33 | s.setblocking(0) 34 | 35 | while True: 36 | now = time.time() 37 | 38 | try: 39 | packet = s.recv(128) 40 | except Exception as e: 41 | if 'Resource temporarily unavailable' not in str(e): 42 | print(e) 43 | pass 44 | else: 45 | printPacket(packet, now, "Received:") 46 | 47 | time.sleep(0.001001) 48 | 49 | if __name__ == "__main__": 50 | # MacAddr piped to ethSender, important it's printed first 51 | printMacAddr() 52 | 53 | print("Listening for packets") 54 | listeneth() -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Docker Layer 2 ICC Bug 2 | 3 | Quick Start: 4 | ```bash 5 | git clone https://github.com/brthor/docker-layer2-icc.git && cd docker-layer2-icc && ./run.sh 6 | ``` 7 | 8 | ## Explanation (What and How) 9 | 10 | When you create a docker container using `docker run`, it is automatically connected to a bridge network. Unless inter-container communication (ICC) was disabled in the docker daemon, every container on that bridge network can communicate with one another via sockets. 11 | 12 | Docker allows you to restrict ICC in two ways: 13 | 1. Restricting it in the kernel with `"icc": false` in `daemon.json` 14 | 2. Creating a network like `docker network create -o com.docker.network.bridge.enable_icc=false noicc` then connecting containers to it with `docker run --network=noicc ...` 15 | 16 | In both cases you expect that all network communications will be blocked between the containers themselves. As demonstrated below, this is not the case. Docker puts in iptables rules that block communications on layer 3, but layer 2 communications are allowed. What this means, is I can still send data between the containers over a socket. 17 | 18 | **Disabling ICC doesn't block raw ethernet frames between containers.** 19 | 20 | This behavior is highly unexpected, and in highly secure environments, likely to be an issue. 21 | 22 | ## Repro Steps 23 | 24 | I reproed this using `Docker-CE for Mac Version 17.09.0-ce-rc3-mac30 (19329)`. 25 | And `Docker-CE on CentOS 7.3 Version 18.02.0-ce` 26 | 27 | ### Automatic Repro 28 | 29 | Use [./run.sh](/run.sh) to run an automatic repro. Your output will look like: 30 | 31 | ```bash 32 | $ ./run.sh 33 | Sending build context to Docker daemon 602.1kB 34 | Step 1/4 : FROM python:3 35 | ---> 336d482502ab 36 | Step 2/4 : RUN apt-get update && apt-get install -y nano 37 | ---> Using cache 38 | ---> 5cee00913b09 39 | Step 3/4 : COPY ./ethListen.py /ethListen.py 40 | ---> Using cache 41 | ---> a2667cd58e69 42 | Step 4/4 : CMD python -u /ethListen.py 43 | ---> Using cache 44 | ---> 9743c680cc72 45 | Successfully built 9743c680cc72 46 | Successfully tagged eth-listener:latest 47 | Sending build context to Docker daemon 602.1kB 48 | Step 1/4 : FROM python:3 49 | ---> 336d482502ab 50 | Step 2/4 : RUN apt-get update && apt-get install -y nano 51 | ---> Using cache 52 | ---> 5cee00913b09 53 | Step 3/4 : COPY ./ethSender.py /ethSender.py 54 | ---> Using cache 55 | ---> e528077d48da 56 | Step 4/4 : CMD python -u /ethSender.py 57 | ---> Using cache 58 | ---> 46061f2a53ca 59 | Successfully built 46061f2a53ca 60 | Successfully tagged eth-sender:latest 61 | eth-listener 62 | eth-sender 63 | 8391d06550a63fb8423be86876cd906a79720f6d52c0d8885ce2e102b8015768 64 | Sent 35-byte Ethernet packet on eth0 65 | XX:XX:XX:XX:XX:XX 66 | ^ Mac Address ^ 67 | Listening for packets 68 | Received: Len: 78 bytes time: 1519123406.9721925 message: `: 69 | $[8 70 | $ 71 | Received: Len: 90 bytes time: 1519123407.0323486 message: `$B:|{ 72 | Received: Len: 90 bytes time: 1519123407.2033708 message: `$B:|{ 73 | Received: Len: 35 bytes time: 1519123407.2230816 message: HELLO from the SENDER 74 | ^C 75 | ``` 76 | 77 | If you see the line 78 | ``` 79 | 80 | Received: Len: 35 bytes time: 1519123407.2230816 message: HELLO from the SENDER 81 | 82 | ``` 83 | 84 | Then the sender successfully sent a raw ethernet frame, despite icc being disabled. 85 | 86 | ### Manual Repro Steps 87 | 88 | 1. Build the Listener image: `docker build -t "eth-listener" -f listener-Dockerfile .` 89 | The [listener](/ethListen.py) listens for raw ethernet frames and prints any received data. 90 | It also finds and prints it's layer 2 address. We will need this value to send it data. 91 | 92 | 2. Build the Sender image: `docker build -t "eth-sender" -f sender-Dockerfile .` 93 | The [sender](/ethSender.py) sends the string "HELLO from the SENDER" to the listener. 94 | We will look for this string amongst the listener output. 95 | 96 | 3. Create the ICC Disabled network: `docker network create -o com.docker.network.bridge.enable_icc=false noicc` 97 | `noicc` is the name of the network. 98 | 99 | 4. Make sure you have two shells open, one for the listening container, and one for the sender. 100 | 101 | 5. Start your listening container: 102 | ```bash 103 | $ docker run -it --network noicc --name eth-listener eth-listener 104 | XX:XX:XX:XX:XX:XX 105 | ^ Mac Address ^ 106 | Listening for packets 107 | ... 108 | ``` 109 | 110 | **Copy the mac address value for the next step.** 111 | 112 | 6. Run the sending container in your second terminal 113 | ```bash 114 | $ docker run -it --network noicc --name eth-sender eth-sender 115 | "Enter the Destination Mac Address." 116 | ``` 117 | 118 | 7. Paste the mac address from the listening container in the sending container terminal. 119 | 120 | 8. Observe the Listening container output for `HELLO from the SENDER` 121 | ``` 122 | $ docker run -it --network noicc --name eth-listener eth-listener 123 | XX:XX:XX:XX:XX:XX 124 | ^ Mac Address ^ 125 | Listening for packets 126 | Received: Len: 78 bytes time: 1519123587.8931577 message: `:vgG\vg 127 | Received: Len: 78 bytes time: 1519123587.9433937 message: `:]qB 128 | Received: Len: 90 bytes time: 1519123588.222616 message: `$: 129 | Received: Len: 35 bytes time: 1519123588.2736714 message: HELLO from the SENDER 130 | Received: Len: 32 bytes time: 1519123588.9437985 message: 131 | ``` 132 | 133 | **Disabling ICC on the bridge network didn't block the raw socket communication as we would have expected** 134 | 135 | These contain more explanation to break it down. 136 | 137 | ## Bug Resolution (Workarounds) 138 | 139 | If you keep containers on the same network bridge, create ebtables rules between containers. For each pair of containers, create a pair of rules, like those below, but replacing the mac addresses with the addresses of your containers. 140 | ``` 141 | sudo ebtables -A FORWARD -d 02:42:ac:13:00:02 -s 02:42:ac:13:00:03 -j DROP 142 | sudo ebtables -A FORWARD -s 02:42:ac:13:00:02 -d 02:42:ac:13:00:03 -j DROP 143 | ``` 144 | 145 | It can be worked around by placing containers on different network bridges. That means using `docker network create` for every container. By default you can only create 31 networks. [Set the subnet on each network manually to get around this limitation](https://loomchild.net/2016/09/04/docker-can-create-only-31-networks-on-a-single-machine/). 146 | --------------------------------------------------------------------------------