├── DPDKPktgen.md ├── README.md ├── SRIOV-FLOWBIFURCATION.md ├── imix.pcap ├── install-dpdk-pktgen.sh ├── monitorpkts.sh ├── output.pcap ├── pktgen-range.pkt └── pktgen.pkt /DPDKPktgen.md: -------------------------------------------------------------------------------- 1 | # DPDK Pktgen 2 | 3 | ## starting 4 | 5 | ``` 6 | /opt/pktgen-20.02.0/app/x86_64-native-linuxapp-gcc/pktgen -- -T -P -m "2.[0]" -f pktgen.pkt 7 | ``` 8 | This means we use CPU core 2 for port 0. ``` -f pktgen.pkt ``` means we have a few set commands in pktgen.pkt 9 | start with an example file that looks like this: 10 | ``` 11 | root@ewr1-x1:~# cat pktgen.pkt 12 | stop 0 13 | set 0 rate 0.1 14 | set 0 ttl 10 15 | set 0 proto udp 16 | set 0 dport 81 17 | set 0 dst mac 44:ec:ce:c1:a8:20 18 | set 0 src mac 00:52:44:11:22:33 19 | set 0 dst ip 10.99.204.8 20 | set 0 src ip 10.99.204.3/30 21 | set 0 size 64 22 | ``` 23 | 24 | Note: you need to run ```start 0``` after this to start 25 | 26 | See this link for all set options 27 | https://pktgen-dpdk.readthedocs.io/en/latest/commands.html#runtime-options-and-commands 28 | 29 | see this link for all pktgen cli option (i.e. core assignments, etc as defined with -m) 30 | https://pktgen-dpdk.readthedocs.io/en/latest/usage_pktgen.html 31 | 32 | ## Pages 33 | 1. page main 34 | ``` 35 | Pktgen:/> page main 36 | ``` 37 | This is the start up page and will show you the current througput, ports, etc. 38 | 39 | 40 | 2. page range 41 | ``` 42 | Pktgen:/> page range 43 | ``` 44 | This will show you the ranges you set up. 45 | Ranges are used for when you want Pktgen to use a range of IPs, Ports, pkt sizes, etc. 46 | Note: that after you change the range settings, you will need to stop and start the test 47 | 48 | ``` 49 | Pktgen:/> stop 0 50 | Pktgen:/> start 0 51 | ``` 52 | 53 | Also know you need to enable and disable range if you'd like to use it. 54 | By default you don't use range 55 | 56 | ``` 57 | Pktgen:/> enable 0 range 58 | ``` 59 | 60 | 61 | ## Traffic patterns 62 | Traffic patterns can be set with ranges. 63 | Remember to enable ranges (not default) 64 | 65 | ``` 66 | Pktgen:/> enable 0 range 67 | ``` 68 | 69 | You need to stop and start after range changes 70 | ``` 71 | stop 0 72 | start 0 73 | ``` 74 | 75 | Set dst port range 76 | ``` 77 | Pktgen:/> range 0 src port start 4000 78 | Pktgen:/> range 0 src port min 4000 79 | Pktgen:/> range 0 src port max 5000 80 | Pktgen:/> range 0 src port inc 1 81 | ``` 82 | Start, minimum, maximum, increment 83 | 84 | Set to UDP 85 | ``` 86 | Pktgen:/> range 0 proto udp 87 | ``` 88 | 89 | Set dst mac 90 | ``` 91 | Pktgen:/> range 0 dst mac start 44:ec:ce:c1:a8:20 92 | ``` 93 | 94 | set src and dst ip 95 | ``` 96 | Pktgen:/> range 0 src ip start 10.99.204.4 97 | Pktgen:/> range 0 dst ip start 10.99.204.1 98 | ``` 99 | 100 | ``` 101 | stop 0 102 | set 0 rate 0.1 103 | range 0 ttl start 10 104 | range 0 dst mac start 44:ec:ce:c1:a8:20 105 | range 0 proto udp 106 | range 0 src port start 4000 107 | range 0 src port inc 0 108 | range 0 src port max 4000 109 | range 0 dst port start 4000 110 | range 0 dst port min 4000 111 | range 0 dst port max 4000 112 | range 0 dst port inc 0 113 | range 0 dst ip start 10.99.204.8 114 | range 0 dst ip inc 0 115 | range 0 dst mac start 44:ec:ce:c1:a8:20 116 | range 0 src ip start 10.99.204.3 117 | range 0 src ip inc 0 118 | range 0 size start 64 119 | range 0 size inc 0 120 | 121 | ``` 122 | ## Read from pcap 123 | 124 | Download for example an imix file: 125 | ``` 126 | wget https://github.com/cisco-system-traffic-generator/trex-core/raw/master/scripts/exp/imix.pcap 127 | ``` 128 | ``` 129 | tshark -r imix.pcap -V | grep 'Frame Length'| sort | uniq -c | sort -n 130 | Running as user "root" and group "root". This could be dangerous. 131 | 9 Frame Length: 1514 bytes (12112 bits) 132 | 33 Frame Length: 590 bytes (4720 bits) 133 | 58 Frame Length: 60 bytes (480 bits) 134 | ``` 135 | See: https://en.wikipedia.org/wiki/Internet_Mix 136 | 137 | | Packet size (incl. IP header) | Packets Distribution (in packets) | 138 | | ------------- |:-------------:| 139 | |40 | 58.333333% | 140 | |576 | 33.333333% | 141 | |1500 | 8.333333% | 142 | 143 | Then rewrite all field to what you need it to be 144 | In this case i'm setting the src and dst mac. And need to rewrite the dst IP to the target host 48.0.0.0/8 > 10.99.204.8/32/ 145 | Also randomizing source IPS 16.0.0.0/8 > 10.10.0.0/16 146 | ``` 147 | tcprewrite \ 148 | --enet-dmac=44:ec:ce:c1:a8:20 \ 149 | --enet-smac=00:52:44:11:22:33 \ 150 | --pnat=16.0.0.0/8:10.10.0.0/16,48.0.0.0/8:10.99.204.8/32 \ 151 | --infile=imix.pcap \ 152 | --outfile=output.pcap 153 | ``` 154 | 155 | check result with ```tcpdump -e -n -r output.pcap``` 156 | 157 | Then start pktgen with the pcap 158 | ``` 159 | /opt/pktgen-20.02.0/app/x86_64-native-linuxapp-gcc/pktgen -- -T -P -m "2.[0]" -s 0:output.pcap 160 | ``` 161 | 162 | Then do this: 163 | ``` 164 | enable 0 pcap 165 | set 0 rate 0.1 166 | start 0 167 | ``` 168 | 169 | Also check ``` page pcap``` not sure why the ips look different in that screen.. 170 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # dpdk_pktgen 2 | Building a high performance - Linux Based Traffic generator with DPDK 3 | 4 | We don’t always have access to hardware-based traffic generators, as they tend to be quite expensive or only available in a lab 5 | The code in this repo allows us to build a software packet generatpor. It is provided as additional information to this blog post 6 | https://toonk.io/building-a-high-performance-linux-based-traffic-generator-with-dpdk/index.html 7 | -------------------------------------------------------------------------------- /SRIOV-FLOWBIFURCATION.md: -------------------------------------------------------------------------------- 1 | Also see: 2 | https://github.com/github/glb-director/blob/master/docs/setup/known-compatible-dpdk.md 3 | 4 | # Enable SRIOV 5 | ``` 6 | echo 1 > /sys/class/net/eno1/device/sriov_numvfs 7 | cat /sys/class/net/eno1/device/sriov_numvfs 8 | 9 | ip link set eno1 vf 0 spoofchk off 10 | ip link set eno1 vf 0 trust on 11 | ethtool --features eno1 ntuple on 12 | 13 | ``` 14 | After you create the VF, identify the new interface like this:, in this case it's em1_0 15 | You can also check ```dmesg -T``` 16 | ``` 17 | root@ewr1-x1:~# lshw -businfo -class network 18 | Bus info Device Class Description 19 | ======================================================= 20 | pci@0000:02:00.0 eno1 network Ethernet Controller X710 for 10GbE backplane 21 | pci@0000:02:00.1 em2 network Ethernet Controller X710 for 10GbE backplane 22 | pci@0000:03:02.0 em1_0 network Illegal Vendor ID 23 | bond0 network Ethernet interface 24 | ``` 25 | bring it up if you'd like ``` ip link set em1_0 up ``` 26 | 27 | # flow bifurcation 28 | Add rule to send to queue 0, which when you have VF SRIOV, will be VF0 29 | note, you can add order with loc key word 30 | ``` 31 | ethtool --config-ntuple eno1 flow-type udp4 dst-port 80 action 0x100000000 32 | Added rule with ID 7679 33 | ``` 34 | 35 | check with 36 | ``` 37 | root@ewr1-x1:~# ethtool -n eno1 38 | 8 RX rings available 39 | Total 1 rules 40 | 41 | Filter: 5 42 | Rule Type: UDP over IPv4 43 | Src IP addr: 0.0.0.0 mask: 255.255.255.255 44 | Dest IP addr: 0.0.0.0 mask: 255.255.255.255 45 | TOS: 0x0 mask: 0xff 46 | Src port: 0 mask: 0xffff 47 | Dest port: 80 mask: 0x0 48 | Action: Direct to VF 0 queue 0 49 | ``` 50 | 51 | To add a drop key: 52 | ``` 53 | ethtool --config-ntuple eno1 flow-type udp4 dst-port 80 action -1 loc 5 54 | ``` 55 | 56 | To delete 57 | ``` 58 | ethtool --config-ntuple eno1 delete 5 59 | ``` 60 | -------------------------------------------------------------------------------- /imix.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/atoonk/dpdk_pktgen/a9eb07b3ebffb4dea236a3a014d5adde31f733a5/imix.pcap -------------------------------------------------------------------------------- /install-dpdk-pktgen.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | export DPDK_VER=20.02 4 | export PKTGEN_VER=20.02.0 5 | export PCI_IF="0000:1a:00.3" 6 | 7 | 8 | if [ `whoami` != 'root' ]; then 9 | echo "Please run this as root..., don't worry about it..." 10 | exit 1 11 | fi 12 | 13 | echo "updating fstab" 14 | r=`grep hugetlbfs /etc/fstab` 15 | if [ $? -eq 1 ]; then 16 | echo "huge /mnt/huge hugetlbfs defaults 0 0" >> /etc/fstab 17 | fi 18 | 19 | if [ ! -d /mnt/huge ]; then 20 | mkdir /mnt/huge 21 | chmod 777 /mnt/huge/ 22 | fi 23 | 24 | 25 | echo "Updating sysctl" 26 | r=`grep nr_hugepages /etc/sysctl.conf` 27 | if [ $? -eq 1 ]; then 28 | echo "vm.nr_hugepages=256" >> /etc/sysctl.conf 29 | # also make sure it is live on this run, in case fstab has been already updated 30 | sysctl -w vm.nr_hugepages=256 31 | fi 32 | 33 | echo "checking for iommu in GRUB" 34 | r=`grep iommu=pt /etc/default/grub` 35 | if [ $? -eq 1 ]; then 36 | echo "iommu is missing from grub" 37 | echo "please edit /etc/default/grub and make to append the below to GRUB_CMDLINE_LINUX" 38 | echo "default_hugepagesz=1G hugepagesz=1G hugepages=8 iommu=pt intel_iommu=on pci=assign-busses" 39 | echo 'example: GRUB_CMDLINE_LINUX="console=tty0 console=ttyS1,115200n8 biosdevname=0 net.ifnames=1 default_hugepagesz=1G hugepagesz=1G hugepages=8 iommu=pt intel_iommu=on pci=assign-busses"' 40 | echo "after that run: update-grub && reboot" 41 | echo "this will reboot your machine!" 42 | echo "other things you may want to add are:" 43 | echo "maxcpus=32" 44 | echo "isolcpus=3-31" 45 | exit 1 46 | fi 47 | 48 | r=`grep intel_iommu=on /etc/default/grub` 49 | if [ $? -eq 1 ]; then 50 | echo "iommu is missing from grub" 51 | echo "please edit /etc/default/grub and make to append the below to GRUB_CMDLINE_LINUX" 52 | echo "default_hugepagesz=1G hugepagesz=1G hugepages=8 iommu=pt intel_iommu=on pci=assign-busses" 53 | 54 | echo 'example: GRUB_CMDLINE_LINUX="console=tty0 console=ttyS1,115200n8 biosdevname=0 net.ifnames=1 default_hugepagesz=1G hugepagesz=1G hugepages=8 iommu=pt intel_iommu=on pci=assign-busses"' 55 | echo "after that run: update-grub && reboot" 56 | echo "this will reboot your machine!" 57 | exit 1 58 | fi 59 | 60 | echo "Going into /opt ..." 61 | cd /opt 62 | 63 | echo "Installing packages..." 64 | apt-get -y update 65 | apt-get -y upgrade 66 | apt-get -y install python2.7 67 | apt-get -y install build-essential ninja-build meson cmake 68 | apt-get -y install libnuma-dev 69 | apt-get -y install pciutils 70 | apt-get -y install libpcap-dev 71 | apt-get -y install liblua5.3-dev 72 | apt-get -y install python3-pyelftools 73 | apt -y install libelf-dev 74 | apt-get -y install linux-headers-`uname -r` || apt -y install linux-headers-generic 75 | 76 | echo "Setting env..." 77 | export RTE_TARGET=x86_64-native-linuxapp-gcc 78 | export RTE_SDK=/opt/dpdk-$DPDK_VER 79 | ln -s /usr/bin/python2.7 /usr/bin/python 80 | 81 | echo "Downloading DPDK..." 82 | if [ ! -f /opt/dpdk-$DPDK_VER.tar.xz ]; then 83 | wget https://fast.dpdk.org/rel/dpdk-$DPDK_VER.tar.xz 84 | fi 85 | 86 | echo "Unpacking DPDK..." 87 | rm -rf dpdk-$DPDK_VER/ 88 | tar xvf dpdk-$DPDK_VER.tar.xz 89 | 90 | echo "Installing DPDK..." 91 | cd dpdk-$DPDK_VER 92 | make config T=x86_64-native-linuxapp-gcc CONFIG_RTE_EAL_IGB_UIO=y 93 | make install T=x86_64-native-linuxapp-gcc DESTDIR=install CONFIG_RTE_EAL_IGB_UIO=y 94 | 95 | cd .. 96 | 97 | echo "Downloading pktgen..." 98 | if [ ! -f /opt/pktgen-$PKTGEN_VER.tar.gz ]; then 99 | wget http://dpdk.org/browse/apps/pktgen-dpdk/snapshot/pktgen-$PKTGEN_VER.tar.gz 100 | fi 101 | 102 | echo "Unpacking pktgen..." 103 | rm -rf pktgen-$PKTGEN_VER/ 104 | tar xvf pktgen-$PKTGEN_VER.tar.gz 105 | 106 | echo "Installing DPDK..." 107 | cd pktgen-$PKTGEN_VER 108 | make 109 | 110 | echo "binding dpdk interface $PCI_IF" 111 | modprobe uio 112 | insmod /opt/dpdk-$DPDK_VER/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko 113 | modprobe vfio-pci 114 | modprobe uio_pci_generic 115 | dpdkdevbind=/opt/dpdk-$DPDK_VER/usertools/dpdk-devbind.py 116 | $dpdkdevbind --force -u $PCI_IF 117 | $dpdkdevbind -b igb_uio $PCI_IF 118 | $dpdkdevbind -s 119 | 120 | echo "To run pktgen:" 121 | echo '/opt/pktgen-$PKTGEN_VER/app/x86_64-native-linuxapp-gcc/pktgen -- -T -P -m "2.[0]"' 122 | echo '/opt/pktgen-$PKTGEN_VER/app/x86_64-native-linuxapp-gcc/pktgen -- -T -P -m "2/4:6/8.[0]"' 123 | 124 | echo " 125 | example commands: 126 | set 0 dst mac e4:43:4b:53:51:83 127 | set 0 rate 1 128 | set 0 size 128 129 | start 0 130 | stop 0 131 | " 132 | -------------------------------------------------------------------------------- /monitorpkts.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | if [ "$1" != "" ]; then 4 | NIC=$1 5 | else 6 | echo "Please provide the network interface you'd like to monitor." 7 | echo "example: $0 eth0" 8 | exit 1 9 | fi 10 | p=0 11 | d=0 12 | while sleep 1; do 13 | r=$(netstat -i | grep $NIC | awk '{print $3,$5}' | grep -v statistics) 14 | p_now=$(echo $r | awk '{print $1}') 15 | d_now=$(echo $r |awk '{print $2}') 16 | 17 | if [ "$p" -gt "0" ]; then 18 | dropped=$((d_now - d)) 19 | rx=$((p_now - p)) 20 | perc_d=$(echo "scale=5;($dropped/$rx)*100" | bc -l) 21 | echo "pps: received: $rx dropped: $dropped $perc_d %" 22 | fi 23 | p=$p_now 24 | d=$d_now 25 | done 26 | -------------------------------------------------------------------------------- /output.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/atoonk/dpdk_pktgen/a9eb07b3ebffb4dea236a3a014d5adde31f733a5/output.pcap -------------------------------------------------------------------------------- /pktgen-range.pkt: -------------------------------------------------------------------------------- 1 | stop 0 2 | set 0 rate 0.1 3 | range 0 ttl start 10 4 | range 0 dst mac start 44:ec:ce:c1:a8:20 5 | 6 | range 0 proto udp 7 | range 0 src port start 4000 8 | range 0 src port min 4000 9 | range 0 src port inc 1 10 | range 0 src port max 5000 11 | 12 | range 0 dst port start 4000 13 | range 0 dst port min 4000 14 | range 0 dst port max 5000 15 | range 0 dst port inc 1 16 | 17 | range 0 dst ip start 10.99.204.8 18 | range 0 dst ip inc 0 19 | range 0 dst mac start 44:ec:ce:c1:a8:20 20 | 21 | range 0 src ip start 10.99.204.3 22 | range 0 src ip inc 0 23 | 24 | range 0 size start 64 25 | range 0 size inc 0 26 | enable 0 range 27 | -------------------------------------------------------------------------------- /pktgen.pkt: -------------------------------------------------------------------------------- 1 | stop 0 2 | set 0 rate 0.1 3 | set 0 ttl 10 4 | set 0 proto udp 5 | set 0 dport 81 6 | set 0 dst mac 44:ec:ce:c1:a8:20 7 | #set 0 src mac 00:52:44:11:22:33 8 | #set 0 src mac 00:00:00:00:00:01 9 | set 0 dst ip 10.99.204.8 10 | set 0 src ip 10.99.204.3/30 11 | set 0 size 64 12 | --------------------------------------------------------------------------------