├── LISA2019_BPF_Performance_Tools.pdf
├── README.md
├── lab1.md
├── lab2.md
├── lab3.md
├── lab4.md
├── lab5.md
└── src
    ├── Makefile
    ├── README.md
    ├── lab001.c
    └── lab003.c


/LISA2019_BPF_Performance_Tools.pdf:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/brendangregg/bpf-perf-workshop/3837dec6e70de9d5db60afce3c7ed54ec49d3e1c/LISA2019_BPF_Performance_Tools.pdf


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
 1 | # BPF Performance Tools Workshop
 2 | 
 3 | This is a 90-minute workshop created by Brendan Gregg, and uses some exercises from the book [BPF Performance Tools: Linux System and Application Observability](http://www.brendangregg.com/bpf-performance-tools-book.html). BPF enables a new class of custom, efficient, and production safe performance analysis tools. Historically BPF stood for Berkeley Packet Filter, but since it has become a general purpose execution environment it is now a technology name and no longer an acronym. BPF is part of the Linux kernel.
 4 | 
 5 | This workshop is composed of slides, exercises, discussion, and demos. This repository contains two of those four parts: the slides and exercises, the other two are only available in class.
 6 | 
 7 | - [USENIX LISA2019 Slides](LISA2019_BPF_Performance_Tools.pdf) (PDF)
 8 | 
 9 | ## Prerequisites
10 | 
11 | You can either:
12 | 
13 | - SSH to a lab instance (see bit of paper), or,
14 | - Setup your own system with [BCC](https://github.com/iovisor/bcc) & [bpftrace](https://github.com/iovisor/bpftrace). Sample instructions below.
15 | 
16 | #### System Setup
17 | 
18 | If you choose to setup your own system, here is a script for Ubuntu:
19 | 
20 | ```
21 | sudo apt-get update
22 | sudo apt-get install sysstat bpfcc-tools bpftrace gcc
23 | git clone https://github.com/brendangregg/bpf-perf-workshop
24 | cd bpf-perf-workshop/src
25 | make && cd ..
26 | ```
27 | 
28 | Just for reference, you can fetch these repos:
29 | 
30 | ```
31 | git clone https://github.com/iovisor/bcc
32 | git clone https://github.com/iovisor/bpftrace
33 | ```
34 | 
35 | ## Checking your system works
36 | 
37 | Testing the opensnoop(8) tool from both BCC and bpftrace (Ubuntu package naming scheme: BCC tools end in -bpfcc, and bpftrace .bt):
38 | 
39 | <pre>
40 | $ <b>sudo bash</b>
41 | # <b>opensnoop-bpfcc</b>
42 | [...]
43 | # <b>opensnoop.bt</b>
44 | [...]
45 | </pre>
46 | 
47 | Those should produce output and not error. Ctrl-C to end each.
48 | 
49 | ## References
50 | 
51 | You may wish to open these in tabs. For labs 1-3 (BCC):
52 | 
53 | - [BCC tools](https://github.com/iovisor/bcc#tools)
54 | - [BCC tutorial](https://github.com/iovisor/bcc/blob/master/docs/tutorial.md)
55 | 
56 | For labs 4-5 (bpftrace):
57 | 
58 | - [bpftrace tutorial](https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md)
59 | - [bpftrace reference guide](https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md)
60 | - [bpftrace tools](https://github.com/iovisor/bpftrace#tools)
61 | - [BPF book tools](https://github.com/brendangregg/bpf-perf-tools-book#tools)
62 | 
63 | ## Labs
64 | 
65 | - [lab1: Investigate latency](lab1.md)
66 | - [lab2: Investigate SSH login performance](lab2.md)
67 | - [lab3: Analyze a CPU issue](lab3.md)
68 | - [lab4: Develop new bpftrace tools](lab4.md)
69 | - [lab5: Advanced bpftrace tools (optional)](lab5.md)
70 | 
71 | Copyright Brendan Gregg, 2019. All rights reserved.
72 | 


--------------------------------------------------------------------------------
/lab1.md:
--------------------------------------------------------------------------------
 1 | # lab1: Investigate latency
 2 | 
 3 | ## Problem Statement
 4 | 
 5 | An application has higher-than expected latency, including latency outliers. Why, and how can performance be improved?
 6 | 
 7 | ## Instructions
 8 | 
 9 | To run the application.
10 | 
11 | ```
12 | cd bpf-perf-workshop
13 | ./lab001 &
14 | ```
15 | 
16 | lab001 simulates a performance issue based on previous real world problems. It does not include application logic: for simplicity, it only executes the problem.
17 | 
18 | While lab001 is running, you can use any BPF tool to investigate performance. See the References below. You will need to run them as root, so `sudo bash` first for a root shell. You can also use traditional tools, including top(1), vmstat(1), mpstat(1), iostat(1), and sar(1), provided they do not perturb performance -- this is a (pretend) production system! That means strace(1) and tcpdump(8) are forbidden. Use BPF equivalents instead.
19 | 
20 | When you have finished your analysis, remember to kill the lab001 process.
21 | 
22 | ## Help! I'm completely lost
23 | 
24 | If you don't know where to start, you can try working through the following checklists:
25 | 
26 | - [0. Before bcc](https://github.com/iovisor/bcc/blob/master/docs/tutorial.md#0-before-bcc)
27 | - [1. General Performance](https://github.com/iovisor/bcc/blob/master/docs/tutorial.md#1-general-performance)
28 | 
29 | These are from my [BCC tutorial](https://github.com/iovisor/bcc/blob/master/docs/tutorial.md). These won't find everything, but are a good start.
30 | 
31 | Later in this workshop I'll introduce methodologies to help you begin performance analysis.
32 | 
33 | ## References
34 | 
35 | - [BCC tools](https://github.com/iovisor/bcc#tools)
36 | - [bpftrace tools](https://github.com/iovisor/bpftrace#tools)
37 | - [BPF book tools](https://github.com/brendangregg/bpf-perf-tools-book#tools)
38 | - [BCC tutorial](https://github.com/iovisor/bcc/blob/master/docs/tutorial.md)
39 | 
40 | Copyright Brendan Gregg, 2019. All rights reserved.
41 | 


--------------------------------------------------------------------------------
/lab2.md:
--------------------------------------------------------------------------------
 1 | # lab2: Investigate SSH login performance
 2 | 
 3 | ## Problem Statement
 4 | 
 5 | Under load, it can take a few seconds to login to your lab system via SSH. Even when idle, it can take 1-2 seconds. Why does it take this long on an idle system, and how can this login time be reduced?
 6 | 
 7 | ## Instructions
 8 | 
 9 | Use two terminal windows. In one, SSH to the target system and run BPF analysis tools. In the other, SSH to the target system while analysis tools are running. You can also run ssh via `time` to create both a login event for analysis, plus provide some timing details:
10 | 
11 | ```
12 | $ time ssh student@IP_ADDRESS echo hello
13 | hello
14 | 
15 | real	0m1.200s
16 | user	0m0.027s
17 | sys	0m0.008s
18 | ```
19 | 
20 | As with [lab1](lab1.md), you can use BPF tools from BCC, bpftrace, and the book for analysis. You an also use traditional tools, except those that perturb performance (e.g., strace(1) and tcpdump(8)).
21 | 
22 | ## References
23 | 
24 | - [BCC tools](https://github.com/iovisor/bcc#tools)
25 | - [bpftrace tools](https://github.com/iovisor/bpftrace#tools)
26 | - [BPF book tools](https://github.com/brendangregg/bpf-perf-tools-book#tools)
27 | - [BCC tutorial](https://github.com/iovisor/bcc/blob/master/docs/tutorial.md)
28 | 
29 | Copyright Brendan Gregg, 2019. All rights reserved.
30 | 


--------------------------------------------------------------------------------
/lab3.md:
--------------------------------------------------------------------------------
 1 | # lab3: Analyze a CPU issue
 2 | 
 3 | ## Problem Statement
 4 | 
 5 | The application has high CPU usage even though it isn't serving many (or any) requests. What is happening? How many potential problems can you identify?
 6 | 
 7 | ## Instructions
 8 | 
 9 | To run the application:
10 | 
11 | ```
12 | cd bpf-perf-workshop
13 | ./lab003 &
14 | ```
15 | 
16 | While running, analyze the application in as much detail as possible. As with [lab1](lab1.md), you can use BPF tools from BCC, bpftrace, and the book for analysis. You an also use traditional tools, except those that perturb performance (e.g., strace(1) and tcpdump(8)).
17 | 
18 | When you have finished your analysis, remember to kill the lab003 process.
19 | 
20 | ## References
21 | 
22 | - [BCC tools](https://github.com/iovisor/bcc#tools)
23 | - [bpftrace tools](https://github.com/iovisor/bpftrace#tools)
24 | - [BPF book tools](https://github.com/brendangregg/bpf-perf-tools-book#tools)
25 | - [BCC tutorial](https://github.com/iovisor/bcc/blob/master/docs/tutorial.md)
26 | 
27 | Copyright Brendan Gregg, 2019. All rights reserved.
28 | 


--------------------------------------------------------------------------------
/lab4.md:
--------------------------------------------------------------------------------
 1 | # lab4: Develop new bpftrace tools
 2 | 
 3 | ## Problem Statement
 4 | 
 5 | Gain experience developing new tools using bpftrace.
 6 | 
 7 | ## Instructions
 8 | 
 9 | You can start with the bpftrace tools in bpftrace/tools, and make copies for editing and customizing. Where noted, some of these labs are from the BPF book. You do not need to complete all of these.
10 | 
11 | A) Modify biolatency.bt to print a linear histogram instead, for the range 0 to 100 milliseconds and a step size of one millisecond. (From the BPF book Ch09.)
12 | 
13 | B) Modify biolatency.bt to print the linear histogram summary every one second. (From the BPF book Ch09.)
14 | 
15 | C) Copy opensnoop.bt to a new file, rmsnoop.bt. Change it to print files that were deleted (tip: the command is called "rm", but the syscall is called something else.) Use rmsnoop.bt to trace an SSH login: what did you see?
16 | 
17 | D) execsnoop.bt only sees new processes that call exec(2) (execve(2)), although some may fork(2) or clone(2) and not exec(2) (e.g., the creation of worker processes). Write a new tool called procsnoop(8) to show all new processes with as many details as possible. You could trace fork() and clone(), or use the sched tracepoints, or something else. (From the BPF book Ch06.)
18 | 
19 | ## References
20 | 
21 | - [bpftrace tutorial](https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md)
22 | - [bpftrace reference guide](https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md)
23 | - [BPF book tools](https://github.com/brendangregg/bpf-perf-tools-book#tools)
24 | - [BPF book](http://www.brendangregg.com/bpf-performance-tools-book.html)
25 | 
26 | Copyright Brendan Gregg, 2019. All rights reserved.
27 | 


--------------------------------------------------------------------------------
/lab5.md:
--------------------------------------------------------------------------------
 1 | # lab5: Advanced bpftrace tools (optional)
 2 | 
 3 | ## Problem Statement
 4 | 
 5 | Gain experience developing advanced tools using bpftrace.
 6 | 
 7 | ## Instructions
 8 | 
 9 | You can start with the bpftrace tools in bpftrace/tools and bpf-perf-tools-book/originals, and make copies for editing and customizing. Where noted, some of these labs are from the BPF book. This is an optional exercise for students who have finished the earlier labs. You do not need to finish these.
10 | 
11 | A) Develop a tool to show the ratio of logical file system I/O (via VFS or the file system interface) vs physical I/O (via block tracepoints). (From the BPF book Ch08.)
12 | 
13 | B) Develop a tool to analyze file descriptor leaks: those that were allocated during tracing but not freed. One possible solution may be to trace the kernel functions __alloc_fd() and __close_fd(). (From the BPF book Ch08.)
14 | 
15 | C) (Unsolved) Develop a tool to show the time between accesses in the page cache as a distribution. What are the challenges with this tool? (From the BPF book Ch08.)
16 | 
17 | The book contains more advanced and unsolved exercises.
18 | 
19 | ## References
20 | 
21 | - [bpftrace tutorial](https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md)
22 | - [bpftrace reference guide](https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md)
23 | - [BPF book tools](https://github.com/brendangregg/bpf-perf-tools-book#tools)
24 | - [BPF book](http://www.brendangregg.com/bpf-performance-tools-book.html)
25 | 
26 | Copyright Brendan Gregg, 2019. All rights reserved.
27 | 


--------------------------------------------------------------------------------
/src/Makefile:
--------------------------------------------------------------------------------
 1 | CC=gcc
 2 | CFLAGS=-O0
 3 | 
 4 | all: lab001 lab003
 5 | 
 6 | lab001: lab001.c
 7 | 	$(CC) -o ../$@ $< $(CFLAGS)
 8 | 
 9 | lab003: lab003.c
10 | 	$(CC) -o ../$@ $< $(CFLAGS) -pthread
11 | 
12 | 


--------------------------------------------------------------------------------
/src/README.md:
--------------------------------------------------------------------------------
1 | Please do not read the source.
2 | 
3 | The whole point of the labs is to make you analyze a black box. It's about the journey you take from knowing nothing to identifying the root cause(s). Reading the code spoils that and will make the lab ineffective.
4 | 


--------------------------------------------------------------------------------
/src/lab001.c:
--------------------------------------------------------------------------------
 1 | #include <sys/types.h>
 2 | #include <sys/stat.h>
 3 | #include <strings.h>
 4 | #include <fcntl.h>
 5 | #include <stdio.h>
 6 | #include <stdlib.h>
 7 | #include <unistd.h>
 8 | 
 9 | const char *datafile = "lab001.log";
10 | 
11 | #define BUFSIZE		(1 * 128)
12 | #define BIGSIZE		(2 * 1024 * 1024)
13 | #define FILESIZE	(50 * 1024 * 1024)
14 | 
15 | ssize_t
16 | os_write(int fd, const void *buf, size_t count)
17 | {
18 | 	ssize_t s;
19 | 	s = write(fd, buf, count);
20 | 	fsync(fd);
21 | 	return s;
22 | }
23 | 
24 | ssize_t
25 | debugdump(int fd, const void *buf, size_t count)
26 | {
27 |         return os_write(fd, buf, count);
28 | }
29 | 
30 | void
31 | write_log(int fd)
32 | {
33 | 	char *buf, *big;
34 | 	long long i;
35 | 	int ret, j;
36 | 
37 | 	buf = malloc(BUFSIZE);
38 | 	big = malloc(BIGSIZE);
39 | 	if (buf == NULL || big == NULL) {
40 | 		printf("ERROR: malloc buffers.\n");
41 | 		exit(1);
42 | 	}
43 | 	bzero(buf, BUFSIZE);
44 | 	bzero(big, BIGSIZE);
45 | 
46 | 	for (;;) {
47 | 		for (i = 0, j = 0; i < FILESIZE;) {
48 | 			if ((j++ % 200) == 50) {
49 | 				ret = debugdump(fd, big, BIGSIZE);
50 | 				i += BIGSIZE;
51 | 			} else {
52 | 				ret = os_write(fd, buf, BUFSIZE);
53 | 				i += BUFSIZE;
54 | 			}
55 | 
56 | 			if (ret < 0) {
57 | 				printf("ERROR: write error.\n");
58 | 				exit(2);
59 | 			}
60 | 		}
61 | 
62 | 		if (lseek(fd, 0, SEEK_SET) < 0) {
63 | 			printf("ERROR: seek() failed.\n");
64 | 			exit(3);
65 | 		}
66 | 	}
67 | 
68 | 	free(buf);
69 | 	free(big);
70 | }
71 | 
72 | int
73 | main()
74 | {
75 | 	int fd;
76 | 
77 | 	if ((fd = open(datafile, O_CREAT | O_WRONLY, 0644)) < 0) {
78 | 		printf("ERROR: writing to %s\n", datafile);
79 | 		exit(1);
80 | 	}
81 | 
82 | 	write_log(fd);
83 | 
84 | 	return (0);
85 | }
86 | 


--------------------------------------------------------------------------------
/src/lab003.c:
--------------------------------------------------------------------------------
  1 | #include <pthread.h>
  2 | #include <string.h>
  3 | #include <stdio.h>
  4 | #include <stdlib.h>
  5 | #include <unistd.h>
  6 | #include <errno.h>
  7 | #include <sys/types.h>
  8 | #include <sys/stat.h>
  9 | #include <fcntl.h>
 10 | #include <unistd.h>
 11 | #include <sys/syscall.h>
 12 | 
 13 | const char *datafile = "lab003.data";
 14 | 
 15 | #define BUFSIZE		(8 * 1024)
 16 | #define FILESIZE	(10 * 1024 * 1024)
 17 | #define THREADS 2
 18 | 
 19 | void
 20 | init_data()
 21 | {
 22 | 	int fd;
 23 | 	long long i;
 24 | 	char buf[BUFSIZE];
 25 | 
 26 | 	if ((fd = open(datafile, O_CREAT | O_WRONLY, 0644)) < 0) {
 27 | 		printf("ERROR: Can't open datafile: %s\n", datafile);
 28 | 		exit(1);
 29 | 	}
 30 | 
 31 | 	for (i = 0; i < FILESIZE / BUFSIZE; i++) {
 32 | 		if (write(fd, buf, BUFSIZE) < 0) {
 33 | 			printf("ERROR: write error.\n");
 34 | 			exit(2);
 35 | 		}
 36 | 	}
 37 | 
 38 | 	close(fd);
 39 | }
 40 | 
 41 | ssize_t
 42 | os_read(int fd, void *buf, size_t count)
 43 | {
 44 | 	// See what happens if you use read() instead of syscall():
 45 | 	//
 46 | 	// return read(fd, buf, count);
 47 | 	return syscall(SYS_read, fd, buf, count);
 48 | }
 49 | 
 50 | void
 51 | load_data()
 52 | {
 53 | 	int fd;
 54 | 	char buf[1];
 55 | 
 56 | 	if ((fd = open(datafile, O_RDONLY)) < 0) {
 57 | 		printf("ERROR: Can't open database: %s\n", datafile);
 58 | 		exit(2);
 59 | 	}
 60 | 
 61 | 	for (;;) { os_read(fd, &buf, 0); }
 62 | 
 63 | 	(void) close(fd);
 64 | }
 65 | 
 66 | void
 67 | import_database()
 68 | {
 69 | 	load_data();
 70 | }
 71 | 
 72 | void *
 73 | thread_start(void *arg)
 74 | {
 75 | 	for (;;) {
 76 | 		import_database();
 77 | 	}
 78 | }
 79 | 
 80 | int
 81 | main(int argc, char *argv[])
 82 | {
 83 | 	int i;
 84 | 	pthread_t threadids[THREADS];
 85 | 	char cmd[80];
 86 | 	pid_t pid;
 87 | 
 88 | 	init_data();
 89 | 
 90 | 	pid = getpid();
 91 | 	sprintf(cmd, "taskset -cp 0 %d >/dev/null\n", pid);
 92 | 	system(cmd);
 93 | 
 94 | 	for (i = 0; i < THREADS; i++) {
 95 | 		if ((pthread_create(&threadids[i], NULL, thread_start,
 96 | 		    NULL) != 0)) {
 97 | 			perror("Thread create failed");
 98 | 		}
 99 | 	}
100 | 
101 | 	for (i = 0; i < THREADS; i++)
102 | 		pthread_join(threadids[i], NULL);
103 | 
104 | 	return 0;
105 | }
106 | 


--------------------------------------------------------------------------------