├── misc.md ├── README.md ├── performance-analysis.md ├── mindset.md ├── debugging-ipsec-racoon.md ├── debugging-networking.md ├── testing-kernel-modules.md ├── bpftrace.md ├── coredump-analysis.md └── debugging-firejail.md /misc.md: -------------------------------------------------------------------------------- 1 | ### Speaking with a PHP-FPM socket via command line 2 | 3 | - https://www.dynatrace.com/news/blog/7-minute-workout-does-your-apache-web-server-need-love/ 4 | - https://serverfault.com/questions/516373/what-is-the-meaning-of-ah00485-scoreboard-is-full-not-at-maxrequestworkers 5 | - https://easyengine.io/tutorials/php/fpm-status-page/ 6 | 7 | ```bash 8 | #SCRIPT_NAME=/status SCRIPT_FILENAME=/status REQUEST_METHOD=GET cgi-fcgi -bind -connect /var/run/php/php7.2-fpm.sock 9 | export SCRIPT_NAME=/status 10 | export SCRIPT_FILENAME=/status 11 | export REQUEST_METHOD=GET 12 | cgi-fcgi -bind -connect /var/run/php/php7.2-fpm.sock 13 | ``` 14 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Linux Debugging Cheat Sheet 2 | state: not finished, but helpful enough to be published. Merge Requests welcome! 3 | 4 | UPDATE: for the current version, check https://debugging.works/blog/network-debugging/. 5 | 6 | This repo contains some useful commands and ideas for linux troubleshooting, performance analysis, ... 7 | 8 | - [How to solve hard (technical) problems](/mindset.md) 9 | - [Network debugging](/debugging-networking.md) 10 | - [Learning & debugging firejail](/debugging-firejail.md) 11 | - [~~Fighting~~ Debugging IPsec on linux (racoon + setkey)](/debugging-ipsec-racoon.md) 12 | - [How to debug a core dump](/coredump-analysis.md) 13 | - [Performance Analysis](/performance-analysis.md) 14 | - [How to build and debug linux kernel modules (easier than you think!)](testing-kernel-modules.md) 15 | - [How to use bpftrace](/bpftrace.md) 16 | - [Some PHP stuff](/misc.md) 17 | 18 | 19 | ## Good Reads 20 | - Brendan Gregg is great. Read and watch everything you catch: http://www.brendangregg.com/ 21 | - Linux Performance Analysis in 60s (video) 22 | - http://www.brendangregg.com/blog/2015-12-03/linux-perf-60s-video.html 23 | - https://netflixtechblog.com/linux-performance-analysis-in-60-000-milliseconds-accc10403c55 24 | - https://www.youtube.com/watch?v=FJW8nGV4jxY&list=PLhhdIMVi0o5RNrf8E2dUijvGpqKLB9TCR 25 | - https://www.slideshare.net/brendangregg/velocity-2015-linux-perf-tools 26 | - http://www.brendangregg.com/Slides/Velocity2015_LinuxPerfTools.pdf 27 | - Networking 28 | - https://blog.cloudflare.com/syn-packet-handling-in-the-wild/ 29 | - https://blog.nelhage.com/2010/08/write-yourself-an-strace-in-70-lines-of-code/ 30 | - https://veithen.io/2014/01/01/how-tcp-backlog-works-in-linux.html 31 | - eBPF 32 | - https://blog.pixielabs.ai/ebpf-http-tracing/ 33 | - else: 34 | - https://veithen.io/2013/11/18/iowait-linux.html 35 | - https://blog.heckel.io/2015/10/18/how-to-create-debian-package-and-debian-repository/ 36 | 37 | -------------------------------------------------------------------------------- /performance-analysis.md: -------------------------------------------------------------------------------- 1 | ## Performance Analysis 2 | This is based (copied) from Brendan Gregg! 3 | 4 | uptime 5 | - load sum averages over 1 minute, 5 minute, and 15 minute 6 | - How is the load changing? 7 | 8 | dmesg | tail 9 | - check for 10 | - oom-killer (Out of memory: Kill process 18694 (perl) score 246 or sacrifice child) 11 | - dropped network packets (TCP: Possible SYN flooding on port 7001. Dropping request. Check SNMP counters.) 12 | - broken disks (ata3.00: failed command: READ FPDMA QUEUED) 13 | 14 | vmstat -w 1 15 | - Package (procps, also includes kill/ps) 16 | - CPU: 17 | - is there idle time (100 - sy - us)? 18 | - high system time? System time is necessary for I/O processing. A high system time average, over 20%, can be interesting to explore further: perhaps the kernel is processing the I/O inefficiently. 19 | - r: # of processes running on CPU and waiting for a turn (without I/O) 20 | - to interpret: an "r" value greater than the CPU count is saturation 21 | - RAM 22 | - free memory in kb. If there are too many digits to count, you have enough free memory 23 | - si,so: swap-ins and swap-outs. If these are non-zero, you’re out of memory 24 | - Disk: 25 | - wa: constant degree of wait I/O points to a disk bottleneck; this is where the CPUs are idle, because tasks are blocked waiting for pending disk I/O 26 | 27 | mpstat -P ALL 1 28 | - Package: sysstat 29 | - CPU time breakdowns per CPU 30 | - check imbalance 31 | - single hot CPU? Single threaded application? 32 | 33 | pidstat 1 34 | - Package: sysstat 35 | - who uses the CPU? 36 | - sys? user? wait? 37 | - %CPU column is the total across all CPUs 38 | 39 | iostat -xz 1 40 | - Package: sysstat 41 | - r/s, w/s, rkB/s, wkB/s: These are the delivered reads, writes, read Kbytes, and write Kbytes per second to the device. 42 | - await: The average time for the I/O in milliseconds. This is the time that the application suffers, as it includes both time queued and time being serviced. Larger than expected average times can be an indicator of device saturation, or device problems. 43 | - %util: Device utilization. This is really a busy percent, showing the time each second that the device was doing work. Values greater than 60% typically lead to poor performance (which should be seen in await), although it depends on the device. Values close to 100% usually indicate saturation. 44 | 45 | 46 | free -h 47 | - Package: procps 48 | - buffers: For the buffer cache, used for block device I/O. 49 | - cached: For the page cache, used by file systems 50 | - free: how much memory is available for starting new applications, without swapping 51 | 52 | sar -n DEV 1 53 | - Package: sysstat 54 | 55 | sar -n TCP,ETCP 1 56 | active/s: Number of locally-initiated TCP connections per second (e.g., via connect()). 57 | passive/s: Number of remotely-initiated TCP connections per second (e.g., via accept()). 58 | retrans/s: Number of TCP retransmits per second. 59 | 60 | Retransmits are a sign of a network or server issue; it may be an unreliable network (e.g., the public Internet), or it may be due a server being overloaded and dropping packets. 61 | -------------------------------------------------------------------------------- /mindset.md: -------------------------------------------------------------------------------- 1 | # How to solve hard (technical) problems 2 | 3 | ### Mindset 4 | 5 | - There are no hard problems. There is just lack of information about how the system works 6 | - Remember that the bug is happening for a logical reason 7 | - Be unreasonably confident in your ability to fix the bug 8 | - The harder the bugs you solve the better you will be 9 | - Every error is an opportunity to learn 10 | 11 | ### Finding the root cause of the problem 12 | 13 | - Try to get the issue reproducible 14 | - Can you reproduce it from the command line? 15 | - It's easier for other people to reproduce the issue 16 | - It's easier to test the fix 17 | - Are there any log files? What's the error message? 18 | - Read the error description. Every word of it. Twice. 19 | - Is there a typo somewhere (command line/configuration/code)? 20 | - Isolate the problem 21 | - Remove some parts of the system and try to reproduce the bug 22 | - Vary one thing at a time while keeping all other things constant 23 | 24 | ### Issue still not fixed? Checklist 25 | 26 | - [ ] Try to tackle hard problems in the morning with a fresh mind and without disruption (before you check mails, chat, ticket system, monitoring) 27 | - [ ] Do you have multiple issues? Try to solve the underlying issue first (like a ssh connection that is dropped every minute) 28 | - [ ] Is it really a problem or just a misunderstanding (works as expected?) 29 | - [ ] Is there a security feature/policy that blocks your work? 30 | - [ ] Get a stable debugging environment 31 | - [ ] Does the problem occur only on a single server? The same thing is working somewhere else? 32 | - [ ] What's the difference? Compare! 33 | - [ ] When did the problem occur first? What has changed? 34 | - [ ] Can you increase the debug log? 35 | - [ ] Do some sanity checks 36 | - [ ] Are you on the right virtual machine? 37 | - [ ] Can you ping the target host? 38 | - [ ] Is DNS still working? 39 | - [ ] Check network traffic with ngrep/tcpdump. Do you see what you expect? 40 | - [ ] Is one of the disks full? 41 | - [ ] Are you editing the right file? 42 | - [ ] Write some garbage and try to compile/a syntax check 43 | - [ ] Check the monitoring system 44 | - [ ] Do other VMs of the customer have problems? 45 | - [ ] Do other VMs running on the same hypervisor have problems? 46 | - [ ] Is the whole data center down? 47 | - [ ] Is the customer logged in on the system? What is he doing (check bash_history and `ps -u`) 48 | 49 | 50 | ### After some time of debugging 51 | 52 | - Force yourself to express the problem in an easy and comprehensible way that a random teddy bear understands it 53 | - Be patient and accept that things just take longer than expected 54 | - Try to understand what happens. Not: endless trial and error 55 | - Is there documentation that can help you understanding the system 56 | - Talk to other people knowing the system better than you 57 | - Take a break (go for a walk, do some exercises, take a deep breath, drink some water, eat some fruits) 58 | - Step back: What's the problem? What's the reason for the problem? What's the actual goal you are trying to achieve? 59 | - You don't have time and stuck on some unrelated details? 60 | - Use a different approach to solve your actual problem 61 | 62 | 63 | ### If you copy/paste from Stackoverflow 64 | 65 | - Don't copy/paste from Stackoverflow without understanding the actual problem 66 | - Don't copy/paste from Stackoverflow without understanding the proposed solution 67 | - If you don't have time for it right now => make a note about it (even after solving it) 68 | - If you don't know what the command/tool is doing read the man page [(https://explainshell.com)](https://explainshell.com/) 69 | - Don't copy/paste commands/code. Type it on your own 70 | 71 | 72 | ### After solving the issue 73 | 1) Well done! I'm glad you didn't give up! 74 | 2) What have you learned? 75 | 2) What were the wrong assumptions? 76 | 3) How can you solve a similar problem in future even faster? 77 | -------------------------------------------------------------------------------- /debugging-ipsec-racoon.md: -------------------------------------------------------------------------------- 1 | ## ~~Fighting~~ Debugging IPsec on linux (racoon + setkey) 2 | [Why IPsec is hard to debug:](https://libreswan.org/wiki/Linux_IPsec_Summit_2018_wishlist#Fixup_XFRM_and_tcpdump) 3 | >The fact that you see some plaintext, but not all plaintext, is the most confusing aspect of IPsec to system administrators, who now believe hey are leaking plaintext. 4 | 5 | The better you know how a system works the better you can debug it. So before debugging IPsec read this: 6 | 7 | - https://wiki.strongswan.org/projects/strongswan/wiki/CorrectTrafficDump 8 | - https://devcentral.f5.com/s/articles/understanding-ikev1-negotiation-on-wireshark-34187 9 | 10 | 11 | #### Phase 1 12 | - [ ] Firewall allows `udp port 500`? Outgoing traffic allowed? 13 | - [ ] Is there communication between the both IPsec gateways? `tcpdump -ni any host ` 14 | - [ ] Are we sending packets to the remote endpoint? 15 | - [ ] Is the remote endpoint talking to us? 16 | - [ ] Are they both talking to each other? We had the problem: We speak IKEv1. They speak IKEv2. Our software was too old to recognize IKEv2 17 | - [ ] Are both using the same proposals? Use Wireshark for dissecting packets and compare parameters. 18 | - [ ] Any errors in `tail -f /var/log/syslog`? 19 | - [ ] Is the secret the same on both endpoints (prevent special characters on really old systems)? 20 | - [ ] We had problems using sha2 (the generated keys for phase2 where truncated at the wrong length) - use sha1 or sha512 21 | 22 | You think it works? 23 | 24 | - check `racoocnctl show-sa esp` (look for state=mature) 25 | - but: tunnel will only be established if there is some traffic 26 | - `ping -I ` 27 | 28 | 29 | #### Phase 2 30 | 31 | Check if the tunnel works: 32 | 33 | - You can ping a remote host? Great! Also check TCP in case of ugly NAT rules! 34 | - You send packets through the tunnel but no esp packets are sent? 35 | - Check it with `tcpdump -ni any host ` 36 | - [ ] Firewall allows `esp` packets in both directions? 37 | - [ ] Both endpoints have phase2 configured properly (ip ranges, crypto, ...)? 38 | - phase2 is encrypted. Wireshark is not that helpful here 39 | - [ ] You read the logs carefully? 40 | - [ ] You increased the debug level (`log notify, debug,and debug2`)? 41 | - [ ] You verified with tcpdump that the traffic that should go into the tunnel looks like you expect? 42 | - [ ] The firewall works as expected? 43 | - [ ] Use `-j TRACE`, `-j NFLOG` or `-j LOG` to see what iptables is trying to match 44 | - [ ] There is a rule in the FORWARD chain that allows the traffic? Rule is used? 45 | - [ ] There is _no_ rule in the `-t nat -L POSTROUTING` chain that NATs your traffic because of a rule like `-t nat -A POSTROUTING -o wan_interface -j MASQUERDE`? This would prevent it from going into the tunnel because source ip does not match anymore 46 | - [ ] Is there someone who can help you? 47 | - [ ] You took a break after the first hour of debugging? It probably won't get better! 48 | - [ ] You yelled at IPsec and asked `Why not Wireguard????`? 49 | - [ ] Traffic goes into the tunnel but there is no response (only outgoing `esp` packages but no incoming ones) 50 | - problem lays on the other side (ping not allowed? Firewall drops packages?) 51 | 52 | 53 | #### Debug commands 54 | `ip xfrm policy` shows the established phase2 connections 55 | `ip xfrm state` shows the keys and bytes used for a phase2 connection 56 | `ip xfrm policy` shows changes 57 | 58 | For debugging with tcpdump/iptables this packet flow overview can help: 59 | ![flow is great](https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg) 60 | 61 | 62 | #### TODOS 63 | - It should be possible to decrypt the phase2 handshakes like described [here](https://wiki.strongswan.org/projects/strongswan/wiki/CorrectTrafficDump) by reading the debug2 output of racoon 64 | 65 | 66 | #### Decrypt IPsec traffic 67 | 1. Use [this script](https://gist.github.com/rectalogic/ee2a48e47584fc0825dad9ffe571ec92) to generate a esp_sa file 68 | - the script puts the output of `ip xfrm state` in a format wireshark likes 69 | 2. Put the file to the right place `scp server:esp_sa ~/.config/wireshark/esp_sa` 70 | 3. Wireshark will automatically use the file. Use the Wireshark filter `!isakmp` 71 | - proof: Wireshark -> Edit -> Preferences -> Protocols -> ESP -> ESP SAs 72 | 73 | -------------------------------------------------------------------------------- /debugging-networking.md: -------------------------------------------------------------------------------- 1 | ## Networking Stuff 2 | 3 | ### Wireshark on remote host 4 | ssh server 'tcpdump -ni any -s0 -U -w - udp port 53' | wireshark -k -i - 5 | 6 | ### Wireshark on remote host over jump server 7 | 8 | ssh -J jumpserver server 'tcpdump -ni any -s0 -U -w - udp port 53' | wireshark -k -i - 9 | 10 | ### Wireshark on remote host over jump server (if the private key for the server is only on the jump host) 11 | 12 | on the jump server: 13 | >ssh server 'tcpdump -ni any -s0 -U -w- udp port 53' > /tmp/packets.pcap 14 | 15 | on our local machine: 16 | >ssh jump "tail -f -c +0 /tmp/packets.pcap" | wireshark -k -i - 17 | 18 | ### Find your traffic in tcpdump 19 | use source ports (NAT does not change the port) for filtering 20 | Explanation: for outgoing connections Linux uses a port out of `ip_local_port_range` 21 | 22 | >kmille@linbox /proc% cat sys/net/ipv4/ip_local_port_range 23 | >32768 60999 24 | 25 | This means: if we use a lower outgoing port we can easily find our packets 26 | 27 | >tcpdump -ni any portrange 2000-3000 28 | >nc -p 2000 -v localhost 8000 29 | >nc -s 192.168.10.70 -p 2000 -v localhost 8000 30 | >curl --local-port 2001 -v localhost:8000 31 | >curl --interface 192.168.10.70 --local-port 2001 -v localhost:8000 32 | 33 | ### Find the dropping firewall 34 | use `mtr` for a simple traceroute with a TCP SYN to 443 on top 35 | shows you where the packet is dropped when there are multiple firewalls and you don't know which is dropping your SYNs 36 | 37 | >mtr --tcp -P 443 server 38 | 39 | ### Useful tcpdump filter 40 | SYN only (someone trys to connect but the firewall drops => retransmission) 41 | >tcpdump -ni any 'tcp[tcpflags] == tcp-syn' 42 | 43 | Another way to find retransmissions: `sar -n ETCP 1` and check the `retrans/s` column 44 | To find which connection/application use 45 | 46 | >watch -n 0.1 'ss -tpn state syn-sent' 47 | 48 | SYN and SYN-ACK (show only newly established tcp connections) 49 | >tcpdump -ni any 'tcp[tcpflags] == tcp-syn or tcp[13]=18' 50 | 51 | SYN and RST (connect to a port which is closed) 52 | >tcpdump -ni any 'tcp[tcpflags] == tcp-syn or tcp[13] & 4!=0' 53 | 54 | SYN and ICMP port unreachable (firewall rejects packet) 55 | >tcpdump -ni any 'tcp[tcpflags] == tcp-syn or icmp[0] = 3' 56 | 57 | SYN AND SYN-ACK AND RST AND ICMP port unreachable 58 | >tcpdump -ni any '(tcp[tcpflags] == tcp-syn or tcp[13]=18) or tcp[13] & 4!=0 or icmp[0] = 3' 59 | 60 | For outgoing connections use [tcpconnect](https://github.com/iovisor/bcc/blob/master/tools/tcpconnect_example.txt)to get the process which is sending the packets. Or `ss -tanp` or `netstat -tanp`. 61 | 62 | ### Debugging Docker network issues 63 | Use `docker run --rm --network=container: -it nicolaka/netshoot` to get a shell with debugging tools in the network namespace of the container, that makes problem. You're not only in the same network as the target container, it also has access to the same ip. So you can just run tcpdump to get the incoming requests. I use these aliases in `~/.bashrc`. 64 | 65 | >dg() { # docker grep 66 | > docker ps --format "{{ .Names }}" | rg $1 67 | >} 68 | > 69 | >de() { # docker exec 70 | > docker exec -it $1 bash 71 | >} 72 | >den() { #docker exec network (run debug container with network of $1 container 73 | > docker run --rm --network=container:$1 -it nicolaka/netshoot 74 | >} 75 | 76 | 77 | ### Debugging iptables 78 | #### Clear and check the counter (pkts/bytes) 79 | >iptables -Z INPUT 80 | >iptables -Z INPUT 1 81 | >watch -n 0.1 iptables -vnL INPUT 82 | 83 | #### Log traffic to dmesg 84 | >iptables -I INPUT --match multiport --sports 2000:3000 -j LOG --log-prefix "our debug traffic" 85 | 86 | 87 | #### Use the nflog interface 88 | >iptables -A FORWARD -p icmp -j NFLOG --nflog-group 5 89 | >wireshark -ni nflog:5 90 | 91 | 92 | #### Trace the iptables rules linux uses for a package 93 | on older systems 94 | >modprobe ipt_LOG 95 | >echo ipt_LOG >/proc/sys/net/netfilter/nf_log/2 96 | 97 | on newer systems 98 | >modprobe nf_log_ipv4 99 | >sysctl net.netfilter.nf_log.2=nf_log_ipv4 100 | 101 | use the `raw` table 102 | use the `OUTPUT` chain for outgoing traffic 103 | use the `PREROUTING` chain for incoming traffic 104 | 105 | for testing 106 | >iptables -t raw -I OUTPUT -p icmp -j TRACE 107 | >iptables -t raw -I PREROUTING -p icmp -j TRACE 108 | 109 | check `dmesg` for some output. If there is no output: 110 | 1) check the `used by` column of lsmod (should be equal to the number of rules int the `raw` table) 111 | >kmille@linbox ~% lsmod | grep nf_log_ipv4 112 | >nf_log_ipv4 16384 2 113 | 114 | 2) check if you use nftables (like Debian buster) 115 | >root@buster:~# lsmod | grep nf_tables 116 | >nf_tables 143360 0 117 | >nfnetlink 16384 1 nf_tables 118 | 119 | use `nft monitor trace` instead of dmesg (on systems using netfilter and not nftables) 120 | other fix: you can use `/usr/sbin/iptables-legacy` instead of the new default `/usr/sbin/iptables-nft` 121 | 122 |
123 | sample output of -j TRACE 124 | 
125 | kmille@linbox ~% dmesg -wHT
126 | ...
127 | [Thu Apr 23 21:55:11 2020] TRACE: raw:OUTPUT:policy:2 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=31519 DF PROTO=ICMP TYPE=8 	CODE=0 ID=11 SEQ=1 UID=1000 GID=100 
128 | [Thu Apr 23 21:55:11 2020] TRACE: nat:OUTPUT:policy:1 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=31519 DF PROTO=ICMP TYPE=8 CODE=0 ID=11 SEQ=1 UID=1000 GID=100 
129 | [Thu Apr 23 21:55:11 2020] TRACE: filter:OUTPUT:policy:2 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=31519 DF PROTO=ICMP TYPE=8 CODE=0 ID=11 SEQ=1 UID=1000 GID=100 
130 | [Thu Apr 23 21:55:11 2020] TRACE: nat:POSTROUTING:rule:1 IN= OUT=wlp3s0 SRC=192.168.10.70 DST=1.1.1.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=31519 DF PROTO=ICMP TYPE=8 CODE=0 ID=11 SEQ=1 UID=1000 GID=100 
131 | [Thu Apr 23 21:55:11 2020] TRACE: raw:PREROUTING:policy:2 IN=wlp3s0 OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:00:00 SRC=1.1.1.1 DST=192.168.10.70 LEN=84 TOS=0x00 PREC=0x00 TTL=59 ID=44374 PROTO=ICMP TYPE=0 CODE=0 ID=11 SEQ=1 
132 | [Thu Apr 23 21:55:11 2020] TRACE: filter:INPUT:rule:3 IN=wlp3s0 OUT= MAC=00:00:00:00:00:00:00:00:00:00:00:00:00:00 SRC=1.1.1.1 DST=192.168.10.70 LEN=84 TOS=0x00 PREC=0x00 TTL=59 ID=44374 PROTO=ICMP TYPE=0 CODE=0 ID=11 SEQ=1 
133 | 
134 | For prettier output
135 | 
136 | kmille@linbox ~% dmesg  | grep TRACE: | egrep -v 'security|raw' | cut -d ' ' -f 3-8,14-17,21-22 | column -t
137 | nat:OUTPUT:policy:1     IN=        OUT=wlp3s0  SRC=192.168.10.70                              DST=193.99.144.80  LEN=84             PROTO=ICMP  TYPE=8  CODE=0  ID=13
138 | filter:OUTPUT:policy:2  IN=        OUT=wlp3s0  SRC=192.168.10.70                              DST=193.99.144.80  LEN=84             PROTO=ICMP  TYPE=8  CODE=0  ID=13
139 | nat:POSTROUTING:rule:1  IN=        OUT=wlp3s0  SRC=192.168.10.70                              DST=193.99.144.80  LEN=84             PROTO=ICMP  TYPE=8  CODE=0  ID=13
140 | filter:INPUT:rule:3     IN=wlp3s0  OUT=        MAC=00:00:00:00:00:00:00:00:00:00:00:00:00:00  SRC=193.99.144.80  DST=192.168.10.70  PROTO=ICMP  TYPE=0  CODE=0  ID=13
141 | 
142 | My mac addresses is redacted on purpose.
143 |
144 |
145 | 146 | for production use the following rules 147 | >iptables -t raw -I OUTPUT -p tcp -m multiport --sports 2000:3000 -j TRACE 148 | >iptables -t raw -I OUTPUT -p tcp -m multiport --dports 2000:3000 -j TRACE 149 | >iptables -t raw -I PREROUTING -p tcp -m multiport --dports 2000:3000 -j TRACE 150 | >iptables -t raw -I PREROUTING -p tcp -m multiport --sports 2000:3000 -j TRACE 151 | 152 | 153 | clear tracing 154 | >iptables -t raw -F 155 | -------------------------------------------------------------------------------- /testing-kernel-modules.md: -------------------------------------------------------------------------------- 1 | # How to debug a Linux Kernel Module (easy) 2 | 3 | ## A simple first test 4 | Try the Hello World exmpale from [cyberciti.biz](https://www.cyberciti.biz/tips/build-linux-kernel-module-against-installed-kernel-source-tree.html) 5 | 6 | First install the kernel header files with `apt-get install kernel-headers-$(uname -r)` 7 | 8 | Content of `hello.c`: 9 | 10 | ```c 11 | #include 12 | #include 13 | 14 | int init_module(void) 15 | { 16 | printk(KERN_INFO "init_module() called\n"); 17 | return 0; 18 | } 19 | 20 | void cleanup_module(void) 21 | { 22 | printk(KERN_INFO "cleanup_module() called\n"); 23 | } 24 | 25 | MODULE_LICENSE("GPL"); 26 | ``` 27 | 28 | 29 | Content of `Makefile`: 30 | 31 | ```Makefile 32 | obj-m := hello.o 33 | KDIR := /lib/modules/$(shell uname -r)/build 34 | PWD := $(shell pwd) 35 | default: 36 | $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules 37 | clean: 38 | rm -rf *.ko 39 | rm -rf *.mod.c 40 | rm -rf *mod.o 41 | rm -rf *.o 42 | 43 | ``` 44 | 45 | Build the module: 46 | >make 47 | >insmod hello.ko 48 | >rmmod hello.ko 49 | >dmesg: 50 | >[ 512.003876] init_module() called 51 | >[ 512.009321] cleanup_module() called 52 | 53 | 54 | 55 | ## Debug our module 56 | for example `nf_log_ipv4` (used for -j TRACE in iptables) 57 | 58 | 1. Get the source code 59 | - download it from Github or 60 | - git clone https://github.com/torvalds/linux 61 | - obviously: code must match the kernel version 62 | 2. Use multiple print statements (also print variables) 63 | - `printk(KERN_INFO "nf_log_ipv4: Passed %s %d \n",__FUNCTION__,__LINE__);` 64 | 3. In the Makefile replace `hello.o` with `nf_log_ipv4.o` 65 | - that's it. just use `make` to build it 66 | 4. Load the module into the kernel with `insmod nf_log_ipv4` 67 | - does not load dependencies => `symbol not found` error 68 | - modprobe automatically loads dependencies (needs a `modprobe` before) 69 | - modprobe loads modules from /lib/modules/$(uname -r)/ ... (insmod from the current dir) 70 | 71 | ### Automate the process 72 | I was testing why `iptables -j TRACE` didn't log something to dmesg. Solution: Debian buster uses nf_tables per default and you get the logs via `nft monitor trace`. If you use the old iptables (`/usr/sbin/iptables-legacy`) the messages will be logged to dmesg. 73 | 74 | Content of `rebuild.sh`: 75 | 76 | ```bash 77 | root@buster:~/mod/hello-example# cat rebuild.sh 78 | #/bin/bash 79 | #iptables=/usr/sbin/iptables-legacy 80 | iptables=iptables 81 | 82 | make clean 83 | make 84 | rmmod nf_log_ipv4 85 | modprobe nf_log_common 86 | insmod nf_log_ipv4.ko 87 | echo nf_log_ipv4 > /proc/sys/net/netfilter/nf_log/2 88 | cat /proc/net/netfilter/nf_log 89 | $iptables -t raw -F 90 | $iptables -t raw -I OUTPUT -p icmp -j TRACE 91 | $iptables -t raw -I PREROUTING -p icmp -j TRACE 92 | lsmod | grep nf_log 93 | ping -c 1 1.1.1.1 94 | $iptables -t raw -vnL 95 | ``` 96 | 97 | 98 | ## Find the source code/which module to look at? 99 | It's very easy to grep the kernel code. Use `strace` to get the user space part: 100 | ```bash 101 | root@linbox ~ strace -e network iptables -vnL 102 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 4 103 | getsockopt(4, SOL_IP, IPT_SO_GET_INFO, "filter\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., [84]) = 0 104 | getsockopt(4, SOL_IP, IPT_SO_GET_ENTRIES, "filter\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., [2984]) = 0 105 | Chain INPUT (policy DROP 0 packets, 0 bytes) 106 | pkts bytes target prot opt in out source destination 107 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 108 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_MATCH, 0x7fff31ce9a90, [30]) = -1 EPROTONOSUPPORT (Protocol not supported) 109 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 110 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_MATCH, "conntrack\0\202\365d\177\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1", [30]) = 0 111 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 112 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_MATCH, 0x7fff31ce9a40, [30]) = -1 EPROTONOSUPPORT (Protocol not supported) 113 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 114 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_MATCH, "conntrack\0\214\365d\177\0\0\1\0\0\0\0\0\0\0\3111\200\365d\2", [30]) = 0 115 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 116 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_MATCH, 0x7fff31ce99f0, [30]) = -1 EPROTONOSUPPORT (Protocol not supported) 117 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 118 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_MATCH, "conntrack\0\227\342\332y\347X\1\0\0\0\0\0\0\0\3111\200\365d\3", [30]) = 0 119 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 120 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_MATCH, 0x7fff31ce99a0, [30]) = -1 EPROTONOSUPPORT (Protocol not supported) 121 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 122 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_MATCH, "conntrack\0\214\365d\177\0\0\0\0\0\0\0\0\0\0\3111\200\365\1\3", [30]) = 0 123 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 124 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_MATCH, "conntrack\0\214\365d\177\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\3", [30]) = 0 125 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 126 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_MATCH, "conntrack\0\214\365d\177\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\2", [30]) = 0 127 | 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID 128 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 129 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_MATCH, 0x7fff31ce9d00, [30]) = -1 EPROTONOSUPPORT (Protocol not supported) 130 | 1472K 1056M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 131 | 1619 97140 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 132 | 0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 133 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 134 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_TARGET, "REJECT\0\0s\371\202\365d\177\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", [30]) = 0 135 | 0 0 REJECT all -- cdark.net * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 136 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 137 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_MATCH, "tcp\0d\177\0\0s\371\202\365d\177\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", [30]) = 0 138 | 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 139 | 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22000 140 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 5 141 | getsockopt(5, SOL_IP, IPT_SO_GET_REVISION_MATCH, "udp\0d\177\0\0s\371\202\365d\177\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", [30]) = 0 142 | 0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:21027 143 | 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:5001 144 | 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080 145 | 63 17777 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 146 | 147 | Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) 148 | pkts bytes target prot opt in out source destination 149 | 150 | Chain OUTPUT (policy ACCEPT 398K packets, 4393M bytes) 151 | pkts bytes target prot opt in out source destination 152 | +++ exited with 0 +++ 153 | root@linbox ~ # 154 | ``` 155 | 156 | To get the kernel space code (for example for IPT_SO_GET_INFO): 157 | 158 | - Google: `torvald/linux IPT_SO_GET_INFO` 159 | - you will find https://github.com/torvalds/linux/blob/master/net/ipv4/netfilter/ip_tables.c 160 | - use https://elixir.bootlin.com/linux/latest/ident/IPT_SO_GET_INFO as search engine 161 | 162 | 163 | 164 | -------------------------------------------------------------------------------- /bpftrace.md: -------------------------------------------------------------------------------- 1 | # How to use bpftrace 2 | 3 | We can trace arbitrary kernel and userspace functions. 4 | 5 | ## Which kernel functions trace? 6 | 7 | `bpftrace -l | grep kprobe` 8 | 9 | Functions, the kernel knows right now: 10 | 11 | `root@buster:/sys/kernel/debug/tracing# cat available_filter_functions ` 12 | 13 | We have the `ac` kernel module loaded: 14 | 15 | `root@buster:/sys/kernel/debug/tracing# lsmod | grep ac` 16 | `ac 16384 0` 17 | 18 | Functions of the kernel module `ac`: 19 | 20 | `root@buster:/sys/kernel/debug/tracing# cat available_filter_functions | grep '\[ac\]' 21 | acpi_ac_get_state [ac] 22 | acpi_ac_resume [ac] 23 | acpi_ac_notify [ac] 24 | acpi_ac_remove [ac] 25 | acpi_ac_battery_notify [ac] 26 | get_ac_property [ac] 27 | acpi_ac_add [ac]` 28 | 29 | 30 | 31 | ## Example 32 | 33 | 1) Let's use `strace` to find some interesting system calls. 34 | 35 | ```bash 36 | root@buster:/home/vagrant# strace -e network iptables-legacy -nL 37 | socket(AF_INET, SOCK_RAW, IPPROTO_RAW) = 4 38 | getsockopt(4, SOL_IP, IPT_SO_GET_INFO, "filter\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., [84]) = 0 39 | getsockopt(4, SOL_IP, IPT_SO_GET_ENTRIES, "filter\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., [672]) = 0 40 | Chain INPUT (policy ACCEPT) 41 | target prot opt source destination 42 | 43 | Chain FORWARD (policy ACCEPT) 44 | target prot opt source destination 45 | 46 | Chain OUTPUT (policy ACCEPT) 47 | target prot opt source destination 48 | +++ exited with 0 +++ 49 | ``` 50 | 51 | 2) Let's find out which kernel we use. Get the kernel source code and checkout the right version. 52 | 53 | ```bash 54 | root@buster:~/mod/linux# uname -a 55 | Linux buster 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1 (2020-01-26) x86_64 GNU/Linux 56 | 57 | root@buster:~/mod/linux# git status 58 | HEAD detached at v4.19-rc8 59 | nothing to commit, working tree clean 60 | ``` 61 | 62 | 3) grep for some function names, constants or variable names 63 | 64 | ```c 65 | root@buster:~/mod/linux# rg IPT_SO_GET_INFO 66 | net/ipv4/ip_sockglue.c 67 | 1563: if (optname >= BPFILTER_IPT_SO_GET_INFO && 68 | 1600: if (optname >= BPFILTER_IPT_SO_GET_INFO && 69 | 70 | net/ipv4/netfilter/ip_tables.c 71 | 1652: case IPT_SO_GET_INFO: 72 | 1698: case IPT_SO_GET_INFO: 73 | 74 | include/uapi/linux/bpfilter.h 75 | 14: BPFILTER_IPT_SO_GET_INFO = 64, 76 | 77 | include/uapi/linux/netfilter_ipv4/ip_tables.h 78 | 140:#define IPT_SO_GET_INFO (IPT_BASE_CTL) 79 | 156:/* The argument to IPT_SO_GET_INFO */ 80 | 81 | ``` 82 | 83 | In `net/ipv4/netfilter/ip_tables.c looks promising` there are two interesting functions: `compat_do_ipt_get_ctl` and `do_ipt_get_ctl`. Let's find out if they are called. 84 | 85 | ```bash 86 | root@buster:~# bpftrace -e 'kprobe:do_ipt_get_ctl { printf("function was called!\n"); }' 87 | Attaching 1 probe... 88 | function was called! 89 | function was called! 90 | ``` 91 | 92 | Let's get some more details. This is how the signature of the `compat_do_ipt_get_ctl` function looks like: 93 | 94 | ```c 95 | static int compat_do_ipt_get_ctl(struct sock *sk, int cmd, void __user *user, int *len) 96 | ``` 97 | 98 | Let's write some code. 99 | 100 | ```c 101 | #include 102 | 103 | kprobe:do_ipt_get_ctl 104 | { 105 | printf("called by %s (pid: %d). and: %d\n", comm, pid, ((sock *)arg0)->__sk_common.skc_family); 106 | } 107 | ``` 108 | 109 | And execute it. 110 | 111 | ```c 112 | root@buster:~# bpftrace mypbraceprogram.bpf 113 | /bpftrace/include/stdarg.h:52:1: warning: null character ignored [-Wnull-character] 114 | /lib/modules/4.19.0-8-amd64/source/arch/x86/include/asm/bitops.h:209:2: error: 'asm goto' constructs are not supported yet 115 | /lib/modules/4.19.0-8-amd64/source/arch/x86/include/asm/bitops.h:256:2: error: 'asm goto' constructs are not supported yet 116 | /lib/modules/4.19.0-8-amd64/source/arch/x86/include/asm/bitops.h:310:2: error: 'asm goto' constructs are not supported yet 117 | /lib/modules/4.19.0-8-amd64/source/arch/x86/include/asm/jump_label.h:23:2: error: 'asm goto' constructs are not supported yet 118 | /lib/modules/4.19.0-8-amd64/source/arch/x86/include/asm/signal.h:24:2: note: array 'sig' declared here 119 | Attaching 1 probe... 120 | called by iptables-legacy (pid: 2981). and: 2 121 | called by iptables-legacy (pid: 2981). and: 2 122 | ``` 123 | 124 | What does the `2` mean? 125 | 126 | ```c 127 | /usr/src/linux-headers-4.19.0-8-common/include/linux/socket.h 128 | 160 /* Supported address families. */ 129 | 161 #define AF_UNSPEC 0 130 | 162 #define AF_UNIX 1 /* Unix domain sockets */ 131 | 163 #define AF_LOCAL 1 /* POSIX name for AF_UNIX */ 132 | 164 #define AF_INET 2 /* Internet IP Protocol */ 133 | 165 #define AF_AX25 3 /* Amateur Radio AX.25 */ 134 | 166 #define AF_IPX 4 /* Novell IPX */ 135 | ``` 136 | 137 | So it's just IPv4. Some explanation of `((sock *)arg0)->__sk_common.skc_family)`. The first parameter of `compat_do_ipt_get_ctl` is a pointer to a `socket`. A `socket` has another data structure of type `sock_common` called `__sk_common`. And ``__sk_common`` has the attribute `skc_family` . You can (have to) find it out by grepping through the code. 138 | 139 | ```c 140 | root@buster:/usr/src/linux-headers-4.19.0-8-common/include# rg '^struct sock \{' 141 | net/sock.h 142 | 327:struct sock { 143 | .... 144 | 239 /** 145 | 240 * struct sock - network layer representation of sockets 146 | 241 * @__sk_common: shared layout with inet_timewait_sock 147 | 242 * @sk_shutdown: mask of %SEND_SHUTDOWN and/or %RCV_SHUTDOWN 148 | 243 * @sk_userlocks: %SO_SNDBUF and %SO_RCVBUF settings 149 | 244 * @sk_lock: synchronizer 150 | 245 * @sk_kern_sock: True if sock is using kernel lock classes 151 | ... 152 | 322 * @sk_rcu: used during RCU grace period 153 | 323 * @sk_clockid: clockid used by time-based scheduling (SO_TXTIME) 154 | 324 * @sk_txtime_deadline_mode: set deadline mode for SO_TXTIME 155 | 325 * @sk_txtime_unused: unused txtime flags 156 | 326 */ 157 | 327 struct sock { 158 | 328 /* 159 | 329 * Now struct inet_timewait_sock also uses sock_common, so please just 160 | 330 * don't add nothing before this first member (__sk_common) --acme 161 | 331 */ 162 | 332 struct sock_common __sk_common; 163 | 333 #define sk_node __sk_common.skc_node 164 | 165 | ... 166 | 167 | 123 * struct sock_common - minimal network layer representation of sockets 168 | 124 * @skc_daddr: Foreign IPv4 addr 169 | 125 * @skc_rcv_saddr: Bound local IPv4 addr 170 | 126 * @skc_hash: hash value used with various protocol lookup tables 171 | 127 * @skc_u16hashes: two u16 hash values used by UDP lookup tables 172 | 128 * @skc_dport: placeholder for inet_dport/tw_dport 173 | 129 * @skc_num: placeholder for inet_num/tw_num 174 | 130 * @skc_family: network address family <----- here it is 175 | 131 * @skc_state: Connection state 176 | ... 177 | 143 * @skc_flags: place holder for sk_flags 178 | 144 * %SO_LINGER (l_onoff), %SO_BROADCAST, %SO_KEEPALIVE, 179 | 145 * %SO_OOBINLINE settings, %SO_TIMESTAMPING settings 180 | 146 * @skc_incoming_cpu: record/match cpu processing incoming packets 181 | 147 * @skc_refcnt: reference count 182 | 148 * 183 | 149 * This is the minimal network layer representation of sockets, the header 184 | 150 * for struct sock and struct inet_timewait_sock. 185 | 151 */ 186 | 152 struct sock_common { 187 | 188 | 189 | 190 | ``` 191 | 192 | Another example: `bpftrace -e 'kprobe:vfs_open { printf("open path: %s\n", str(((path *)arg0)->dentry->d_name.name)); }'` 193 | 194 | 195 | 196 | # Useful links 197 | 198 | - https://www.joyfulbikeshedding.com/blog/2019-01-31-full-system-dynamic-tracing-on-linux-using-ebpf-and-bpftrace.html 199 | 200 | - https://lwn.net/Articles/793749/ 201 | 202 | - http://www.brendangregg.com/BPF/bpftrace-cheat-sheet.html 203 | - https://github.com/iovisor/bpftrace/blob/master/docs/tutorial_one_liners.md 204 | - https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md#6-tracepoint-static-tracing-kernel-level-arguments 205 | - https://jvns.ca/blog/2017/07/05/linux-tracing-systems/ 206 | - https://piware.de/post/2020-02-28-bpftrace/ -------------------------------------------------------------------------------- /coredump-analysis.md: -------------------------------------------------------------------------------- 1 | ## Some tools, links and snippets for debugging software on (Arch) Linux 2 | 3 | On Arch Linux, we get "unlimited" core dumps. They are stored in the `/var/lib/systemd/coredump` directory. 4 | 5 | ```bash 6 | kmille@linbox:~ ulimit -c 7 | unlimited 8 | kmille@linbox:~ cat /proc/sys/kernel/core_pattern 9 | |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h 10 | kmille@linbox:~ file /var/lib/systemd/coredump/* | head -n 1 11 | /var/lib/systemd/coredump/core.a\x2eout.1000.cb780d16ec3e434aaaa6a69c01e0abe8.1270928.1611849944000000.zst: Zstandard compressed data (v0.8+), Dictionary ID: None 12 | ``` 13 | 14 | List the core dumps. 15 | 16 | ```bash 17 | kmille@linbox:~ coredumpctl list 18 | 19 | TIME PID UID GID SIG COREFILE EXE 20 | ... 21 | Wed 2021-01-27 15:02:03 CET 489025 1000 100 11 present /usr/bin/urxvt 22 | Wed 2021-01-27 15:19:08 CET 491085 1000 100 11 present /usr/bin/urxvt 23 | Wed 2021-01-27 15:20:53 CET 434557 1000 100 11 present /usr/bin/urxvt 24 | Wed 2021-01-27 16:39:27 CET 492570 1000 100 11 present /usr/bin/urxvt 25 | Wed 2021-01-27 18:50:17 CET 495418 1000 100 11 present /usr/bin/urxvt 26 | Wed 2021-01-27 18:51:12 CET 499149 1000 100 11 present /usr/bin/urxvt 27 | ``` 28 | 29 | Dump the core file the current working directory. 30 | 31 | ```bash 32 | kmille@linbox:~ coredumpctl dump 499149 --output 499149.core 33 | PID: 499149 (urxvt) 34 | UID: 1000 (kmille) 35 | GID: 100 (users) 36 | Signal: 11 (SEGV) 37 | Timestamp: Wed 2021-01-27 18:51:12 CET (7min ago) 38 | Command Line: urxvt 39 | Executable: /usr/bin/urxvt 40 | Control Group: /user.slice/user-1000.slice/session-1.scope 41 | Unit: session-1.scope 42 | Slice: user-1000.slice 43 | Session: 1 44 | Owner UID: 1000 (kmille) 45 | Boot ID: cb780d16ec3e434aaaa6a69c01e0abe8 46 | Machine ID: 2ea1cb1e148d4172ad4915fa6e5d48c9 47 | Hostname: linbox 48 | Storage: /var/lib/systemd/coredump/core.urxvt.1000.cb780d16ec3e434aaaa6a69c01e0abe8.499149.1611769872000000.zst 49 | Message: Process 499149 (urxvt) of user 1000 dumped core. 50 | Stack trace of thread 499149: 51 | #0 0x00007f25969b34ed getenv (libc.so.6 + 0x3f4ed) 52 | #1 0x00007f2596bd25c6 perl_destruct (libperl.so + 0x735c6) 53 | #2 0x0000555f62069336 _ZN16rxvt_perl_interpD1Ev (urxvt + 0x4d336) 54 | #3 0x00007f25969b3db7 __run_exit_handlers (libc.so.6 + 0x3fdb7) 55 | #4 0x00007f25969b3f5e exit (libc.so.6 + 0x3ff5e) 56 | #5 0x00007f259699c159 __libc_start_main (libc.so.6 + 0x28159) 57 | #6 0x0000555f6204acfe _start (urxvt + 0x2ecfe) 58 | 59 | kmille@linbox: file 499149.core 60 | 499149.core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'urxvt', real uid: 1000, effective uid: 1000, real gid: 100, effective gid: 100, execfn: '/usr/bin/urxvt', platform: 'x86_64' 61 | ``` 62 | 63 | load urxvt with the core dump into gdb. 64 | 65 | ```bash 66 | kmille@linbox: gdb -c 499149.core $(which urxvt) 67 | ... 68 | Reading symbols from /usr/bin/urxvt... 69 | (No debugging symbols found in /usr/bin/urxvt) 70 | [New LWP 499149] 71 | [Thread debugging using libthread_db enabled] 72 | Using host libthread_db library "/usr/lib/libthread_db.so.1". 73 | Core was generated by `urxvt'. 74 | Program terminated with signal SIGSEGV, Segmentation fault. 75 | #0 0x00007f25969b34ed in getenv () from /usr/lib/libc.so.6 76 | (gdb) bt 77 | #0 0x00007f25969b34ed in getenv () from /usr/lib/libc.so.6 78 | #1 0x00007f2596bd25c6 in perl_destruct () from /usr/lib/perl5/5.32/core_perl/CORE/libperl.so 79 | #2 0x0000555f62069336 in rxvt_perl_interp::~rxvt_perl_interp() () 80 | #3 0x00007f25969b3db7 in __run_exit_handlers () from /usr/lib/libc.so.6 81 | #4 0x00007f25969b3f5e in exit () from /usr/lib/libc.so.6 82 | #5 0x00007f259699c159 in __libc_start_main () from /usr/lib/libc.so.6 83 | #6 0x0000555f6204acfe in _start () 84 | (gdb) 85 | ``` 86 | At this point we don't have more information than in the output of ` coredumpctl dump 499149`. We still don't know which line in urxvt caused the segmentation fault. The problem is that gdb does not find the symbols (`No debugging symbols found in /usr/bin/urxvt`). 87 | 88 | Some ways to find out if a binary contains debug symbols. 89 | 90 | ```bash 91 | kmille@linbox:linux-debugging objdump --syms $(which urxvt) 92 | 93 | /usr/bin/urxvt: file format elf64-x86-64 94 | 95 | SYMBOL TABLE: 96 | no symbols 97 | 98 | kmille@linbox:linux-debugging: readelf --debug-dump=decodedline $(which urxvt) 99 | kmille@linbox:linux-debugging: 100 | 101 | ``` 102 | 103 | Let's build urxvt by our own and add debug symbols. 104 | 105 | ```bash 106 | kmille@linbox:tmp asp checkout rxvt-unicode 107 | Cloning into 'rxvt-unicode'... 108 | done. 109 | kmille@linbox:tmp cd rxvt-unicode 110 | kmille@linbox:rxvt-unicode cd repos/community-x86_64 111 | kmille@linbox:community-x86_64 vim PKGBUILD # add options=(debug !strip) 112 | kmille@linbox:community-x86_64 makepkg -i # this will build and install the package 113 | kmille@linbox:community-x86_64 makepkg -ef # if you edit the source and want to rebuild: 114 | # use -e to use the local source code and skip the prepare() function 115 | ``` 116 | 117 | Now our gdb shows us the line in which the error occurs (./rxvtperl.xs:379). 118 | 119 | ```bash 120 | (gdb) bt 121 | #0 0x00007ffff774d4ed in getenv () from /usr/lib/libc.so.6 122 | #1 0x00007ffff796c5c6 in perl_destruct () from /usr/lib/perl5/5.32/core_perl/CORE/libperl.so 123 | #2 0x00005555555a1336 in rxvt_perl_interp::~rxvt_perl_interp (this=, __in_chrg=) at ./rxvtperl.xs:379 124 | #3 0x00007ffff774ddb7 in __run_exit_handlers () from /usr/lib/libc.so.6 125 | #4 0x00007ffff774df5e in exit () from /usr/lib/libc.so.6 126 | #5 0x00007ffff7736159 in __libc_start_main () from /usr/lib/libc.so.6 127 | #6 0x0000555555582cfe in _start () 128 | (gdb) 129 | ``` 130 | 131 | I use ` auto-load safe-path /` in my global ~/.gdbinit. It allows me to use .gdbinit of the current working directory. 132 | 133 | ```bash 134 | kmille@linbox: cat ~/.gdbinit 135 | set auto-load safe-path / 136 | ``` 137 | 138 | The segfault happens if `getenv` is called. Let's use `man 3 getenv` to get the parameters of it. 139 | 140 | char *getenv(const char *name); 141 | 142 | We can use `man syscall` to find out where the arguments are stored. We are interested in the first (and only argument). We can find it in the `rdi` register. 143 | 144 | Arch/ABI Instruction System Ret Ret Error Notes 145 | x86-64 syscall rax rax rdx - 5 146 | 147 | Arch/ABI arg1 arg2 arg3 arg4 arg5 arg6 arg7 Notes 148 | x86-64 rdi rsi rdx r10 r8 r9 - 149 | 150 | Let's script gdb. I want to see all calls to `getenv` including the parameter. `commands 1` refers to the first breakpoint (set by `break getenv`). The `set breakpoint pending on` is important because when we set the breakpoint the `getenv` function is not resolved at this time. 151 | 152 | ```bash 153 | kmille@linbox:cat .gdbinit 154 | set breakpoint pending on 155 | set pagination off 156 | 157 | break getenv 158 | commands 1 159 | x/s $rdi 160 | continue 161 | end 162 | run 163 | ``` 164 | 165 | The output of gdb looks like this: 166 | 167 | ```bash 168 | kmille@linbox:debugging-urxvt gdb -q $(which urxvt) 169 | Reading symbols from /usr/bin/urxvt... 170 | (No debugging symbols found in /usr/bin/urxvt) 171 | Function "getenv" not defined. 172 | Breakpoint 1 (getenv) pending. 173 | [Thread debugging using libthread_db enabled] 174 | Using host libthread_db library "/usr/lib/libthread_db.so.1". 175 | 176 | Breakpoint 1, __GI_getenv (name=0x7ffff7219210 "G_MESSAGES_PREFIXED") at getenv.c:34 177 | 34 getenv.c: No such file or directory. 178 | 0x7ffff7219210: "G_MESSAGES_PREFIXED" 179 | 180 | Breakpoint 1, __GI_getenv (name=0x7ffff7219240 "G_DEBUG") at getenv.c:34 181 | 34 in getenv.c 182 | 0x7ffff7219240: "G_DEBUG" 183 | 184 | Breakpoint 1, __GI_getenv (name=0x7ffff7222378 "G_SLICE") at getenv.c:34 185 | ``` 186 | 187 | In my case the urxvt terminal appears as normal. No problem/segfault so far. If I close urxvt, the segfault happens. 188 | 189 | ```bash 190 | Breakpoint 1, __GI_getenv (name=name@entry=0x7ffff7a504d6 "PERL_DESTRUCT_LEVEL") at getenv.c:34 191 | 34 in getenv.c 192 | 0x7ffff7a504d6: "PERL_DESTRUCT_LEVEL" 193 | 194 | Program received signal SIGSEGV, Segmentation fault. 195 | __GI_getenv (name=0x7ffff7a504d8 "RL_DESTRUCT_LEVEL", name@entry=0x7ffff7a504d6 "PERL_DESTRUCT_LEVEL") at getenv.c:84 196 | 84 in getenv.c 197 | (gdb) 198 | ``` 199 | 200 | A workaround was to start urxvt with [PERL_DESTRUCT_LEVEL=2](https://perldoc.perl.org/perlhacktips#PERL_DESTRUCT_LEVEL) as environment variable. 201 | 202 | ``` 203 | kmille@linbox:~ urxvt 204 | zsh: segmentation fault (core dumped) urxvt 205 | kmille@linbox:~ PERL_DESTRUCT_LEVEL=2 urxvt 206 | kmille@linbox:~ 207 | ``` 208 | 209 | The bug was already known. It was discussed [here](https://bugzilla.redhat.com/show_bug.cgi?id=1894917) and [here](http://lists.schmorp.de/pipermail/rxvt-unicode/2020q2/002583.html). I didn't understood the underlying problem but I learned how to dig deeper. I will need it in the future so I made this document. I proposed a patch in the [Arch bug report](https://bugs.archlinux.org/task/67691#comment196105). 210 | 211 | ## Some useful things 212 | 213 | - use `set style enabled off` if you have redshift on 214 | - use `disas ` and you will see the exact position (marked with =>) where the segfault happens 215 | - use `directory ` in the gdbinit to tell gdb where the source code is 216 | - then use `gdb -tui` 217 | - use `gdb -q` to hide the license message at the start of gdb 218 | - use `break basic.c:101 ` to set a breakpoint in a dedicated file and line 219 | - use `jump +1` to skip a line of c code (like an assert) 220 | - build software with `-ggdb3` to get debug symbols 221 | - use `print argv[0]` and `ptype argv` 222 | - use `info locals`, `info args`, `info registers` 223 | - use `x/gx` (giant hex) to get the address of a variable 224 | - check the valgrind output `valgrind --leak-check=full --show-leak-kinds=all --track-origins=yes --verbose --log-file=valgrind-out.txt urxvt` 225 | 226 | ## Resources 227 | - http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html 228 | - https://jvns.ca/blog/2018/04/28/debugging-a-segfault-on-linux/ 229 | - https://wiki.archlinux.org/index.php/Core_dump 230 | - https://wiki.archlinux.org/index.php/Debug_-_Getting_Traces 231 | - https://wiki.archlinux.org/index.php/Debugging 232 | - https://stackoverflow.com/questions/5134891/how-do-i-use-valgrind-to-find-memory-leaks 233 | - GDB cheatsheet: https://gist.github.com/rkubik/b96c23bd8ed58333de37f2b8cd052c30 234 | 235 | ## TODOs and ideas 236 | 237 | - set conditional breakpoints 238 | - use eBPF to trace all getenv calls 239 | - the ltrace output was just empty. I expected to see calls to getenv 240 | - use strace to debug segfaults 241 | - learn to use `vcs` 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | -------------------------------------------------------------------------------- /debugging-firejail.md: -------------------------------------------------------------------------------- 1 | # Learning & debugging firejail 2 | 3 | So, what it firejail? 4 | 5 | > Firejail is a SUID program that reduces the risk of security breaches by restricting the running environment of untrusted applications using Linux namespaces and seccomp-bpf. It allows a process and all its descendants to have their own private view of the globally shared kernel resources, such as the network stack, process table, mount table. 6 | 7 | It's basically an easy-to-use sandbox of the risk of running everything as root (because it's a setuid binary, even if it drops privileges later). I really like firejail, but there are some things I don't like: 8 | 9 | - The website is outdated!? The man page [of the website](https://firejail.wordpress.com/features-3/man-firejail/) are older than the man page of the [Arch Linux package](https://man.archlinux.org/man/firejail.1). Newer features like `--nettrace` are not listed on the website. Things like `--audit` has been moved to the `jailcheck` tool. 10 | - The docs are good, but sometimes not super clear. Some more examples and command line output would be nice. For example: how to disable _all_ [protocols](https://man.archlinux.org/man/firejail.1#protocol=protocol,protocol,protocol)? `protocol`, `protocol none `, `net none`? I still don't know. 11 | - Sometimes it's hard to debug a failing profile 12 | 13 | That's why I want to share some of my learnings. 14 | 15 | # How to learn about firejail features 16 | 17 | 1. Create an empty firejail profile called `test`: `touch ~/.config/firejail/test.profile` 18 | 2. Run a shell in the `test` sandbox: `firejail --profile=test bash` 19 | 3. Play around in the shell: what works and what not (some things like /boot are blacklisted by default) 20 | 4. Modify `test.profile`: use `blacklist`, `whitelist`, `noblacklist`, `mkdir`, `private-bin`, `net none`, `private`, ... 21 | 5. Check out the [man page](https://man.archlinux.org/man/firejail.1#NAME) and find out, how things behave 22 | 23 | # Debugging firejail profiles 24 | 25 | ### join a sandbox 26 | 27 | You can start a shell in the sandbox of a running application. You can use `firejail list` and then `firejail --join=id` or start the program with a dedicated name (sometimes the program name works too - like `--join=keepassxc`). 28 | 29 | ```bash 30 | kmille@linbox:tmp firejail --name=keepass keepassxc 31 | Reading profile /etc/firejail/keepassxc.profile 32 | Reading profile /home/kmille/.config/firejail/keepassxc.local 33 | .... 34 | kmille@linbox:~ firejail --join=keepass 35 | Switching to pid 124932, the first child process inside the sandbox 36 | Changing root to /proc/124932/root 37 | Child process initialized in 21.92 ms 38 | linbox% ls 39 | zsh: command not found: ls 40 | ``` 41 | 42 | If there are debugging tools (like ls) missing, the profile uses `private-bin`. You have to restart the application and add the desired debug tools: 43 | 44 | ```bash 45 | kmille@linbox:tmp firejail --name=keepass --private-bin=ls keepassxc 46 | Reading profile /etc/firejail/keepassxc.profile 47 | ... 48 | kmille@linbox:~ firejail --join=keepass 49 | Switching to pid 125282, the first child process inside the sandbox 50 | Changing root to /proc/125282/root 51 | Child process initialized in 22.68 ms 52 | linbox% cd /usr/bin 53 | linbox% ls 54 | keepassxc keepassxc-cli keepassxc-proxy ls 55 | ``` 56 | 57 | Then you can inspect the filesystem and check the whitelisted/blacklisted files. 58 | 59 | You can also use `--join-filesystem` (there is also `--join-network`). This will give you a shell within the sandbox. It doesn't work in my case (probably because of other hardening things /o\\). If I try to join a `ping` process: 60 | 61 | ```bash 62 | kmille@linbox:~ firejail --join=74748 63 | Switching to pid 74750, the first child process inside the sandbox 64 | Changing root to /proc/74750/root 65 | Child process initialized in 2.27 ms 66 | /usr/bin/zsh: error while loading shared libraries: libncursesw.so.6: cannot open shared object file: No such file or directory 67 | kmille@linbox:~ sudo firejail --join-filesystem=74748 68 | Switching to pid 74750, the first child process inside the sandbox 69 | Changing root to /proc/74750/root 70 | Error: cannot open /proc/self/fd directory 71 | kmille@linbox:~ 72 | ``` 73 | 74 | ### tracelog filesystem violations 75 | 76 | You can call `firejail --tracelog application` or use `tracelog` in the profile. Filesystem access violations will then be logged: 77 | 78 | ```bash 79 | kmille@linbox:~ firejail --join=keepassxc 80 | ... 81 | linbox% pwd 82 | /run/user/1000 83 | linbox% cd app 84 | cd: permission denied: app 85 | ... 86 | kmille@linbox:log journalctl -n0 -f 87 | Nov 17 11:06:07 linbox firejail[126044]: blacklist violation - sandbox 126015, name keepassxc, exe 3, syscall stat, path . 88 | Nov 17 11:06:07 linbox firejail[126044]: blacklist violation - sandbox 126015, name keepassxc, exe 3, syscall chdir, path /run/user/1000/app 89 | Nov 17 11:06:07 linbox firejail[126044]: blacklist violation - sandbox 126015, name keepassxc, exe 3, syscall stat, path . 90 | ``` 91 | 92 | After the `keepasxc` update, the socket (to communicate with the browser) moved from `/run/user/1000/` to `/run/user/1000/app/` ([commit](https://github.com/keepassxreboot/keepassxc/commit/40316ac7b9bb5276c3d3ae1be2c3d808db503e3c)), but the firejail profile was not yet updated at that time. The directory was blacklisted in `disable-common.inc` (`blacklist ${RUNUSER}/app`). 93 | 94 | Some more details: in my case, the log entries you see in the `journalctl` output were generated by my manual debugging. Internally, `kepassxc` calls [`QDir().mkpath(subPath)` ](https://github.com/keepassxreboot/keepassxc/blob/12be175d583fbfac5a7b6b250a3bb5f792925285/src/browser/BrowserShared.cpp#L42) and the return value is not checked ([docs](https://doc.qt.io/qt-6/qdir.html#mkpath)). So `keepassxc` could have caught this error and print an error message to the user. Also, the call to `mkpath` (which results in a mkdir syscall) does not lead to a log entry by `--tracelog`. 95 | 96 | ### trace open, access and connect system calls 97 | 98 | ```bash 99 | kmille@linbox:tmp firejail --name=keepass --trace=fire.log keepassxc 100 | Reading profile /etc/firejail/keepassxc.profile 101 | ... 102 | kmille@linbox:tmp cat fire.log| grep BrowserServer -A 1 103 | 15:keepassxc:unlink /run/user/1000/org.keepassxc.KeePassXC.BrowserServer:0 104 | 15:keepassxc:mkdir /run/user/1000/app/org.keepassxc.KeePassXC/.VKFFOA:-1 105 | ``` 106 | 107 | If we use `--trace`, we can see that the to call `mkdir` fails (-1 is the return value of the syscall, check [`man 2 mkdir`](https://man7.org/linux/man-pages/man2/mkdir.2.html)). I still don't know where the `.VKFFOA` is coming from. 108 | 109 | ## Some other debug techniques 110 | 111 | - Run firejail with `--debug`. It shows you all it does (a lot) 112 | - Proven to be reliable, but not the smartest approach: use trial and error by commenting lines the firejail profile. 113 | 1. Run the application without firejail. If it also fails, firejail is not the problem 114 | 2. Make a backup of the profile and modify the real one (you can also just copy it to `~/.config/firejail`) 115 | 1. `sudo -s; cd /etc/firejail; cp keepassxc.profile keepassxc.profile.bak; vim keepassxc.profile ` 116 | 3. Try the first half: comment all lines with `include`, `blacklist` and `whitelist`. Problem solved? Use binary search to find out the exact line 117 | 4. Black/Whitelists are not responsible for the problem? Re-enable the original `blacklist`/`whitelist` configuration and comment the half of the lower part of the configuration (`dbus-user`, `private-*`, ...) 118 | 5. move on until you find the responsible line in the profile 119 | 6. fix it permanently by submitting an issue upstream or using `~/.config/firejail/application.local` 120 | 121 | # Managing files in a sandbox 122 | 123 | ### Put a file into a sandbox 124 | 125 | ```bash 126 | kmille@linbox:~ firejail --put=keepassxc /etc/passwd ~/passwd 127 | Switching to pid 130345, the first child process inside the sandbox 128 | ``` 129 | 130 | ```bash 131 | kmille@linbox:~ firejail --ls=keepassxc ~ 132 | drwx------ 1000 1000 260 . 133 | drwxr-xr-x root root 60 .. 134 | -rw------- 1000 1000 106 .Xauthority 135 | drwxr-xr-x 1000 1000 80 .cache 136 | drwxr-xr-x 1000 1000 260 .config 137 | -rw------- 1000 1000 42 .histfile 138 | -rw-r--r-- 1000 1000 26 .inputrc 139 | drwxr-xr-x 1000 1000 60 .local 140 | drwxr-xr-x 1000 1000 60 .mozilla 141 | drwxr-xr-x 1000 1000 60 cloud 142 | -rw------- 1000 1000 1790 passwd <---- this file is new 143 | ``` 144 | 145 | ```bash 146 | kmille@linbox:~ firejail --cat=keepassxc ~/passwd 147 | root:x:0:0::/root:/usr/bin/zsh 148 | ... 149 | ``` 150 | 151 | ### Get a file out of a sandbox 152 | 153 | ```bash 154 | kmille@linbox:~ firejail --get=keepassxc passwd 155 | Switching to pid 130345, the first child process inside the sandbox 156 | kmille@linbox:~ ls passwd 157 | -rw------- 1 kmille kmille 1.8K Nov 17 11:31 passwd 158 | ``` 159 | 160 | ### Using firejail with AppArmor 161 | 162 | ```bash 163 | kmille@linbox:~ sudo systemctl start apparmor 164 | kmille@linbox:~ sudo systemctl enable apparmor 165 | kmille@linbox:~ sudo apparmor_parser -r /etc/apparmor.d/firejail-default 166 | kmille@linbox:~ sudo aa-enforce firejail-default 167 | kmille@linbox:~ sudo aa-status 168 | apparmor module is loaded. 169 | 64 profiles are loaded. 170 | 64 profiles are in enforce mode. 171 | /usr/lib/apache2/mpm-prefork/apache2 172 | ... 173 | firejail-default 174 | 0 profiles are in complain mode. 175 | 0 profiles are in kill mode. 176 | 0 profiles are in unconfined mode. 177 | 7 processes have profiles defined. 178 | 7 processes are in enforce mode. 179 | /usr/lib/firefox/firefox (95771) firejail-default 180 | /usr/lib/firefox/firefox (95844) firejail-default 181 | /usr/lib/firefox/firefox (95867) firejail-default 182 | /usr/lib/firefox/firefox (95913) firejail-default 183 | /usr/lib/firefox/firefox (95959) firejail-default 184 | /usr/lib/firefox/firefox (95962) firejail-default 185 | /usr/lib/firefox/firefox (95967) firejail-default 186 | 0 processes are in complain mode. 187 | 0 processes are unconfined but have a profile defined. 188 | 0 processes are in mixed mode. 189 | 0 processes are in kill mode. 190 | ``` 191 | 192 | ### Run a second instance of signal-desktop 193 | 194 | Run `signal-desktop` with a dedicated, persistent home directory. You have to create `/home/kmille/.config/Signal2` manually before. `signal-desktop` also supports `--user-data-dir` to use it with a different home directory/account. 195 | 196 | ```ini 197 | kmille@linbox:~ cat ~/.local/share/applications/signal-desktop2.desktop 198 | [Desktop Entry] 199 | Type=Application 200 | Name=Signal2 201 | Comment=Signal - Private Messenger 202 | Comment[de]=Signal - Sicherer Messenger 203 | Icon=signal-desktop 204 | #Exec=signal-desktop -- %u 205 | Exec=firejail --private=/home/kmille/.config/Signal2 --profile=/etc/firejail/signal-desktop.profile /usr/bin/signal-desktop 206 | Terminal=false 207 | Categories=Network;InstantMessaging; 208 | StartupWMClass=Signal 209 | MimeType=x-scheme-handler/sgnl;x-scheme-handler/signalcaptcha; 210 | Keywords=sgnl;chat;im;messaging;messenger;sms;security;privat; 211 | X-GNOME-UsesNotifications=true 212 | ``` 213 | 214 | ```bash 215 | kmille@linbox:~ alias signal2 216 | signal2='firejail --private=/home/kmille/.config/Signal2 --profile=/etc/firejail/signal-desktop.profile /usr/bin/signal-desktop' 217 | ``` 218 | 219 | ## jailjack 220 | 221 | `sudo jailcheck` does a basic audit for all running sandboxes: 222 | 223 | ```bash 224 | kmille@linbox:firejail sudo jailcheck 225 | .... 226 | 5459:kmille::/usr/bin/firejail / 227 | 132309:kmille:keepassxc:/usr/bin/firejail /usr/bin/keepassxc 228 | Warning: AppArmor not enabled 229 | Virtual dirs: /home/kmille, /tmp, /var/tmp, /dev, /etc, /bin, /usr/share 230 | Networking: disabled 231 | ``` 232 | 233 | This means that `/home/kmille`, `/etc` and the other virtual directories are fresh. Only dedicated stuff is bound/copied to these directories. Things like `lib` looks like on the host system. More infos in the [man page](https://firejail.wordpress.com/man-jailcheck/). 234 | 235 | ### whitelist firejail users 236 | 237 | You can reduce the risk of the setuid binary by whitelisting the users, firejail will work with. You can use [`firecfg --add-users`](https://man7.org/linux/man-pages/man1/firecfg.1.html). You can also use `force-nonewprivs yes` under special [circumstances](https://firejail.wordpress.com/documentation-2/basic-usage/#suid). 238 | 239 | ```bash 240 | kmille@linbox:~ cat /etc/firejail/firejail.users 241 | kmille 242 | kmille@linbox:~ sudo -u nobody bash 243 | [sudo] password for kmille: 244 | [nobody@linbox ~]$ gnome-calculator 245 | Error: the user is not allowed to use Firejail. 246 | Please add the user in /etc/firejail/firejail.users file, 247 | either by running "sudo firecfg", or by editing the file directly. 248 | See "man firejail-users" for more details. 249 | Authorization required, but no authorization protocol specified 250 | (gnome-calculator:99058): Gtk-WARNING **: 17:49:20.595: cannot open display: :0 251 | [nobody@linbox ~]$ 252 | ``` 253 | 254 | # How to create own profiles 255 | 256 | - Use the template profile in [`/usr/share/doc/firejail/profile.template`](https://github.com/netblue30/firejail/blob/master/etc/templates/profile.template) and read the comments 257 | - https://github.com/netblue30/firejail/wiki/Creating-Profiles 258 | - https://firejail.wordpress.com/documentation-2/building-custom-profiles/ 259 | - https://github.com/netblue30/firejail/wiki/Creating-overrides 260 | - You can use `--build` 261 | 262 | ```bash 263 | kmille@linbox: firejail --build mpv ~/Downloads/test.mp3 264 | (+) Audio --aid=1 (pcm_s24le 2ch 44100Hz) 265 | AO: [pulse] 44100Hz stereo 2ch s32 266 | A: 00:00:00 / 00:00:01 (69%) 267 | 268 | Exiting... (End of file) 269 | --- Built profile begins after this line --- 270 | # Save this file as "application.profile" (change "application" with the 271 | # program name) in ~/.config/firejail directory. Firejail will find it 272 | # automatically every time you sandbox your application. 273 | # 274 | # Run "firejail application" to test it. In the file there are 275 | # some other commands you can try. Enable them by removing the "#". 276 | 277 | # Firejail profile for mpv 278 | # Persistent local customizations 279 | #include mpv.local 280 | # Persistent global definitions 281 | #include globals.local 282 | 283 | ### Basic Blacklisting ### 284 | ### Enable as many of them as you can! A very important one is 285 | ### "disable-exec.inc". This will make among other things your home 286 | ### and /tmp directories non-executable. 287 | include disable-common.inc # dangerous directories like ~/.ssh and ~/.gnupg 288 | #include disable-devel.inc # development tools such as gcc and gdb 289 | #include disable-exec.inc # non-executable directories such as /var, /tmp, and /home 290 | #include disable-interpreters.inc # perl, python, lua etc. 291 | include disable-programs.inc # user configuration for programs such as firefox, vlc etc. 292 | #include disable-shell.inc # sh, bash, zsh etc. 293 | #include disable-xdg.inc # standard user directories: Documents, Pictures, Videos, Music 294 | 295 | ### Home Directory Whitelisting ### 296 | ### If something goes wrong, this section is the first one to comment out. 297 | ### Instead, you'll have to relay on the basic blacklisting above. 298 | whitelist ${HOME}/.config/pipewire 299 | whitelist ${HOME}/Downloads 300 | whitelist ${HOME}/Downloads/test.mp3 301 | include whitelist-common.inc 302 | 303 | ### Filesystem Whitelisting ### 304 | include whitelist-run-common.inc 305 | whitelist ${RUNUSER}/pulse 306 | include whitelist-runuser-common.inc 307 | include whitelist-usr-share-common.inc 308 | include whitelist-var-common.inc 309 | 310 | #apparmor # if you have AppArmor running, try this one! 311 | caps.drop all 312 | ipc-namespace 313 | netfilter 314 | #no3d # disable 3D acceleration 315 | #nodvd # disable DVD and CD devices 316 | #nogroups # disable supplementary user groups 317 | #noinput # disable input devices 318 | nonewprivs 319 | noroot 320 | #notv # disable DVB TV devices 321 | #nou2f # disable U2F devices 322 | #novideo # disable video capture devices 323 | protocol unix, 324 | net none 325 | seccomp !chroot # allowing chroot, just in case this is an Electron app 326 | shell none 327 | #tracelog # send blacklist violations to syslog 328 | 329 | #disable-mnt # no access to /mnt, /media, /run/mount and /run/media 330 | private-bin mpv, 331 | #private-cache # run with an empty ~/.cache directory 332 | private-dev 333 | private-etc machine-id,pulse,pipewire,fonts,mpv,login.defs, 334 | #private-lib 335 | private-tmp 336 | 337 | #dbus-user none 338 | #dbus-system none 339 | 340 | #memory-deny-write-execute 341 | kmille@linbox:firejail 342 | ``` 343 | 344 | # Resources 345 | 346 | ### About the benefits/risks of using firejail 347 | 348 | - https://github.com/netblue30/firejail/issues/3046 349 | - https://madaidans-insecurities.github.io/linux.html#firejail 350 | 351 | ### General 352 | 353 | - https://wiki.archlinux.org/title/firejail 354 | - https://github.com/netblue30/firejail/wiki/Debugging-Firejail 355 | - https://firejail.wordpress.com/features-3/man-firejail/ 356 | 357 | 358 | 359 | --------------------------------------------------------------------------------