├── CONTRIBUTING.md ├── LICENSE.txt ├── Makefile ├── README.md ├── SECURITY.md ├── buildrpm ├── 1.1 │ └── fdr.spec ├── 1.2 │ └── fdr.spec └── 1.3 │ └── fdr.spec ├── fdr ├── fdr.service ├── fdrd.c ├── fdrd.man └── samples ├── nfs └── nfs.logrotate /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing to this repository 2 | 3 | We welcome your contributions! There are multiple ways to contribute. 4 | 5 | ## Opening issues 6 | 7 | For bugs or enhancement requests, please file a GitHub issue unless it's 8 | security related. When filing a bug remember that the better written the bug is, 9 | the more likely it is to be fixed. If you think you've found a security 10 | vulnerability, do not raise a GitHub issue and follow the instructions in our 11 | [security policy](./SECURITY.md). 12 | 13 | ## Contributing code 14 | 15 | We welcome your code contributions. Before submitting code via a pull request, 16 | you will need to have signed the [Oracle Contributor Agreement][OCA] (OCA) and 17 | your commits need to include the following line using the name and e-mail 18 | address you used to sign the OCA: 19 | 20 | ```text 21 | Signed-off-by: Your Name 22 | ``` 23 | 24 | This can be automatically added to pull requests by committing with `--sign-off` 25 | or `-s`, e.g. 26 | 27 | ```text 28 | git commit --signoff 29 | ``` 30 | 31 | Only pull requests from committers that can be verified as having signed the OCA 32 | can be accepted. 33 | 34 | ## Pull request process 35 | 36 | 1. Ensure there is an issue created to track and discuss the fix or enhancement 37 | you intend to submit. 38 | 1. Fork this repository. 39 | 1. Create a branch in your fork to implement the changes. We recommend using 40 | the issue number as part of your branch name, e.g. `1234-fixes`. 41 | 1. Ensure that any documentation is updated with the changes that are required 42 | by your change. 43 | 1. Ensure that any samples are updated if the base image has been changed. 44 | 1. Submit the pull request. *Do not leave the pull request blank*. Explain exactly 45 | what your changes are meant to do and provide simple steps on how to validate. 46 | your changes. Ensure that you reference the issue you created as well. 47 | 1. We will assign the pull request to 2-3 people for review before it is merged. 48 | 49 | ## Code of conduct 50 | 51 | Follow the [Golden Rule](https://en.wikipedia.org/wiki/Golden_Rule). If you'd 52 | like more specific guidelines, see the [Contributor Covenant Code of Conduct][COC]. 53 | 54 | [OCA]: https://oca.opensource.oracle.com 55 | [COC]: https://www.contributor-covenant.org/version/1/4/code-of-conduct/ 56 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2019,2023 Oracle and/or its affiliates. All rights reserved. 2 | 3 | The Universal Permissive License (UPL), Version 1.0 4 | 5 | Subject to the condition set forth below, permission is hereby granted to any 6 | person obtaining a copy of this software, associated documentation and/or data 7 | (collectively the "Software"), free of charge and under any and all copyright 8 | rights in the Software, and any and all patent rights owned or freely 9 | licensable by each licensor hereunder covering either (i) the unmodified 10 | Software as contributed to or provided by such licensor, or (ii) the Larger 11 | Works (as defined below), to deal in both 12 | 13 | (a) the Software, and 14 | (b) any piece of software and/or hardware listed in the lrgrwrks.txt file if 15 | one is included with the Software (each a "Larger Work" to which the Software 16 | is contributed by such licensors), 17 | 18 | without restriction, including without limitation the rights to copy, create 19 | derivative works of, display, perform, and distribute the Software and make, 20 | use, sell, offer for sale, import, export, have made, and have sold the 21 | Software and the Larger Work(s), and to sublicense the foregoing rights on 22 | either these or other terms. 23 | 24 | This license is subject to the following condition: 25 | The above copyright notice and either this complete permission notice or at 26 | a minimum a reference to the UPL must be included in all copies or 27 | substantial portions of the Software. 28 | 29 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 30 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 31 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 32 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 33 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 34 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 35 | SOFTWARE. 36 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Licensed under the Universal Permissive License v 1.0 as shown at 2 | # https://oss.oracle.com/licenses/upl. 3 | # 4 | CFLAGS := -g 5 | 6 | PREFIX := /usr 7 | SBINDIR := $(PREFIX)/sbin 8 | DATADIR := $(PREFIX)/share 9 | MANDIR8 := $(DATADIR)/man/man8 10 | INSTALL := install 11 | 12 | RPMBUILD_DIR ?= $(HOME) 13 | LATEST_VERS ?= 1.3 14 | 15 | all: fdrd 16 | 17 | fdrd: fdrd.c 18 | $(CC) $(CPPFLAGS) $(CFLAGS) -o $@ $< 19 | 20 | clean: 21 | rm -f fdrd 22 | 23 | install: fdrd 24 | mkdir -p $(DESTDIR)$(SBINDIR) 25 | $(INSTALL) -m 0755 fdrd $(DESTDIR)$(SBINDIR) 26 | mkdir -p $(DESTDIR)$(MANDIR8) 27 | $(INSTALL) -m 0644 fdrd.man $(DESTDIR)$(MANDIR8) 28 | mkdir -p $(DESTDIR)$(DATADIR)/fdr/samples 29 | $(INSTALL) -m 0644 README.md $(DESTDIR)$(DATADIR)/fdr/README 30 | $(INSTALL) -m 0644 samples/nfs $(DESTDIR)$(DATADIR)/fdr/samples 31 | $(INSTALL) -m 0644 samples/nfs.logrotate $(DESTDIR)$(DATADIR)/fdr/samples 32 | 33 | uninstall: 34 | rm -rf $(SBINDIR)/fdrd 35 | 36 | tarball: clean 37 | tar --transform "s/^./fdr-$(LATEST_VERS)/" \ 38 | --xz -cf $(RPMBUILD_DIR)/SOURCES/fdr-$(LATEST_VERS).tar.xz . 39 | 40 | release: 41 | git tag -f fdr-$(LATEST_VERS) 42 | git archive --format=tar --prefix=fdr-$(LATEST_VERS)/ fdr-$(LATEST_VERS) \ 43 | | ( cd /tmp ; tar xf - ) 44 | (cd /tmp ; tar cJf $(RPMBUILD_DIR)/SOURCES/fdr-$(LATEST_VERS).tar.xz \ 45 | fdr-$(LATEST_VERS)) 46 | 47 | rpm: tarball 48 | cp buildrpm/$(LATEST_VERS)/fdr.spec \ 49 | $(RPMBUILD_DIR)/rpmbuild/SPECS/fdr.spec 50 | rpmbuild -bb $(RPMBUILD_DIR)/rpmbuild/SPECS/fdr.spec 51 | 52 | srpm: tarball 53 | cp buildrpm/$(LATEST_VERS)/fdr.spec \ 54 | $(RPMBUILD_DIR)/rpmbuild/SPECS/fdr.spec 55 | rpmbuild -bs $(RPMBUILD_DIR)/rpmbuild/SPECS/fdr.spec 56 | 57 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Flight Data Recorder 2 | 3 | ## Description 4 | 5 | The flight data recorder (fdr) is a daemon which enables ftrace probes, 6 | harvests ftrace data and (optionally) writes the data to a file. 7 | 8 | The behavior of fdr is defined by configuration files stored in 9 | ```/etc/fdr.d```. During service startup, fdr will process each file in 10 | the directory which has the suffix of .conf. If new config files 11 | are added to the fdr.d directory, then the service must be restarted 12 | to recognize the new configuration information. 13 | 14 | fdr is controlled by ```systemd(8)``` on systems where systemd is 15 | available. Error messages from fdr can be viewed via systemctl, 16 | for example, ```systemctl status -l fdr```. 17 | 18 | ## Configuration File Syntax 19 | 20 | The following keywords and options are recognized 21 | 22 | ### instance iname [buffer-size] 23 | 24 | Create a new ftrace instance called "iname". This instance 25 | will appear in ```/sys/kernel/debug/tracing/instances``` 26 | 27 | The optional buffer-size parameter can be used to control 28 | the size of the ftrace buffers for this instance in the 29 | kernel. A suffix of 'k', 'K', 'm', 'M', 'g' or 'G' may be 30 | used to specify kilobytes, megabytes or gigabytes. 31 | 32 | ### modprobe module-name 33 | 34 | Force the named module to be loaded by fdr. This can be 35 | useful when the module is normally loaded on demand and 36 | the probes cannot be enabled until the module is loaded. 37 | 38 | ### enable subsystem-name/probe-name [filter] 39 | 40 | Enable an ftrace probe in the specified subsystem. Both 41 | the subsystem name and probe name are defined by the kernel. 42 | 43 | The optional filter parameter allows an ftrace filter to 44 | be set as well. This will limit the amount of data being 45 | emitted. The syntax of the filter language is 46 | defined by ftrace itself and the parameters are defined 47 | by the static tracepoint being enabled in the kernel. 48 | 49 | ### enable subsystem-name/all 50 | 51 | Enable all ftrace probes for the subsystem. 52 | 53 | ### disable subsystem-name/probe-name 54 | 55 | Disable an ftrace probe in the specified subsystem. This 56 | can be useful to disable selective probes when the "ALL" 57 | keyword has been used. 58 | 59 | ### disable subsystem-name/all 60 | 61 | Disable all probes in the specified subsystem. 62 | 63 | ### saveto file-name [maxsize] 64 | 65 | Save the output of enabled probes to the named file. If 66 | the optional maxsize parameter is given, the daemon will 67 | initiate a log rotation, see [Log Rotation](README.md#log-rotation) below. 68 | A suffix 69 | of 'k', 'K', 'm', 'M', 'g' or 'G' may be used to specify 70 | kilobytes, megabytes or gigabytes. 71 | 72 | If no saveto directive is present, then fdr will create the 73 | instance and enable the probes. In this case, the data 74 | can be harvested manually by reading: 75 | 76 | ``` 77 | /sys/kernel/debug/tracing/instances/iname/trace_pipe 78 | ``` 79 | 80 | The ftrace buffers in the kernel are circular. If no 81 | process harvests the data, new data will overwrite old data. 82 | 83 | ### minfree value 84 | 85 | Limit the output by the daemon based on free space in the 86 | file system for the save file. If free space percentage is 87 | below the specified value, no output will be written. 88 | 89 | If no minfree directive is present, fdr will use 5% by 90 | default. 91 | 92 | ## Log Rotation 93 | 94 | fdr can use ```logrotate(8)``` to manage the output files. By convention, 95 | ``` /etc/logrotate.d/instance-name ``` controls the behavior of logrotate. 96 | 97 | fdr will also invoke logrotate directly at startup and when reaching 98 | the maxsize limit for the save file. 99 | 100 | ## See Also 101 | 102 | [trace-cmd](https://lwn.net/Articles/410200/) 103 | 104 | [ftrace documentation](https://www.kernel.org/doc/Documentation/trace/ftrace.txt) 105 | 106 | ## Building & Installing 107 | 108 | A Makefile is provided with this repository to facilitate building 109 | and installing fdr. Simply type `make` to build fdr and type 110 | `make install` to install it 111 | 112 | The Makefile depends on the C compiler (provided by the gcc rpm), 113 | the `install` tool (provided by the coreutils rpm) as well as 114 | make inself (provided by make rpm). 115 | 116 | The source code itself depends on standard header files such 117 | as `````` (provided by glibc-headers). 118 | 119 | ## Contributing 120 | 121 | This project welcomes contributions from the community. Before submitting a pull request, please [review our contribution guide](./CONTRIBUTING.md) 122 | 123 | ## Security 124 | 125 | Please consult the [security guide](./SECURITY.md) for our responsible security vulnerability disclosure process 126 | 127 | ## License 128 | 129 | This repository is licensed under the "Universal Permissive 130 | License" (UPL). See [LICENSE](/LICENSE) in this repository for 131 | more information. 132 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | # Reporting security vulnerabilities 2 | 3 | Oracle values the independent security research community and believes that 4 | responsible disclosure of security vulnerabilities helps us ensure the security 5 | and privacy of all our users. 6 | 7 | Please do NOT raise a GitHub Issue to report a security vulnerability. If you 8 | believe you have found a security vulnerability, please submit a report to 9 | [secalert_us@oracle.com][1] preferably with a proof of concept. Please review 10 | some additional information on [how to report security vulnerabilities to Oracle][2]. 11 | We encourage people who contact Oracle Security to use email encryption using 12 | [our encryption key][3]. 13 | 14 | We ask that you do not use other channels or contact the project maintainers 15 | directly. 16 | 17 | Non-vulnerability related security issues including ideas for new or improved 18 | security features are welcome on GitHub Issues. 19 | 20 | ## Security updates, alerts and bulletins 21 | 22 | Security updates will be released on a regular cadence. Many of our projects 23 | will typically release security fixes in conjunction with the 24 | Oracle Critical Patch Update program. Additional 25 | information, including past advisories, is available on our [security alerts][4] 26 | page. 27 | 28 | ## Security-related information 29 | 30 | We will provide security related information such as a threat model, considerations 31 | for secure use, or any known security issues in our documentation. Please note 32 | that labs and sample code are intended to demonstrate a concept and may not be 33 | sufficiently hardened for production use. 34 | 35 | [1]: mailto:secalert_us@oracle.com 36 | [2]: https://www.oracle.com/corporate/security-practices/assurance/vulnerability/reporting.html 37 | [3]: https://www.oracle.com/security-alerts/encryptionkey.html 38 | [4]: https://www.oracle.com/security-alerts/ 39 | -------------------------------------------------------------------------------- /buildrpm/1.1/fdr.spec: -------------------------------------------------------------------------------- 1 | Name: fdr 2 | URL: https://github.com/oracle/fdr.git 3 | Version: 1 4 | Release: 1 5 | Summary: Flight Data Recorder 6 | License: UPL 7 | Source0: %{name}-%{version}.tar.gz 8 | 9 | %description 10 | The flight data recorder, a daemon which enables ftrace probes and harvests the data 11 | 12 | %prep 13 | %setup 14 | %build 15 | make 16 | 17 | %install 18 | mkdir -p %{buildroot}/usr/sbin 19 | install -m 755 fdrd %{buildroot}/usr/sbin 20 | 21 | mkdir -p %{buildroot}/etc/fdr.d/samples 22 | install -m 644 README.md %{buildroot}/etc/fdr.d/README 23 | install -m 644 samples/nfs %{buildroot}/etc/fdr.d/samples 24 | 25 | mkdir -p %{buildroot}/usr/lib/systemd/system 26 | install -m 644 fdr.service %{buildroot}/usr/lib/systemd/system 27 | 28 | %files 29 | /usr/sbin/fdrd 30 | /usr/lib/systemd/system/fdr.service 31 | /etc/fdr.d/README 32 | /etc/fdr.d/samples/nfs 33 | 34 | %preun 35 | systemctl stop fdr 36 | systemctl disable fdr 37 | 38 | %changelog 39 | # 1.1 - initial release to github 40 | 41 | -------------------------------------------------------------------------------- /buildrpm/1.2/fdr.spec: -------------------------------------------------------------------------------- 1 | Name: fdr 2 | URL: https://github.com/oracle/fdr.git 3 | Version: 1.2 4 | Release: 0 5 | Summary: Flight Data Recorder 6 | License: UPL 7 | Source0: %{name}-%{version}.tar.xz 8 | 9 | %description 10 | The flight data recorder, a daemon which enables ftrace probes and harvests the data 11 | 12 | %prep 13 | %setup 14 | %build 15 | make 16 | 17 | %install 18 | mkdir -p %{buildroot}/usr/sbin 19 | install -m 755 fdrd %{buildroot}/usr/sbin 20 | 21 | mkdir -p %{buildroot}/etc/fdr.d/samples 22 | install -m 644 README.md %{buildroot}/etc/fdr.d/README 23 | install -m 644 samples/nfs %{buildroot}/etc/fdr.d/samples 24 | install -m 644 samples/nfs.logrotate %{buildroot}/etc/fdr.d/samples 25 | 26 | mkdir -p %{buildroot}/usr/lib/systemd/system 27 | install -m 644 fdr.service %{buildroot}/usr/lib/systemd/system 28 | 29 | %files 30 | /usr/sbin/fdrd 31 | /usr/lib/systemd/system/fdr.service 32 | /etc/fdr.d/README 33 | /etc/fdr.d/samples/nfs 34 | /etc/fdr.d/samples/nfs.logrotate 35 | 36 | %preun 37 | systemctl stop fdr 38 | systemctl disable fdr 39 | 40 | %changelog 41 | # 1.1 - initial release to github 42 | # 1.2 - Add logrotate sample 43 | 44 | -------------------------------------------------------------------------------- /buildrpm/1.3/fdr.spec: -------------------------------------------------------------------------------- 1 | Summary: A daemon which enables ftrace probes and harvests the data 2 | Name: fdr 3 | URL: https://github.com/oracle/fdr.git 4 | Version: 1.3 5 | Release: 2%{?dist} 6 | License: UPL 7 | Source0: http://people.redhat.com/steved/fdr/%{name}-%{version}.tar.xz 8 | 9 | BuildRequires: gcc 10 | BuildRequires: make 11 | BuildRequires: sed 12 | BuildRequires: systemd-rpm-macros 13 | Requires: systemd 14 | 15 | %description 16 | The flight data recorder, a daemon which enables ftrace probes 17 | and harvests the data 18 | 19 | %prep 20 | %autosetup 21 | 22 | %build 23 | sed -i -e "s:^CFLAGS.*:CFLAGS = %{optflags}:" Makefile 24 | %make_build 25 | 26 | 27 | %install 28 | mkdir -p %{buildroot}/%{_sbindir} 29 | install -m 755 fdrd %{buildroot}/%{_sbindir} 30 | 31 | mkdir -p %{buildroot}%{_datadir}/fdr/samples 32 | install -m 644 samples/nfs %{buildroot}/%{_datadir}/fdr/samples 33 | install -m 644 samples/nfs.logrotate %{buildroot}/%{_datadir}/fdr/samples 34 | 35 | mkdir -p %{buildroot}/%{_unitdir} 36 | install -m 644 %{name}.service %{buildroot}/%{_unitdir}/%{name}.service 37 | 38 | mkdir -p %{buildroot}/%{_mandir}/man8 39 | install -m 644 fdrd.man %{buildroot}/%{_mandir}/man8/fdrd.8 40 | 41 | %post 42 | %systemd_post %{name}.service 43 | 44 | %preun 45 | %systemd_preun %{name}.service 46 | 47 | %postun 48 | %systemd_postun_with_restart %{name}.service 49 | 50 | %files 51 | %{_sbindir}/fdrd 52 | %{_unitdir}/fdr.service 53 | %{_datadir}/fdr/samples/nfs 54 | %{_datadir}/fdr/samples/nfs.logrotate 55 | %{_mandir}/man8/* 56 | %doc README.md 57 | %license LICENSE 58 | 59 | %changelog 60 | * Fri Jun 25 2021 Bill Baker 1.3-2 61 | - Initial commit 62 | -------------------------------------------------------------------------------- /fdr: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # 3 | # Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved. 4 | # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. 5 | # 6 | # Licensed under the Universal Permissive License v 1.0 as shown at 7 | # https://oss.oracle.com/licenses/upl. 8 | 9 | # Source function library. 10 | . /etc/rc.d/init.d/functions 11 | 12 | uid=`id | cut -d\( -f1 | cut -d= -f2` 13 | 14 | case "$1" in 15 | start) 16 | [ -x /usr/sbin/fdrd ] || exit 5 17 | 18 | # Only root can start the service 19 | [ $uid -ne 0 ] && exit 4 20 | 21 | # check to see if tracefs is mounted and where 22 | tpath=`grep tracefs /etc/mtab | awk '{ print $2 }'` 23 | [ -z "$tpath" ] && mount -t tracefs nodev /sys/kernel/tracing 24 | tpath=`grep tracefs /etc/mtab | awk '{ print $2 }'` 25 | [ -z "$tpath" ] && exit 5 26 | 27 | action $"Starting fdr services " 28 | daemon fdrd -d $tpath 2> /var/log/fdr.log 29 | RETVAL=$? 30 | echo 31 | ;; 32 | stop) 33 | # Only root can stop the service 34 | [ $uid -ne 0 ] && exit 4 35 | 36 | # Stop daemons. 37 | echo -n $"Shutting down fdr daemon: " 38 | killproc fdrd -15 39 | RETVAL=$? 40 | echo 41 | ;; 42 | status) 43 | status fdrd 44 | RETVAL=$? 45 | ;; 46 | restart ) 47 | $0 stop 48 | $0 start 49 | RETVAL=$? 50 | echo 51 | ;; 52 | *) 53 | echo $"Usage: fdr {start|stop|status|restart}" 54 | RETVAL=2 55 | ;; 56 | esac 57 | 58 | exit $RETVAL 59 | -------------------------------------------------------------------------------- /fdr.service: -------------------------------------------------------------------------------- 1 | 2 | # Licensed under the Universal Permissive License v 1.0 as shown at 3 | # https://oss.oracle.com/licenses/upl. 4 | 5 | [Unit] 6 | Description=Flight Data Recorder 7 | 8 | [Service] 9 | Type=forking 10 | RemainAfterExit=yes 11 | ExecStart=/usr/sbin/fdrd 12 | 13 | [Install] 14 | WantedBy=multi-user.target 15 | 16 | -------------------------------------------------------------------------------- /fdrd.c: -------------------------------------------------------------------------------- 1 | /* 2 | * fdrd.c - Flight Data Recorder Daemon 3 | * 4 | * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved. 5 | * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. 6 | * 7 | * The Universal Permissive License (UPL), Version 1.0 8 | * 9 | * Subject to the condition set forth below, permission is hereby granted to any 10 | * person obtaining a copy of this software, associated documentation and/or 11 | * data (collectively the "Software"), free of charge and under any and all 12 | * copyright rights in the Software, and any and all patent rights owned or 13 | * freely licensable by each licensor hereunder covering either (i) the 14 | * unmodified Software as contributed to or provided by such licensor, or (ii) 15 | * the Larger Works (as defined below), to deal in both 16 | * 17 | * (a) the Software, and 18 | * 19 | * (b) any piece of software and/or hardware listed in the lrgrwrks.txt file if 20 | * one is included with the Software each a "Larger Work" to which the Software 21 | * is contributed by such licensors), 22 | * 23 | * without restriction, including without limitation the rights to copy, create 24 | * derivative works of, display, perform, and distribute the Software and make, 25 | * use, sell, offer for sale, import, export, have made, and have sold the 26 | * Software and the Larger Work(s), and to sublicense the foregoing rights on 27 | * either these or other terms. 28 | * 29 | * This license is subject to the following condition: 30 | * 31 | * The above copyright notice and either this complete permission notice or at a 32 | * minimum a reference to the UPL must be included in all copies or substantial 33 | * portions of the Software. 34 | * 35 | * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 36 | * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 37 | * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 38 | * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 39 | * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 40 | * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 41 | * SOFTWARE. 42 | */ 43 | 44 | #include 45 | #include 46 | #include 47 | #include 48 | #include 49 | #include 50 | #include 51 | #include 52 | #include 53 | #include 54 | #include 55 | #include 56 | #include 57 | 58 | /* 59 | * Notes about process management in this daemon: 60 | * 61 | * Each ftrace instance (as specified by the various config files) will 62 | * have a process assigned to it, it addition to the main, parent process 63 | * started by systemd itself. If the instance has a saveto directive, 64 | * then that process will hang around reading the trace_pipe and writing 65 | * the data to the file. If there is no saveto directive for that instance 66 | * the process will exit. 67 | * 68 | * The parent process will handle SIGTERM from systemd and attempt to 69 | * clean up the ftrace buffer. 70 | */ 71 | 72 | /* 73 | * Syntax of the config file(s) 74 | * 75 | * instance ftrace-instance-name [bufsize] 76 | * modprobe module-name 77 | * enable subsystem/probe-name 78 | * enable subsystem/all 79 | * disable subsystem/probe-name 80 | * disable subsystem/all 81 | * saveto filename [maxsize] 82 | * minfree pct 83 | * 84 | * FUTURE 85 | * 86 | * globbing of probe names 87 | * better handling of logrotate failures 88 | * eliminate modprobe directive, use modprobe.d 89 | * apply ordering rules to the list 90 | * how do I prevent data loss if the system crashes after I've harvested 91 | * data from the kernel, but before it is written to the saveto file? 92 | * There ought to be a way to add new config files (and instances) without 93 | * completely restarting the service. 94 | */ 95 | #define BUFSIZE 256 96 | #define FDR_CONFIG_DIR "/etc/fdr.d" 97 | 98 | #define INST_DIR "/sys/kernel/debug/tracing/instances" 99 | 100 | #define INSTANCE "instance" 101 | #define INSTANCE_T 0 102 | #define MODPROBE "modprobe" 103 | #define MODPROBE_T 1 104 | #define ENABLE "enable" 105 | #define ENABLE_T 2 106 | #define DISABLE "disable" 107 | #define DISABLE_T 3 108 | #define SAVETO "saveto" 109 | #define SAVETO_T 4 110 | #define LOGROT "logrotate" 111 | #define LOGROT_T 5 112 | #define MINFREE "minfree" 113 | #define MINFREE_T 6 114 | 115 | #define MINFREE_DEFAULT 5 116 | #define MAXSIZE_DEFAULT INT_MAX 117 | 118 | /* exit codes */ 119 | #define EC_MKDIR 1 120 | #define EC_SYSTEM 2 121 | #define EC_SYNTAX 3 122 | #define EC_BADTYPE1 4 123 | #define EC_OPEN 5 124 | #define EC_WRITE1 6 125 | #define EC_OPENLOG 7 126 | #define EC_OPENTRACE 8 127 | #define EC_FSTAT 9 128 | #define EC_MALLOC 10 129 | #define EC_BADVERB 11 130 | #define EC_FORK 12 131 | #define EC_BADTYPE2 13 132 | #define EC_BADARGS 14 133 | #define EC_EXEC 15 134 | 135 | /* 136 | * anchor -> instance -> instance -> anchor 137 | * | | 138 | * V V 139 | * item item 140 | * item item 141 | * | | 142 | * V V 143 | */ 144 | 145 | struct item { 146 | struct item *forw; 147 | int typ; 148 | char verb[BUFSIZE]; 149 | char target[BUFSIZE]; 150 | char fpath[BUFSIZE]; 151 | char optarg[BUFSIZE]; 152 | int line; 153 | }; 154 | 155 | struct instance { 156 | struct instance *forw; 157 | struct instance *back; 158 | struct item *ifirst; 159 | struct item *ilast; 160 | char iname[BUFSIZE]; 161 | char dname[BUFSIZE]; /* name of the ftrace directory */ 162 | int fud; 163 | unsigned long bufsize; /* ftrace buffer size, per cpu */ 164 | long maxsize; /* max file size of saveto target */ 165 | int minfree; /* minimum free space percentage */ 166 | }; 167 | 168 | struct { 169 | struct instance *forw; 170 | struct instance *back; 171 | int numi; 172 | } anchor = { (struct instance *) &anchor, (struct instance *) &anchor, 0 }; 173 | 174 | int verbose; 175 | int got_sighup; 176 | char inst_dir[BUFSIZE]; 177 | 178 | void 179 | sighandler(int signo, siginfo_t *sp, void *vp) 180 | { 181 | struct instance *insp; 182 | 183 | if (verbose > 1) 184 | fprintf(stderr, "signal %d received, cleaning up\n", signo); 185 | 186 | for (insp = anchor.forw; (void *)insp != (void *)&anchor; 187 | insp = insp->forw) { 188 | 189 | if (verbose > 1) 190 | fprintf(stderr, "removing %s\n", insp->dname); 191 | 192 | if (rmdir(insp->dname)) 193 | fprintf(stderr, "rmdir %s failed, errno %d\n", 194 | insp->dname, errno); 195 | } 196 | exit(0); 197 | } 198 | 199 | void 200 | sighup(int signo, siginfo_t *sp, void *vp) 201 | { 202 | if (verbose > 1) 203 | fprintf(stderr, "SIGHUP received\n"); 204 | /* 205 | * This will cause saveto to react 206 | */ 207 | got_sighup = 1; 208 | } 209 | 210 | unsigned long 211 | getvalue(char *arg) 212 | { 213 | unsigned long x; 214 | char *ep; 215 | 216 | x = strtoul(arg, &ep, 0); 217 | if (*ep == 'k' || *ep == 'K') 218 | x *= 1024; 219 | else if (*ep == 'm' || *ep == 'M') 220 | x *= 1024*1024; 221 | else if (*ep == 'g' || *ep == 'G') 222 | x *= 1024*1024*1024; 223 | 224 | return x; 225 | } 226 | 227 | void 228 | instance(struct instance *insp, struct item *itp) 229 | { 230 | char bs[BUFSIZE], value[BUFSIZE]; 231 | int fd, sl; 232 | 233 | fprintf(stderr, "creating: %s\n", insp->dname); 234 | (void) rmdir(insp->dname); 235 | errno = 0; 236 | /* 237 | * If mkdir fails w/ EEXIST, then just use that, instead 238 | * of just having outright failure. It's probably a leftover 239 | * from some previous instantation of the daemon that didn't 240 | * get cleaned up properly. 241 | */ 242 | if (mkdir(insp->dname, 0700) && errno != EEXIST) { 243 | perror(insp->dname); 244 | exit(1); 245 | } 246 | if (insp->bufsize) { 247 | fprintf(stderr, "%s: bufsize %lu\n", itp->target, 248 | insp->bufsize); 249 | snprintf(bs, sizeof(bs), "%s/buffer_size_kb", insp->dname); 250 | 251 | fd = open(bs, O_WRONLY); 252 | if (fd < 0) 253 | perror(bs); 254 | else { 255 | snprintf(value, sizeof(value), "%lu", insp->bufsize); 256 | sl = strlen(value); 257 | if (write(fd, value, sl) == -1) 258 | perror(bs); 259 | close(fd); 260 | } 261 | 262 | } 263 | } 264 | 265 | void 266 | load_module(struct item *itp) 267 | { 268 | char cmdline[BUFSIZE]; 269 | 270 | snprintf(cmdline, sizeof(cmdline), "modprobe %s", itp->target); 271 | if (system(cmdline)) { 272 | perror(cmdline); 273 | exit(EC_SYSTEM); 274 | } 275 | } 276 | 277 | void 278 | enable_or_disable(struct instance *insp, struct item *itp) 279 | { 280 | /* 281 | * need to parse out subsys/probe from subsys/all 282 | */ 283 | char fname[BUFSIZE], *sp, *wbuf, *msg; 284 | int fd, len; 285 | 286 | if (verbose > 1 && itp->optarg[0]) 287 | fprintf(stderr, "eod: optarg: %s\n", itp->optarg); 288 | 289 | sp = strchr(itp->target, '/'); 290 | 291 | if (!sp) { 292 | fprintf(stderr, "missing slash on line %d in %s\n", 293 | itp->line, itp->fpath); 294 | exit(EC_SYNTAX); 295 | } 296 | sp++; 297 | if (strncmp(sp, "all", 3) == 0) { 298 | /* .../subsystem/enable */ 299 | *--sp = '\0'; 300 | snprintf(fname, sizeof(fname), "%s/%s/events/%s/enable", 301 | inst_dir, insp->iname, itp->target); 302 | } else { 303 | /* .../subsystem/probe-name */ 304 | snprintf(fname, sizeof(fname), "%s/%s/events/%s/enable", 305 | inst_dir, insp->iname, itp->target); 306 | } 307 | 308 | if (itp->typ == ENABLE_T) { 309 | wbuf = "1"; 310 | msg = "enable"; 311 | } else if (itp->typ == DISABLE_T) { 312 | wbuf = "0"; 313 | msg = "disable"; 314 | } else { 315 | fprintf(stderr, "internal error 2, bad typ %d\n", itp->typ); 316 | exit(EC_BADTYPE2); 317 | } 318 | 319 | if (verbose > 1) 320 | fprintf(stderr, "%s: %s\n", msg, fname); 321 | 322 | fd = open(fname, O_WRONLY); 323 | if (fd < 0) { 324 | if (errno == ENOENT) { 325 | fprintf(stderr, "%s: no such probe\n", itp->target); 326 | /* not fatal, just return */ 327 | return; 328 | } 329 | perror(fname); 330 | exit(EC_OPEN); 331 | } 332 | 333 | if (write(fd, wbuf, 1) != 1) { 334 | perror("write"); 335 | exit(EC_WRITE1); 336 | } 337 | close(fd); 338 | 339 | /* Enable a filter if one was specified */ 340 | if (itp->optarg[0] == '\0') 341 | return; 342 | 343 | snprintf(fname, sizeof(fname), "%s/%s/events/%s/filter", 344 | inst_dir, insp->iname, itp->target); 345 | 346 | fprintf(stderr, "applying filter '%s' to %s\n", itp->optarg, fname); 347 | 348 | fd = open(fname, O_WRONLY); 349 | if (fd < 0) { 350 | perror(fname); 351 | return; 352 | } 353 | len = strlen(itp->optarg); 354 | if (write(fd, itp->optarg, len) != len) { 355 | perror(fname); 356 | return; 357 | } 358 | close(fd); 359 | } 360 | 361 | int 362 | openw(char *f) 363 | { 364 | int fd; 365 | 366 | fd = open(f, O_WRONLY|O_CREAT|O_TRUNC, 0400); 367 | if (fd < 0) { 368 | perror(f); 369 | exit(EC_OPENLOG); 370 | } 371 | return (fd); 372 | } 373 | 374 | /* 375 | * Invoke logrotate(8) directly. This is necessary because fdr may 376 | * generate so much data as to make the crontab initiated log rotations 377 | * to not be able to keep up. 378 | */ 379 | void 380 | rotate(struct instance *insp) 381 | { 382 | struct stat s; 383 | char cfile[BUFSIZE]; 384 | int status; 385 | pid_t pid; 386 | char *args[4]; 387 | 388 | /* 389 | * Check to see if a logrotate(8) config file exists 390 | * with this instance name in /etc/logrotate.d. If 391 | * so, then use this. Otherwise, do it by hand. 392 | */ 393 | snprintf(cfile, sizeof(cfile), "/etc/logrotate.d/%s", insp->iname); 394 | 395 | if (verbose > 1) 396 | fprintf(stderr, "looking for %s\n", cfile); 397 | 398 | if (stat(cfile, &s) == 0) { 399 | pid = fork(); 400 | if (pid == 0) { 401 | args[0] = "logrotate"; 402 | args[1] = "-f"; 403 | args[2] = cfile; 404 | args[3] = NULL; 405 | 406 | if (execv("/usr/sbin/logrotate", args)) { 407 | perror("cannot exec logrotate"); 408 | exit(EC_EXEC); 409 | } 410 | } else if (pid == -1) { 411 | perror("fork"); 412 | } else { 413 | if (waitpid(pid, &status, 0) == pid) { 414 | if (!(WIFEXITED(status) && 415 | WEXITSTATUS(status) == 0)) { 416 | fprintf(stderr, "logrotate failed %d\n", 417 | status); 418 | } 419 | } 420 | } 421 | } 422 | } 423 | 424 | /* 425 | * If the number of probes is too high for the workload, we don't want to 426 | * flood the log with messages. Return 1 if the message should be throttled 427 | * and 0 if it should be emitted. 428 | */ 429 | int 430 | throttle(int *counter) 431 | { 432 | return (((*counter)++ % 1000) == 0) ? 0 : 1; 433 | } 434 | 435 | void 436 | saveto(struct instance *insp, struct item *itp) 437 | { 438 | char fname[BUFSIZE]; 439 | int rfd, wfd, n, rsize, pctf, logoro = 0; 440 | char *buf; 441 | struct stat s; 442 | struct statvfs vfs; 443 | struct sigaction sa; 444 | int warn1 = 0, warn2 = 0, warn3 = 0; 445 | 446 | /* 447 | * If the saveto file already exists, then rotate it. This avoids 448 | * the rather unfortunate scenario where the system reboots and then 449 | * immediately overwrites the old data. 450 | */ 451 | if (stat(itp->target, &s) == 0 && s.st_size > 0) { 452 | if (verbose) 453 | fprintf(stderr, "rotating %s\n", itp->target); 454 | rotate(insp); 455 | } 456 | 457 | snprintf(fname, sizeof(fname), "%s/%s/trace_pipe", 458 | inst_dir, insp->iname); 459 | 460 | fprintf(stderr, "saving from %s to %s\n", fname, itp->target); 461 | 462 | sa.sa_sigaction = sighup; 463 | sa.sa_flags = 0; 464 | if (sigaction(SIGHUP, &sa, NULL)) 465 | perror("sigaction SIGHUP, continuing"); 466 | 467 | rfd = open(fname, O_RDONLY); 468 | if (rfd < 0) { 469 | perror(fname); 470 | exit(EC_OPENTRACE); 471 | } 472 | if (fstat(rfd, &s)) { 473 | perror(fname); 474 | exit(EC_FSTAT); 475 | } 476 | rsize = s.st_blksize; 477 | buf = malloc(rsize); 478 | if (buf == NULL) { 479 | perror("malloc"); 480 | exit(EC_MALLOC); 481 | } 482 | insp->fud = rfd; 483 | 484 | wfd = openw(itp->target); 485 | 486 | for (;;) { 487 | n = read(rfd, buf, rsize); 488 | 489 | if (n == -1 && got_sighup) { 490 | if (verbose > 1) 491 | fprintf(stderr, "got SIGHUP from read\n"); 492 | got_sighup = 0; 493 | /* 494 | * Most likely a SIGHUP from logrotate(8), we 495 | * need to close the current file and make a 496 | * new one. That way, the name of the current 497 | * log file is preserved in spite of the action 498 | * of logrotate. 499 | */ 500 | (void) close(wfd); 501 | wfd = openw(itp->target); 502 | continue; 503 | } 504 | if (n == 0) 505 | break; 506 | 507 | if (fstat(wfd, &s) == 0) { 508 | /* 509 | * If the file has been unlinked, then close the 510 | * fd so that free space will be released. 511 | */ 512 | if (s.st_nlink == 0) { 513 | fprintf(stderr, "closing %s\n", itp->target); 514 | (void) close(wfd); 515 | wfd = openw(itp->target); 516 | 517 | } else if (s.st_size > insp->maxsize) { 518 | if (!throttle(&warn1)) { 519 | fprintf(stderr, "file size for %s ", 520 | itp->target); 521 | fprintf(stderr, "exceeded, rotating\n"); 522 | } 523 | close(wfd); 524 | rotate(insp); 525 | wfd = openw(itp->target); 526 | } 527 | } 528 | /* 529 | * Be careful not to run the file system out of space, 530 | * hold 5% in reserve. 531 | */ 532 | if (fstatvfs(wfd, &vfs) == 0) { 533 | pctf = (vfs.f_bavail * 100) / vfs.f_blocks; 534 | if (pctf <= insp->minfree) { 535 | if (!throttle(&warn2)) 536 | fprintf(stderr, 537 | "free space too low for %s\n", 538 | itp->target); 539 | 540 | /* Try a log rotation. */ 541 | if (logoro == 0) { 542 | close(wfd); 543 | rotate(insp); 544 | wfd = openw(itp->target); 545 | logoro++; 546 | } 547 | } else 548 | logoro = 0; 549 | 550 | } 551 | if (write(wfd, buf, n) != n) 552 | if (!throttle(&warn3)) { 553 | perror(itp->target); 554 | } 555 | } 556 | } 557 | 558 | 559 | int 560 | read_config_file(const char *fpath, const struct stat *sb, int typeflag) 561 | { 562 | FILE *f; 563 | char buf[BUFSIZE], save_buf[BUFSIZE], *bp; 564 | int l; 565 | struct item *itp; 566 | struct instance *insp; 567 | char *cp, *verbp = NULL, *targetp = NULL, *optarg = NULL; 568 | char *savep; 569 | 570 | if (typeflag != FTW_F) 571 | return 0; 572 | 573 | cp = strstr(fpath, ".conf"); 574 | if (cp == NULL) 575 | return 0; 576 | 577 | l = strlen(fpath); 578 | /* Weed out stuff like foo.conf.OLD */ 579 | if (cp != fpath+l-strlen(".conf")) 580 | return 0; 581 | 582 | fprintf(stderr, "reading %s\n", fpath); 583 | 584 | f = fopen(fpath, "r"); 585 | if (f == NULL) { 586 | perror(fpath); 587 | return 1; 588 | } 589 | 590 | insp = malloc(sizeof(struct instance)); 591 | if (insp == NULL) { 592 | perror("malloc"); 593 | return 1; 594 | } 595 | insque(insp, &anchor); 596 | insp->ifirst = insp->ilast = NULL; 597 | 598 | for (l = 1; fgets(buf, sizeof(buf), f) != NULL; l++) { 599 | 600 | if (buf[0] == '#' || buf[0] == '\n') 601 | continue; 602 | 603 | buf[strlen(buf)-1] = '\0'; 604 | /* 605 | * Save a copy of buf since strtok_r modifies the original. 606 | */ 607 | strncpy(save_buf, buf, sizeof(save_buf)); 608 | 609 | /* 610 | * The structure of the directives is: 611 | * verb target [optional-argument] 612 | */ 613 | bp = buf; 614 | verbp = targetp = optarg = NULL; 615 | while ((cp = strtok_r(bp, " \t", &savep)) != NULL) { 616 | bp = NULL; /* for next call to strtok */ 617 | if (verbp == NULL) { 618 | verbp = cp; 619 | continue; 620 | } else if (targetp == NULL) { 621 | targetp = cp; 622 | continue; 623 | } else if (optarg == NULL) { 624 | optarg = &save_buf[cp - buf]; 625 | 626 | if (verbose > 1) 627 | fprintf(stderr, "optarg: %s\n", optarg); 628 | break; 629 | } 630 | } 631 | 632 | if (verbp == NULL || targetp == NULL) { 633 | /* no verbs found, skip the line */ 634 | continue; 635 | } 636 | 637 | itp = malloc(sizeof(struct item)); 638 | if (itp == NULL) { 639 | perror("malloc"); 640 | exit(EC_MALLOC); 641 | } 642 | 643 | (void) strncpy(itp->verb, verbp, sizeof(itp->verb)); 644 | (void) strncpy(itp->target, targetp, sizeof(itp->target)); 645 | /* fpath could to be stored in the instance */ 646 | (void) strncpy(itp->fpath, fpath, sizeof(itp->fpath)); 647 | itp->line = l; 648 | 649 | /* 650 | * this could be table driven 651 | * could move these checks inside the strtok loop 652 | */ 653 | if (strncmp(verbp, INSTANCE, sizeof(INSTANCE)-1) == 0) { 654 | (void) strncpy(insp->iname, targetp, 655 | sizeof(insp->iname)); 656 | snprintf(insp->dname, sizeof(insp->dname), "%s/%s", 657 | inst_dir, itp->target); 658 | itp->typ = INSTANCE_T; 659 | insp->bufsize = 0; /* take the ftrace default */ 660 | if (optarg) { 661 | insp->bufsize = getvalue(optarg); 662 | fprintf(stderr, "bufsize: %lu\n", insp->bufsize); 663 | } 664 | 665 | anchor.numi++; 666 | } else if (strncmp(verbp, MODPROBE, sizeof(MODPROBE)-1) == 0) 667 | itp->typ = MODPROBE_T; 668 | else if (strncmp(verbp, ENABLE, sizeof(ENABLE)-1) == 0) { 669 | itp->typ = ENABLE_T; 670 | if (optarg) 671 | strncpy(itp->optarg, optarg, 672 | sizeof(itp->optarg)); 673 | else 674 | itp->optarg[0] = '\0'; 675 | } else if (strncmp(verbp, DISABLE, sizeof(DISABLE)-1) == 0) 676 | itp->typ = DISABLE_T; 677 | else if (strncmp(verbp, SAVETO, sizeof(SAVETO)-1) == 0) { 678 | itp->typ = SAVETO_T; 679 | if (optarg) 680 | insp->maxsize = getvalue(optarg); 681 | else 682 | insp->maxsize = MAXSIZE_DEFAULT; 683 | 684 | if (verbose > 1) 685 | fprintf(stderr, "maxsize: %ld\n", 686 | insp->maxsize); 687 | 688 | } else if (strncmp(verbp, MINFREE, sizeof(MINFREE)-1) == 0) { 689 | itp->typ = MINFREE_T; 690 | insp->minfree = MINFREE_DEFAULT; 691 | if (targetp) 692 | insp->minfree = atoi(targetp); 693 | 694 | if (insp->minfree > 100 || insp->minfree <= 0) 695 | insp->minfree = MINFREE_DEFAULT; 696 | 697 | if (verbose) 698 | fprintf(stderr, "minfree: %d\n", 699 | insp->minfree); 700 | 701 | } else { 702 | fprintf(stderr, "BAD verb at %d in %s\n", l, fpath); 703 | exit(EC_BADVERB); 704 | } 705 | 706 | if (insp->ifirst == NULL) { 707 | /* empty, but put it on the front */ 708 | insp->ifirst = insp->ilast = itp; 709 | itp->forw = NULL; 710 | } else { 711 | /* put it on the end */ 712 | itp->forw = NULL; 713 | insp->ilast->forw = itp; 714 | insp->ilast = itp; 715 | } 716 | } 717 | return 0; 718 | } 719 | 720 | void 721 | make_one_instance(struct instance *insp) 722 | { 723 | pid_t p; 724 | struct item *itp; 725 | struct sigaction sa; 726 | 727 | if (verbose > 1) 728 | fprintf(stderr, "creating instance for %s\n", insp->iname); 729 | 730 | fflush(stdout); 731 | fflush(stderr); 732 | 733 | p = fork(); 734 | if (p == (pid_t) -1) { 735 | perror("fork"); 736 | exit(EC_FORK); 737 | } 738 | if (p) 739 | return; /* parent just returns */ 740 | 741 | sa.sa_handler = SIG_DFL; 742 | sa.sa_flags = 0; 743 | if (sigaction(SIGTERM, &sa, NULL) || 744 | sigaction(SIGINT, &sa, NULL)) 745 | perror("sigaction"); 746 | 747 | 748 | for (itp = insp->ifirst; itp != NULL; itp = itp->forw) 749 | switch (itp->typ) { 750 | case INSTANCE_T: 751 | instance(insp, itp); 752 | break; 753 | case MODPROBE_T: 754 | load_module(itp); 755 | break; 756 | case ENABLE_T: 757 | case DISABLE_T: 758 | enable_or_disable(insp, itp); 759 | break; 760 | case SAVETO_T: 761 | saveto(insp, itp); 762 | break; 763 | case MINFREE_T: 764 | /* ignore it, already set in the instance */ 765 | break; 766 | default: 767 | fprintf(stderr, "internal error 1, bad typ %d\n", 768 | itp->typ); 769 | exit(EC_BADTYPE1); 770 | } 771 | 772 | fprintf(stderr, "instance %s exiting\n", insp->iname); 773 | exit(0); 774 | } 775 | 776 | int 777 | main(int argc, char **argv) 778 | { 779 | struct sigaction sa; 780 | struct instance *insp; 781 | int opt, ret; 782 | 783 | ret = daemon(1, 1); 784 | if (ret) 785 | return ret; 786 | 787 | while ((opt = getopt(argc, argv, "vd:")) != -1) 788 | switch (opt) { 789 | case 'v': 790 | verbose++; 791 | break; 792 | case 'd': 793 | snprintf(inst_dir, sizeof(inst_dir), "%s/instances", 794 | optarg); 795 | break; 796 | default: 797 | fprintf(stderr, "%s: [-v] [-d instance-dir-name]\n", 798 | argv[0]); 799 | exit(EC_BADARGS); 800 | } 801 | 802 | if (inst_dir[0] == '\0') 803 | strncpy(inst_dir, INST_DIR, sizeof(inst_dir)); 804 | 805 | sa.sa_sigaction = sighandler; 806 | sa.sa_flags = 0; 807 | if (sigaction(SIGTERM, &sa, NULL) || 808 | sigaction(SIGINT, &sa, NULL)) 809 | perror("sigaction"); /* continue anyway */ 810 | 811 | /* 812 | * Ignore SIGHUP because logrotate will send it when the log 813 | * files are rotated. The procs handling saveto() will deal 814 | * with it. 815 | */ 816 | sa.sa_handler = SIG_IGN; 817 | sa.sa_flags = 0; 818 | if (sigaction(SIGHUP, &sa, NULL)) 819 | perror("sigaction SIGHUP"); /* continue anyway */ 820 | 821 | ftw(FDR_CONFIG_DIR, read_config_file, 1); 822 | /* 823 | * At this point, we've ingested all the config files and 824 | * no syntax errors were detected. Create a separate process 825 | * for each instance. This could be done with pthreads. 826 | */ 827 | for (insp = anchor.forw; (void *)insp != (void *)&anchor; 828 | insp = insp->forw) 829 | make_one_instance(insp); 830 | 831 | /* Just hang out and let systemd terminate us via SIGTERM. */ 832 | pause(); 833 | exit(0); 834 | } 835 | -------------------------------------------------------------------------------- /fdrd.man: -------------------------------------------------------------------------------- 1 | .\" 2 | .\" nfsd(8) 3 | .\" 4 | .\" Copyright (C) XXX 5 | .TH fdrd 8 "15 June 2021" 6 | .SH NAME 7 | fdrd \- Flight Data Recorder daemon 8 | .SH SYNOPSIS 9 | .BI "/usr/bin/fdrd" 10 | .SH DESCRIPTION 11 | The 12 | .B fdrd 13 | program is a daemon which enables ftrace probes, 14 | harvests ftrace data and (optionally) writes the data to a file. 15 | .P 16 | The behavior of fdr is defined by configuration files stored in 17 | .B /etc/fdr.d. 18 | During service startup, fdr will process each file in 19 | the directory which has the suffix of .conf. If new config files 20 | are added to the fdr.d directory, then the service must be restarted 21 | to recognize the new configuration information. 22 | .P 23 | fdr is controlled by 24 | .B systemd 25 | (8) on systems where systemd is 26 | available. Error messages from fdr can be viewed via systemctl, 27 | for example, 28 | .P 29 | .B 30 | systemctl status -l fdr 31 | .SH Configuration File Syntax 32 | 33 | The following keywords and options are recognized 34 | 35 | .B instance 36 | iname [buffer-size] 37 | .P 38 | Create a new ftrace instance called "iname". This instance 39 | will appear in 40 | .B /sys/kernel/debug/tracing/instances 41 | .P 42 | The optional buffer-size parameter can be used to control 43 | the size of the ftrace buffers for this instance in the 44 | kernel. A suffix of 'k', 'K', 'm', 'M', 'g' or 'G' may be 45 | used to specify kilobytes, megabytes or gigabytes. 46 | .P 47 | .B modprobe 48 | module-name 49 | .P 50 | Force the named module to be loaded by fdr. This can be 51 | useful when the module is normally loaded on demand and 52 | the probes cannot be enabled until the module is loaded. 53 | .P 54 | .B enable 55 | subsystem-name/probe-name [filter] 56 | .P 57 | Enable an ftrace probe in the specified subsystem. Both 58 | the subsystem name and probe name are defined by the kernel. 59 | .P 60 | The optional filter parameter allows an ftrace filter to 61 | be set as well. This will limit the amount of data being 62 | emitted. The syntax of the filter language is 63 | defined by ftrace itself and the parameters are defined 64 | by the static tracepoint being enabled in the kernel. 65 | .P 66 | .B enable 67 | subsystem-name/all 68 | .P 69 | Enable all ftrace probes for the subsystem. 70 | .P 71 | .P 72 | .B disable 73 | subsystem-name/probe-name 74 | .P 75 | Disable an ftrace probe in the specified subsystem. This 76 | can be useful to disable selective probes when the "ALL" 77 | keyword has been used. 78 | .P 79 | .B disable 80 | subsystem-name/all 81 | .P 82 | Disable all probes in the specified subsystem. 83 | .P 84 | .B saveto 85 | file-name [maxsize] 86 | .P 87 | Save the output of enabled probes to the named file. If 88 | the optional maxsize parameter is given, the daemon will 89 | initiate a log rotation, see 90 | .B Log Rotation 91 | below. 92 | A suffix 93 | of 'k', 'K', 'm', 'M', 'g' or 'G' may be used to specify 94 | kilobytes, megabytes or gigabytes. 95 | .P 96 | If no saveto directive is present, then fdr will create the 97 | instance and enable the probes. In this case, the data 98 | can be harvested manually by reading: 99 | .P 100 | /sys/kernel/debug/tracing/instances/iname/trace_pipe 101 | .P 102 | The ftrace buffers in the kernel are circular. If no 103 | process harvests the data, new data will overwrite old data. 104 | .P 105 | .B minfree 106 | value 107 | .P 108 | Limit the output by the daemon based on free space in the 109 | file system for the save file. If free space percentage is 110 | below the specified value, no output will be written. 111 | .P 112 | If no minfree directive is present, fdr will use 5% by 113 | default. 114 | .SH LOG ROTATION 115 | .P 116 | fdr can use logrotate(8) to manage the output files. By convention, 117 | /etc/logrotate.d/instance-name controls the behavior of logrotate. 118 | .P 119 | fdr will also invoke logrotate directly at startup and when reaching 120 | the maxsize limit for the save file. 121 | .SH SEE ALSO 122 | .P 123 | https://lwn.net/Articles/410200/ 124 | .P 125 | https://www.kernel.org/doc/Documentation/trace/ftrace.txt 126 | .SH AUTHOR 127 | Bill Baker and Calum Mackay 128 | -------------------------------------------------------------------------------- /samples/nfs: -------------------------------------------------------------------------------- 1 | # 2 | # Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved. 3 | # DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER. 4 | # 5 | # Licensed under the Universal Permissive License v 1.0 as shown at 6 | # https://oss.oracle.com/licenses/upl 7 | # 8 | # This is a sample config file for flight data recorder to enable 9 | # probes in the nfs client. To use this, copy the file to 10 | # /etc/fdr.d/nfs.conf and start fdr with systemctl start fdr 11 | # 12 | instance nfs 13 | # modprobe nfs4 will drag in sunrpc & nfs as deps 14 | modprobe nfsv4 15 | # 16 | # enable nfs4/nfs4_cb_recall 17 | enable nfs4/nfs4_create_session 18 | enable nfs4/nfs4_destroy_clientid 19 | enable nfs4/nfs4_destroy_session 20 | enable nfs4/nfs4_exchange_id 21 | enable nfs4/nfs4_get_fs_locations 22 | enable nfs4/nfs4_open_expired 23 | enable nfs4/nfs4_open_reclaim 24 | enable nfs4/nfs4_reclaim_complete 25 | enable nfs4/nfs4_reclaim_delegation 26 | enable nfs4/nfs4_renew 27 | enable nfs4/nfs4_secinfo 28 | enable nfs4/nfs4_setclientid 29 | enable nfs4/nfs4_setclientid_confirm 30 | enable nfs4/nfs4_test_delegation_stateid 31 | enable nfs4/nfs4_test_lock_stateid 32 | enable nfs4/nfs4_test_open_stateid 33 | # the nfs4_xdr_status only exists in more recent kernels, v5.x 34 | #enable nfs4/nfs4_xdr_status (error != 0) 35 | enable nfs4/nfs4_xdr_status (error != 0 && (!(error == 2 && op == 15))) 36 | # 37 | enable sunrpc/rpc__auth_tooweak 38 | enable sunrpc/rpc_bad_callhdr 39 | enable sunrpc/rpc__bad_creds 40 | enable sunrpc/rpc_bad_verifier 41 | enable sunrpc/rpc_bind_status 42 | enable sunrpc/rpc_connect_status 43 | enable sunrpc/rpc__garbage_args 44 | enable sunrpc/rpc__mismatch 45 | enable sunrpc/rpc__proc_unavail 46 | enable sunrpc/rpc__prog_mismatch 47 | enable sunrpc/rpc__prog_unavail 48 | enable sunrpc/rpc_socket_close 49 | enable sunrpc/rpc_socket_connect 50 | enable sunrpc/rpc_socket_error 51 | enable sunrpc/rpc_socket_reset_connection 52 | enable sunrpc/rpc_socket_shutdown 53 | enable sunrpc/rpc_socket_state_change 54 | enable sunrpc/rpc__stale_creds 55 | enable sunrpc/rpc__unparsable 56 | enable sunrpc/rpc_xdr_alignment 57 | enable sunrpc/rpc_xdr_overflow 58 | enable sunrpc/xprt_ping 59 | enable sunrpc/xprt_timer 60 | # 61 | saveto /var/log/nfs.log 128k 62 | -------------------------------------------------------------------------------- /samples/nfs.logrotate: -------------------------------------------------------------------------------- 1 | /var/log/nfs.log { 2 | missingok 3 | create 4 | daily 5 | rotate 4 6 | maxsize 128k 7 | nodateext 8 | postrotate 9 | pkill -x -HUP fdrd 10 | endscript 11 | } 12 | --------------------------------------------------------------------------------