├── Makefile ├── might_crash.cpp ├── appveyor.yml ├── enable-local-dumps.reg ├── shippable.yml ├── circle.yml ├── .travis.yml ├── test.bat └── README.md /Makefile: -------------------------------------------------------------------------------- 1 | 2 | 3 | all: build 4 | 5 | build: 6 | $(CXX) -o test might_crash.cpp -g -O0 -DDEBUG 7 | 8 | clean: 9 | rm -rf ./test 10 | rm -rf ./test.dSYM -------------------------------------------------------------------------------- /might_crash.cpp: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | int main() { 4 | if( getenv("CRASH_PLEASE") ) { 5 | // this will crash 6 | delete (int*)0xffffffff; 7 | } 8 | } -------------------------------------------------------------------------------- /appveyor.yml: -------------------------------------------------------------------------------- 1 | environment: 2 | matrix: 3 | - msvs_toolset: 12 4 | platform: x64 5 | - msvs_toolset: 12 6 | platform: x86 7 | 8 | #os: Visual Studio 2015 RC 9 | 10 | install: 11 | - test.bat 12 | 13 | build: OFF 14 | test: OFF 15 | deploy: OFF 16 | -------------------------------------------------------------------------------- /enable-local-dumps.reg: -------------------------------------------------------------------------------- 1 | Windows Registry Editor Version 5.00 2 | 3 | [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps] 4 | 5 | [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\might_crash.exe] 6 | "DumpType"=dword:00000002 7 | -------------------------------------------------------------------------------- /shippable.yml: -------------------------------------------------------------------------------- 1 | build: 2 | pre_ci: 3 | - docker build -t logbt-precise -f Dockerfile.ubuntu-precise . 4 | - cat /proc/sys/kernel/core_pattern 5 | 6 | language: none 7 | 8 | before_install: 9 | - echo "before_install" 10 | - git clone http://github.com/mapbox/logbt 11 | - cat /proc/sys/kernel/core_pattern 12 | 13 | install: 14 | - pwd 15 | - cd logbt 16 | - ./bin/logbt --test 17 | 18 | before_script: 19 | - echo "before_script" 20 | 21 | script: 22 | - echo "script" -------------------------------------------------------------------------------- /circle.yml: -------------------------------------------------------------------------------- 1 | machine: 2 | environment: 3 | CRASH_PLEASE: boooooooom 4 | 5 | checkout: 6 | post: 7 | - gcc --version 8 | 9 | ## Customize dependencies 10 | dependencies: 11 | pre: 12 | - sudo apt-get -y update && sudo apt-get -y install gdb 13 | 14 | cache_directories: 15 | - "./mason_packages" 16 | 17 | test: 18 | pre: 19 | - ulimit -c 20 | - ulimit -c unlimited -S 21 | post: 22 | - RESULT=0 23 | # Compile our demo program which will crash if 24 | # the CRASH_PLEASE environment variable is set (to anything) 25 | - make 26 | # Run the program to prompt a crash 27 | # Note: we capture the return code of the program here and add 28 | # `|| true` to ensure that travis continues past the crash 29 | - ./test 30 | - ls 31 | - RESULT=$(./test > /dev/null)$? || true 32 | - if [[ ${RESULT} == 0 ]]; then echo "\\o/ our test worked without problems"; else echo "ruhroh test returned an errorcode of $RESULT"; fi; 33 | # If the program returned an error code, now we check for a 34 | # core file in the current working directory and dump the backtrace out 35 | - for i in $(find ./ -maxdepth 1 -name 'core*' -print); do gdb $(pwd)/test core* -ex "thread apply all bt" -ex "set pagination 0" -batch; done; 36 | # now we should present travis with the original 37 | # error code so the run cleanly stops 38 | - if [[ ${RESULT} != 0 ]]; then exit $RESULT ; fi; 39 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: cpp 2 | 3 | before_install: 4 | - node -e "console.log(process.arch)" || true 5 | # What is the current file size max for core files? 6 | # It is usually 0, which means no core file will be dumped if there is a crash 7 | - ulimit -c 8 | - ulimit -a -S 9 | - ulimit -a -H 10 | - cat /proc/sys/kernel/core_pattern 11 | - cat /etc/default/apport 12 | 13 | script: 14 | # Compile our demo program which will crash if 15 | # the CRASH_PLEASE environment variable is set (to anything) 16 | - make 17 | # Set the ulimit 18 | - ulimit -c unlimited -S 19 | # Run the program to prompt a crash 20 | # Note: we capture the return code of the program here and add 21 | # `|| true` to ensure that travis continues past the crash 22 | - CRASH_PLEASE=1 ./test || RESULT=$? 23 | - ls -l 24 | # For the purpose of this repo we assume the test will crash so let's 25 | # assert that the RESULT is 139 (exit code for a segfault) 26 | # For a real repo you would likely assert the RESULT is zero 27 | - if [[ ${RESULT} != 139 ]]; then echo "expected segfault and 139 exit but instead exited with ${RESULT}" && exit 1; fi; 28 | - for i in $(find ./ -maxdepth 1 -name 'core*' -print); do gdb $(pwd)/test core* -ex "thread apply all bt" -ex "set pagination 0" -batch; done; 29 | 30 | addons: 31 | apt: 32 | packages: 33 | - gdb 34 | - apport 35 | 36 | matrix: 37 | include: 38 | - os: linux 39 | arch: amd64 40 | - os: linux 41 | arch: arm64 42 | - os: linux 43 | arch: arm64-graviton2 44 | virt: vm 45 | - os: linux 46 | arch: arm64-graviton2 47 | virt: lxd 48 | - os: linux 49 | arch: amd64 50 | -------------------------------------------------------------------------------- /test.bat: -------------------------------------------------------------------------------- 1 | ::@echo off 2 | 3 | SET EL=0 4 | 5 | 6 | reg import enable-local-dumps.reg 7 | IF %ERRORLEVEL% NEQ 0 ECHO could not enable local dumps && GOTO ERROR 8 | 9 | if "%msvs_toolset%"=="" SET msvs_toolset=12 10 | if "%platform%"=="" SET platform=x64 11 | 12 | ECHO msvs_toolset^: %msvs_toolset% 13 | ECHO platform^: %platform% 14 | 15 | :: https://msdn.microsoft.com/en-us/library/f2ccy3wt.aspx 16 | ECHO "setting path for msbuild" 17 | SET PATH=C:\Program Files (x86)\MSBuild\%msvs_toolset%.0\bin;%PATH% 18 | IF %ERRORLEVEL% NEQ 0 GOTO ERROR 19 | 20 | ECHO "setting path for VC" 21 | SET PATH=C:\Program Files (x86)\Microsoft Visual Studio %msvs_toolset%.0\VC\bin;%PATH% 22 | IF %ERRORLEVEL% NEQ 0 GOTO ERROR 23 | 24 | ::ECHO "installing windbg" 25 | ::call choco install windbg 26 | 27 | ECHO "setting path for windbg" 28 | set PATH=C:\Program Files (x86)\Windows Kits\8.1\Debuggers\%platform%;%PATH% 29 | IF %ERRORLEVEL% NEQ 0 GOTO ERROR 30 | 31 | dir "C:\Program Files (x86)\Windows Kits\8.1\Debuggers\%platform%" 32 | 33 | :: note: will override %PLATFORM% to upper case 34 | ECHO "enabling VC env %platform%" 35 | if %platform% == x64 CALL "C:\Program Files (x86)\Microsoft Visual Studio %msvs_toolset%.0\VC\vcvarsall.bat" amd64 36 | IF %ERRORLEVEL% NEQ 0 GOTO ERROR 37 | if %platform% == x86 CALL "C:\Program Files (x86)\Microsoft Visual Studio %msvs_toolset%.0\VC\vcvarsall.bat" amd64_x86 38 | IF %ERRORLEVEL% NEQ 0 GOTO ERROR 39 | ECHO "VC env %platform% enabled" 40 | 41 | ECHO "compiling test program" 42 | cl /MD /Zi /Fmight_crash.pdb /nologo /EHsc /DEBUG /D DEBUG might_crash.cpp 43 | IF %ERRORLEVEL% NEQ 0 GOTO ERROR 44 | 45 | ECHO "running test program" 46 | .\might_crash 47 | IF %ERRORLEVEL% NEQ 0 GOTO ERROR 48 | 49 | :: expected to crash so we ignore any error so the script continues 50 | ECHO "running test program again" 51 | set CRASH_PLEASE=1 52 | .\might_crash 53 | 54 | :: https://msdn.microsoft.com/en-us/library/windows/hardware/ff542967(v=vs.85).aspx 55 | ::SET WIN_SDK_ROOT=C:\Program Files\Microsoft SDKs\Windows 56 | ::set WINDOWS_SDK_VERSION=v7.1 57 | ::dir "%WIN_SDK_ROOT%" 58 | ::dir "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%" 59 | ::dir "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Bin" 60 | ::dir "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Redist\amd64" 61 | 62 | ::ECHO "Running WindowsSdkVer.exe" 63 | ::call "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Setup\WindowsSdkVer.exe" -q -version:%WINDOWS_SDK_VERSION% 64 | ::ECHO "Running SetEnv.cmd" 65 | ::call "%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Bin\SetEnv.cmd" /%platform% /release 66 | 67 | :: https://msdn.microsoft.com/en-us/library/windows/desktop/bb787181(v=vs.85).aspx 68 | :: http://www.codeproject.com/Articles/31997/How-to-Use-and-Understand-the-Windows-Console-Debu 69 | IF EXIST %LOCALAPPDATA%\CrashDumps ( 70 | ECHO CrashDumps found 71 | dir %LOCALAPPDATA%\CrashDumps 72 | ::windbg 73 | ::ntsd 74 | for %%f in (%LOCALAPPDATA%\CrashDumps\*) do cdb -c "k;q" -G -v -y .\ -i .\ -z "%%f" 75 | ) 76 | 77 | GOTO DONE 78 | 79 | :ERROR 80 | echo --------- ERROR! ------------ 81 | ECHO ERRORLEVEL %ERRORLEVEL% 82 | SET EL=%ERRORLEVEL% 83 | 84 | :DONE 85 | echo --------- DONE ------------ 86 | EXIT /b %EL% 87 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## travis-coredump 2 | 3 | [![Build Status](https://travis-ci.org/springmeyer/travis-coredump.svg?branch=master)](https://travis-ci.org/springmeyer/travis-coredump) 4 | 5 | WARNING: only `sudo:true` machines can produce coredumps currently on travis (https://github.com/springmeyer/travis-coredump/issues/6) 6 | 7 | On travis you might see a cryptic error like: 8 | 9 | /home/travis/build.sh: line 41: 2300 Segmentation fault 10 | 11 | Or: 12 | 13 | *** glibc detected *** ./test: double free or corruption (top): *** 14 | 15 | Yay, your app segfaulted or hit a double-free. And you can't reproduce locally. And running your program in the gdb interpreter won't work on a remote machine. What now? 16 | 17 | You have come to the right place. 18 | 19 | This repo contains a demo of: 20 | 21 | - How to enable coredumps for linux (`ulimit -c unlimited`) 22 | - A c++ program that will simulate a crash 23 | - The `gdb` incantation to make the crash backtrace visible in the travis logs 24 | 25 | See `.travis.yml` for detailed instructions you can copy into your own `.travis.yml`. See sample logs at . This repo is configured so that with two runs in the travis matix: 26 | 27 | - One run tests a program that does not crash and therefore the build should cleanly pass 28 | - The other run tests a program that does crash, reports the backtrace, then exits the run with the same errorcode as generated by the crash. 29 | 30 | The backtrace should reveal to you the file and line number of the bug: 31 | 32 | ``` 33 | Core was generated by `./test'. 34 | Program terminated with signal 11, Segmentation fault. 35 | #0 0x00007f438532302c in free () from /lib/x86_64-linux-gnu/libc.so.6 36 | Thread 1 (LWP 2611): 37 | #0 0x00007f438532302c in free () from /lib/x86_64-linux-gnu/libc.so.6 38 | #1 0x00000000004005d6 in main () at might_crash.cpp:6 39 | ``` 40 | 41 | ## Running locally 42 | 43 | To experiment locally: 44 | 45 | 1) Build the test program 46 | 47 | ```sh 48 | make 49 | ``` 50 | 51 | 2) Enable core files 52 | 53 | ```sh 54 | ulimit -c unlimited -S 55 | ``` 56 | 57 | 3) Run test program and make it crash 58 | 59 | ``` 60 | CRASH_PLEASE=1 ./test 61 | ``` 62 | 63 | You should see a nasty error show up with `(core dumped)` in the last line. 64 | 65 | 4) Generate a backtrace 66 | 67 | Great, so now that you've confirmed it crashed, let's get more fancy. 68 | 69 | To script the auto generation of the backtrace we can collect the PID and use that to find the core file and generate a backtrace: 70 | 71 | 72 | #### OS X 73 | 74 | On OS X for this to work (unfortunely) you need a very recent lldb version. 75 | ```sh 76 | CRASH_PLEASE=1 ./test & pid=$! && fg; 77 | lldb --core /cores/core.$pid --batch --one-line "bt" 78 | ``` 79 | 80 | #### Linux 81 | 82 | ```sh 83 | sudo apt-get install gdb 84 | CRASH_PLEASE=1 ./test 85 | gdb $(pwd)/test core -ex "thread apply all bt" -ex "set pagination 0" -batch 86 | ``` 87 | 88 | 89 | ### Other platforms 90 | 91 | If you are on OS X, you don't need to worry about these steps, just look inside: 92 | 93 | ~/Library/Logs/DiagnosticReports/ 94 | 95 | And a backtrace should be present for any program that just crashed. 96 | 97 | If you are on Windows, let me know if you have tips on generating backtraces for crashes on the command line. 98 | --------------------------------------------------------------------------------