.
675 |
--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
1 |
2 |
3 |
4 |
5 | # ReproDroid
6 | ReproDroid is a framework which can be used to create, refine and execute reproducible benchmarks for Android app analysis tools.
7 |
8 |
9 | !!! Update can be found in the Errata section below !!!
10 |
11 |
12 | ## Toolchain
13 | The complete ReproDroid framework consists of BREW and its underlying AQL-System which uses the AQL.
14 | The picture below summarizes how the framework works.
15 | BREW takes a set of apps or a complete benchmark as input and issues one AQL-Query per benchmark case.
16 | Then, one query after another arrives at an AQL-System which produces one AQL-Answer per query.
17 | To do so, it uses analysis tools specified in BREW's configuration file.
18 | All AQL-Answers are gathered by BREW. Based on these answers a final report for e.g. a benchmark is carried out.
19 |
20 |
21 |
22 |
23 |
24 | The tools and results presented in the proposing paper can be downloaded for inspection here.
25 | In order to work with the framework, we suggest to download the up-to-date version of [BREW](https://FoelliX.github.io/BREW "Benchmark Refinement and Execution Wizard"). The underlying [AQL-System](https://FoelliX.github.io/AQL-System "System using the Analysis Query Language") is also available in a newer version.
26 |
27 |
28 |
29 | ## Tools
30 | To refine benchmarks and to determine the associated results the *Benchmark Refinement and Execution Wizard (BREW)* has been used. There are two versions available for download:
31 | - [BREW-Website](https://FoelliX.github.io/BREW) or [Github-Project](https://github.com/FoelliX/BREW) (Up-to-date versions)
32 | - [BREW](https://uni-paderborn.sciebo.de/s/ZmlRvtzI6pVYHVP/download?path=%2Ftools&files=BREW.zip) *(Version used for the computation of the results below)*
33 |
34 | A tutorial on how to fully load ReproDroid benchmark results can be found [here](https://github.com/FoelliX/BREW/wiki/ReproDroid)
35 |
36 | A documentation of the *Android App Analysis Query Langauge (AQL)* as well as the *AQL-System* using it is also obtainable online:
37 | - [AQL-System-Website](https://FoelliX.github.io/AQL-System) or [Github-Project](https://github.com/FoelliX/AQL-System) *(Up-to-date versions)*
38 | - [AQL-System](https://uni-paderborn.sciebo.de/s/ZmlRvtzI6pVYHVP/download?path=%2Ftools&files=AQLSystem.zip) *(Version used by BREW to compute the results below)*
39 |
40 | #### Remarks
41 | None of the six evaluated tools are contained in either of these tools.
42 | How to set up a configuration file in order to use a tool is explained in this [tutorial](https://github.com/FoelliX/BREW/wiki).
43 | The six evaluated tools themselves can be downloaded from their associated websites:
44 | - Amandroid: [https://bintray.com/arguslab/maven/argus-saf/3.1.2](https://bintray.com/arguslab/maven/argus-saf/3.1.2)
45 | - DIALDroid: [https://github.com/dialdroid-android/DIALDroid](https://github.com/dialdroid-android/DIALDroid)
46 | - DidFail: [https://www.cert.org/secure-coding/tools/didfail.cfm](https://www.cert.org/secure-coding/tools/didfail.cfm)
47 | - DroidSafe: [https://mit-pac.github.io/droidsafe-src](https://mit-pac.github.io/droidsafe-src)
48 | - FlowDroid: [https://github.com/secure-software-engineering/soot-infoflow-android/wiki](https://github.com/secure-software-engineering/soot-infoflow-android/wiki)
49 | - IccTA: [https://sites.google.com/site/icctawebpage/source-and-usage](https://sites.google.com/site/icctawebpage/source-and-usage)
50 |
51 |
52 |
53 | ## Results
54 | All result determined with ReproDroid can be found in this section.
55 |
56 | ### Droidbench
57 | The refined versions of DroidBench 2.0 and 3.0 as well as the extended DroidBench version can be downloaded here.
58 | Every download includes:
59 |
60 | - Benchmark
61 | - BREW benchmark file *(.ser file)*
62 | - Benchmark Cases *(.apk files)*
63 | - Groundtruth *(Expected results in AQL format: .xml)*
64 | - Source Code *(Eclipse/Android Studio project directories/archives)*
65 | - Results
66 | - BREW benchmark file including result-summary *(data/data.ser)*
67 | - AQL-Answers per app *(data/storage/*.xml)*
68 | - Expected and actual AQL-Answers per benchmark case *(output)*
69 | - Logfile *(log.txt)*
70 |
71 | #### Download
72 | - [DroidBench (extended)](https://uni-paderborn.sciebo.de/s/ZmlRvtzI6pVYHVP/download?path=%2Fbenchmarks&files=DroidBenchExtended.zip)
73 | - [DroidBench 3.0](https://uni-paderborn.sciebo.de/s/ZmlRvtzI6pVYHVP/download?path=%2Fbenchmarks&files=DroidBench30.zip)
74 | - [DroidBench 2.0](https://uni-paderborn.sciebo.de/s/ZmlRvtzI6pVYHVP/download?path=%2Fbenchmarks&files=DroidBench20.zip)
75 |
76 | ### Extensions for DroidBench
77 | The Feature-Checking and Intent-Matching benchmark extensions can be downloaded here. Both are available for Android API 19 and 26.
78 | Every download includes:
79 |
80 | - Benchmark
81 | - BREW benchmark file (.ser file)
82 | - Benchmark Cases (.apk files)
83 | - Groundtruth (Expected results in AQL format: .xml)
84 | - Source Code (Android Studio project directories/archives)
85 | - Results
86 | - BREW benchmark file including result-summary (data/data.ser)
87 | - AQL-Answers per app (data/storage/*.xml)
88 | - Expected and actual AQL-Answers per benchmark case (output)
89 | - Logfile (log.txt)
90 |
91 | #### Download
92 | - [Feature-Checking (API 19)](https://uni-paderborn.sciebo.de/s/ZmlRvtzI6pVYHVP/download?path=%2Fbenchmarks&files=FeatureChecking_API19.zip)
93 | - [Feature-Checking (API 26)](https://uni-paderborn.sciebo.de/s/ZmlRvtzI6pVYHVP/download?path=%2Fbenchmarks&files=FeatureChecking_API26.zip)
94 | - [Intent-Matching (API 19)](https://uni-paderborn.sciebo.de/s/ZmlRvtzI6pVYHVP/download?path=%2Fbenchmarks&files=IntentMatching_API19.zip)
95 | - [Intent-Matching (API 26)](https://uni-paderborn.sciebo.de/s/ZmlRvtzI6pVYHVP/download?path=%2Fbenchmarks&files=IntentMatching_API26.zip)
96 |
97 |
98 | ### ICC-Bench
99 | The refined version of ICC-Bench 2.0 can be downloaded here.
100 | It includes:
101 |
102 | - Benchmark
103 | - BREW benchmark file *(.ser file)*
104 | - Benchmark Cases *(.apk files)*
105 | - Groundtruth *(Expected results in AQL format: .xml)*
106 | - Source Code *(Project directories)*
107 | - Results
108 | - BREW benchmark file including result-summary *(data/data.ser)*
109 | - AQL-Answers per app *(data/storage/*.xml)*
110 | - Expected and actual AQL-Answers per benchmark case *(output)*
111 | - Logfile *(log.txt)*
112 |
113 | #### Download
114 | - [ICC-Bench 2.0](https://uni-paderborn.sciebo.de/s/ZmlRvtzI6pVYHVP/download?path=%2Fbenchmarks&files=ICCBench20.zip)
115 |
116 | ### DIALDroidBench
117 | The iteratively refined version of DIALDroidBench can be downloaded here. It includes:
118 |
119 | - Benchmark
120 | - BREW benchmark file (.ser file)
121 | - Benchmark Cases (.apk files)
122 | - Groundtruth as far as known (Expected results in AQL format: .xml)
123 | - Source Code (Decompiled .apks)
124 | - Results
125 | - BREW benchmark file including result-summary (data/data.ser)
126 | - AQL-Answers per app (data/storage/*.xml)
127 |
128 | #### Download
129 | - [DIALDroidBench](https://uni-paderborn.sciebo.de/s/ZmlRvtzI6pVYHVP/download?path=%2Fbenchmarks&files=DIALDroidBench.zip)
130 |
131 |
132 |
133 | ## Errata
134 | All benchmarks above which are based on DroidBench contain four tiny bugs (mislabeled).
135 |
136 | | Category | Benchmark Case | Wrong Label | Correct Label |
137 | | -------- | -------------- | ----------- | ------------- |
138 | | Aliasing | SimpleAliasing1 | Negative / Not-Expected Case | Positive / Expected Case |
139 | | UnreachableCode | UnreachableBoth | Positive / Expected Case | Negative / Not-Expected Case |
140 | | UnreachableCode | UnreachableSink1 | Positive / Expected Case | Negative / Not-Expected Case |
141 | | UnreachableCode | UnreachableSource1 | Positive / Expected Case | Negative / Not-Expected Case |
142 |
143 | Furthermore the results in the category Reflection were incorrectly reported.
144 | The filter included the category `Reflection_ICC` in the category `Reflection` - a simple (sub-)string matching mistake.
145 | (The results for most benchmarks and all tools above will be re-evaluated and published here asap - still might take a while.)
146 |
147 | ### DroidBench 3.0 (updated) + TaintBench
148 | Here you find an updated version of the [DroidBench 3.0](http://FoelliX.de/downloads/benchmarks/data_droidbench30.zip) benchmark ([DroidBench website](https://github.com/secure-software-engineering/DroidBench/tree/develop)) and the new [TaintBench](http://FoelliX.de/downloads/benchmarks/data_taintbench10.zip) benchmark ([TaintBench website](https://TaintBench.github.io/)) - to open you need [BREW version 2.0.0](https://github.com/FoelliX/BREW/releases) or newer.
149 | This are the two benchmarks we recommend to use for your Android taint analysis tool evaluation.
150 |
151 | **Download**
152 | - [DroidBench 3.0](http://FoelliX.de/downloads/benchmarks/data_droidbench30.zip)
153 | - [TaintBench](http://FoelliX.de/downloads/benchmarks/data_taintbench10.zip)
154 | - [BREW](https://github.com/FoelliX/BREW/releases)
155 |
156 |
157 |
158 | ## Publications
159 | - *Do Android Taint Analysis Tools Keep Their Promises?* (Felix Pauck, Eric Bodden, Heike Wehrheim)
160 | ESEC/FSE 2018 [https://dl.acm.org/citation.cfm?id=3236029](https://dl.acm.org/citation.cfm?id=3236029)
161 | - *Together Strong: Cooperative Android App Analysis* (Felix Pauck, Heike Wehrheim)
162 | ESEC/FSE 2019 [https://dl.acm.org/citation.cfm?id=3338915](https://dl.acm.org/citation.cfm?id=3338915)
163 | - *TaintBench: Automatic real-world malware benchmarking of Android taint analyses* (Linghui Luo, Felix Pauck, ...)
164 | EMSE 2022 [https://link.springer.com/article/10.1007%2Fs10664-021-10013-5](https://link.springer.com/article/10.1007%2Fs10664-021-10013-5)
165 |
166 | # Contact
167 | **Felix Pauck** (FoelliX)
168 | Paderborn University
169 | fpauck@mail.uni-paderborn.de
170 | [http://www.FelixPauck.de](http://www.FelixPauck.de)
--------------------------------------------------------------------------------
/_config.yml:
--------------------------------------------------------------------------------
1 | theme: jekyll-theme-slate
2 | title: ReproDroid
3 | description: Android Benchmark Reproduction Framework
4 | show_downloads: false
5 | logo: https://foellix.github.io/ReproDroid/logo.png
--------------------------------------------------------------------------------
/config_toolset1.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 | /path/to/Android/platforms
4 | 4
5 |
6 |
7 | 1
8 | /not/important
9 |
10 | /also/not/important
11 | /guess/what
12 | 0
13 | 2
14 |
15 | IntraAppFlows
16 |
17 |
18 |
19 |
20 | 1
21 | /not/important
22 |
23 | /also/not/important
24 | /guess/what
25 | 0
26 | 2
27 |
28 | COMBINE
29 |
30 |
31 |
--------------------------------------------------------------------------------
/config_toolset2.xml:
--------------------------------------------------------------------------------
1 |
2 |
3 | /path/to/Android/platforms
4 | 4
5 |
6 |
7 | 1
8 | /not/important
9 |
10 | /also/not/important
11 | /guess/what
12 | 0
13 | 2
14 |
15 | IntraAppFlows, InterAppFlows
16 |
17 |
18 |
--------------------------------------------------------------------------------
/logo.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/FoelliX/ReproDroid/88374106e6b11169df87accf4f21bba12e60a362/logo.png
--------------------------------------------------------------------------------
/overview.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/FoelliX/ReproDroid/88374106e6b11169df87accf4f21bba12e60a362/overview.png
--------------------------------------------------------------------------------