├── .gitignore ├── BUILDING ├── HACKING ├── HISTORY.md ├── INSTALLABLES ├── PLANNING ├── README.md ├── TIMEFRAME ├── mascot_konqi-dev-qt.png └── shashlik-structure.svg /.gitignore: -------------------------------------------------------------------------------- 1 | *.kdev4 2 | -------------------------------------------------------------------------------- /BUILDING: -------------------------------------------------------------------------------- 1 | This file describes how to build Shashlik, the simulated Android environment. 2 | 3 | 4 | 5 | Getting the Sources 6 | =================== 7 | 8 | Shashlik uses Dalvik, the Android runtime interpreter, to run the 9 | applications. In addition to Dalvik itself we need to have a number of 10 | dependencies from the Google Android source tree to be able to build 11 | it. 12 | 13 | Dalvik is normally built inside the full Android source tree and we want to 14 | minimize the dependency tree when building on a normal Linux system. Here is 15 | the list of sources that are necessary to build Shashlik. For each component 16 | below you should be able to just cut and paste the command lines into a shell. 17 | 18 | The source trees should be placed next to each other in a common subdirectory, 19 | e.g. $HOME/SHASHLIK or something similar. In the rest of this file we will 20 | assume that you are using this location. If not, you will have to adjust the 21 | instructions below accordingly. 22 | 23 | cd 24 | mkdir SHASHLIK 25 | cd SHASHLIK 26 | 27 | Create the main directory for the sources. We recommend using a separate 28 | directory for the source tree because you will also have the install directory 29 | at this level (see below under Building the sources). 30 | 31 | mkdir src 32 | cd src 33 | 34 | Then proceed to download the git and svn repositories as follows. 35 | 36 | 37 | 1. The 9 repositories from the Shashlik project on github. 38 | 39 | Shashlik has 8 repos from the Google Android sources copied into Github. These 40 | 8 repos are Dalvik, Android Core, Android Libcore, Frameworks Native, Frameworks Base, 41 | libnativehelper, telephony and skia. 42 | The changes in these repositories will be kept as small as possible so that it will 43 | be easier to upgrade to a later version later. The main code written for Shashlik 44 | is created in the 9th repository of the project named (surprise!) shaslik. 45 | 46 | The main page of the project is https://github.com/shashlik. Here you will 47 | find links to all 9 repositories. Since the shashlik repos are private you 48 | will need to give your username and password to clone them. 49 | 50 | git clone https://github.com/shashlik/shashlik.git 51 | git clone --branch shashlik-kitkat https://github.com/shashlik/dalvik.git 52 | git clone --branch shashlik-kitkat https://github.com/shashlik/android-core.git 53 | git clone --branch shashlik-kitkat https://github.com/shashlik/android-libcore.git 54 | git clone --branch shashlik-kitkat https://github.com/shashlik/android-frameworks-native.git 55 | git clone --branch shashlik-kitkat https://github.com/shashlik/android-frameworks-base.git 56 | git clone --branch shashlik-kitkat https://github.com/shashlik/android-libnativehelper.git 57 | git clone --branch shashlik-kitkat https://github.com/shashlik/android-telephony.git 58 | git clone --branch shashlik-kitkat https://github.com/shashlik/android-skia.git 59 | 60 | 2. safe-iop 61 | 62 | Safe IOP is also a Google library which is described like this on the project 63 | home page at https://code.google.com/p/safe-iop/: 64 | 65 | "This library provides a collection of (macro-based) functions for 66 | performing safe integer operations across platform and architecture with a 67 | straightforward API." 68 | 69 | This library is kept in subversion. To check it out, do the following: 70 | 71 | svn checkout http://safe-iop.googlecode.com/svn/trunk/ safe-iop 72 | cd safe-iop 73 | svn switch http://safe-iop.googlecode.com/svn/tags/r0.3.1 74 | cd .. 75 | 76 | We will use a specific release, in this case 0.3.1, to get a stable platform. 77 | This is the reason for the svn switch. 78 | 79 | 80 | 3. bionic 81 | 82 | Bionic is a derivation of the BSD's standard C library code that was 83 | originally developed by Google for their Android[2] operating system 84 | (Wikipedia). It is a stripped-down version of libc that contains exactly what 85 | is needed on Android. It is necessary to build Dalvik, although there is a 86 | project to port Dalvik to GNU libc. 87 | 88 | There is a project called dvk (https://code.google.com/p/dvk/) whose goal is 89 | to port dalvik to other CPU architectures and away from needing bionic but we 90 | have not gone down that route. 91 | 92 | git clone https://android.googlesource.com/platform/bionic 93 | 94 | We use the kitkat tag for this one, meaning you need to run: 95 | 96 | cd bionic 97 | git checkout android-4.4.4_r2.0.1 98 | cd .. 99 | 100 | which will leave you with a detached head 101 | 102 | 103 | 4. openssl 104 | 105 | openssl is the premiere secure socket layer library used by free software projects, 106 | and android has made certain modifications to it which makes it difficult to simply 107 | use the one provided by most linux distributions. So, we use theit modified version. 108 | 109 | git clone https://android.googlesource.com/platform/external/openssl 110 | cd openssl 111 | git checkout android-4.4.4_r2.0.1 112 | cd .. 113 | 114 | 5. ext libraries 115 | 116 | A set of java libraries used by android and packed together into one jar. 117 | 118 | git clone https://android.googlesource.com/platform/external/nist-sip 119 | git clone https://android.googlesource.com/platform/external/apache-http 120 | git clone https://android.googlesource.com/platform/external/tagsoup 121 | git clone https://android.googlesource.com/platform/external/libphonenumber 122 | git clone https://android.googlesource.com/platform/external/okhttp 123 | git clone https://android.googlesource.com/platform/external/bouncycastle 124 | 125 | for all of the above: 126 | cd nist-sip 127 | git checkout android-4.4.4_r2.0.1 128 | cd .. 129 | cd apache-http 130 | git checkout android-4.4.4_r2.0.1 131 | cd .. 132 | cd tagsoup 133 | git checkout android-4.4.4_r2.0.1 134 | cd .. 135 | cd libphonenumber 136 | git checkout android-4.4.4_r2.0.1 137 | cd .. 138 | cd okhttp 139 | git checkout android-4.4.4_r2.0.1 140 | cd .. 141 | cd bouncycastle 142 | git checkout android-4.4.4_r2.0.1 143 | cd .. 144 | 145 | 6. jni requirements 146 | 147 | git clone https://android.googlesource.com/platform/frameworks/opt/net/voip 148 | git clone https://android.googlesource.com/platform/external/liblzf 149 | git clone https://android.googlesource.com/platform/external/protobuf 150 | git clone https://android.googlesource.com/platform/external/sqlite 151 | git clone https://android.googlesource.com/platform/frameworks/opt/emoji android-emoji 152 | git clone https://android.googlesource.com/platform/build android-build 153 | cd voip 154 | git checkout android-4.4.4_r2.0.1 155 | cd .. 156 | cd liblzf 157 | git checkout android-4.4.4_r2.0.1 158 | cd .. 159 | cd protobuf 160 | git checkout android-4.4.4_r2.0.1 161 | cd .. 162 | cd sqlite 163 | git checkout android-4.4.4_r2.0.1 164 | cd .. 165 | cd android-emoji 166 | git checkout android-4.4.4_r2.0.1 167 | cd .. 168 | cd android-build 169 | git checkout android-4.4.4_r2.0.1 170 | cd .. 171 | 172 | We also require the Android 5.1 version of libhardware and master head of android 173 | core (for libbase) 174 | git clone https://android.googlesource.com/platform/hardware/libhardware 175 | git clone https://android.googlesource.com/platform/system/core core-head 176 | cd libhardware 177 | git checkout android-5.1.0_r1 178 | cd .. 179 | 180 | The following libraries are not currently used, but will be required once RenderScript 181 | support is functional. 182 | git clone https://android.googlesource.com/platform/frameworks/compile/libbcc 183 | git clone https://android.googlesource.com/platform/frameworks/rs 184 | cd libbcc 185 | git checkout android-4.4.4_r2.0.1 186 | cd .. 187 | cd rs 188 | git checkout android-4.4.4_r2.0.1 189 | cd .. 190 | 191 | 7. Graphics assistance libraries 192 | 193 | These libraries are an attempt to get SurfaceFlinger to work on normal Linux, and so 194 | we piggyback on that work. Minor changes have been made to the drm_gralloc module, and 195 | until those are upstreamed or otherwise fixed, we keep a local clone of it. 196 | 197 | git clone https://android.googlesource.com/platform/hardware/libhardware_legacy 198 | git clone git://anongit.freedesktop.org/mesa/mesa 199 | git clone https://github.com/shashlik/drm_gralloc.git 200 | cd drm_gralloc 201 | git checkout lollipop-x86 202 | cd .. 203 | 204 | The following are from https://wiki.linaro.org/BenjaminGaignard/HWComposer_DRM and 205 | they are a part of an attempt to get things running on a full blood linux as well 206 | which of course aligns very well with our efforts. 207 | 208 | git clone git://git.linaro.org/people/benjamin.gaignard/gralloc.git 209 | git clone https://github.com/shashlik/hwcomposer-linaro.git hwcomposer 210 | 211 | 8. Gentroid 212 | 213 | This code is a clone and modification of the gentroid code, used in the leinir2 branch 214 | to build jni 215 | 216 | git clone https://github.com/shashlik/gentroid-frameworks-base.git 217 | 218 | 9. JarJar 219 | 220 | This is a tool used for doing modifications to jar files, such as relocating packages 221 | to other sub-domains (for example, org.conscrypt.* becomes com.android.conscrypt...). 222 | 223 | Leave this file in the base source directory so the build system can find it. As google 224 | code is shutting down, this may need to relocate at some point (other mirrors exist, 225 | but this is the canonical release address for the file) 226 | 227 | wget https://jarjar.googlecode.com/files/jarjar-1.4.jar 228 | 229 | Building the Sources 230 | ==================== 231 | 232 | The goal of Shashlik is to create a stand-alone Android runtime 233 | environment. This basically means Dalvik with a number of support features 234 | such as emulated sensors and a screen. 235 | 236 | Dalvik itself needs a number of support classes and libraries from other parts 237 | of the Android sources. Unfortunately the dependencies themselves drag in more 238 | dependencies and so on until most of Android is present. It turns out that 239 | Dalvik itself does not use that many classes and functions in the dependency 240 | libraries so by selecting to build only those classes that are actually needed 241 | we can keep down the number of repositories that we need to check out. 242 | 243 | In this description we assume that you have downloaded the sources above and 244 | that they are all in $HOME/shashlik. 245 | 246 | At this time the instructions assume that you are building on a standard Linux 247 | desktop system with an installed software development kit. Refer to your 248 | distribution on how to install this. As far as we know there are no 249 | distribution specific problems or peculiarities. 250 | 251 | 252 | Prerequisites 253 | ------------- 254 | 255 | Aside from using a normal Linux development system (make, gcc, etc) you need 256 | a variety of components that might not be on everybodys computers: 257 | - CMake, version 2.8.8 minimum 258 | - Qt 5.3 259 | - KDE Frameworks 5 (KF5), specifically the following modules: 260 | I18n WindowSystem Service Package Declarative Plasma PlasmaQuick Wayland 261 | - Kernel headers 262 | - Java SDK version 6 (or newer, we're just forcing target/source) 263 | - flex and bison 264 | 265 | - 32 bit development headers for all the following libraries, used by the android VM 266 | Fontconfig 267 | Freetype 268 | giflib 269 | libjpeg62 270 | libwebp 271 | libpng - version 1.2 272 | harfbuzz 273 | glu-devel 274 | libcap 275 | zlib-devel 276 | ffi 277 | icuuc 278 | icui18n 279 | expat 280 | crypto 281 | openssl 282 | Qt5Core 283 | GLESv2 284 | GLESv1_CM 285 | selinux 286 | Wayland EGL 287 | Qt 5.3 modules: Core, Gui, Widgets, X11Extras 288 | Note! All the libraries above must be 32bit versions, as they are used by the 289 | (always) 32 bit android runtime 290 | 291 | for openSuse, you can run the following commands to get what you need: 292 | sudo zypper install -t pattern devel_c_cpp devel_java devel_qt5 devel_kde_frameworks 293 | sudo zypper install libharfbuzz0-32bit libharfbuzz-icu0-32bit glibc-devel-32bit \ 294 | libstdc++-devel-32bit gcc-32bit gcc-c++-32bit libwebp-devel libwebpdecoder1-32bit \ 295 | libwebpdemux1-32bit libwebpmux1-32bit fontconfig-devel-32bit zlib-devel-32bit \ 296 | freetype2-devel-32bit glu-devel-32bit giflib-devel-32bit libpng12-devel-32bit \ 297 | libpng12-compat-devel-32bit libjpeg62-32bit libjpeg62-devel-32bit \ 298 | libicu-devel-32bit libdrm-devel-32bit libcap-devel \ 299 | libQt5Core-devel-32bit libffi48-devel-32bit libexpat-devel-32bit libopenssl-devel-32bit \ 300 | java-1_7_0-openjdk-devel jarjar libselinux-devel \ 301 | libQt5Widgets-devel-32bit libqt5-qtx11extras-devel-32bit \ 302 | libwayland-egl-devel-32bit kwayland-devel-32bit 303 | 304 | This instruction will from here on assume that you have these installed 305 | correctly. If not, you will have to install them (probably build yourself) 306 | using their respective installation instructions. 307 | 308 | NOTE: Even after installing Qt5.3 and KF5 correctly for normal usage you will 309 | still have to add a specific environment variable to be able to use them 310 | for development using CMake: 311 | 312 | export CMAKE_PREFIX_PATH=$KF5:$CMAKE_PREFIX_PATH 313 | 314 | We also suggest to create an environment variable with the path of the 315 | Shashlik top directory. 316 | 317 | export SHASHLIK=$HOME/SHASHLIK 318 | 319 | In the rest of this document we shall use this variable in our examples. 320 | 321 | 322 | Branches 323 | -------- 324 | 325 | We are trying to minimize the changes in each component as much as 326 | possible. However some changes are necessary and cannot be avoided. So for 327 | each dependency that does need to be changed, we have created a branch called 328 | shashlik-kitkat. 329 | 330 | Some of the components have release tags that we are using so that we can have 331 | a stable build. Here is a list of the branches and tags that we are using. If 332 | you have checked out the repositories using the commands given above they 333 | should be on the correct branch already. 334 | 335 | * shashlik: branch master 336 | * dalvik: branch shashlik-kitkat 337 | * android-core: branch shashlik-kitkat 338 | * android-libcore: branch shashlik-kitkat 339 | * android-frameworks-native: branch shashlik-kitkat 340 | * android-frameworks-base: branch shashlik-kitkat 341 | 342 | * frameworks-native branch kitkat-release 343 | * safe-iop: tag http://safe-iop.googlecode.com/svn/tags/r0.3.1 344 | * libnativehelper: branch kitkat-release 345 | * bionic: tag android-4.4.4_r2.0.1 346 | * skia: tag android-4.4.4_r2.0.1 347 | * openssl: tag android-4.4.4_r2.0.1 348 | 349 | As you can see above, the repo shashlik itself is the one where we try to put 350 | all of our changes so that should be on master branch unless you are working 351 | on some new feature in a feature branch. 352 | 353 | 354 | Building 355 | -------- 356 | 357 | Building Shashlik is slightly more complicated than building a normal source 358 | tree. This is because we are taking sources from the Android source tree and 359 | build them in a way that they are not designed for. 360 | 361 | You can install anywhere you like, basically, but we highly recommend that you 362 | do so by creating a SHASHLIK/install directory. Note that the install directory 363 | should be _outside_ the shashlik repo source tree but under the SHASHLIK top 364 | directory. 365 | 366 | cd $SHASHLIK 367 | mkdir install 368 | 369 | For building, the procedure is in itself similar to how you normally compile 370 | any other cmake based project, however the directory structure is vital. The 371 | build system expects the location of the sources to be as suggested in Getting 372 | The Sources above. In other words, you should have a directory structure 373 | similar to the following tree. Those marked with * are directories you should 374 | be creating, all the others are git clones (and that one svn checkout): 375 | 376 | + SHASHLIK * 377 | + install * 378 | + src * 379 | + android-core 380 | + android-frameworks-native 381 | + android-frameworks-base 382 | + android-libcore 383 | + bionic 384 | + dalvik 385 | + libnativehelper 386 | + safe-iop 387 | + (etc, other checkouts) 388 | + shashlik 389 | + build * 390 | 391 | 392 | Lastly you will have to build the shashlik sources that depend on the selected 393 | Android libraries that were created in the builddeps directory. This is done 394 | in the $SHASHLIK/src/shashlik/build/ directory and it is really easy: 395 | 396 | cd $SHASHLIK/src/shashlik/build 397 | cmake .. -DCMAKE_INSTALL_PREFIX=$SHASHLIK/install 398 | make install 399 | 400 | 401 | The main control for the shashlik android runtime is shashlik-controller, the 402 | only binary installed outside of the android root. Run this command with --help 403 | to see what options you've got. This binary currently makes assumptions regarding 404 | the setup of your system. These assumptions will be removed at a later date. 405 | 406 | To run an application, run as follows: 407 | shashlik-controller --start 408 | shashlik-controller --launchapk=/path/to/apk-file.apk 409 | -------------------------------------------------------------------------------- /HACKING: -------------------------------------------------------------------------------- 1 | Shashlik project standards 2 | 3 | 4 | This file describes the coding and workflow standards of the Shashlik 5 | project. We want to keep formality at a minimum but we still need some common 6 | standards to keep things from coming into chaos 7 | 8 | 9 | General 10 | ------- 11 | 12 | The project aims at creating a runtime environment for Android applications 13 | packaged into APK packages. Our first target environment is going to be 14 | standard desktop Linux, both 32 and 64 bit systems. To do this, we are using 15 | the following resources: 16 | 17 | 1. System libraries. These should be used as much as possible. 18 | 19 | 2. Imported sources from the Android source tree. These will comprise tools 20 | such as the Dalvik java runtime and libraries such as libdex to handle dex 21 | files. 22 | 23 | 3. Code that we write ourselves. This should only be done if absolutely 24 | necessary. 25 | 26 | The organization of the project is that we use as few Android repos as 27 | possible and use as few libraries from the ones that we do choose as 28 | possible. We also try to use them as much as is and only make minimal changes 29 | to them. 30 | 31 | There are a number of repos from Android that we use totally as is. Then there 32 | are a few repos that we import into this project and actually have to make 33 | changes to. At the time of writing, these repos are: 34 | - dalvik 35 | - android-libcore 36 | - android-core 37 | 38 | These repos have been imported into github and should be checked out from 39 | github together with the shashlik repo. 40 | 41 | For all unchanged repos we use specific tags. For all repos with changes (the 42 | 3 above) we create a branch with the name pattern shashlik- 43 | where is the name of a specific Android release that we base 44 | our runtime environment on. At the time of the writing, this is 45 | shashlik-kitkat. 46 | 47 | For building the sources, we use CMake and it's all done inside shashlik 48 | itself. For details about how to build the sources, see the instructions in 49 | the file BUILDING. 50 | 51 | 52 | Coding standards 53 | ---------------- 54 | 55 | For code that we write ourselves we are generally following two well 56 | established standards: 57 | 58 | * The kdelibs coding standards for C++: 59 | https://techbase.kde.org/Policies/Kdelibs_Coding_Style 60 | 61 | * The plasma coding standards for QML: 62 | https://community.kde.org/Plasma/QMLStyle 63 | 64 | When it comes to filenames and classes, the filenames should be the same as 65 | the class name. So the class FooBar should be in FooBar.h and FooBar.cpp (and 66 | possibly also FooBar_p.h for the private class if there is one and it's used 67 | by other classes too). Only have one class per file. 68 | 69 | Dialogs should be constructed using Qt Designer. 70 | 71 | Spacing is good. Keep the sources light and not too condensed 72 | (From the Krita coding standards). 73 | 74 | 75 | Workflow 76 | -------- 77 | 78 | Our workflow can be summarized in a few simple points: 79 | 80 | * New features and other significant changes should be developed in a git 81 | branch. The name of the branch should follow the naming scheme 82 | part-feature-developer (taken from the Calligra standards). Example: 83 | tools-launcher-ingwa for a branch that implements or enhances the launcher 84 | in the tools part of the sources (at the time of the writing there is no 85 | such area, but it illustrates the point). 86 | 87 | * Before merging into master all branches should be reviewed. The same goes 88 | for smaller changes if they are deep in the libraries. 89 | 90 | * Branches should be merged with the following command line: 91 | 92 | git checkout master 93 | git merge --squash 94 | 95 | This way, the branch will be merged as one commit and we will not have 96 | unbuildable nodes along the master branch. 97 | 98 | Workflow specific to releases is still TBD. 99 | -------------------------------------------------------------------------------- /HISTORY.md: -------------------------------------------------------------------------------- 1 | 13.7.2014 - Inge Wallin 2 | 3 | 4 | We currently have: 5 | 6 | * the complete dalvik vm (all dependencies it has from android ported to vanilla linux) 7 | * pixelflinger 8 | * properties server 9 | * complete documentation on the build system and how to make that go 10 | 11 | 12 | (6:32:00 PM) ingwa2_: What to expect from the updates would be a more or less complete dalvik bytecode interpreter. 13 | (6:32:22 PM) ingwa2_: There are 9 parts of it, of which 2 are building in the current checked-in sources. 14 | 15 | 16 | 17 | ---- 18 | 19 | 11.7.2014 - Aaron Seigo 20 | 21 | Currently the entire dalvik system builds including the tools. What we are 22 | left with is a complete system with headers, libraries and binaries installed 23 | to the host system. 24 | 25 | Left to merge is the settings kded5 plugin and building the java-based tools. 26 | This will be in by tomorrow. 27 | 28 | Some notes to help understand progress a bit: 29 | 30 | * some of the commits represent multiple experiments at doing it in different 31 | ways. two examples of this are: 32 | 33 | * the bionic-included memory allocator. i tried including the files from 34 | bionic directly into the vm library by adding them to the dalvik repository. 35 | this worked, but as bionic is also moving around my concern is this would lead 36 | to de-synchronization between our build and the bugfixes and performance 37 | improvements that go into bionic. so then i did an out-of-tree build that 38 | depends on bionic (which is what it does now, in fact). there were two 39 | variants on that as well: one where the configuration of the allocator was done 40 | in the CMake build system and one where the configuration was done in the 41 | dalvik VM's DlMalloc.cpp file directly. the last is the variant chosen 42 | 43 | * assembler. this was pretty annoying as there were three options: a) 44 | generate assembler from the stubs myself and patch that into the build; b) use 45 | the pre-included assembler files with modifications where needed; c) use the 46 | generic C implementation. i actually went with the latter until i realized 47 | after futzing with it that this was not going to be any easier that pulling in 48 | the assembler directly. so then i had to figure out how, in the Android 49 | environment, the assembler for a given platform is actually pulled in (which 50 | is when i realized there was both an (a) and a (b) option ... *sigh*), and 51 | eventually settled on c. 52 | 53 | As the documentation is almost always the code itself, some of these decisions 54 | took brute-force experimentation and reading through the sources while 55 | consulting mailing list archives for hints and ideas. 56 | 57 | 58 | ---- 59 | 60 | 15.5.2014 61 | 62 | 63 | I have the dalvik stack building and have been working on the runtime bits 64 | that the applications running on top of dalvik assume are there. Some of these 65 | components were actually implemented in the Linux kernel itself for what seems 66 | to be two reasons: 67 | 68 | * performance on very limited hardware 69 | * security (you can more easily guarantee certain things when running in the 70 | kernel) 71 | 72 | It isn't realistic to expect people to patch their kernel to become more 73 | Android like, if that is even possible (which it wouldn't be in many 74 | situations .. patches would be kernel version specific, etc) so I've been 75 | pushing forward with the userspace replacements as planned. Fortunately dalvik 76 | itself has hooks to run inside an emulated environment so Android app devs can 77 | work on "real" operating systems and hardware to write their apps and then 78 | deploy for testing once the application basically works. I'm taking advantage 79 | of those hooks as they are exactly what we need :) 80 | 81 | 82 | ---- 83 | 84 | 25.4.2014 85 | 86 | 87 | == Planning and research 88 | 89 | I identified which repositories will require patches that we carry downstream and got that number down to just three. While this took some time to complete, it has allowed me to work straight ahead since without interruptions or surprises. 90 | 91 | 92 | == Build System 93 | 94 | The out-of-tree buildsystem approach is working very well. It has gone through 95 | a few revisions, and there are still some "cleanliness" things I'd like 96 | undertake over time, but for now it is allowing me to cherry pick the parts we 97 | need for Dalvik and port only what is absolutely required (right down to the 98 | file level: some libraries are currently built without some API they would have 99 | on Android). 100 | 101 | I'm keeping the documentation up to date on how to build things, and that will 102 | expand again in detail as I get closer to finished (no point in documenting 103 | things that are still moving around). 104 | 105 | 106 | == Build progress 107 | 108 | All the pieces outside of Dalvik are now building as required and I am 109 | currently working on the core Dalvik virtual machine. It nearly compiles 110 | already, in fact. The last bits are some things it relies on from bionic 111 | (Android's libc) which I started porting over yesterday. (Bionic is not an 112 | acceptable dependency as it replaces libc!) This will be finished on Monday and 113 | then the vm should build and link successfully. 114 | 115 | The remainder of the pieces in the dalvik repository should follow quickly as 116 | they are tiny (mostly helpers and dev tools) which means that before the end 117 | of this month I will have reached the first milestone, as intended: produce a 118 | Dalvik that builds on vanilla Linux on x86. 119 | 120 | 121 | == Next month 122 | 123 | Next month's [May] work is to produce the necessary runtime components. That 124 | includes: 125 | 126 | * build of the java components 127 | * write a properties server (this is provided by the android kernel and is 128 | entirely non-standard, but a user-space server is supported if it is built 129 | with the right options, which is what the build system currently does) 130 | * write a launch service (this is a long-running daemon that bootstraps the vm 131 | and then launches apk's on request) 132 | * use the above to get an apk running! 133 | 134 | The two daemons will end up in the shashlik repository in a tier3 build 135 | directory (next to tier1 and tier2 which are used to build the Dalvik runtime) 136 | 137 | I'm not 100% sure how much work the above will be, though I know it will fit 138 | within the next month's hours. I might have time next month to already start 139 | on the Plasma integration, but I can't yet promise that. 140 | 141 | 142 | ---- 143 | 144 | 23.4.2014 145 | 146 | Just another quick update: all the legwork I did up until now is finally paying 147 | off. I'm already into building dalvik itself. There was more porting work than 148 | I hoped (lots of things assumed components from android, like their custom 149 | libc (bionic) in subtle ways), but I'm getting there and have broken through a 150 | few important barriers and am on to parts of the dalvik runtime itself. 151 | 152 | 153 | ---- 154 | 155 | 18.4.2014 156 | 157 | lots of initial work to get to the starting point. 158 | 159 | I have not synced anything to the git repo because I will actually need at 160 | least 3 more repos, possibly more. These will include: 161 | 162 | * android-core 163 | * android-libcore 164 | * dalvik 165 | 166 | They all need (and are getting) patches, and the only sane way to track 167 | upstream and keep those patches maintainable is to clone each of the 168 | repositories individually. 169 | 170 | There are two other hard dependencies which I may be able to avoid changing or 171 | even requiring after all the patching is in place: 172 | 173 | * libnativehelper 174 | * safe-iop 175 | 176 | The shashlik repository will still be needed, though: I am putting the build 177 | system in the shashlik git repo. This will give us a "meta build" that lives 178 | in the shashlik repo that will build all the necessary pieces of the above 179 | mentioned repositories. This keeps all those files out of the cloned repos, 180 | which means the cloned repos will *only* have the needed patches, but no new 181 | files due to the build system. .. it should be much easier to track changes 182 | over time that way. 183 | 184 | So the shashlik repo will end up with documentation and a set of build files 185 | which will require the other repos above checked out to actually build 186 | everything. (Perhaps more repos .. but currently I am managing to keep other 187 | depedencies out .. there is a LOT of old and odd cruft in these repositories, 188 | things from the early days of android and even things for hardware platforms 189 | and operating systems we do not need to support) 190 | 191 | 192 | 193 | ---- 194 | 195 | 9.4.2014 196 | 197 | Making pretty great progress at the moment on shashlik. I've been documenting 198 | the dependcies of dalvik which has sent me through the bowels of the android 199 | base system, but severing what bits of hard-coded ties are at that level seem 200 | pretty straight forward at this point. 201 | 202 | Some useful decision points have come and gone now, including which version of 203 | dalvik to focus on: the version that comes with kitkat. I don't want to be 204 | chasing their development in master, so will instead be working on their 205 | kitkat branch. 206 | 207 | 208 | The roadmap / gameplan is: 209 | 210 | * Enumerate components and dependencies 211 | 212 | There are 9 items on this list, and I've worked through the first two. Working 213 | my way up the stack should become easier as lower level items that are already 214 | ready will be re-used, so each successive entry should have less and less work 215 | involved. 216 | 217 | 218 | * Build components 219 | 220 | After enumerating a component set, I will move to this stage; once the 221 | component set builds, I will return to "Enumerate components and dependencies" 222 | to work on the next entry. 223 | 224 | This is where source code actually starts getting modified, and at this point I 225 | will import a fork of the relevant code base into a new git repository and 226 | start working on it there. It will be regularly rebased on the upstream repos 227 | to keep up with changes there. 228 | 229 | I have decided to start out by moving the build system to cmake. The reason 230 | for this is multiple: 231 | 232 | - The Android build system is based on the old autotools stuff which is just 233 | horrible 234 | - It means not having to touch their build system, so changes in it won't 235 | affect us 236 | - As new files appear in the repositories over time, it will be evident what 237 | needs review (and porting if needed) as it will be missing from the cmake 238 | build system 239 | - It will be much much simpler to maintain in the long run 240 | 241 | The builds are all quite simple looking. For the base "libdex" library in 242 | dalvik, which is the first item in the components list, it looks like I'll have 243 | to come up with just 5 cmake files, each of which makes a single library .. 244 | which is child's play :) 245 | 246 | Getting things to actually build and link will be a bit more work. There are 247 | areas where they assume they are running on an android system, for instance, 248 | and that will need some reimplementation of functions to work on a regular 249 | Linux system. Thought, to be honest, it's not as bad as I feared it might be. 250 | 251 | 252 | * Implement system integration points 253 | 254 | During the first stage enumeration I am also documenting system integration 255 | points I find. To get things building, things will mostly just get stubbed in. 256 | Once the libraries and necessary binaries are all building, I will work on 257 | implementing the useful system integration points. Some system integration 258 | points are simply not interesting, however (such as the hooks used to do 259 | system startup). 260 | 261 | Some hooks will be crucial, however, such as the hooks into surfaceflinger to 262 | get openGL surfaces for the apps to render into, and those will take the bulk 263 | of the effort I think. 264 | 265 | There will also likely be areas like the handling of the "back" and "home" 266 | software keys which apps expect to work a specific way under Android. This list 267 | will grow and shrink as I move stepwise through the first two phases. above 268 | 269 | 270 | * Create launch daemon 271 | 272 | The way Dalvik works (or .. is expected to work anyways ..) is that there is a 273 | long-running instance of the VM that is used as a launcher. When an app 274 | starts, this launcher forks itself into another process (which Dalvik 275 | apparently is optimized for...) and starts the app. The launcher then manages 276 | the apps as needed. 277 | 278 | 279 | * Test against real world apps 280 | 281 | Once there is a launch daemon, then we can start testing against real world 282 | apps. I expect this to reveal what remaining system integration work needs to 283 | be done on a practical level. 284 | 285 | 286 | * Plasma workspace integration 287 | 288 | Once things are running somewhat usefully, integration with Plasma will come 289 | next. That includes form factor awareness, taskbar integration, application 290 | launcher integration, windowed modes, etc. Some of these things will be a bit 291 | "funny" for the Android apps as not all the concepts used on the various form 292 | factors Plasma supports map to the Android concept. This is why Google does 293 | not use Android for their Chrome Books, for instance. In any case, I've 294 | compiled a short list of targets for this as well. 295 | 296 | 297 | ---- 298 | 299 | 1.4.2014 300 | 301 | Shashlik Project started 302 | -------------------------------------------------------------------------------- /INSTALLABLES: -------------------------------------------------------------------------------- 1 | This is a list of what is built and installed from the Shashlik source code 2 | 3 | bin/ 4 | shashlik-launcher This is the runner of Android APK files. 5 | dexdump* Dumps the contents of dex files 6 | dexlist* ? 7 | hprof-conv* ? 8 | Only on non-64 bit systems: 9 | gdbjithelper ? 10 | dexopt ? 11 | dmtracedump ? 12 | 13 | lib/ 14 | libdvm.so The Dalvik Virtual Machine itself 15 | libapk.so Support library for handling APK files 16 | libdex.so Support library for handling DEX files 17 | libpixelflinger.so Graphic library 18 | 19 | Support libraries: 20 | libcorkscrew.so 21 | libnativehelper.so Support library for Dalvik classes 22 | liblog.so Logging 23 | libsafe-iop.so platform-independent integer operations 24 | libutils.so 25 | libcutils.so 26 | -------------------------------------------------------------------------------- /PLANNING: -------------------------------------------------------------------------------- 1 | ============================ 2 | Goal: Get Flappy Bird to run 3 | ============================ 4 | 5 | Inge Wallin 6 | 7 | Components 8 | + DalvikVM 9 | - core.jar, framework.jar, ext.jar 10 | + Zygote / Launcher 11 | - Preload Java classes 12 | + Installer? 13 | + ...? 14 | 15 | System Services 16 | + Settings 17 | + Surface flinger? 18 | + Window Manager? 19 | + Activity Manager? 20 | + ...? 21 | 22 | I/O 23 | + Screen OpenGL 24 | + touch Drag events from the window? 25 | 26 | 27 | Open Questions 28 | + How does an app get access to the resources in the APK? 29 | A: installed files in the file system 30 | - OR - provided by reading the APK directly. 31 | + How does an app paint on the window? 32 | A: The application asks for an OpenGL window, gets one (which should be 33 | embedded into a standard X/Wayland window and handled by the normal 34 | window manager (kwin). 35 | + How to communicate between services/apps when there is no kernel bridge? 36 | A: settings: Over a named pipe 37 | A: serviceflinger: protocol buffers 38 | + What do we need to do with the DEX before running it? 39 | A: not much(?) 40 | 41 | ---------------------------------------------------------------- 42 | 43 | Launcher 44 | + Replacement for Android's Zygote 45 | + start the VM 46 | + load the application into the VM 47 | + load deps of the application 48 | 49 | Surface Flinger 50 | + Create a window with an OpenGL canvas/context 51 | + ...? 52 | 53 | ---------------------------------------------------------------- 54 | 55 | Estimations are not definite since we are still learning a lot about how the 56 | Android sources are organized. But right now this is the best estimate. 57 | 58 | Development Activities 59 | + Create Launcher 60 | - Build the parts from zygote that will be reused STEP 1: 1 week (0.5 due July, 24th 2014) 61 | - Write porting code STEP 2: 2 weeks (0.6 due August, 7th 2014) 62 | . Start VM 63 | . Load system java classes 64 | . Load application provided classes 65 | . Load application 66 | + Write a Java Native Interface (JNI) STEP 3: 2 weeks (0.7 due August, 21st 2014) 67 | This includes the input from kbd and touch 68 | + Write "Surface Flinger" STEP 4: 1 week (0.8 due August, 28th 2014) 69 | 70 | 71 | ================================================================ 72 | Goal: Flappy bird flies 6 weeks (0.8 due August, 28th 2014) 73 | ================================================================ 74 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | shashlik 2 | ======== 3 | 4 | Android Simulated Environment 5 | 6 | The roadmap / plan is: 7 | 8 | * Enumerate components and dependencies 9 | 10 | There are 9 items on this list, and I've worked through the first two. Working 11 | my way up the stack should become easier as lower level items that are already 12 | ready will be re-used, so each successive entry should have less and less work 13 | involved. 14 | 15 | 16 | * Build components 17 | 18 | After enumerating a component set, I will move to this stage; once the 19 | component set builds, I will return to "Enumerate components and dependencies" 20 | to work on the next entry. 21 | 22 | This is where source code actually starts getting modified, and at this point I 23 | will import a fork of the relevant code base into a new git repository and 24 | start working on it there. It will be regularly rebased on the upstream repos 25 | to keep up with changes there. 26 | 27 | I have decided to start out by moving the build system to cmake. The reason 28 | for this is multiple: 29 | 30 | - The Android build system is based on the old autotools stuff which is just 31 | horrible 32 | - It means not having to touch their build system, so changes in it won't 33 | affect us 34 | - As new files appear in the repositories over time, it will be evident what 35 | needs review (and porting if needed) as it will be missing from the cmake 36 | build system 37 | - It will be much much simpler to maintain in the long run 38 | 39 | The builds are all quite simple looking. For the base "libdex" library in 40 | dalvik, which is the first item in the components list, it looks like I'll have 41 | to come up with just 5 cmake files, each of which makes a single library .. 42 | which is child's play :) 43 | 44 | Getting things to actually build and link will be a bit more work. There are 45 | areas where they assume they are running on an android system, for instance, 46 | and that will need some reimplementation of functions to work on a regular 47 | Linux system. Thought, to be honest, it's not as bad as I feared it might be. 48 | 49 | The code quality is not awesome in many places, but we already knew that 50 | about Android :/ 51 | 52 | 53 | * Implement system integration points 54 | 55 | During the first stage enumeration I am also documenting system integration 56 | points I find. To get things building, things will mostly just get stubbed in. 57 | Once the libraries and necessary binaries are all building, I will work on 58 | implementing the useful system integration points. Some system integration 59 | points are simply not interesting, however (such as the hooks used to do 60 | system startup). 61 | 62 | Some hooks will be crucial, however, such as the hooks into surfaceflinger to 63 | get openGL surfaces for the apps to render into, and those will take the bulk 64 | of the effort I think. 65 | 66 | There will also likely be areas like the handling of the "back" and "home" 67 | software keys which apps expect to work a specific way under Android. This list 68 | will grow and shrink as I move stepwise through the first two phases. above 69 | 70 | 71 | * Create launch daemon 72 | 73 | The way Dalvik works (or .. is expected to work anyways ..) is that there is a 74 | long-running instance of the VM that is used as a launcher. When an app 75 | starts, this launcher forks itself into another process (which Dalvik 76 | apparently is optimized for...) and starts the app. The launcher then manages 77 | the apps as needed. 78 | 79 | 80 | * Test against real world apps 81 | 82 | Once there is a launch daemon, then we can start testing against real world 83 | apps. I expect this to reveal what remaining system integration work needs to 84 | be done on a practical level. 85 | 86 | 87 | * Plasma workspace integration 88 | 89 | Once things are running somewhat usefully, integration with Plasma will come 90 | next. That includes form factor awareness, taskbar integration, application 91 | launcher integration, windowed modes, etc. Some of these things will be a bit 92 | "funny" for the Android apps as not all the concepts used on the various form 93 | factors Plasma supports map to the Android concept. This is why Google does 94 | not use Android for their Chrome Books, for instance. In any case, I've 95 | compiled a short list of targets for this as well. 96 | 97 | 98 | * Interesting Links: 99 | * http://www.gamasutra.com/blogs/RuthanielVandenNaar/20140612/219170/Current_of_Androidx86_PC_OS_interview_with_ChihWei_Huang.php 100 | -------------------------------------------------------------------------------- /TIMEFRAME: -------------------------------------------------------------------------------- 1 | ==================================================================== 2 | Short term goals and milestones 3 | ==================================================================== 4 | 5 | Dan Leinir 6 | 7 | These are the items that we are working on now and in the near future. 8 | Most of these are picked from the medium term goals below. The task 9 | list here is more detailed than below. 10 | 11 | Next milestone: M2: Run a dex file inside the VM 12 | 13 | 14 | Immediate stuff 15 | * Add the HAVE_ definitions to the appropriate (top?) level. Done 16 | 17 | 18 | Tasks to reach next milestone 19 | * Preparations 20 | + Use the AndroidRuntime class from frameworks-native Done 21 | instead of copying it 22 | + Create a binder library but without connections to ---- 23 | the kernel 24 | + Create the system .jar and .odex files (core, framework, ---- 25 | ext (at least, more may be required)) 26 | + Recreate dex-preopt as cmake command ---- 27 | (script requires full android build system, prereq for ^) 28 | 29 | * Check if the provided file is a correct APK ---- 30 | + Check for the manifest 31 | + Check the META-INF/MANIFEST.MF file 32 | + Check for assets/ 33 | + Check for classes.dex 34 | * Unpack the APK into a temp directory ---- 35 | * Get the VM started ---- 36 | * Load system java classes ---- 37 | * Load application provided dex ---- 38 | 39 | 40 | ==================================================================== 41 | Medium term goals and milestones 42 | ==================================================================== 43 | 44 | 45 | Build the components of Dalvik 46 | * vm library DONE 47 | * Binaries DONE 48 | + dexlist Done 49 | + dexopt (when using JIT) Done 50 | + dexdump Done 51 | * tools DONE 52 | 53 | New tools 54 | * properties server (see android-core/libcutils/properties.c) DONE 55 | 56 | 57 | 58 | Milestone 1: Basic launcher done (v 0.5) DONE 59 | -------------------------------------------------------------------- 60 | 61 | 62 | Handle APK file 63 | * Check if the provided file is a correct APK ---- 64 | * Unpack the APK into a temp directory ---- 65 | * Get the VM started ---- 66 | * Load system java classes ---- 67 | * Load application provided dex ---- 68 | 69 | 70 | Milestone 2: Run a dex file inside the VM (v 0.6) ---- 71 | -------------------------------------------------------------------- 72 | 73 | 74 | Write a Java Native Interface (JNI) 75 | * Screen output 76 | * Kbd Input 77 | * Touch screen input 78 | This includes the input from kbd and touch 79 | 80 | 81 | Milestone 3: Functional Java interface (JNI) (v 0.7) ---- 82 | -------------------------------------------------------------------- 83 | 84 | 85 | Write Surfaceflinger 86 | * Create a OpenGL workspace manager ---- 87 | 88 | 89 | Milestone 4: Flappy bird runs (v 0.8) ---- 90 | -------------------------------------------------------------------- 91 | 92 | 93 | ==================================================================== 94 | Long term goals and milestones 95 | ==================================================================== 96 | 97 | These are not yet settled. 98 | 99 | 100 | * Emulated sensors 101 | * Real Installer 102 | * JIT? 103 | * Rebase on Later Android versions? 104 | -------------------------------------------------------------------------------- /mascot_konqi-dev-qt.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/shashlik/old-docs/b6f028494610014573ee3f57835b7b7fce85a6ae/mascot_konqi-dev-qt.png -------------------------------------------------------------------------------- /shashlik-structure.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 19 | 21 | 28 | 34 | 35 | 42 | 48 | 49 | 56 | 62 | 63 | 70 | 76 | 77 | 78 | 100 | 102 | 103 | 105 | image/svg+xml 106 | 108 | 109 | 110 | 111 | 112 | 117 | 128 | 131 | Our code 143 | 149 | 152 | propertiesd 163 | libshashlikutils 174 | libapk 185 | hwcomposer.shashlik 196 | 197 | CMake build system 208 | 210 | Shashlik Launcher 221 | Shashlik JNI 232 | Shashlik Controller 243 | 244 | 245 | 251 | Used by CMake during build and install steps to generate the java frameworks 278 | 281 | Android system libraries 292 | CMake Build System 304 | 310 | System dependencies 321 | Android support libraries 332 | Android development tools 343 | Build system tools 354 | Android virtual machine 365 | Our libraries 376 | Shashlik Launcher 387 | Java Libraries 398 | 404 | 405 | 408 | Build .jar (Java 1.6) 419 | Android Java Libraries 431 | 437 | Generate java interface classes (aidl) 448 | Translate java bytecode to android bytecode (dx) 459 | During build: 470 | Post-install: 481 | Pre-optimise (dexopt) 492 | Build resource packages (aapt) 503 | 504 | 507 | Shashlik Launcher 519 | 525 | Launch 536 | APK? 547 | Unpack and install 558 | Instantiate Runtime 569 | Run? 580 | Class? 591 | Request launch 602 | 607 | 612 | 617 | 622 | 628 | 634 | JNI takes over 645 | 650 | 656 | 657 | 665 | 671 | 677 | 683 | 686 | Android code 698 | 701 | Bionic 712 | safe-iop 723 | libutils 734 | libcutils 745 | Skia 756 | libpixelfinger 767 | libnativehelper 778 | liblog 789 | libjavacore 800 | libdex 811 | libcorkscrew 822 | libbinder 833 | libandroidfw 844 | libjni 855 | Support Libraries 866 | 871 | etc... 882 | 883 | 886 | hprof-conv 897 | dexlist 908 | dexdump 919 | dmtracedump 930 | gdbjithelper 941 | Development tools 952 | 957 | 958 | 961 | dx 972 | aidl 983 | dexopt 994 | Build system tools 1005 | 1010 | aapt 1021 | 1022 | 1025 | Dalvik 1036 | java frameworks- bouncycastle- conscrypt- core- ext- framework- framework2- okhttp 1075 | Android Virtual Machine 1086 | 1091 | 1092 | 1098 | Android sources fetched separately and builtusing our CMake build system, allowing fortight control of all used functionalityand dependencies 1125 | We only build explicitly requiredbits, which a standard libc doesnot supply (e.g. strlcat, strlcpy) 1147 | Only the explicitly required partsare built, from the android specificbranch in the android external repository 1169 | Modified to bepossible to build withouta full Android systemunderneath. 1196 | 1202 | 1208 | 1214 | 1220 | 1226 | 1232 | 1238 | 1239 | Shashlik 1250 | anatomy of 1261 | the 1272 | 1275 | 1278 | Java Native Interface 1290 | 1296 | Initialise JNI environment 1307 | Start Virtual Machine 1318 | Load system frameworks 1329 | (≅ "boot android") 1340 | Locate main function in passed class 1351 | Run the main function 1362 | 1363 | 1369 | 1370 | 1376 | 1379 | Shashlik Controller 1391 | 1397 | Launch APK 1408 | 1414 | Launch Activity Manager 1425 | Start 1436 | Stop 1447 | Restart 1458 | 1461 | Zygote 1472 | Servicemanager 1483 | SurfaceFlinger 1494 | InstallD 1505 | 1506 | Shashlik Controller UI 1517 | Launch options 1528 | 1534 | 1540 | 1546 | 1552 | 1558 | 1564 | 1570 | 1576 | 1582 | 1588 | 1594 | 1600 | 1601 | 1607 | 1613 | 1619 | 1620 | 1621 | --------------------------------------------------------------------------------