├── apm.png ├── libs.png ├── types.png ├── jni_files.png ├── ndk_build.gif ├── structure.png └── README.md /apm.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mail2chromium/Android-Native-Development-For-WebRTC/HEAD/apm.png -------------------------------------------------------------------------------- /libs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mail2chromium/Android-Native-Development-For-WebRTC/HEAD/libs.png -------------------------------------------------------------------------------- /types.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mail2chromium/Android-Native-Development-For-WebRTC/HEAD/types.png -------------------------------------------------------------------------------- /jni_files.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mail2chromium/Android-Native-Development-For-WebRTC/HEAD/jni_files.png -------------------------------------------------------------------------------- /ndk_build.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mail2chromium/Android-Native-Development-For-WebRTC/HEAD/ndk_build.gif -------------------------------------------------------------------------------- /structure.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mail2chromium/Android-Native-Development-For-WebRTC/HEAD/structure.png -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # "WebRTC Native Stack is considered as a signal chain as close to the Hardware Abstraction Layer (HAL) as possible." 2 | 3 | ----- 4 | 5 | **Getting Started** 6 | 7 | ------ 8 | 9 | This repository involves a **[Step by Step Guide to Build Android App based on WebRTC Native Stack](https://github.com/mail2chromium/Android-Native-Development-For-WebRTC).** As we all know, Android Programs run into Dalvik Virtual Machine. Native Development tool (NDK) allows users to execute some of the program using native code languages such as `C/C++`. 10 | 11 | For *Compilation and Building the WebRTC Library for Android*, you should have to look into this refernce: 12 | 13 | - [Compile_WebRTC_Library_For_Android](https://github.com/mail2chromium/Compile_WebRTC_Library_For_Android) 14 | 15 | 16 | For *real-time Communication and AudioProcessing* in Android, I will recommend you to must visit these refernces: 17 | 18 | - [Android_Realtime_Communication_Using_WebRTC](https://github.com/mail2chromium/Android_Realtime_Communication_Using_WebRTC) 19 | - [Android-Audio-Processing-Using-WebRTC](https://github.com/mail2chromium/Android-Audio-Processing-Using-WebRTC) 20 | 21 | ------ 22 | 23 | ### Content of this Document 24 | 25 | ----- 26 | 27 | - [NDK (Native Development Kit)](#ndk) 28 | 29 | - [Why NDK?](#why-ndk) 30 | 31 | - [What is JNI?](#what-is-JNI) 32 | 33 | - [NDK and JNI Workaround](#ndk-and-JNI-Workaround) 34 | 35 | - [Installation and Configurations](#installation-and-configurations) 36 | 37 | - [WebRTC with Android](#webRTC-with-Android) 38 | 39 | - [Creating an Android Project](#creating-an-Android-Project) 40 | 41 | - [Importing WebRTC Source to Android Project](#importing-WebRTC-Source-to-Android-Project) 42 | 43 | - [Write a Java class that uses native code of C/C++](#write-a-Java-class-that-uses-native-code-of-C/C++) 44 | 45 | - [Compile the Java Program Apm.java & Generate the C/C++ Header File](#compile-the-Java-Program-Apm.java-&-Generate-the-C/C++-Header-File) 46 | 47 | - [WebRTC JNI APM Wrapper](#webRTC-JNI-APM-Wrapper) 48 | 49 | - [Android Make file](#android-make-file) 50 | 51 | - [Build Shared Library via Android NDK](#build-Shared-Library-via-Android-NDK) 52 | 53 | - [Run with gradle](#run-with-gradle) 54 | 55 | 56 | ------ 57 | 58 | ### NDK: 59 | 60 | ------ 61 | 62 | [NDK (Native Development Kit)](https://developer.android.com/ndk/guides) includes the following features such as; 63 | 64 | - Tool and build files needed to generate a native code base from `C/C++`. 65 | - Embed consistent native libraries in application packages files (.apk files) that can be deployed on Android devices. 66 | - Supports a number of native system header files and libraries for the Android platform. 67 | 68 | Here you can select & download the latest or lagecy NDK package for your development platform i.e. [NDK Download](https://developer.android.com/ndk/downloads) 69 | 70 | ------ 71 | 72 | ### Why NDK? 73 | 74 | ------ 75 | 76 | NDk finds its best use to squeeze extra performance out to device, to achieve low latency or run computationally intensive applications on android platform. Here are some other reasons why we use NDK in our development; 77 | 78 | - Easy to port libraries written in `C/C++` can be easily reused on other embedded platforms. 79 | - Code protection, because the java code of apk can be easily decompiled, and the `C/C++` library is difficult to reverse. 80 | - Call third-party `C/C++` libraries in the NDK, because most open source libraries are written in `C/C++` code, 81 | such as WebRTC, TarsosDSP, Ffmpeg, OpenCV, and a lot of other machine learning stuff etc. 82 | 83 | ------ 84 | 85 | ### What is JNI? 86 | 87 | ------ 88 | 89 | [JNI (Java Native Interface)](https://docs.oracle.com/javase/8/docs/technotes/guides/jni/index.html) provides several APIs to implement the communication between Java and other languages (mainly `C/C++`). 90 | 91 | Drawback: There is one of the major side effect of JNI is, *The Program is no longer Cross-Platform*. To make your program Cross-Platform, you must recompile the native language part in different system environments. The program is no longer absolutely secure, and *improper* use of native code may cause the entire program to crash. As a general rule, you should focus your local methods on a few classes. This reduces the coupling between `JAVA` and `C`. 92 | 93 |  In fact, Java and C/C++ are static type languages, the main reason that java and C cannot communicate is the problem of data types. JNI plays a role in data conversion. The corresponding relationship is shown in the following table: 94 | 95 | ![JNI Types and Data Sturctures](https://github.com/mail2chromium/Android-Native-Development-For-WebRTC/blob/master/types.png) 96 | 97 | ----- 98 | 99 | ### NDK and JNI Workaround: 100 |   101 | ----- 102 | 103 | To put it simply, NDK is an extension toolkit developed by JNI. It uses NDK to compile corresponding local [static or shared libraries](https://stackoverflow.com/questions/2649334/difference-between-static-and-shared-libraries) that can be run for different mobile devices (`ARM, x86 ...`). In order to improve the sorting efficiency and use `C/C ++` in android, here are different steps for this process as follows: 104 | 105 | **1.** SDK development (write java code, write code containing native keywords to start JNI)([Native Methods](http://journals.ecs.soton.ac.uk/java/tutorial/native/index.html)) . 106 | 107 | **2.** JNI development (according to the JNI coding specification, write a local interaction with java Code, the role is to talk about the `C/C++` data type conversion into Java can recognize). 108 | 109 | **3.** `C/C++` development (write business logic, or call the local API or library provided by the NDK to complete the development of specific functions). 110 | 111 | **4.** NDK compilation ( Write the .mk file, compile and debug, modify the .mk file, and optimize the compilation results for the specific platform (`ARM/X86`). 112 | 113 | **5.** SDK compilation and packaging. 114 | 115 | ----- 116 | 117 | ### Installation and Configurations 118 | 119 | ----- 120 | 121 | Here we start with; 122 | 123 | #### 1. Download Source Code: 124 | 125 | As we know that WebRTC implementation is done in `C/C++` languages. It needs to integrate with android application via Java Native Interface(JNI). WebRTC source available at [Github WebRTC Source Code](https://github.com/JumpingYang001/webrtc.git) as well as official WebRTC [download page](http://webrtc.github.io/webrtc-org/native-code/development/). 126 | 127 | #### 2. Setup Android NDK: 128 | 129 | To integrate WebRTC code via JNI, first I needs setup android NDK. Download NDK from [here](https://developer.android.com/ndk/downloads) and extract it. Main thing to consider here is, make sure NDK path is being setup in environment variables in windows i.e. [see](https://subscription.packtpub.com/book/application_development/9781849691505/1/ch01lvl1sec09/setting-up-an-android-ndk-development-environment-in-windows). 130 | 131 | To test whether ndk-build is working or not, check these commands in the terminal like; 132 | 133 | ``` 134 | ndk-stack 135 | ndk-build --help 136 | ``` 137 | 138 | **Example:** 139 | 140 | Before digging into WebRTC Native Stack, If you don't have a basic understanding of NDK development, 141 | I must recommend you to follow this tutorial i.e [Hello-JNI](https://github.com/android/ndk-samples/tree/master/hello-jni). 142 | 143 | ----- 144 | 145 | ### WebRTC with Android: 146 | 147 | ------ 148 | 149 | Following are the steps that I have followed to integrate WebRTC source code with my android application. 150 | 151 | #### 1- [Creating an Android Project](https://developer.android.com/training/basics/firstapp/creating-project) 152 | 153 | #### 2- Importing WebRTC Source to Android Project: 154 | 155 | I have a seperated directory called `webrtc` inside the jni directory and added WebRTC source code for it. WebRTC source code can be cloned from [github](https://github.com/JumpingYang001/webrtc.git) or download via official [download page](http://webrtc.github.io/webrtc-org/native-code/development/). 156 | 157 | ![Android WebRTC Project Structure](https://github.com/mail2chromium/Android-Native-Development-For-WebRTC/blob/master/structure.png) 158 | 159 | Make sure your project structure follow the same hierarchy as given above. 160 | 161 | ----- 162 | 163 | #### 3- Write a Java class that uses native code of `C/C++`: 164 | 165 | ------ 166 | 167 | To call JNI based C function I need to write JNI wrappers for them. I kept the written wrappers in com_webrtc_audioprocessing_Apm_ package. Read more about JNI from [here](https://docs.oracle.com/javase/8/docs/technotes/guides/jni/spec/jniTOC.html). 168 | Following is the JNI wrapper for WebRTC Audio processing. I have maintained and updated `Apm.java` in my Android Project as given here; 169 | 170 | ![Android WebRTC Project Structure](https://github.com/mail2chromium/Android-Native-Development-For-WebRTC/blob/master/apm.png) 171 | 172 | This file includes the all the native instance method, via keyword native which denotes that this method is implemented in another language. A native method does not contain a body. These native methods shall be found in the native library loaded. Here are the list of native methods inculded inside the `Apm.java` class; 173 | 174 | ``` 175 | // apm constructor 176 | private native boolean nativeCreateApmInstance(boolean aecExtendFilter, boolean speechIntelligibilityEnhance, boolean delayAgnostic, boolean beamforming, boolean nextGenerationAec, boolean experimentalNs, boolean experimentalAgc); 177 | private native void nativeFreeApmInstance(); 178 | private native int high_pass_filter_enable(boolean enable); 179 | 180 | // aec and aecm 181 | private native int aec_enable(boolean enable); 182 | private native int aec_set_suppression_level(int level); //[0, 1, 2] 183 | private native int aec_clock_drift_compensation_enable(boolean enable); 184 | private native int aecm_enable(boolean enable); 185 | private native int aecm_set_suppression_level(int level); //[0, 1, 2, 3, 4] 186 | private native int ns_enable(boolean enable); 187 | private native int ns_set_level(int level); // [0, 1, 2, 3] 188 | 189 | // agc 190 | private native int agc_enable(boolean enable); 191 | private native int agc_set_target_level_dbfs(int level); //[0,31] 192 | private native int agc_set_compression_gain_db(int gain); //[0,90] 193 | private native int agc_enable_limiter(boolean enable); 194 | private native int agc_set_analog_level_limits(int minimum, int maximum); // limit to [0, 65535] 195 | private native int agc_set_mode(int mode); // [0, 1, 2] 196 | private native int agc_set_stream_analog_level(int level); 197 | private native int agc_stream_analog_level(); 198 | 199 | // vad 200 | private native int vad_enable(boolean enable); 201 | private native int vad_set_likelihood(int likelihood); 202 | private native boolean vad_stream_has_voice(); 203 | 204 | // Capture and render audio stream 205 | private native int ProcessStream(short[] nearEnd, int offset); //Local data// 16K, 16Bits, 10ms 206 | private native int ProcessReverseStream(short[] farEnd, int offset); // Remote data // 16K, 16Bits, 10ms 207 | private native int set_stream_delay_ms(int delay); 208 | 209 | // resampling 210 | private native boolean SamplingInit(int inFreq, int outFreq, long num_channels); 211 | private native int SamplingReset(int inFreq, int outFreq, long num_channels); 212 | private native int SamplingResetIfNeeded(int inFreq, int outFreq, long num_channels); 213 | private native int SamplingPush(short[] samplesIn, long lengthIn, short[] samplesOut,long maxLen, long outLen); 214 | private native boolean SamplingDestroy(); 215 | 216 | ``` 217 | 218 | The static initializer invokes `System.loadLibrary()` to load the native library "webrtc_apms" (which contains all native methods) during the class loading. It will be mapped to " `libwebrtc_apms.dll` " in Windows; or " `libwebrtc_apms.so` " in (Unix/Mac-OS X). This library shall be included in **libs** directory automatically after you run this command in terminal i.e. `ndk-build`. The program will throw a UnsatisfiedLinkError if the library cannot be found in runtime. 219 | 220 | ----- 221 | 222 | #### 4- Compile the Java Program Apm.java & Generate the C/C++ Header File: 223 | 224 | ----- 225 | 226 | Starting from JDK 8, you should use "javac -h" to compile the Java program AND generate C/C++ header file called `com_webrtc_audioprocessing_Apm.h` as follows: 227 | 228 | ``` 229 | javac -h . Apm.java 230 | ``` 231 | 232 | The "-h dir" option generates C/C++ header and places it in the directory specified (in the above example, '.' for the current directory). 233 | 234 | Before JDK 8, you need to compile the Java program using javac and generate C/C++ header using a dedicated javah utility, as follows. The javah utility is no longer available in JDK 10. 235 | 236 | ``` 237 | javac Apm.java 238 | javah ApmJNI 239 | ``` 240 | 241 | Now place this header file (`com_webrtc_audioprocessing_Apm.h`) inside the following i.e. `src/main/jni/` 242 | 243 | ----- 244 | 245 | #### 5- WebRTC JNI APM Wrapper 246 | 247 | ----- 248 | 249 | By using those generated library files and header files I have written JNI based C functions to **preprocess and postprocess** the audio data. These JNI files prefixed with `com_webrtc_audioprocessing_Apm` which is the java package path of the JNI wrappers. 250 | 251 | ![Android WebRTC Project Structure](https://github.com/mail2chromium/Android-Native-Development-For-WebRTC/blob/master/jni_files.png) 252 | 253 | Here is the JNI based audio processing module which is [android_apm_wrapper.cpp](https://github.com/mail2chromium/Android-Audio-Processing-Using-WebRTC/blob/master/app/src/main/jni/android_apm_wrapper.cpp), you can inspect this cpp-file. 254 | 255 | ------ 256 | 257 | #### 6- Android Make File 258 | 259 | ------ 260 | 261 | I need to build a native shared library (`.so`) with my JNI C/C++ functions in order to call them from JAVA(with JNI wrapper). To do that I have written android make file `Android.mk`. Since my C/C++ functions using WebRTC library files, I have integrated them with the make file as well. 262 | Following is the make file to build shared library `Android.mk`; 263 | 264 | ``` 265 | 266 | MY_WEBRTC_ROOT_PATH := $(call my-dir) 267 | 268 | LOCAL_CFLAGS += -DWEBRTC_POSIX 269 | 270 | include $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/aec/Android.mk 271 | include $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/aecm/Android.mk 272 | include $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/agc/Android.mk 273 | include $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/ns/Android.mk 274 | include $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/utility/Android.mk 275 | include $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/vad/Android.mk 276 | include $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/Android.mk 277 | include $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/intelligibility/Android.mk 278 | include $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/transient/Android.mk 279 | include $(MY_WEBRTC_ROOT_PATH)/modules/audio_processing/beamformer/Android.mk 280 | include $(MY_WEBRTC_ROOT_PATH)/common_audio/vad/Android.mk 281 | include $(MY_WEBRTC_ROOT_PATH)/common_audio/signal_processing/Android.mk 282 | include $(MY_WEBRTC_ROOT_PATH)/common_audio/Android.mk 283 | include $(MY_WEBRTC_ROOT_PATH)/common_audio/resampler/Android.mk 284 | include $(MY_WEBRTC_ROOT_PATH)/modules/audio_coding/codecs/isac/main/source/Android.mk 285 | include $(MY_WEBRTC_ROOT_PATH)/modules/audio_coding/codecs/Android.mk 286 | include $(MY_WEBRTC_ROOT_PATH)/system_wrappers/source/Android.mk 287 | include $(MY_WEBRTC_ROOT_PATH)/base/Android.mk 288 | 289 | 290 | LOCAL_PATH := $(call my-dir) 291 | 292 | include $(CLEAR_VARS) 293 | 294 | # Here you can change the name of your shared library 295 | LOCAL_MODULE := libwebrtc_apms 296 | 297 | LOCAL_MODULE_TAGS := optional 298 | 299 | LOCAL_LDLIBS += -llog 300 | LOCAL_LDLIBS += -landroid 301 | LOCAL_CFLAGS += -O0 302 | LOCAL_CPPFLAGS += -std=c++11 303 | LOCAL_CFLAGS += -DWEBRTC_ANDROID 304 | 305 | LOCAL_C_INCLUDES := $(JNI_PATH) 306 | 307 | # WebRTC Apm JNI Wrapper inculde C++ Functions and their body 308 | LOCAL_SRC_FILES := $(JNI_PATH)/android_apm_wrapper.cpp 309 | 310 | # Here, I basically loaded all the static libraries for different modules i.e. aec, aecm, agc, vad, ns 311 | LOCAL_WHOLE_STATIC_LIBRARIES := libwebrtc_aec \ 312 | libwebrtc_aecm \ 313 | libwebrtc_agc \ 314 | libwebrtc_ns \ 315 | libwebrtc_utility \ 316 | libwebrtc_vad \ 317 | libwebrtc_apm \ 318 | libwebrtc_common_audio_vad \ 319 | libwebrtc_spl \ 320 | libwebrtc_codecs_isac \ 321 | libwebrtc_common_audio \ 322 | libwebrtc_common_audio_resampler \ 323 | libwebrtc_system_wrapper \ 324 | libwebrtc_base \ 325 | libwebrtc_intelligibility \ 326 | libwebrtc_transient \ 327 | libwebrtc_beamformer \ 328 | libwebrtc_codecs 329 | 330 | LOCAL_PRELINK_MODULE := false 331 | 332 | include $(BUILD_SHARED_LIBRARY) 333 | 334 | 335 | ``` 336 | 337 | ---- 338 | 339 | #### 7- Build Shared Library via Android NDK 340 | 341 | ---- 342 | 343 | Now I can build the shared library via android NDK. To build the shared library, follow these steps; 344 | 345 | - Go to the directory of your Apm wrapper such as 346 | 347 | ``` 348 | 349 | path/src/main/jni/android_apm_wrapper.cpp 350 | 351 | ``` 352 | 353 | - Open the command prompt or terminal and build ndk by this command; 354 | 355 | ``` 356 | 357 | ndk-build 358 | 359 | ``` 360 | 361 | Here is the workaround of the `ndk-build` command: 362 | 363 | 364 | ![Android WebRTC Project Structure](https://github.com/mail2chromium/Android-Native-Development-For-WebRTC/blob/master/ndk_build.gif) 365 | 366 | 367 | This command will generates shared library files inside `app/src/main/libs` and `app/src/main/obj` directories. The shared library name defined in the make file with `LOCAL_MODULE := libwebrtc_apms` 368 | 369 | ![Android WebRTC Project Structure](https://github.com/mail2chromium/Android-Native-Development-For-WebRTC/blob/master/libs.png) 370 | 371 | 372 | Now everything is ready for *WebRTC APM Module* to pre-process and post-process audio data. 373 | 374 | 1. Push input PCM data into `ProcessCaptureStream` to process in place. 375 | 2. Get the processed PCM data from `ProcessCaptureStream` and send to far-end. 376 | 3. The far end pushed the received data into `ProcessRenderStream`. 377 | 378 | ---- 379 | 380 | #### 8- Run with gradle 381 | 382 | ---- 383 | 384 | Now I have integrated WbeRTC APM with my application. I can run it with gradle(in android studio). To run it with gradle I need to manually define the JNI library directory in build.gradle file 385 | 386 | 387 | ``` 388 | 389 | android { 390 | 391 | -- 392 | 393 | sourceSets { 394 | main { 395 | jniLibs.srcDir ‘src/main/libs’ 396 | jni.srcDirs = [] 397 | } 398 | } 399 | 400 | -- 401 | 402 | } 403 | 404 | 405 | ``` 406 | 407 | I have published all important source codes inculding an example which mentioned in this article on [Android-Audio-Processing-Using-WebRTC](https://github.com/mail2chromium/Android-Audio-Processing-Using-WebRTC). Please refer them if you want. 408 | --------------------------------------------------------------------------------