├── .gitignore ├── INSTALL.md ├── LICENSE ├── Makefile.am ├── README.md ├── autogen.sh ├── configure.ac ├── m4 └── ax_pthread.m4 ├── ndff.c ├── ndff.spec ├── ndff.sysconfig ├── ndffd.in ├── protos.txt └── tests ├── do.sh └── pcap ├── 6in4tunnel.pcap ├── BGP_Cisco_hdlc_slarp.pcap ├── BGP_redist.pcap ├── EAQ.pcap ├── Instagram.pcap ├── KakaoTalk_chat.pcap ├── KakaoTalk_talk.pcap ├── Meu.pcap ├── NTPv2.pcap ├── NTPv3.pcap ├── NTPv4.pcap ├── Oscar.pcap ├── README.txt ├── Torcedor.pcap ├── bittorrent.pcap ├── bt_search.pcap ├── google_ssl.pcap ├── http_ipv6.pcap ├── mpeg.pcap ├── mpegts.pcap ├── ocs.pcap ├── quic.pcap ├── quickplay.pcap ├── skype.pcap ├── skype_no_unknown.pcap ├── snapchat.pcap ├── starcraft_battle.pcap ├── teredo.pcap ├── waze.pcap ├── webex.pcap ├── whatsapp_login_call.pcap ├── whatsapp_login_chat.pcap └── whatsapp_voice_and_message.pcap /.gitignore: -------------------------------------------------------------------------------- 1 | # Object files 2 | *.o 3 | *.ko 4 | *.obj 5 | *.elf 6 | 7 | # Libraries 8 | *.lib 9 | *.a 10 | 11 | # Shared objects (inc. Windows DLLs) 12 | *.dll 13 | *.so 14 | *.so.* 15 | *.dylib 16 | 17 | # Executables 18 | *.exe 19 | *.out 20 | *.app 21 | *.i*86 22 | *.x86_64 23 | *.hex 24 | 25 | 26 | *.[oa] 27 | *~ 28 | *o.cmd 29 | *.lo 30 | *o.cmd 31 | *.swp 32 | *.la 33 | .libs 34 | .dirstamp 35 | stamp-h1 36 | /configure 37 | /config.guess 38 | /config.h 39 | /config.h.in 40 | /config.sub 41 | /config.log 42 | /config.status 43 | /Makefile.in 44 | /depcomp 45 | /install-sh 46 | /ltmain.sh 47 | /missing 48 | /Makefile 49 | /libndpi.pc 50 | /libtool 51 | /aclocal.m4 52 | /m4/libtool.m4 53 | /m4/ltoptions.m4 54 | /m4/ltsugar.m4 55 | /m4/ltversion.m4 56 | /m4/lt~obsolete.m4 57 | /.deps/ 58 | /autom4te.cache/ 59 | /ndff 60 | tags 61 | -------------------------------------------------------------------------------- /INSTALL.md: -------------------------------------------------------------------------------- 1 | Install 2 | =================== 3 | 4 | ## Prerequisites 5 | - GNU autotools/libtool 6 | - libpcap or PF_RING (optional but recommended) 7 | - [nDPI](http://www.ntop.org/products/deep-packet-inspection/ndpi/) 8 | 9 | ## autotools/libtool/libpcap 10 | ### On Ubuntu/Debian 11 | ``` 12 | # apt-get install build-essential 13 | # apt-get install git autoconf automake autogen libpcap-dev libtool pkg-config 14 | ``` 15 | 16 | ### On Fedora/CentOS 17 | ``` 18 | # yum install kernel-devel 19 | # yum groupinstall "Development tools" 20 | # yum install git autoconf automake autogen libpcap-devel libtool pkgconfig 21 | ``` 22 | 23 | ## nDPI 24 | ### From source 25 | ``` 26 | $ git clone https://github.com/ntop/nDPI.git 27 | $ cd nDPI 28 | $ ./autogen.sh 29 | $ ./configure 30 | $ make 31 | $ sudo make install 32 | ``` 33 | 34 | ### From package 35 | ``` 36 | $ sudo rpm -ivh https://forensics.cert.org/centos/cert/6.5/x86_64/nDPI-1.7.1-1.el6.x86_64.rpm 37 | ``` 38 | 39 | ## PF_RING (optional but recommended) 40 | Refer to [packages.ntop.org](http://packages.ntop.org/) 41 | 42 | ### On Ubuntu/Debian 43 | ``` 44 | $ wget http://apt-stable.ntop.org/14.04/all/apt-ntop-stable.deb 45 | $ sudo dpkg -i apt-ntop-stable.deb 46 | $ apt-get clean all 47 | $ apt-get update 48 | $ apt-get install pfring 49 | ``` 50 | 51 | ### On CentOS 52 | ``` 53 | # cat /etc/yum.repos.d/ntop.repo 54 | [ntop] 55 | name=ntop packages 56 | baseurl=http://packages.ntop.org/centos-stable/$releasever/$basearch/ 57 | enabled=1 58 | gpgcheck=1 59 | gpgkey=http://packages.ntop.org/centos-stable/RPM-GPG-KEY-deri 60 | [ntop-noarch] 61 | name=ntop packages 62 | baseurl=http://packages.ntop.org/centos-stable/$releasever/noarch/ 63 | enabled=1 64 | gpgcheck=1 65 | gpgkey=http://packages.ntop.org/centos-stable/RPM-GPG-KEY-deri 66 | and also install the /etc/yum.repos.d/epel.repo extra repositories 67 | # cat /etc/yum.repos.d/epel.repo 68 | [epel] 69 | name=Extra Packages for Enterprise Linux X - $basearch 70 | mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-X&arch=$basearch 71 | failovermethod=priority 72 | enabled=1 73 | gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-X 74 | and 75 | # cd /etc/yum.repos.d/ 76 | # wget https://copr.fedoraproject.org/coprs/saltstack/zeromq4/repo/epel-X/saltstack-zeromq4-epel-X.repo 77 | ``` 78 | Note: replace X with 6 (for CentOS 6) or 7 (for CentOS 7) then do: 79 | 80 | then do: 81 | ``` 82 | # yum erase zeromq3 (Do this once to make sure zeromq3 is not installed) 83 | # yum clean all 84 | # yum update 85 | # yum install pfring 86 | ``` 87 | 88 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 3, 29 June 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | Preamble 9 | 10 | The GNU General Public License is a free, copyleft license for 11 | software and other kinds of works. 12 | 13 | The licenses for most software and other practical works are designed 14 | to take away your freedom to share and change the works. By contrast, 15 | the GNU General Public License is intended to guarantee your freedom to 16 | share and change all versions of a program--to make sure it remains free 17 | software for all its users. We, the Free Software Foundation, use the 18 | GNU General Public License for most of our software; it applies also to 19 | any other work released this way by its authors. You can apply it to 20 | your programs, too. 21 | 22 | When we speak of free software, we are referring to freedom, not 23 | price. Our General Public Licenses are designed to make sure that you 24 | have the freedom to distribute copies of free software (and charge for 25 | them if you wish), that you receive source code or can get it if you 26 | want it, that you can change the software or use pieces of it in new 27 | free programs, and that you know you can do these things. 28 | 29 | To protect your rights, we need to prevent others from denying you 30 | these rights or asking you to surrender the rights. Therefore, you have 31 | certain responsibilities if you distribute copies of the software, or if 32 | you modify it: responsibilities to respect the freedom of others. 33 | 34 | For example, if you distribute copies of such a program, whether 35 | gratis or for a fee, you must pass on to the recipients the same 36 | freedoms that you received. You must make sure that they, too, receive 37 | or can get the source code. And you must show them these terms so they 38 | know their rights. 39 | 40 | Developers that use the GNU GPL protect your rights with two steps: 41 | (1) assert copyright on the software, and (2) offer you this License 42 | giving you legal permission to copy, distribute and/or modify it. 43 | 44 | For the developers' and authors' protection, the GPL clearly explains 45 | that there is no warranty for this free software. For both users' and 46 | authors' sake, the GPL requires that modified versions be marked as 47 | changed, so that their problems will not be attributed erroneously to 48 | authors of previous versions. 49 | 50 | Some devices are designed to deny users access to install or run 51 | modified versions of the software inside them, although the manufacturer 52 | can do so. This is fundamentally incompatible with the aim of 53 | protecting users' freedom to change the software. The systematic 54 | pattern of such abuse occurs in the area of products for individuals to 55 | use, which is precisely where it is most unacceptable. Therefore, we 56 | have designed this version of the GPL to prohibit the practice for those 57 | products. If such problems arise substantially in other domains, we 58 | stand ready to extend this provision to those domains in future versions 59 | of the GPL, as needed to protect the freedom of users. 60 | 61 | Finally, every program is threatened constantly by software patents. 62 | States should not allow patents to restrict development and use of 63 | software on general-purpose computers, but in those that do, we wish to 64 | avoid the special danger that patents applied to a free program could 65 | make it effectively proprietary. To prevent this, the GPL assures that 66 | patents cannot be used to render the program non-free. 67 | 68 | The precise terms and conditions for copying, distribution and 69 | modification follow. 70 | 71 | TERMS AND CONDITIONS 72 | 73 | 0. Definitions. 74 | 75 | "This License" refers to version 3 of the GNU General Public License. 76 | 77 | "Copyright" also means copyright-like laws that apply to other kinds of 78 | works, such as semiconductor masks. 79 | 80 | "The Program" refers to any copyrightable work licensed under this 81 | License. Each licensee is addressed as "you". "Licensees" and 82 | "recipients" may be individuals or organizations. 83 | 84 | To "modify" a work means to copy from or adapt all or part of the work 85 | in a fashion requiring copyright permission, other than the making of an 86 | exact copy. The resulting work is called a "modified version" of the 87 | earlier work or a work "based on" the earlier work. 88 | 89 | A "covered work" means either the unmodified Program or a work based 90 | on the Program. 91 | 92 | To "propagate" a work means to do anything with it that, without 93 | permission, would make you directly or secondarily liable for 94 | infringement under applicable copyright law, except executing it on a 95 | computer or modifying a private copy. Propagation includes copying, 96 | distribution (with or without modification), making available to the 97 | public, and in some countries other activities as well. 98 | 99 | To "convey" a work means any kind of propagation that enables other 100 | parties to make or receive copies. Mere interaction with a user through 101 | a computer network, with no transfer of a copy, is not conveying. 102 | 103 | An interactive user interface displays "Appropriate Legal Notices" 104 | to the extent that it includes a convenient and prominently visible 105 | feature that (1) displays an appropriate copyright notice, and (2) 106 | tells the user that there is no warranty for the work (except to the 107 | extent that warranties are provided), that licensees may convey the 108 | work under this License, and how to view a copy of this License. If 109 | the interface presents a list of user commands or options, such as a 110 | menu, a prominent item in the list meets this criterion. 111 | 112 | 1. Source Code. 113 | 114 | The "source code" for a work means the preferred form of the work 115 | for making modifications to it. "Object code" means any non-source 116 | form of a work. 117 | 118 | A "Standard Interface" means an interface that either is an official 119 | standard defined by a recognized standards body, or, in the case of 120 | interfaces specified for a particular programming language, one that 121 | is widely used among developers working in that language. 122 | 123 | The "System Libraries" of an executable work include anything, other 124 | than the work as a whole, that (a) is included in the normal form of 125 | packaging a Major Component, but which is not part of that Major 126 | Component, and (b) serves only to enable use of the work with that 127 | Major Component, or to implement a Standard Interface for which an 128 | implementation is available to the public in source code form. A 129 | "Major Component", in this context, means a major essential component 130 | (kernel, window system, and so on) of the specific operating system 131 | (if any) on which the executable work runs, or a compiler used to 132 | produce the work, or an object code interpreter used to run it. 133 | 134 | The "Corresponding Source" for a work in object code form means all 135 | the source code needed to generate, install, and (for an executable 136 | work) run the object code and to modify the work, including scripts to 137 | control those activities. However, it does not include the work's 138 | System Libraries, or general-purpose tools or generally available free 139 | programs which are used unmodified in performing those activities but 140 | which are not part of the work. For example, Corresponding Source 141 | includes interface definition files associated with source files for 142 | the work, and the source code for shared libraries and dynamically 143 | linked subprograms that the work is specifically designed to require, 144 | such as by intimate data communication or control flow between those 145 | subprograms and other parts of the work. 146 | 147 | The Corresponding Source need not include anything that users 148 | can regenerate automatically from other parts of the Corresponding 149 | Source. 150 | 151 | The Corresponding Source for a work in source code form is that 152 | same work. 153 | 154 | 2. Basic Permissions. 155 | 156 | All rights granted under this License are granted for the term of 157 | copyright on the Program, and are irrevocable provided the stated 158 | conditions are met. This License explicitly affirms your unlimited 159 | permission to run the unmodified Program. The output from running a 160 | covered work is covered by this License only if the output, given its 161 | content, constitutes a covered work. This License acknowledges your 162 | rights of fair use or other equivalent, as provided by copyright law. 163 | 164 | You may make, run and propagate covered works that you do not 165 | convey, without conditions so long as your license otherwise remains 166 | in force. You may convey covered works to others for the sole purpose 167 | of having them make modifications exclusively for you, or provide you 168 | with facilities for running those works, provided that you comply with 169 | the terms of this License in conveying all material for which you do 170 | not control copyright. Those thus making or running the covered works 171 | for you must do so exclusively on your behalf, under your direction 172 | and control, on terms that prohibit them from making any copies of 173 | your copyrighted material outside their relationship with you. 174 | 175 | Conveying under any other circumstances is permitted solely under 176 | the conditions stated below. Sublicensing is not allowed; section 10 177 | makes it unnecessary. 178 | 179 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 180 | 181 | No covered work shall be deemed part of an effective technological 182 | measure under any applicable law fulfilling obligations under article 183 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or 184 | similar laws prohibiting or restricting circumvention of such 185 | measures. 186 | 187 | When you convey a covered work, you waive any legal power to forbid 188 | circumvention of technological measures to the extent such circumvention 189 | is effected by exercising rights under this License with respect to 190 | the covered work, and you disclaim any intention to limit operation or 191 | modification of the work as a means of enforcing, against the work's 192 | users, your or third parties' legal rights to forbid circumvention of 193 | technological measures. 194 | 195 | 4. Conveying Verbatim Copies. 196 | 197 | You may convey verbatim copies of the Program's source code as you 198 | receive it, in any medium, provided that you conspicuously and 199 | appropriately publish on each copy an appropriate copyright notice; 200 | keep intact all notices stating that this License and any 201 | non-permissive terms added in accord with section 7 apply to the code; 202 | keep intact all notices of the absence of any warranty; and give all 203 | recipients a copy of this License along with the Program. 204 | 205 | You may charge any price or no price for each copy that you convey, 206 | and you may offer support or warranty protection for a fee. 207 | 208 | 5. Conveying Modified Source Versions. 209 | 210 | You may convey a work based on the Program, or the modifications to 211 | produce it from the Program, in the form of source code under the 212 | terms of section 4, provided that you also meet all of these conditions: 213 | 214 | a) The work must carry prominent notices stating that you modified 215 | it, and giving a relevant date. 216 | 217 | b) The work must carry prominent notices stating that it is 218 | released under this License and any conditions added under section 219 | 7. This requirement modifies the requirement in section 4 to 220 | "keep intact all notices". 221 | 222 | c) You must license the entire work, as a whole, under this 223 | License to anyone who comes into possession of a copy. This 224 | License will therefore apply, along with any applicable section 7 225 | additional terms, to the whole of the work, and all its parts, 226 | regardless of how they are packaged. This License gives no 227 | permission to license the work in any other way, but it does not 228 | invalidate such permission if you have separately received it. 229 | 230 | d) If the work has interactive user interfaces, each must display 231 | Appropriate Legal Notices; however, if the Program has interactive 232 | interfaces that do not display Appropriate Legal Notices, your 233 | work need not make them do so. 234 | 235 | A compilation of a covered work with other separate and independent 236 | works, which are not by their nature extensions of the covered work, 237 | and which are not combined with it such as to form a larger program, 238 | in or on a volume of a storage or distribution medium, is called an 239 | "aggregate" if the compilation and its resulting copyright are not 240 | used to limit the access or legal rights of the compilation's users 241 | beyond what the individual works permit. Inclusion of a covered work 242 | in an aggregate does not cause this License to apply to the other 243 | parts of the aggregate. 244 | 245 | 6. Conveying Non-Source Forms. 246 | 247 | You may convey a covered work in object code form under the terms 248 | of sections 4 and 5, provided that you also convey the 249 | machine-readable Corresponding Source under the terms of this License, 250 | in one of these ways: 251 | 252 | a) Convey the object code in, or embodied in, a physical product 253 | (including a physical distribution medium), accompanied by the 254 | Corresponding Source fixed on a durable physical medium 255 | customarily used for software interchange. 256 | 257 | b) Convey the object code in, or embodied in, a physical product 258 | (including a physical distribution medium), accompanied by a 259 | written offer, valid for at least three years and valid for as 260 | long as you offer spare parts or customer support for that product 261 | model, to give anyone who possesses the object code either (1) a 262 | copy of the Corresponding Source for all the software in the 263 | product that is covered by this License, on a durable physical 264 | medium customarily used for software interchange, for a price no 265 | more than your reasonable cost of physically performing this 266 | conveying of source, or (2) access to copy the 267 | Corresponding Source from a network server at no charge. 268 | 269 | c) Convey individual copies of the object code with a copy of the 270 | written offer to provide the Corresponding Source. This 271 | alternative is allowed only occasionally and noncommercially, and 272 | only if you received the object code with such an offer, in accord 273 | with subsection 6b. 274 | 275 | d) Convey the object code by offering access from a designated 276 | place (gratis or for a charge), and offer equivalent access to the 277 | Corresponding Source in the same way through the same place at no 278 | further charge. You need not require recipients to copy the 279 | Corresponding Source along with the object code. If the place to 280 | copy the object code is a network server, the Corresponding Source 281 | may be on a different server (operated by you or a third party) 282 | that supports equivalent copying facilities, provided you maintain 283 | clear directions next to the object code saying where to find the 284 | Corresponding Source. Regardless of what server hosts the 285 | Corresponding Source, you remain obligated to ensure that it is 286 | available for as long as needed to satisfy these requirements. 287 | 288 | e) Convey the object code using peer-to-peer transmission, provided 289 | you inform other peers where the object code and Corresponding 290 | Source of the work are being offered to the general public at no 291 | charge under subsection 6d. 292 | 293 | A separable portion of the object code, whose source code is excluded 294 | from the Corresponding Source as a System Library, need not be 295 | included in conveying the object code work. 296 | 297 | A "User Product" is either (1) a "consumer product", which means any 298 | tangible personal property which is normally used for personal, family, 299 | or household purposes, or (2) anything designed or sold for incorporation 300 | into a dwelling. In determining whether a product is a consumer product, 301 | doubtful cases shall be resolved in favor of coverage. For a particular 302 | product received by a particular user, "normally used" refers to a 303 | typical or common use of that class of product, regardless of the status 304 | of the particular user or of the way in which the particular user 305 | actually uses, or expects or is expected to use, the product. A product 306 | is a consumer product regardless of whether the product has substantial 307 | commercial, industrial or non-consumer uses, unless such uses represent 308 | the only significant mode of use of the product. 309 | 310 | "Installation Information" for a User Product means any methods, 311 | procedures, authorization keys, or other information required to install 312 | and execute modified versions of a covered work in that User Product from 313 | a modified version of its Corresponding Source. The information must 314 | suffice to ensure that the continued functioning of the modified object 315 | code is in no case prevented or interfered with solely because 316 | modification has been made. 317 | 318 | If you convey an object code work under this section in, or with, or 319 | specifically for use in, a User Product, and the conveying occurs as 320 | part of a transaction in which the right of possession and use of the 321 | User Product is transferred to the recipient in perpetuity or for a 322 | fixed term (regardless of how the transaction is characterized), the 323 | Corresponding Source conveyed under this section must be accompanied 324 | by the Installation Information. But this requirement does not apply 325 | if neither you nor any third party retains the ability to install 326 | modified object code on the User Product (for example, the work has 327 | been installed in ROM). 328 | 329 | The requirement to provide Installation Information does not include a 330 | requirement to continue to provide support service, warranty, or updates 331 | for a work that has been modified or installed by the recipient, or for 332 | the User Product in which it has been modified or installed. Access to a 333 | network may be denied when the modification itself materially and 334 | adversely affects the operation of the network or violates the rules and 335 | protocols for communication across the network. 336 | 337 | Corresponding Source conveyed, and Installation Information provided, 338 | in accord with this section must be in a format that is publicly 339 | documented (and with an implementation available to the public in 340 | source code form), and must require no special password or key for 341 | unpacking, reading or copying. 342 | 343 | 7. Additional Terms. 344 | 345 | "Additional permissions" are terms that supplement the terms of this 346 | License by making exceptions from one or more of its conditions. 347 | Additional permissions that are applicable to the entire Program shall 348 | be treated as though they were included in this License, to the extent 349 | that they are valid under applicable law. If additional permissions 350 | apply only to part of the Program, that part may be used separately 351 | under those permissions, but the entire Program remains governed by 352 | this License without regard to the additional permissions. 353 | 354 | When you convey a copy of a covered work, you may at your option 355 | remove any additional permissions from that copy, or from any part of 356 | it. (Additional permissions may be written to require their own 357 | removal in certain cases when you modify the work.) You may place 358 | additional permissions on material, added by you to a covered work, 359 | for which you have or can give appropriate copyright permission. 360 | 361 | Notwithstanding any other provision of this License, for material you 362 | add to a covered work, you may (if authorized by the copyright holders of 363 | that material) supplement the terms of this License with terms: 364 | 365 | a) Disclaiming warranty or limiting liability differently from the 366 | terms of sections 15 and 16 of this License; or 367 | 368 | b) Requiring preservation of specified reasonable legal notices or 369 | author attributions in that material or in the Appropriate Legal 370 | Notices displayed by works containing it; or 371 | 372 | c) Prohibiting misrepresentation of the origin of that material, or 373 | requiring that modified versions of such material be marked in 374 | reasonable ways as different from the original version; or 375 | 376 | d) Limiting the use for publicity purposes of names of licensors or 377 | authors of the material; or 378 | 379 | e) Declining to grant rights under trademark law for use of some 380 | trade names, trademarks, or service marks; or 381 | 382 | f) Requiring indemnification of licensors and authors of that 383 | material by anyone who conveys the material (or modified versions of 384 | it) with contractual assumptions of liability to the recipient, for 385 | any liability that these contractual assumptions directly impose on 386 | those licensors and authors. 387 | 388 | All other non-permissive additional terms are considered "further 389 | restrictions" within the meaning of section 10. If the Program as you 390 | received it, or any part of it, contains a notice stating that it is 391 | governed by this License along with a term that is a further 392 | restriction, you may remove that term. If a license document contains 393 | a further restriction but permits relicensing or conveying under this 394 | License, you may add to a covered work material governed by the terms 395 | of that license document, provided that the further restriction does 396 | not survive such relicensing or conveying. 397 | 398 | If you add terms to a covered work in accord with this section, you 399 | must place, in the relevant source files, a statement of the 400 | additional terms that apply to those files, or a notice indicating 401 | where to find the applicable terms. 402 | 403 | Additional terms, permissive or non-permissive, may be stated in the 404 | form of a separately written license, or stated as exceptions; 405 | the above requirements apply either way. 406 | 407 | 8. Termination. 408 | 409 | You may not propagate or modify a covered work except as expressly 410 | provided under this License. Any attempt otherwise to propagate or 411 | modify it is void, and will automatically terminate your rights under 412 | this License (including any patent licenses granted under the third 413 | paragraph of section 11). 414 | 415 | However, if you cease all violation of this License, then your 416 | license from a particular copyright holder is reinstated (a) 417 | provisionally, unless and until the copyright holder explicitly and 418 | finally terminates your license, and (b) permanently, if the copyright 419 | holder fails to notify you of the violation by some reasonable means 420 | prior to 60 days after the cessation. 421 | 422 | Moreover, your license from a particular copyright holder is 423 | reinstated permanently if the copyright holder notifies you of the 424 | violation by some reasonable means, this is the first time you have 425 | received notice of violation of this License (for any work) from that 426 | copyright holder, and you cure the violation prior to 30 days after 427 | your receipt of the notice. 428 | 429 | Termination of your rights under this section does not terminate the 430 | licenses of parties who have received copies or rights from you under 431 | this License. If your rights have been terminated and not permanently 432 | reinstated, you do not qualify to receive new licenses for the same 433 | material under section 10. 434 | 435 | 9. Acceptance Not Required for Having Copies. 436 | 437 | You are not required to accept this License in order to receive or 438 | run a copy of the Program. Ancillary propagation of a covered work 439 | occurring solely as a consequence of using peer-to-peer transmission 440 | to receive a copy likewise does not require acceptance. However, 441 | nothing other than this License grants you permission to propagate or 442 | modify any covered work. These actions infringe copyright if you do 443 | not accept this License. Therefore, by modifying or propagating a 444 | covered work, you indicate your acceptance of this License to do so. 445 | 446 | 10. Automatic Licensing of Downstream Recipients. 447 | 448 | Each time you convey a covered work, the recipient automatically 449 | receives a license from the original licensors, to run, modify and 450 | propagate that work, subject to this License. You are not responsible 451 | for enforcing compliance by third parties with this License. 452 | 453 | An "entity transaction" is a transaction transferring control of an 454 | organization, or substantially all assets of one, or subdividing an 455 | organization, or merging organizations. If propagation of a covered 456 | work results from an entity transaction, each party to that 457 | transaction who receives a copy of the work also receives whatever 458 | licenses to the work the party's predecessor in interest had or could 459 | give under the previous paragraph, plus a right to possession of the 460 | Corresponding Source of the work from the predecessor in interest, if 461 | the predecessor has it or can get it with reasonable efforts. 462 | 463 | You may not impose any further restrictions on the exercise of the 464 | rights granted or affirmed under this License. For example, you may 465 | not impose a license fee, royalty, or other charge for exercise of 466 | rights granted under this License, and you may not initiate litigation 467 | (including a cross-claim or counterclaim in a lawsuit) alleging that 468 | any patent claim is infringed by making, using, selling, offering for 469 | sale, or importing the Program or any portion of it. 470 | 471 | 11. Patents. 472 | 473 | A "contributor" is a copyright holder who authorizes use under this 474 | License of the Program or a work on which the Program is based. The 475 | work thus licensed is called the contributor's "contributor version". 476 | 477 | A contributor's "essential patent claims" are all patent claims 478 | owned or controlled by the contributor, whether already acquired or 479 | hereafter acquired, that would be infringed by some manner, permitted 480 | by this License, of making, using, or selling its contributor version, 481 | but do not include claims that would be infringed only as a 482 | consequence of further modification of the contributor version. For 483 | purposes of this definition, "control" includes the right to grant 484 | patent sublicenses in a manner consistent with the requirements of 485 | this License. 486 | 487 | Each contributor grants you a non-exclusive, worldwide, royalty-free 488 | patent license under the contributor's essential patent claims, to 489 | make, use, sell, offer for sale, import and otherwise run, modify and 490 | propagate the contents of its contributor version. 491 | 492 | In the following three paragraphs, a "patent license" is any express 493 | agreement or commitment, however denominated, not to enforce a patent 494 | (such as an express permission to practice a patent or covenant not to 495 | sue for patent infringement). To "grant" such a patent license to a 496 | party means to make such an agreement or commitment not to enforce a 497 | patent against the party. 498 | 499 | If you convey a covered work, knowingly relying on a patent license, 500 | and the Corresponding Source of the work is not available for anyone 501 | to copy, free of charge and under the terms of this License, through a 502 | publicly available network server or other readily accessible means, 503 | then you must either (1) cause the Corresponding Source to be so 504 | available, or (2) arrange to deprive yourself of the benefit of the 505 | patent license for this particular work, or (3) arrange, in a manner 506 | consistent with the requirements of this License, to extend the patent 507 | license to downstream recipients. "Knowingly relying" means you have 508 | actual knowledge that, but for the patent license, your conveying the 509 | covered work in a country, or your recipient's use of the covered work 510 | in a country, would infringe one or more identifiable patents in that 511 | country that you have reason to believe are valid. 512 | 513 | If, pursuant to or in connection with a single transaction or 514 | arrangement, you convey, or propagate by procuring conveyance of, a 515 | covered work, and grant a patent license to some of the parties 516 | receiving the covered work authorizing them to use, propagate, modify 517 | or convey a specific copy of the covered work, then the patent license 518 | you grant is automatically extended to all recipients of the covered 519 | work and works based on it. 520 | 521 | A patent license is "discriminatory" if it does not include within 522 | the scope of its coverage, prohibits the exercise of, or is 523 | conditioned on the non-exercise of one or more of the rights that are 524 | specifically granted under this License. You may not convey a covered 525 | work if you are a party to an arrangement with a third party that is 526 | in the business of distributing software, under which you make payment 527 | to the third party based on the extent of your activity of conveying 528 | the work, and under which the third party grants, to any of the 529 | parties who would receive the covered work from you, a discriminatory 530 | patent license (a) in connection with copies of the covered work 531 | conveyed by you (or copies made from those copies), or (b) primarily 532 | for and in connection with specific products or compilations that 533 | contain the covered work, unless you entered into that arrangement, 534 | or that patent license was granted, prior to 28 March 2007. 535 | 536 | Nothing in this License shall be construed as excluding or limiting 537 | any implied license or other defenses to infringement that may 538 | otherwise be available to you under applicable patent law. 539 | 540 | 12. No Surrender of Others' Freedom. 541 | 542 | If conditions are imposed on you (whether by court order, agreement or 543 | otherwise) that contradict the conditions of this License, they do not 544 | excuse you from the conditions of this License. If you cannot convey a 545 | covered work so as to satisfy simultaneously your obligations under this 546 | License and any other pertinent obligations, then as a consequence you may 547 | not convey it at all. For example, if you agree to terms that obligate you 548 | to collect a royalty for further conveying from those to whom you convey 549 | the Program, the only way you could satisfy both those terms and this 550 | License would be to refrain entirely from conveying the Program. 551 | 552 | 13. Use with the GNU Affero General Public License. 553 | 554 | Notwithstanding any other provision of this License, you have 555 | permission to link or combine any covered work with a work licensed 556 | under version 3 of the GNU Affero General Public License into a single 557 | combined work, and to convey the resulting work. The terms of this 558 | License will continue to apply to the part which is the covered work, 559 | but the special requirements of the GNU Affero General Public License, 560 | section 13, concerning interaction through a network will apply to the 561 | combination as such. 562 | 563 | 14. Revised Versions of this License. 564 | 565 | The Free Software Foundation may publish revised and/or new versions of 566 | the GNU General Public License from time to time. Such new versions will 567 | be similar in spirit to the present version, but may differ in detail to 568 | address new problems or concerns. 569 | 570 | Each version is given a distinguishing version number. If the 571 | Program specifies that a certain numbered version of the GNU General 572 | Public License "or any later version" applies to it, you have the 573 | option of following the terms and conditions either of that numbered 574 | version or of any later version published by the Free Software 575 | Foundation. If the Program does not specify a version number of the 576 | GNU General Public License, you may choose any version ever published 577 | by the Free Software Foundation. 578 | 579 | If the Program specifies that a proxy can decide which future 580 | versions of the GNU General Public License can be used, that proxy's 581 | public statement of acceptance of a version permanently authorizes you 582 | to choose that version for the Program. 583 | 584 | Later license versions may give you additional or different 585 | permissions. However, no additional obligations are imposed on any 586 | author or copyright holder as a result of your choosing to follow a 587 | later version. 588 | 589 | 15. Disclaimer of Warranty. 590 | 591 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY 592 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT 593 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY 594 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, 595 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 596 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM 597 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF 598 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 599 | 600 | 16. Limitation of Liability. 601 | 602 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 603 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS 604 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY 605 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE 606 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF 607 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD 608 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), 609 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF 610 | SUCH DAMAGES. 611 | 612 | 17. Interpretation of Sections 15 and 16. 613 | 614 | If the disclaimer of warranty and limitation of liability provided 615 | above cannot be given local legal effect according to their terms, 616 | reviewing courts shall apply local law that most closely approximates 617 | an absolute waiver of all civil liability in connection with the 618 | Program, unless a warranty or assumption of liability accompanies a 619 | copy of the Program in return for a fee. 620 | 621 | END OF TERMS AND CONDITIONS 622 | 623 | How to Apply These Terms to Your New Programs 624 | 625 | If you develop a new program, and you want it to be of the greatest 626 | possible use to the public, the best way to achieve this is to make it 627 | free software which everyone can redistribute and change under these terms. 628 | 629 | To do so, attach the following notices to the program. It is safest 630 | to attach them to the start of each source file to most effectively 631 | state the exclusion of warranty; and each file should have at least 632 | the "copyright" line and a pointer to where the full notice is found. 633 | 634 | 635 | Copyright (C) 636 | 637 | This program is free software: you can redistribute it and/or modify 638 | it under the terms of the GNU General Public License as published by 639 | the Free Software Foundation, either version 3 of the License, or 640 | (at your option) any later version. 641 | 642 | This program is distributed in the hope that it will be useful, 643 | but WITHOUT ANY WARRANTY; without even the implied warranty of 644 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 645 | GNU General Public License for more details. 646 | 647 | You should have received a copy of the GNU General Public License 648 | along with this program. If not, see . 649 | 650 | Also add information on how to contact you by electronic and paper mail. 651 | 652 | If the program does terminal interaction, make it output a short 653 | notice like this when it starts in an interactive mode: 654 | 655 | Copyright (C) 656 | This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 657 | This is free software, and you are welcome to redistribute it 658 | under certain conditions; type `show c' for details. 659 | 660 | The hypothetical commands `show w' and `show c' should show the appropriate 661 | parts of the General Public License. Of course, your program's commands 662 | might be different; for a GUI interface, you would use an "about box". 663 | 664 | You should also get your employer (if you work as a programmer) or school, 665 | if any, to sign a "copyright disclaimer" for the program, if necessary. 666 | For more information on this, and how to apply and follow the GNU GPL, see 667 | . 668 | 669 | The GNU General Public License does not permit incorporating your program 670 | into proprietary programs. If your program is a subroutine library, you 671 | may consider it more useful to permit linking proprietary applications with 672 | the library. If this is what you want to do, use the GNU Lesser General 673 | Public License instead of this License. But first, please read 674 | . 675 | 676 | ------------------------------------------------------------------------- 677 | 678 | /* 679 | * Copyright 2002-2006 Damien Miller All rights reserved. 680 | * 681 | * Redistribution and use in source and binary forms, with or without 682 | * modification, are permitted provided that the following conditions 683 | * are met: 684 | * 1. Redistributions of source code must retain the above copyright 685 | * notice, this list of conditions and the following disclaimer. 686 | * 2. Redistributions in binary form must reproduce the above copyright 687 | * notice, this list of conditions and the following disclaimer in the 688 | * documentation and/or other materials provided with the distribution. 689 | * 690 | * THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR 691 | * IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES 692 | * OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 693 | * IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, 694 | * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT 695 | * NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, 696 | * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY 697 | * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 698 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF 699 | * THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 700 | */ 701 | -------------------------------------------------------------------------------- /Makefile.am: -------------------------------------------------------------------------------- 1 | ACLOCAL_AMFLAGS = -I m4 2 | 3 | bin_PROGRAMS = ndff 4 | 5 | AM_CPPFLAGS = @PCAP_INC@ 6 | AM_CFLAGS = @PTHREAD_CFLAGS@ @JSON_C_CFLAGS@ @NDPI_CFLAGS@ @MSGPACK_CFLAGS@ 7 | 8 | LDADD = @JSON_C_LIBS@ @PTHREAD_LIBS@ @PCAP_LIB@ @NDPI_LIBS@ @MSGPACK_LIBS@ 9 | AM_LDFLAGS = -static 10 | 11 | ndff_SOURCES = ndff.c 12 | 13 | init_ddir = $(sysconfdir)/rc.d/init.d 14 | init_d_SCRIPTS = ndffd 15 | CLEANFILES = $(init_d_SCRIPTS) 16 | 17 | ndff.o: ndff.c 18 | 19 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ndff (nDPI for fluentd) 2 | =================== 3 | ndff is a flow collector with nDPI library for fluentd. 4 | ndff has been implemented on the basis of ndpiReader which is an example in nDPI. 5 | 6 | 7 | Overview 8 | -------- 9 | 10 | ![b44f24cc-f5b5-11e5-8e98-b641ec22cce4](https://cloud.githubusercontent.com/assets/2253692/14105522/0d5b111c-f5e7-11e5-88dd-cef1cf914614.png) 11 | 12 | 13 | [nDPI](http://www.ntop.org/products/deep-packet-inspection/ndpi/) is an open source LGPLv3 library for deep-packet inspection. 14 | ndpiReader is an example in nDPI which has the following features. 15 | - Detect the protocol from packets (pcap files or devices) 16 | - Apply a BPF filter for filtering selected traffic 17 | - Export the content of packets to a file in JSON format 18 | - etc... 19 | 20 | It's very userful if the results of nDPI can be analyzed in fluentd. 21 | So, ndff aggregates packets as a flow like NetFlow and forwards the results to the fluentd server in the form of JSON or MessagePack. 22 | 23 | Configuration 24 | -------- 25 | ### Requirements 26 | - Common 27 | - nDPI 28 | - From source 29 | - GNU tools (autogen, automake, autoconf, libtool, pkg-config) 30 | - GNU C compiler (gcc) 31 | 32 | ### Building and Installing 33 | #### Quick Start 34 | Install ndff from binary (CentOS 6). 35 | ``` 36 | $ sudo yum -y install libpcap msgpack json-c 37 | $ sudo rpm -ivh https://forensics.cert.org/centos/cert/6.5/x86_64/nDPI-1.7.1-1.el6.x86_64.rpm 38 | $ sudo rpm -ivh https://github.com/knqyf263/ndff/releases/download/0.0.2/ndff-0.0.2-1.x86_64.rpm 39 | ``` 40 | And run 41 | ``` 42 | $ sudo ndff -i eth0 -s 127.0.0.1 -p 24224 -m msgpack 43 | ``` 44 | 45 | #### From source 46 | Refer to [INSTALL.md](https://github.com/knqyf263/ndff/blob/master/INSTALL.md) 47 | 48 | Then, install [json-c](https://github.com/json-c/json-c) or [msgpack-c](https://github.com/msgpack/msgpack-c). 49 | In the case of json-c: 50 | ``` 51 | $ git clone https://github.com/json-c/json-c.git 52 | $ cd json-c 53 | $ sh autogen.sh 54 | $ ./configure 55 | $ make 56 | $ make install 57 | ``` 58 | 59 | Finally, install ndff from git repository. 60 | ``` 61 | $ git clone https://github.com/knqyf263/ndff.git 62 | $ cd ndff 63 | $ ./autogen.sh 64 | $ make 65 | $ sudo make install 66 | ``` 67 | 68 | 69 | ### Options 70 | 71 | ``` 72 | $ ndff 73 | ndff -i [-s ] [-m ] [-f ] 74 | [-p ][-P ][-t ][-q][-d][-D][-h][-T][-v ] 75 | [-n ] [-w ] 76 | 77 | Usage: 78 | -i | Specify a pcap file/playlist to read packets from or a device for live capture (comma-separated list) 79 | -m | Specify a protocol to send messages to the server (json or msgpack) 80 | -f | Specify a BPF filter for filtering selected traffic 81 | -s | Specify a server for fluentd (If not, ndff runs in the dry-run mode) 82 | -p | Specify a port for fluentd (default: 24224) 83 | -P .protos | Specify a protocol file (eg. protos.txt) 84 | -n | Number of threads. Default: number of interfaces in -i. Ignored with pcap files. 85 | -g | Thread affinity mask (one core id per thread) 86 | -d | Daemonize (run in background) 87 | -D | Disable protocol guess and use only DPI 88 | -q | Quiet mode 89 | -t | Specify a tag for fluentd (default: ndpi.flow) 90 | -T | Dissect GTP/TZSP tunnels 91 | -r | Print nDPI version and git revision 92 | -w | Write test output on the specified file. This is useful for 93 | | testing purposes in order to compare results across runs 94 | -h | This help 95 | -v <1|2> | Verbose 'unknown protocol' packet print. 1=verbose, 2=very verbose 96 | ``` 97 | 98 | 99 | Usage 100 | -------- 101 | Run ndff in foreground or background 102 | 103 | ### Command-line 104 | 105 | #### dry-run (live traffic capture) 106 | Export results to stdout. 107 | ``` 108 | $ sudo ndff -v 2 -i eth0 109 | [WARN] No server is specified. This is dry-run mode. 110 | [INFO] Capturing live traffic from device eth0... 111 | [INFO] Running thread 0... 112 | 113 | 1 ICMPV6 [2001:db8::253]:0 <-> [2001:db8::22]:0 [VLAN: 262][proto: 102/ICMPV6][1 pkts/90 bytes] 114 | 2 UDP 192.0.2.10:1985 <-> 192.0.2.11:1985 [VLAN: 262][proto: 125/Skype][1 pkts/98 bytes] 115 | 3 VRRP 192.0.2.10:0 <-> 192.0.2.11:0 [VLAN: 262][proto: 73/VRRP][1 pkts/60 bytes] 116 | 4 UDP 192.0.2.12:1985 <-> 192.0.2.13:1985 [proto: 125/Skype][1 pkts/94 bytes] 117 | 5 TCP 192.0.2.14:22 <-> 192.0.2.15:51462 [proto: 92/SSH][11 pkts/1510 bytes] 118 | ``` 119 | 120 | #### dry-run (from pcap file) 121 | Export results to the file. 122 | ``` 123 | $ sudo ndff -v 2 -i tests/pcap/bittorrent.pcap -w bittorrent.txt 124 | [WARN] No server is specified. This is dry-run mode. 125 | [INFO] Reading packets from pcap file tests/pcap/bittorrent.pcap... 126 | [INFO] Running thread 0... 127 | 128 | $ head -n5 bittorrent.txt 129 | 1 TCP 192.168.1.3:52888 <-> 82.58.216.115:38305 [proto: 37/BitTorrent][1 pkts/134 bytes][BT Hash: dcfcdccfb9e670ccc3dd40c78c161f2bea243126] 130 | 2 TCP 192.168.1.3:52887 <-> 82.57.97.83:53137 [proto: 37/BitTorrent][1 pkts/134 bytes][BT Hash: dcfcdccfb9e670ccc3dd40c78c161f2bea243126] 131 | 3 TCP 192.168.1.3:52895 <-> 83.216.184.241:51413 [proto: 37/BitTorrent][1 pkts/134 bytes][BT Hash: dcfcdccfb9e670ccc3dd40c78c161f2bea243126] 132 | 4 TCP 79.53.228.2:14627 <-> 192.168.1.3:52896 [proto: 37/BitTorrent][1 pkts/134 bytes][BT Hash: dcfcdccfb9e670ccc3dd40c78c161f2bea243126] 133 | 5 TCP 192.168.1.3:52894 <-> 120.62.33.241:39332 [proto: 37/BitTorrent][1 pkts/134 bytes][BT Hash: dcfcdccfb9e670ccc3dd40c78c161f2bea243126] 134 | ``` 135 | 136 | 137 | #### send messages (JSON) 138 | Send to the fluentd server. 139 | ``` 140 | $ sudo ndff -q -i eth0 -s fluentd.example.com -p 22425 -t json.ndpi.flow -m json 141 | Capturing live traffic from device eth0... 142 | Running thread 0... 143 | ``` 144 | 145 | then output becomes as below at the fluentd server 146 | ``` 147 | 2016-03-29 14:41:23 +0900 ndpi.flow: {"protocol":"ICMPV6","src_addr":"2001:DB8::1234","src_port":0,"dst_addr":"2001:DB8::5678","dst_port":0,"detected_protocol":102,"protocol_name":"ICMPV6","out_pkts":1,"out_bytes":86,"in_pkts":0,"in_bytes":0,"first_switched":1459230083,"last_switched":1459230083,"server_name":""} 148 | 2016-03-29 14:41:00 +0900 ndpi.flow: {"protocol":"UDP","src_addr":"192.0.2.2","src_port":1985,"dst_addr":"192.0.2.3","dst_port":1985,"detected_protocol":125,"protocol_name":"Skype","out_pkts":16,"out_bytes":1568,"in_pkts":0,"in_bytes":0,"first_switched":1459230060,"last_switched":1459230100,"server_name":""} 149 | 2016-03-29 14:41:35 +0900 ndpi.flow: {"protocol":"TCP","src_addr":"192.0.2.4","src_port":49751,"dst_addr":"192.0.2.5","dst_port":80,"detected_protocol":7,"protocol_name":"HTTP","out_pkts":6,"out_bytes":514,"in_pkts":4,"in_bytes":816,"first_switched":1459230095,"last_switched":1459230096,"server_name":"google.co.jp"} 150 | 2016-03-29 14:41:35 +0900 ndpi.flow: {"protocol":"UDP","src_addr":"192.0.2.4,"src_port":36605,"dst_addr":"192.0.2.6","dst_port":53,"master_protocol":5,"detected_protocol":126,"protocol_name":"DNS.Google","out_pkts":2,"out_bytes":152,"in_pkts":2,"in_bytes":488,"first_switched":1459230095,"last_switched":1459230095,"server_name":"www.google.co.jp"} 151 | ``` 152 | 153 | #### send messages (MessagePack) 154 | ``` 155 | $ sudo ndff -q -i eth0 -s fluentd.example.com -p 22425 -t msgpack.ndpi.flow -m msgpack 156 | Capturing live traffic from device eth0... 157 | Running thread 0... 158 | ``` 159 | 160 | then output becomes as below at the fluentd server 161 | ``` 162 | 2016-03-29 14:41:23 +0900 ndpi.flow: {"protocol":"ICMPV6","src_addr":"2001:DB8::1234","src_port":0,"dst_addr":"2001:DB8::5678","dst_port":0,"detected_protocol":102,"protocol_name":"ICMPV6","out_pkts":1,"out_bytes":86,"in_pkts":0,"in_bytes":0,"first_switched":1459230083,"last_switched":1459230083,"server_name":""} 163 | 2016-03-29 14:41:00 +0900 ndpi.flow: {"protocol":"UDP","src_addr":"192.0.2.2","src_port":1985,"dst_addr":"192.0.2.3","dst_port":1985,"detected_protocol":125,"protocol_name":"Skype","out_pkts":16,"out_bytes":1568,"in_pkts":0,"in_bytes":0,"first_switched":1459230060,"last_switched":1459230100,"server_name":""} 164 | 2016-03-29 14:41:35 +0900 ndpi.flow: {"protocol":"TCP","src_addr":"192.0.2.4","src_port":49751,"dst_addr":"192.0.2.5","dst_port":80,"detected_protocol":7,"protocol_name":"HTTP","out_pkts":6,"out_bytes":514,"in_pkts":4,"in_bytes":816,"first_switched":1459230095,"last_switched":1459230096,"server_name":"google.co.jp"} 165 | 2016-03-29 14:41:35 +0900 ndpi.flow: {"protocol":"UDP","src_addr":"192.0.2.4,"src_port":36605,"dst_addr":"192.0.2.6","dst_port":53,"master_protocol":5,"detected_protocol":126,"protocol_name":"DNS.Google","out_pkts":2,"out_bytes":152,"in_pkts":2,"in_bytes":488,"first_switched":1459230095,"last_switched":1459230095,"server_name":"www.google.co.jp"} 166 | ``` 167 | 168 | ### Daemon 169 | config file: `/etc/sysconfig/ndff` 170 | 171 | ``` 172 | $ sudo vim /etc/sysconfig/ndff 173 | # Config file for ndff startup 174 | # Options passed to the ndff program 175 | OPTIONS="-i eth0 -s 127.0.0.1 -p 24224 -t ndff.flow -m json" 176 | 177 | $ sudo /etc/rc.d/init.d/ndff start 178 | Starting ndff: [ OK ] 179 | ``` 180 | 181 | Contributing 182 | -------- 183 | 184 | 1. Fork it 185 | 2. Create your feature branch (`git checkout -b my-new-feature`) 186 | 3. Commit your changes (`git commit -am 'Added some feature'`) 187 | 4. Push to the branch (`git push origin my-new-feature`) 188 | 5. Create new Pull Request 189 | 190 | License 191 | -------- 192 | `ndff` is licensed under The GNU General Public License Version 3. See the [LICENSE](https://github.com/knqyf263/ndff/blob/master/LICENSE) file for details. 193 | -------------------------------------------------------------------------------- /autogen.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | 4 | /bin/rm -f configure config.h config.h.in src/lib/Makefile.in 5 | 6 | AUTOCONF=$(which autoconf) 7 | AUTOMAKE=$(which automake) 8 | LIBTOOL=$(which libtool) 9 | LIBTOOLIZE=$(which libtoolize) 10 | AUTORECONF=$(which autoreconf) 11 | 12 | if test -z $AUTOCONF; then 13 | echo "autoconf is missing: please install it and try again" 14 | exit 15 | fi 16 | 17 | if test -z $AUTOMAKE; then 18 | echo "automake is missing: please install it and try again" 19 | exit 20 | fi 21 | 22 | if test -z $LIBTOOL && test -z $LIBTOOLIZE ; then 23 | echo "libtool and libtoolize is missing: please install it and try again" 24 | exit 25 | fi 26 | 27 | if test -z $AUTORECONF; then 28 | echo "autoreconf is missing: please install it and try again" 29 | exit 30 | fi 31 | 32 | autoreconf -ivf 33 | ./configure 34 | -------------------------------------------------------------------------------- /configure.ac: -------------------------------------------------------------------------------- 1 | AC_INIT([ndff], [0.0.2]) 2 | 3 | AC_CONFIG_MACRO_DIR([m4]) 4 | 5 | AM_INIT_AUTOMAKE([foreign subdir-objects]) 6 | 7 | LT_INIT 8 | 9 | AC_PROG_CC 10 | AX_PTHREAD 11 | 12 | if test -d ".git"; then : 13 | GIT_TAG=`git log -1 --format=%h` 14 | GIT_DATE=`git log -1 --format=%cd` 15 | # 16 | # On CentOS 6 `git rev-list HEAD --count` does not work 17 | # 18 | # 19 | GIT_NUM=`git log --pretty=oneline | wc -l | tr -d '[[:space:]]'` 20 | GIT_BRANCH=`git rev-parse --abbrev-ref HEAD` 21 | GIT_RELEASE="${PACKAGE_VERSION}-${GIT_BRANCH}-${GIT_NUM}-${GIT_TAG}" 22 | else 23 | GIT_RELEASE="${PACKAGE_VERSION}" 24 | GIT_DATE=`date` 25 | fi 26 | 27 | AC_DEFINE_UNQUOTED(NDFF_GIT_RELEASE, "${GIT_RELEASE}", [GIT Release]) 28 | AC_DEFINE_UNQUOTED(NDFF_GIT_DATE, "${GIT_DATE}", [Last GIT change]) 29 | 30 | AC_CHECK_HEADERS([netinet/in.h stdint.h stdlib.h string.h unistd.h]) 31 | 32 | PCAP_HOME=$HOME/PF_RING/userland 33 | 34 | if test -d $PCAP_HOME; then : 35 | echo -n "" 36 | else 37 | PCAP_HOME=`pwd`/../../PF_RING/userland 38 | fi 39 | SHORT_MACHINE=`uname -m | cut -b1-3` 40 | if test $SHORT_MACHINE = "arm"; then 41 | LIBNUMA="" 42 | else 43 | LIBNUMA="-lnuma" 44 | fi 45 | 46 | if test -f $PCAP_HOME/libpcap/libpcap.a; then : 47 | echo "Using libpcap from $PCAP_HOME" 48 | PCAP_INC="-I $PCAP_HOME/libpcap" 49 | PCAP_LIB="$PCAP_HOME/libpcap/libpcap.a $PCAP_HOME/lib/libpfring.a $LIBNUMA `$PCAP_HOME/lib/pfring_config --libs`" 50 | 51 | AC_CHECK_LIB([rt], [clock_gettime], [PCAP_LIB="$PCAP_LIB -lrt"]) 52 | AC_CHECK_LIB([nl], [nl_handle_alloc], [PCAP_LIB="$PCAP_LIB -lnl"]) 53 | else 54 | AC_CHECK_LIB([pcap], [pcap_open_live], [PCAP_LIB="-lpcap"]) 55 | 56 | if test $ac_cv_lib_pcap_pcap_open_live = "no"; then : 57 | echo "" 58 | echo "ERROR: Missing libpcap(-dev) library required to compile the example application" 59 | echo "ERROR: Please install it and try again" 60 | exit 61 | fi 62 | 63 | if test -f /usr/local/lib/libpfring.so; then : 64 | PCAP_LIB="$PCAP_LIB -lpfring" 65 | fi 66 | 67 | fi 68 | 69 | export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig 70 | PKG_PROG_PKG_CONFIG 71 | PKG_CHECK_MODULES(NDPI, libndpi) 72 | PKG_CHECK_MODULES(MSGPACK, msgpack) 73 | 74 | if test -d /usr/local/include/json-c/; then : 75 | CFLAGS="$CFLAGS -I/usr/local/include/json-c/" 76 | LDFLAGS="$LDFLAGS -L/usr/local/lib -ljson-c" 77 | else 78 | if ! test -z "$PKG_CONFIG"; then : 79 | PKG_CHECK_MODULES(JSON_C, json-c) 80 | fi 81 | fi 82 | 83 | 84 | OLD_LIBS=$LIBS 85 | LIBS="-L/opt/napatech3/lib $LIBS" 86 | AC_CHECK_LIB([ntapi], 87 | [NT_Init], 88 | [PCAP_LIB="$PCAP_LIB -L/opt/napatech3/lib -lntapi"], 89 | [], [] ) 90 | LIBS=$OLD_LIBS 91 | 92 | AC_CHECK_LIB(json-c, json_object_new_object, ,) 93 | AC_CHECK_LIB(msgpackc, main, ,) 94 | AC_CHECK_LIB(pthread, pthread_setaffinity_np, AC_DEFINE_UNQUOTED(HAVE_PTHREAD_SETAFFINITY_NP, 1, [libc has pthread_setaffinity_np])) 95 | AC_CHECK_LIB(ndpi, main, AC_DEFINE_UNQUOTED(HAVE_NDPI, 1, [The nDPI library is present]), [echo "Error: Required library libndpi not found. Install it and try again."; exit -1]) 96 | 97 | AC_CONFIG_FILES([Makefile]) 98 | AC_CONFIG_HEADERS(config.h) 99 | AC_SUBST(GIT_RELEASE) 100 | AC_SUBST(SVN_DATE) 101 | AC_SUBST(PCAP_INC) 102 | AC_SUBST(PCAP_LIB) 103 | AC_SUBST(PTHREAD) 104 | AC_SUBST(HAVE_PTHREAD_SETAFFINITY_NP) 105 | 106 | AC_CONFIG_FILES([ndffd], [chmod +x ndffd]) 107 | 108 | AC_OUTPUT 109 | -------------------------------------------------------------------------------- /m4/ax_pthread.m4: -------------------------------------------------------------------------------- 1 | # =========================================================================== 2 | # http://www.gnu.org/software/autoconf-archive/ax_pthread.html 3 | # =========================================================================== 4 | # 5 | # SYNOPSIS 6 | # 7 | # AX_PTHREAD([ACTION-IF-FOUND[, ACTION-IF-NOT-FOUND]]) 8 | # 9 | # DESCRIPTION 10 | # 11 | # This macro figures out how to build C programs using POSIX threads. It 12 | # sets the PTHREAD_LIBS output variable to the threads library and linker 13 | # flags, and the PTHREAD_CFLAGS output variable to any special C compiler 14 | # flags that are needed. (The user can also force certain compiler 15 | # flags/libs to be tested by setting these environment variables.) 16 | # 17 | # Also sets PTHREAD_CC to any special C compiler that is needed for 18 | # multi-threaded programs (defaults to the value of CC otherwise). (This 19 | # is necessary on AIX to use the special cc_r compiler alias.) 20 | # 21 | # NOTE: You are assumed to not only compile your program with these flags, 22 | # but also link it with them as well. e.g. you should link with 23 | # $PTHREAD_CC $CFLAGS $PTHREAD_CFLAGS $LDFLAGS ... $PTHREAD_LIBS $LIBS 24 | # 25 | # If you are only building threads programs, you may wish to use these 26 | # variables in your default LIBS, CFLAGS, and CC: 27 | # 28 | # LIBS="$PTHREAD_LIBS $LIBS" 29 | # CFLAGS="$CFLAGS $PTHREAD_CFLAGS" 30 | # CC="$PTHREAD_CC" 31 | # 32 | # In addition, if the PTHREAD_CREATE_JOINABLE thread-attribute constant 33 | # has a nonstandard name, defines PTHREAD_CREATE_JOINABLE to that name 34 | # (e.g. PTHREAD_CREATE_UNDETACHED on AIX). 35 | # 36 | # Also HAVE_PTHREAD_PRIO_INHERIT is defined if pthread is found and the 37 | # PTHREAD_PRIO_INHERIT symbol is defined when compiling with 38 | # PTHREAD_CFLAGS. 39 | # 40 | # ACTION-IF-FOUND is a list of shell commands to run if a threads library 41 | # is found, and ACTION-IF-NOT-FOUND is a list of commands to run it if it 42 | # is not found. If ACTION-IF-FOUND is not specified, the default action 43 | # will define HAVE_PTHREAD. 44 | # 45 | # Please let the authors know if this macro fails on any platform, or if 46 | # you have any other suggestions or comments. This macro was based on work 47 | # by SGJ on autoconf scripts for FFTW (http://www.fftw.org/) (with help 48 | # from M. Frigo), as well as ac_pthread and hb_pthread macros posted by 49 | # Alejandro Forero Cuervo to the autoconf macro repository. We are also 50 | # grateful for the helpful feedback of numerous users. 51 | # 52 | # Updated for Autoconf 2.68 by Daniel Richard G. 53 | # 54 | # LICENSE 55 | # 56 | # Copyright (c) 2008 Steven G. Johnson 57 | # Copyright (c) 2011 Daniel Richard G. 58 | # 59 | # This program is free software: you can redistribute it and/or modify it 60 | # under the terms of the GNU General Public License as published by the 61 | # Free Software Foundation, either version 3 of the License, or (at your 62 | # option) any later version. 63 | # 64 | # This program is distributed in the hope that it will be useful, but 65 | # WITHOUT ANY WARRANTY; without even the implied warranty of 66 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General 67 | # Public License for more details. 68 | # 69 | # You should have received a copy of the GNU General Public License along 70 | # with this program. If not, see . 71 | # 72 | # As a special exception, the respective Autoconf Macro's copyright owner 73 | # gives unlimited permission to copy, distribute and modify the configure 74 | # scripts that are the output of Autoconf when processing the Macro. You 75 | # need not follow the terms of the GNU General Public License when using 76 | # or distributing such scripts, even though portions of the text of the 77 | # Macro appear in them. The GNU General Public License (GPL) does govern 78 | # all other use of the material that constitutes the Autoconf Macro. 79 | # 80 | # This special exception to the GPL applies to versions of the Autoconf 81 | # Macro released by the Autoconf Archive. When you make and distribute a 82 | # modified version of the Autoconf Macro, you may extend this special 83 | # exception to the GPL to apply to your modified version as well. 84 | 85 | #serial 21 86 | 87 | AU_ALIAS([ACX_PTHREAD], [AX_PTHREAD]) 88 | AC_DEFUN([AX_PTHREAD], [ 89 | AC_REQUIRE([AC_CANONICAL_HOST]) 90 | AC_LANG_PUSH([C]) 91 | ax_pthread_ok=no 92 | 93 | # We used to check for pthread.h first, but this fails if pthread.h 94 | # requires special compiler flags (e.g. on True64 or Sequent). 95 | # It gets checked for in the link test anyway. 96 | 97 | # First of all, check if the user has set any of the PTHREAD_LIBS, 98 | # etcetera environment variables, and if threads linking works using 99 | # them: 100 | if test x"$PTHREAD_LIBS$PTHREAD_CFLAGS" != x; then 101 | save_CFLAGS="$CFLAGS" 102 | CFLAGS="$CFLAGS $PTHREAD_CFLAGS" 103 | save_LIBS="$LIBS" 104 | LIBS="$PTHREAD_LIBS $LIBS" 105 | AC_MSG_CHECKING([for pthread_join in LIBS=$PTHREAD_LIBS with CFLAGS=$PTHREAD_CFLAGS]) 106 | AC_TRY_LINK_FUNC([pthread_join], [ax_pthread_ok=yes]) 107 | AC_MSG_RESULT([$ax_pthread_ok]) 108 | if test x"$ax_pthread_ok" = xno; then 109 | PTHREAD_LIBS="" 110 | PTHREAD_CFLAGS="" 111 | fi 112 | LIBS="$save_LIBS" 113 | CFLAGS="$save_CFLAGS" 114 | fi 115 | 116 | # We must check for the threads library under a number of different 117 | # names; the ordering is very important because some systems 118 | # (e.g. DEC) have both -lpthread and -lpthreads, where one of the 119 | # libraries is broken (non-POSIX). 120 | 121 | # Create a list of thread flags to try. Items starting with a "-" are 122 | # C compiler flags, and other items are library names, except for "none" 123 | # which indicates that we try without any flags at all, and "pthread-config" 124 | # which is a program returning the flags for the Pth emulation library. 125 | 126 | ax_pthread_flags="pthreads none -Kthread -kthread lthread -pthread -pthreads -mthreads pthread --thread-safe -mt pthread-config" 127 | 128 | # The ordering *is* (sometimes) important. Some notes on the 129 | # individual items follow: 130 | 131 | # pthreads: AIX (must check this before -lpthread) 132 | # none: in case threads are in libc; should be tried before -Kthread and 133 | # other compiler flags to prevent continual compiler warnings 134 | # -Kthread: Sequent (threads in libc, but -Kthread needed for pthread.h) 135 | # -kthread: FreeBSD kernel threads (preferred to -pthread since SMP-able) 136 | # lthread: LinuxThreads port on FreeBSD (also preferred to -pthread) 137 | # -pthread: Linux/gcc (kernel threads), BSD/gcc (userland threads) 138 | # -pthreads: Solaris/gcc 139 | # -mthreads: Mingw32/gcc, Lynx/gcc 140 | # -mt: Sun Workshop C (may only link SunOS threads [-lthread], but it 141 | # doesn't hurt to check since this sometimes defines pthreads too; 142 | # also defines -D_REENTRANT) 143 | # ... -mt is also the pthreads flag for HP/aCC 144 | # pthread: Linux, etcetera 145 | # --thread-safe: KAI C++ 146 | # pthread-config: use pthread-config program (for GNU Pth library) 147 | 148 | case ${host_os} in 149 | solaris*) 150 | 151 | # On Solaris (at least, for some versions), libc contains stubbed 152 | # (non-functional) versions of the pthreads routines, so link-based 153 | # tests will erroneously succeed. (We need to link with -pthreads/-mt/ 154 | # -lpthread.) (The stubs are missing pthread_cleanup_push, or rather 155 | # a function called by this macro, so we could check for that, but 156 | # who knows whether they'll stub that too in a future libc.) So, 157 | # we'll just look for -pthreads and -lpthread first: 158 | 159 | ax_pthread_flags="-pthreads pthread -mt -pthread $ax_pthread_flags" 160 | ;; 161 | 162 | darwin*) 163 | ax_pthread_flags="-pthread $ax_pthread_flags" 164 | ;; 165 | esac 166 | 167 | # Clang doesn't consider unrecognized options an error unless we specify 168 | # -Werror. We throw in some extra Clang-specific options to ensure that 169 | # this doesn't happen for GCC, which also accepts -Werror. 170 | 171 | AC_MSG_CHECKING([if compiler needs -Werror to reject unknown flags]) 172 | save_CFLAGS="$CFLAGS" 173 | ax_pthread_extra_flags="-Werror" 174 | CFLAGS="$CFLAGS $ax_pthread_extra_flags -Wunknown-warning-option -Wsizeof-array-argument" 175 | AC_COMPILE_IFELSE([AC_LANG_PROGRAM([int foo(void);],[foo()])], 176 | [AC_MSG_RESULT([yes])], 177 | [ax_pthread_extra_flags= 178 | AC_MSG_RESULT([no])]) 179 | CFLAGS="$save_CFLAGS" 180 | 181 | if test x"$ax_pthread_ok" = xno; then 182 | for flag in $ax_pthread_flags; do 183 | 184 | case $flag in 185 | none) 186 | AC_MSG_CHECKING([whether pthreads work without any flags]) 187 | ;; 188 | 189 | -*) 190 | AC_MSG_CHECKING([whether pthreads work with $flag]) 191 | PTHREAD_CFLAGS="$flag" 192 | ;; 193 | 194 | pthread-config) 195 | AC_CHECK_PROG([ax_pthread_config], [pthread-config], [yes], [no]) 196 | if test x"$ax_pthread_config" = xno; then continue; fi 197 | PTHREAD_CFLAGS="`pthread-config --cflags`" 198 | PTHREAD_LIBS="`pthread-config --ldflags` `pthread-config --libs`" 199 | ;; 200 | 201 | *) 202 | AC_MSG_CHECKING([for the pthreads library -l$flag]) 203 | PTHREAD_LIBS="-l$flag" 204 | ;; 205 | esac 206 | 207 | save_LIBS="$LIBS" 208 | save_CFLAGS="$CFLAGS" 209 | LIBS="$PTHREAD_LIBS $LIBS" 210 | CFLAGS="$CFLAGS $PTHREAD_CFLAGS $ax_pthread_extra_flags" 211 | 212 | # Check for various functions. We must include pthread.h, 213 | # since some functions may be macros. (On the Sequent, we 214 | # need a special flag -Kthread to make this header compile.) 215 | # We check for pthread_join because it is in -lpthread on IRIX 216 | # while pthread_create is in libc. We check for pthread_attr_init 217 | # due to DEC craziness with -lpthreads. We check for 218 | # pthread_cleanup_push because it is one of the few pthread 219 | # functions on Solaris that doesn't have a non-functional libc stub. 220 | # We try pthread_create on general principles. 221 | AC_LINK_IFELSE([AC_LANG_PROGRAM([#include 222 | static void routine(void *a) { a = 0; } 223 | static void *start_routine(void *a) { return a; }], 224 | [pthread_t th; pthread_attr_t attr; 225 | pthread_create(&th, 0, start_routine, 0); 226 | pthread_join(th, 0); 227 | pthread_attr_init(&attr); 228 | pthread_cleanup_push(routine, 0); 229 | pthread_cleanup_pop(0) /* ; */])], 230 | [ax_pthread_ok=yes], 231 | []) 232 | 233 | LIBS="$save_LIBS" 234 | CFLAGS="$save_CFLAGS" 235 | 236 | AC_MSG_RESULT([$ax_pthread_ok]) 237 | if test "x$ax_pthread_ok" = xyes; then 238 | break; 239 | fi 240 | 241 | PTHREAD_LIBS="" 242 | PTHREAD_CFLAGS="" 243 | done 244 | fi 245 | 246 | # Various other checks: 247 | if test "x$ax_pthread_ok" = xyes; then 248 | save_LIBS="$LIBS" 249 | LIBS="$PTHREAD_LIBS $LIBS" 250 | save_CFLAGS="$CFLAGS" 251 | CFLAGS="$CFLAGS $PTHREAD_CFLAGS" 252 | 253 | # Detect AIX lossage: JOINABLE attribute is called UNDETACHED. 254 | AC_MSG_CHECKING([for joinable pthread attribute]) 255 | attr_name=unknown 256 | for attr in PTHREAD_CREATE_JOINABLE PTHREAD_CREATE_UNDETACHED; do 257 | AC_LINK_IFELSE([AC_LANG_PROGRAM([#include ], 258 | [int attr = $attr; return attr /* ; */])], 259 | [attr_name=$attr; break], 260 | []) 261 | done 262 | AC_MSG_RESULT([$attr_name]) 263 | if test "$attr_name" != PTHREAD_CREATE_JOINABLE; then 264 | AC_DEFINE_UNQUOTED([PTHREAD_CREATE_JOINABLE], [$attr_name], 265 | [Define to necessary symbol if this constant 266 | uses a non-standard name on your system.]) 267 | fi 268 | 269 | AC_MSG_CHECKING([if more special flags are required for pthreads]) 270 | flag=no 271 | case ${host_os} in 272 | aix* | freebsd* | darwin*) flag="-D_THREAD_SAFE";; 273 | osf* | hpux*) flag="-D_REENTRANT";; 274 | solaris*) 275 | if test "$GCC" = "yes"; then 276 | flag="-D_REENTRANT" 277 | else 278 | # TODO: What about Clang on Solaris? 279 | flag="-mt -D_REENTRANT" 280 | fi 281 | ;; 282 | esac 283 | AC_MSG_RESULT([$flag]) 284 | if test "x$flag" != xno; then 285 | PTHREAD_CFLAGS="$flag $PTHREAD_CFLAGS" 286 | fi 287 | 288 | AC_CACHE_CHECK([for PTHREAD_PRIO_INHERIT], 289 | [ax_cv_PTHREAD_PRIO_INHERIT], [ 290 | AC_LINK_IFELSE([AC_LANG_PROGRAM([[#include ]], 291 | [[int i = PTHREAD_PRIO_INHERIT;]])], 292 | [ax_cv_PTHREAD_PRIO_INHERIT=yes], 293 | [ax_cv_PTHREAD_PRIO_INHERIT=no]) 294 | ]) 295 | AS_IF([test "x$ax_cv_PTHREAD_PRIO_INHERIT" = "xyes"], 296 | [AC_DEFINE([HAVE_PTHREAD_PRIO_INHERIT], [1], [Have PTHREAD_PRIO_INHERIT.])]) 297 | 298 | LIBS="$save_LIBS" 299 | CFLAGS="$save_CFLAGS" 300 | 301 | # More AIX lossage: compile with *_r variant 302 | if test "x$GCC" != xyes; then 303 | case $host_os in 304 | aix*) 305 | AS_CASE(["x/$CC"], 306 | [x*/c89|x*/c89_128|x*/c99|x*/c99_128|x*/cc|x*/cc128|x*/xlc|x*/xlc_v6|x*/xlc128|x*/xlc128_v6], 307 | [#handle absolute path differently from PATH based program lookup 308 | AS_CASE(["x$CC"], 309 | [x/*], 310 | [AS_IF([AS_EXECUTABLE_P([${CC}_r])],[PTHREAD_CC="${CC}_r"])], 311 | [AC_CHECK_PROGS([PTHREAD_CC],[${CC}_r],[$CC])])]) 312 | ;; 313 | esac 314 | fi 315 | fi 316 | 317 | test -n "$PTHREAD_CC" || PTHREAD_CC="$CC" 318 | 319 | AC_SUBST([PTHREAD_LIBS]) 320 | AC_SUBST([PTHREAD_CFLAGS]) 321 | AC_SUBST([PTHREAD_CC]) 322 | 323 | # Finally, execute ACTION-IF-FOUND/ACTION-IF-NOT-FOUND: 324 | if test x"$ax_pthread_ok" = xyes; then 325 | ifelse([$1],,[AC_DEFINE([HAVE_PTHREAD],[1],[Define if you have POSIX threads libraries and header files.])],[$1]) 326 | : 327 | else 328 | ax_pthread_ok=no 329 | $2 330 | fi 331 | AC_LANG_POP 332 | ])dnl AX_PTHREAD 333 | -------------------------------------------------------------------------------- /ndff.c: -------------------------------------------------------------------------------- 1 | /* 2 | * ndff.c 3 | * 4 | * Copyright 2002-2006 Damien Miller All rights reserved. 5 | * Copyright (C) 2011-15 - ntop.org 6 | * Copyright (C) 2009-2011 by ipoque GmbH 7 | * Copyright (C) 2014 - Matteo Bogo (JSON support) 8 | * Copyright (C) 2016 Teppei Fukuda 9 | * Copyright (C) 2016 DeNA Co., Ltd. 10 | * 11 | * Distributed under The GNU General Public License Version 3. 12 | * (See accompanying file LICENSE or copy at 13 | * http://www.gnu.org/licenses/) 14 | */ 15 | 16 | #ifdef linux 17 | #define _GNU_SOURCE 18 | #include 19 | #endif 20 | #include 21 | #include 22 | #ifdef WIN32 23 | #include /* winsock.h is included automatically */ 24 | #include 25 | #include 26 | #include 27 | #define getopt getopt____ 28 | #else 29 | #include 30 | #include 31 | #endif 32 | #include 33 | #include 34 | #include 35 | #include 36 | #include 37 | #include 38 | #include 39 | #include 40 | #include 41 | #include 42 | #include 43 | #include 44 | #include 45 | 46 | #include "config.h" 47 | #include 48 | 49 | #ifdef HAVE_LIBMSGPACKC 50 | #include 51 | #endif 52 | 53 | #ifdef HAVE_LIBJSON_C 54 | #include 55 | #endif 56 | 57 | 58 | /* Timeouts */ 59 | #define TCP_TIMEOUT 3600 60 | #define TCP_RST_TIMEOUT 120 61 | #define TCP_FIN_TIMEOUT 300 62 | #define UDP_TIMEOUT 300 63 | #define ICMP_TIMEOUT 300 64 | #define GENERAL_TIMEOUT 3600 65 | #define MAXIMUM_LIFETIME (3600*24*7) 66 | 67 | #define MAX_NUM_READER_THREADS 16 68 | #define IDLE_SCAN_PERIOD 10 /* msec (use detection_tick_resolution = 1000) */ 69 | #define IDLE_SCAN_BUDGET 1024 70 | #define NUM_ROOTS 512 71 | #define GTP_U_V1_PORT 2152 72 | #define TZSP_PORT 37008 73 | #define MAX_NDPI_FLOWS 200000000 74 | 75 | #ifndef ETH_P_IP 76 | #define ETH_P_IP 0x0800 /* IPv4 */ 77 | #endif 78 | 79 | #ifndef ETH_P_IPv6 80 | #define ETH_P_IPV6 0x86dd /* IPv6 */ 81 | #endif 82 | 83 | #define SLARP 0x8035 /* Cisco Slarp */ 84 | #define CISCO_D_PROTO 0x2000 /* Cisco Discovery Protocol */ 85 | 86 | #define VLAN 0x8100 87 | #define MPLS_UNI 0x8847 88 | #define MPLS_MULTI 0x8848 89 | #define PPPoE 0x8864 90 | #define SNAP 0xaa 91 | 92 | /* mask for FCF */ 93 | #define WIFI_DATA 0x2 /* 0000 0010 */ 94 | #define FCF_TYPE(fc) (((fc) >> 2) & 0x3) /* 0000 0011 = 0x3 */ 95 | #define FCF_SUBTYPE(fc) (((fc) >> 4) & 0xF) /* 0000 1111 = 0xF */ 96 | #define FCF_TO_DS(fc) ((fc) & 0x0100) 97 | #define FCF_FROM_DS(fc) ((fc) & 0x0200) 98 | 99 | /* mask for Bad FCF presence */ 100 | #define BAD_FCS 0x50 /* 0101 0000 */ 101 | 102 | /** 103 | * @brief Set main components necessary to the detection 104 | */ 105 | static void setup_detection(u_int16_t thread_id); 106 | static int setup_socket(u_int16_t thread_id); 107 | 108 | /** 109 | * Client parameters 110 | */ 111 | static char *_pcap_file[MAX_NUM_READER_THREADS]; /**< Ingress pcap file/interafaces */ 112 | static FILE *playlist_fp[MAX_NUM_READER_THREADS] = { NULL }; /**< Ingress playlist */ 113 | static FILE *results_file = NULL; 114 | static char *results_path = NULL; 115 | static char *_server_addr = NULL; /**< server ip address */ 116 | static int _server_port = 24224; /**< server port */ 117 | static char *_tag = "ndpi.flow"; /**< tag for fluentd */ 118 | static char *_bpf_filter = NULL; /**< bpf filter */ 119 | static char *_protoFilePath = NULL; /**< Protocol file path */ 120 | static u_int8_t live_capture = 0; 121 | static u_int8_t undetected_flows_deleted = 0; 122 | /** 123 | * User preferences 124 | */ 125 | static u_int8_t enable_protocol_guess = 1, verbose = 0, nDPI_traceLevel = 0, json_flag = 0, msgpack_flag = 0, dryrun_flag = 0; 126 | static u_int16_t decode_tunnels = 0; 127 | static u_int8_t shutdown_app = 0, quiet_mode = 0; 128 | static u_int8_t num_threads = 1; 129 | static u_int32_t current_ndpi_memory = 0, max_ndpi_memory = 0; 130 | static char* server; 131 | #ifdef linux 132 | static int core_affinity[MAX_NUM_READER_THREADS]; 133 | #endif 134 | 135 | static struct timeval pcap_start, pcap_end; 136 | 137 | /** 138 | * Detection parameters 139 | */ 140 | static u_int32_t detection_tick_resolution = 1000; 141 | 142 | static u_int32_t num_flows; 143 | 144 | struct reader_thread { 145 | struct ndpi_detection_module_struct *ndpi_struct; 146 | void *ndpi_flows_root[NUM_ROOTS]; 147 | char _pcap_error_buffer[PCAP_ERRBUF_SIZE]; 148 | pcap_t *_pcap_handle; 149 | u_int64_t last_time; 150 | u_int64_t last_idle_scan_time; 151 | u_int32_t idle_scan_idx; 152 | u_int32_t num_idle_flows; 153 | pthread_t pthread; 154 | int _pcap_datalink_type; 155 | 156 | struct ndpi_flow *idle_flows[IDLE_SCAN_BUDGET]; 157 | }; 158 | 159 | static struct reader_thread ndpi_thread_info[MAX_NUM_READER_THREADS]; 160 | 161 | /** 162 | * @brief ID tracking 163 | */ 164 | typedef struct ndpi_id { 165 | u_int8_t ip[4]; // Ip address 166 | struct ndpi_id_struct *ndpi_id; // nDpi worker structure 167 | } ndpi_id_t; 168 | 169 | static u_int32_t size_id_struct = 0; // ID tracking structure size 170 | 171 | // flow tracking 172 | typedef struct ndpi_flow { 173 | u_int32_t lower_ip; 174 | u_int32_t upper_ip; 175 | u_int16_t lower_port; 176 | u_int16_t upper_port; 177 | u_int8_t detection_completed, protocol; 178 | u_int16_t vlan_id; 179 | struct ndpi_flow_struct *ndpi_flow; 180 | char lower_name[48], upper_name[48]; 181 | u_int8_t ip_version; 182 | u_int64_t first_seen; 183 | u_int64_t last_seen; 184 | u_int64_t expires_at; 185 | u_int64_t out_bytes; 186 | u_int64_t in_bytes; 187 | u_int32_t out_pkts; 188 | u_int32_t in_pkts; 189 | u_int8_t src_to_dst_direction; 190 | u_int8_t fin_rst_received; 191 | 192 | // result only, not used for flow identification 193 | ndpi_protocol detected_protocol; 194 | 195 | char host_server_name[192]; 196 | 197 | struct { 198 | char client_certificate[48], server_certificate[48]; 199 | } ssl; 200 | 201 | void *src_id, *dst_id; 202 | } ndpi_flow_t; 203 | 204 | 205 | static u_int32_t size_flow_struct = 0; 206 | 207 | 208 | static char _sockfd[MAX_NUM_READER_THREADS]; 209 | 210 | static void help(u_int long_help) { 211 | printf("ndff -i [-s ] [-m ] [-f ]\n" 212 | " [-p ][-P ][-t ][-q][-d][-D][-h][-T][-v ]\n" 213 | " [-n ] [-w ] \n\n" 214 | "Usage:\n" 215 | " -i | Specify a pcap file/playlist to read packets from or a device for live capture (comma-separated list)\n" 216 | " -m | Specify a protocol to send messages to the server (json or msgpack)\n" 217 | " -f | Specify a BPF filter for filtering selected traffic\n" 218 | " -s | Specify a server for fluentd (If not, ndff runs in the dry-run mode)\n" 219 | " -p | Specify a port for fluentd (default: 24224)\n" 220 | " -P .protos | Specify a protocol file (eg. protos.txt)\n" 221 | " -n | Number of threads. Default: number of interfaces in -i. Ignored with pcap files.\n" 222 | #ifdef linux 223 | " -g | Thread affinity mask (one core id per thread)\n" 224 | #endif 225 | " -d | Daemonize (run in background)\n" 226 | " -D | Disable protocol guess and use only DPI\n" 227 | " -q | Quiet mode\n" 228 | " -t | Specify a tag for fluentd (default: ndpi.flow)\n" 229 | " -T | Dissect GTP/TZSP tunnels\n" 230 | " -r | Print nDPI version and git revision\n" 231 | " -w | Write test output on the specified file. This is useful for\n" 232 | " | testing purposes in order to compare results across runs\n" 233 | " -h | This help\n" 234 | " -v <1|2> | Verbose 'unknown protocol' packet print. 1=verbose, 2=very verbose\n"); 235 | 236 | if(long_help) { 237 | printf("\n\nSupported protocols:\n"); 238 | num_threads = 1; 239 | setup_detection(0); 240 | ndpi_dump_protocols(ndpi_thread_info[0].ndpi_struct); 241 | } 242 | 243 | exit(!long_help); 244 | } 245 | 246 | /* ***************************************************** */ 247 | 248 | void output(int priority, const char *format, ... ) { 249 | va_list arg; 250 | 251 | va_start(arg, format); 252 | if(!quiet_mode){ 253 | va_list arg2; 254 | va_copy (arg2, arg); 255 | vprintf(format, arg2); 256 | va_end(arg2); 257 | } 258 | vsyslog(priority, format, arg); 259 | va_end(arg); 260 | } 261 | 262 | /* ***************************************************** */ 263 | 264 | static void parse_options(int argc, char **argv) { 265 | char *__pcap_file = NULL, *bind_mask = NULL; 266 | int thread_id, opt; 267 | #ifdef linux 268 | u_int num_cores = sysconf(_SC_NPROCESSORS_ONLN); 269 | #endif 270 | 271 | while ((opt = getopt(argc, argv, "dDf:g:i:hp:P:l:s:t:Tv:V:n:rp:m:w:q")) != EOF) { 272 | switch (opt) { 273 | case 'd': 274 | if (daemon(0, 0) != 0) { 275 | output(LOG_ERR, "%s\n", "[ERROR] daemonize failed"); 276 | exit(1); 277 | } 278 | break; 279 | 280 | case 'D': 281 | enable_protocol_guess = 0; 282 | break; 283 | 284 | case 'i': 285 | _pcap_file[0] = optarg; 286 | break; 287 | 288 | case 'f': 289 | _bpf_filter = optarg; 290 | break; 291 | 292 | case 'g': 293 | bind_mask = optarg; 294 | break; 295 | 296 | case 'n': 297 | num_threads = atoi(optarg); 298 | break; 299 | 300 | case 'p': 301 | _server_port = atoi(optarg); 302 | break; 303 | 304 | case 'P': 305 | _protoFilePath = optarg; 306 | break; 307 | 308 | case 's': 309 | _server_addr = optarg; 310 | break; 311 | 312 | case 't': 313 | _tag = optarg; 314 | break; 315 | 316 | case 'T': 317 | decode_tunnels = 1; 318 | break; 319 | 320 | case 'r': 321 | printf("%s\n- nDPI (%s)\n", PACKAGE_STRING, ndpi_revision()); 322 | exit(0); 323 | 324 | case 'v': 325 | verbose = atoi(optarg); 326 | break; 327 | 328 | case 'V': 329 | printf("%d\n",atoi(optarg) ); 330 | nDPI_traceLevel = atoi(optarg); 331 | break; 332 | 333 | case 'h': 334 | help(1); 335 | break; 336 | 337 | case 'm': 338 | if(strcmp(optarg, "json") == 0){ 339 | #ifndef HAVE_LIBJSON_C 340 | output(LOG_WARNING, "%s\n", "[WARN] this copy of ndff has been compiled without JSON-C: json export disabled"); 341 | #else 342 | json_flag = 1; 343 | #endif 344 | }else if(strcmp((char *)optarg, "msgpack") == 0){ 345 | #ifndef HAVE_LIBMSGPACKC 346 | output(LOG_WARNING, "%s\n", "[WARN] this copy of ndff has been compiled without msgpack-c: msgpack export disabled"); 347 | #else 348 | msgpack_flag = 1; 349 | #endif 350 | }else{ 351 | help(0); 352 | } 353 | break; 354 | 355 | case 'w': 356 | results_path = strdup(optarg); 357 | if((results_file = fopen(results_path, "w")) == NULL) { 358 | output(LOG_ERR, "[ERROR] Unable to write in file %s: quitting\n", results_path); 359 | return; 360 | } 361 | break; 362 | 363 | case 'q': 364 | quiet_mode = 1; 365 | break; 366 | 367 | default: 368 | help(0); 369 | break; 370 | } 371 | } 372 | 373 | // check parameters 374 | if(_pcap_file[0] == NULL || strcmp(_pcap_file[0], "") == 0) { 375 | help(0); 376 | }else if(_server_addr == NULL || strcmp(_server_addr, "") == 0) { 377 | output(LOG_WARNING, "%s\n", "[WARN] No server is specified. This is dry-run mode."); 378 | dryrun_flag = 1; 379 | }else if(!json_flag && !msgpack_flag){ 380 | output(LOG_WARNING, "%s\n", "[WARN] No protocol is specified. This is dry-run mode."); 381 | dryrun_flag = 1; 382 | } 383 | 384 | if(strchr(_pcap_file[0], ',')) { /* multiple ingress interfaces */ 385 | num_threads = 0; /* setting number of threads = number of interfaces */ 386 | __pcap_file = strtok(_pcap_file[0], ","); 387 | while (__pcap_file != NULL && num_threads < MAX_NUM_READER_THREADS) { 388 | _pcap_file[num_threads++] = __pcap_file; 389 | __pcap_file = strtok(NULL, ","); 390 | } 391 | } else { 392 | if(num_threads > MAX_NUM_READER_THREADS) num_threads = MAX_NUM_READER_THREADS; 393 | for(thread_id = 1; thread_id < num_threads; thread_id++) 394 | _pcap_file[thread_id] = _pcap_file[0]; 395 | } 396 | 397 | #ifdef linux 398 | for(thread_id = 0; thread_id < num_threads; thread_id++) 399 | core_affinity[thread_id] = -1; 400 | 401 | if(num_cores > 1 && bind_mask != NULL) { 402 | char *core_id = strtok(bind_mask, ":"); 403 | thread_id = 0; 404 | while (core_id != NULL && thread_id < num_threads) { 405 | core_affinity[thread_id++] = atoi(core_id) % num_cores; 406 | core_id = strtok(NULL, ":"); 407 | } 408 | } 409 | #endif 410 | } 411 | 412 | /* ***************************************************** */ 413 | 414 | static void debug_printf(u_int32_t protocol, void *id_struct, 415 | ndpi_log_level_t log_level, 416 | const char *format, ...) { 417 | va_list va_ap; 418 | #ifndef WIN32 419 | struct tm result; 420 | #endif 421 | 422 | if(log_level <= nDPI_traceLevel) { 423 | char buf[8192], out_buf[8192]; 424 | char theDate[32]; 425 | const char *extra_msg = ""; 426 | time_t theTime = time(NULL); 427 | 428 | va_start (va_ap, format); 429 | 430 | if(log_level == NDPI_LOG_ERROR) 431 | extra_msg = "ERROR: "; 432 | else if(log_level == NDPI_LOG_TRACE) 433 | extra_msg = "TRACE: "; 434 | else 435 | extra_msg = "DEBUG: "; 436 | 437 | memset(buf, 0, sizeof(buf)); 438 | strftime(theDate, 32, "%d/%b/%Y %H:%M:%S", localtime_r(&theTime,&result) ); 439 | vsnprintf(buf, sizeof(buf)-1, format, va_ap); 440 | 441 | snprintf(out_buf, sizeof(out_buf), "%s %s%s", theDate, extra_msg, buf); 442 | printf("%s", out_buf); 443 | fflush(stdout); 444 | } 445 | 446 | va_end(va_ap); 447 | } 448 | 449 | /* ***************************************************** */ 450 | 451 | static void *malloc_wrapper(size_t size) { 452 | current_ndpi_memory += size; 453 | 454 | if(current_ndpi_memory > max_ndpi_memory) 455 | max_ndpi_memory = current_ndpi_memory; 456 | 457 | return malloc(size); 458 | } 459 | 460 | /* ***************************************************** */ 461 | 462 | static void free_wrapper(void *freeable) { 463 | free(freeable); 464 | } 465 | 466 | /* ***************************************************** */ 467 | 468 | static char* ipProto2Name(u_short proto_id) { 469 | static char proto[8]; 470 | 471 | switch(proto_id) { 472 | case IPPROTO_TCP: 473 | return("TCP"); 474 | break; 475 | case IPPROTO_UDP: 476 | return("UDP"); 477 | break; 478 | case IPPROTO_ICMP: 479 | return("ICMP"); 480 | break; 481 | case IPPROTO_ICMPV6: 482 | return("ICMPV6"); 483 | break; 484 | case 112: 485 | return("VRRP"); 486 | break; 487 | case IPPROTO_IGMP: 488 | return("IGMP"); 489 | break; 490 | } 491 | 492 | snprintf(proto, sizeof(proto), "%u", proto_id); 493 | return(proto); 494 | } 495 | 496 | /* ***************************************************** */ 497 | 498 | /* 499 | * A faster replacement for inet_ntoa(). 500 | */ 501 | char* intoaV4(unsigned int addr, char* buf, u_short bufLen) { 502 | char *cp, *retStr; 503 | uint byte; 504 | int n; 505 | 506 | cp = &buf[bufLen]; 507 | *--cp = '\0'; 508 | 509 | n = 4; 510 | do { 511 | byte = addr & 0xff; 512 | *--cp = byte % 10 + '0'; 513 | byte /= 10; 514 | if(byte > 0) { 515 | *--cp = byte % 10 + '0'; 516 | byte /= 10; 517 | if(byte > 0) 518 | *--cp = byte + '0'; 519 | } 520 | *--cp = '.'; 521 | addr >>= 8; 522 | } while (--n > 0); 523 | 524 | /* Convert the string to lowercase */ 525 | retStr = (char*)(cp+1); 526 | 527 | return(retStr); 528 | } 529 | 530 | /* ***************************************************** */ 531 | 532 | static void print_flow(u_int16_t thread_id, struct ndpi_flow *flow) { 533 | FILE *out = results_file ? results_file : stdout; 534 | 535 | fprintf(out, "\t%u", ++num_flows); 536 | 537 | char *src_name, *dst_name; 538 | u_int16_t src_port, dst_port; 539 | if(flow->src_to_dst_direction) { 540 | src_name = flow->lower_name, dst_name = flow->upper_name; 541 | src_port = flow->lower_port, dst_port = flow->upper_port; 542 | } else { 543 | src_name = flow->upper_name, dst_name = flow->lower_name; 544 | src_port = flow->upper_port, dst_port = flow->lower_port; 545 | } 546 | 547 | fprintf(out, "\t%s %s%s%s:%u <-> %s%s%s:%u ", 548 | ipProto2Name(flow->protocol), 549 | (flow->ip_version == 6) ? "[" : "", 550 | src_name, 551 | (flow->ip_version == 6) ? "]" : "", 552 | ntohs(src_port), 553 | (flow->ip_version == 6) ? "[" : "", 554 | dst_name, 555 | (flow->ip_version == 6) ? "]" : "", 556 | ntohs(dst_port)); 557 | 558 | if(flow->vlan_id > 0) fprintf(out, "[VLAN: %u]", flow->vlan_id); 559 | 560 | if(flow->detected_protocol.master_protocol) { 561 | char buf[64]; 562 | 563 | fprintf(out, "[proto: %u.%u/%s]", 564 | flow->detected_protocol.master_protocol, flow->detected_protocol.protocol, 565 | ndpi_protocol2name(ndpi_thread_info[thread_id].ndpi_struct, 566 | flow->detected_protocol, buf, sizeof(buf))); 567 | } else 568 | fprintf(out, "[proto: %u/%s]", 569 | flow->detected_protocol.protocol, 570 | ndpi_get_proto_name(ndpi_thread_info[thread_id].ndpi_struct, flow->detected_protocol.protocol)); 571 | 572 | fprintf(out, "[Up: %u pkts/%llu bytes, Down: %u pkts/%llu bytes]", 573 | flow->out_pkts, (long long unsigned int)flow->out_bytes, 574 | flow->in_pkts, (long long unsigned int)flow->in_bytes); 575 | 576 | if(flow->host_server_name[0] != '\0') fprintf(out, "[Host: %s]", flow->host_server_name); 577 | if(flow->ssl.client_certificate[0] != '\0') fprintf(out, "[SSL client: %s]", flow->ssl.client_certificate); 578 | if(flow->ssl.server_certificate[0] != '\0') fprintf(out, "[SSL server: %s]", flow->ssl.server_certificate); 579 | 580 | fprintf(out, "\n"); 581 | } 582 | 583 | /* ***************************************************** */ 584 | 585 | static int isgraph_string(char* c) { 586 | int i; 587 | for(i = 0; i < strlen(c); i++){ 588 | if(!isgraph(c[i])){ 589 | return 0; 590 | } 591 | } 592 | return 1; 593 | } 594 | 595 | /* ***************************************************** */ 596 | 597 | static void write_socket(u_int16_t thread_id, char *buf, int length){ 598 | int n, retry_count = 0; 599 | for(retry_count = 0; retry_count < 3; retry_count++){ 600 | n = write(_sockfd[thread_id], buf, length); 601 | if(n == length){ 602 | break; 603 | } 604 | output(LOG_WARNING, "%s\n", "[WARN] Failed to connect to server - retrying in 5 sec..."); 605 | sleep(5); 606 | close(_sockfd[thread_id]); 607 | setup_socket(thread_id); 608 | } 609 | if(retry_count >= 3){ 610 | output(LOG_ERR, "%s\n", "[ERROR] Maximum connection retry count has been exceeded"); 611 | exit(1); 612 | } 613 | } 614 | 615 | /* ***************************************************** */ 616 | 617 | #ifdef HAVE_LIBMSGPACKC 618 | static int get_map_size(struct ndpi_flow *flow) { 619 | int size = 13; 620 | if(flow->detected_protocol.master_protocol) size++; 621 | if(isgraph_string(flow->host_server_name)) size++; 622 | if((flow->ssl.client_certificate[0] != '\0') || (flow->ssl.server_certificate[0] != '\0')) size++; 623 | return size; 624 | } 625 | 626 | /* ***************************************************** */ 627 | 628 | static void msgpack_pack_string(msgpack_packer pk, char *string) { 629 | msgpack_pack_str(&pk, strlen(string)); 630 | msgpack_pack_str_body(&pk, string, strlen(string)); 631 | } 632 | 633 | /* ***************************************************** */ 634 | 635 | static void msgpack_pack_kv(msgpack_packer pk, char *key, char *value) { 636 | msgpack_pack_string(pk, key); 637 | msgpack_pack_string(pk, value); 638 | } 639 | 640 | /* ***************************************************** */ 641 | 642 | static void send_msgpack(u_int16_t thread_id, struct ndpi_flow *flow) { 643 | msgpack_sbuffer sbuf; 644 | msgpack_sbuffer_init(&sbuf); 645 | 646 | msgpack_packer pk; 647 | msgpack_packer_init(&pk, &sbuf, msgpack_sbuffer_write); 648 | 649 | char *src_name, *dst_name; 650 | u_int16_t src_port, dst_port; 651 | 652 | if(flow->src_to_dst_direction) { 653 | src_name = flow->lower_name, dst_name = flow->upper_name; 654 | src_port = flow->lower_port, dst_port = flow->upper_port; 655 | } else { 656 | src_name = flow->upper_name, dst_name = flow->lower_name; 657 | src_port = flow->upper_port, dst_port = flow->lower_port; 658 | } 659 | 660 | msgpack_pack_array(&pk, 3); 661 | 662 | msgpack_pack_string(pk, _tag); 663 | msgpack_pack_uint64(&pk, flow->first_seen / detection_tick_resolution); 664 | msgpack_pack_map(&pk, get_map_size(flow)); 665 | 666 | msgpack_pack_kv(pk, "protocol", ipProto2Name(flow->protocol)); 667 | 668 | msgpack_pack_kv(pk, "src_addr", src_name); 669 | 670 | msgpack_pack_string(pk, "src_port"); 671 | msgpack_pack_uint16(&pk, ntohs(src_port)); 672 | 673 | msgpack_pack_kv(pk, "dst_addr", dst_name); 674 | 675 | msgpack_pack_string(pk, "dst_port"); 676 | msgpack_pack_uint16(&pk, ntohs(dst_port)); 677 | 678 | if(flow->detected_protocol.master_protocol){ 679 | msgpack_pack_string(pk, "master_protocol"); 680 | msgpack_pack_int(&pk, flow->detected_protocol.master_protocol); 681 | } 682 | 683 | msgpack_pack_string(pk, "detected_protocol"); 684 | msgpack_pack_int(&pk, flow->detected_protocol.protocol); 685 | 686 | if(flow->detected_protocol.master_protocol) { 687 | char tmp[256]; 688 | 689 | snprintf(tmp, sizeof(tmp), "%s.%s", 690 | ndpi_get_proto_name(ndpi_thread_info[thread_id].ndpi_struct, flow->detected_protocol.master_protocol), 691 | ndpi_get_proto_name(ndpi_thread_info[thread_id].ndpi_struct, flow->detected_protocol.protocol)); 692 | 693 | msgpack_pack_kv(pk,"protocol_name", tmp); 694 | } else 695 | msgpack_pack_kv(pk,"protocol_name", ndpi_get_proto_name(ndpi_thread_info[thread_id].ndpi_struct, 696 | flow->detected_protocol.protocol)); 697 | 698 | msgpack_pack_string(pk, "out_pkts"); 699 | msgpack_pack_int(&pk, flow->out_pkts); 700 | msgpack_pack_string(pk, "out_bytes"); 701 | msgpack_pack_int(&pk, flow->out_bytes); 702 | msgpack_pack_string(pk, "in_pkts"); 703 | msgpack_pack_int(&pk, flow->in_pkts); 704 | msgpack_pack_string(pk, "in_bytes"); 705 | msgpack_pack_int(&pk, flow->in_bytes); 706 | msgpack_pack_string(pk, "first_switched"); 707 | msgpack_pack_int64(&pk, flow->first_seen / detection_tick_resolution); 708 | msgpack_pack_string(pk, "last_switched"); 709 | msgpack_pack_int64(&pk, flow->last_seen / detection_tick_resolution); 710 | 711 | if(isgraph_string(flow->host_server_name)){ 712 | msgpack_pack_kv(pk, "server_name", flow->host_server_name); 713 | } 714 | 715 | if((flow->ssl.client_certificate[0] != '\0') || (flow->ssl.server_certificate[0] != '\0')) { 716 | msgpack_pack_string(pk, "ssl"); 717 | if((flow->ssl.client_certificate[0] != '\0') && (flow->ssl.server_certificate[0] != '\0')) { 718 | msgpack_pack_map(&pk, 2); 719 | }else{ 720 | msgpack_pack_map(&pk, 1); 721 | } 722 | if(flow->ssl.client_certificate[0] != '\0') 723 | msgpack_pack_kv(pk, "client", flow->ssl.client_certificate); 724 | 725 | if(flow->ssl.server_certificate[0] != '\0') 726 | msgpack_pack_kv(pk, "server", flow->ssl.server_certificate); 727 | } 728 | 729 | write_socket(thread_id, sbuf.data, sbuf.size); 730 | msgpack_sbuffer_destroy(&sbuf); 731 | } 732 | #endif 733 | 734 | /* ***************************************************** */ 735 | 736 | #ifdef HAVE_LIBJSON_C 737 | static void send_json(u_int16_t thread_id, struct ndpi_flow *flow) { 738 | json_object *jObj; 739 | 740 | jObj = json_object_new_object(); 741 | json_object *jarray = json_object_new_array(); 742 | 743 | char *src_name, *dst_name; 744 | u_int16_t src_port, dst_port; 745 | 746 | if(flow->src_to_dst_direction) { 747 | src_name = flow->lower_name, dst_name = flow->upper_name; 748 | src_port = flow->lower_port, dst_port = flow->upper_port; 749 | } else { 750 | src_name = flow->upper_name, dst_name = flow->lower_name; 751 | src_port = flow->upper_port, dst_port = flow->lower_port; 752 | } 753 | 754 | json_object_object_add(jObj,"protocol",json_object_new_string(ipProto2Name(flow->protocol))); 755 | json_object_object_add(jObj,"src_addr",json_object_new_string(src_name)); 756 | json_object_object_add(jObj,"src_port",json_object_new_int(ntohs(src_port))); 757 | json_object_object_add(jObj,"dst_addr",json_object_new_string(dst_name)); 758 | json_object_object_add(jObj,"dst_port",json_object_new_int(ntohs(dst_port))); 759 | 760 | if(flow->detected_protocol.master_protocol) 761 | json_object_object_add(jObj,"master_protocol",json_object_new_int(flow->detected_protocol.master_protocol)); 762 | 763 | json_object_object_add(jObj,"detected_protocol",json_object_new_int(flow->detected_protocol.protocol)); 764 | 765 | if(flow->detected_protocol.master_protocol) { 766 | char tmp[256]; 767 | 768 | snprintf(tmp, sizeof(tmp), "%s.%s", 769 | ndpi_get_proto_name(ndpi_thread_info[thread_id].ndpi_struct, flow->detected_protocol.master_protocol), 770 | ndpi_get_proto_name(ndpi_thread_info[thread_id].ndpi_struct, flow->detected_protocol.protocol)); 771 | 772 | json_object_object_add(jObj,"protocol_name", 773 | json_object_new_string(tmp)); 774 | } else 775 | json_object_object_add(jObj,"protocol_name", 776 | json_object_new_string(ndpi_get_proto_name(ndpi_thread_info[thread_id].ndpi_struct, 777 | flow->detected_protocol.protocol))); 778 | 779 | json_object_object_add(jObj,"out_pkts",json_object_new_int(flow->out_pkts)); 780 | json_object_object_add(jObj,"out_bytes",json_object_new_int(flow->out_bytes)); 781 | json_object_object_add(jObj,"in_pkts",json_object_new_int(flow->in_pkts)); 782 | json_object_object_add(jObj,"in_bytes",json_object_new_int(flow->in_bytes)); 783 | json_object_object_add(jObj,"first_switched", json_object_new_int64((signed)(flow->first_seen / detection_tick_resolution))); 784 | json_object_object_add(jObj,"last_switched", json_object_new_int64((signed)(flow->first_seen / detection_tick_resolution))); 785 | 786 | if(isgraph_string(flow->host_server_name)){ 787 | json_object_object_add(jObj,"server_name",json_object_new_string(flow->host_server_name)); 788 | } 789 | 790 | if((flow->ssl.client_certificate[0] != '\0') || (flow->ssl.server_certificate[0] != '\0')) { 791 | json_object *sjObj = json_object_new_object(); 792 | 793 | if(flow->ssl.client_certificate[0] != '\0') 794 | json_object_object_add(sjObj, "client", json_object_new_string(flow->ssl.client_certificate)); 795 | 796 | if(flow->ssl.server_certificate[0] != '\0') 797 | json_object_object_add(sjObj, "server", json_object_new_string(flow->ssl.server_certificate)); 798 | 799 | json_object_object_add(jObj, "ssl", sjObj); 800 | } 801 | 802 | json_object_array_add(jarray, json_object_new_string(_tag)); 803 | json_object_array_add(jarray, json_object_new_int64((signed)(flow->first_seen / detection_tick_resolution))); 804 | json_object_array_add(jarray, jObj); 805 | 806 | char* body = (char *)json_object_to_json_string(jarray); 807 | write_socket(thread_id, body, strlen(body)); 808 | 809 | json_object_put(jarray); 810 | } 811 | #endif 812 | 813 | /* ***************************************************** */ 814 | 815 | static void send_flow(u_int16_t thread_id, struct ndpi_flow *flow) { 816 | if(json_flag){ 817 | #ifdef HAVE_LIBJSON_C 818 | send_json(thread_id, flow); 819 | #endif 820 | }else if(msgpack_flag){ 821 | #ifdef HAVE_LIBMSGPACKC 822 | send_msgpack(thread_id, flow); 823 | #endif 824 | } 825 | } 826 | 827 | 828 | /* ***************************************************** */ 829 | 830 | static void free_ndpi_flow(struct ndpi_flow *flow) { 831 | if(flow->ndpi_flow) { ndpi_free_flow(flow->ndpi_flow); flow->ndpi_flow = NULL; } 832 | if(flow->src_id) { ndpi_free(flow->src_id); flow->src_id = NULL; } 833 | if(flow->dst_id) { ndpi_free(flow->dst_id); flow->dst_id = NULL; } 834 | 835 | } 836 | 837 | /* ***************************************************** */ 838 | 839 | static void ndpi_flow_freer(void *node) { 840 | struct ndpi_flow *flow = (struct ndpi_flow*)node; 841 | 842 | free_ndpi_flow(flow); 843 | ndpi_free(flow); 844 | } 845 | 846 | /* ***************************************************** */ 847 | 848 | static void node_print_unknown_proto_walker(const void *node, ndpi_VISIT which, int depth, void *user_data) { 849 | struct ndpi_flow *flow = *(struct ndpi_flow**)node; 850 | u_int16_t thread_id = *((u_int16_t*)user_data); 851 | 852 | if(flow->detected_protocol.protocol != NDPI_PROTOCOL_UNKNOWN) return; 853 | 854 | if((which == ndpi_preorder) || (which == ndpi_leaf)) /* Avoid walking the same node multiple times */ 855 | print_flow(thread_id, flow); 856 | } 857 | 858 | /* ***************************************************** */ 859 | 860 | static void node_print_known_proto_walker(const void *node, ndpi_VISIT which, int depth, void *user_data) { 861 | struct ndpi_flow *flow = *(struct ndpi_flow**)node; 862 | u_int16_t thread_id = *((u_int16_t*)user_data); 863 | 864 | if(flow->detected_protocol.protocol == NDPI_PROTOCOL_UNKNOWN) return; 865 | 866 | if((which == ndpi_preorder) || (which == ndpi_leaf)) /* Avoid walking the same node multiple times */ 867 | print_flow(thread_id, flow); 868 | } 869 | 870 | /* ***************************************************** */ 871 | 872 | static u_int16_t node_guess_undetected_protocol(u_int16_t thread_id, struct ndpi_flow *flow) { 873 | flow->detected_protocol = ndpi_guess_undetected_protocol(ndpi_thread_info[thread_id].ndpi_struct, 874 | flow->protocol, 875 | ntohl(flow->lower_ip), 876 | ntohs(flow->lower_port), 877 | ntohl(flow->upper_ip), 878 | ntohs(flow->upper_port)); 879 | 880 | return(flow->detected_protocol.protocol); 881 | } 882 | 883 | /* ***************************************************** */ 884 | 885 | static void node_proto_guess_walker(const void *node, ndpi_VISIT which, int depth, void *user_data) { 886 | struct ndpi_flow *flow = *(struct ndpi_flow **) node; 887 | u_int16_t thread_id = *((u_int16_t *) user_data); 888 | 889 | if((which == ndpi_preorder) || (which == ndpi_leaf)) { /* Avoid walking the same node multiple times */ 890 | if(enable_protocol_guess) { 891 | if(flow->detected_protocol.protocol == NDPI_PROTOCOL_UNKNOWN) { 892 | node_guess_undetected_protocol(thread_id, flow); 893 | } 894 | } 895 | 896 | } 897 | } 898 | 899 | /* ***************************************************** */ 900 | 901 | static void node_idle_scan_walker(const void *node, ndpi_VISIT which, int depth, void *user_data) { 902 | struct ndpi_flow *flow = *(struct ndpi_flow **) node; 903 | u_int16_t thread_id = *((u_int16_t *) user_data); 904 | 905 | if(ndpi_thread_info[thread_id].num_idle_flows == IDLE_SCAN_BUDGET) /* TODO optimise with a budget-based walk */ 906 | return; 907 | 908 | if((which == ndpi_preorder) || (which == ndpi_leaf)) { /* Avoid walking the same node multiple times */ 909 | if(flow->expires_at < ndpi_thread_info[thread_id].last_time) { 910 | /* update stats */ 911 | node_proto_guess_walker(node, which, depth, user_data); 912 | 913 | if((flow->detected_protocol.protocol == NDPI_PROTOCOL_UNKNOWN) && !undetected_flows_deleted) 914 | undetected_flows_deleted = 1; 915 | 916 | if(verbose > 1) print_flow(thread_id, flow); 917 | if(!dryrun_flag) send_flow(thread_id, flow); 918 | 919 | free_ndpi_flow(flow); 920 | 921 | /* adding to a queue (we can't delete it from the tree inline ) */ 922 | ndpi_thread_info[thread_id].idle_flows[ndpi_thread_info[thread_id].num_idle_flows++] = flow; 923 | } 924 | } 925 | } 926 | 927 | 928 | /* ***************************************************** */ 929 | 930 | static void node_expire_all_walker(const void *node, ndpi_VISIT which, int depth, void *user_data) { 931 | struct ndpi_flow *flow = *(struct ndpi_flow **) node; 932 | u_int16_t thread_id = *((u_int16_t *) user_data); 933 | 934 | if((which == ndpi_preorder) || (which == ndpi_leaf)) { /* Avoid walking the same node multiple times */ 935 | node_proto_guess_walker(node, which, depth, user_data); 936 | 937 | if(verbose > 1) print_flow(thread_id, flow); 938 | if(!dryrun_flag) send_flow(thread_id, flow); 939 | 940 | free_ndpi_flow(flow); 941 | } 942 | } 943 | 944 | 945 | /* ***************************************************** */ 946 | 947 | static int node_cmp(const void *a, const void *b) { 948 | struct ndpi_flow *fa = (struct ndpi_flow*)a; 949 | struct ndpi_flow *fb = (struct ndpi_flow*)b; 950 | 951 | if(fa->vlan_id < fb->vlan_id ) return(-1); else { if(fa->vlan_id > fb->vlan_id ) return(1); } 952 | if(fa->lower_ip < fb->lower_ip ) return(-1); else { if(fa->lower_ip > fb->lower_ip ) return(1); } 953 | if(fa->lower_port < fb->lower_port) return(-1); else { if(fa->lower_port > fb->lower_port) return(1); } 954 | if(fa->upper_ip < fb->upper_ip ) return(-1); else { if(fa->upper_ip > fb->upper_ip ) return(1); } 955 | if(fa->upper_port < fb->upper_port) return(-1); else { if(fa->upper_port > fb->upper_port) return(1); } 956 | if(fa->protocol < fb->protocol ) return(-1); else { if(fa->protocol > fb->protocol ) return(1); } 957 | 958 | return(0); 959 | } 960 | 961 | /* ***************************************************** */ 962 | 963 | static void remove_idle_flows(u_int16_t thread_id){ 964 | /* remove idle flows (unfortunately we cannot do this inline) */ 965 | while (ndpi_thread_info[thread_id].num_idle_flows > 0) { 966 | /* search and delete the idle flow from the "ndpi_flow_root" (see struct reader thread) - here flows are the node of a b-tree */ 967 | ndpi_tdelete(ndpi_thread_info[thread_id].idle_flows[--ndpi_thread_info[thread_id].num_idle_flows], &ndpi_thread_info[thread_id].ndpi_flows_root[ndpi_thread_info[thread_id].idle_scan_idx], node_cmp); 968 | 969 | /* free the memory associated to idle flow in "idle_flows" - (see struct reader thread)*/ 970 | free_ndpi_flow(ndpi_thread_info[thread_id].idle_flows[ndpi_thread_info[thread_id].num_idle_flows]); 971 | ndpi_free(ndpi_thread_info[thread_id].idle_flows[ndpi_thread_info[thread_id].num_idle_flows]); 972 | } 973 | } 974 | 975 | /* ***************************************************** */ 976 | 977 | static struct ndpi_flow *get_ndpi_flow(u_int16_t thread_id, 978 | const u_int8_t version, 979 | u_int16_t vlan_id, 980 | const struct ndpi_iphdr *iph, 981 | const struct ndpi_ipv6hdr *iph6, 982 | u_int16_t ip_offset, 983 | u_int16_t ipsize, 984 | u_int16_t l4_packet_len, 985 | struct ndpi_tcphdr **tcph, 986 | struct ndpi_udphdr **udph, 987 | u_int16_t *sport, u_int16_t *dport, 988 | struct ndpi_id_struct **src, 989 | struct ndpi_id_struct **dst, 990 | u_int8_t *proto, 991 | u_int8_t **payload, 992 | u_int16_t *payload_len, 993 | u_int8_t *src_to_dst_direction) { 994 | u_int32_t idx, l4_offset; 995 | u_int32_t lower_ip; 996 | u_int32_t upper_ip; 997 | u_int16_t lower_port; 998 | u_int16_t upper_port; 999 | struct ndpi_flow flow; 1000 | void *ret; 1001 | u_int8_t *l3, *l4; 1002 | 1003 | /* 1004 | Note: to keep things simple 1005 | we handle IPv6 a-la-IPv4. 1006 | */ 1007 | if(version == 4) { 1008 | if(ipsize < 20) 1009 | return NULL; 1010 | 1011 | if((iph->ihl * 4) > ipsize || ipsize < ntohs(iph->tot_len) 1012 | || (iph->frag_off & htons(0x1FFF)) != 0) 1013 | return NULL; 1014 | 1015 | l4_offset = iph->ihl * 4; 1016 | l3 = (u_int8_t*)iph; 1017 | } else { 1018 | l4_offset = sizeof(struct ndpi_ipv6hdr); 1019 | l3 = (u_int8_t*)iph6; 1020 | } 1021 | 1022 | if(iph->saddr < iph->daddr) { 1023 | lower_ip = iph->saddr; 1024 | upper_ip = iph->daddr; 1025 | *src_to_dst_direction = 1; 1026 | } else { 1027 | lower_ip = iph->daddr; 1028 | upper_ip = iph->saddr; 1029 | *src_to_dst_direction = 0; 1030 | } 1031 | 1032 | *proto = iph->protocol; 1033 | l4 = ((u_int8_t *) l3 + l4_offset); 1034 | 1035 | if(iph->protocol == 6 && l4_packet_len >= 20) { 1036 | u_int tcp_len; 1037 | 1038 | // tcp 1039 | *tcph = (struct ndpi_tcphdr *)l4; 1040 | *sport = ntohs((*tcph)->source), *dport = ntohs((*tcph)->dest); 1041 | 1042 | if(iph->saddr < iph->daddr) { 1043 | lower_port = (*tcph)->source, upper_port = (*tcph)->dest; 1044 | } else { 1045 | lower_port = (*tcph)->dest; 1046 | upper_port = (*tcph)->source; 1047 | 1048 | if(iph->saddr == iph->daddr) { 1049 | if(lower_port > upper_port) { 1050 | u_int16_t p = lower_port; 1051 | 1052 | lower_port = upper_port; 1053 | upper_port = p; 1054 | } 1055 | } 1056 | } 1057 | 1058 | tcp_len = ndpi_min(4*(*tcph)->doff, l4_packet_len); 1059 | *payload = &l4[tcp_len]; 1060 | *payload_len = ndpi_max(0, l4_packet_len-4*(*tcph)->doff); 1061 | } else if(iph->protocol == 17 && l4_packet_len >= 8) { 1062 | // udp 1063 | *udph = (struct ndpi_udphdr *)l4; 1064 | *sport = ntohs((*udph)->source), *dport = ntohs((*udph)->dest); 1065 | *payload = &l4[sizeof(struct ndpi_udphdr)]; 1066 | *payload_len = ndpi_max(0, l4_packet_len-sizeof(struct ndpi_udphdr)); 1067 | 1068 | if(iph->saddr < iph->daddr) { 1069 | lower_port = (*udph)->source, upper_port = (*udph)->dest; 1070 | } else { 1071 | lower_port = (*udph)->dest, upper_port = (*udph)->source; 1072 | 1073 | 1074 | if(iph->saddr == iph->daddr) { 1075 | if(lower_port > upper_port) { 1076 | u_int16_t p = lower_port; 1077 | 1078 | lower_port = upper_port; 1079 | upper_port = p; 1080 | } 1081 | } 1082 | } 1083 | 1084 | *sport = ntohs(lower_port), *dport = ntohs(upper_port); 1085 | } else { 1086 | // non tcp/udp protocols 1087 | lower_port = 0; 1088 | upper_port = 0; 1089 | } 1090 | 1091 | flow.protocol = iph->protocol, flow.vlan_id = vlan_id; 1092 | flow.lower_ip = lower_ip, flow.upper_ip = upper_ip; 1093 | flow.lower_port = lower_port, flow.upper_port = upper_port; 1094 | 1095 | idx = (vlan_id + lower_ip + upper_ip + iph->protocol + lower_port + upper_port) % NUM_ROOTS; 1096 | ret = ndpi_tfind(&flow, &ndpi_thread_info[thread_id].ndpi_flows_root[idx], node_cmp); 1097 | 1098 | if(ret == NULL) { 1099 | struct ndpi_flow *newflow = (struct ndpi_flow*)malloc(sizeof(struct ndpi_flow)); 1100 | 1101 | if(newflow == NULL) { 1102 | output(LOG_ERR, "[ERROR] %s(1): not enough memory\n", __FUNCTION__); 1103 | return(NULL); 1104 | } 1105 | 1106 | memset(newflow, 0, sizeof(struct ndpi_flow)); 1107 | newflow->protocol = iph->protocol, newflow->vlan_id = vlan_id; 1108 | newflow->lower_ip = lower_ip, newflow->upper_ip = upper_ip; 1109 | newflow->lower_port = lower_port, newflow->upper_port = upper_port; 1110 | newflow->ip_version = version; 1111 | newflow->src_to_dst_direction = *src_to_dst_direction; 1112 | 1113 | if(version == 4) { 1114 | inet_ntop(AF_INET, &lower_ip, newflow->lower_name, sizeof(newflow->lower_name)); 1115 | inet_ntop(AF_INET, &upper_ip, newflow->upper_name, sizeof(newflow->upper_name)); 1116 | } else { 1117 | inet_ntop(AF_INET6, &iph6->ip6_src, newflow->lower_name, sizeof(newflow->lower_name)); 1118 | inet_ntop(AF_INET6, &iph6->ip6_dst, newflow->upper_name, sizeof(newflow->upper_name)); 1119 | } 1120 | 1121 | if((newflow->ndpi_flow = malloc_wrapper(size_flow_struct)) == NULL) { 1122 | output(LOG_ERR, "[ERROR] %s(2): not enough memory\n", __FUNCTION__); 1123 | free(newflow); 1124 | return(NULL); 1125 | } else 1126 | memset(newflow->ndpi_flow, 0, size_flow_struct); 1127 | 1128 | if((newflow->src_id = malloc_wrapper(size_id_struct)) == NULL) { 1129 | output(LOG_ERR, "[ERROR] %s(3): not enough memory\n", __FUNCTION__); 1130 | free(newflow); 1131 | return(NULL); 1132 | } else 1133 | memset(newflow->src_id, 0, size_id_struct); 1134 | 1135 | if((newflow->dst_id = malloc_wrapper(size_id_struct)) == NULL) { 1136 | output(LOG_ERR, "[ERROR] %s(4): not enough memory\n", __FUNCTION__); 1137 | free(newflow); 1138 | return(NULL); 1139 | } else 1140 | memset(newflow->dst_id, 0, size_id_struct); 1141 | 1142 | ndpi_tsearch(newflow, &ndpi_thread_info[thread_id].ndpi_flows_root[idx], node_cmp); /* Add */ 1143 | 1144 | *src = newflow->src_id, *dst = newflow->dst_id; 1145 | 1146 | return newflow; 1147 | } else { 1148 | struct ndpi_flow *flow = *(struct ndpi_flow**)ret; 1149 | 1150 | if(flow->lower_ip == lower_ip && flow->upper_ip == upper_ip 1151 | && flow->lower_port == lower_port && flow->upper_port == upper_port) 1152 | *src = flow->src_id, *dst = flow->dst_id; 1153 | else 1154 | *src = flow->dst_id, *dst = flow->src_id; 1155 | 1156 | return flow; 1157 | } 1158 | } 1159 | 1160 | /* ***************************************************** */ 1161 | 1162 | static struct ndpi_flow *get_ndpi_flow6(u_int16_t thread_id, 1163 | u_int16_t vlan_id, 1164 | const struct ndpi_ipv6hdr *iph6, 1165 | u_int16_t ip_offset, 1166 | struct ndpi_tcphdr **tcph, 1167 | struct ndpi_udphdr **udph, 1168 | u_int16_t *sport, u_int16_t *dport, 1169 | struct ndpi_id_struct **src, 1170 | struct ndpi_id_struct **dst, 1171 | u_int8_t *proto, 1172 | u_int8_t **payload, 1173 | u_int16_t *payload_len, 1174 | u_int8_t *src_to_dst_direction) { 1175 | struct ndpi_iphdr iph; 1176 | 1177 | memset(&iph, 0, sizeof(iph)); 1178 | iph.version = 4; 1179 | iph.saddr = iph6->ip6_src.u6_addr.u6_addr32[2] + iph6->ip6_src.u6_addr.u6_addr32[3]; 1180 | iph.daddr = iph6->ip6_dst.u6_addr.u6_addr32[2] + iph6->ip6_dst.u6_addr.u6_addr32[3]; 1181 | iph.protocol = iph6->ip6_ctlun.ip6_un1.ip6_un1_nxt; 1182 | 1183 | if(iph.protocol == 0x3C /* IPv6 destination option */) { 1184 | u_int8_t *options = (u_int8_t*)iph6 + sizeof(const struct ndpi_ipv6hdr); 1185 | 1186 | iph.protocol = options[0]; 1187 | } 1188 | 1189 | return(get_ndpi_flow(thread_id, 6, vlan_id, &iph, iph6, ip_offset, 1190 | sizeof(struct ndpi_ipv6hdr), 1191 | ntohs(iph6->ip6_ctlun.ip6_un1.ip6_un1_plen), 1192 | tcph, udph, sport, dport, 1193 | src, dst, proto, payload, payload_len, src_to_dst_direction)); 1194 | } 1195 | 1196 | /* ***************************************************** */ 1197 | 1198 | static int setup_socket(u_int16_t thread_id) { 1199 | int len, ret, result, sockfd; 1200 | struct sockaddr_in address; 1201 | 1202 | /*クライアント用ソケット作成*/ 1203 | if((sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0) { 1204 | output(LOG_ERR, "%s\n", "[ERROR] Could not create socket"); 1205 | close(sockfd); 1206 | return 0; 1207 | } 1208 | 1209 | /*サーバ側と同じ名前でソケットの名前を指定*/ 1210 | address.sin_family = AF_INET; 1211 | address.sin_addr.s_addr = inet_addr(_server_addr); 1212 | if (address.sin_addr.s_addr == 0xffffffff) { 1213 | struct addrinfo hints; 1214 | struct addrinfo *result; 1215 | 1216 | memset(&hints, 0, sizeof(struct addrinfo)); 1217 | if((ret = getaddrinfo(_server_addr, NULL, &hints, &result)) != 0){ 1218 | output(LOG_ERR, "[ERROR] %s\n", gai_strerror(ret)); 1219 | close(sockfd); 1220 | return 0; 1221 | } 1222 | address.sin_addr.s_addr = ((struct sockaddr_in *)(result->ai_addr))->sin_addr.s_addr; 1223 | freeaddrinfo(result); 1224 | } 1225 | address.sin_port = htons(_server_port); 1226 | len = sizeof(address); 1227 | 1228 | /*クライアントのソケットとサーバのソケットの接続*/ 1229 | if((connect(sockfd, (struct sockaddr *)&address, len)) == -1){ 1230 | output(LOG_ERR, "%s\n", "[ERROR] Could not connect to server"); 1231 | close(sockfd); 1232 | return 0; 1233 | } 1234 | _sockfd[thread_id] = sockfd; 1235 | return 1; 1236 | } 1237 | 1238 | /* ***************************************************** */ 1239 | 1240 | static void setup_detection(u_int16_t thread_id) { 1241 | NDPI_PROTOCOL_BITMASK all; 1242 | 1243 | memset(&ndpi_thread_info[thread_id], 0, sizeof(ndpi_thread_info[thread_id])); 1244 | 1245 | // init global detection structure 1246 | ndpi_thread_info[thread_id].ndpi_struct = ndpi_init_detection_module(detection_tick_resolution, 1247 | malloc_wrapper, free_wrapper, NULL); 1248 | if(ndpi_thread_info[thread_id].ndpi_struct == NULL) { 1249 | output(LOG_ERR, "%s\n", "[ERROR] global structure initialization failed"); 1250 | exit(-1); 1251 | } 1252 | 1253 | /* ndpi_thread_info[thread_id].ndpi_struct->http_dont_dissect_response = 1; */ 1254 | 1255 | // enable all protocols 1256 | NDPI_BITMASK_SET_ALL(all); 1257 | ndpi_set_protocol_detection_bitmask2(ndpi_thread_info[thread_id].ndpi_struct, &all); 1258 | 1259 | // allocate memory for id and flow tracking 1260 | size_id_struct = sizeof(struct ndpi_id_struct); 1261 | size_flow_struct = sizeof(struct ndpi_flow_struct); 1262 | 1263 | if(_protoFilePath != NULL) 1264 | ndpi_load_protocols_file(ndpi_thread_info[thread_id].ndpi_struct, _protoFilePath); 1265 | } 1266 | 1267 | /* ***************************************************** */ 1268 | 1269 | static void terminate_detection(u_int16_t thread_id) { 1270 | int i; 1271 | 1272 | for(i=0; ilast_seen - flow->first_seen > MAXIMUM_LIFETIME * detection_tick_resolution) { 1291 | flow->expires_at = 0; 1292 | }else if (proto == IPPROTO_TCP) { 1293 | /* TCP flows */ 1294 | if (tcph->rst) { 1295 | /* Reset TCP flows */ 1296 | flow->fin_rst_received = 1; 1297 | flow->expires_at = flow->last_seen + TCP_RST_TIMEOUT * detection_tick_resolution; 1298 | }else if (tcph->fin){ 1299 | /* Finished TCP flows */ 1300 | flow->fin_rst_received = 1; 1301 | flow->expires_at = flow->last_seen + TCP_FIN_TIMEOUT * detection_tick_resolution; 1302 | }else if (!flow->fin_rst_received) { 1303 | /* TCP flows */ 1304 | flow->expires_at = flow->last_seen + TCP_TIMEOUT * detection_tick_resolution; 1305 | } 1306 | }else if (proto == IPPROTO_UDP) { 1307 | /* UDP flows */ 1308 | flow->expires_at = flow->last_seen + UDP_TIMEOUT * detection_tick_resolution; 1309 | }else if ((proto == IPPROTO_ICMP) || (proto == IPPROTO_ICMPV6)) { 1310 | /* ICMP flows */ 1311 | flow->expires_at = flow->last_seen + ICMP_TIMEOUT * detection_tick_resolution; 1312 | }else{ 1313 | /* Everything else */ 1314 | flow->expires_at = flow->last_seen + GENERAL_TIMEOUT * detection_tick_resolution; 1315 | } 1316 | } 1317 | 1318 | /* ***************************************************** */ 1319 | 1320 | static unsigned int packet_processing(u_int16_t thread_id, 1321 | const u_int64_t time, 1322 | u_int16_t vlan_id, 1323 | const struct ndpi_iphdr *iph, 1324 | struct ndpi_ipv6hdr *iph6, 1325 | u_int16_t ip_offset, 1326 | u_int16_t ipsize, u_int16_t rawsize) { 1327 | struct ndpi_id_struct *src, *dst; 1328 | struct ndpi_flow *flow; 1329 | struct ndpi_flow_struct *ndpi_flow = NULL; 1330 | u_int8_t proto; 1331 | struct ndpi_tcphdr *tcph = NULL; 1332 | struct ndpi_udphdr *udph = NULL; 1333 | u_int16_t sport, dport, payload_len; 1334 | u_int8_t *payload; 1335 | u_int8_t src_to_dst_direction= 1; 1336 | 1337 | if(iph) 1338 | flow = get_ndpi_flow(thread_id, 4, vlan_id, iph, NULL, 1339 | ip_offset, ipsize, 1340 | ntohs(iph->tot_len) - (iph->ihl * 4), 1341 | &tcph, &udph, &sport, &dport, 1342 | &src, &dst, &proto, 1343 | &payload, &payload_len, &src_to_dst_direction); 1344 | else 1345 | flow = get_ndpi_flow6(thread_id, vlan_id, iph6, ip_offset, 1346 | &tcph, &udph, &sport, &dport, 1347 | &src, &dst, &proto, 1348 | &payload, &payload_len, &src_to_dst_direction); 1349 | 1350 | if(flow != NULL) { 1351 | ndpi_flow = flow->ndpi_flow; 1352 | // Add to the download packets and bytes if the current direction is different from the first direction. 1353 | if(flow->src_to_dst_direction ^ src_to_dst_direction){ 1354 | flow->in_pkts++, flow->in_bytes += rawsize; 1355 | }else{ 1356 | flow->out_pkts++, flow->out_bytes += rawsize; 1357 | } 1358 | flow->last_seen = time; 1359 | if(flow->first_seen == 0){ 1360 | flow->first_seen = time; 1361 | } 1362 | } else { 1363 | return(0); 1364 | } 1365 | 1366 | flow_update_expiry(flow, tcph, udph, proto); 1367 | if(flow->detection_completed) return(0); 1368 | 1369 | flow->detected_protocol = ndpi_detection_process_packet(ndpi_thread_info[thread_id].ndpi_struct, ndpi_flow, 1370 | iph ? (uint8_t *)iph : (uint8_t *)iph6, 1371 | ipsize, time, src, dst); 1372 | 1373 | if(flow->detected_protocol.protocol != NDPI_PROTOCOL_UNKNOWN){ 1374 | flow->detection_completed = 1; 1375 | 1376 | if((flow->detected_protocol.protocol == NDPI_PROTOCOL_UNKNOWN) && (ndpi_flow->num_stun_udp_pkts > 0)) 1377 | ndpi_set_detected_protocol(ndpi_thread_info[thread_id].ndpi_struct, ndpi_flow, NDPI_PROTOCOL_STUN, NDPI_PROTOCOL_UNKNOWN); 1378 | 1379 | snprintf(flow->host_server_name, sizeof(flow->host_server_name), "%s", flow->ndpi_flow->host_server_name); 1380 | 1381 | if((proto == IPPROTO_TCP) && (flow->detected_protocol.protocol != NDPI_PROTOCOL_DNS)) { 1382 | snprintf(flow->ssl.client_certificate, sizeof(flow->ssl.client_certificate), "%s", flow->ndpi_flow->protos.ssl.client_certificate); 1383 | snprintf(flow->ssl.server_certificate, sizeof(flow->ssl.server_certificate), "%s", flow->ndpi_flow->protos.ssl.server_certificate); 1384 | } 1385 | 1386 | if(flow->ndpi_flow != NULL) free_ndpi_flow(flow); 1387 | 1388 | if(verbose > 1) { 1389 | if(enable_protocol_guess) { 1390 | if(flow->detected_protocol.protocol == NDPI_PROTOCOL_UNKNOWN) { 1391 | flow->detected_protocol.protocol = node_guess_undetected_protocol(thread_id, flow), 1392 | flow->detected_protocol.master_protocol = NDPI_PROTOCOL_UNKNOWN; 1393 | } 1394 | } 1395 | } 1396 | } 1397 | 1398 | if(ndpi_thread_info[thread_id].last_idle_scan_time + IDLE_SCAN_PERIOD < ndpi_thread_info[thread_id].last_time) { 1399 | /* scan for idle flows */ 1400 | ndpi_twalk(ndpi_thread_info[thread_id].ndpi_flows_root[ndpi_thread_info[thread_id].idle_scan_idx], node_idle_scan_walker, &thread_id); 1401 | 1402 | /* remove idle flows */ 1403 | remove_idle_flows(thread_id); 1404 | 1405 | if(++ndpi_thread_info[thread_id].idle_scan_idx == NUM_ROOTS) ndpi_thread_info[thread_id].idle_scan_idx = 0; 1406 | ndpi_thread_info[thread_id].last_idle_scan_time = ndpi_thread_info[thread_id].last_time; 1407 | } 1408 | 1409 | return 0; 1410 | } 1411 | 1412 | /* ***************************************************** */ 1413 | 1414 | static void close_pcap_file(u_int16_t thread_id) { 1415 | if(ndpi_thread_info[thread_id]._pcap_handle != NULL) { 1416 | pcap_close(ndpi_thread_info[thread_id]._pcap_handle); 1417 | 1418 | } 1419 | } 1420 | 1421 | /* ***************************************************** */ 1422 | 1423 | static void break_pcap_loop(u_int16_t thread_id) { 1424 | if(ndpi_thread_info[thread_id]._pcap_handle != NULL) { 1425 | pcap_breakloop(ndpi_thread_info[thread_id]._pcap_handle); 1426 | } 1427 | } 1428 | 1429 | /* ***************************************************** */ 1430 | 1431 | // executed for each packet in the pcap file 1432 | void sigproc(int sig) { 1433 | static int called = 0; 1434 | int thread_id; 1435 | 1436 | if(called) return; else called = 1; 1437 | shutdown_app = 1; 1438 | 1439 | for(thread_id=0; thread_idts.tv_sec, pcap_start.tv_usec = header->ts.tv_usec; 1585 | pcap_end.tv_sec = header->ts.tv_sec, pcap_end.tv_usec = header->ts.tv_usec; 1586 | } 1587 | 1588 | /* setting time */ 1589 | time = ((uint64_t) header->ts.tv_sec) * detection_tick_resolution + 1590 | header->ts.tv_usec / (1000000 / detection_tick_resolution); 1591 | 1592 | /* safety check */ 1593 | if(ndpi_thread_info[thread_id].last_time > time) { 1594 | /* printf("\nWARNING: timestamp bug in the pcap file (ts delta: %llu, repairing)\n", ndpi_thread_info[thread_id].last_time - time); */ 1595 | time = ndpi_thread_info[thread_id].last_time; 1596 | } 1597 | /* update last time value */ 1598 | ndpi_thread_info[thread_id].last_time = time; 1599 | 1600 | /*** check Data Link type ***/ 1601 | int datalink_type = ndpi_thread_info[thread_id]._pcap_datalink_type; 1602 | 1603 | datalink_check: 1604 | switch(datalink_type) { 1605 | case DLT_NULL : 1606 | if(ntohl(*((u_int32_t*)&packet[eth_offset])) == 2) 1607 | type = ETH_P_IP; 1608 | else 1609 | type = ETH_P_IPV6; 1610 | 1611 | ip_offset = 4 + eth_offset; 1612 | 1613 | /* Cisco PPP in HDLC-like framing - 50 */ 1614 | case DLT_PPP_SERIAL: 1615 | chdlc = (struct ndpi_chdlc *) &packet[eth_offset]; 1616 | ip_offset = sizeof(struct ndpi_chdlc); /* CHDLC_OFF = 4 */ 1617 | type = ntohs(chdlc->proto_code); 1618 | break; 1619 | 1620 | /* Cisco PPP with HDLC framing - 104 */ 1621 | case DLT_C_HDLC: 1622 | chdlc = (struct ndpi_chdlc *) &packet[eth_offset]; 1623 | ip_offset = sizeof(struct ndpi_chdlc); /* CHDLC_OFF = 4 */ 1624 | type = ntohs(chdlc->proto_code); 1625 | break; 1626 | 1627 | /* IEEE 802.3 Ethernet - 1 */ 1628 | case DLT_EN10MB : 1629 | ethernet = (struct ndpi_ethhdr *) &packet[eth_offset]; 1630 | ip_offset = sizeof(struct ndpi_ethhdr) + eth_offset; 1631 | check = ntohs(ethernet->h_proto); 1632 | 1633 | if(check <= 1500) 1634 | pyld_eth_len = check; 1635 | else if (check >= 1536) 1636 | type = check; 1637 | 1638 | if(pyld_eth_len != 0) { 1639 | /* check for LLC layer with SNAP extension */ 1640 | if(packet[ip_offset] == SNAP) { 1641 | llc = (struct ndpi_llc_header *)(&packet[ip_offset]); 1642 | type = llc->snap.proto_ID; 1643 | ip_offset += + 8; 1644 | } 1645 | } 1646 | break; 1647 | 1648 | /* Linux Cooked Capture - 113 */ 1649 | case DLT_LINUX_SLL : 1650 | type = (packet[eth_offset+14] << 8) + packet[eth_offset+15]; 1651 | ip_offset = 16 + eth_offset; 1652 | break; 1653 | 1654 | /* Radiotap link-layer - 127 */ 1655 | case DLT_IEEE802_11_RADIO : 1656 | radiotap = (struct ndpi_radiotap_header *) &packet[eth_offset]; 1657 | radio_len = radiotap->len; 1658 | 1659 | /* Check Bad FCS presence */ 1660 | if((radiotap->flags & BAD_FCS) == BAD_FCS) { 1661 | malformed_pkts += 1; 1662 | return; 1663 | } 1664 | 1665 | fcs = header->len - 4; 1666 | 1667 | /* Calculate 802.11 header length (variable) */ 1668 | wifi = (struct ndpi_wifi_header*)( packet + eth_offset + radio_len); 1669 | fc = wifi->fc; 1670 | 1671 | /* check wifi data presence */ 1672 | if(FCF_TYPE(fc) == WIFI_DATA) { 1673 | if((FCF_TO_DS(fc) && FCF_FROM_DS(fc) == 0x0) || 1674 | (FCF_TO_DS(fc) == 0x0 && FCF_FROM_DS(fc))) 1675 | wifi_len = 26; /* + 4 byte fcs */ 1676 | } else /* no data frames */ 1677 | break; 1678 | 1679 | /* Check ether_type from LLC */ 1680 | llc = (struct ndpi_llc_header*)(packet + eth_offset + wifi_len + radio_len); 1681 | if(llc->dsap == SNAP) 1682 | type = ntohs(llc->snap.proto_ID); 1683 | 1684 | /* Set IP header offset */ 1685 | ip_offset = wifi_len + radio_len + sizeof(struct ndpi_llc_header) + eth_offset; 1686 | break; 1687 | 1688 | case DLT_RAW: 1689 | ip_offset = eth_offset = 0; 1690 | break; 1691 | 1692 | default: 1693 | /* printf("Unknown datalink %d\n", datalink_type); */ 1694 | return; 1695 | } 1696 | 1697 | /* check ether type */ 1698 | if(type == VLAN) { 1699 | vlan_id = ((packet[ip_offset] << 8) + packet[ip_offset+1]) & 0xFFF; 1700 | type = (packet[ip_offset+2] << 8) + packet[ip_offset+3]; 1701 | ip_offset += 4; 1702 | vlan_packet = 1; 1703 | } else if(type == MPLS_UNI || type == MPLS_MULTI) { 1704 | mpls = (struct ndpi_mpls_header *) &packet[ip_offset]; 1705 | label = ntohl(mpls->label); 1706 | /* label = ntohl(*((u_int32_t*)&packet[ip_offset])); */ 1707 | type = ETH_P_IP, ip_offset += 4; 1708 | 1709 | while((label & 0x100) != 0x100) { 1710 | ip_offset += 4; 1711 | label = ntohl(mpls->label); 1712 | } 1713 | } 1714 | else if(type == SLARP) { 1715 | slarp = (struct ndpi_slarp *) &packet[ip_offset]; 1716 | if(slarp->slarp_type == 0x02 || slarp->slarp_type == 0x00 || slarp->slarp_type == 0x01) { 1717 | /* TODO if info are needed */ 1718 | } 1719 | slarp_pkts++; 1720 | } 1721 | else if(type == CISCO_D_PROTO) { 1722 | cdp = (struct ndpi_cdp *) &packet[ip_offset]; 1723 | cdp_pkts++; 1724 | } 1725 | else if(type == PPPoE) { 1726 | type = ETH_P_IP; 1727 | ip_offset += 8; 1728 | } 1729 | 1730 | 1731 | iph_check: 1732 | /* Check and set IP header size and total packet length */ 1733 | iph = (struct ndpi_iphdr *) &packet[ip_offset]; 1734 | 1735 | /* just work on Ethernet packets that contain IP */ 1736 | if(type == ETH_P_IP && header->caplen >= ip_offset) { 1737 | frag_off = ntohs(iph->frag_off); 1738 | 1739 | proto = iph->protocol; 1740 | if(header->caplen < header->len) { 1741 | static u_int8_t cap_warning_used = 0; 1742 | 1743 | if(cap_warning_used == 0) { 1744 | output(LOG_WARNING, "%s\n", "[WARN] packet capture size is smaller than packet size, DETECTION MIGHT NOT WORK CORRECTLY"); 1745 | cap_warning_used = 1; 1746 | } 1747 | } 1748 | } 1749 | 1750 | if(iph->version == 4) { 1751 | ip_len = ((u_short)iph->ihl * 4); 1752 | iph6 = NULL; 1753 | 1754 | if(iph->protocol == 41) { 1755 | ip_offset += ip_len; 1756 | goto iph_check; 1757 | } 1758 | 1759 | if((frag_off & 0x3FFF) != 0) { 1760 | static u_int8_t ipv4_frags_warning_used = 0; 1761 | if(ipv4_frags_warning_used == 0) { 1762 | output(LOG_WARNING, "%s\n", "[WARN] IPv4 fragments has not been supported yet"); 1763 | ipv4_frags_warning_used = 1; 1764 | } 1765 | return; 1766 | } 1767 | } else if(iph->version == 6) { 1768 | iph6 = (struct ndpi_ipv6hdr *)&packet[ip_offset]; 1769 | proto = iph6->ip6_ctlun.ip6_un1.ip6_un1_nxt; 1770 | ip_len = sizeof(struct ndpi_ipv6hdr); 1771 | 1772 | if(proto == 0x3C /* IPv6 destination option */) { 1773 | 1774 | u_int8_t *options = (u_int8_t*)&packet[ip_offset+ip_len]; 1775 | proto = options[0]; 1776 | ip_len += 8 * (options[1] + 1); 1777 | } 1778 | iph = NULL; 1779 | 1780 | } else { 1781 | static u_int8_t ipv4_warning_used = 0; 1782 | 1783 | v4_warning: 1784 | if(ipv4_warning_used == 0) { 1785 | if(!quiet_mode) 1786 | output(LOG_WARNING, "%s\n", "[WARN] only IPv4/IPv6 packets are supported by ndff, all other packets will be discarded"); 1787 | ipv4_warning_used = 1; 1788 | } 1789 | return; 1790 | } 1791 | 1792 | if(decode_tunnels && (proto == IPPROTO_UDP)) { 1793 | struct ndpi_udphdr *udp = (struct ndpi_udphdr *)&packet[ip_offset+ip_len]; 1794 | u_int16_t sport = ntohs(udp->source), dport = ntohs(udp->dest); 1795 | 1796 | if((sport == GTP_U_V1_PORT) || (dport == GTP_U_V1_PORT)) { 1797 | /* Check if it's GTPv1 */ 1798 | u_int offset = ip_offset+ip_len+sizeof(struct ndpi_udphdr); 1799 | u_int8_t flags = packet[offset]; 1800 | u_int8_t message_type = packet[offset+1]; 1801 | 1802 | if((((flags & 0xE0) >> 5) == 1 /* GTPv1 */) && 1803 | (message_type == 0xFF /* T-PDU */)) { 1804 | 1805 | ip_offset = ip_offset+ip_len+sizeof(struct ndpi_udphdr)+8; /* GTPv1 header len */ 1806 | if(flags & 0x04) ip_offset += 1; /* next_ext_header is present */ 1807 | if(flags & 0x02) ip_offset += 4; /* sequence_number is present (it also includes next_ext_header and pdu_number) */ 1808 | if(flags & 0x01) ip_offset += 1; /* pdu_number is present */ 1809 | 1810 | iph = (struct ndpi_iphdr *) &packet[ip_offset]; 1811 | 1812 | if(iph->version != 4) { 1813 | goto v4_warning; 1814 | } 1815 | } 1816 | } else if((sport == TZSP_PORT) || (dport == TZSP_PORT)) { 1817 | /* https://en.wikipedia.org/wiki/TZSP */ 1818 | u_int offset = ip_offset+ip_len+sizeof(struct ndpi_udphdr); 1819 | u_int8_t version = packet[offset]; 1820 | u_int8_t type = packet[offset+1]; 1821 | u_int16_t encapsulates = ntohs(*((u_int16_t*)&packet[offset+2])); 1822 | 1823 | if((version == 1) && (type == 0) && (encapsulates == 1)) { 1824 | u_int8_t stop = 0; 1825 | 1826 | offset += 4; 1827 | 1828 | while((!stop) && (offset < header->caplen)) { 1829 | u_int8_t tag_type = packet[offset]; 1830 | u_int8_t tag_len; 1831 | 1832 | switch(tag_type) { 1833 | case 0: /* PADDING Tag */ 1834 | tag_len = 1; 1835 | break; 1836 | case 1: /* END Tag */ 1837 | tag_len = 1, stop = 1; 1838 | break; 1839 | default: 1840 | tag_len = packet[offset+1]; 1841 | break; 1842 | } 1843 | 1844 | offset += tag_len; 1845 | 1846 | if(offset >= header->caplen) 1847 | return; /* Invalid packet */ 1848 | else { 1849 | eth_offset = offset; 1850 | goto datalink_check; 1851 | } 1852 | } 1853 | } 1854 | } 1855 | } 1856 | 1857 | /* process the packet */ 1858 | packet_processing(thread_id, time, vlan_id, iph, iph6, 1859 | ip_offset, header->len - ip_offset, header->len); 1860 | } 1861 | 1862 | /* ******************************************************************** */ 1863 | 1864 | static void run_pcap_loop(u_int16_t thread_id) { 1865 | if((!shutdown_app) && (ndpi_thread_info[thread_id]._pcap_handle != NULL)) 1866 | pcap_loop(ndpi_thread_info[thread_id]._pcap_handle, -1, &pcap_packet_callback, (u_char*)&thread_id); 1867 | } 1868 | 1869 | /* ******************************************************************** */ 1870 | 1871 | void *processing_thread(void *_thread_id) { 1872 | long thread_id = (long) _thread_id; 1873 | 1874 | #if defined(linux) && defined(HAVE_PTHREAD_SETAFFINITY_NP) 1875 | if(core_affinity[thread_id] >= 0) { 1876 | cpu_set_t cpuset; 1877 | CPU_ZERO(&cpuset); 1878 | CPU_SET(core_affinity[thread_id], &cpuset); 1879 | 1880 | if(pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpuset) != 0) 1881 | fprintf(stderr, "[ERROR] while binding thread %ld to core %d\n", thread_id, core_affinity[thread_id]); 1882 | else { 1883 | output(LOG_INFO, "[INFO] Running thread %ld on core %d...\n", thread_id, core_affinity[thread_id]); 1884 | } 1885 | } else 1886 | #endif 1887 | output(LOG_INFO, "[INFO] Running thread %ld...\n", thread_id); 1888 | 1889 | pcap_loop: 1890 | run_pcap_loop(thread_id); 1891 | 1892 | if(playlist_fp[thread_id] != NULL) { /* playlist: read next file */ 1893 | char filename[256]; 1894 | 1895 | if(get_next_pcap_file_from_playlist(thread_id, filename, sizeof(filename)) == 0 && 1896 | (ndpi_thread_info[thread_id]._pcap_handle = pcap_open_offline(filename, ndpi_thread_info[thread_id]._pcap_error_buffer)) != NULL) { 1897 | configure_pcap_handle(thread_id); 1898 | goto pcap_loop; 1899 | } 1900 | } 1901 | 1902 | return NULL; 1903 | } 1904 | 1905 | /* ******************************************************************** */ 1906 | 1907 | void run() { 1908 | struct timeval begin, end; 1909 | u_int64_t tot_usec; 1910 | long thread_id; 1911 | 1912 | for(thread_id = 0; thread_id < num_threads; thread_id++) { 1913 | setup_detection(thread_id); 1914 | if(!dryrun_flag) { 1915 | if(setup_socket(thread_id) == 0) exit(1); 1916 | } 1917 | open_pcap_file_or_device(thread_id); 1918 | } 1919 | 1920 | gettimeofday(&begin, NULL); 1921 | 1922 | /* Running processing threads */ 1923 | for(thread_id = 0; thread_id < num_threads; thread_id++) 1924 | pthread_create(&ndpi_thread_info[thread_id].pthread, NULL, processing_thread, (void *) thread_id); 1925 | 1926 | /* Waiting for completion */ 1927 | for(thread_id = 0; thread_id < num_threads; thread_id++) 1928 | pthread_join(ndpi_thread_info[thread_id].pthread, NULL); 1929 | 1930 | gettimeofday(&end, NULL); 1931 | tot_usec = end.tv_sec*1000000 + end.tv_usec - (begin.tv_sec*1000000 + begin.tv_usec); 1932 | 1933 | for(thread_id = 0; thread_id < num_threads; thread_id++) { 1934 | close_pcap_file(thread_id); 1935 | terminate_detection(thread_id); 1936 | close(_sockfd[thread_id]); 1937 | } 1938 | } 1939 | 1940 | /* ***************************************************** */ 1941 | 1942 | int main(int argc, char **argv) { 1943 | int i; 1944 | 1945 | memset(ndpi_thread_info, 0, sizeof(ndpi_thread_info)); 1946 | memset(&pcap_start, 0, sizeof(pcap_start)); 1947 | memset(&pcap_end, 0, sizeof(pcap_end)); 1948 | 1949 | parse_options(argc, argv); 1950 | 1951 | signal(SIGINT, sigproc); 1952 | 1953 | run(); 1954 | 1955 | if(results_path) free(results_path); 1956 | if(results_file) fclose(results_file); 1957 | 1958 | return 0; 1959 | } 1960 | 1961 | /* ****************************************************** */ 1962 | -------------------------------------------------------------------------------- /ndff.spec: -------------------------------------------------------------------------------- 1 | Summary: ndff - nDPI for fluentd 2 | Name: ndff 3 | Version: 0.0.2 4 | Release: 1 5 | Group: DeNA-Security 6 | Vendor: DeNA Security Dept 7 | License: GPL 8 | Source0: ndff-%{version}.tar.gz 9 | Source1: ndffd.in 10 | Source2: ndff.sysconfig 11 | BuildRoot: %{_tmppath}/%{name}-%{version}-buildroot 12 | BuildRequires: autoconf,automake,libtool,pkgconfig 13 | 14 | %description 15 | ndff package 16 | 17 | %prep 18 | %setup -q 19 | 20 | %build 21 | #./autogen.sh 22 | %configure 23 | make 24 | 25 | %install 26 | rm -rf $RPM_BUILD_ROOT 27 | %makeinstall 28 | 29 | mkdir -p $RPM_BUILD_ROOT%{_initrddir} 30 | install -m 755 %{SOURCE1} $RPM_BUILD_ROOT%{_initrddir}/ndffd 31 | mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/sysconfig 32 | install -m 644 %{SOURCE2} $RPM_BUILD_ROOT%{_sysconfdir}/sysconfig/ndff 33 | 34 | %clean 35 | rm -rf %{buildroot} 36 | 37 | %files 38 | %defattr(-,root,root) 39 | %{_bindir}/* 40 | %config %{_initrddir}/* 41 | %config %{_sysconfdir}/sysconfig/* 42 | 43 | %changelog 44 | * Sun Mar 13 2016 put.a.feud.pike011235@gmail.com 45 | - Initial package 46 | 47 | -------------------------------------------------------------------------------- /ndff.sysconfig: -------------------------------------------------------------------------------- 1 | # Config file for ndff startup 2 | 3 | # Options passed to the ndff program 4 | # Example: export JSON data from traffic on eth0 to localhost on port 24224 5 | OPTIONS="-i eth0 -s 127.0.0.1 -p 24224 -t ndff.flow -m json" 6 | -------------------------------------------------------------------------------- /ndffd.in: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # ndff - nDPI for fluentd 4 | # 5 | # chkconfig: 2345 95 05 6 | # description: Starts and stops the ndff 7 | 8 | # Source function library. 9 | . /etc/init.d/functions 10 | 11 | NDFF_CONF=/etc/sysconfig/ndff 12 | NDFF_LOCK=/var/lock/subsys/ndff 13 | NDFF_PROG=/usr/bin/ndff 14 | OPTIONS="-i eth0" 15 | 16 | # Source config 17 | if [ -f $NDFF_CONF ]; then 18 | . $NDFF_CONF 19 | fi 20 | 21 | [ -x $NDFF_PROG ] || exit 0 22 | 23 | RETVAL=0 24 | 25 | start() { 26 | echo -n $"Starting ndff: " 27 | daemon $NDFF_PROG $OPTIONS -d 28 | RETVAL=$? 29 | echo 30 | [ $RETVAL -eq 0 ] && touch $NDFF_LOCK 31 | return $RETVAL 32 | } 33 | stop() { 34 | echo -n $"Shutting down ndff: " 35 | killproc $NDFF_PROG 36 | RETVAL=$? 37 | echo 38 | [ $RETVAL -eq 0 ] && rm -f $NDFF_LOCK 39 | return $RETVAL 40 | } 41 | restart() { 42 | stop 43 | start 44 | } 45 | 46 | case "$1" in 47 | start) 48 | start 49 | ;; 50 | stop) 51 | stop 52 | ;; 53 | status) 54 | status $NDFF_PROG 55 | ;; 56 | restart|reload) 57 | restart 58 | ;; 59 | condrestart) 60 | [ -f $NDFF_LOCK ] && restart || : 61 | ;; 62 | *) 63 | echo $"Usage: $0 {start|stop|status|restart|condrestart}" 64 | exit 1 65 | esac 66 | 67 | exit $? 68 | -------------------------------------------------------------------------------- /protos.txt: -------------------------------------------------------------------------------- 1 | # Format: 2 | # :,:,.....@ 3 | 4 | tcp:81,tcp:8181@HTTP 5 | udp:5061-5062@SIP 6 | tcp:860,udp:860,tcp:3260,udp:3260@iSCSI 7 | tcp:3000@ntop 8 | 9 | # Subprotocols 10 | # Format: 11 | # host:"",host:"",.....@ 12 | 13 | host:"googlesyndacation.com"@Google 14 | host:"venere.com"@Venere 15 | host:"kataweb.it",host:"repubblica.it"@Repubblica 16 | host:"ntop"@ntop 17 | # IP based Subprotocols 18 | # Format: 19 | # ip:,ip:,.....@ 20 | 21 | ip:213.75.170.11@CustomProtocol 22 | 23 | 24 | 25 | -------------------------------------------------------------------------------- /tests/do.sh: -------------------------------------------------------------------------------- 1 | READER=../ndff 2 | 3 | RC=0 4 | PCAPS=`cd pcap; /bin/ls *.pcap` 5 | 6 | build_results() { 7 | for f in $PCAPS; do 8 | # create result files if not present 9 | if [ ! -f result/$f.out ]; then 10 | $READER -q -i pcap/$f -w result/$f.out -v 1 11 | fi 12 | done 13 | } 14 | 15 | check_results() { 16 | for f in $PCAPS; do 17 | if [ -f result/$f.out ]; then 18 | CMD="$READER -q -i pcap/$f -w /tmp/reader.out -v 1" 19 | $CMD 20 | NUM_DIFF=`diff result/$f.out /tmp/reader.out | wc -l` 21 | 22 | if [ $NUM_DIFF -eq 0 ]; then 23 | printf "%-32s\tOK\n" "$f" 24 | else 25 | printf "%-32s\tERROR\n" "$f" 26 | echo "$CMD" 27 | diff result/$f.out /tmp/reader.out 28 | RC=1 29 | fi 30 | 31 | /bin/rm /tmp/reader.out 32 | fi 33 | done 34 | } 35 | 36 | build_results 37 | check_results 38 | 39 | exit $RC 40 | -------------------------------------------------------------------------------- /tests/pcap/6in4tunnel.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/6in4tunnel.pcap -------------------------------------------------------------------------------- /tests/pcap/BGP_Cisco_hdlc_slarp.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/BGP_Cisco_hdlc_slarp.pcap -------------------------------------------------------------------------------- /tests/pcap/BGP_redist.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/BGP_redist.pcap -------------------------------------------------------------------------------- /tests/pcap/EAQ.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/EAQ.pcap -------------------------------------------------------------------------------- /tests/pcap/Instagram.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/Instagram.pcap -------------------------------------------------------------------------------- /tests/pcap/KakaoTalk_chat.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/KakaoTalk_chat.pcap -------------------------------------------------------------------------------- /tests/pcap/KakaoTalk_talk.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/KakaoTalk_talk.pcap -------------------------------------------------------------------------------- /tests/pcap/Meu.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/Meu.pcap -------------------------------------------------------------------------------- /tests/pcap/NTPv2.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/NTPv2.pcap -------------------------------------------------------------------------------- /tests/pcap/NTPv3.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/NTPv3.pcap -------------------------------------------------------------------------------- /tests/pcap/NTPv4.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/NTPv4.pcap -------------------------------------------------------------------------------- /tests/pcap/Oscar.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/Oscar.pcap -------------------------------------------------------------------------------- /tests/pcap/README.txt: -------------------------------------------------------------------------------- 1 | Place here test pcaps used for regressions testing 2 | -------------------------------------------------------------------------------- /tests/pcap/Torcedor.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/Torcedor.pcap -------------------------------------------------------------------------------- /tests/pcap/bittorrent.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/bittorrent.pcap -------------------------------------------------------------------------------- /tests/pcap/bt_search.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/bt_search.pcap -------------------------------------------------------------------------------- /tests/pcap/google_ssl.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/google_ssl.pcap -------------------------------------------------------------------------------- /tests/pcap/http_ipv6.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/http_ipv6.pcap -------------------------------------------------------------------------------- /tests/pcap/mpeg.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/mpeg.pcap -------------------------------------------------------------------------------- /tests/pcap/mpegts.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/mpegts.pcap -------------------------------------------------------------------------------- /tests/pcap/ocs.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/ocs.pcap -------------------------------------------------------------------------------- /tests/pcap/quic.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/quic.pcap -------------------------------------------------------------------------------- /tests/pcap/quickplay.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/quickplay.pcap -------------------------------------------------------------------------------- /tests/pcap/skype.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/skype.pcap -------------------------------------------------------------------------------- /tests/pcap/skype_no_unknown.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/skype_no_unknown.pcap -------------------------------------------------------------------------------- /tests/pcap/snapchat.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/snapchat.pcap -------------------------------------------------------------------------------- /tests/pcap/starcraft_battle.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/starcraft_battle.pcap -------------------------------------------------------------------------------- /tests/pcap/teredo.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/teredo.pcap -------------------------------------------------------------------------------- /tests/pcap/waze.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/waze.pcap -------------------------------------------------------------------------------- /tests/pcap/webex.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/webex.pcap -------------------------------------------------------------------------------- /tests/pcap/whatsapp_login_call.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/whatsapp_login_call.pcap -------------------------------------------------------------------------------- /tests/pcap/whatsapp_login_chat.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/whatsapp_login_chat.pcap -------------------------------------------------------------------------------- /tests/pcap/whatsapp_voice_and_message.pcap: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/knqyf263/ndff/427999751413f6a6a13f62b09c527abeee14456a/tests/pcap/whatsapp_voice_and_message.pcap --------------------------------------------------------------------------------