├── CHANGELOG.md ├── CNAME ├── LICENSE ├── README.md ├── _config.yml ├── config ├── kibana.ndjson └── plugin.yml ├── docker-compose.yml ├── docs ├── BUILD.md ├── ELASTICSEARCH.md ├── ESXI_Traffic_Mirroring.md ├── KIBANA.md ├── PF_RING.md └── images │ ├── Kernel_Upgrade_Reboot.png │ ├── delete_isolated_image.png │ ├── exsi_vSwitch.png │ ├── exsi_vSwitch_config.png │ ├── favicon.ico │ ├── logo.png │ └── logo_white.png ├── rules ├── README.md ├── apps.json └── nmap-service-probes └── test ├── logstash_tcp_send_test.py ├── nmap-service-probes ├── nmap-service-probes-patch.json ├── nmap-service-probes-patch.py └── nmap-service-probes_patch_202006181420 /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | 修改日志 2 | ========== 3 | 4 | 1.0.0 (2019-12-23) 5 | ------------------ 6 | - 初始版本。 -------------------------------------------------------------------------------- /CNAME: -------------------------------------------------------------------------------- 1 | dsolab.org -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | GNU GENERAL PUBLIC LICENSE 2 | Version 3, 29 June 2007 3 | 4 | Copyright (C) 2007 Free Software Foundation, Inc. 5 | Everyone is permitted to copy and distribute verbatim copies 6 | of this license document, but changing it is not allowed. 7 | 8 | Preamble 9 | 10 | The GNU General Public License is a free, copyleft license for 11 | software and other kinds of works. 12 | 13 | The licenses for most software and other practical works are designed 14 | to take away your freedom to share and change the works. By contrast, 15 | the GNU General Public License is intended to guarantee your freedom to 16 | share and change all versions of a program--to make sure it remains free 17 | software for all its users. We, the Free Software Foundation, use the 18 | GNU General Public License for most of our software; it applies also to 19 | any other work released this way by its authors. You can apply it to 20 | your programs, too. 21 | 22 | When we speak of free software, we are referring to freedom, not 23 | price. Our General Public Licenses are designed to make sure that you 24 | have the freedom to distribute copies of free software (and charge for 25 | them if you wish), that you receive source code or can get it if you 26 | want it, that you can change the software or use pieces of it in new 27 | free programs, and that you know you can do these things. 28 | 29 | To protect your rights, we need to prevent others from denying you 30 | these rights or asking you to surrender the rights. Therefore, you have 31 | certain responsibilities if you distribute copies of the software, or if 32 | you modify it: responsibilities to respect the freedom of others. 33 | 34 | For example, if you distribute copies of such a program, whether 35 | gratis or for a fee, you must pass on to the recipients the same 36 | freedoms that you received. You must make sure that they, too, receive 37 | or can get the source code. And you must show them these terms so they 38 | know their rights. 39 | 40 | Developers that use the GNU GPL protect your rights with two steps: 41 | (1) assert copyright on the software, and (2) offer you this License 42 | giving you legal permission to copy, distribute and/or modify it. 43 | 44 | For the developers' and authors' protection, the GPL clearly explains 45 | that there is no warranty for this free software. For both users' and 46 | authors' sake, the GPL requires that modified versions be marked as 47 | changed, so that their problems will not be attributed erroneously to 48 | authors of previous versions. 49 | 50 | Some devices are designed to deny users access to install or run 51 | modified versions of the software inside them, although the manufacturer 52 | can do so. This is fundamentally incompatible with the aim of 53 | protecting users' freedom to change the software. The systematic 54 | pattern of such abuse occurs in the area of products for individuals to 55 | use, which is precisely where it is most unacceptable. Therefore, we 56 | have designed this version of the GPL to prohibit the practice for those 57 | products. If such problems arise substantially in other domains, we 58 | stand ready to extend this provision to those domains in future versions 59 | of the GPL, as needed to protect the freedom of users. 60 | 61 | Finally, every program is threatened constantly by software patents. 62 | States should not allow patents to restrict development and use of 63 | software on general-purpose computers, but in those that do, we wish to 64 | avoid the special danger that patents applied to a free program could 65 | make it effectively proprietary. To prevent this, the GPL assures that 66 | patents cannot be used to render the program non-free. 67 | 68 | The precise terms and conditions for copying, distribution and 69 | modification follow. 70 | 71 | TERMS AND CONDITIONS 72 | 73 | 0. Definitions. 74 | 75 | "This License" refers to version 3 of the GNU General Public License. 76 | 77 | "Copyright" also means copyright-like laws that apply to other kinds of 78 | works, such as semiconductor masks. 79 | 80 | "The Program" refers to any copyrightable work licensed under this 81 | License. Each licensee is addressed as "you". "Licensees" and 82 | "recipients" may be individuals or organizations. 83 | 84 | To "modify" a work means to copy from or adapt all or part of the work 85 | in a fashion requiring copyright permission, other than the making of an 86 | exact copy. The resulting work is called a "modified version" of the 87 | earlier work or a work "based on" the earlier work. 88 | 89 | A "covered work" means either the unmodified Program or a work based 90 | on the Program. 91 | 92 | To "propagate" a work means to do anything with it that, without 93 | permission, would make you directly or secondarily liable for 94 | infringement under applicable copyright law, except executing it on a 95 | computer or modifying a private copy. Propagation includes copying, 96 | distribution (with or without modification), making available to the 97 | public, and in some countries other activities as well. 98 | 99 | To "convey" a work means any kind of propagation that enables other 100 | parties to make or receive copies. Mere interaction with a user through 101 | a computer network, with no transfer of a copy, is not conveying. 102 | 103 | An interactive user interface displays "Appropriate Legal Notices" 104 | to the extent that it includes a convenient and prominently visible 105 | feature that (1) displays an appropriate copyright notice, and (2) 106 | tells the user that there is no warranty for the work (except to the 107 | extent that warranties are provided), that licensees may convey the 108 | work under this License, and how to view a copy of this License. If 109 | the interface presents a list of user commands or options, such as a 110 | menu, a prominent item in the list meets this criterion. 111 | 112 | 1. Source Code. 113 | 114 | The "source code" for a work means the preferred form of the work 115 | for making modifications to it. "Object code" means any non-source 116 | form of a work. 117 | 118 | A "Standard Interface" means an interface that either is an official 119 | standard defined by a recognized standards body, or, in the case of 120 | interfaces specified for a particular programming language, one that 121 | is widely used among developers working in that language. 122 | 123 | The "System Libraries" of an executable work include anything, other 124 | than the work as a whole, that (a) is included in the normal form of 125 | packaging a Major Component, but which is not part of that Major 126 | Component, and (b) serves only to enable use of the work with that 127 | Major Component, or to implement a Standard Interface for which an 128 | implementation is available to the public in source code form. A 129 | "Major Component", in this context, means a major essential component 130 | (kernel, window system, and so on) of the specific operating system 131 | (if any) on which the executable work runs, or a compiler used to 132 | produce the work, or an object code interpreter used to run it. 133 | 134 | The "Corresponding Source" for a work in object code form means all 135 | the source code needed to generate, install, and (for an executable 136 | work) run the object code and to modify the work, including scripts to 137 | control those activities. However, it does not include the work's 138 | System Libraries, or general-purpose tools or generally available free 139 | programs which are used unmodified in performing those activities but 140 | which are not part of the work. For example, Corresponding Source 141 | includes interface definition files associated with source files for 142 | the work, and the source code for shared libraries and dynamically 143 | linked subprograms that the work is specifically designed to require, 144 | such as by intimate data communication or control flow between those 145 | subprograms and other parts of the work. 146 | 147 | The Corresponding Source need not include anything that users 148 | can regenerate automatically from other parts of the Corresponding 149 | Source. 150 | 151 | The Corresponding Source for a work in source code form is that 152 | same work. 153 | 154 | 2. Basic Permissions. 155 | 156 | All rights granted under this License are granted for the term of 157 | copyright on the Program, and are irrevocable provided the stated 158 | conditions are met. This License explicitly affirms your unlimited 159 | permission to run the unmodified Program. The output from running a 160 | covered work is covered by this License only if the output, given its 161 | content, constitutes a covered work. This License acknowledges your 162 | rights of fair use or other equivalent, as provided by copyright law. 163 | 164 | You may make, run and propagate covered works that you do not 165 | convey, without conditions so long as your license otherwise remains 166 | in force. You may convey covered works to others for the sole purpose 167 | of having them make modifications exclusively for you, or provide you 168 | with facilities for running those works, provided that you comply with 169 | the terms of this License in conveying all material for which you do 170 | not control copyright. Those thus making or running the covered works 171 | for you must do so exclusively on your behalf, under your direction 172 | and control, on terms that prohibit them from making any copies of 173 | your copyrighted material outside their relationship with you. 174 | 175 | Conveying under any other circumstances is permitted solely under 176 | the conditions stated below. Sublicensing is not allowed; section 10 177 | makes it unnecessary. 178 | 179 | 3. Protecting Users' Legal Rights From Anti-Circumvention Law. 180 | 181 | No covered work shall be deemed part of an effective technological 182 | measure under any applicable law fulfilling obligations under article 183 | 11 of the WIPO copyright treaty adopted on 20 December 1996, or 184 | similar laws prohibiting or restricting circumvention of such 185 | measures. 186 | 187 | When you convey a covered work, you waive any legal power to forbid 188 | circumvention of technological measures to the extent such circumvention 189 | is effected by exercising rights under this License with respect to 190 | the covered work, and you disclaim any intention to limit operation or 191 | modification of the work as a means of enforcing, against the work's 192 | users, your or third parties' legal rights to forbid circumvention of 193 | technological measures. 194 | 195 | 4. Conveying Verbatim Copies. 196 | 197 | You may convey verbatim copies of the Program's source code as you 198 | receive it, in any medium, provided that you conspicuously and 199 | appropriately publish on each copy an appropriate copyright notice; 200 | keep intact all notices stating that this License and any 201 | non-permissive terms added in accord with section 7 apply to the code; 202 | keep intact all notices of the absence of any warranty; and give all 203 | recipients a copy of this License along with the Program. 204 | 205 | You may charge any price or no price for each copy that you convey, 206 | and you may offer support or warranty protection for a fee. 207 | 208 | 5. Conveying Modified Source Versions. 209 | 210 | You may convey a work based on the Program, or the modifications to 211 | produce it from the Program, in the form of source code under the 212 | terms of section 4, provided that you also meet all of these conditions: 213 | 214 | a) The work must carry prominent notices stating that you modified 215 | it, and giving a relevant date. 216 | 217 | b) The work must carry prominent notices stating that it is 218 | released under this License and any conditions added under section 219 | 7. This requirement modifies the requirement in section 4 to 220 | "keep intact all notices". 221 | 222 | c) You must license the entire work, as a whole, under this 223 | License to anyone who comes into possession of a copy. This 224 | License will therefore apply, along with any applicable section 7 225 | additional terms, to the whole of the work, and all its parts, 226 | regardless of how they are packaged. This License gives no 227 | permission to license the work in any other way, but it does not 228 | invalidate such permission if you have separately received it. 229 | 230 | d) If the work has interactive user interfaces, each must display 231 | Appropriate Legal Notices; however, if the Program has interactive 232 | interfaces that do not display Appropriate Legal Notices, your 233 | work need not make them do so. 234 | 235 | A compilation of a covered work with other separate and independent 236 | works, which are not by their nature extensions of the covered work, 237 | and which are not combined with it such as to form a larger program, 238 | in or on a volume of a storage or distribution medium, is called an 239 | "aggregate" if the compilation and its resulting copyright are not 240 | used to limit the access or legal rights of the compilation's users 241 | beyond what the individual works permit. Inclusion of a covered work 242 | in an aggregate does not cause this License to apply to the other 243 | parts of the aggregate. 244 | 245 | 6. Conveying Non-Source Forms. 246 | 247 | You may convey a covered work in object code form under the terms 248 | of sections 4 and 5, provided that you also convey the 249 | machine-readable Corresponding Source under the terms of this License, 250 | in one of these ways: 251 | 252 | a) Convey the object code in, or embodied in, a physical product 253 | (including a physical distribution medium), accompanied by the 254 | Corresponding Source fixed on a durable physical medium 255 | customarily used for software interchange. 256 | 257 | b) Convey the object code in, or embodied in, a physical product 258 | (including a physical distribution medium), accompanied by a 259 | written offer, valid for at least three years and valid for as 260 | long as you offer spare parts or customer support for that product 261 | model, to give anyone who possesses the object code either (1) a 262 | copy of the Corresponding Source for all the software in the 263 | product that is covered by this License, on a durable physical 264 | medium customarily used for software interchange, for a price no 265 | more than your reasonable cost of physically performing this 266 | conveying of source, or (2) access to copy the 267 | Corresponding Source from a network server at no charge. 268 | 269 | c) Convey individual copies of the object code with a copy of the 270 | written offer to provide the Corresponding Source. This 271 | alternative is allowed only occasionally and noncommercially, and 272 | only if you received the object code with such an offer, in accord 273 | with subsection 6b. 274 | 275 | d) Convey the object code by offering access from a designated 276 | place (gratis or for a charge), and offer equivalent access to the 277 | Corresponding Source in the same way through the same place at no 278 | further charge. You need not require recipients to copy the 279 | Corresponding Source along with the object code. If the place to 280 | copy the object code is a network server, the Corresponding Source 281 | may be on a different server (operated by you or a third party) 282 | that supports equivalent copying facilities, provided you maintain 283 | clear directions next to the object code saying where to find the 284 | Corresponding Source. Regardless of what server hosts the 285 | Corresponding Source, you remain obligated to ensure that it is 286 | available for as long as needed to satisfy these requirements. 287 | 288 | e) Convey the object code using peer-to-peer transmission, provided 289 | you inform other peers where the object code and Corresponding 290 | Source of the work are being offered to the general public at no 291 | charge under subsection 6d. 292 | 293 | A separable portion of the object code, whose source code is excluded 294 | from the Corresponding Source as a System Library, need not be 295 | included in conveying the object code work. 296 | 297 | A "User Product" is either (1) a "consumer product", which means any 298 | tangible personal property which is normally used for personal, family, 299 | or household purposes, or (2) anything designed or sold for incorporation 300 | into a dwelling. In determining whether a product is a consumer product, 301 | doubtful cases shall be resolved in favor of coverage. For a particular 302 | product received by a particular user, "normally used" refers to a 303 | typical or common use of that class of product, regardless of the status 304 | of the particular user or of the way in which the particular user 305 | actually uses, or expects or is expected to use, the product. A product 306 | is a consumer product regardless of whether the product has substantial 307 | commercial, industrial or non-consumer uses, unless such uses represent 308 | the only significant mode of use of the product. 309 | 310 | "Installation Information" for a User Product means any methods, 311 | procedures, authorization keys, or other information required to install 312 | and execute modified versions of a covered work in that User Product from 313 | a modified version of its Corresponding Source. The information must 314 | suffice to ensure that the continued functioning of the modified object 315 | code is in no case prevented or interfered with solely because 316 | modification has been made. 317 | 318 | If you convey an object code work under this section in, or with, or 319 | specifically for use in, a User Product, and the conveying occurs as 320 | part of a transaction in which the right of possession and use of the 321 | User Product is transferred to the recipient in perpetuity or for a 322 | fixed term (regardless of how the transaction is characterized), the 323 | Corresponding Source conveyed under this section must be accompanied 324 | by the Installation Information. But this requirement does not apply 325 | if neither you nor any third party retains the ability to install 326 | modified object code on the User Product (for example, the work has 327 | been installed in ROM). 328 | 329 | The requirement to provide Installation Information does not include a 330 | requirement to continue to provide support service, warranty, or updates 331 | for a work that has been modified or installed by the recipient, or for 332 | the User Product in which it has been modified or installed. Access to a 333 | network may be denied when the modification itself materially and 334 | adversely affects the operation of the network or violates the rules and 335 | protocols for communication across the network. 336 | 337 | Corresponding Source conveyed, and Installation Information provided, 338 | in accord with this section must be in a format that is publicly 339 | documented (and with an implementation available to the public in 340 | source code form), and must require no special password or key for 341 | unpacking, reading or copying. 342 | 343 | 7. Additional Terms. 344 | 345 | "Additional permissions" are terms that supplement the terms of this 346 | License by making exceptions from one or more of its conditions. 347 | Additional permissions that are applicable to the entire Program shall 348 | be treated as though they were included in this License, to the extent 349 | that they are valid under applicable law. If additional permissions 350 | apply only to part of the Program, that part may be used separately 351 | under those permissions, but the entire Program remains governed by 352 | this License without regard to the additional permissions. 353 | 354 | When you convey a copy of a covered work, you may at your option 355 | remove any additional permissions from that copy, or from any part of 356 | it. (Additional permissions may be written to require their own 357 | removal in certain cases when you modify the work.) You may place 358 | additional permissions on material, added by you to a covered work, 359 | for which you have or can give appropriate copyright permission. 360 | 361 | Notwithstanding any other provision of this License, for material you 362 | add to a covered work, you may (if authorized by the copyright holders of 363 | that material) supplement the terms of this License with terms: 364 | 365 | a) Disclaiming warranty or limiting liability differently from the 366 | terms of sections 15 and 16 of this License; or 367 | 368 | b) Requiring preservation of specified reasonable legal notices or 369 | author attributions in that material or in the Appropriate Legal 370 | Notices displayed by works containing it; or 371 | 372 | c) Prohibiting misrepresentation of the origin of that material, or 373 | requiring that modified versions of such material be marked in 374 | reasonable ways as different from the original version; or 375 | 376 | d) Limiting the use for publicity purposes of names of licensors or 377 | authors of the material; or 378 | 379 | e) Declining to grant rights under trademark law for use of some 380 | trade names, trademarks, or service marks; or 381 | 382 | f) Requiring indemnification of licensors and authors of that 383 | material by anyone who conveys the material (or modified versions of 384 | it) with contractual assumptions of liability to the recipient, for 385 | any liability that these contractual assumptions directly impose on 386 | those licensors and authors. 387 | 388 | All other non-permissive additional terms are considered "further 389 | restrictions" within the meaning of section 10. If the Program as you 390 | received it, or any part of it, contains a notice stating that it is 391 | governed by this License along with a term that is a further 392 | restriction, you may remove that term. If a license document contains 393 | a further restriction but permits relicensing or conveying under this 394 | License, you may add to a covered work material governed by the terms 395 | of that license document, provided that the further restriction does 396 | not survive such relicensing or conveying. 397 | 398 | If you add terms to a covered work in accord with this section, you 399 | must place, in the relevant source files, a statement of the 400 | additional terms that apply to those files, or a notice indicating 401 | where to find the applicable terms. 402 | 403 | Additional terms, permissive or non-permissive, may be stated in the 404 | form of a separately written license, or stated as exceptions; 405 | the above requirements apply either way. 406 | 407 | 8. Termination. 408 | 409 | You may not propagate or modify a covered work except as expressly 410 | provided under this License. Any attempt otherwise to propagate or 411 | modify it is void, and will automatically terminate your rights under 412 | this License (including any patent licenses granted under the third 413 | paragraph of section 11). 414 | 415 | However, if you cease all violation of this License, then your 416 | license from a particular copyright holder is reinstated (a) 417 | provisionally, unless and until the copyright holder explicitly and 418 | finally terminates your license, and (b) permanently, if the copyright 419 | holder fails to notify you of the violation by some reasonable means 420 | prior to 60 days after the cessation. 421 | 422 | Moreover, your license from a particular copyright holder is 423 | reinstated permanently if the copyright holder notifies you of the 424 | violation by some reasonable means, this is the first time you have 425 | received notice of violation of this License (for any work) from that 426 | copyright holder, and you cure the violation prior to 30 days after 427 | your receipt of the notice. 428 | 429 | Termination of your rights under this section does not terminate the 430 | licenses of parties who have received copies or rights from you under 431 | this License. If your rights have been terminated and not permanently 432 | reinstated, you do not qualify to receive new licenses for the same 433 | material under section 10. 434 | 435 | 9. Acceptance Not Required for Having Copies. 436 | 437 | You are not required to accept this License in order to receive or 438 | run a copy of the Program. Ancillary propagation of a covered work 439 | occurring solely as a consequence of using peer-to-peer transmission 440 | to receive a copy likewise does not require acceptance. However, 441 | nothing other than this License grants you permission to propagate or 442 | modify any covered work. These actions infringe copyright if you do 443 | not accept this License. Therefore, by modifying or propagating a 444 | covered work, you indicate your acceptance of this License to do so. 445 | 446 | 10. Automatic Licensing of Downstream Recipients. 447 | 448 | Each time you convey a covered work, the recipient automatically 449 | receives a license from the original licensors, to run, modify and 450 | propagate that work, subject to this License. You are not responsible 451 | for enforcing compliance by third parties with this License. 452 | 453 | An "entity transaction" is a transaction transferring control of an 454 | organization, or substantially all assets of one, or subdividing an 455 | organization, or merging organizations. If propagation of a covered 456 | work results from an entity transaction, each party to that 457 | transaction who receives a copy of the work also receives whatever 458 | licenses to the work the party's predecessor in interest had or could 459 | give under the previous paragraph, plus a right to possession of the 460 | Corresponding Source of the work from the predecessor in interest, if 461 | the predecessor has it or can get it with reasonable efforts. 462 | 463 | You may not impose any further restrictions on the exercise of the 464 | rights granted or affirmed under this License. For example, you may 465 | not impose a license fee, royalty, or other charge for exercise of 466 | rights granted under this License, and you may not initiate litigation 467 | (including a cross-claim or counterclaim in a lawsuit) alleging that 468 | any patent claim is infringed by making, using, selling, offering for 469 | sale, or importing the Program or any portion of it. 470 | 471 | 11. Patents. 472 | 473 | A "contributor" is a copyright holder who authorizes use under this 474 | License of the Program or a work on which the Program is based. The 475 | work thus licensed is called the contributor's "contributor version". 476 | 477 | A contributor's "essential patent claims" are all patent claims 478 | owned or controlled by the contributor, whether already acquired or 479 | hereafter acquired, that would be infringed by some manner, permitted 480 | by this License, of making, using, or selling its contributor version, 481 | but do not include claims that would be infringed only as a 482 | consequence of further modification of the contributor version. For 483 | purposes of this definition, "control" includes the right to grant 484 | patent sublicenses in a manner consistent with the requirements of 485 | this License. 486 | 487 | Each contributor grants you a non-exclusive, worldwide, royalty-free 488 | patent license under the contributor's essential patent claims, to 489 | make, use, sell, offer for sale, import and otherwise run, modify and 490 | propagate the contents of its contributor version. 491 | 492 | In the following three paragraphs, a "patent license" is any express 493 | agreement or commitment, however denominated, not to enforce a patent 494 | (such as an express permission to practice a patent or covenant not to 495 | sue for patent infringement). To "grant" such a patent license to a 496 | party means to make such an agreement or commitment not to enforce a 497 | patent against the party. 498 | 499 | If you convey a covered work, knowingly relying on a patent license, 500 | and the Corresponding Source of the work is not available for anyone 501 | to copy, free of charge and under the terms of this License, through a 502 | publicly available network server or other readily accessible means, 503 | then you must either (1) cause the Corresponding Source to be so 504 | available, or (2) arrange to deprive yourself of the benefit of the 505 | patent license for this particular work, or (3) arrange, in a manner 506 | consistent with the requirements of this License, to extend the patent 507 | license to downstream recipients. "Knowingly relying" means you have 508 | actual knowledge that, but for the patent license, your conveying the 509 | covered work in a country, or your recipient's use of the covered work 510 | in a country, would infringe one or more identifiable patents in that 511 | country that you have reason to believe are valid. 512 | 513 | If, pursuant to or in connection with a single transaction or 514 | arrangement, you convey, or propagate by procuring conveyance of, a 515 | covered work, and grant a patent license to some of the parties 516 | receiving the covered work authorizing them to use, propagate, modify 517 | or convey a specific copy of the covered work, then the patent license 518 | you grant is automatically extended to all recipients of the covered 519 | work and works based on it. 520 | 521 | A patent license is "discriminatory" if it does not include within 522 | the scope of its coverage, prohibits the exercise of, or is 523 | conditioned on the non-exercise of one or more of the rights that are 524 | specifically granted under this License. You may not convey a covered 525 | work if you are a party to an arrangement with a third party that is 526 | in the business of distributing software, under which you make payment 527 | to the third party based on the extent of your activity of conveying 528 | the work, and under which the third party grants, to any of the 529 | parties who would receive the covered work from you, a discriminatory 530 | patent license (a) in connection with copies of the covered work 531 | conveyed by you (or copies made from those copies), or (b) primarily 532 | for and in connection with specific products or compilations that 533 | contain the covered work, unless you entered into that arrangement, 534 | or that patent license was granted, prior to 28 March 2007. 535 | 536 | Nothing in this License shall be construed as excluding or limiting 537 | any implied license or other defenses to infringement that may 538 | otherwise be available to you under applicable patent law. 539 | 540 | 12. No Surrender of Others' Freedom. 541 | 542 | If conditions are imposed on you (whether by court order, agreement or 543 | otherwise) that contradict the conditions of this License, they do not 544 | excuse you from the conditions of this License. If you cannot convey a 545 | covered work so as to satisfy simultaneously your obligations under this 546 | License and any other pertinent obligations, then as a consequence you may 547 | not convey it at all. For example, if you agree to terms that obligate you 548 | to collect a royalty for further conveying from those to whom you convey 549 | the Program, the only way you could satisfy both those terms and this 550 | License would be to refrain entirely from conveying the Program. 551 | 552 | 13. Use with the GNU Affero General Public License. 553 | 554 | Notwithstanding any other provision of this License, you have 555 | permission to link or combine any covered work with a work licensed 556 | under version 3 of the GNU Affero General Public License into a single 557 | combined work, and to convey the resulting work. The terms of this 558 | License will continue to apply to the part which is the covered work, 559 | but the special requirements of the GNU Affero General Public License, 560 | section 13, concerning interaction through a network will apply to the 561 | combination as such. 562 | 563 | 14. Revised Versions of this License. 564 | 565 | The Free Software Foundation may publish revised and/or new versions of 566 | the GNU General Public License from time to time. Such new versions will 567 | be similar in spirit to the present version, but may differ in detail to 568 | address new problems or concerns. 569 | 570 | Each version is given a distinguishing version number. If the 571 | Program specifies that a certain numbered version of the GNU General 572 | Public License "or any later version" applies to it, you have the 573 | option of following the terms and conditions either of that numbered 574 | version or of any later version published by the Free Software 575 | Foundation. If the Program does not specify a version number of the 576 | GNU General Public License, you may choose any version ever published 577 | by the Free Software Foundation. 578 | 579 | If the Program specifies that a proxy can decide which future 580 | versions of the GNU General Public License can be used, that proxy's 581 | public statement of acceptance of a version permanently authorizes you 582 | to choose that version for the Program. 583 | 584 | Later license versions may give you additional or different 585 | permissions. However, no additional obligations are imposed on any 586 | author or copyright holder as a result of your choosing to follow a 587 | later version. 588 | 589 | 15. Disclaimer of Warranty. 590 | 591 | THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY 592 | APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT 593 | HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY 594 | OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, 595 | THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR 596 | PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM 597 | IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF 598 | ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 599 | 600 | 16. Limitation of Liability. 601 | 602 | IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING 603 | WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS 604 | THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY 605 | GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE 606 | USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF 607 | DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD 608 | PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), 609 | EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF 610 | SUCH DAMAGES. 611 | 612 | 17. Interpretation of Sections 15 and 16. 613 | 614 | If the disclaimer of warranty and limitation of liability provided 615 | above cannot be given local legal effect according to their terms, 616 | reviewing courts shall apply local law that most closely approximates 617 | an absolute waiver of all civil liability in connection with the 618 | Program, unless a warranty or assumption of liability accompanies a 619 | copy of the Program in return for a fee. 620 | 621 | END OF TERMS AND CONDITIONS 622 | 623 | How to Apply These Terms to Your New Programs 624 | 625 | If you develop a new program, and you want it to be of the greatest 626 | possible use to the public, the best way to achieve this is to make it 627 | free software which everyone can redistribute and change under these terms. 628 | 629 | To do so, attach the following notices to the program. It is safest 630 | to attach them to the start of each source file to most effectively 631 | state the exclusion of warranty; and each file should have at least 632 | the "copyright" line and a pointer to where the full notice is found. 633 | 634 | 635 | Copyright (C) 636 | 637 | This program is free software: you can redistribute it and/or modify 638 | it under the terms of the GNU General Public License as published by 639 | the Free Software Foundation, either version 3 of the License, or 640 | (at your option) any later version. 641 | 642 | This program is distributed in the hope that it will be useful, 643 | but WITHOUT ANY WARRANTY; without even the implied warranty of 644 | MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 645 | GNU General Public License for more details. 646 | 647 | You should have received a copy of the GNU General Public License 648 | along with this program. If not, see . 649 | 650 | Also add information on how to contact you by electronic and paper mail. 651 | 652 | If the program does terminal interaction, make it output a short 653 | notice like this when it starts in an interactive mode: 654 | 655 | Copyright (C) 656 | This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. 657 | This is free software, and you are welcome to redistribute it 658 | under certain conditions; type `show c' for details. 659 | 660 | The hypothetical commands `show w' and `show c' should show the appropriate 661 | parts of the General Public License. Of course, your program's commands 662 | might be different; for a GUI interface, you would use an "about box". 663 | 664 | You should also get your employer (if you work as a programmer) or school, 665 | if any, to sign a "copyright disclaimer" for the program, if necessary. 666 | For more information on this, and how to apply and follow the GNU GPL, see 667 | . 668 | 669 | The GNU General Public License does not permit incorporating your program 670 | into proprietary programs. If your program is a subroutine library, you 671 | may consider it more useful to permit linking proprietary applications with 672 | the library. If this is what you want to do, use the GNU Lesser General 673 | Public License instead of this License. But first, please read 674 | . 675 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ## ![](docs/images/logo.png) 2 | 3 | [![Python 3.x](https://img.shields.io/badge/python-3.x-yellow.svg)](https://www.python.org/) [![Ruby 2.5](https://img.shields.io/badge/ruby-2.5-red.svg)](https://www.ruby-lang.org/) [![Go 1.13.x](https://img.shields.io/badge/go-1.x-green.svg)](https://github.com/golang/go) [![Java 1.8](https://img.shields.io/badge/java-1.8-red.svg)](https://www.java.com/) [![Platform linux&docker](https://img.shields.io/badge/platform-linux&docker-9cf.svg)](https://www.docker.com/) [![License](https://img.shields.io/badge/license-GPLv3-red.svg)](https://github.com/DSO-Lab/passets/raw/master/LICENSE) 4 | 5 | 6 | ## 概述 7 | 8 | passets 是一套由 [DSO 安全实验室（DSO Security Lab）](http://www.dsolab.org)开源的被动资产识别框架，用于基于被动流量的网络资产发现。整体框架包括四大组件： 9 | 10 | **[流量采集模块](https://github.com/DSO-Lab/passets-sensor)** 11 | > 提供流量采集及基本的流量过滤和处理。 12 | 13 | **数据存储模块（ELK）** 14 | > 采用成熟的 [Elasticsearch](docs/ELASTICSEARCH.md) + [Logstash](https://github.com/DSO-Lab/passets-logstash) + [Kibana](docs/KIBANA.md) 框架， 负责采集数据的基本加工、存储及索引等。 15 | 16 | **[数据清洗模块](https://github.com/DSO-Lab/passets-filter)** 17 | > 根据数据清洗插件对数据进行深度清洗、标记和重组。 18 | 19 | **[数据发布模块](https://github.com/DSO-Lab/passets-api)** 20 | > 负责向第三方应用提供采集的资产数据。 21 | 22 | 23 | ## 特性 24 | 25 | * 从采集到数据清洗全部支持分布式架构 26 | * 基于流量的资产指纹识别 27 | * 支持多种硬件架构（x86、ARM） 28 | * 容器化部署，上手容易 29 | * 更多... 30 | 31 | 32 | ## 环境 33 | 34 | ### 软件 35 | 36 | - Linux 37 | - [Docker](https://www.docker.com/) 38 | - [docker-compose](https://github.com/docker/compose) 39 | 40 | ### 硬件 41 | 42 | 用户需要自行准备一台至少包含两块网卡的计算机设备。其中一个网卡用来接收交换机镜像的流量，另一个网卡则用于通过Web API向其它应用提供数据。 43 | 44 | 用户提供的计算机设备最小配置要求如下（默认保留近30天数据）： 45 | 46 | | 参数项 | 百兆网(推荐配置) | 千兆网(推荐配置) | 47 | | ------ | ---------------- | ---------------- | 48 | | CPU | 四核 | 八核 | 49 | | 内存 | 8G | 16G | 50 | | 存储 | 100G | 500G | 51 | 52 | ## 部署 53 | 54 | 依赖环境安装： 55 | 56 | ```bash 57 | curl -fsSL https://get.docker.com/ | sh 58 | yum -y install docker-compose 59 | systemctl start docker 60 | systemctl enable docker 61 | ``` 62 | 63 | ### 单机快速部署 64 | 65 | **第一步**：点击 [这里](https://github.com/DSO-Lab/passets/archive/master.tar.gz) 下载最新的部署文件包并解压缩： 66 | 67 | ```bash 68 | curl -L https://github.com/DSO-Lab/passets/archive/master.tar.gz -o master.tar.gz 69 | tar -zxf master.tar.gz 70 | cd passets-master/ 71 | ``` 72 | 73 | **第二步**：修改 [`docker-compose`] 配置文件中下列参数： 74 | 75 | ``` 76 | services: 77 | api: 78 | ... 79 | environment: 80 | ... 81 | # 配置API访问密钥，该密钥用于通过 API 获取资产数据 82 | SECRET= 83 | ... 84 | sensor: 85 | ... 86 | environment： 87 | # 流量镜像网卡配置 88 | - interface=<网卡编号> 89 | ``` 90 | 91 | **第三步**：获取最新的指纹库、IP定位数据库 92 | 93 | 获取最新的指纹库，保存到当前目录下的 rules 目录下。 94 | ``` bash 95 | # Wappalyzer 指纹库 96 | curl -L https://github.com/AliasIO/Wappalyzer/raw/master/src/apps.json -o ./rules/apps.json 97 | 98 | # NMAP 指纹库 99 | curl -L https://svn.nmap.org/nmap/nmap-service-probes -o ./rules/nmap-service-probes 100 | ``` 101 | 102 | IP 地址数据库的更新请参阅 [passets-logstash](https://github.com/DSO-Lab/passets-logstash) 的文档说明。 103 | 104 | **第四步**：定义自己的内部IP地址 105 | 106 | 内部IP地址默认已配置为通用局域网IP地址，用户可以通过修改 `docker-compose.yml` 中 Logstash 容器的 `INNER_IP_LIST` 环境变量来自定义内部 IP 地址列表。格式为“<起始IP>-<结束IP>,IP”（单条记录支持IP范围和单个IP，多条记录用半角逗号分隔，不能有空格），配置示例如下： 107 | 108 | ``` 109 | version: "3" 110 | 111 | services: 112 | ... ... 113 | 114 | logstash: 115 | ... ... 116 | environment: 117 | ... ... 118 | - INNER_IP_LIST=10.0.0.0-10.255.255.255,172.16.0.0-172.31.255.255,192.168.0.0-192.168.255.255,169.254.0.0-169.254.255.255,127.0.0.1-127.0.0.255 119 | ``` 120 | 121 | **第五步**：创建数据、日志目录并赋权 122 | 123 | 在 docker-compose.yml 文件同级目录下创建下列目录： 124 | ``` bash 125 | mkdir -p data/elasticsearch/nodes 126 | mkdir -p data/logstash 127 | mkdir -p data/kibana 128 | mkdir -p data/logs 129 | chown -R 1000 data 130 | # E.L.K. 默认使用 uid 为 1000的用户运行 131 | ``` 132 | 133 | **第六步**：启动应用 134 | 135 | ``` bash 136 | docker-compose up -d 137 | ``` 138 | 139 | 启动时将首先从 Docker HUB 上下载各模块的镜像文件，然后运行，可能需要等待一段时间。等待系统启动完成后，可以通过下面的地址访问 API 接口： 140 | 141 | http://x.x.x.x:8081/swagger-ui.html 142 | 143 | 或者通过集成的 Kibana 进行可视化查询或展示（[Kibana 配置方法示例](docs/KIBANA.md)）： 144 | 145 | http://x.x.x.x:5601/ 146 | 147 | ### 集群部署 148 | 149 | 采用集群部署时，需要根据需要创建各个模块的 docker-compose 配置文件，配置文件中需要调整的主要为环境变量和映射卷，主要配置参数请参考下表： 150 | 151 | #### Elasticsearch 152 | 153 | | 类型 | 参数名 | 参数说明 154 | |--------------|---------------------|--------------------------------------------------| 155 | | environment | discovery.type | 单节点模式时需配置为 single-node 156 | | volumes | /usr/share/elasticsearch/data | Elasticsearch 数据目录 157 | | volumes | /usr/share/elasticsearch/logs | Elasticsearch 日志目录 158 | | port | 9200 | 对外开放的 ES 端口，仅集群模式下主节点需要配置 159 | 160 | #### Logstash 161 | 162 | | 类型 | 参数名 | 参数说明 163 | |--------------|---------------------|--------------------------------------------------| 164 | | environment | ELASTICSEARCH_URL | ES 服务器地址:端口 165 | | environment | ELASTICSEARCH_INDEX | ES 索引前缀，系统会自动在后面追加 “-yyyyMMdd”，不建议修改 166 | | environment | INNER_IP_LIST | 用于定义企业的内部IP地址，支持地址段和地址，多个以逗号分隔，不允许出现空格 167 | | volumes | /usr/share/logstash/data | Logstash 数据目录 168 | | volumes | /usr/share/logstash/logs | Logstash 日志目录 169 | 170 | #### Kibana 171 | 172 | | 类型 | 参数名 | 参数说明 173 | |--------------|------------------------|--------------------------------------------------| 174 | | environment | ELASTICSEARCH_HOSTS | ES 服务器 URL，示例：http://127.0.0.1:9200 175 | | environment | I18N_LOCALE | Kibana 使用的界面语言 176 | | volumes | /usr/share/kibana/data | Kibana 数据目录 177 | | volumes | /usr/share/kibana/logs | Kibana 日志目录 178 | | port | 5601 | 对外开放的 Kibana 端口 179 | 180 | #### Passets-Sensor 181 | 182 | | 类型 | 参数名 | 参数说明 183 | |--------------|---------------------|--------------------------------------------------| 184 | | environment | interface | 流量采集网口编号，与宿主机相同，例如：eth0、ens160等 185 | | environment | ip | Logstash 服务器地址（只有集群模式才需要修改） 186 | | environment | port | Logstash 服务器接收数据的端口 187 | | environment | tag | 记录标识，分布式场景下用于识别数据来自哪个采集模块 188 | | environment | cache | 采集器缓存的已处理记录数，缓存时间120秒，先进先出 189 | | environment | timeout | 采集器内存回收超时时间，超过指定的时间，采集器自动回收内存 190 | | environment | debug | 调试模式开关，生产环境禁用，将产生大量输出 191 | 192 | #### Passets-Filter 193 | 194 | | 类型 | 参数名 | 参数说明 195 | |--------------|---------------------|--------------------------------------------------| 196 | | environment | ELASTICSEARCH_URL | ES 服务器地址:端口 197 | | environment | ELASTICSEARCH_INDEX | ES 索引前缀，需与Logstash配置保持一致 198 | | environment | THREADS | 清洗线程数，默认10，建议不超过20 199 | | environment | RANGE | 首次处理数据的时间提前量，单位分钟，默认15 200 | | environment | BATCH_SIZE | 每线程单次处理的数据条数，默认20条 201 | | environment | CACHE_SIZE | 清洗模块缓存的已处理记录数，默认1024 202 | | environment | CACHE_TTL | 清洗模块缓存的过期时间，单位秒，默认600 203 | | environment | MODE | 工作模式：1-主节点，0-从节点，默认为1 204 | | environment | DEBUG | 调试模式开关，0以上整数，生产环境禁用 205 | | volumes | /opt/filter/rules/ | Wappalyzer、NMAP指纹库文件 206 | | volumes | /opt/filter/config/plugin.yml | 清洗插件配置，控制插件的启用和关闭，可不配置此项 207 | 208 | #### Passets-Api 209 | 210 | | 类型 | 参数名 | 参数说明 211 | |--------------|---------------------|--------------------------------------------------| 212 | | environment | ELASTICSEARCH_URL | ES 服务器地址:端口 213 | | environment | ELASTICSEARCH_INDEX | ES 索引前缀，需与Logstash配置保持一致 214 | | environment | SECRET | API 认证密钥，任意不含空格的字符串，该密钥用于填充 API 模块的 X-Auth-Secret 参数，实现 API 访问认证 215 | | port | 8080 | 对外开放的 API 端口 216 | 217 | 218 | **注** 此处仅列出了与各模块正常运行密切相关的一些配置参数，实际配置中还有一些跟容器相关的参数需要配置，例如： network、restart、depends_on、container_name、image等，请参考 [docker-compose](docker-compose.yml) 配置文件根据实际情况进行配置。 219 | 220 | ## 帮助文档 221 | 222 | 详细配置说明参见 [```docs```](./docs) 目录中的文档。 223 | 224 | 225 | ## 目录结构 226 | 227 | ``` 228 | ├─ docker-compose.yml # docker-compose 配置文件（X86_64） 229 | ├─ config # 配置文件 230 | │ ├─ kibana.ndjson # Kibana 配置文件 231 | │ └─ plugin.yml # 数据清洗模块配置文件 232 | ├─ docs # 文档 233 | │ └─ ... ... 234 | ├─ rules # 指纹库 235 | │ ├─ apps.json # Wappalyzer 开源Web指纹库 236 | │ └─ nmap-service-probes # NMAP 指纹库 237 | └─data # 容器数据目录，需自行创建 238 | ├─ elasticsearch # Elasticsearch 数据 239 | ├─ logstash # Logstash 数据 240 | ├─ kibana # Kibana 数据 241 | └─ logs # E.L.K 日志 242 | ``` 243 | 244 | ## 链接 245 | 246 | * [修改日志](./CHANGELOG.md) 247 | * [缺陷跟踪](https://github.com/DSO-Lab/passets/issues) 248 | * [软件授权](./LICENSE) 249 | -------------------------------------------------------------------------------- /_config.yml: -------------------------------------------------------------------------------- 1 | theme: jekyll-theme-architect -------------------------------------------------------------------------------- /config/kibana.ndjson: -------------------------------------------------------------------------------- 1 | {"attributes":{"fields":"[{\"name\":\"@timestamp\",\"type\":\"date\",\"esTypes\":[\"date\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"@version\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"_id\",\"type\":\"string\",\"esTypes\":[\"_id\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_index\",\"type\":\"string\",\"esTypes\":[\"_index\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_score\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_source\",\"type\":\"_source\",\"esTypes\":[\"_source\"],\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_type\",\"type\":\"string\",\"esTypes\":[\"_type\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"apps.categories.id\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"apps.categories.name\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"apps.categories.name.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"apps.categories.name\",\"subType\":\"multi\"},{\"name\":\"apps.confidence\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"apps.name\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"apps.name.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"apps.name\",\"subType\":\"multi\"},{\"name\":\"body\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"body.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"body\",\"subType\":\"multi\"},{\"name\":\"code\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"code.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"code\",\"subType\":\"multi\"},{\"name\":\"data\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"data.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"data\",\"subType\":\"multi\"},{\"name\":\"geoip.city_name\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"geoip.city_name.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"geoip.city_name\",\"subType\":\"multi\"},{\"name\":\"geoip.country_name\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"geoip.country_name.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"geoip.country_name\",\"subType\":\"multi\"},{\"name\":\"geoip.location\",\"type\":\"geo_point\",\"esTypes\":[\"geo_point\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"header\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"header.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"header\",\"subType\":\"multi\"},{\"name\":\"host\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"host.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"host\",\"subType\":\"multi\"},{\"name\":\"inner\",\"type\":\"boolean\",\"esTypes\":[\"boolean\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ip\",\"type\":\"ip\",\"esTypes\":[\"ip\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ip_str\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"ip_str.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"ip_str\",\"subType\":\"multi\"},{\"name\":\"method\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"method.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"method\",\"subType\":\"multi\"},{\"name\":\"path\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"path.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"path\",\"subType\":\"multi\"},{\"name\":\"port\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"pro\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"pro.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"pro\",\"subType\":\"multi\"},{\"name\":\"server\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"server.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"server\",\"subType\":\"multi\"},{\"name\":\"site\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"site.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"site\",\"subType\":\"multi\"},{\"name\":\"state\",\"type\":\"number\",\"esTypes\":[\"long\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"tag\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"tag.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"tag\",\"subType\":\"multi\"},{\"name\":\"tags\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"tags.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"tags\",\"subType\":\"multi\"},{\"name\":\"title\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"title.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"title\",\"subType\":\"multi\"},{\"name\":\"type\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"type.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"type\",\"subType\":\"multi\"},{\"name\":\"url\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"url.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"url\",\"subType\":\"multi\"},{\"name\":\"url_tpl\",\"type\":\"string\",\"esTypes\":[\"text\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"url_tpl.keyword\",\"type\":\"string\",\"esTypes\":[\"keyword\"],\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true,\"parent\":\"url_tpl\",\"subType\":\"multi\"}]","timeFieldName":"@timestamp","title":"logstash-passets*"},"id":"2b1d8bf0-22f5-11ea-86b1-e14502bf26c7","migrationVersion":{"index-pattern":"6.5.0"},"references":[],"type":"index-pattern","updated_at":"2020-08-07T10:43:19.093Z","version":"WzU4MDI4OSwyMF0="} 2 | {"attributes":{"description":"平台监测到的所有IP地址。","kibanaSavedObjectMeta":{"searchSourceJSON":"{\n \"query\": {\n \"query\": \"\",\n \"language\": \"kuery\"\n },\n \"filter\": [],\n \"indexRefName\": \"kibanaSavedObjectMeta.searchSourceJSON.index\"\n}"},"title":"IP 数量","uiStateJSON":"{}","version":1,"visState":"{\n \"title\": \"IP 数量\",\n \"type\": \"metric\",\n \"params\": {\n \"metric\": {\n \"percentageMode\": false,\n \"useRanges\": false,\n \"colorSchema\": \"Blues\",\n \"metricColorMode\": \"Labels\",\n \"colorsRange\": [\n {\n \"type\": \"range\",\n \"from\": 0,\n \"to\": 100000\n },\n {\n \"type\": \"range\",\n \"from\": 100000,\n \"to\": 1000000\n }\n ],\n \"labels\": {\n \"show\": false\n },\n \"invertColors\": true,\n \"style\": {\n \"bgFill\": \"#000\",\n \"bgColor\": false,\n \"labelColor\": true,\n \"subText\": \"\",\n \"fontSize\": 40\n }\n },\n \"dimensions\": {\n \"metrics\": [\n {\n \"type\": \"vis_dimension\",\n \"accessor\": 0,\n \"format\": {\n \"id\": \"number\",\n \"params\": {}\n }\n }\n ]\n },\n \"addTooltip\": true,\n \"addLegend\": false,\n \"type\": \"metric\"\n },\n \"aggs\": [\n {\n \"id\": \"1\",\n \"enabled\": true,\n \"type\": \"cardinality\",\n \"schema\": \"metric\",\n \"params\": {\n \"field\": \"ip\",\n \"customLabel\": \"IP 地址数\"\n }\n }\n ]\n}"},"id":"6216afb0-1030-11ea-a7f8-65c9feeb13c9","migrationVersion":{"visualization":"7.4.2"},"references":[{"id":"2b1d8bf0-22f5-11ea-86b1-e14502bf26c7","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2020-08-07T10:47:30.507Z","version":"WzU4MDI5MywyMF0="} 3 | {"attributes":{"description":"平台监测到的全部应用数量。","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"应用数量","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"应用数量\",\"type\":\"metric\",\"params\":{\"metric\":{\"percentageMode\":false,\"useRanges\":false,\"colorSchema\":\"Blues\",\"metricColorMode\":\"Labels\",\"colorsRange\":[{\"type\":\"range\",\"from\":0,\"to\":100000},{\"type\":\"range\",\"from\":100000,\"to\":1000000}],\"labels\":{\"show\":false},\"invertColors\":true,\"style\":{\"bgFill\":\"#000\",\"bgColor\":false,\"labelColor\":true,\"subText\":\"\",\"fontSize\":40}},\"dimensions\":{\"metrics\":[{\"type\":\"vis_dimension\",\"accessor\":0,\"format\":{\"id\":\"number\",\"params\":{}}}]},\"addTooltip\":true,\"addLegend\":false,\"type\":\"metric\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"cardinality\",\"schema\":\"metric\",\"params\":{\"field\":\"host.keyword\",\"customLabel\":\"服务数\"}}]}"},"id":"d372c450-1030-11ea-a7f8-65c9feeb13c9","migrationVersion":{"visualization":"7.4.2"},"references":[{"id":"2b1d8bf0-22f5-11ea-86b1-e14502bf26c7","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2020-08-07T10:42:05.878Z","version":"WzU4MDI3NywyMF0="} 4 | {"attributes":{"columns":["title","url","header"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\n \"highlightAll\": true,\n \"version\": true,\n \"query\": {\n \"query\": \"pro: \\\"HTTP\\\"\",\n \"language\": \"kuery\"\n },\n \"filter\": [],\n \"indexRefName\": \"kibanaSavedObjectMeta.searchSourceJSON.index\"\n}"},"sort":[["@timestamp","desc"]],"title":"HTTP","version":1},"id":"becf02a0-1276-11ea-951a-fda8bb7aefb8","migrationVersion":{"search":"7.4.0"},"references":[{"id":"2b1d8bf0-22f5-11ea-86b1-e14502bf26c7","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"search","updated_at":"2020-08-07T10:42:05.878Z","version":"WzU4MDI3OCwyMF0="} 5 | {"attributes":{"description":"平台监测到的全部URL（已排除参数变化的情况）。","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"URL 数量","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"URL 数量\",\"type\":\"metric\",\"params\":{\"metric\":{\"percentageMode\":false,\"useRanges\":false,\"colorSchema\":\"Blues\",\"metricColorMode\":\"Labels\",\"colorsRange\":[{\"type\":\"range\",\"from\":0,\"to\":100000},{\"type\":\"range\",\"from\":100000,\"to\":1000000}],\"labels\":{\"show\":false},\"invertColors\":true,\"style\":{\"bgFill\":\"#000\",\"bgColor\":false,\"labelColor\":true,\"subText\":\"\",\"fontSize\":40}},\"dimensions\":{\"metrics\":[{\"type\":\"vis_dimension\",\"accessor\":0,\"format\":{\"id\":\"number\",\"params\":{}}}]},\"addTooltip\":true,\"addLegend\":false,\"type\":\"metric\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"cardinality\",\"schema\":\"metric\",\"params\":{\"field\":\"url_tpl.keyword\",\"customLabel\":\"URL数\"}}]}"},"id":"17d2ff80-1030-11ea-a7f8-65c9feeb13c9","migrationVersion":{"visualization":"7.4.2"},"references":[{"id":"becf02a0-1276-11ea-951a-fda8bb7aefb8","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2020-08-07T10:42:05.878Z","version":"WzU4MDI3OSwyMF0="} 6 | {"attributes":{"description":"监测到的各时段数据量变化。","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"title":"数据发现时序图","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"数据发现时序图\",\"type\":\"area\",\"params\":{\"type\":\"area\",\"grid\":{\"categoryLines\":true,\"valueAxis\":\"ValueAxis-1\"},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"filter\":true,\"truncate\":100,\"rotate\":0},\"title\":{}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"数量\"}}],\"seriesParams\":[{\"show\":\"true\",\"type\":\"area\",\"mode\":\"stacked\",\"data\":{\"label\":\"数量\",\"id\":\"1\"},\"drawLinesBetweenPoints\":true,\"showCircles\":true,\"interpolate\":\"cardinal\",\"valueAxis\":\"ValueAxis-1\"}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"top\",\"times\":[],\"addTimeMarker\":true,\"thresholdLine\":{\"show\":true,\"value\":10,\"width\":1,\"style\":\"dashed\",\"color\":\"#DB1374\"},\"labels\":{},\"dimensions\":{\"x\":{\"accessor\":0,\"format\":{\"id\":\"date\",\"params\":{\"pattern\":\"HH:mm\"}},\"params\":{\"date\":true,\"interval\":\"PT10M\",\"format\":\"HH:mm\",\"bounds\":{\"min\":\"2019-11-28T07:37:17.742Z\",\"max\":\"2019-11-29T07:37:17.742Z\"}},\"aggType\":\"date_histogram\"},\"y\":[{\"accessor\":1,\"format\":{\"id\":\"number\"},\"params\":{},\"aggType\":\"count\"}]}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"数量\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"timeRange\":{\"from\":\"now-24h\",\"to\":\"now\"},\"useNormalizedEsInterval\":true,\"scaleMetricValues\":true,\"interval\":\"s\",\"drop_partials\":true,\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\"时间\"}}]}"},"id":"d9ac6c40-1279-11ea-951a-fda8bb7aefb8","migrationVersion":{"visualization":"7.4.2"},"references":[{"id":"2b1d8bf0-22f5-11ea-86b1-e14502bf26c7","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"visualization","updated_at":"2020-08-07T10:42:05.878Z","version":"WzU4MDI4MCwyMF0="} 7 | {"attributes":{"description":"访问最频繁的外部站点。","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"inner:false \",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"热门外部站点","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"热门外部站点\",\"type\":\"pie\",\"params\":{\"type\":\"pie\",\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":true,\"labels\":{\"show\":true,\"values\":false,\"last_level\":true,\"truncate\":100},\"dimensions\":{\"metric\":{\"accessor\":1,\"format\":{\"id\":\"number\"},\"params\":{},\"aggType\":\"count\"},\"buckets\":[{\"accessor\":0,\"format\":{\"id\":\"string\"},\"params\":{},\"aggType\":\"significant_terms\"}]}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"数量\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"site.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":15,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\",\"customLabel\":\"站点\"}}]}"},"id":"cafdcb90-26c3-11ea-b265-b9ada7395b5f","migrationVersion":{"visualization":"7.4.2"},"references":[{"id":"becf02a0-1276-11ea-951a-fda8bb7aefb8","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2020-08-07T10:42:05.878Z","version":"WzU4MDI4MSwyMF0="} 8 | {"attributes":{"description":"使用最多的应用组件。","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"热门应用指纹","uiStateJSON":"{}","version":1,"visState":"{\"title\":\"热门应用指纹\",\"type\":\"tagcloud\",\"params\":{\"scale\":\"log\",\"orientation\":\"right angled\",\"minFontSize\":10,\"maxFontSize\":80,\"showLabel\":false,\"metric\":{\"type\":\"vis_dimension\",\"accessor\":1,\"format\":{\"id\":\"string\",\"params\":{}}},\"bucket\":{\"type\":\"vis_dimension\",\"accessor\":0,\"format\":{\"id\":\"terms\",\"params\":{\"id\":\"string\",\"otherBucketLabel\":\"Other\",\"missingBucketLabel\":\"Missing\"}}}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"数量\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"apps.name.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":50,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"Missing\"}}]}"},"id":"3068f150-127c-11ea-951a-fda8bb7aefb8","migrationVersion":{"visualization":"7.4.2"},"references":[{"id":"becf02a0-1276-11ea-951a-fda8bb7aefb8","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2020-08-07T10:42:05.878Z","version":"WzU4MDI4MiwyMF0="} 9 | {"attributes":{"columns":["ip","port"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\n \"highlightAll\": true,\n \"version\": true,\n \"query\": {\n \"query\": \"pro: \\\"TCP\\\"\",\n \"language\": \"kuery\"\n },\n \"filter\": [],\n \"indexRefName\": \"kibanaSavedObjectMeta.searchSourceJSON.index\"\n}"},"sort":[["@timestamp","desc"]],"title":"TCP","version":1},"id":"a04356b0-1276-11ea-951a-fda8bb7aefb8","migrationVersion":{"search":"7.4.0"},"references":[{"id":"2b1d8bf0-22f5-11ea-86b1-e14502bf26c7","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"}],"type":"search","updated_at":"2020-08-07T10:42:05.878Z","version":"WzU4MDI4MywyMF0="} 10 | {"attributes":{"description":"访问最频繁的内部应用服务端口。","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"inner:true\",\"language\":\"kuery\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"热门内部应用","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":null,\"direction\":null}}}}","version":1,"visState":"{\"title\":\"热门内部应用\",\"type\":\"table\",\"params\":{\"perPage\":10,\"showPartialRows\":false,\"showMetricsAtAllLevels\":false,\"sort\":{\"columnIndex\":null,\"direction\":null},\"showTotal\":false,\"totalFunc\":\"sum\",\"percentageCol\":\"\",\"dimensions\":{\"metrics\":[{\"accessor\":2,\"format\":{\"id\":\"number\"},\"params\":{},\"aggType\":\"count\"}],\"buckets\":[{\"accessor\":0,\"format\":{\"id\":\"terms\",\"params\":{\"id\":\"string\",\"otherBucketLabel\":\"Other\",\"missingBucketLabel\":\"Missing\"}},\"params\":{},\"aggType\":\"terms\"},{\"accessor\":1,\"format\":{\"id\":\"terms\",\"params\":{\"id\":\"number\",\"otherBucketLabel\":\"Other\",\"missingBucketLabel\":\"Missing\"}},\"params\":{},\"aggType\":\"terms\"}]}},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"数量\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"host.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":10,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"--\",\"customLabel\":\"应用\"}}]}"},"id":"fb682d30-102e-11ea-a7f8-65c9feeb13c9","migrationVersion":{"visualization":"7.4.2"},"references":[{"id":"a04356b0-1276-11ea-951a-fda8bb7aefb8","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2020-08-07T10:42:05.878Z","version":"WzU4MDI4NCwyMF0="} 11 | {"attributes":{"description":"访问最频繁的内部网站。","kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"inner: true\"},\"filter\":[]}"},"savedSearchRefName":"search_0","title":"热门内部站点","uiStateJSON":"{\"vis\":{\"params\":{\"sort\":{\"columnIndex\":2,\"direction\":\"desc\"}}}}","version":1,"visState":"{\"title\":\"热门内部站点\",\"type\":\"table\",\"params\":{\"dimensions\":{\"buckets\":[{\"accessor\":0,\"aggType\":\"terms\",\"format\":{\"id\":\"terms\",\"params\":{\"id\":\"string\",\"missingBucketLabel\":\"--\",\"otherBucketLabel\":\"Other\"}},\"params\":{}}],\"metrics\":[{\"accessor\":1,\"aggType\":\"count\",\"format\":{\"id\":\"number\"},\"params\":{}}]},\"perPage\":10,\"percentageCol\":\"\",\"showMetricsAtAllLevels\":false,\"showPartialRows\":false,\"showTotal\":false,\"sort\":{\"columnIndex\":2,\"direction\":\"desc\"},\"totalFunc\":\"sum\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"数量\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"bucket\",\"params\":{\"field\":\"site.keyword\",\"orderBy\":\"1\",\"order\":\"desc\",\"size\":10,\"otherBucket\":false,\"otherBucketLabel\":\"Other\",\"missingBucket\":false,\"missingBucketLabel\":\"--\",\"customLabel\":\"站点\"}}]}"},"id":"3c6a17a0-102d-11ea-a7f8-65c9feeb13c9","migrationVersion":{"visualization":"7.4.2"},"references":[{"id":"becf02a0-1276-11ea-951a-fda8bb7aefb8","name":"search_0","type":"search"}],"type":"visualization","updated_at":"2020-08-07T10:42:05.878Z","version":"WzU4MDI4NSwyMF0="} 12 | {"attributes":{"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filter\":[]}"},"optionsJSON":"{\"hidePanelTitles\":false,\"useMargins\":true}","panelsJSON":"[{\"version\":\"7.5.1\",\"gridData\":{\"x\":0,\"y\":0,\"w\":16,\"h\":6,\"i\":\"9fed3d05-87ab-464d-8e6e-3aea69d5e3d5\"},\"panelIndex\":\"9fed3d05-87ab-464d-8e6e-3aea69d5e3d5\",\"embeddableConfig\":{},\"panelRefName\":\"panel_0\"},{\"version\":\"7.5.1\",\"gridData\":{\"x\":16,\"y\":0,\"w\":16,\"h\":6,\"i\":\"1e0b9c6c-be64-4e82-b74a-39e77741f0ab\"},\"panelIndex\":\"1e0b9c6c-be64-4e82-b74a-39e77741f0ab\",\"embeddableConfig\":{},\"panelRefName\":\"panel_1\"},{\"version\":\"7.5.1\",\"gridData\":{\"x\":32,\"y\":0,\"w\":16,\"h\":6,\"i\":\"1f04cf67-155d-41d5-8124-cae58b530571\"},\"panelIndex\":\"1f04cf67-155d-41d5-8124-cae58b530571\",\"embeddableConfig\":{},\"panelRefName\":\"panel_2\"},{\"version\":\"7.5.1\",\"gridData\":{\"x\":0,\"y\":6,\"w\":24,\"h\":15,\"i\":\"015d0777-70ea-4e78-96c3-4a906a36fa22\"},\"panelIndex\":\"015d0777-70ea-4e78-96c3-4a906a36fa22\",\"embeddableConfig\":{},\"panelRefName\":\"panel_3\"},{\"version\":\"7.5.1\",\"gridData\":{\"x\":24,\"y\":6,\"w\":24,\"h\":15,\"i\":\"737298ba-2778-4ab2-8801-4afed132f0ef\"},\"panelIndex\":\"737298ba-2778-4ab2-8801-4afed132f0ef\",\"embeddableConfig\":{},\"panelRefName\":\"panel_4\"},{\"version\":\"7.5.1\",\"gridData\":{\"x\":0,\"y\":21,\"w\":24,\"h\":17,\"i\":\"fdc7af63-2b8b-431a-ac16-8b4ccaece0b0\"},\"panelIndex\":\"fdc7af63-2b8b-431a-ac16-8b4ccaece0b0\",\"embeddableConfig\":{},\"panelRefName\":\"panel_5\"},{\"version\":\"7.5.1\",\"gridData\":{\"x\":36,\"y\":21,\"w\":12,\"h\":17,\"i\":\"e80b12b0-d9b8-4eb0-9d3a-63ad611c8b99\"},\"panelIndex\":\"e80b12b0-d9b8-4eb0-9d3a-63ad611c8b99\",\"embeddableConfig\":{},\"panelRefName\":\"panel_6\"},{\"version\":\"7.5.1\",\"gridData\":{\"x\":24,\"y\":21,\"w\":12,\"h\":17,\"i\":\"479ba67a-eb98-4d5e-a6b0-b87127e6bac1\"},\"panelIndex\":\"479ba67a-eb98-4d5e-a6b0-b87127e6bac1\",\"embeddableConfig\":{},\"panelRefName\":\"panel_7\"}]","timeRestore":false,"title":"Default","version":1},"id":"130f0440-102f-11ea-a7f8-65c9feeb13c9","migrationVersion":{"dashboard":"7.3.0"},"references":[{"id":"6216afb0-1030-11ea-a7f8-65c9feeb13c9","name":"panel_0","type":"visualization"},{"id":"d372c450-1030-11ea-a7f8-65c9feeb13c9","name":"panel_1","type":"visualization"},{"id":"17d2ff80-1030-11ea-a7f8-65c9feeb13c9","name":"panel_2","type":"visualization"},{"id":"d9ac6c40-1279-11ea-951a-fda8bb7aefb8","name":"panel_3","type":"visualization"},{"id":"cafdcb90-26c3-11ea-b265-b9ada7395b5f","name":"panel_4","type":"visualization"},{"id":"3068f150-127c-11ea-951a-fda8bb7aefb8","name":"panel_5","type":"visualization"},{"id":"fb682d30-102e-11ea-a7f8-65c9feeb13c9","name":"panel_6","type":"visualization"},{"id":"3c6a17a0-102d-11ea-a7f8-65c9feeb13c9","name":"panel_7","type":"visualization"}],"type":"dashboard","updated_at":"2020-08-07T10:42:05.878Z","version":"WzU4MDI4NiwyMF0="} 13 | {"attributes":{"bounds":{"coordinates":[[[-141.45294,77.96551],[-141.45294,-32.97593],[170.60474,-32.97593],[170.60474,77.96551],[-141.45294,77.96551]]],"type":"Polygon"},"description":"","layerListJSON":"[{\"sourceDescriptor\":{\"type\":\"EMS_TMS\",\"isAutoSelect\":true},\"id\":\"b7a700a1-563c-4016-b199-7e20c31305ab\",\"label\":null,\"minZoom\":0,\"maxZoom\":24,\"alpha\":1,\"visible\":true,\"applyGlobalQuery\":true,\"style\":{},\"type\":\"VECTOR_TILE\"},{\"sourceDescriptor\":{\"id\":\"400c2799-a6de-4071-854c-f659bfe91e9a\",\"type\":\"ES_SEARCH\",\"geoField\":\"geoip.location\",\"filterByMapBounds\":true,\"tooltipProperties\":[\"ip\",\"geoip.country_name\",\"geoip.city_name\"],\"useTopHits\":false,\"topHitsSize\":1,\"sortField\":\"@timestamp\",\"sortOrder\":\"desc\",\"indexPatternRefName\":\"layer_1_source_index_pattern\"},\"style\":{\"type\":\"VECTOR\",\"properties\":{\"fillColor\":{\"type\":\"STATIC\",\"options\":{\"color\":\"#F37020\"}},\"lineColor\":{\"type\":\"STATIC\",\"options\":{\"color\":\"#FFF\"}},\"lineWidth\":{\"type\":\"STATIC\",\"options\":{\"size\":1}},\"iconSize\":{\"type\":\"STATIC\",\"options\":{\"size\":8}},\"iconOrientation\":{\"type\":\"STATIC\",\"options\":{\"orientation\":0}},\"symbol\":{\"options\":{\"symbolizeAs\":\"circle\",\"symbolId\":\"airfield\"}}}},\"id\":\"3e597057-951f-4c0e-9e3a-7eb5b8d9f6f8\",\"label\":\"外部IP\",\"minZoom\":0,\"maxZoom\":24,\"alpha\":0.75,\"visible\":true,\"applyGlobalQuery\":true,\"type\":\"VECTOR\",\"query\":{\"query\":\"inner: false \",\"language\":\"kuery\"}}]","mapStateJSON":"{\"zoom\":1.69,\"center\":{\"lon\":14.5759,\"lat\":42.45292},\"timeFilters\":{\"from\":\"now-15m\",\"to\":\"now\"},\"refreshConfig\":{\"isPaused\":false,\"interval\":0},\"query\":{\"language\":\"kuery\",\"query\":\"\"},\"filters\":[]}","title":"外部IP热力图","uiStateJSON":"{\"isLayerTOCOpen\":true,\"openTOCDetails\":[]}"},"id":"4937fec0-1ff0-11ea-b09c-5b1868b70b29","migrationVersion":{"map":"7.5.0"},"references":[{"id":"2b1d8bf0-22f5-11ea-86b1-e14502bf26c7","name":"layer_1_source_index_pattern","type":"index-pattern"}],"type":"map","updated_at":"2020-08-07T10:50:12.757Z","version":"WzU4MDI5NywyMF0="} 14 | {"exportedCount":13,"missingRefCount":0,"missingReferences":[]} -------------------------------------------------------------------------------- /config/plugin.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # 根节点：插件名称，与plugins目录下的文件名对应 3 | # 二级节点： 4 | # - enable: 是否启用插件 5 | # - index: 插件的处理顺序，0以上的整数，数值越小 6 | 7 | # ip: 8 | # enable: false 9 | # index: 1 10 | # inner_ips: 11 | # - 10.0.0.0-10.255.255.255 12 | # - 172.16.0.0-172.31.255.255 13 | # - 192.168.0.0-192.168.255.255 14 | # - 169.254.0.0-169.254.255.255 15 | # - 127.0.0.1-127.0.0.255 16 | 17 | # geoip: 18 | # enable: false 19 | # index: 2 20 | 21 | # url: 22 | # enable: false 23 | # index: 3 24 | 25 | wappalyzer: 26 | enable: true 27 | index: 1 28 | 29 | nmap: 30 | enable: true 31 | index: 2 32 | 33 | 34 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | api: 5 | image: dsolab/passets-api:1.0.0 6 | container_name: passets-api 7 | environment: 8 | - TZ=CST-8 9 | - ELASTICSEARCH_URL=passets-elasticsearch:9200 10 | - ELASTICSEARCH_INDEX=logstash-passets 11 | - SECRET= 12 | ports: 13 | - 8081:8080 14 | networks: 15 | - passets 16 | restart: unless-stopped 17 | depends_on: 18 | - elasticsearch 19 | 20 | elasticsearch: 21 | image: elasticsearch:7.5.1 22 | container_name: passets-elasticsearch 23 | environment: 24 | - TZ=CST-8 25 | - discovery.type=single-node 26 | ports: 27 | - 9200:9200 28 | volumes: 29 | - ./data/elasticsearch:/usr/share/elasticsearch/data 30 | - ./data/logs:/usr/share/elasticsearch/logs 31 | networks: 32 | - passets 33 | restart: unless-stopped 34 | 35 | logstash: 36 | image: dsolab/passets-logstash:1.0.0 37 | container_name: passets-logstash 38 | environment: 39 | - TZ=CST-8 40 | - ELASTICSEARCH_URL=passets-elasticsearch:9200 41 | - ELASTICSEARCH_INDEX=logstash-passets 42 | - INNER_IP_LIST=10.0.0.0-10.255.255.255,172.16.0.0-172.31.255.255,192.168.0.0-192.168.255.255,169.254.0.0-169.254.255.255,127.0.0.1-127.0.0.255 43 | ports: 44 | - 5044:5044/tcp 45 | volumes: 46 | - ./data/logstash:/usr/share/logstash/data 47 | - ./data/logs:/usr/share/logstash/logs 48 | restart: unless-stopped 49 | depends_on: 50 | - elasticsearch 51 | networks: 52 | passets: 53 | ipv4_address: 172.20.20.100 54 | 55 | filter: 56 | image: dsolab/passets-filter:1.0.0 57 | container_name: passets-filter 58 | environment: 59 | - TZ=CST-8 60 | - ELASTICSEARCH_URL=passets-elasticsearch:9200 61 | - ELASTICSEARCH_INDEX=logstash-passets 62 | - THREADS=20 63 | - CACHE_SIZE=256 64 | - DEBUG=0 65 | volumes: 66 | - ./rules/:/opt/filter/rules/ 67 | - ./config/plugin.yml:/opt/filter/config/plugin.yml 68 | networks: 69 | - passets 70 | restart: unless-stopped 71 | depends_on: 72 | - elasticsearch 73 | 74 | kibana: 75 | image: kibana:7.5.1 76 | container_name: passets-kibana 77 | environment: 78 | - TZ=CST-8 79 | - ELASTICSEARCH_HOSTS=http://passets-elasticsearch:9200 80 | - I18N_LOCALE=zh-CN 81 | volumes: 82 | - ./data/logs:/usr/share/kibana/logs 83 | - ./data/kibana:/usr/share/kibana/data 84 | ports: 85 | - "5601:5601" 86 | networks: 87 | - passets 88 | restart: unless-stopped 89 | 90 | sensor: 91 | image: dsolab/passets-sensor:1.0.0 92 | container_name: passets-sensor 93 | environment: 94 | - TZ=CST-8 95 | - port=5044 96 | - tag=sensor 97 | - interface=ens192 98 | - ip=172.20.20.100 99 | - debug=off 100 | - cache=1024 101 | - timeout=3600 102 | restart: unless-stopped 103 | privileged: true 104 | depends_on: 105 | - logstash 106 | network_mode: host 107 | 108 | networks: 109 | passets: 110 | driver: bridge 111 | ipam: 112 | driver: default 113 | config: 114 | - subnet: 172.20.20.0/24 115 | -------------------------------------------------------------------------------- /docs/BUILD.md: -------------------------------------------------------------------------------- 1 | # 镜像构建说明 2 | 3 | ## 如何构建、发布跨平台 Docker 镜像 4 | 5 | 为了保持用户在不同平台使用 Passets 的操纵一致性，需要将不同平台的容器镜像获取方式统一，减少使用者由于选择平台不当导致的容器启动错误问题。 6 | 7 | 操作步骤： 8 | 9 | **第一步** 10 | 11 | 从 GitHUB 上为各个模块添加一个Release版本，版本号和tag均采用 “`数字.数字.数字`” 格式，添加完成后，DockerHUB 会自动开始构建对应版本的镜像。 12 | 13 | **第二步** 14 | 15 | 等待第一步各模块的镜像构建完成后，在其它平台下构建对应的镜像文件，镜像的tag统一在对应版本号后面加上后缀 “`_<架构名>`”，然后提交到 DockerHUB 。例如：`dsolab/passets-api:1.0.1_armv7`。 16 | ``` 17 | # 构建 18 | docker build -t dsolab/passets-:_ . 19 | # 提交到 Docker HUB （很重要） 20 | docker push dsolab/passets-:_ 21 | ``` 22 | 23 | **第三步** 24 | 25 | 借助 Docker 的 manifest 列表功能来实现多平台镜像融合机制。 26 | 27 | 在操作合并镜像的主机上启用 Docker 的 manifest 功能（实验性功能）： 28 | ``` 29 | vi ~/.docker/config.json 30 | { 31 | ... ... 32 | "experimental": "enabled" 33 | } 34 | ``` 35 | 36 | 删除缓存的 manifest 文件： 37 | ``` 38 | rm -rf ~/.docker/manifests/* 39 | # 如果本地镜像列表中包含要融合的镜像，可能还需要执行下面的操作 40 | docker manifest inspect dsolab/passets-:_ 41 | ``` 42 | 43 | 创建多版本融合 manifests： 44 | ``` 45 | docker manifest create dsolab/passets-: dsolab/passets-: dsolab/passets-:_ -a 46 | ``` 47 | 48 | 创建成功后核对 manifest 中对应平台镜像的 SHA256 值与Dockerhub上是否正确： 49 | ``` 50 | docker manifest inspect dsolab/passets-: 51 | ``` 52 | 53 | 确认无误后，提交 manifests: 54 | ``` 55 | docker manifest push dsolab/passets-: -p 56 | ``` 57 | 58 | **第四步** 59 | 60 | 从 DockerHUB 上删除孤立的镜像，仅保留融合后的镜像。 61 | ![](images/delete_isolated_image.png) -------------------------------------------------------------------------------- /docs/ELASTICSEARCH.md: -------------------------------------------------------------------------------- 1 | # Elasticsearch 使用说明 2 | 3 | Elasticsearch 7.4.1安装，详细说明，可以参见[官方支持文档](https://www.elastic.co/guide/en/elasticsearch/reference/7.4/docker.html)。 4 | 5 | ## 获取镜像 6 | 7 | 通过下面的命令从 Docker HUB 上获取 Elasticsearch 镜像： 8 | 9 | ``` 10 | docker pull elasticsearch:7.4.1 11 | ``` 12 | 13 | ## 创建数据、日志目录 14 | 15 | ``` 16 | mkdir data/elasticsearch -p -m 777 17 | mkdir data/logs -p -m 777 18 | ``` 19 | 20 | **注意**: 目录权限必须为777 21 | 22 | ## 容器启动 23 | 24 | ### 单节点 ES 容器启动 25 | 26 | ``` 27 | docker run -p 9200:9200 -e "TZ=Asia/Shanghai" -e "discovery.type=single-node" -v "$(pwd)/data/logs:/usr/share/elasticsearch/logs" -v "$(pwd)/data/elasticsearch:/usr/share/elasticsearch/data" -d elasticsearch:7.4.1 28 | ``` 29 | 30 | ### 采用 Docker Compose 启动单节点 ES 容器 31 | 32 | ``` 33 | version: "3" 34 | 35 | services: 36 | elasticsearch: 37 | image: elasticsearch:7.4.1 38 | container_name: passets-elasticsearch 39 | environment: 40 | - TZ=Asia/Shanghai 41 | - discovery.type=single-node 42 | ports: 43 | - "9200:9200" 44 | volumes: 45 | - ./data/logs:/usr/share/elasticsearch/logs 46 | - ./data/elasticsearch:/usr/share/elasticsearch/data 47 | restart: unless-stopped 48 | ``` 49 | 50 | 启动容器： 51 | 52 | ``` 53 | docker-compose up 54 | ``` 55 | 56 | ### 采用 Docker Compose 启动 ES 容器集群 57 | 58 | 多个容器节点需要启动时推荐使用 `Docker Compose` 来进行。 59 | 60 | 首先，创建 `elasticsearch.yml` 配置文件： 61 | 62 | ``` 63 | http.cors.enabled: true 64 | http.cors.allow-origin: "*" 65 | ``` 66 | 67 | 然后，创建 `docker-compose.yml` 文件： 68 | 69 | ``` 70 | version: "3" 71 | 72 | services: 73 | elasticsearch: 74 | image: elasticsearch:7.4.1 75 | container_name: passets-elasticsearch 76 | environment: 77 | - TZ=Asia/Shanghai 78 | - cluster.name=docker-cluster #集群名称，两个节点需要保持一致 79 | - bootstrap.memory_lock=true #可根据需要配置 80 | - node.name=node-0 #节点名称，与另外一个节点区分开 81 | - xpack.security.enabled=false 82 | ports: 83 | - "9200:9200" 84 | volumes: 85 | - ./data/logs:/usr/share/elasticsearch/logs 86 | - ./data/elasticsearch:/usr/share/elasticsearch/data 87 | - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml 88 | restart: unless-stopped 89 | 90 | elasticsearch2: 91 | image: elasticsearch:7.4.1 92 | container_name: passets-elasticsearch2 93 | environment: 94 | - TZ=Asia/Shanghai 95 | - cluster.name=docker-cluster #集群名称，两个节点需要保持一致 96 | - bootstrap.memory_lock=true #可根据需要配置 97 | - node.name=node-1 #节点名称，与另外一个节点区分开 98 | - xpack.security.enabled=false 99 | - discovery.zen.ping.unicast.hosts=passets-elasticsearch # 指定主节点的容器名 100 | volumes: 101 | - ./data/logs1:/usr/share/elasticsearch/logs 102 | - ./data/elasticsearch1:/usr/share/elasticsearch/data 103 | - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml 104 | restart: unless-stopped 105 | ``` 106 | 107 | 启动容器： 108 | 109 | ``` 110 | docker-compose up 111 | ``` -------------------------------------------------------------------------------- /docs/ESXI_Traffic_Mirroring.md: -------------------------------------------------------------------------------- 1 | ### ESXI场景流量镜像配置 2 | 3 | **第一步**：创建虚拟交换机网络，并分配独立物理网口，该网口用于接收流量镜像数据使用。 4 | 5 | ![exsi_vSwitch](images/exsi_vSwitch.png) 6 | 7 | **第二步**：虚拟交换机属性配置，开启”混杂模式“。 8 | 9 | ![exsi_vSwitch_config](images/exsi_vSwitch_config.png) -------------------------------------------------------------------------------- /docs/KIBANA.md: -------------------------------------------------------------------------------- 1 | # Kibana 使用说明 2 | 3 | Passets 被动资产发现框架内置了 `Kibana`，可以方便用户进行数据的可视化展示。 4 | 5 | 为了安全考虑，可以将 Kibana 设置为仅对 `127.0.0.1` 开放，用户可以通过修改 `docker-compose.yml` 文件中的端口映射配置： 6 | 7 | ``` 8 | services: 9 | ... 10 | kibana: 11 | ... 12 | ports: 13 | - "127.0.0.1:5601:5601" 14 | ``` 15 | 16 | 修改配置后需要执行下面重新部署操作，使其生效： 17 | 18 | ``` 19 | docker-compose down 20 | docker-compose up -d 21 | ``` 22 | 23 | 初始化 Kibana 后，需要进行必要的配置，为便于使用可以通过导入配置文件完成以下步骤，已包含项目预定义的一套统计图表。 24 | 25 | ``` 26 | 1、访问 Discover 模块，并自定义索引表达式为 passets 27 | 2、根据数据分析需求，创建Dashboard统计图表 28 | ``` 29 | 30 | 通过 “`Management/Saved Objects`” 页面的 “`Import`” 功能导入。 31 | 32 | [下载默认配置文件](../config/kibana.ndjson) 33 | 34 | 也可以使用下面的请求通过程序导入 Kibana 配置，实现自动化配置： 35 | 36 | ``` 37 | POST /api/saved_objects/_import?overwrite=true HTTP/1.1 38 | Host: 127.0.0.1:5601 39 | Connection: keep-alive 40 | Content-Length: 36098 41 | Origin: http://127.0.0.1:5601 42 | kbn-version: 7.4.1 43 | User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36 44 | Content-Type: multipart/form-data; boundary=----WebKitFormBoundarylDGWyMO66tGqmSoA 45 | Accept: */* 46 | Referer: http://127.0.0.1:5601/app/kibana 47 | Accept-Encoding: gzip, deflate 48 | Accept-Language: zh-CN,zh;q=0.9 49 | 50 | ------WebKitFormBoundarylDGWyMO66tGqmSoA 51 | Content-Disposition: form-data; name="file"; filename="kibana.ndjson" 52 | Content-Type: application/octet-stream 53 | 54 | <导出的配置文件内容> 55 | ------WebKitFormBoundarylDGWyMO66tGqmSoA-- 56 | ``` 57 | 58 | 导入成功后，重新访问或刷新页面，即可使用`Discover` 和 `Dashboard`等模块。 59 | 60 | ``` 61 | # 数据检索功能页面 62 | http://x.x.x.x:5601/app/kibana#/discover 63 | 64 | # 数据统计仪表板页面(近3天数据) 65 | http://10.87.222.222:5601/app/kibana#/dashboard/130f0440-102f-11ea-a7f8-65c9feeb13c9?embed=true&_g=(refreshInterval:(pause:!t,value:10),time:(from:now-3d,to:now))&_a=(description:'',filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!t,useMargins:!t),viewMode:view) 66 | ``` 67 | 68 | -------------------------------------------------------------------------------- /docs/PF_RING.md: -------------------------------------------------------------------------------- 1 | ### PF_RING安装 2 | 3 | pf_ring需要在docker所在的宿主机上安装，详细说明，可以参见官方支持文档。 4 | 5 | ``` 6 | http://packages.ntop.org/ 7 | ``` 8 | 9 | ### 安装参考 - CentOS 7.3 10 | 11 | **第一步**：安装pf_ring 12 | 13 | **在线环境安装pf_ring** 14 | 15 | ``` 16 | yum -y install wget net-tools 17 | cd /etc/yum.repos.d/ 18 | wget http://packages.ntop.org/centos/ntop.repo -O ntop.repo 19 | rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm 20 | yum clean all 21 | yum update 22 | yum -y install pfring-dkms 23 | ``` 24 | 25 | **离线环境安装pf_ring** 26 | 27 | pf_ring离线yum源制作： 28 | 29 | ``` 30 | mkdir -p /opt/yum-repo 31 | 32 | yum install --downloadonly --downloaddir=/opt/yum-repo net-tools wget createrepo 33 | yum -y install wget net-tools createrepo 34 | cd /etc/yum.repos.d/ 35 | wget http://packages.ntop.org/centos/ntop.repo -O ntop.repo 36 | rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm 37 | yum clean all 38 | 39 | yum update --downloadonly --downloaddir=/opt/yum-repo 40 | yum install --downloadonly --downloaddir=/opt/yum-repo pfring-dkms 41 | 42 | createrepo -pdo /opt/yum-repo /opt/yum-repo 43 | createrepo --update /opt/yum-repo 44 | 45 | tar zcvf /opt/yum-repo.tar.gz /opt/yum-repo/ 46 | ``` 47 | 48 | pf_ring离线yum源安装： 49 | 50 | ``` 51 | vi /etc/yum.repos.d/localyum.repo 52 | 53 | [localyum] 54 | name=localyum 55 | baseurl=file:///opt/yum-repo 56 | enable=1 57 | gpgcheck=0 58 | ``` 59 | 60 | ``` 61 | tar zxvf yum-repo.tar.gz -C /opt/ 62 | 63 | mkdir /etc/yum.repos.d/bak 64 | mv /etc/yum.repos.d/CentOS-* /etc/yum.repos.d/bak/ 65 | yum clean all 66 | yum makecache 67 | yum update 68 | rm -rf /etc/yum.repos.d/CentOS-* 69 | ``` 70 | 71 | ``` 72 | yum install net-tools wget createrepo pfring-dkms 73 | ``` 74 | 75 | **第二步**：内核升级后重启使配置生效（不同版本系统升级后的内核版本不一样，根据实际情况选择） 76 | 77 | 方法一：重启，手工选择。 78 | 79 | 宿主机重启，grub引导时选择升级后的内核启动： 80 | 81 | ![Kernel_Upgrade_Reboot](images/Kernel_Upgrade_Reboot.png) 82 | 83 | 方法二：修改默认grub配置，重启。 84 | 85 | ```bash 86 | # 查看所有的内核版本 87 | cat /boot/grub2/grub.cfg | grep "menuentry" 88 | 89 | # 修改默认启动内核 90 | grub2-set-default 'CentOS Linux (3.10.0-1127.18.2.el7.x86_64) 7 (Core)' 91 | 92 | # 检查配置是否生效 93 | grub2-editenv list 94 | ``` 95 | 96 | **第三步**：pf_ring启动、功能测试: 97 | 98 | ``` 99 | systemctl start pf_ring 100 | systemctl enable pf_ring 101 | 102 | pfcount -i <流量镜像网卡> 103 | ``` 104 | 105 | #### 网卡驱动兼容 106 | 107 | pf_ring网卡兼容性官方介绍： 108 | 109 | ``` 110 | https://www.ntop.org/products/packet-capture/pf_ring/ 111 | ``` 112 | 113 | intel千兆网卡支持清单： 114 | 115 | ``` 116 | https://www.intel.com/content/www/us/en/support/articles/000005480/network-and-i-o/ethernet-products.html 117 | ``` 118 | 119 | intel万兆网卡支持清单： 120 | 121 | ``` 122 | https://downloadcenter.intel.com/download/14687/Intel-Network-Adapter-Driver-for-PCIe-Intel-10-Gigabit-Ethernet-Network-Connections-Under-Linux-?wapkw=ixgbe 123 | ``` 124 | 125 | 126 | 127 | 128 | 129 | -------------------------------------------------------------------------------- /docs/images/Kernel_Upgrade_Reboot.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DSO-Lab/passets/5508d215e5b1d357d971a552f539b112034c2bf1/docs/images/Kernel_Upgrade_Reboot.png -------------------------------------------------------------------------------- /docs/images/delete_isolated_image.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DSO-Lab/passets/5508d215e5b1d357d971a552f539b112034c2bf1/docs/images/delete_isolated_image.png -------------------------------------------------------------------------------- /docs/images/exsi_vSwitch.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DSO-Lab/passets/5508d215e5b1d357d971a552f539b112034c2bf1/docs/images/exsi_vSwitch.png -------------------------------------------------------------------------------- /docs/images/exsi_vSwitch_config.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DSO-Lab/passets/5508d215e5b1d357d971a552f539b112034c2bf1/docs/images/exsi_vSwitch_config.png -------------------------------------------------------------------------------- /docs/images/favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DSO-Lab/passets/5508d215e5b1d357d971a552f539b112034c2bf1/docs/images/favicon.ico -------------------------------------------------------------------------------- /docs/images/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DSO-Lab/passets/5508d215e5b1d357d971a552f539b112034c2bf1/docs/images/logo.png -------------------------------------------------------------------------------- /docs/images/logo_white.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/DSO-Lab/passets/5508d215e5b1d357d971a552f539b112034c2bf1/docs/images/logo_white.png -------------------------------------------------------------------------------- /rules/README.md: -------------------------------------------------------------------------------- 1 | # 简介 2 | 3 | 此文件夹用于放置 Wappalyzer、Nmap 指纹规则文件。 4 | 5 | # 规则库获取 6 | 7 | 从下面的位置可以获取对应的规则库文件： 8 | 9 | ## apps.json 10 | 11 | **Wappalyzer 指纹库** 12 | 13 | https://github.com/AliasIO/Wappalyzer/raw/master/src/apps.json 14 | 15 | ## nmap-service-probes 16 | 17 | **NMAP 指纹库** 18 | 19 | https://github.com/nmap/nmap/raw/master/nmap-service-probes 20 | 21 | -------------------------------------------------------------------------------- /test/logstash_tcp_send_test.py: -------------------------------------------------------------------------------- 1 | #-*- coding:utf-8 -*- 2 | 3 | import socket 4 | import random 5 | import json 6 | 7 | server_ip = '192.168.199.10' 8 | server_port = 5044 9 | 10 | http_json = {"pro": "HTTP", "tag": "lo", "ip": "2001:DB8:0:23:8:0:0:1881", "port": 80, "method": "GET", "code": "200", "type": "text/html; charset=utf-8", "server": "", "header": "Date: Wed, 18 Nov 2020 10:54:24 GMT\r\nContent-Length: 955\r\nContent-Type: text/html; charset=utf-8", "url": "http://[2001:db8:0:23:8::1881]/", "body": "\n\n\n\n \n Passets -- \u7ba1\u7406\u5e73\u53f0\n \n\n \n \n\n\n\n \n\n\n\n"} 11 | tcp_json = {"pro": "TCP", "tag": "lo", "ip": "220.194.238.58", "port": 47808, "data": "16030300550200005103035fbc76d565873c0a90c9c1c8421c4479a0dd93db0e669493440b67dd3f1a70ce205fbb71e157361cb06810d81fdd2c9fd5cc1f64e054d5a9f4899bd3fd6978233bc028000009ff0100010000170000"} 12 | https_json = {"pro": "TCP", "tag": "lo", "ip": "59.111.239.61", "port": 443, "data": "160303007a020000760303344f5d69a6dcae82902adb80930fc61ad9094c70f8b7c1fbaa47f462df2241b62012bb5f0f6be9c0dd502c1413fa0144d664892ff6c4b9af58d50e2f237a18e136130100002e00330024001d00203157283f3310335bc869052990ec5019691faf6471c1001b9bf1289550aad040002b000203041403030001011703030c31cd164a8e5d11a3507b4582d966fd94e0b3bc586b7100ac02284d39e4acd81a52748d5fb575c0abe0e32ef63b888fafc04c29cc9c6f170b2d7a949f889b83291dd4c7faed27b436c6f2fe77617ee9224187ac2eb1b883d6316a5084231866bbd1e97f46f6dabcfe11744d83c53267e218c8b6d6bdc4e44a5f34e8ddc392ab14ccd7df299b148436620518572d198a9304fb2e672b687fcc13b8a919d041d62aeec60021932db4fbe36e9ca0c14a66b416219b0087e24e1bc9275798328870aeb6d13b6e81c000be00943d7efcf784d776db6fd8e2f12b2ee406d442854c880ce81792bf34650b3e0e588ca401bd7158e8844084590859a13b5426eddb565d29c535111c7013356e75e294e9190d655652566b184ae16b8da689fc07fb34541f0b1cb832ed7829df38db0c18cc2d49fa3b974b9932fcd3d32d8bba36c8833af833cbb41427b8c2fe4e2ec3596703457e353ebe36e938478d30a48c88f99a548fdfb9fcdac31c5f8b3fc213d296b5751a796bba9e293b1b9346db324c3632a12552e96ff3ad36aba49a565b9830f2db87e00b60c27259bc07db51deff299d1a21b685f7b5003d98e5d0c84012847391cd043f1ef68f6f6c63c2018f1fdef702d9207eaa0a314d1f1ccbd2d1f14820f44e4571f282c5605c920db1250f6bd232ef798f66646061b34485d1d26a82046e3e071610adbb9d4dce72161c60cda003146013771d7c931ea0eeb2f1040c4ba3575376fc5c3a63cd2362ba15a08e39bcc935b4f1fec3688bee9cfb3b2b28cbeefa239067d87276afa07f0ad0bc56143d9664b2eecfd2c86fce61ff6ede618cedf1eb3aba654a7243423a1517a96d5b6a8f91f769ac362483a7c0dc1e4207c155ab312c996be7ebc984c241fce8a5582c06296f17afd744745a5a802a08930781feed07542061bda19e0686494cd45a0ff1a36aad428bb57425fb02fdedefa785af9d28aa8ad64cbd3357b160b75ee8bb78beb8a223983c7009d52c25d36e02d8d29e43ddcead42f8a2dc02d275718cf2f3b615d7387d1fb435a8a78c252df71520dc10fadee0531136cc5d13477d2bcac5198824b191821eeada86aae1381c3665037d521e0eb8feb91a37bf6b2831170080dfc996e2bb3a64c9fc15ebfac7e6adb84cac824f8f372f4fac52a60673eeaca8a099bbb59d3822252f0d26a5784079685d2bb595a9c8e53746ea0c09773198230383a594cae4f023e650ab626abdf5b7428e3b0e4c71359c7f2a23df54325b8cbd845edfbc5bd7b21665974f942275fecc2ed638bf53e690fb199080b95542b7770e64f2c19023e76c37c88044affae6685d08e90df3cd9d05fd7ea59071366d7b7708023412cefbbb18de8848c26278a6c4ea9b62e5fc057e41ac933167717f7cb6e5a48f7db88f90fbfae1cfb673e4c6e6b93a095ea75d6be5ed2c6dcba424528c47c1f11cc0fe9c4c53c8cdeb0cbb524c32285961b3a7f9f297e4df3e760df17c09e208218a7959dd74ab4f398a457ca3b229db5489ffce89a0890671fb630191ab9047f78a0823dcb6da46f12435060fd3ae80dfeff4cd25a4bcbb123264e59e63b1ccae5d8cd58da95dd139d91231fff2610a9410513dbcd2002d7bd2c8de2b0b9e8d0cbbe18c4bfabe1e65379a07179846d95a2f751fc0694253d3b744edc91005e27d113c4d4046c3a1a87de427dbad89d754b07d54025cf398aa976d8888e7bdfe26dfe3cf5a35558f7d2cb0a63ce3e426ebe0168a5eaf1ee98c7ea5a6af76875823f7b2f46441f819d4e6a7a0824e7c4fff8a3614952658ad669a219a690c56e62d3dadc6fa0f57e7dfeec2b8c3da6d13378da6cea4086de9e2"} 13 | msg_list = [http_json,tcp_json,https_json] 14 | # TCP Send 15 | class _tcp_msg_send: 16 | def __init__(self,server_ip,server_port): 17 | self.server_ip = server_ip 18 | self.server_port = server_port 19 | self.tcp_client = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 20 | #心跳维护 21 | self.tcp_client.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) 22 | self.tcp_client.connect((self.server_ip, self.server_port)) 23 | 24 | def info(self,msg): 25 | try: 26 | self.tcp_client.send(msg.encode()+b"\n") 27 | return True 28 | except socket.error as e: 29 | if e.errno == 32: 30 | return False 31 | # self.tcp_client.close() 32 | 33 | 34 | send_obj = _tcp_msg_send(server_ip,server_port) 35 | while 1: 36 | msg = random.choice(msg_list) 37 | send_obj.info(json.dumps(msg)) -------------------------------------------------------------------------------- /test/nmap-service-probes-patch.json: -------------------------------------------------------------------------------- 1 | { 2 | "apps": [{ 3 | "probe": "Probe TCP TerminalServerCookie q|\x03\0\0*%\xe0\0\0\0\0\0Cookie: mstshash=nmap\r\n\x01\0\x08\0\x03\0\0\0|", 4 | "src_conf": "match ms-wbt-server m|^\x03\0\0\x13\x0e\xd0\0\0\x124\0\x02\x1f\x08\0\x02\0\0\0| p/Microsoft Terminal Services/ o/Windows/ cpe:/o:microsoft:windows/a", 5 | "dst_conf": "match ms-wbt-server m|^\x03\0\0\x13\x0e\xd0\0\0\x12.\0\x02.\x08\0\x02\0\0\0| p/Microsoft Terminal Services/ o/Windows/ cpe:/o:microsoft:windows/a", 6 | "action": "update", 7 | "data": "030000130ed000001234000209080002000000" 8 | }, 9 | { 10 | "probe": "Probe TCP NULL q||", 11 | "src_conf": "", 12 | "dst_conf": "match smtp m|^220 ([-.+\w]+) Esmtp (.+)| p/smtp/", 13 | "action": "add", 14 | "data": "32323020736d74702e71712e636f6d2045736d7470205151204d61696c205365727665720d0a" 15 | }, 16 | { 17 | "probe": "Probe TCP SMBProgNeg q|\0\0\0\xa4\xff\x53\x4d\x42\x72\0\0\0\0\x08\x01\x40\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x40\x06\0\0\x01\0\0\x81\0\x02PC NETWORK PROGRAM 1.0\0\x02MICROSOFT NETWORKS 1.03\0\x02MICROSOFT NETWORKS 3.0\0\x02LANMAN1.0\0\x02LM1.2X002\0\x02Samba\0\x02NT LANMAN 1.0\0\x02NT LM 0.12\0|", 18 | "src_conf": "match microsoft-ds m|^\0\0\0\x61\xffSMBr\0\0\0\0\x88\x03\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0@\x06\0\0\x01\0\x11\x06\0\x02\x0a\0\x01\0\x04\x11\0\0\xff\xff\0\0....\0\x03\0\0..........\x08\x1c\0........\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0$|s p/Xerox printer microsoft-ds/ d/printer/", 19 | "dst_conf": "match microsoft-ds m|^\0\0\0\xf8\xfe\x53\x4d\x42\x40\0\0\0\0\0\0\0\0\0(.*?)| p/SMB/", 20 | "action": "add", 21 | "data": "000000f8fe534d4240000000000000000000010001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000041000300ff020000d12df313baa703459832f1f62049c80a07000000000080000000800000008000d9cadf2f8f3fd6015f1b3a076e30d6018000780000000000607606062b0601050502a06c306aa03c303a060a2b06010401823702021e06092a864882f71201020206092a864886f712010202060a2a864886f71201020203060a2b06010401823702020aa32a3028a0261b246e6f745f646566696e65645f696e5f5246433431373840706c656173655f69676e6f7265" 22 | }, 23 | { 24 | "probe": "Probe TCP NULL q||", 25 | "src_conf": "match ssh m|^SSH-([\d.]+)-OpenSSH_([\w._-]+) \.\n| p/OpenSSH/ v/$2/ i/protocol $1/ cpe:/a:openbsd:openssh:$2/", 26 | "dst_conf": "match ssh m|^SSH-([\d.]+)-OpenSSH_([\w._-]+)| p/OpenSSH/ v/$2/ i/protocol $1/ cpe:/a:openbsd:openssh:$2/", 27 | "action": "update", 28 | "data": "5353482d322e302d4f70656e5353485f372e340d0a" 29 | } 30 | ] 31 | } -------------------------------------------------------------------------------- /test/nmap-service-probes-patch.py: -------------------------------------------------------------------------------- 1 | # coding:utf-8 2 | 3 | import subprocess 4 | import sys 5 | import time 6 | import getopt 7 | import os 8 | import json 9 | 10 | split_str = "##############################NEXT PROBE##############################" 11 | 12 | def Usage(): 13 | print(''' 14 | ################################################################### 15 | # nmap-service-probes-patch # 16 | ################################################################### 17 | ------------------------------------------------------------------- 18 | Usage: 19 | python %s [options] ... 20 | 21 | -n nmap-service-probes file 22 | -p nmap-service-probes-patch file 23 | ------------------------------------------------------------------- 24 | '''%os.path.basename(__file__)) 25 | sys.exit() 26 | 27 | 28 | def nmap_patch(nmap_probes_list,patch_json): 29 | patch_nmap_probe_obj = open(nmap_file+"_patch_"+time.strftime("%Y%m%d%H%M", time.localtime()),"a",encoding="utf-8") 30 | 31 | for patch in patch_json["apps"]: 32 | probe = patch["probe"] 33 | src_conf = patch["src_conf"] 34 | dst_conf = patch["dst_conf"] 35 | action = patch["action"] 36 | 37 | for n in range(len(nmap_probes_list)): 38 | probe_data = nmap_probes_list[n] 39 | if probe in probe_data: 40 | if action == "update": 41 | if src_conf in probe_data: 42 | probe_data = probe_data.replace(src_conf,dst_conf) 43 | else: 44 | action = "add" 45 | if action == "add": 46 | probe_data += '\n'+dst_conf+'\n' 47 | nmap_probes_list[n] = probe_data 48 | 49 | for n in range(len(nmap_probes_list)): 50 | if n == 0: 51 | probe_data = nmap_probes_list[n] 52 | else: 53 | if probe_data[-1] != "\n": 54 | probe_data = "\n"+split_str+nmap_probes_list[n] 55 | else: 56 | probe_data = split_str+nmap_probes_list[n] 57 | patch_nmap_probe_obj.write(probe_data) 58 | patch_nmap_probe_obj.flush() 59 | 60 | def main(): 61 | 62 | with open(nmap_file,"r",encoding="utf-8") as n_obj: 63 | nmap_probes_list = n_obj.read().split(split_str) 64 | 65 | with open(patch_file,"r",encoding="utf-8") as p_obj: 66 | patch_json = json.loads(p_obj.read().replace('\\','\\\\')) 67 | 68 | nmap_patch(nmap_probes_list,patch_json) 69 | 70 | if __name__ == '__main__': 71 | 72 | nmap_file = 'nmap-service-probes' 73 | patch_file = 'nmap-service-probes-patch.json' 74 | 75 | try: 76 | opts,args = getopt.getopt(sys.argv[1:],'n: p:') 77 | except: 78 | Usage() 79 | 80 | for o, a in opts: 81 | if o == "-n": 82 | nmap_file = str(a) 83 | if o == "-p": 84 | patch_file = str(a) 85 | 86 | if nmap_file and patch_file: 87 | main() 88 | else: 89 | Usage() --------------------------------------------------------------------------------