├── .editorconfig ├── .github └── workflows │ └── ci.yaml ├── .gitignore ├── CHANGELOG.md ├── LICENSE ├── Makefile ├── README.md ├── README_ZH.md ├── VERSION ├── cache ├── cache.go ├── cache_lru.go └── cache_test.go ├── docs ├── loggz.png ├── logs.png ├── t1.png ├── t2.png ├── t3.png ├── t4.png ├── t5.png ├── t6.png ├── t7.png └── tatb.png ├── go.mod ├── go.sum ├── handler ├── benchmark_test.go ├── example_test.go ├── handler.go ├── handler_test.go ├── largest_average_time_uris.go ├── largest_percent_time_uris.go ├── most_frequent_status.go ├── most_visited_fields.go ├── most_visited_locations.go └── pv_uv.go ├── ioutil ├── example_test.go ├── files.go ├── files_test.go └── log.go ├── main.go ├── parser ├── benchmark_test.go ├── log_info.go ├── parser.go └── parser_test.go └── testdata ├── GeoLite2-City-Test.mmdb ├── access.json.log ├── access.json.log.1.gz └── access.log /.editorconfig: -------------------------------------------------------------------------------- 1 | root = true 2 | 3 | [*] 4 | charset = utf-8 5 | end_of_line = lf 6 | insert_final_newline = true 7 | indent_style = space 8 | indent_size = 4 9 | max_line_length = 100 10 | trim_trailing_whitespace = true 11 | 12 | [*.java] 13 | indent_size = 4 14 | 15 | [*.go] 16 | indent_style = tab 17 | indent_size = 1 18 | tab_width = 4 19 | 20 | [{*.js,*.json,*.css}] 21 | indent_size = 2 22 | 23 | [*.sh] 24 | indent_size = 2 25 | 26 | [{*.cql,*.ddl,*.sql}] 27 | indent_size = 2 28 | 29 | [*.xml] 30 | indent_size = 4 31 | 32 | [{*.yaml,*.yml}] 33 | indent_size = 2 34 | 35 | [*.init] 36 | indent_size = 4 37 | 38 | [*.http] 39 | indent_size = 2 40 | 41 | [*.md] 42 | indent_size = 2 43 | -------------------------------------------------------------------------------- /.github/workflows/ci.yaml: -------------------------------------------------------------------------------- 1 | name: ci 2 | 3 | on: 4 | push: 5 | pull_request: 6 | branches: 7 | - master 8 | 9 | jobs: 10 | ci: 11 | name: Continuous Integration 12 | runs-on: ubuntu-20.04 13 | steps: 14 | - name: Checkout project 15 | uses: actions/checkout@v4 16 | - name: Setup Go 17 | uses: actions/setup-go@v4 18 | with: 19 | go-version: '1.21' 20 | - name: Go test 21 | run: go test ./... -race -coverprofile=coverage.txt -covermode=atomic -v 22 | - name: Upload coverage to Codecov 23 | uses: codecov/codecov-action@v3 24 | with: 25 | token: ${{ secrets.CODECOV_TOKEN }} 26 | files: ./coverage.txt 27 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | 3 | bin/ 4 | coverage.txt 5 | 6 | ### IntelliJ IDEA ### 7 | .idea 8 | *.iws 9 | *.iml 10 | *.ipr 11 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | ## v1.0.1 4 | 5 | ### Features 6 | 7 | - Support multiple log format configurations 8 | - combined (Nginx default configuration) 9 | - JSON 10 | 11 | ### Change 12 | 13 | - Rename project name to nginx-log-analyzer 14 | 15 | ### Full Changelog 16 | 17 | [v1.0.0...v1.0.1](https://github.com/fantasticmao/nginx-log-analyzer/compare/v1.0.0...v1.0.1) 18 | 19 | ## v1.0.0 20 | 21 | ### Features 22 | 23 | - Filter logs based on the request time 24 | - Analyze multiple files at the same time 25 | - Analyze .gz compressed files 26 | - Support a variety 27 | of [statistical indicators](https://github.com/fantasticmao/nginx-log-analyzer#specify-the-analysis-type--t) -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Attribution-ShareAlike 4.0 International 2 | 3 | ======================================================================= 4 | 5 | Creative Commons Corporation ("Creative Commons") is not a law firm and 6 | does not provide legal services or legal advice. Distribution of 7 | Creative Commons public licenses does not create a lawyer-client or 8 | other relationship. Creative Commons makes its licenses and related 9 | information available on an "as-is" basis. Creative Commons gives no 10 | warranties regarding its licenses, any material licensed under their 11 | terms and conditions, or any related information. Creative Commons 12 | disclaims all liability for damages resulting from their use to the 13 | fullest extent possible. 14 | 15 | Using Creative Commons Public Licenses 16 | 17 | Creative Commons public licenses provide a standard set of terms and 18 | conditions that creators and other rights holders may use to share 19 | original works of authorship and other material subject to copyright 20 | and certain other rights specified in the public license below. The 21 | following considerations are for informational purposes only, are not 22 | exhaustive, and do not form part of our licenses. 23 | 24 | Considerations for licensors: Our public licenses are 25 | intended for use by those authorized to give the public 26 | permission to use material in ways otherwise restricted by 27 | copyright and certain other rights. Our licenses are 28 | irrevocable. Licensors should read and understand the terms 29 | and conditions of the license they choose before applying it. 30 | Licensors should also secure all rights necessary before 31 | applying our licenses so that the public can reuse the 32 | material as expected. Licensors should clearly mark any 33 | material not subject to the license. This includes other CC- 34 | licensed material, or material used under an exception or 35 | limitation to copyright. More considerations for licensors: 36 | wiki.creativecommons.org/Considerations_for_licensors 37 | 38 | Considerations for the public: By using one of our public 39 | licenses, a licensor grants the public permission to use the 40 | licensed material under specified terms and conditions. If 41 | the licensor's permission is not necessary for any reason--for 42 | example, because of any applicable exception or limitation to 43 | copyright--then that use is not regulated by the license. Our 44 | licenses grant only permissions under copyright and certain 45 | other rights that a licensor has authority to grant. Use of 46 | the licensed material may still be restricted for other 47 | reasons, including because others have copyright or other 48 | rights in the material. A licensor may make special requests, 49 | such as asking that all changes be marked or described. 50 | Although not required by our licenses, you are encouraged to 51 | respect those requests where reasonable. More_considerations 52 | for the public: 53 | wiki.creativecommons.org/Considerations_for_licensees 54 | 55 | ======================================================================= 56 | 57 | Creative Commons Attribution-ShareAlike 4.0 International Public 58 | License 59 | 60 | By exercising the Licensed Rights (defined below), You accept and agree 61 | to be bound by the terms and conditions of this Creative Commons 62 | Attribution-ShareAlike 4.0 International Public License ("Public 63 | License"). To the extent this Public License may be interpreted as a 64 | contract, You are granted the Licensed Rights in consideration of Your 65 | acceptance of these terms and conditions, and the Licensor grants You 66 | such rights in consideration of benefits the Licensor receives from 67 | making the Licensed Material available under these terms and 68 | conditions. 69 | 70 | 71 | Section 1 -- Definitions. 72 | 73 | a. Adapted Material means material subject to Copyright and Similar 74 | Rights that is derived from or based upon the Licensed Material 75 | and in which the Licensed Material is translated, altered, 76 | arranged, transformed, or otherwise modified in a manner requiring 77 | permission under the Copyright and Similar Rights held by the 78 | Licensor. For purposes of this Public License, where the Licensed 79 | Material is a musical work, performance, or sound recording, 80 | Adapted Material is always produced where the Licensed Material is 81 | synched in timed relation with a moving image. 82 | 83 | b. Adapter's License means the license You apply to Your Copyright 84 | and Similar Rights in Your contributions to Adapted Material in 85 | accordance with the terms and conditions of this Public License. 86 | 87 | c. BY-SA Compatible License means a license listed at 88 | creativecommons.org/compatiblelicenses, approved by Creative 89 | Commons as essentially the equivalent of this Public License. 90 | 91 | d. Copyright and Similar Rights means copyright and/or similar rights 92 | closely related to copyright including, without limitation, 93 | performance, broadcast, sound recording, and Sui Generis Database 94 | Rights, without regard to how the rights are labeled or 95 | categorized. For purposes of this Public License, the rights 96 | specified in Section 2(b)(1)-(2) are not Copyright and Similar 97 | Rights. 98 | 99 | e. Effective Technological Measures means those measures that, in the 100 | absence of proper authority, may not be circumvented under laws 101 | fulfilling obligations under Article 11 of the WIPO Copyright 102 | Treaty adopted on December 20, 1996, and/or similar international 103 | agreements. 104 | 105 | f. Exceptions and Limitations means fair use, fair dealing, and/or 106 | any other exception or limitation to Copyright and Similar Rights 107 | that applies to Your use of the Licensed Material. 108 | 109 | g. License Elements means the license attributes listed in the name 110 | of a Creative Commons Public License. The License Elements of this 111 | Public License are Attribution and ShareAlike. 112 | 113 | h. Licensed Material means the artistic or literary work, database, 114 | or other material to which the Licensor applied this Public 115 | License. 116 | 117 | i. Licensed Rights means the rights granted to You subject to the 118 | terms and conditions of this Public License, which are limited to 119 | all Copyright and Similar Rights that apply to Your use of the 120 | Licensed Material and that the Licensor has authority to license. 121 | 122 | j. Licensor means the individual(s) or entity(ies) granting rights 123 | under this Public License. 124 | 125 | k. Share means to provide material to the public by any means or 126 | process that requires permission under the Licensed Rights, such 127 | as reproduction, public display, public performance, distribution, 128 | dissemination, communication, or importation, and to make material 129 | available to the public including in ways that members of the 130 | public may access the material from a place and at a time 131 | individually chosen by them. 132 | 133 | l. Sui Generis Database Rights means rights other than copyright 134 | resulting from Directive 96/9/EC of the European Parliament and of 135 | the Council of 11 March 1996 on the legal protection of databases, 136 | as amended and/or succeeded, as well as other essentially 137 | equivalent rights anywhere in the world. 138 | 139 | m. You means the individual or entity exercising the Licensed Rights 140 | under this Public License. Your has a corresponding meaning. 141 | 142 | 143 | Section 2 -- Scope. 144 | 145 | a. License grant. 146 | 147 | 1. Subject to the terms and conditions of this Public License, 148 | the Licensor hereby grants You a worldwide, royalty-free, 149 | non-sublicensable, non-exclusive, irrevocable license to 150 | exercise the Licensed Rights in the Licensed Material to: 151 | 152 | a. reproduce and Share the Licensed Material, in whole or 153 | in part; and 154 | 155 | b. produce, reproduce, and Share Adapted Material. 156 | 157 | 2. Exceptions and Limitations. For the avoidance of doubt, where 158 | Exceptions and Limitations apply to Your use, this Public 159 | License does not apply, and You do not need to comply with 160 | its terms and conditions. 161 | 162 | 3. Term. The term of this Public License is specified in Section 163 | 6(a). 164 | 165 | 4. Media and formats; technical modifications allowed. The 166 | Licensor authorizes You to exercise the Licensed Rights in 167 | all media and formats whether now known or hereafter created, 168 | and to make technical modifications necessary to do so. The 169 | Licensor waives and/or agrees not to assert any right or 170 | authority to forbid You from making technical modifications 171 | necessary to exercise the Licensed Rights, including 172 | technical modifications necessary to circumvent Effective 173 | Technological Measures. For purposes of this Public License, 174 | simply making modifications authorized by this Section 2(a) 175 | (4) never produces Adapted Material. 176 | 177 | 5. Downstream recipients. 178 | 179 | a. Offer from the Licensor -- Licensed Material. Every 180 | recipient of the Licensed Material automatically 181 | receives an offer from the Licensor to exercise the 182 | Licensed Rights under the terms and conditions of this 183 | Public License. 184 | 185 | b. Additional offer from the Licensor -- Adapted Material. 186 | Every recipient of Adapted Material from You 187 | automatically receives an offer from the Licensor to 188 | exercise the Licensed Rights in the Adapted Material 189 | under the conditions of the Adapter's License You apply. 190 | 191 | c. No downstream restrictions. You may not offer or impose 192 | any additional or different terms or conditions on, or 193 | apply any Effective Technological Measures to, the 194 | Licensed Material if doing so restricts exercise of the 195 | Licensed Rights by any recipient of the Licensed 196 | Material. 197 | 198 | 6. No endorsement. Nothing in this Public License constitutes or 199 | may be construed as permission to assert or imply that You 200 | are, or that Your use of the Licensed Material is, connected 201 | with, or sponsored, endorsed, or granted official status by, 202 | the Licensor or others designated to receive attribution as 203 | provided in Section 3(a)(1)(A)(i). 204 | 205 | b. Other rights. 206 | 207 | 1. Moral rights, such as the right of integrity, are not 208 | licensed under this Public License, nor are publicity, 209 | privacy, and/or other similar personality rights; however, to 210 | the extent possible, the Licensor waives and/or agrees not to 211 | assert any such rights held by the Licensor to the limited 212 | extent necessary to allow You to exercise the Licensed 213 | Rights, but not otherwise. 214 | 215 | 2. Patent and trademark rights are not licensed under this 216 | Public License. 217 | 218 | 3. To the extent possible, the Licensor waives any right to 219 | collect royalties from You for the exercise of the Licensed 220 | Rights, whether directly or through a collecting society 221 | under any voluntary or waivable statutory or compulsory 222 | licensing scheme. In all other cases the Licensor expressly 223 | reserves any right to collect such royalties. 224 | 225 | 226 | Section 3 -- License Conditions. 227 | 228 | Your exercise of the Licensed Rights is expressly made subject to the 229 | following conditions. 230 | 231 | a. Attribution. 232 | 233 | 1. If You Share the Licensed Material (including in modified 234 | form), You must: 235 | 236 | a. retain the following if it is supplied by the Licensor 237 | with the Licensed Material: 238 | 239 | i. identification of the creator(s) of the Licensed 240 | Material and any others designated to receive 241 | attribution, in any reasonable manner requested by 242 | the Licensor (including by pseudonym if 243 | designated); 244 | 245 | ii. a copyright notice; 246 | 247 | iii. a notice that refers to this Public License; 248 | 249 | iv. a notice that refers to the disclaimer of 250 | warranties; 251 | 252 | v. a URI or hyperlink to the Licensed Material to the 253 | extent reasonably practicable; 254 | 255 | b. indicate if You modified the Licensed Material and 256 | retain an indication of any previous modifications; and 257 | 258 | c. indicate the Licensed Material is licensed under this 259 | Public License, and include the text of, or the URI or 260 | hyperlink to, this Public License. 261 | 262 | 2. You may satisfy the conditions in Section 3(a)(1) in any 263 | reasonable manner based on the medium, means, and context in 264 | which You Share the Licensed Material. For example, it may be 265 | reasonable to satisfy the conditions by providing a URI or 266 | hyperlink to a resource that includes the required 267 | information. 268 | 269 | 3. If requested by the Licensor, You must remove any of the 270 | information required by Section 3(a)(1)(A) to the extent 271 | reasonably practicable. 272 | 273 | b. ShareAlike. 274 | 275 | In addition to the conditions in Section 3(a), if You Share 276 | Adapted Material You produce, the following conditions also apply. 277 | 278 | 1. The Adapter's License You apply must be a Creative Commons 279 | license with the same License Elements, this version or 280 | later, or a BY-SA Compatible License. 281 | 282 | 2. You must include the text of, or the URI or hyperlink to, the 283 | Adapter's License You apply. You may satisfy this condition 284 | in any reasonable manner based on the medium, means, and 285 | context in which You Share Adapted Material. 286 | 287 | 3. You may not offer or impose any additional or different terms 288 | or conditions on, or apply any Effective Technological 289 | Measures to, Adapted Material that restrict exercise of the 290 | rights granted under the Adapter's License You apply. 291 | 292 | 293 | Section 4 -- Sui Generis Database Rights. 294 | 295 | Where the Licensed Rights include Sui Generis Database Rights that 296 | apply to Your use of the Licensed Material: 297 | 298 | a. for the avoidance of doubt, Section 2(a)(1) grants You the right 299 | to extract, reuse, reproduce, and Share all or a substantial 300 | portion of the contents of the database; 301 | 302 | b. if You include all or a substantial portion of the database 303 | contents in a database in which You have Sui Generis Database 304 | Rights, then the database in which You have Sui Generis Database 305 | Rights (but not its individual contents) is Adapted Material, 306 | 307 | including for purposes of Section 3(b); and 308 | c. You must comply with the conditions in Section 3(a) if You Share 309 | all or a substantial portion of the contents of the database. 310 | 311 | For the avoidance of doubt, this Section 4 supplements and does not 312 | replace Your obligations under this Public License where the Licensed 313 | Rights include other Copyright and Similar Rights. 314 | 315 | 316 | Section 5 -- Disclaimer of Warranties and Limitation of Liability. 317 | 318 | a. UNLESS OTHERWISE SEPARATELY UNDERTAKEN BY THE LICENSOR, TO THE 319 | EXTENT POSSIBLE, THE LICENSOR OFFERS THE LICENSED MATERIAL AS-IS 320 | AND AS-AVAILABLE, AND MAKES NO REPRESENTATIONS OR WARRANTIES OF 321 | ANY KIND CONCERNING THE LICENSED MATERIAL, WHETHER EXPRESS, 322 | IMPLIED, STATUTORY, OR OTHER. THIS INCLUDES, WITHOUT LIMITATION, 323 | WARRANTIES OF TITLE, MERCHANTABILITY, FITNESS FOR A PARTICULAR 324 | PURPOSE, NON-INFRINGEMENT, ABSENCE OF LATENT OR OTHER DEFECTS, 325 | ACCURACY, OR THE PRESENCE OR ABSENCE OF ERRORS, WHETHER OR NOT 326 | KNOWN OR DISCOVERABLE. WHERE DISCLAIMERS OF WARRANTIES ARE NOT 327 | ALLOWED IN FULL OR IN PART, THIS DISCLAIMER MAY NOT APPLY TO YOU. 328 | 329 | b. TO THE EXTENT POSSIBLE, IN NO EVENT WILL THE LICENSOR BE LIABLE 330 | TO YOU ON ANY LEGAL THEORY (INCLUDING, WITHOUT LIMITATION, 331 | NEGLIGENCE) OR OTHERWISE FOR ANY DIRECT, SPECIAL, INDIRECT, 332 | INCIDENTAL, CONSEQUENTIAL, PUNITIVE, EXEMPLARY, OR OTHER LOSSES, 333 | COSTS, EXPENSES, OR DAMAGES ARISING OUT OF THIS PUBLIC LICENSE OR 334 | USE OF THE LICENSED MATERIAL, EVEN IF THE LICENSOR HAS BEEN 335 | ADVISED OF THE POSSIBILITY OF SUCH LOSSES, COSTS, EXPENSES, OR 336 | DAMAGES. WHERE A LIMITATION OF LIABILITY IS NOT ALLOWED IN FULL OR 337 | IN PART, THIS LIMITATION MAY NOT APPLY TO YOU. 338 | 339 | c. The disclaimer of warranties and limitation of liability provided 340 | above shall be interpreted in a manner that, to the extent 341 | possible, most closely approximates an absolute disclaimer and 342 | waiver of all liability. 343 | 344 | 345 | Section 6 -- Term and Termination. 346 | 347 | a. This Public License applies for the term of the Copyright and 348 | Similar Rights licensed here. However, if You fail to comply with 349 | this Public License, then Your rights under this Public License 350 | terminate automatically. 351 | 352 | b. Where Your right to use the Licensed Material has terminated under 353 | Section 6(a), it reinstates: 354 | 355 | 1. automatically as of the date the violation is cured, provided 356 | it is cured within 30 days of Your discovery of the 357 | violation; or 358 | 359 | 2. upon express reinstatement by the Licensor. 360 | 361 | For the avoidance of doubt, this Section 6(b) does not affect any 362 | right the Licensor may have to seek remedies for Your violations 363 | of this Public License. 364 | 365 | c. For the avoidance of doubt, the Licensor may also offer the 366 | Licensed Material under separate terms or conditions or stop 367 | distributing the Licensed Material at any time; however, doing so 368 | will not terminate this Public License. 369 | 370 | d. Sections 1, 5, 6, 7, and 8 survive termination of this Public 371 | License. 372 | 373 | 374 | Section 7 -- Other Terms and Conditions. 375 | 376 | a. The Licensor shall not be bound by any additional or different 377 | terms or conditions communicated by You unless expressly agreed. 378 | 379 | b. Any arrangements, understandings, or agreements regarding the 380 | Licensed Material not stated herein are separate from and 381 | independent of the terms and conditions of this Public License. 382 | 383 | 384 | Section 8 -- Interpretation. 385 | 386 | a. For the avoidance of doubt, this Public License does not, and 387 | shall not be interpreted to, reduce, limit, restrict, or impose 388 | conditions on any use of the Licensed Material that could lawfully 389 | be made without permission under this Public License. 390 | 391 | b. To the extent possible, if any provision of this Public License is 392 | deemed unenforceable, it shall be automatically reformed to the 393 | minimum extent necessary to make it enforceable. If the provision 394 | cannot be reformed, it shall be severed from this Public License 395 | without affecting the enforceability of the remaining terms and 396 | conditions. 397 | 398 | c. No term or condition of this Public License will be waived and no 399 | failure to comply consented to unless expressly agreed to by the 400 | Licensor. 401 | 402 | d. Nothing in this Public License constitutes or may be interpreted 403 | as a limitation upon, or waiver of, any privileges and immunities 404 | that apply to the Licensor or You, including from the legal 405 | processes of any jurisdiction or authority. 406 | 407 | 408 | ======================================================================= 409 | 410 | Creative Commons is not a party to its public 411 | licenses. Notwithstanding, Creative Commons may elect to apply one of 412 | its public licenses to material it publishes and in those instances 413 | will be considered the “Licensor.” The text of the Creative Commons 414 | public licenses is dedicated to the public domain under the CC0 Public 415 | Domain Dedication. Except for the limited purpose of indicating that 416 | material is shared under a Creative Commons public license or as 417 | otherwise permitted by the Creative Commons policies published at 418 | creativecommons.org/policies, Creative Commons does not authorize the 419 | use of the trademark "Creative Commons" or any other trademark or logo 420 | of Creative Commons without its prior written consent including, 421 | without limitation, in connection with any unauthorized modifications 422 | to any of its public licenses or any other arrangements, 423 | understandings, or agreements concerning use of licensed material. For 424 | the avoidance of doubt, this paragraph does not form part of the 425 | public licenses. 426 | 427 | Creative Commons may be contacted at creativecommons.org. -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | NAME=nginx-log-analyzer 2 | BIN_DIR=bin 3 | VERSION=$(shell cat VERSION) 4 | BUILD_TIME=$(shell date -u) 5 | COMMIT_HASH=$(shell git rev-parse --short HEAD) 6 | GO_BUILD=CGO_ENABLED=0 go build -trimpath -ldflags \ 7 | '-X "main.Version=$(VERSION)" \ 8 | -X "main.BuildTime=$(BUILD_TIME)" \ 9 | -X "main.CommitHash=$(COMMIT_HASH)" \ 10 | -w -s' 11 | 12 | PLATFORM_LIST=darwin-amd64 darwin-arm64 linux-amd64 linux-armv5 linux-armv6 linux-armv7 linux-armv8 windows-amd64 13 | 14 | default: build 15 | 16 | .PHONY: default 17 | 18 | build: darwin-amd64 linux-amd64 windows-amd64 # Most used 19 | 20 | build-all: $(PLATFORM_LIST) 21 | 22 | .PHONY: build build-all 23 | 24 | darwin-amd64: 25 | GOARCH=amd64 GOOS=darwin $(GO_BUILD) -o $(BIN_DIR)/$(NAME)-$@ 26 | 27 | darwin-arm64: 28 | GOARCH=arm64 GOOS=darwin $(GO_BUILD) -o $(BIN_DIR)/$(NAME)-$@ 29 | 30 | linux-amd64: 31 | GOARCH=amd64 GOOS=linux $(GO_BUILD) -o $(BIN_DIR)/$(NAME)-$@ 32 | 33 | linux-armv5: 34 | GOARCH=arm GOOS=linux GOARM=5 $(GO_BUILD) -o $(BIN_DIR)/$(NAME)-$@ 35 | 36 | linux-armv6: 37 | GOARCH=arm GOOS=linux GOARM=6 $(GO_BUILD) -o $(BIN_DIR)/$(NAME)-$@ 38 | 39 | linux-armv7: 40 | GOARCH=arm GOOS=linux GOARM=7 $(GO_BUILD) -o $(BIN_DIR)/$(NAME)-$@ 41 | 42 | linux-armv8: 43 | GOARCH=arm64 GOOS=linux $(GO_BUILD) -o $(BIN_DIR)/$(NAME)-$@ 44 | 45 | windows-amd64: 46 | GOARCH=amd64 GOOS=windows $(GO_BUILD) -o $(BIN_DIR)/$(NAME)-$@.exe 47 | 48 | test: 49 | go test ./... -race -coverprofile=coverage.txt -covermode=atomic -v 50 | 51 | .PHONY: test 52 | 53 | clean: 54 | rm $(BIN_DIR)/* 55 | 56 | .PHONY: clean 57 | 58 | help: 59 | @echo 'make clean: clean project' 60 | @echo 'make test: compile and test project' 61 | @echo 'make [build]: compile and build project' 62 | @echo 'make build-all: compile and build project for all platform' 63 | 64 | .PHONY: clean 65 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Nginx-Log-Analyzer 2 | 3 | [![Actions Status](https://github.com/fantasticmao/nginx-log-analyzer/workflows/ci/badge.svg)](https://github.com/fantasticmao/nginx-log-analyzer/actions) 4 | [![codecov](https://codecov.io/gh/fantasticmao/nginx-log-analyzer/branch/main/graph/badge.svg)](https://codecov.io/gh/fantasticmao/nginx-log-analyzer) 5 | ![Go Version](https://img.shields.io/github/go-mod/go-version/fantasticmao/nginx-log-analyzer) 6 | [![Go Report Card](https://goreportcard.com/badge/github.com/fantasticmao/nginx-log-analyzer)](https://goreportcard.com/report/github.com/fantasticmao/nginx-log-analyzer) 7 | [![Release](https://img.shields.io/github/v/release/fantasticmao/nginx-log-analyzer)](https://github.com/fantasticmao/nginx-log-analyzer/releases) 8 | [![License](https://img.shields.io/github/license/fantasticmao/nginx-log-analyzer)](https://github.com/fantasticmao/nginx-log-analyzer/blob/main/LICENSE) 9 | 10 | README [English](README.md) | [中文](README_ZH.md) 11 | 12 | ## What is it 13 | 14 | Nginx-Log-Analyzer is a lightweight (simplistic) log analyzer, used to analyze Nginx access logs for myself. 15 | 16 | Nginx-Log-Analyzer is written in Go programming language, needs only a 2 MB executable file to run, currently supported 17 | features are as follows: 18 | 19 | - [x] Filter logs based on the request time 20 | - [x] Support multiple log format configurations 21 | - combined (Nginx default configuration) 22 | - JSON 23 | - [x] Analyze multiple files at the same time 24 | - [x] Analyze .gz compressed files 25 | - [x] Support a variety of [statistical indicators](#specify-the-analysis-type--t) 26 | 27 | ### Advantages compared to [GoAccess](https://goaccess.io/) 28 | 29 | GoAccess is an excellent and powerful real-time web log analyzer, interactive viewer that runs in a terminal in \*nix 30 | systems or through your browser. But as far as I know, GoAccess seems does not support counting URI response time by 31 | percentile, Nginx-Log-Analyzer supports this feature. 32 | 33 | If I knew about GoAccess before developing Nginx-Log-Analyzer, I might choose to use it directly. GoAccess is so 34 | powerful, I love GoAccess. 35 | 36 | ### Advantages compared to [ELK](https://www.elastic.co/cn/what-is/elk-stack) 37 | 38 | Although ELK is powerful, it is troublesome to install and configure, and it also has certain requirements for machine 39 | performance. Nginx-Log-Analyzer is more lightweight and easier to use, suitable for some simple log analysis scenarios. 40 | 41 | ## Quick start 42 | 43 | ### Installation 44 | 45 | Just download the binary executable file for the corresponding platform from the 46 | GitHub [Release](https://github.com/fantasticmao/nginx-log-analyzer/releases) page of Nginx-Log-Analyzer. 47 | 48 | #### GeoIP2 and GeoLite2 49 | 50 | [GeoIP2](https://www.maxmind.com/en/geoip2-city) is a commercial IP geolocation database, need to pay to use 51 | it. [GeoLite2](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data) is a free and low-precision version of 52 | GeoIP2, distribute by [Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/deed.en) 53 | license, download by logging in to the [MaxMind](https://www.maxmind.com/en/accounts/current/geoip/downloads) official 54 | website. 55 | 56 | When using Nginx-Log-Analyzer, if you need to resolve the geographic location of the IP (that is, use the `-t 4` 57 | mode), then you will need to download the GeoIP2 or GeoLite2 City database file, save it to the `City.mmdb` file in the 58 | default configuration directory `${HOME}/.config/nginx-log-analyzer/`. The corresponding shell commands are as follows: 59 | 60 | ```shell 61 | ~$ mkdir -p ${HOME}/.config/nginx-log-analyzer 62 | ~$ tar -xzf GeoLite2-City_20211109.tar.gz 63 | ~$ cp GeoLite2-City_20211109/GeoLite2-City.mmdb ${HOME}/.config/nginx-log-analyzer/City.mmdb 64 | ``` 65 | 66 | #### Configure Nginx 67 | 68 | Nginx-Log-Analyzer parses Nginx access logs in combined format by default, which means that the logs will contain the 69 | following fields: 70 | 71 | - $remote_addr 72 | - $remote_user 73 | - $time_local 74 | - $request 75 | - $status 76 | - $body_bytes_sent 77 | - $http_referer 78 | - $http_user_agent 79 | 80 | When using Nginx-Log-Analyzer, if you need more types of [statistical indicators](#specify-the-analysis-type--t), then 81 | you will need to use the `-lf json` option to specify the log parsing mode to the JSON format, and need to add the 82 | following `log_format` and `access_log` directives in the Nginx configuration: 83 | 84 | ```text 85 | log_format json_log escape=json '{"remote_addr":"$remote_addr",' 86 | '"time_local":"$time_local",' 87 | '"request":"$request",' 88 | '"status":$status,' 89 | '"body_bytes_sent":$body_bytes_sent,' 90 | '"http_user_agent":"$http_user_agent",' 91 | '"request_time":$request_time}'; 92 | access_log /path/to/access.json.log json_log; 93 | ``` 94 | 95 | - The `log_format` directive can only appear in the `http` context; 96 | - The `access_log` directive could appear in the `http`, `server`, `location` context, and should use the `log_format` 97 | declared above; 98 | - You can make multiple `access_log`s at the same time without deleting the original configuration. e.g. 99 | ```text 100 | access_log /path/to/access.log; 101 | access_log /path/to/access.json.log json_log; 102 | ``` 103 | 104 | Related document: http://nginx.org/en/docs/http/ngx_http_log_module.html 105 | 106 | ### Command line options 107 | 108 | #### show version -v 109 | 110 | The `-v` options show Nginx-Log-Analyzer's build version, build time, and Git Commit at build time. 111 | 112 | #### specify the configuration directory -d 113 | 114 | The `-d` option specify the configuration directory that Nginx-Log-Analyzer required at runtime, the default value 115 | is `${HOME}/.config/nginx-log-analyzer/`. 116 | 117 | #### specify the log format -lf 118 | 119 | The `-lf` option specify the log format parsed by Nginx-Log-Analyzer, available values are combined and json, the 120 | default value is combined. 121 | 122 | #### specify the analysis type -t 123 | 124 | The `-t` option specify the type of this analysis, the analysis type and corresponding statistical indicators are as 125 | follows: 126 | 127 | | Supported | Analysis Type `-t` | Statistical Indicators | Required Fields or Libraries | 128 | | --------- | ------------------ | -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | 129 | | ✅ | 0 | PV and UV | $remote_addr | 130 | | ✅ | 1 | Most visited IPs | $remote_addr | 131 | | ✅ | 2 | Most visited URIs | $request | 132 | | ✅ | 3 | Most visited User-Agents | $http_user_agent | 133 | | ✅ | 4 | Most visited user countries and cities | $remote_addr, MaxMind [GeoIP2](https://www.maxmind.com/en/geoip2-city) or [GeoLite2](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data) City Database | 134 | | ✅ | 5 | Most frequent response status | $status, $request | 135 | | ✅ | 6 | Largest average response time URIs | $request, $request_time | 136 | | ✅ | 7 | Largest percentile response time URIs, e.g. p1(min), p50(median), p95, p100(max) | $request, $request_time | 137 | 138 | #### limit the analysis start and end time -ta -tb 139 | 140 | `-ta` and `-tb` options are used to filter logs based on the request time, `ta` is the abbreviation of time after, `tb` 141 | is the abbreviation of time before. 142 | 143 | `-ta` and `-tb` options required the $time_local field in `log_format` directive of Nginx configuration. 144 | 145 | #### limit the output lines number -n -n2 146 | 147 | `-n` and `-n2` options are used to limit the number of output lines of Nginx-Log-Analyzer, `-n2` option only works 148 | in `-t 4` mode. 149 | 150 | #### specify the percentile value -p 151 | 152 | The `-p` option specify the percentile value in the `-t 7` mode, the default value is 95. 153 | 154 | ### Usages 155 | 156 | #### Filter logs based on the request time 157 | 158 | ![image](docs/tatb.png) 159 | 160 | #### Analyze multiple files at the same time 161 | 162 | ![image](docs/logs.png) 163 | 164 | #### Analyze .gz compressed files 165 | 166 | ![image](docs/loggz.png) 167 | 168 | #### Count the most visited IPs 169 | 170 | ![image](docs/t1.png) 171 | 172 | #### Count the most visited URIs 173 | 174 | ![image](docs/t2.png) 175 | 176 | #### Count the most visited User-Agents 177 | 178 | ![image](docs/t3.png) 179 | 180 | #### Count the most visited countries and cities 181 | 182 | ![image](docs/t4.png) 183 | 184 | #### Count the most frequently response status 185 | 186 | ![image](docs/t5.png) 187 | 188 | #### count the largest URI average response times 189 | 190 | ![image](docs/t6.png) 191 | 192 | #### count the largest URI percentile response times 193 | 194 | ![image](docs/t7.png) 195 | 196 | ## FQA 197 | 198 | Q: Will it support real-time analysis in the future? 199 | 200 | A: No. If you want this feature, it is recommended to use solutions such as GoAccess, ELK, Grafana + Time Series DBMS. 201 | 202 | ## License 203 | 204 | GeoLite2 Database [License](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data#license) 205 | 206 | Nginx-Log-Analyzer [License](https://github.com/fantasticmao/nginx-log-analyzer/blob/main/LICENSE) 207 | 208 | Copyright (c) 2021 fantasticmao 209 | -------------------------------------------------------------------------------- /README_ZH.md: -------------------------------------------------------------------------------- 1 | # Nginx-Log-Analyzer 2 | 3 | [![Actions Status](https://github.com/fantasticmao/nginx-log-analyzer/workflows/ci/badge.svg)](https://github.com/fantasticmao/nginx-log-analyzer/actions) 4 | [![codecov](https://codecov.io/gh/fantasticmao/nginx-log-analyzer/branch/main/graph/badge.svg)](https://codecov.io/gh/fantasticmao/nginx-log-analyzer) 5 | ![Go Version](https://img.shields.io/github/go-mod/go-version/fantasticmao/nginx-log-analyzer) 6 | [![Go Report Card](https://goreportcard.com/badge/github.com/fantasticmao/nginx-log-analyzer)](https://goreportcard.com/report/github.com/fantasticmao/nginx-log-analyzer) 7 | [![Release](https://img.shields.io/github/v/release/fantasticmao/nginx-log-analyzer)](https://github.com/fantasticmao/nginx-log-analyzer/releases) 8 | [![License](https://img.shields.io/github/license/fantasticmao/nginx-log-analyzer)](https://github.com/fantasticmao/nginx-log-analyzer/blob/main/LICENSE) 9 | 10 | README [English](README.md) | [中文](README_ZH.md) 11 | 12 | ## 这是什么 13 | 14 | Nginx-Log-Analyzer 是一个轻量的（简陋的）的日志分析工具，用于满足我自己对 Nginx 访问日志的分析需求。 15 | 16 | Nginx-Log-Analyzer 采用 Go 语言来编写，运行时只需一个 2 MB 左右的可执行文件，目前支持的功能特性如下： 17 | 18 | - [x] 基于请求时间筛选数据 19 | - [x] 支持多种日志格式配置 20 | - combined（Nginx 默认配置） 21 | - JSON 22 | - [x] 支持同时分析多个文件 23 | - [x] 支持分析 .gz 压缩文件 24 | - [x] 支持多种 [统计指标](#指定分析类型--t) 25 | 26 | ### 和 [GoAccess](https://goaccess.io/) 相比有什么优势 27 | 28 | GoAccess 是一个优秀和强大的实时 web 日志分析工具，支持以命令行或者浏览器的两种交互方式。不过据我所知，GoAccess 似乎不支持按百分位统计 URI 响应时间，Nginx-Log-Analyzer 支持这个特性。 29 | 30 | 如果在开发 Nginx-Log-Analyzer 之前，我知道有 GoAccess 的话，可能我会直接使用它了。GoAccess 很强大，我爱 GoAccess。 31 | 32 | ### 和 [ELK](https://www.elastic.co/cn/what-is/elk-stack) 相比有什么优势 33 | 34 | ELK 虽然功能强大，但安装和配置比较麻烦，对机器性能也有一定要求。Nginx-Log-Analyzer 更加轻量，使用起来更加简单，适用于一些简单的日志分析场景。 35 | 36 | ## 快速开始 37 | 38 | ### 下载安装 39 | 40 | 在 Nginx-Log-Analyzer 的 GitHub [Release](https://github.com/fantasticmao/nginx-log-analyzer/releases) 41 | 页面中，下载对应平台的二进制可执行文件即可。 42 | 43 | #### GeoIP2 和 GeoLite2 44 | 45 | [GeoIP2](https://www.maxmind.com/en/geoip2-city) 是商业版的 IP 46 | 地理定位的数据库，需要付费才能使用。[GeoLite2](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data) 是免费版和低精度版的 47 | GeoIP2，以 [署名-相同方式共享 4.0 国际](https://creativecommons.org/licenses/by-sa/4.0/deed.zh) 48 | 许可证发行，在 [MaxMind](https://www.maxmind.com/en/accounts/current/geoip/downloads) 官网登录即可下载。 49 | 50 | 在使用 Nginx-Log-Analyzer 时，如果需要解析 IP 的地理位置（即使用 `-t 4` 模式），则需要额外下载 GeoIP2 或者 GeoLite2 51 | 的城市数据库文件，保存至默认配置目录 `${HOME}/.config/nginx-log-analyzer/` 中的 `City.mmdb` 文件。对应的 shell 命令如下： 52 | 53 | ```shell 54 | ~$ mkdir -p ${HOME}/.config/nginx-log-analyzer 55 | ~$ tar -xzf GeoLite2-City_20211109.tar.gz 56 | ~$ cp GeoLite2-City_20211109/GeoLite2-City.mmdb ${HOME}/.config/nginx-log-analyzer/City.mmdb 57 | ``` 58 | 59 | #### 配置 Nginx 60 | 61 | Nginx-Log-Analyzer 默认解析 combined 格式的 Nginx 访问日志，这意味着日志中将包含以下字段： 62 | 63 | - $remote_addr 64 | - $remote_user 65 | - $time_local 66 | - $request 67 | - $status 68 | - $body_bytes_sent 69 | - $http_referer 70 | - $http_user_agent 71 | 72 | 在使用 Nginx-Log-Analyzer 时，如果需要更多类型的 [统计指标](#指定分析类型--t)，则需要使用 `-lf json` 选项指定 JSON 格式的日志解析模式， 并且需要在 Nginx 73 | 配置中添加如下的 `log_format` 和 `access_log` 指令： 74 | 75 | ```text 76 | log_format json_log escape=json '{"remote_addr":"$remote_addr",' 77 | '"time_local":"$time_local",' 78 | '"request":"$request",' 79 | '"status":$status,' 80 | '"body_bytes_sent":$body_bytes_sent,' 81 | '"http_user_agent":"$http_user_agent",' 82 | '"request_time":$request_time}'; 83 | access_log /path/to/access.json.log json_log; 84 | ``` 85 | 86 | - `log_format` 指令只能出现在 `http` 上下文中； 87 | - `access_log` 指令可以出现在 `http`、`server`、`location` 等上下文中，并且需要使用如上声明的 `log_format`； 88 | - 可以同时使用多个 `access_log` 指令，而不用删除原先已有的配置。例如： 89 | ```text 90 | access_log /path/to/access.log; 91 | access_log /path/to/access.json.log json_log; 92 | ``` 93 | 94 | 相关文档: http://nginx.org/en/docs/http/ngx_http_log_module.html 95 | 96 | ### 命令行选项 97 | 98 | #### 显示版本 -v 99 | 100 | `-v` 选项依次显示 Nginx-Log-Analyzer 的构建版本、构建时间、构建时的 Git Commit。 101 | 102 | #### 指定配置目录 -d 103 | 104 | `-d` 选项可以指定 Nginx-Log-Analyzer 运行时需要的配置目录，默认的配置目录为 `${HOME}/.config/nginx-log-analyzer/`。 105 | 106 | #### 指定日志格式 -lf 107 | 108 | `-lf` 选项可以指定 Nginx-Log-Analyzer 解析的日志格式，可用的值为 combined 和 json，默认值为 combined。 109 | 110 | #### 指定分析类型 -t 111 | 112 | `-t` 选项可以指定本次分析的类型，具体的分析类型和对应的统计指标如下表： 113 | 114 | | 是否支持 | 分析类型 `-t` | 统计指标 | 需要的字段或者依赖 | 115 | | -------- | ------------- | ------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | 116 | | ✅ | 0 | PV 和 UV | $remote_addr | 117 | | ✅ | 1 | 访问最多的 IP | $remote_addr | 118 | | ✅ | 2 | 访问最多的 URI | $request | 119 | | ✅ | 3 | 访问最多的 User-Agent | $http_user_agent | 120 | | ✅ | 4 | 访问最多的国家和城市 | $remote_addr、MaxMind [GeoIP2](https://www.maxmind.com/en/geoip2-city) 或者 [GeoLite2](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data) 城市数据库 | 121 | | ✅ | 5 | 频率最高的响应状态码 | $status、$request | 122 | | ✅ | 6 | 最大 URI 平均响应时间 | $request、$request_time | 123 | | ✅ | 7 | 最大 URI 百分位响应时间，例如 P1(最小)，P50(中位)，P95，P100(最大) | $request、$request_time | 124 | 125 | #### 限制请求时间 -ta -tb 126 | 127 | `-ta` 和 `-tb` 选项可以基于请求时间来过滤日志数据，`ta` 是 time after 的缩写，`tb` 是 time before 的缩写。 128 | 129 | `-ta` 和 `-tb` 选项需要在 Nginx 的 `log_format` 中配置 $time_local 字段。 130 | 131 | #### 限制输出行数 -n -n2 132 | 133 | `-n` 和 `-n2` 选项可以限制 Nginx-Log-Analyzer 的输出行数，`-n2` 仅对 `-t 4` 模式生效。 134 | 135 | #### 指定百分位值 -p 136 | 137 | `-p` 选项可以指定 `-t 7` 模式中的百分位值，默认值为 95。 138 | 139 | ### 使用示例 140 | 141 | #### 基于请求时间过滤数据 142 | 143 | ![image](docs/tatb.png) 144 | 145 | #### 同时分析多个文件 146 | 147 | ![image](docs/logs.png) 148 | 149 | #### 分析 .gz 压缩文件 150 | 151 | ![image](docs/loggz.png) 152 | 153 | #### 统计访问最多的 IP 154 | 155 | ![image](docs/t1.png) 156 | 157 | #### 统计访问最多的 URI 158 | 159 | ![image](docs/t2.png) 160 | 161 | #### 统计访问最多的 User-Agent 162 | 163 | ![image](docs/t3.png) 164 | 165 | #### 统计访问最多的国家和城市 166 | 167 | ![image](docs/t4.png) 168 | 169 | #### 统计频率最高的响应状态码 170 | 171 | ![image](docs/t5.png) 172 | 173 | #### 统计最大 URI 平均响应时间 174 | 175 | ![image](docs/t6.png) 176 | 177 | #### 统计最大 URI P90 响应时间 178 | 179 | ![image](docs/t7.png) 180 | 181 | ## 常见的问题和回答 182 | 183 | 问：未来是否会支持实时解析？ 184 | 185 | 答：不会支持。如果想要这个特性，建议使用 GoAccess、ELK、Grafana + 时序数据库之类的方案。 186 | 187 | ## 版权声明 188 | 189 | GeoLite2 Database [版权声明](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data#license) 190 | 191 | Nginx-Log-Analyzer [版权声明](https://github.com/fantasticmao/nginx-log-analyzer/blob/main/LICENSE) 192 | 193 | Copyright (c) 2021 fantasticmao 194 | -------------------------------------------------------------------------------- /VERSION: -------------------------------------------------------------------------------- 1 | v1.0.1 2 | -------------------------------------------------------------------------------- /cache/cache.go: -------------------------------------------------------------------------------- 1 | package cache 2 | 3 | type Key interface{} 4 | 5 | type Cache interface { 6 | Len() int 7 | 8 | Get(key Key) interface{} 9 | 10 | Put(key Key, value interface{}) 11 | } 12 | -------------------------------------------------------------------------------- /cache/cache_lru.go: -------------------------------------------------------------------------------- 1 | package cache 2 | 3 | import "container/list" 4 | 5 | type LruCache struct { 6 | capacity int 7 | list *list.List 8 | cache map[Key]*list.Element 9 | } 10 | 11 | type lruEntry struct { 12 | key Key 13 | value interface{} 14 | } 15 | 16 | func NewLruCache(capacity int) *LruCache { 17 | return &LruCache{ 18 | capacity: capacity, 19 | list: list.New(), 20 | cache: make(map[Key]*list.Element), 21 | } 22 | } 23 | 24 | func (c *LruCache) Len() int { 25 | if c.list == nil || c.cache == nil { 26 | return 0 27 | } 28 | return c.list.Len() 29 | } 30 | 31 | func (c *LruCache) Get(key Key) interface{} { 32 | if c.list == nil || c.cache == nil { 33 | return nil 34 | } 35 | 36 | element, ok := c.cache[key] 37 | if !ok { 38 | return nil 39 | } 40 | c.list.MoveToFront(element) 41 | return element.Value.(*lruEntry).value 42 | } 43 | 44 | func (c *LruCache) Put(key Key, value interface{}) { 45 | if c.list == nil || c.cache == nil { 46 | return 47 | } 48 | 49 | element, ok := c.cache[key] 50 | if ok { // update 51 | c.list.MoveToFront(element) 52 | element.Value.(*lruEntry).value = value 53 | return 54 | } else { // insert 55 | element = c.list.PushFront(&lruEntry{key: key, value: value}) 56 | c.cache[key] = element 57 | if c.list.Len() > c.capacity { // evict 58 | c.removeOldest() 59 | } 60 | } 61 | } 62 | 63 | func (c *LruCache) removeOldest() { 64 | oldestElement := c.list.Back() 65 | c.list.Remove(oldestElement) 66 | delete(c.cache, oldestElement.Value.(*lruEntry).key) 67 | } 68 | -------------------------------------------------------------------------------- /cache/cache_test.go: -------------------------------------------------------------------------------- 1 | package cache 2 | 3 | import ( 4 | "github.com/stretchr/testify/assert" 5 | "testing" 6 | ) 7 | 8 | func TestNewLruCache(t *testing.T) { 9 | c := NewLruCache(3) 10 | c.Put(1, 1) 11 | c.Put(2, 2) 12 | c.Put(3, 3) 13 | c.Put(4, 4) 14 | assert.Equal(t, 3, c.Len()) 15 | assert.Nil(t, c.Get(1)) 16 | assert.Equal(t, 2, c.Get(2)) 17 | assert.Equal(t, 3, c.Get(3)) 18 | assert.Equal(t, 4, c.Get(4)) 19 | } 20 | -------------------------------------------------------------------------------- /docs/loggz.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fantasticmao/nginx-log-analyzer/0f5422a12ca7c88f6791089d14e7afc13fc2d3b6/docs/loggz.png -------------------------------------------------------------------------------- /docs/logs.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fantasticmao/nginx-log-analyzer/0f5422a12ca7c88f6791089d14e7afc13fc2d3b6/docs/logs.png -------------------------------------------------------------------------------- /docs/t1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fantasticmao/nginx-log-analyzer/0f5422a12ca7c88f6791089d14e7afc13fc2d3b6/docs/t1.png -------------------------------------------------------------------------------- /docs/t2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fantasticmao/nginx-log-analyzer/0f5422a12ca7c88f6791089d14e7afc13fc2d3b6/docs/t2.png -------------------------------------------------------------------------------- /docs/t3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fantasticmao/nginx-log-analyzer/0f5422a12ca7c88f6791089d14e7afc13fc2d3b6/docs/t3.png -------------------------------------------------------------------------------- /docs/t4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fantasticmao/nginx-log-analyzer/0f5422a12ca7c88f6791089d14e7afc13fc2d3b6/docs/t4.png -------------------------------------------------------------------------------- /docs/t5.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fantasticmao/nginx-log-analyzer/0f5422a12ca7c88f6791089d14e7afc13fc2d3b6/docs/t5.png -------------------------------------------------------------------------------- /docs/t6.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fantasticmao/nginx-log-analyzer/0f5422a12ca7c88f6791089d14e7afc13fc2d3b6/docs/t6.png -------------------------------------------------------------------------------- /docs/t7.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fantasticmao/nginx-log-analyzer/0f5422a12ca7c88f6791089d14e7afc13fc2d3b6/docs/t7.png -------------------------------------------------------------------------------- /docs/tatb.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fantasticmao/nginx-log-analyzer/0f5422a12ca7c88f6791089d14e7afc13fc2d3b6/docs/tatb.png -------------------------------------------------------------------------------- /go.mod: -------------------------------------------------------------------------------- 1 | module github.com/fantasticmao/nginx-log-analyzer 2 | 3 | go 1.21.3 4 | 5 | require ( 6 | github.com/oschwald/geoip2-golang v1.9.0 7 | github.com/stretchr/testify v1.8.4 8 | ) 9 | 10 | require ( 11 | github.com/davecgh/go-spew v1.1.1 // indirect 12 | github.com/oschwald/maxminddb-golang v1.12.0 // indirect 13 | github.com/pmezard/go-difflib v1.0.0 // indirect 14 | golang.org/x/sys v0.14.0 // indirect 15 | gopkg.in/yaml.v3 v3.0.1 // indirect 16 | ) 17 | -------------------------------------------------------------------------------- /go.sum: -------------------------------------------------------------------------------- 1 | github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= 2 | github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= 3 | github.com/oschwald/geoip2-golang v1.9.0 h1:uvD3O6fXAXs+usU+UGExshpdP13GAqp4GBrzN7IgKZc= 4 | github.com/oschwald/geoip2-golang v1.9.0/go.mod h1:BHK6TvDyATVQhKNbQBdrj9eAvuwOMi2zSFXizL3K81Y= 5 | github.com/oschwald/maxminddb-golang v1.12.0 h1:9FnTOD0YOhP7DGxGsq4glzpGy5+w7pq50AS6wALUMYs= 6 | github.com/oschwald/maxminddb-golang v1.12.0/go.mod h1:q0Nob5lTCqyQ8WT6FYgS1L7PXKVVbgiymefNwIjPzgY= 7 | github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= 8 | github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= 9 | github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= 10 | github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= 11 | golang.org/x/sys v0.14.0 h1:Vz7Qs629MkJkGyHxUlRHizWJRG2j8fbQKjELVSNhy7Q= 12 | golang.org/x/sys v0.14.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= 13 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= 14 | gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= 15 | gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= 16 | gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= 17 | -------------------------------------------------------------------------------- /handler/benchmark_test.go: -------------------------------------------------------------------------------- 1 | package handler 2 | 3 | import ( 4 | "math/rand" 5 | "testing" 6 | ) 7 | 8 | var ( 9 | // see https://github.com/maxmind/MaxMind-DB/blob/main/source-data/GeoLite2-City-Test.json 10 | ips = []string{ 11 | "2.125.160.216", "67.43.156.0", "81.2.69.142", 12 | "81.2.69.144", "89.160.20.112", "175.16.199.0", 13 | "2001:218::", "2001:252::", "2001:230::", 14 | } 15 | ) 16 | 17 | func BenchmarkQueryIpLocation(b *testing.B) { 18 | handler := NewMostVisitedLocationsHandler("../testdata/GeoLite2-City-Test.mmdb", limit) 19 | 20 | for i, l := 0, len(ips); i < b.N; i++ { 21 | ip := ips[rand.Intn(l)] 22 | _, _ = handler.queryIpLocation(ip) 23 | } 24 | } 25 | 26 | func BenchmarkCachedQueryIpLocation(b *testing.B) { 27 | handler := NewMostVisitedLocationsHandler("../testdata/GeoLite2-City-Test.mmdb", limit) 28 | 29 | for i, l := 0, len(ips); i < b.N; i++ { 30 | ip := ips[rand.Intn(l)] 31 | _, _ = handler.cachedQueryIpLocation(ip) 32 | } 33 | } 34 | -------------------------------------------------------------------------------- /handler/example_test.go: -------------------------------------------------------------------------------- 1 | package handler 2 | 3 | import "github.com/fantasticmao/nginx-log-analyzer/parser" 4 | 5 | func ExampleNewPvAndUvHandler() { 6 | handler := NewPvAndUvHandler() 7 | handler.Input(&parser.LogInfo{RemoteAddr: ip1}) 8 | handler.Input(&parser.LogInfo{RemoteAddr: ip1}) 9 | handler.Input(&parser.LogInfo{RemoteAddr: ip1}) 10 | handler.Input(&parser.LogInfo{RemoteAddr: ip2}) 11 | handler.Input(&parser.LogInfo{RemoteAddr: ip2}) 12 | handler.Input(&parser.LogInfo{RemoteAddr: ip3}) 13 | handler.Output(limit) 14 | // Output: 15 | // PV: 6 16 | // UV: 3 17 | } 18 | 19 | func ExampleNewMostVisitedIpsHandler() { 20 | handler := NewMostVisitedFieldsHandler(AnalysisTypeVisitedIps) 21 | handler.Input(&parser.LogInfo{RemoteAddr: ip1}) 22 | handler.Input(&parser.LogInfo{RemoteAddr: ip1}) 23 | handler.Input(&parser.LogInfo{RemoteAddr: ip1}) 24 | handler.Input(&parser.LogInfo{RemoteAddr: ip2}) 25 | handler.Input(&parser.LogInfo{RemoteAddr: ip2}) 26 | handler.Input(&parser.LogInfo{RemoteAddr: ip3}) 27 | handler.Output(limit) 28 | // Output: 29 | // "192.168.1.1" hits: 3 30 | // "192.168.1.2" hits: 2 31 | // "192.168.1.3" hits: 1 32 | } 33 | 34 | func ExampleNewMostVisitedUrisHandler() { 35 | handler := NewMostVisitedFieldsHandler(AnalysisTypeVisitedUris) 36 | handler.Input(&parser.LogInfo{Request: uri1}) 37 | handler.Input(&parser.LogInfo{Request: uri1}) 38 | handler.Input(&parser.LogInfo{Request: uri1}) 39 | handler.Input(&parser.LogInfo{Request: uri2}) 40 | handler.Input(&parser.LogInfo{Request: uri2}) 41 | handler.Input(&parser.LogInfo{Request: uri3}) 42 | handler.Output(limit) 43 | // Output: 44 | // "GET /name/Tom HTTP/2.0" hits: 3 45 | // "GET /name/Sam HTTP/2.0" hits: 2 46 | // "GET /name/Bob HTTP/2.0" hits: 1 47 | } 48 | 49 | func ExampleNewMostVisitedUserAgentsHandler() { 50 | handler := NewMostVisitedFieldsHandler(AnalysisTypeVisitedUserAgents) 51 | handler.Input(&parser.LogInfo{HttpUserAgent: userAgent1}) 52 | handler.Input(&parser.LogInfo{HttpUserAgent: userAgent1}) 53 | handler.Input(&parser.LogInfo{HttpUserAgent: userAgent1}) 54 | handler.Input(&parser.LogInfo{HttpUserAgent: userAgent2}) 55 | handler.Input(&parser.LogInfo{HttpUserAgent: userAgent2}) 56 | handler.Input(&parser.LogInfo{HttpUserAgent: userAgent3}) 57 | handler.Output(limit) 58 | // Output: 59 | // "iOS" hits: 3 60 | // "Android" hits: 2 61 | // "Windows" hits: 1 62 | } 63 | 64 | func ExampleNewMostVisitedLocationsHandler() { 65 | handler := NewMostVisitedLocationsHandler("../testdata/GeoLite2-City-Test.mmdb", limit) 66 | 67 | // see https://github.com/maxmind/MaxMind-DB/blob/main/source-data/GeoLite2-City-Test.json 68 | handler.Input(&parser.LogInfo{RemoteAddr: "175.16.199.0"}) // China -> Changchun 69 | handler.Input(&parser.LogInfo{RemoteAddr: "175.16.199.0"}) 70 | handler.Input(&parser.LogInfo{RemoteAddr: "175.16.199.0"}) 71 | handler.Input(&parser.LogInfo{RemoteAddr: "2.125.160.216"}) // United Kingdom -> Boxford 72 | handler.Input(&parser.LogInfo{RemoteAddr: "2.125.160.216"}) 73 | handler.Input(&parser.LogInfo{RemoteAddr: "2001:218::"}) // Japan -> unknown 74 | handler.Output(limit) 75 | // Output: 76 | // [中国 China] hits: 3 77 | // |--[长春 Changchun] hits: 3 78 | // | |--"175.16.199.0" hits: 3 79 | // [United Kingdom] hits: 2 80 | // |--[Boxford] hits: 2 81 | // | |--"2.125.160.216" hits: 2 82 | // [日本 Japan] hits: 1 83 | // |--[unknown] hits: 1 84 | // | |--"2001:218::" hits: 1 85 | } 86 | 87 | func ExampleNewMostFrequentStatusHandler() { 88 | handler := NewMostFrequentStatusHandler() 89 | handler.Input(&parser.LogInfo{Status: responseStatus1, Request: uri1}) 90 | handler.Input(&parser.LogInfo{Status: responseStatus1, Request: uri1}) 91 | handler.Input(&parser.LogInfo{Status: responseStatus2, Request: uri1}) 92 | handler.Input(&parser.LogInfo{Status: responseStatus2, Request: uri1}) 93 | handler.Input(&parser.LogInfo{Status: responseStatus3, Request: uri1}) 94 | handler.Input(&parser.LogInfo{Status: responseStatus3, Request: uri1}) 95 | handler.Input(&parser.LogInfo{Status: responseStatus1, Request: uri2}) 96 | handler.Input(&parser.LogInfo{Status: responseStatus2, Request: uri2}) 97 | handler.Input(&parser.LogInfo{Status: responseStatus3, Request: uri3}) 98 | handler.Output(limit) 99 | // Output: 100 | // 200 hits: 3 101 | // |--"GET /name/Tom HTTP/2.0" hits: 2 102 | // |--"GET /name/Sam HTTP/2.0" hits: 1 103 | // 302 hits: 3 104 | // |--"GET /name/Tom HTTP/2.0" hits: 2 105 | // |--"GET /name/Sam HTTP/2.0" hits: 1 106 | // 404 hits: 3 107 | // |--"GET /name/Tom HTTP/2.0" hits: 2 108 | // |--"GET /name/Bob HTTP/2.0" hits: 1 109 | } 110 | 111 | func ExampleNewLargestAverageTimeUrisHandler() { 112 | handler := NewLargestAverageTimeUrisHandler() 113 | handler.Input(&parser.LogInfo{Request: uri1, RequestTime: responseTime1}) 114 | handler.Input(&parser.LogInfo{Request: uri1, RequestTime: responseTime2}) 115 | handler.Input(&parser.LogInfo{Request: uri1, RequestTime: responseTime3}) 116 | handler.Input(&parser.LogInfo{Request: uri2, RequestTime: responseTime1}) 117 | handler.Input(&parser.LogInfo{Request: uri2, RequestTime: responseTime2}) 118 | handler.Input(&parser.LogInfo{Request: uri3, RequestTime: responseTime1}) 119 | handler.Output(limit) 120 | // Output: 121 | // "GET /name/Tom HTTP/2.0" average response-time: 0.200 122 | // "GET /name/Sam HTTP/2.0" average response-time: 0.150 123 | // "GET /name/Bob HTTP/2.0" average response-time: 0.100 124 | } 125 | 126 | func ExampleNewLargestPercentTimeUrisHandler() { 127 | handler := NewLargestPercentTimeUrisHandler(30) 128 | handler.Input(&parser.LogInfo{Request: uri1, RequestTime: responseTime1}) 129 | handler.Input(&parser.LogInfo{Request: uri1, RequestTime: responseTime2}) 130 | handler.Input(&parser.LogInfo{Request: uri1, RequestTime: responseTime3}) 131 | handler.Input(&parser.LogInfo{Request: uri2, RequestTime: responseTime2}) 132 | handler.Input(&parser.LogInfo{Request: uri2, RequestTime: responseTime3}) 133 | handler.Input(&parser.LogInfo{Request: uri3, RequestTime: responseTime3}) 134 | handler.Output(limit) 135 | // Output: 136 | // "GET /name/Bob HTTP/2.0" P30.00 response-time: 0.300 137 | // "GET /name/Sam HTTP/2.0" P30.00 response-time: 0.200 138 | // "GET /name/Tom HTTP/2.0" P30.00 response-time: 0.100 139 | } 140 | -------------------------------------------------------------------------------- /handler/handler.go: -------------------------------------------------------------------------------- 1 | package handler 2 | 3 | import ( 4 | "github.com/fantasticmao/nginx-log-analyzer/parser" 5 | ) 6 | 7 | const ( 8 | AnalysisTypePvAndUv = iota 9 | AnalysisTypeVisitedIps 10 | AnalysisTypeVisitedUris 11 | AnalysisTypeVisitedUserAgents 12 | AnalysisTypeVisitedLocations 13 | AnalysisTypeResponseStatus 14 | AnalysisTypeAverageTimeUris 15 | AnalysisTypePercentTimeUris 16 | ) 17 | 18 | type Handler interface { 19 | Input(info *parser.LogInfo) 20 | 21 | Output(limit int) 22 | } 23 | -------------------------------------------------------------------------------- /handler/handler_test.go: -------------------------------------------------------------------------------- 1 | package handler 2 | 3 | import ( 4 | "github.com/fantasticmao/nginx-log-analyzer/parser" 5 | "github.com/stretchr/testify/assert" 6 | "testing" 7 | ) 8 | 9 | var ( 10 | limit = 15 11 | 12 | ip1 = "192.168.1.1" 13 | ip2 = "192.168.1.2" 14 | ip3 = "192.168.1.3" 15 | 16 | uri1 = "GET /name/Tom HTTP/2.0" 17 | uri2 = "GET /name/Sam HTTP/2.0" 18 | uri3 = "GET /name/Bob HTTP/2.0" 19 | 20 | userAgent1 = "iOS" 21 | userAgent2 = "Android" 22 | userAgent3 = "Windows" 23 | 24 | responseStatus1 = 200 25 | responseStatus2 = 302 26 | responseStatus3 = 404 27 | 28 | responseTime1 = 0.1 29 | responseTime2 = 0.2 30 | responseTime3 = 0.3 31 | ) 32 | 33 | func TestNewPvAndUvHandler(t *testing.T) { 34 | handler := NewPvAndUvHandler() 35 | handler.Input(&parser.LogInfo{RemoteAddr: ip1}) 36 | handler.Input(&parser.LogInfo{RemoteAddr: ip1}) 37 | handler.Input(&parser.LogInfo{RemoteAddr: ip1}) 38 | handler.Input(&parser.LogInfo{RemoteAddr: ip2}) 39 | handler.Input(&parser.LogInfo{RemoteAddr: ip2}) 40 | handler.Input(&parser.LogInfo{RemoteAddr: ip3}) 41 | 42 | assert.Equal(t, int32(6), handler.pv) 43 | assert.Equal(t, int32(3), handler.uv) 44 | } 45 | 46 | func TestNewMostVisitedIpsHandler(t *testing.T) { 47 | handler := NewMostVisitedFieldsHandler(AnalysisTypeVisitedIps) 48 | handler.Input(&parser.LogInfo{RemoteAddr: ip1}) 49 | handler.Input(&parser.LogInfo{RemoteAddr: ip1}) 50 | handler.Input(&parser.LogInfo{RemoteAddr: ip1}) 51 | handler.Input(&parser.LogInfo{RemoteAddr: ip2}) 52 | handler.Input(&parser.LogInfo{RemoteAddr: ip2}) 53 | handler.Input(&parser.LogInfo{RemoteAddr: ip3}) 54 | 55 | assert.Equal(t, 3, handler.countMap[ip1]) 56 | assert.Equal(t, 2, handler.countMap[ip2]) 57 | assert.Equal(t, 1, handler.countMap[ip3]) 58 | } 59 | 60 | func TestNewMostVisitedUrisHandler(t *testing.T) { 61 | handler := NewMostVisitedFieldsHandler(AnalysisTypeVisitedUris) 62 | handler.Input(&parser.LogInfo{Request: uri1}) 63 | handler.Input(&parser.LogInfo{Request: uri1}) 64 | handler.Input(&parser.LogInfo{Request: uri1}) 65 | handler.Input(&parser.LogInfo{Request: uri2}) 66 | handler.Input(&parser.LogInfo{Request: uri2}) 67 | handler.Input(&parser.LogInfo{Request: uri3}) 68 | 69 | assert.Equal(t, 3, handler.countMap[uri1]) 70 | assert.Equal(t, 2, handler.countMap[uri2]) 71 | assert.Equal(t, 1, handler.countMap[uri3]) 72 | } 73 | 74 | func TestNewMostVisitedUserAgentsHandler(t *testing.T) { 75 | handler := NewMostVisitedFieldsHandler(AnalysisTypeVisitedUserAgents) 76 | handler.Input(&parser.LogInfo{HttpUserAgent: userAgent1}) 77 | handler.Input(&parser.LogInfo{HttpUserAgent: userAgent1}) 78 | handler.Input(&parser.LogInfo{HttpUserAgent: userAgent1}) 79 | handler.Input(&parser.LogInfo{HttpUserAgent: userAgent2}) 80 | handler.Input(&parser.LogInfo{HttpUserAgent: userAgent2}) 81 | handler.Input(&parser.LogInfo{HttpUserAgent: userAgent3}) 82 | 83 | assert.Equal(t, 3, handler.countMap[userAgent1]) 84 | assert.Equal(t, 2, handler.countMap[userAgent2]) 85 | assert.Equal(t, 1, handler.countMap[userAgent3]) 86 | } 87 | 88 | func TestNewMostVisitedLocationsHandler(t *testing.T) { 89 | handler := NewMostVisitedLocationsHandler("../testdata/GeoLite2-City-Test.mmdb", limit) 90 | assert.NotNil(t, handler.geoLite2Db) 91 | 92 | // see https://github.com/maxmind/MaxMind-DB/blob/main/source-data/GeoLite2-City-Test.json 93 | handler.Input(&parser.LogInfo{RemoteAddr: "175.16.199.0"}) // China -> Changchun 94 | handler.Input(&parser.LogInfo{RemoteAddr: "175.16.199.0"}) 95 | handler.Input(&parser.LogInfo{RemoteAddr: "175.16.199.0"}) 96 | handler.Input(&parser.LogInfo{RemoteAddr: "2.125.160.216"}) // United Kingdom -> Boxford 97 | handler.Input(&parser.LogInfo{RemoteAddr: "2.125.160.216"}) 98 | handler.Input(&parser.LogInfo{RemoteAddr: "2001:218::"}) // Japan -> unknown 99 | 100 | assert.Equal(t, 3, handler.countryCountMap["中国 China"]) 101 | assert.Equal(t, 2, handler.countryCountMap["United Kingdom"]) 102 | assert.Equal(t, 1, handler.countryCountMap["日本 Japan"]) 103 | 104 | assert.Equal(t, 3, handler.countryCityCountMap["中国 China"]["长春 Changchun"]) 105 | assert.Equal(t, 2, handler.countryCityCountMap["United Kingdom"]["Boxford"]) 106 | assert.Equal(t, 1, handler.countryCityCountMap["日本 Japan"]["unknown"]) 107 | 108 | assert.Equal(t, 3, handler.countryCityIpCountMap["中国 China"]["长春 Changchun"]["175.16.199.0"]) 109 | assert.Equal(t, 2, handler.countryCityIpCountMap["United Kingdom"]["Boxford"]["2.125.160.216"]) 110 | assert.Equal(t, 1, handler.countryCityIpCountMap["日本 Japan"]["unknown"]["2001:218::"]) 111 | } 112 | 113 | func TestNewMostFrequentStatusHandler(t *testing.T) { 114 | handler := NewMostFrequentStatusHandler() 115 | handler.Input(&parser.LogInfo{Status: responseStatus1, Request: uri1}) 116 | handler.Input(&parser.LogInfo{Status: responseStatus1, Request: uri1}) 117 | handler.Input(&parser.LogInfo{Status: responseStatus2, Request: uri1}) 118 | handler.Input(&parser.LogInfo{Status: responseStatus2, Request: uri1}) 119 | handler.Input(&parser.LogInfo{Status: responseStatus3, Request: uri1}) 120 | handler.Input(&parser.LogInfo{Status: responseStatus3, Request: uri1}) 121 | handler.Input(&parser.LogInfo{Status: responseStatus1, Request: uri2}) 122 | handler.Input(&parser.LogInfo{Status: responseStatus2, Request: uri2}) 123 | handler.Input(&parser.LogInfo{Status: responseStatus3, Request: uri3}) 124 | 125 | assert.Equal(t, 3, handler.statusCountMap[responseStatus1]) 126 | assert.Equal(t, 3, handler.statusCountMap[responseStatus2]) 127 | assert.Equal(t, 3, handler.statusCountMap[responseStatus3]) 128 | 129 | assert.Equal(t, 2, handler.statusUriCountMap[responseStatus1][uri1]) 130 | assert.Equal(t, 1, handler.statusUriCountMap[responseStatus1][uri2]) 131 | assert.Equal(t, 2, handler.statusUriCountMap[responseStatus2][uri1]) 132 | assert.Equal(t, 1, handler.statusUriCountMap[responseStatus2][uri2]) 133 | assert.Equal(t, 2, handler.statusUriCountMap[responseStatus3][uri1]) 134 | assert.Equal(t, 1, handler.statusUriCountMap[responseStatus3][uri3]) 135 | } 136 | 137 | func TestNewLargestAverageTimeUrisHandler(t *testing.T) { 138 | handler := NewLargestAverageTimeUrisHandler() 139 | handler.Input(&parser.LogInfo{Request: uri1, RequestTime: responseTime1}) 140 | handler.Input(&parser.LogInfo{Request: uri1, RequestTime: responseTime2}) 141 | handler.Input(&parser.LogInfo{Request: uri1, RequestTime: responseTime3}) 142 | handler.Input(&parser.LogInfo{Request: uri2, RequestTime: responseTime1}) 143 | handler.Input(&parser.LogInfo{Request: uri2, RequestTime: responseTime2}) 144 | handler.Input(&parser.LogInfo{Request: uri3, RequestTime: responseTime1}) 145 | 146 | assert.Equal(t, []float64{responseTime1, responseTime2, responseTime3}, handler.timeCostListMap[uri1]) 147 | assert.Equal(t, []float64{responseTime1, responseTime2}, handler.timeCostListMap[uri2]) 148 | assert.Equal(t, []float64{responseTime1}, handler.timeCostListMap[uri3]) 149 | } 150 | 151 | func TestNewLargestPercentTimeUrisHandler(t *testing.T) { 152 | handler := NewLargestPercentTimeUrisHandler(50) 153 | handler.Input(&parser.LogInfo{Request: uri1, RequestTime: responseTime1}) 154 | handler.Input(&parser.LogInfo{Request: uri1, RequestTime: responseTime2}) 155 | handler.Input(&parser.LogInfo{Request: uri1, RequestTime: responseTime3}) 156 | handler.Input(&parser.LogInfo{Request: uri2, RequestTime: responseTime2}) 157 | handler.Input(&parser.LogInfo{Request: uri2, RequestTime: responseTime3}) 158 | handler.Input(&parser.LogInfo{Request: uri3, RequestTime: responseTime3}) 159 | 160 | assert.Equal(t, []float64{responseTime1, responseTime2, responseTime3}, handler.timeCostListMap[uri1]) 161 | assert.Equal(t, []float64{responseTime2, responseTime3}, handler.timeCostListMap[uri2]) 162 | assert.Equal(t, []float64{responseTime3}, handler.timeCostListMap[uri3]) 163 | } 164 | -------------------------------------------------------------------------------- /handler/largest_average_time_uris.go: -------------------------------------------------------------------------------- 1 | package handler 2 | 3 | import ( 4 | "fmt" 5 | "github.com/fantasticmao/nginx-log-analyzer/parser" 6 | "sort" 7 | ) 8 | 9 | type LargestAverageTimeUrisHandler struct { 10 | timeCostListMap map[string][]float64 11 | } 12 | 13 | func NewLargestAverageTimeUrisHandler() *LargestAverageTimeUrisHandler { 14 | return &LargestAverageTimeUrisHandler{ 15 | timeCostListMap: make(map[string][]float64), 16 | } 17 | } 18 | 19 | func (handler *LargestAverageTimeUrisHandler) Input(info *parser.LogInfo) { 20 | if _, ok := handler.timeCostListMap[info.Request]; ok { 21 | handler.timeCostListMap[info.Request] = append(handler.timeCostListMap[info.Request], info.RequestTime) 22 | } else { 23 | array := []float64{info.RequestTime} 24 | handler.timeCostListMap[info.Request] = array 25 | } 26 | } 27 | 28 | func (handler *LargestAverageTimeUrisHandler) Output(limit int) { 29 | timeCostMap := make(map[string]float64) 30 | for uri, costList := range handler.timeCostListMap { 31 | var sum = 0.0 32 | for _, cost := range costList { 33 | sum += cost 34 | } 35 | timeCostMap[uri] = sum / float64(len(costList)) 36 | } 37 | 38 | keys := make([]string, 0, len(timeCostMap)) 39 | for k := range timeCostMap { 40 | keys = append(keys, k) 41 | } 42 | 43 | sort.Slice(keys, func(i, j int) bool { 44 | return timeCostMap[keys[i]] > timeCostMap[keys[j]] 45 | }) 46 | 47 | for i := 0; i < limit && i < len(keys); i++ { 48 | fmt.Printf("\"%v\" average response-time: %.3f\n", keys[i], timeCostMap[keys[i]]) 49 | } 50 | } 51 | -------------------------------------------------------------------------------- /handler/largest_percent_time_uris.go: -------------------------------------------------------------------------------- 1 | package handler 2 | 3 | import ( 4 | "fmt" 5 | "github.com/fantasticmao/nginx-log-analyzer/ioutil" 6 | "github.com/fantasticmao/nginx-log-analyzer/parser" 7 | "math" 8 | "sort" 9 | ) 10 | 11 | type LargestPercentTimeUrisHandler struct { 12 | percentile float64 13 | timeCostListMap map[string][]float64 14 | } 15 | 16 | func NewLargestPercentTimeUrisHandler(percentile float64) *LargestPercentTimeUrisHandler { 17 | if percentile <= 0 || percentile > 100 { 18 | ioutil.Fatal("illegal argument percentile: %.3f\n", percentile) 19 | return nil 20 | } 21 | return &LargestPercentTimeUrisHandler{ 22 | percentile: percentile, 23 | timeCostListMap: make(map[string][]float64), 24 | } 25 | } 26 | 27 | func (handler *LargestPercentTimeUrisHandler) Input(info *parser.LogInfo) { 28 | if _, ok := handler.timeCostListMap[info.Request]; ok { 29 | handler.timeCostListMap[info.Request] = append(handler.timeCostListMap[info.Request], info.RequestTime) 30 | } else { 31 | array := []float64{info.RequestTime} 32 | handler.timeCostListMap[info.Request] = array 33 | } 34 | } 35 | 36 | func (handler *LargestPercentTimeUrisHandler) Output(limit int) { 37 | timeCostMap := make(map[string]float64) 38 | for uri, costList := range handler.timeCostListMap { 39 | sort.Float64s(costList) 40 | 41 | // according to https://stackoverflow.com/questions/41413544/calculate-percentile-from-a-long-array 42 | index := int(math.Ceil(handler.percentile/100*float64(len(costList))) - 1) 43 | timeCostMap[uri] = costList[index] 44 | } 45 | 46 | keys := make([]string, 0, len(timeCostMap)) 47 | for k := range timeCostMap { 48 | keys = append(keys, k) 49 | } 50 | 51 | sort.Slice(keys, func(i, j int) bool { 52 | return timeCostMap[keys[i]] > timeCostMap[keys[j]] 53 | }) 54 | 55 | for i := 0; i < limit && i < len(keys); i++ { 56 | fmt.Printf("\"%v\" P%.2f response-time: %.3f\n", keys[i], handler.percentile, timeCostMap[keys[i]]) 57 | } 58 | } 59 | -------------------------------------------------------------------------------- /handler/most_frequent_status.go: -------------------------------------------------------------------------------- 1 | package handler 2 | 3 | import ( 4 | "fmt" 5 | "github.com/fantasticmao/nginx-log-analyzer/parser" 6 | "sort" 7 | ) 8 | 9 | type MostFrequentStatusHandler struct { 10 | // status -> count 11 | statusCountMap map[int]int 12 | // status -> uri -> count 13 | statusUriCountMap map[int]map[string]int 14 | } 15 | 16 | func NewMostFrequentStatusHandler() *MostFrequentStatusHandler { 17 | return &MostFrequentStatusHandler{ 18 | statusCountMap: make(map[int]int), 19 | statusUriCountMap: make(map[int]map[string]int), 20 | } 21 | } 22 | 23 | func (handler *MostFrequentStatusHandler) Input(info *parser.LogInfo) { 24 | if _, ok := handler.statusUriCountMap[info.Status]; !ok { 25 | handler.statusCountMap[info.Status] = 1 26 | handler.statusUriCountMap[info.Status] = make(map[string]int) 27 | } else { 28 | handler.statusCountMap[info.Status]++ 29 | } 30 | 31 | if _, ok := handler.statusUriCountMap[info.Status][info.Request]; !ok { 32 | handler.statusUriCountMap[info.Status][info.Request] = 1 33 | } else { 34 | handler.statusUriCountMap[info.Status][info.Request]++ 35 | } 36 | } 37 | 38 | func (handler *MostFrequentStatusHandler) Output(limit int) { 39 | statusCountKeys := make([]int, 0, len(handler.statusCountMap)) 40 | for k := range handler.statusCountMap { 41 | statusCountKeys = append(statusCountKeys, k) 42 | } 43 | sort.Ints(statusCountKeys) 44 | 45 | for _, status := range statusCountKeys { 46 | count := handler.statusCountMap[status] 47 | uriCountMap := handler.statusUriCountMap[status] 48 | fmt.Printf("%v hits: %v\n", status, count) 49 | 50 | uriCountKeys := make([]string, 0, len(uriCountMap)) 51 | for k := range uriCountMap { 52 | uriCountKeys = append(uriCountKeys, k) 53 | } 54 | sort.Slice(uriCountKeys, func(i, j int) bool { 55 | return uriCountMap[uriCountKeys[i]] > uriCountMap[uriCountKeys[j]] 56 | }) 57 | 58 | for i := 0; i < limit && i < len(uriCountKeys); i++ { 59 | uri := uriCountKeys[i] 60 | fmt.Printf(" |--\"%v\" hits: %v\n", uri, uriCountMap[uri]) 61 | } 62 | } 63 | } 64 | -------------------------------------------------------------------------------- /handler/most_visited_fields.go: -------------------------------------------------------------------------------- 1 | package handler 2 | 3 | import ( 4 | "fmt" 5 | "github.com/fantasticmao/nginx-log-analyzer/ioutil" 6 | "github.com/fantasticmao/nginx-log-analyzer/parser" 7 | "sort" 8 | ) 9 | 10 | type MostVisitedFieldsHandler struct { 11 | analysisType int 12 | countMap map[string]int 13 | } 14 | 15 | func NewMostVisitedFieldsHandler(analysisType int) *MostVisitedFieldsHandler { 16 | return &MostVisitedFieldsHandler{ 17 | analysisType: analysisType, 18 | countMap: make(map[string]int), 19 | } 20 | } 21 | 22 | func (handler *MostVisitedFieldsHandler) Input(info *parser.LogInfo) { 23 | var field string 24 | switch handler.analysisType { 25 | case AnalysisTypeVisitedIps: 26 | field = info.RemoteAddr 27 | case AnalysisTypeVisitedUris: 28 | field = info.Request 29 | case AnalysisTypeVisitedUserAgents: 30 | field = info.HttpUserAgent 31 | default: 32 | ioutil.Fatal("unsupported analysis type: %v\n", handler.analysisType) 33 | return 34 | } 35 | 36 | if _, ok := handler.countMap[field]; ok { 37 | handler.countMap[field]++ 38 | } else { 39 | handler.countMap[field] = 1 40 | } 41 | } 42 | 43 | func (handler *MostVisitedFieldsHandler) Output(limit int) { 44 | keys := make([]string, 0, len(handler.countMap)) 45 | for k := range handler.countMap { 46 | keys = append(keys, k) 47 | } 48 | 49 | sort.Slice(keys, func(i, j int) bool { 50 | return handler.countMap[keys[i]] > handler.countMap[keys[j]] 51 | }) 52 | 53 | for i := 0; i < limit && i < len(keys); i++ { 54 | fmt.Printf("\"%v\" hits: %v\n", keys[i], handler.countMap[keys[i]]) 55 | } 56 | } 57 | -------------------------------------------------------------------------------- /handler/most_visited_locations.go: -------------------------------------------------------------------------------- 1 | package handler 2 | 3 | import ( 4 | "fmt" 5 | "github.com/fantasticmao/nginx-log-analyzer/cache" 6 | "github.com/fantasticmao/nginx-log-analyzer/ioutil" 7 | "github.com/fantasticmao/nginx-log-analyzer/parser" 8 | "github.com/oschwald/geoip2-golang" 9 | "net" 10 | "sort" 11 | "strings" 12 | ) 13 | 14 | const ( 15 | countryChina = "China" 16 | countryJapan = "Japan" 17 | areaHongKong = "Hong Kong" 18 | areaTaiwan = "Taiwan" 19 | cityUnknown = "unknown" 20 | ) 21 | const ( 22 | languageEn = "en" 23 | languageJa = "ja" 24 | languageZhCn = "zh-CN" 25 | ) 26 | 27 | type MostVisitedLocationsHandler struct { 28 | limitSecond int 29 | geoLite2Db *geoip2.Reader 30 | ipLocationCache cache.Cache 31 | // country -> count 32 | countryCountMap map[string]int 33 | // country -> city -> count 34 | countryCityCountMap map[string]map[string]int 35 | // country -> city -> ip -> count 36 | countryCityIpCountMap map[string]map[string]map[string]int 37 | } 38 | 39 | type locationEntry struct { 40 | country string 41 | city string 42 | } 43 | 44 | func NewMostVisitedLocationsHandler(dbFile string, limitSecond int) *MostVisitedLocationsHandler { 45 | db, err := geoip2.Open(dbFile) 46 | if err != nil { 47 | ioutil.Fatal("open MaxMind-DB error: %v\n", err.Error()) 48 | return nil 49 | } 50 | return &MostVisitedLocationsHandler{ 51 | limitSecond: limitSecond, 52 | geoLite2Db: db, 53 | ipLocationCache: cache.NewLruCache(1000), 54 | countryCountMap: make(map[string]int), 55 | countryCityCountMap: make(map[string]map[string]int), 56 | countryCityIpCountMap: make(map[string]map[string]map[string]int), 57 | } 58 | } 59 | 60 | func (handler *MostVisitedLocationsHandler) Input(info *parser.LogInfo) { 61 | country, city := handler.queryIpLocation(info.RemoteAddr) 62 | 63 | // save or update by country 64 | if _, ok := handler.countryCityIpCountMap[country]; !ok { 65 | handler.countryCountMap[country] = 1 66 | handler.countryCityCountMap[country] = make(map[string]int) 67 | handler.countryCityIpCountMap[country] = make(map[string]map[string]int) 68 | } else { 69 | handler.countryCountMap[country]++ 70 | } 71 | 72 | // save or update by city 73 | if _, ok := handler.countryCityIpCountMap[country][city]; !ok { 74 | handler.countryCityCountMap[country][city] = 1 75 | handler.countryCityIpCountMap[country][city] = make(map[string]int) 76 | } else { 77 | handler.countryCityCountMap[country][city]++ 78 | } 79 | 80 | // save or update by ip address 81 | if _, ok := handler.countryCityIpCountMap[country][city][info.RemoteAddr]; !ok { 82 | handler.countryCityIpCountMap[country][city][info.RemoteAddr] = 1 83 | } else { 84 | handler.countryCityIpCountMap[country][city][info.RemoteAddr]++ 85 | } 86 | } 87 | 88 | func (handler *MostVisitedLocationsHandler) Output(limit int) { 89 | defer handler.geoLite2Db.Close() 90 | 91 | countryCountKeys := make([]string, 0, len(handler.countryCityIpCountMap)) 92 | for k := range handler.countryCityIpCountMap { 93 | countryCountKeys = append(countryCountKeys, k) 94 | } 95 | sort.Slice(countryCountKeys, func(i, j int) bool { 96 | return handler.countryCountMap[countryCountKeys[i]] > handler.countryCountMap[countryCountKeys[j]] 97 | }) 98 | 99 | for i := 0; i < len(countryCountKeys); i++ { 100 | country := countryCountKeys[i] 101 | cityIpCountMap := handler.countryCityIpCountMap[country] 102 | fmt.Printf("[%v] hits: %v\n", country, handler.countryCountMap[country]) 103 | 104 | cityCountKeys := make([]string, 0, len(cityIpCountMap)) 105 | for k := range cityIpCountMap { 106 | cityCountKeys = append(cityCountKeys, k) 107 | } 108 | sort.Slice(cityCountKeys, func(i, j int) bool { 109 | return handler.countryCityCountMap[country][cityCountKeys[i]] > handler.countryCityCountMap[country][cityCountKeys[j]] 110 | }) 111 | 112 | for j := 0; j < handler.limitSecond && j < len(cityCountKeys); j++ { 113 | city := cityCountKeys[j] 114 | ipCountMap := cityIpCountMap[city] 115 | fmt.Printf(" |--[%v] hits: %v\n", city, handler.countryCityCountMap[country][city]) 116 | 117 | ipCountKeys := make([]string, 0, len(ipCountMap)) 118 | for k := range ipCountMap { 119 | ipCountKeys = append(ipCountKeys, k) 120 | } 121 | sort.Slice(ipCountKeys, func(i, j int) bool { 122 | return ipCountMap[ipCountKeys[i]] > ipCountMap[ipCountKeys[j]] 123 | }) 124 | 125 | for k := 0; k < limit && k < len(ipCountKeys); k++ { 126 | ip := ipCountKeys[k] 127 | fmt.Printf(" | |--\"%v\" hits: %v\n", ip, ipCountMap[ip]) 128 | } 129 | } 130 | } 131 | } 132 | 133 | func (handler *MostVisitedLocationsHandler) queryIpLocation(ip string) (string, string) { 134 | record, err := handler.geoLite2Db.City(net.ParseIP(ip)) 135 | if record == nil { 136 | ioutil.Fatal("query from MaxMind-DB error: record is nil\n") 137 | return "", "" 138 | } 139 | if err != nil { 140 | ioutil.Fatal("query from MaxMind-DB error: %v\n", err.Error()) 141 | return "", "" 142 | } 143 | 144 | country := record.Country.Names[languageEn] 145 | city := record.City.Names[languageEn] 146 | if city == "" { 147 | city = cityUnknown 148 | } 149 | 150 | if strings.EqualFold(countryChina, country) || strings.EqualFold(areaHongKong, country) || 151 | strings.EqualFold(areaTaiwan, country) { 152 | country = fmt.Sprintf("%s %s", record.Country.Names[languageZhCn], country) 153 | if city != cityUnknown && record.City.Names[languageZhCn] != "" { 154 | city = fmt.Sprintf("%s %s", record.City.Names[languageZhCn], city) 155 | } 156 | } else if strings.EqualFold(countryJapan, country) { 157 | country = fmt.Sprintf("%s %s", record.Country.Names[languageJa], country) 158 | if city != cityUnknown && record.City.Names[languageJa] != "" { 159 | city = fmt.Sprintf("%s %s", record.City.Names[languageJa], city) 160 | } 161 | } 162 | return country, city 163 | } 164 | 165 | func (handler *MostVisitedLocationsHandler) cachedQueryIpLocation(ip string) (string, string) { 166 | data := handler.ipLocationCache.Get(ip) 167 | if data != nil { 168 | return data.(*locationEntry).country, data.(*locationEntry).city 169 | } else { // cache missed 170 | country, city := handler.queryIpLocation(ip) 171 | handler.ipLocationCache.Put(ip, &locationEntry{country: country, city: city}) 172 | return country, city 173 | } 174 | } 175 | -------------------------------------------------------------------------------- /handler/pv_uv.go: -------------------------------------------------------------------------------- 1 | package handler 2 | 3 | import ( 4 | "fmt" 5 | "github.com/fantasticmao/nginx-log-analyzer/parser" 6 | ) 7 | 8 | type PvAndUvHandler struct { 9 | pv int32 10 | uv int32 11 | uniqMap map[string]bool 12 | } 13 | 14 | func NewPvAndUvHandler() *PvAndUvHandler { 15 | return &PvAndUvHandler{ 16 | pv: 0, 17 | uv: 0, 18 | uniqMap: make(map[string]bool), 19 | } 20 | } 21 | 22 | func (handler *PvAndUvHandler) Input(info *parser.LogInfo) { 23 | handler.pv++ 24 | if _, ok := handler.uniqMap[info.RemoteAddr]; !ok { 25 | handler.uv++ 26 | handler.uniqMap[info.RemoteAddr] = true 27 | } 28 | } 29 | 30 | func (handler *PvAndUvHandler) Output(limit int) { 31 | fmt.Printf("PV: %v\n", handler.pv) 32 | fmt.Printf("UV: %v\n", handler.uv) 33 | } 34 | -------------------------------------------------------------------------------- /ioutil/example_test.go: -------------------------------------------------------------------------------- 1 | package ioutil 2 | 3 | import "os" 4 | 5 | func ExampleFatal() { 6 | fatal(os.Stdout, func(i int) {}, "log: %v\n", "Hello, World!") 7 | // Output: 8 | // log: Hello, World! 9 | } 10 | -------------------------------------------------------------------------------- /ioutil/files.go: -------------------------------------------------------------------------------- 1 | package ioutil 2 | 3 | import ( 4 | "bufio" 5 | "compress/gzip" 6 | "os" 7 | "path/filepath" 8 | "strings" 9 | ) 10 | 11 | func OpenFile(path string) (*os.File, bool) { 12 | file, err := os.Open(path) 13 | if err != nil { 14 | Fatal("open file error: %v\n", err.Error()) 15 | return nil, false 16 | } 17 | 18 | ext := filepath.Ext(file.Name()) 19 | return file, strings.EqualFold(".gz", ext) 20 | } 21 | 22 | func ReadFile(file *os.File, isGzip bool) *bufio.Reader { 23 | if isGzip { 24 | gzipReader, err := gzip.NewReader(file) 25 | if err != nil { 26 | Fatal("gzip new reader error: %v\n", err.Error()) 27 | return nil 28 | } 29 | return bufio.NewReader(gzipReader) 30 | } else { 31 | return bufio.NewReader(file) 32 | } 33 | } 34 | -------------------------------------------------------------------------------- /ioutil/files_test.go: -------------------------------------------------------------------------------- 1 | package ioutil 2 | 3 | import ( 4 | "github.com/stretchr/testify/assert" 5 | "testing" 6 | ) 7 | 8 | func TestOpenFile(t *testing.T) { 9 | file, isGzip := OpenFile("../testdata/access.log") 10 | assert.NotNil(t, file) 11 | assert.False(t, isGzip) 12 | 13 | file, isGzip = OpenFile("../testdata/access.json.log") 14 | assert.NotNil(t, file) 15 | assert.False(t, isGzip) 16 | 17 | file, isGzip = OpenFile("../testdata/access.json.log.1.gz") 18 | assert.NotNil(t, file) 19 | assert.True(t, isGzip) 20 | } 21 | 22 | func TestReadFile(t *testing.T) { 23 | file, isGzip := OpenFile("../testdata/access.log") 24 | reader := ReadFile(file, isGzip) 25 | assert.NotNil(t, reader) 26 | 27 | file, isGzip = OpenFile("../testdata/access.json.log") 28 | reader = ReadFile(file, isGzip) 29 | assert.NotNil(t, reader) 30 | 31 | file, isGzip = OpenFile("../testdata/access.json.log.1.gz") 32 | reader = ReadFile(file, isGzip) 33 | assert.NotNil(t, reader) 34 | } 35 | -------------------------------------------------------------------------------- /ioutil/log.go: -------------------------------------------------------------------------------- 1 | package ioutil 2 | 3 | import ( 4 | "fmt" 5 | "io" 6 | "os" 7 | ) 8 | 9 | func Fatal(format string, a ...interface{}) { 10 | fatal(os.Stderr, os.Exit, format, a...) 11 | } 12 | 13 | func fatal(w io.Writer, exit func(int), format string, a ...interface{}) { 14 | _, _ = fmt.Fprintf(w, format, a...) 15 | exit(1) 16 | } 17 | -------------------------------------------------------------------------------- /main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "flag" 5 | "fmt" 6 | "github.com/fantasticmao/nginx-log-analyzer/handler" 7 | "github.com/fantasticmao/nginx-log-analyzer/ioutil" 8 | "github.com/fantasticmao/nginx-log-analyzer/parser" 9 | "io" 10 | "os" 11 | "path" 12 | "time" 13 | ) 14 | 15 | var ( 16 | logFiles []string 17 | showVersion bool 18 | configDir string 19 | analysisType int 20 | limit int 21 | limitSecond int 22 | percentile float64 23 | timeAfter string 24 | timeBefore string 25 | logFormat string 26 | ) 27 | 28 | var ( 29 | Name = "nginx-log-analyzer" 30 | Version string 31 | BuildTime string 32 | CommitHash string 33 | ) 34 | 35 | func init() { 36 | flag.BoolVar(&showVersion, "v", false, "show current version") 37 | flag.StringVar(&configDir, "d", "", "specify the configuration directory") 38 | flag.IntVar(&analysisType, "t", 0, "specify the analysis type, see documentation for more details:\nhttps://github.com/fantasticmao/nginx-log-analyzer#specify-the-analysis-type--t") 39 | flag.IntVar(&limit, "n", 15, "limit the output lines number") 40 | flag.IntVar(&limitSecond, "n2", 15, "limit the secondary output lines number in '-t 4' mode") 41 | flag.Float64Var(&percentile, "p", 95, "specify the percentile value in '-t 7' mode") 42 | flag.StringVar(&timeAfter, "ta", "", "limit the analysis start time, in format of RFC3339 e.g. '2021-11-01T00:00:00+08:00'") 43 | flag.StringVar(&timeBefore, "tb", "", "limit the analysis end time, in format of RFC3339 e.g. '2021-11-02T00:00:00+08:00'") 44 | flag.StringVar(&logFormat, "lf", "combined", "specify the nginx log format, value should be 'combined' or 'json'") 45 | flag.Parse() 46 | logFiles = flag.Args() 47 | } 48 | 49 | func main() { 50 | if showVersion { 51 | fmt.Printf("%v %v build at %v on commit %v\n", Name, Version, BuildTime, CommitHash) 52 | return 53 | } 54 | 55 | if configDir == "" { 56 | homeDir, err := os.UserHomeDir() 57 | if err != nil { 58 | ioutil.Fatal("get user home directory error: %v\n", err.Error()) 59 | return 60 | } 61 | configDir = path.Join(homeDir, ".config", Name) 62 | } 63 | 64 | var ( 65 | since, util time.Time 66 | err error 67 | ) 68 | if timeAfter != "" { 69 | since, err = time.Parse(time.RFC3339, timeAfter) 70 | if err != nil { 71 | ioutil.Fatal("parse start time error: %v\n", err.Error()) 72 | return 73 | } 74 | } 75 | if timeBefore != "" { 76 | util, err = time.Parse(time.RFC3339, timeBefore) 77 | if err != nil { 78 | ioutil.Fatal("parse end time error: %v\n", err.Error()) 79 | return 80 | } 81 | } 82 | 83 | p := newLogParser() 84 | h := newLogHandler() 85 | process(logFiles, p, h, since, util) 86 | } 87 | 88 | func newLogHandler() handler.Handler { 89 | switch analysisType { 90 | case handler.AnalysisTypePvAndUv: 91 | return handler.NewPvAndUvHandler() 92 | case handler.AnalysisTypeVisitedIps: 93 | return handler.NewMostVisitedFieldsHandler(analysisType) 94 | case handler.AnalysisTypeVisitedUris: 95 | return handler.NewMostVisitedFieldsHandler(analysisType) 96 | case handler.AnalysisTypeVisitedUserAgents: 97 | return handler.NewMostVisitedFieldsHandler(analysisType) 98 | case handler.AnalysisTypeVisitedLocations: 99 | const dbFile = "City.mmdb" 100 | return handler.NewMostVisitedLocationsHandler(path.Join(configDir, dbFile), limitSecond) 101 | case handler.AnalysisTypeResponseStatus: 102 | return handler.NewMostFrequentStatusHandler() 103 | case handler.AnalysisTypeAverageTimeUris: 104 | return handler.NewLargestAverageTimeUrisHandler() 105 | case handler.AnalysisTypePercentTimeUris: 106 | return handler.NewLargestPercentTimeUrisHandler(percentile) 107 | default: 108 | ioutil.Fatal("unsupported analysis type: %v\n", analysisType) 109 | return nil 110 | } 111 | } 112 | 113 | func newLogParser() parser.Parser { 114 | switch logFormat { 115 | case parser.LogFormatTypeCombined: 116 | return parser.NewCombinedParser() 117 | case parser.LogFormatTypeJson: 118 | return parser.NewJsonParser() 119 | default: 120 | ioutil.Fatal("unsupported log format : %v\n", logFormat) 121 | return nil 122 | } 123 | } 124 | 125 | func process(logFiles []string, p parser.Parser, h handler.Handler, since, util time.Time) { 126 | for _, logFile := range logFiles { 127 | // 1. open and read file 128 | file, isGzip := ioutil.OpenFile(logFile) 129 | reader := ioutil.ReadFile(file, isGzip) 130 | for { 131 | data, err := reader.ReadBytes('\n') 132 | if err == io.EOF { 133 | break 134 | } else if err != nil { 135 | ioutil.Fatal("read file error: %v\n", err.Error()) 136 | return 137 | } 138 | 139 | // 2. parse line 140 | logInfo := p.ParseLog(data) 141 | 142 | // 3. datetime filter 143 | if !since.IsZero() || !util.IsZero() { 144 | logTime := parser.ParseTime(logInfo.TimeLocal) 145 | if !since.IsZero() && logTime.Before(since) { 146 | // go to next line 147 | continue 148 | } 149 | if !util.IsZero() && logTime.After(util) { 150 | // go to next file 151 | break 152 | } 153 | } 154 | 155 | // 4. process data 156 | h.Input(logInfo) 157 | } 158 | 159 | // 5. close file handler 160 | err := file.Close() 161 | if err != nil { 162 | ioutil.Fatal("close file error: %v\n", err.Error()) 163 | return 164 | } 165 | } 166 | 167 | // 5. print result 168 | h.Output(limit) 169 | } 170 | -------------------------------------------------------------------------------- /parser/benchmark_test.go: -------------------------------------------------------------------------------- 1 | package parser 2 | 3 | import "testing" 4 | 5 | func BenchmarkJsonParser(b *testing.B) { 6 | p := NewJsonParser() 7 | for i := 0; i < b.N; i++ { 8 | p.ParseLog(jsonLog) 9 | } 10 | } 11 | 12 | func BenchmarkCombinedParser(b *testing.B) { 13 | p := NewCombinedParser() 14 | for i := 0; i < b.N; i++ { 15 | p.ParseLog(combinedLog) 16 | } 17 | } 18 | -------------------------------------------------------------------------------- /parser/log_info.go: -------------------------------------------------------------------------------- 1 | package parser 2 | 3 | type LogInfo struct { 4 | RemoteAddr string `json:"remote_addr"` 5 | RemoteUser string `json:"remote_user"` 6 | TimeLocal string `json:"time_local"` 7 | Request string `json:"request"` 8 | Status int `json:"status"` 9 | BodyBytesSent int `json:"body_bytes_sent"` 10 | HttpReferer string `json:"http_referer"` 11 | HttpUserAgent string `json:"http_user_agent"` 12 | RequestTime float64 `json:"request_time"` 13 | } 14 | -------------------------------------------------------------------------------- /parser/parser.go: -------------------------------------------------------------------------------- 1 | package parser 2 | 3 | import ( 4 | "bytes" 5 | "encoding/json" 6 | "github.com/fantasticmao/nginx-log-analyzer/ioutil" 7 | "strconv" 8 | "time" 9 | ) 10 | 11 | const ( 12 | LogFormatTypeCombined = "combined" 13 | LogFormatTypeJson = "json" 14 | ) 15 | 16 | func ParseTime(timeLocal string) time.Time { 17 | t, err := time.Parse("02/Jan/2006:15:04:05 -0700", timeLocal) 18 | if err != nil { 19 | ioutil.Fatal("parse log time error: %v\n", err.Error()) 20 | } 21 | return t 22 | } 23 | 24 | type Parser interface { 25 | ParseLog(line []byte) *LogInfo 26 | } 27 | 28 | type JsonParser struct { 29 | } 30 | 31 | func NewJsonParser() *JsonParser { 32 | return &JsonParser{} 33 | } 34 | 35 | func (parser *JsonParser) ParseLog(line []byte) *LogInfo { 36 | logInfo := &LogInfo{} 37 | err := json.Unmarshal(line[:len(line)-1], logInfo) 38 | if err != nil { 39 | ioutil.Fatal("parse json log error: %v\n", err.Error()) 40 | return nil 41 | } 42 | return logInfo 43 | } 44 | 45 | type CombinedParser struct { 46 | delimiters [][]byte 47 | } 48 | 49 | func NewCombinedParser() *CombinedParser { 50 | // 0 1 2 3 4 5 6 7 51 | // log_format combined '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'; 52 | // | | | | | | | | 53 | // ' - ' ' [' '] "' '" ' ' ' ' "' '" "' '"\n' 54 | var delimiters = [][]byte{ 55 | []byte(" - "), []byte(" ["), []byte("] \""), []byte("\" "), 56 | []byte(" "), []byte(" \""), []byte("\" \""), []byte("\"\n"), 57 | } 58 | return &CombinedParser{ 59 | delimiters: delimiters, 60 | } 61 | } 62 | 63 | func (parser *CombinedParser) ParseLog(line []byte) *LogInfo { 64 | var ( 65 | variables = make([]string, 0, 8) 66 | i = 0 // variable start index 67 | j = 0 // variable end index 68 | k = 0 // delimiters and variables index 69 | ) 70 | for k < len(parser.delimiters) && j <= len(line)-len(parser.delimiters[k]) { 71 | if bytes.Equal(line[j:j+len(parser.delimiters[k])], parser.delimiters[k]) { 72 | variables = append(variables, string(line[i:j])) 73 | j = j + len(parser.delimiters[k]) 74 | i = j 75 | k++ 76 | } else { 77 | j++ 78 | } 79 | } 80 | if k != len(parser.delimiters) { 81 | ioutil.Fatal("parse combined log error: %v\n", string(line)) 82 | } 83 | status, err := strconv.Atoi(variables[4]) 84 | if err != nil { 85 | ioutil.Fatal("convert $status to int error: %v\n", variables[4]) 86 | } 87 | bodyBytesSent, err := strconv.Atoi(variables[5]) 88 | if err != nil { 89 | ioutil.Fatal("convert $body_bytes_sent to int error: %v\n", variables[5]) 90 | } 91 | return &LogInfo{ 92 | RemoteAddr: variables[0], 93 | RemoteUser: variables[1], 94 | TimeLocal: variables[2], 95 | Request: variables[3], 96 | Status: status, 97 | BodyBytesSent: bodyBytesSent, 98 | HttpReferer: variables[6], 99 | HttpUserAgent: variables[7], 100 | } 101 | } 102 | -------------------------------------------------------------------------------- /parser/parser_test.go: -------------------------------------------------------------------------------- 1 | package parser 2 | 3 | import ( 4 | "github.com/stretchr/testify/assert" 5 | "testing" 6 | ) 7 | 8 | var ( 9 | jsonLog = []byte("{\"remote_addr\":\"66.102.6.200\",\"time_local\":\"15/Nov/2021:13:44:10 +0800\",\"request\":\"GET / HTTP/1.1\",\"status\":200,\"body_bytes_sent\":1603,\"http_user_agent\":\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon\",\"request_time\":0.20}\n") 10 | combinedLog = []byte("66.102.6.200 - - [15/Nov/2021:13:44:10 +0800] \"GET / HTTP/1.1\" 200 1603 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon\"\n") 11 | ) 12 | 13 | func TestParseTime(t *testing.T) { 14 | datetime := ParseTime("01/Nov/2021:00:00:00 +0800") 15 | assert.NotNil(t, datetime) 16 | assert.Equal(t, int64(1635696000000), datetime.UnixMilli()) 17 | } 18 | 19 | func TestParseLogJson(t *testing.T) { 20 | logInfo := NewJsonParser().ParseLog(jsonLog) 21 | assert.NotNil(t, logInfo) 22 | assert.Equal(t, "66.102.6.200", logInfo.RemoteAddr) 23 | assert.Equal(t, "", logInfo.RemoteUser) 24 | assert.Equal(t, "15/Nov/2021:13:44:10 +0800", logInfo.TimeLocal) 25 | assert.Equal(t, "GET / HTTP/1.1", logInfo.Request) 26 | assert.Equal(t, 200, logInfo.Status) 27 | assert.Equal(t, 1603, logInfo.BodyBytesSent) 28 | assert.Equal(t, "", logInfo.HttpReferer) 29 | assert.Equal(t, "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon", logInfo.HttpUserAgent) 30 | assert.Equal(t, 0.20, logInfo.RequestTime) 31 | } 32 | 33 | func TestParseLogCombined(t *testing.T) { 34 | logInfo := NewCombinedParser().ParseLog(combinedLog) 35 | assert.NotNil(t, logInfo) 36 | assert.Equal(t, "66.102.6.200", logInfo.RemoteAddr) 37 | assert.Equal(t, "-", logInfo.RemoteUser) 38 | assert.Equal(t, "15/Nov/2021:13:44:10 +0800", logInfo.TimeLocal) 39 | assert.Equal(t, "GET / HTTP/1.1", logInfo.Request) 40 | assert.Equal(t, 200, logInfo.Status) 41 | assert.Equal(t, 1603, logInfo.BodyBytesSent) 42 | assert.Equal(t, "-", logInfo.HttpReferer) 43 | assert.Equal(t, "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon", logInfo.HttpUserAgent) 44 | } 45 | -------------------------------------------------------------------------------- /testdata/GeoLite2-City-Test.mmdb: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fantasticmao/nginx-log-analyzer/0f5422a12ca7c88f6791089d14e7afc13fc2d3b6/testdata/GeoLite2-City-Test.mmdb -------------------------------------------------------------------------------- /testdata/access.json.log: -------------------------------------------------------------------------------- 1 | {"time_local":"01/Nov/2021:00:00:01 +0800","remote_addr":"192.168.1.1","request_time":0.010,"request":"GET /name/Tom HTTP/2.0","status":200,"body_bytes_sent":100,"http_user_agent":"iOS"} 2 | {"time_local":"01/Nov/2021:00:00:02 +0800","remote_addr":"192.168.1.1","request_time":0.020,"request":"GET /name/Tom HTTP/2.0","status":200,"body_bytes_sent":100,"http_user_agent":"iOS"} 3 | {"time_local":"01/Nov/2021:00:00:03 +0800","remote_addr":"192.168.1.1","request_time":0.030,"request":"GET /name/Tom HTTP/2.0","status":200,"body_bytes_sent":100,"http_user_agent":"iOS"} 4 | {"time_local":"01/Nov/2021:00:00:10 +0800","remote_addr":"192.168.1.2","request_time":0.010,"request":"GET /name/Sam HTTP/2.0","status":302,"body_bytes_sent":100,"http_user_agent":"Android"} 5 | {"time_local":"01/Nov/2021:00:00:11 +0800","remote_addr":"192.168.1.2","request_time":0.020,"request":"GET /name/Sam HTTP/2.0","status":200,"body_bytes_sent":100,"http_user_agent":"Android"} 6 | {"time_local":"01/Nov/2021:00:00:20 +0800","remote_addr":"192.168.1.3","request_time":0.040,"request":"GET /name/Bob HTTP/2.0","status":200,"body_bytes_sent":100,"http_user_agent":"Windows"} 7 | {"time_local":"01/Nov/2021:00:00:21 +0800","remote_addr":"192.168.1.3","request_time":0.050,"request":"GET /name/Bob HTTP/2.0","status":404,"body_bytes_sent":100,"http_user_agent":"Windows"} 8 | {"time_local":"01/Nov/2021:00:00:22 +0800","remote_addr":"192.168.1.3","request_time":0.060,"request":"GET /name/Bob HTTP/2.0","status":200,"body_bytes_sent":100,"http_user_agent":"Windows"} 9 | -------------------------------------------------------------------------------- /testdata/access.json.log.1.gz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/fantasticmao/nginx-log-analyzer/0f5422a12ca7c88f6791089d14e7afc13fc2d3b6/testdata/access.json.log.1.gz -------------------------------------------------------------------------------- /testdata/access.log: -------------------------------------------------------------------------------- 1 | 192.168.1.1 - - [01/Nov/2021:00:00:00 +0800] "GET /name/Tom HTTP/2.0" 200 100 "https://www.google.com" "iOS" 2 | 192.168.1.1 - - [01/Nov/2021:00:00:01 +0800] "GET /name/Tom HTTP/2.0" 200 100 "https://www.google.com" "iOS" 3 | 192.168.1.1 - - [01/Nov/2021:00:00:02 +0800] "GET /name/Tom HTTP/2.0" 200 100 "https://www.google.com" "iOS" 4 | 192.168.1.2 - - [01/Nov/2021:00:00:10 +0800] "GET /name/Sam HTTP/2.0" 200 100 "https://www.bing.com" "Android" 5 | 192.168.1.2 - - [01/Nov/2021:00:00:11 +0800] "GET /name/Sam HTTP/2.0" 200 100 "https://www.bing.com" "Android" 6 | 192.168.1.3 - - [01/Nov/2021:00:00:20 +0800] "GET /name/Bob HTTP/2.0" 200 100 "https://www.google.com" "Windows" 7 | 192.168.1.3 - - [01/Nov/2021:00:00:21 +0800] "GET /name/Bob HTTP/2.0" 404 100 "https://www.google.com" "Windows" 8 | 192.168.1.3 - - [01/Nov/2021:00:00:22 +0800] "GET /name/Bob HTTP/2.0" 200 100 "https://www.google.com" "Windows" 9 | --------------------------------------------------------------------------------