├── .gitignore ├── LICENSE ├── README.md ├── build.sh ├── client ├── Dockerfile ├── main.go └── run.sh ├── config.ini ├── glenda_space_medium.jpg ├── httopd ├── Dockerfile ├── cli.go ├── draw.go ├── httopd ├── main.go ├── parse.go ├── stats.go └── watch.go ├── server ├── Dockerfile ├── app.py ├── requirements.txt └── run.sh ├── start.sh └── stop.sh /.gitignore: -------------------------------------------------------------------------------- 1 | logs 2 | httopd/httopd 3 | loglist.txt 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2014 Tony Worm 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy of 6 | this software and associated documentation files (the "Software"), to deal in 7 | the Software without restriction, including without limitation the rights to 8 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of 9 | the Software, and to permit persons to whom the Software is furnished to do so, 10 | subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS 17 | FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR 18 | COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER 19 | IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN 20 | CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | httopd - top for httpd logs 2 | ==================================== 3 | 4 | pronounced "hopped" like a gopher 5 | 6 | ![boing!](https://raw.github.com/verdverm/httopd/master/glenda_space_medium.jpg) 7 | 8 | 9 | Dependencies 10 | ------------ 11 | 12 | 1. Docker 13 | 2. *nix 14 | 15 | Installation 16 | ------------- 17 | `go get github.com/verdverm/httopd/httopd` 18 | 19 | Running 20 | ----------- 21 | 22 | watching a single log file 23 | `httopd -fn="/abs/path/to/log/file"` 24 | 25 | watching a list of log files 26 | `httopd -fnList="path/to/list/file"` 27 | 28 | list file format 29 | ``` 30 | /abs/path/to/log/access.log 31 | /abs/path/to/site/log/access.log 32 | /abs/path/to/site/log/access.log 33 | ``` 34 | 35 | 36 | Simulator 37 | ------------ 38 | 39 | 1. `git clone https://github.com/verdverm/httopd && cd httopd` 40 | 2. `sudo build.sh` 41 | 3. `sudo run.sh` 42 | 43 | 44 | `sudo` is required for the docker commands (unless you run a non-sudo docker setup) 45 | 46 | Enhancements / Issues / Todo's 47 | ------------------------------ 48 | 49 | Want to help out? Add or remove items from the following list. 50 | 51 | - log file format 52 | -- multiple files / domains... how to handle? 53 | -- only handling access.log default from nginx 54 | -- what about error.log ? 55 | -- other log providers ? 56 | - monitor multiple log files when there are multiple sites / server blocks 57 | - there are race conditions on the statistics 58 | -- there is a single reader and a single writer 59 | -- shouldn't be too much of an issues since only one party reads and only one party writes 60 | - page stats only update when new line_data shows up for that page 61 | - add history and alerts to errors 62 | - backfill with 10 minutes of history on startup 63 | - sort by columns 64 | - config file or directory inspection for log files / log directories 65 | - ML triggers & alerts 66 | - more configurable triggers / set from CLI / save to file? 67 | - better log line parser 68 | -- simpler and more flexible 69 | -- only tested with nginx, need to check apache and others 70 | - stats should be kept in something like an R data frame 71 | -- so that aggregates can be calculated more easily 72 | -- can rely on an external library for stats calculations 73 | - there may be some errors now resulting from multiple log files 74 | -- multiple watchers are firing linedata to the stats over channels 75 | -- if they arrive in a 'bad' order (reverse alternations over a minute boundary) 76 | -- we will switch bins, but the next arrival (last minute) will end up in the current bin, not the last one 77 | -- should really determine the bin by the time stamp, and then index to it 78 | -- right now we are just adding to the current bin 79 | -- this will be an issue with backfilling, which once resolved, we can backfile file by file, and not worry about time 80 | 81 | Subdir Details 82 | ------------ 83 | 84 | #### server 85 | 86 | This docker contains a Python-Flask site. 87 | 88 | #### client 89 | 90 | This docker contains a http-client simulator. 91 | 92 | #### httpod 93 | 94 | This folder contains the httpod program code. 95 | 96 | #### logs 97 | 98 | a temporary directory created and destroyed by the simulator 99 | 100 | References 101 | --------------- 102 | 103 | 1. [Logging Control In W3C httpd - Logfile Format](http://www.w3.org/Daemon/User/Config/Logging.html#common-logfile-format) 104 | 2. [termbox-go](https://github.com/nsf/termbox-go) 105 | -------------------------------------------------------------------------------- /build.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e 4 | 5 | # printf "\n\n\nbuilding server\n---------------------------\n" 6 | # cd server 7 | # docker build -t verdverm/httopd-server . 8 | # cd .. 9 | 10 | printf "\n\n\nbuilding client\n---------------------------\n" 11 | cd client 12 | docker build -t verdverm/httopd-client . 13 | cd .. 14 | 15 | # printf "\n\n\nbuilding monitor\n---------------------------\n" 16 | # cd monitor 17 | # docker build -t verdverm/httopd-monitor . 18 | # cd .. 19 | -------------------------------------------------------------------------------- /client/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM google/golang:latest 2 | MAINTAINER verdverm@gmail.com 3 | 4 | ADD . /gopath 5 | WORKDIR /gopath 6 | 7 | CMD /gopath/run.sh 8 | -------------------------------------------------------------------------------- /client/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "flag" 5 | "fmt" 6 | "io/ioutil" 7 | "math" 8 | "math/rand" 9 | "net/http" 10 | "time" 11 | ) 12 | 13 | var ( 14 | numClient = flag.Int("c", 16, "number of client routines") 15 | meanReqTime = flag.Float64("mean", 2.0, "mean time between request per client") 16 | sdevReqTime = flag.Float64("sdev", 1.0, "mean time between request per client") 17 | hostStr = flag.String("host", "localhost:8080", "host:port for request") 18 | mean, sdev float64 19 | ) 20 | 21 | func main() { 22 | flag.Parse() 23 | rand.Seed(time.Now().UnixNano()) 24 | 25 | mean, sdev = *meanReqTime, *sdevReqTime 26 | 27 | fmt.Println("http client simulator", *hostStr) 28 | 29 | for i := 1; i < *numClient; i++ { 30 | go run(i) 31 | } 32 | run(0) 33 | } 34 | 35 | func run(id int) { 36 | for { 37 | url := genUrl() 38 | fmt.Println(id, url) 39 | getUrl(url) 40 | sleep() 41 | } 42 | } 43 | 44 | func genUrl() string { 45 | // generate random number in the range [1:5] 46 | // where 5 has half the probability of the other values 47 | // (5 is a 404 page) 48 | r := rand.Int() 49 | i := ((r % 9) + 2) / 2 50 | url := "http://" + *hostStr + "/page" + fmt.Sprintf("%d", i) 51 | // add extra level on page 52 | if r%2 == 0 { 53 | url += "/asubpage" 54 | } 55 | if r%3 == 0 { 56 | url += "/asubsubpage" 57 | } 58 | return url 59 | } 60 | 61 | func getUrl(url string) { 62 | resp, err := http.Get(url) 63 | if err != nil { 64 | fmt.Println("Get Error: ", err) 65 | } 66 | defer resp.Body.Close() 67 | _, err = ioutil.ReadAll(resp.Body) 68 | if err != nil { 69 | fmt.Println("ReadAll Error: ", err) 70 | } 71 | } 72 | 73 | func sleep() { 74 | f := rand.NormFloat64()*sdev + mean 75 | i := int(math.Floor(f + 0.5)) 76 | dur := time.Second * time.Duration(i) 77 | time.Sleep(dur) 78 | } 79 | -------------------------------------------------------------------------------- /client/run.sh: -------------------------------------------------------------------------------- 1 | set -e 2 | 3 | # docker related... to get the gateway IP 4 | # export HOSTURL=$(netstat -nr | grep 'UG' | awk '{print $2}') 5 | # echo "HOSTURL: $HOSTURL" 6 | # go build 7 | 8 | echo "client 8080" 9 | go run *.go -host="localhost:8080" & 10 | 11 | echo "client 8081" 12 | go run *.go -host="localhost:8081" & 13 | 14 | echo "client 8082" 15 | go run *.go -host="localhost:8082" 16 | 17 | -------------------------------------------------------------------------------- /config.ini: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/verdverm/httopd/c27315a2e2b8c5e58b0abf5360a367964512a563/config.ini -------------------------------------------------------------------------------- /glenda_space_medium.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/verdverm/httopd/c27315a2e2b8c5e58b0abf5360a367964512a563/glenda_space_medium.jpg -------------------------------------------------------------------------------- /httopd/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM google/golang:latest 2 | MAINTAINER verdverm@gmail.com 3 | 4 | VOLUME ["/logdir"] 5 | 6 | ADD . /gopath 7 | WORKDIR /gopath 8 | 9 | CMD go run main.go 10 | -------------------------------------------------------------------------------- /httopd/cli.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | // "fmt" 5 | "time" 6 | 7 | "github.com/nsf/termbox-go" 8 | ) 9 | 10 | var quit = false 11 | 12 | func startCLI() { 13 | err := termbox.Init() 14 | if err != nil { 15 | panic(err) 16 | } 17 | termbox.SetInputMode(termbox.InputEsc) 18 | redraw_all() 19 | 20 | // capture and process events from the CLI 21 | eventChan := make(chan termbox.Event, 16) 22 | go handleEvents(eventChan) 23 | go func() { 24 | for { 25 | ev := termbox.PollEvent() 26 | eventChan <- ev 27 | } 28 | }() 29 | 30 | // start update (redraw) ticker 31 | timer := time.Tick(time.Millisecond * 100) 32 | for { 33 | select { 34 | case <-timer: 35 | redraw_all() 36 | } 37 | } 38 | } 39 | 40 | const edit_box_width = 30 41 | 42 | var alertDetailsView = false 43 | var alertPage = "" 44 | 45 | func handleEvents(eventChan chan termbox.Event) { 46 | for { 47 | ev := <-eventChan 48 | switch ev.Type { 49 | case termbox.EventKey: 50 | switch ev.Key { 51 | 52 | case termbox.KeyEnter: 53 | alertDetailsView = true 54 | // FIX THIS 55 | // alertPage = fmt.Sprintf("page%d", selectedRow+1) 56 | 57 | case termbox.KeyArrowDown: 58 | // if alertDetailsView { 59 | // maxSelectedRow = len(siteStats.AlertHist[alertPage]) - 1 60 | // if siteStats.OpenAlerts[alertPage] != nil { 61 | // maxSelectedRow++ 62 | // } 63 | // } else { 64 | // maxSelectedRow = len(knownPages) - 1 65 | // } 66 | if !alertDetailsView && selectedRow < maxSelectedRow { 67 | selectedRow++ 68 | } 69 | case termbox.KeyArrowUp: 70 | if !alertDetailsView && selectedRow > 0 { 71 | selectedRow-- 72 | } 73 | case termbox.KeyHome: 74 | selectedRow = 0 75 | case termbox.KeyEnd: 76 | selectedRow = maxSelectedRow 77 | 78 | case termbox.KeyEsc: 79 | if alertDetailsView { 80 | maxSelectedRow = len(knownPages) 81 | alertDetailsView = false 82 | } else { 83 | goto endfunc 84 | } 85 | case termbox.KeyCtrlQ: 86 | goto endfunc 87 | 88 | default: 89 | if ev.Ch != 0 { 90 | // edit_box.InsertRune(ev.Ch) 91 | } 92 | } 93 | case termbox.EventError: 94 | panic(ev.Err) 95 | } 96 | } 97 | endfunc: 98 | quit = true 99 | } 100 | -------------------------------------------------------------------------------- /httopd/draw.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "strings" 6 | "time" 7 | 8 | "github.com/nsf/termbox-go" 9 | ) 10 | 11 | const coldef = termbox.ColorDefault 12 | 13 | var startTime time.Time 14 | 15 | func init() { 16 | startTime = time.Now() 17 | } 18 | 19 | var w, h int 20 | 21 | var tmpLogFn = "/home/tony/gocode/src/github.com/verdverm/httopd/logs/host.httopd-1.access.log" 22 | 23 | var knownPages = []string{ 24 | "page1", 25 | "page2", 26 | "page3", 27 | "page4", 28 | } 29 | 30 | var selectedRow = 0 31 | var colHeaderRow = 7 32 | var minSelectedRow = colHeaderRow + 1 33 | var maxSelectedRow = colHeaderRow + 1 34 | 35 | func redraw_all() { 36 | termbox.Clear(coldef, coldef) 37 | w, h = termbox.Size() 38 | 39 | drawCurrentTime(1, 0) 40 | 41 | // temporary, this is changing 42 | drawRetCodes(1, 2) 43 | drawErrStats(30, 2) 44 | 45 | // convert to 'detailsView' 46 | if alertDetailsView { 47 | // determine details to draw 48 | alertPage = "page1" 49 | ss := siteStats.Logs[tmpLogFn] 50 | 51 | drawSectionDetails(1, 7, alertPage, ss) 52 | // draw page details (more detailed stats & alert hist) 53 | // or 54 | // draw logfile details (aggregates of the page details) 55 | } else { 56 | drawColumnHeaders(1, colHeaderRow) 57 | y := colHeaderRow + 1 58 | for _, f := range siteStats.LogNames { 59 | ss := siteStats.Logs[f] 60 | drawPageStats(1, y, ss) 61 | y += len(ss.PageStats) 62 | } 63 | maxSelectedRow = y - 1 64 | } 65 | 66 | drawFooter() 67 | 68 | // midy := h / 2 69 | // midx := (w - edit_box_width) / 2 70 | 71 | termbox.HideCursor() 72 | 73 | tbprint(w-6, h-1, coldef, termbox.ColorBlue, "ʕ◔ϖ◔ʔ") 74 | termbox.Flush() 75 | } 76 | 77 | func fill(x, y, w, h int, cell termbox.Cell) { 78 | for ly := 0; ly < h; ly++ { 79 | for lx := 0; lx < w; lx++ { 80 | termbox.SetCell(x+lx, y+ly, cell.Ch, cell.Fg, cell.Bg) 81 | } 82 | } 83 | } 84 | 85 | func tbprint(x, y int, fg, bg termbox.Attribute, msg string) { 86 | for _, c := range msg { 87 | termbox.SetCell(x, y, c, fg, bg) 88 | x++ 89 | } 90 | } 91 | 92 | var knownErrs = []string{ 93 | "parse", 94 | "nildata", 95 | } 96 | 97 | func drawCurrentTime(x, y int) { 98 | now := time.Now() 99 | since := now.Sub(startTime) 100 | h := int(since.Hours()) 101 | m := int(since.Minutes()) % 60 102 | s := int(since.Seconds()) % 60 103 | timeStr := fmt.Sprintf("Now: %-24s Watching: %3d:%02d:%02d", now.Format(DATEPRINT), h, m, s) 104 | for i, c := range timeStr { 105 | termbox.SetCell(x+i, y, c, coldef, coldef) 106 | } 107 | } 108 | 109 | func drawErrStats(x, y int) { 110 | colTitle := "Error Count" 111 | for i, c := range colTitle { 112 | termbox.SetCell(x+i, y, c, coldef, coldef) 113 | } 114 | y++ 115 | 116 | for _, err := range knownErrs { 117 | errStr := fmt.Sprintf("%-8s %5d", err, siteStats.ErrStats[err]) 118 | for i, c := range errStr { 119 | termbox.SetCell(x+i, y, c, coldef, coldef) 120 | } 121 | y++ 122 | } 123 | selRow := fmt.Sprintf("%-8s %5d %5d %5d", "selRow", selectedRow, minSelectedRow, maxSelectedRow) 124 | for i, c := range selRow { 125 | termbox.SetCell(x+i, y, c, coldef, coldef) 126 | } 127 | 128 | } 129 | 130 | var knownCodes = []string{ 131 | "200", 132 | "404", 133 | } 134 | 135 | func drawRetCodes(x, y int) { 136 | // temporary, want to calc global here 137 | ss := siteStats.Logs[tmpLogFn] 138 | 139 | colTitle := "Code Count" 140 | for i, c := range colTitle { 141 | termbox.SetCell(x+i, y, c, coldef, coldef) 142 | } 143 | y++ 144 | 145 | total := 0 146 | for _, code := range knownCodes { 147 | total += ss.RetCodes[code] 148 | errStr := fmt.Sprintf("%-8s %5d", code, ss.RetCodes[code]) 149 | for i, c := range errStr { 150 | termbox.SetCell(x+i, y, c, coldef, coldef) 151 | } 152 | y++ 153 | } 154 | 155 | totalStr := fmt.Sprintf("%-8s %5d", "total", total) 156 | for i, c := range totalStr { 157 | termbox.SetCell(x+i, y, c, coldef, coldef) 158 | } 159 | } 160 | 161 | func drawColumnHeaders(x, y int) { 162 | columnHeaders := fmt.Sprintf( 163 | "%-4s %-24s %-6s %-6s %-48s", 164 | "CID", "Page", "Alerts", "Count", "Hits / min", 165 | ) 166 | 167 | for i := 0; i < w; i++ { 168 | termbox.SetCell(i, y, ' ', coldef, termbox.ColorBlue) 169 | } 170 | for i, c := range columnHeaders { 171 | termbox.SetCell(x+i, y, c, coldef, termbox.ColorBlue) 172 | } 173 | } 174 | func drawPageStats(x, y int, ss *SiteStats) { 175 | 176 | if selectedRow < minSelectedRow { 177 | selectedRow = minSelectedRow 178 | } 179 | 180 | // draw log row 181 | 182 | fg_col, bg_col := coldef, coldef 183 | if y == selectedRow { 184 | fg_col = termbox.ColorBlack 185 | bg_col = termbox.ColorYellow 186 | } 187 | 188 | // print log file name 189 | lpos := strings.LastIndex(ss.LogName, "/") + 1 190 | lfn_short := ss.LogName[lpos:] 191 | str := fmt.Sprintf("%-4d %-24s ", y, lfn_short) 192 | for i := 0; i < w; i++ { 193 | termbox.SetCell(i, y, ' ', fg_col, bg_col) 194 | } 195 | xcnt := x 196 | for _, c := range str { 197 | termbox.SetCell(xcnt, y, c, fg_col, bg_col) 198 | xcnt++ 199 | } 200 | y++ 201 | 202 | // draw page sections 203 | for _, page := range knownPages { 204 | hs := ss.PageStats[page] 205 | hist := make([]int, len(hs.HistBins)) 206 | for i := len(hs.HistBins) - 1; i >= 0; i-- { 207 | hist[i] = hs.HistBins[i].Count 208 | } 209 | 210 | xcnt := x 211 | 212 | fg_col, bg_col = coldef, coldef 213 | if y == selectedRow { 214 | fg_col = termbox.ColorBlack 215 | bg_col = termbox.ColorYellow 216 | } 217 | 218 | // print page name 219 | str := fmt.Sprintf("%-4d %-24s ", y, " "+page) 220 | for i := 0; i < w; i++ { 221 | termbox.SetCell(i, y, ' ', fg_col, bg_col) 222 | } 223 | for _, c := range str { 224 | termbox.SetCell(xcnt, y, c, fg_col, bg_col) 225 | xcnt++ 226 | } 227 | 228 | // print alerts 229 | alerts := ss.AlertHist[page] 230 | alertCount := len(alerts) 231 | alert_fg := fg_col 232 | alert_bg := bg_col 233 | if a := ss.OpenAlerts[page]; a != nil { 234 | alert_fg = termbox.ColorDefault 235 | alert_bg = termbox.ColorRed 236 | alertCount++ 237 | } 238 | str = fmt.Sprintf("%6d ", alertCount) 239 | for _, c := range str { 240 | termbox.SetCell(xcnt, y, c, alert_fg, alert_bg) 241 | xcnt++ 242 | } 243 | 244 | // print hit infomation 245 | str = fmt.Sprintf("%6d [ ", hs.Total) 246 | for i := len(hist) - 1; i >= 0; i-- { 247 | str += fmt.Sprintf("%3d ", hist[i]) 248 | } 249 | str += "]" 250 | for _, c := range str { 251 | termbox.SetCell(xcnt, y, c, fg_col, bg_col) 252 | xcnt++ 253 | } 254 | 255 | y++ 256 | } 257 | 258 | } 259 | 260 | const alertDateFormat = "01-02 15:04" 261 | 262 | func drawSectionDetails(x, y int, alertsPage string, ss *SiteStats) { 263 | 264 | // draw the details 265 | colTitle := "Type Start End Details" 266 | for i := 0; i < w; i++ { 267 | termbox.SetCell(i, y, ' ', coldef, termbox.ColorBlue) 268 | } 269 | for i, c := range colTitle { 270 | termbox.SetCell(x+i, y, c, coldef, termbox.ColorBlue) 271 | } 272 | y++ 273 | 274 | if selectedRow < 0 { 275 | selectedRow = 0 276 | } 277 | 278 | fg_col, bg_col := coldef, coldef 279 | 280 | if alert := ss.OpenAlerts[alertPage]; alert != nil { 281 | bstr, estr := alert.BeginTime.Format(alertDateFormat), " " 282 | if alert.BeginTime.Before(alert.EndTime) { 283 | estr = alert.EndTime.Format(alertDateFormat) 284 | } 285 | str := fmt.Sprintf("%-12s %-12s %-12s %s", alert.Type, bstr, estr, alert.Detail) 286 | for i, c := range str { 287 | termbox.SetCell(x+i, y, c, fg_col, bg_col) 288 | } 289 | y++ 290 | } 291 | 292 | alerts := ss.AlertHist[alertPage] 293 | for P := len(alerts) - 1; P >= 0; P-- { 294 | alert := alerts[P] 295 | hs := ss.PageStats[alertPage] 296 | hist := make([]int, len(hs.HistBins)) 297 | for i := len(hs.HistBins) - 1; i >= 0; i-- { 298 | hist[i] = hs.HistBins[i].Count 299 | } 300 | 301 | fg_col, bg_col := coldef, coldef 302 | 303 | // print alert 304 | str := fmt.Sprintf("%-12s %-12s %-12s %s", alert.Type, alert.BeginTime.Format(alertDateFormat), alert.EndTime.Format(alertDateFormat), alert.Detail) 305 | for i, c := range str { 306 | termbox.SetCell(x+i, y, c, fg_col, bg_col) 307 | } 308 | y++ 309 | } 310 | 311 | } 312 | 313 | func drawFooter() { 314 | footerText := " Esc:Back Ctrl-Q:Quit Enter:Detail" //"<_sort_> " 315 | for i := 0; i < w; i++ { 316 | termbox.SetCell(i, h-1, ' ', coldef, termbox.ColorBlue) 317 | } 318 | for i, c := range footerText { 319 | termbox.SetCell(i, h-1, c, coldef, termbox.ColorBlue) 320 | } 321 | 322 | } 323 | -------------------------------------------------------------------------------- /httopd/httopd: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/verdverm/httopd/c27315a2e2b8c5e58b0abf5360a367964512a563/httopd/httopd -------------------------------------------------------------------------------- /httopd/main.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "flag" 5 | "fmt" 6 | "io/ioutil" 7 | "runtime/debug" 8 | "time" 9 | 10 | // "github.com/go-fsnotify/fsnotify" 11 | "github.com/nsf/termbox-go" 12 | ) 13 | 14 | var ( 15 | fn = flag.String("fn", "", "log file to monitor") 16 | fnList = flag.String("fnList", "", "text file with list of log files to monitor") 17 | ) 18 | 19 | func main() { 20 | flag.Parse() 21 | 22 | defer func() { 23 | if e := recover(); e != nil { 24 | termbox.Close() 25 | trace := fmt.Sprintf("%s: %s", e, debug.Stack()) // line 20 26 | ioutil.WriteFile("trace.txt", []byte(trace), 0644) 27 | } 28 | }() 29 | 30 | // var err error 31 | // watcher, err = fsnotify.NewWatcher() 32 | // if err != nil { 33 | // panic(err) 34 | // } 35 | // defer watcher.Close() 36 | line_chan := make(chan *LineRaw, 64) 37 | data_chan := make(chan *LineData, 64) 38 | 39 | if *fnList != "" { 40 | fmt.Println("Starting watchers") 41 | go startWatcherList(*fnList, line_chan) 42 | } else if *fn != "" { 43 | fmt.Println("Starting watcher") 44 | go startWatcher(*fn, line_chan) 45 | } else { 46 | fmt.Println("must specify log file(s) to watch") 47 | return 48 | } 49 | 50 | numParsers := 1 51 | for i := 0; i < numParsers; i++ { 52 | go startParser(line_chan, data_chan) 53 | } 54 | 55 | // can only have one of these right now 56 | go startStats(data_chan) 57 | 58 | // View & Cmd loops 59 | go startCLI() // streams CLI commands to the main loop 60 | 61 | for { 62 | // select {} 63 | time.Sleep(time.Millisecond * 100) 64 | 65 | if quit == true { 66 | break 67 | } 68 | } 69 | 70 | termbox.Close() 71 | } 72 | -------------------------------------------------------------------------------- /httopd/parse.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bytes" 5 | "fmt" 6 | "strconv" 7 | "time" 8 | ) 9 | 10 | type LineRaw struct { 11 | line []byte 12 | logfile string 13 | 14 | // planned for dealing with multiple &| custom log formats 15 | // and a generalized parser which may require a more complicated stuct of it's own 16 | logfmt string // format sting info for parsing (or perhaps the ID of the parser to use) 17 | } 18 | 19 | type LineData struct { 20 | Logfile string // from the logfile `name` in LineRaw.logfile 21 | Date time.Time 22 | 23 | RequestStr string 24 | RequestMethod string 25 | SectionStr string 26 | 27 | Status string 28 | ContentLen int 29 | 30 | RemoteHost string 31 | Rfc931 string 32 | AuthUser string 33 | } 34 | 35 | const DATELAYOUT = "02/Jan/2006:15:04:05 -0700" 36 | const DATEPRINT = "Jan 02, 2006 15:04:05" 37 | 38 | func ParseLineData(raw *LineRaw) (*LineData, error) { 39 | defer func() { 40 | if e := recover(); e != nil { 41 | siteStats.ErrStats["parse"]++ 42 | } 43 | }() 44 | 45 | var err error 46 | ld := new(LineData) 47 | ld.Logfile = raw.logfile 48 | line := raw.line 49 | 50 | // find date delimiters and parse 51 | //------------------------------- 52 | lsb_pos := bytes.IndexRune(line, '[') 53 | rsb_pos := bytes.IndexRune(line, ']') 54 | date_str := string(line[lsb_pos+1 : rsb_pos]) 55 | ld.Date, err = time.Parse(DATELAYOUT, date_str) 56 | if err != nil { 57 | return nil, err 58 | } 59 | 60 | // fmt.Println(string(line)) 61 | 62 | // parse first three fields 63 | //------------------------- 64 | last_pos := 0 65 | curr_pos := bytes.IndexRune(line[last_pos:lsb_pos], ' ') + last_pos 66 | ld.RemoteHost = string(line[last_pos:curr_pos]) 67 | last_pos = curr_pos + 1 68 | 69 | curr_pos = bytes.IndexRune(line[last_pos:lsb_pos], ' ') + last_pos 70 | ld.Rfc931 = string(line[last_pos:curr_pos]) 71 | last_pos = curr_pos + 1 72 | 73 | curr_pos = bytes.IndexRune(line[last_pos:lsb_pos], ' ') + last_pos 74 | ld.AuthUser = string(line[last_pos:curr_pos]) 75 | 76 | // parse request string 77 | //----------------- 78 | lqt_pos := bytes.IndexRune(line, '"') + 1 79 | rqt_pos := bytes.IndexRune(line[lqt_pos:], '"') + lqt_pos 80 | req := line[lqt_pos:rqt_pos] 81 | ld.RequestStr = string(req) 82 | 83 | // parse RequestMethod from RequestStr 84 | rrm_pos := bytes.IndexRune(req, ' ') 85 | ld.RequestMethod = string(req[0:rrm_pos]) 86 | 87 | // parse SectionStr from RequestStr 88 | lreq_pos := bytes.IndexRune(req, '/') + 1 89 | rreq_pos := bytes.IndexAny(req[lreq_pos:], "/ ") + lreq_pos 90 | ld.SectionStr = string(req[lreq_pos:rreq_pos]) 91 | 92 | // parse status and content-length (the next two fields) 93 | last_pos = rqt_pos + 2 // skip past '"' and first ' ' 94 | curr_pos = bytes.IndexRune(line[last_pos:], ' ') + last_pos 95 | ld.Status = string(line[last_pos:curr_pos]) 96 | last_pos = curr_pos + 1 97 | curr_pos = bytes.IndexRune(line[last_pos:], ' ') + last_pos 98 | clenStr := string(line[last_pos:curr_pos]) 99 | clenInt, err := strconv.ParseInt(clenStr, 10, 64) 100 | if err != nil { 101 | return nil, err 102 | } 103 | ld.ContentLen = int(clenInt) 104 | 105 | // there are two remaining fields, which I am omitting 106 | 107 | return ld, nil 108 | } 109 | 110 | func startParser(line_chan chan *LineRaw, data_chan chan *LineData) { 111 | for { 112 | select { 113 | case line := <-line_chan: 114 | // fmt.Print(line) 115 | ld, err := ParseLineData(line) 116 | if err != nil { 117 | fmt.Println(err) 118 | } else { 119 | data_chan <- ld 120 | } 121 | } 122 | if quit == true { 123 | return 124 | } 125 | } 126 | } 127 | -------------------------------------------------------------------------------- /httopd/stats.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "fmt" 5 | "time" 6 | ) 7 | 8 | const ALERT_THRESHOLD = 35 9 | 10 | type Recorder func(*HistStats, *LineData) 11 | type Trigger func(*HistStats) bool 12 | 13 | // main "DS" 14 | type MainDS struct { 15 | LogNames []string 16 | Logs map[string]*SiteStats 17 | ErrStats map[string]int 18 | 19 | // this alerts is not in use, but is planned to move all alerts here 20 | AlertHist map[string][]*PageAlert 21 | } 22 | 23 | var siteStats MainDS 24 | 25 | func init() { 26 | siteStats.Logs = make(map[string]*SiteStats) 27 | siteStats.ErrStats = make(map[string]int) 28 | } 29 | 30 | type SiteStats struct { 31 | LogName string 32 | // use PageStats here to avoid duplication of history data stuct 33 | RetCodes map[string]int 34 | PageStats map[string]HistStats 35 | PageErrors map[string]HistStats 36 | 37 | OpenAlerts map[string]*PageAlert 38 | AlertHist map[string][]*PageAlert 39 | } 40 | 41 | // called in startWatcher 42 | func addSiteStats(logfn string) { 43 | ss := new(SiteStats) 44 | ss.LogName = logfn 45 | 46 | // not thread safe 47 | siteStats.LogNames = append(siteStats.LogNames, logfn) 48 | siteStats.Logs[logfn] = ss 49 | 50 | ss.RetCodes = make(map[string]int) 51 | 52 | ss.PageStats = make(map[string]HistStats) 53 | ss.AlertHist = make(map[string][]*PageAlert) 54 | ss.OpenAlerts = make(map[string]*PageAlert) 55 | } 56 | 57 | var ( 58 | BinSize = time.Minute // size of a bin 59 | HistoryLen = 10 // number of bins 60 | ) 61 | 62 | type HistBin struct { 63 | Start time.Time 64 | Bytes int 65 | Count int 66 | } 67 | 68 | type HistStats struct { 69 | HistBins []HistBin 70 | LastTime time.Time 71 | Total int 72 | } 73 | 74 | type PageAlert struct { 75 | Type string 76 | Detail string 77 | 78 | BeginTime time.Time 79 | EndTime time.Time 80 | } 81 | 82 | func startStats(data_chan chan *LineData) { 83 | 84 | for { 85 | select { 86 | case ld := <-data_chan: 87 | if ld == nil { 88 | siteStats.ErrStats["nildata"]++ 89 | continue 90 | } 91 | 92 | updateStats(ld) 93 | checkAlerts(ld) 94 | 95 | } 96 | } 97 | } 98 | 99 | func updateStats(ld *LineData) { 100 | code := ld.Status 101 | page := ld.SectionStr 102 | logfn := ld.Logfile 103 | siteStats.Logs[logfn].RetCodes[code]++ 104 | 105 | hs := siteStats.Logs[logfn].PageStats[page] 106 | hs.Total++ 107 | if hs.LastTime.Minute() != ld.Date.Minute() { 108 | // time for new bin 109 | if len(hs.HistBins) > 9 { 110 | hs.HistBins = hs.HistBins[1:] 111 | } 112 | d := ld.Date 113 | ldTime := time.Date(d.Year(), d.Month(), d.Day(), d.Hour(), d.Minute(), d.Minute(), 0, time.UTC) 114 | bin := HistBin{ 115 | Start: ldTime, 116 | Count: 1, 117 | } 118 | hs.HistBins = append(hs.HistBins, bin) 119 | hs.LastTime = ldTime 120 | } else { 121 | // continue with current bin 122 | l := len(hs.HistBins) - 1 123 | hs.HistBins[l].Count++ 124 | hs.HistBins[l].Bytes += ld.ContentLen 125 | } 126 | siteStats.Logs[logfn].PageStats[page] = hs 127 | } 128 | 129 | func checkAlerts(ld *LineData) { 130 | page := ld.SectionStr 131 | logfn := ld.Logfile 132 | 133 | hs := siteStats.Logs[logfn].PageStats[page] 134 | bins := hs.HistBins 135 | l := len(bins) 136 | 137 | // make sure we have enough history 138 | if l < 3 { 139 | return 140 | } 141 | 142 | // calc trigger for last two COMPLETE minutes 143 | ave := (bins[l-3].Count + bins[l-2].Count) / 2 144 | 145 | alert := siteStats.Logs[logfn].OpenAlerts[page] 146 | if alert == nil && ave >= ALERT_THRESHOLD { 147 | // check to open a new alert 148 | alert = new(PageAlert) 149 | alert.Type = "High Traffic" 150 | alert.Detail = fmt.Sprintf("hits per minute > %d", ALERT_THRESHOLD) 151 | d := ld.Date 152 | ldTime := time.Date(d.Year(), d.Month(), d.Day(), d.Hour(), d.Minute(), d.Minute(), 0, time.UTC) 153 | alert.BeginTime = ldTime 154 | siteStats.Logs[logfn].OpenAlerts[page] = alert 155 | } else if alert != nil && ave <= ALERT_THRESHOLD && 156 | alert.BeginTime.Minute() != ld.Date.Minute() { 157 | // check to close the current alert 158 | d := ld.Date 159 | ldTime := time.Date(d.Year(), d.Month(), d.Day(), d.Hour(), d.Minute(), d.Minute(), 0, time.UTC) 160 | alert.EndTime = ldTime 161 | alerts := siteStats.Logs[logfn].AlertHist[page] 162 | alerts = append(alerts, alert) 163 | siteStats.Logs[logfn].AlertHist[page] = alerts 164 | siteStats.Logs[logfn].OpenAlerts[page] = nil 165 | } 166 | 167 | } 168 | -------------------------------------------------------------------------------- /httopd/watch.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "bytes" 5 | "fmt" 6 | "io/ioutil" 7 | "os" 8 | 9 | "github.com/go-fsnotify/fsnotify" 10 | ) 11 | 12 | var watcher *fsnotify.Watcher 13 | 14 | // setup and defferd closing in main() 15 | 16 | func startWatcher(filename string, out chan *LineRaw) { 17 | // file stuff 18 | file, err := os.Open(filename) 19 | if err != nil { 20 | panic(err) 21 | } 22 | fi, err := os.Stat(filename) 23 | if err != nil { 24 | panic(err) 25 | } 26 | 27 | // heres where backfill stuff would happen 28 | last_sz := fi.Size() 29 | 30 | // go into watcher loops 31 | watcher, err = fsnotify.NewWatcher() 32 | if err != nil { 33 | panic(err) 34 | } 35 | defer watcher.Close() 36 | err = watcher.Add(filename) 37 | if err != nil { 38 | panic(err) 39 | } 40 | 41 | // main event loop 42 | for { 43 | select { 44 | case event := <-watcher.Events: 45 | if event.Op&fsnotify.Write == fsnotify.Write { 46 | // get updated size 47 | fi, err := os.Stat(filename) 48 | if err != nil { 49 | panic(err) 50 | } 51 | curr_sz := fi.Size() 52 | sz_chg := curr_sz - last_sz 53 | 54 | // make buf for reading 55 | buf := make([]byte, sz_chg) 56 | // read and check for errors 57 | n, err := file.ReadAt(buf, last_sz) 58 | if err != nil { 59 | fmt.Println("readat error:", err) 60 | } 61 | if n != int(sz_chg) { 62 | fmt.Println("n:", n, "!=", "sz_chg:", sz_chg) 63 | } 64 | 65 | // send data out 66 | out <- &LineRaw{ 67 | line: buf, 68 | logfile: filename, 69 | } 70 | 71 | // update counters 72 | last_sz = curr_sz 73 | } 74 | case err := <-watcher.Errors: 75 | fmt.Println("watch error:", err) 76 | } 77 | // check for ending 78 | if quit == true { 79 | return 80 | } 81 | } 82 | } 83 | 84 | func startWatcherList(listfile string, out chan *LineRaw) { 85 | listbytes, err := ioutil.ReadFile(listfile) 86 | if err != nil { 87 | panic(err) 88 | } 89 | lines := bytes.Fields(listbytes) 90 | for _, logfile := range lines { 91 | filename := string(logfile) 92 | fmt.Println(" watching: ", filename) 93 | addSiteStats(filename) 94 | go startWatcher(filename, out) 95 | } 96 | } 97 | -------------------------------------------------------------------------------- /server/Dockerfile: -------------------------------------------------------------------------------- 1 | # FROM google/debian:wheezy 2 | FROM dockerfile/ubuntu:latest 3 | MAINTAINER verdverm@gmail.com 4 | 5 | # Update stuff 6 | RUN apt-get update 7 | RUN apt-get upgrade -y 8 | 9 | # Install Nginx. 10 | RUN \ 11 | add-apt-repository -y ppa:nginx/stable && \ 12 | apt-get update && \ 13 | apt-get --no-install-recommends install -y nginx && \ 14 | echo "\ndaemon off;" >> /etc/nginx/nginx.conf && \ 15 | chown -R www-data:www-data /var/lib/nginx 16 | 17 | # Define mountable directories. 18 | VOLUME ["/data", "/etc/nginx/sites-enabled", "/var/log/nginx"] 19 | 20 | # Install Python Setuptools 21 | RUN apt-get --no-install-recommends install -y python-setuptools build-essential python-dev 22 | 23 | # Install pip 24 | RUN easy_install pip 25 | 26 | ADD . /httopd 27 | WORKDIR /httopd 28 | 29 | # Install requirements.txt 30 | RUN pip install -r requirements.txt 31 | 32 | # Expose ports. 33 | EXPOSE 5000 8080 34 | 35 | CMD /httopd/run.sh 36 | -------------------------------------------------------------------------------- /server/app.py: -------------------------------------------------------------------------------- 1 | #!flask/bin/python 2 | from flask import Flask, make_response 3 | import os 4 | 5 | app = Flask(__name__) 6 | 7 | 8 | @app.errorhandler(404) 9 | def not_found(error): 10 | return make_response('Nada Dawg!' , 404) 11 | 12 | @app.errorhandler(400) 13 | def not_found(error): 14 | print error 15 | return make_response('What you say Dawg?!', 400) 16 | 17 | @app.route('/') 18 | @app.route('/index') 19 | def index(): 20 | return "Yo Dawg!" 21 | 22 | @app.route('/page1') 23 | @app.route('/page1/') 24 | @app.route('/page1//') 25 | def page1(subpage = "", subsubpage = ""): 26 | res = "Yo Dawg! You found page1" 27 | if (subpage is not "" ): 28 | res += "/" + subpage 29 | if (subsubpage is not "" ): 30 | res += "/" + subsubpage 31 | return res 32 | 33 | @app.route('/page2') 34 | @app.route('/page2/') 35 | @app.route('/page2//') 36 | def page2(subpage = "", subsubpage = ""): 37 | res = "Yo Dawg! You found page2" 38 | if (subpage is not "" ): 39 | res += "/" + subpage 40 | if (subsubpage is not "" ): 41 | res += "/" + subsubpage 42 | return res 43 | 44 | @app.route('/page3') 45 | @app.route('/page3/') 46 | @app.route('/page3//') 47 | def page3(subpage = "", subsubpage = ""): 48 | res = "Yo Dawg! You found page3" 49 | if (subpage is not "" ): 50 | res += "/" + subpage 51 | if (subsubpage is not "" ): 52 | res += "/" + subsubpage 53 | return res 54 | 55 | @app.route('/page4') 56 | @app.route('/page4/') 57 | @app.route('/page4//') 58 | def page4(subpage = "", subsubpage = ""): 59 | res = "Yo Dawg! You found page4" 60 | if (subpage is not "" ): 61 | res += "/" + subpage 62 | if (subsubpage is not "" ): 63 | res += "/" + subsubpage 64 | return res 65 | 66 | name = "" 67 | 68 | if __name__ == '__main__': 69 | port = int(os.environ['PORT']) 70 | app.run(host='0.0.0.0', port=port, debug=True) 71 | 72 | -------------------------------------------------------------------------------- /server/requirements.txt: -------------------------------------------------------------------------------- 1 | flask 2 | -------------------------------------------------------------------------------- /server/run.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | set -e 4 | 5 | # docker related... to get the gateway IP 6 | export AHOST=$(netstat -nr | grep 'UG' | awk '{print $2}') 7 | echo "AHOST: $AHOST" 8 | 9 | ( 10 | echo "# Auto-generated : do not touch" 11 | echo "" 12 | # echo "upstream site.localhost {" 13 | # printf " server ${AHOST}:5000;\n" 14 | # echo "}" 15 | echo "server {" 16 | echo "" 17 | echo " listen 8080;" 18 | echo " server_name httopd-1;" 19 | echo " access_log /var/log/nginx/host.httopd-1.access.log;" 20 | echo "" 21 | echo " location / {" 22 | echo " proxy_set_header Host \$host;" 23 | echo " proxy_set_header X-Real-IP \$remote_addr;" 24 | echo " proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;" 25 | echo "" 26 | echo " proxy_pass http://127.0.0.1:5001;" 27 | echo " }" 28 | echo "}" 29 | echo "server {" 30 | echo "" 31 | echo " listen 8081;" 32 | echo " server_name httopd-2;" 33 | echo " access_log /var/log/nginx/host.httopd-2.access.log;" 34 | echo "" 35 | echo " location / {" 36 | echo " proxy_set_header Host \$host;" 37 | echo " proxy_set_header X-Real-IP \$remote_addr;" 38 | echo " proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;" 39 | echo "" 40 | echo " proxy_pass http://127.0.0.1:5002;" 41 | echo " }" 42 | echo "}" 43 | echo "server {" 44 | echo "" 45 | echo " listen 8082;" 46 | echo " server_name httopd-3;" 47 | echo " access_log /var/log/nginx/host.httopd-3.access.log;" 48 | echo "" 49 | echo " location / {" 50 | echo " proxy_set_header Host \$host;" 51 | echo " proxy_set_header X-Real-IP \$remote_addr;" 52 | echo " proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;" 53 | echo "" 54 | echo " proxy_pass http://127.0.0.1:5003;" 55 | echo " }" 56 | echo "}" 57 | ) > dlb.cfg 58 | 59 | sudo cp dlb.cfg /etc/nginx/sites-enabled/default 60 | 61 | nginx & 62 | 63 | export PORT=5001 64 | python app.py & 65 | 66 | export PORT=5002 67 | python app.py & 68 | 69 | export PORT=5003 70 | python app.py 71 | -------------------------------------------------------------------------------- /start.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # set -e 4 | 5 | curr_dir="$(pwd)/$(dirname $0)" 6 | echo "curr_dir = $curr_dir" 7 | 8 | mkdir -p $curr_dir/logs 9 | 10 | echo "Starting server httopd!" 11 | docker run -d --name httopd-server \ 12 | -p 8080:8080 \ 13 | -p 8081:8081 \ 14 | -p 8082:8082 \ 15 | -v $curr_dir/logs:/var/log/nginx \ 16 | verdverm/httopd-server > /dev/null 17 | 18 | echo "Starting client httopd!" 19 | docker run -d --name httopd-client \ 20 | --net host \ 21 | verdverm/httopd-client > /dev/null 22 | 23 | # echo "Starting monitor httopd!" 24 | # docker run -i -t --name httopd-monitor \ 25 | # verdverm/httopd-monitor 26 | -------------------------------------------------------------------------------- /stop.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | echo "stopping httopd-*!" 4 | echo "----------------" 5 | 6 | docker rm -f httopd-client 7 | docker rm -f httopd-server 8 | 9 | rm -rf $(dirname $0)/logs 10 | --------------------------------------------------------------------------------