├── .gitignore ├── .travis.yml ├── LICENSE ├── README.md ├── lap.go ├── lap_test.go └── test_data ├── input_1.txt ├── input_2.txt ├── input_3.txt ├── input_4.txt ├── input_5.txt ├── output_1_xml.txt ├── output_2_xml.txt ├── output_3_xml.txt ├── output_4_xml.txt └── output_5_xml.txt /.gitignore: -------------------------------------------------------------------------------- 1 | *.log 2 | lap 3 | ldap_access_parser 4 | *.swp 5 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: go 2 | 3 | go: 4 | - tip 5 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2015 Aidan Rowe 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | LDAP Access-log Parser [![Build Status](https://travis-ci.org/aidan-/ldap-access-parser.svg)](https://travis-ci.org/aidan-/ldap-access-parser/) 2 | ====================== 3 | LDAP Access-log Parser (LAP) is a simple program designed to parse 389 Directory Server (also Fedora Directory Server, Red Hat Directory Server, etc) access logs into individual context aware events ready to be sent to upstream services like ElasticSearch for further analysis. 4 | 5 | This application doesn't handle the sending of data to specific endpoints, but that can easily be achieved by piping to something like [log-courier](https://github.com/driskell/log-courier) or [logstash](https://www.elastic.co/products/logstash). 6 | 7 | Usage 8 | ----- 9 | Use the `-h` flag to view application usage. 10 | 11 | ``` 12 | Usage of ./lap: 13 | -format string 14 | format to output log events. possible values are 'json' or 'xml'. (default "json") 15 | -tail 16 | tail the log file to receive future events 17 | ``` 18 | 19 | You can begin parsing logs with a simple command: 20 | ``` 21 | ./lap -tail /path/to/access_log 22 | ``` 23 | 24 | This continue to parse the file even if it gets rotated. 25 | 26 | Design 27 | ------ 28 | The applications functionality and output format tries to follow the design specification laid out [here](http://directory.fedoraproject.org/docs/389ds/design/audit-events.html) as much as possible, however there are a few use-cases which were not covered in the design spec that needed to be defined. 29 | 30 | More detailed information about the access log format is available in the [Red Hat Directory Server documentation](https://access.redhat.com/documentation/en-US/Red_Hat_Directory_Server/8.1/html/Configuration_and_Command_Reference/logs-reference.html). 31 | 32 | Example Output 33 | -------------- 34 | Given the raw access log input of: 35 | ``` 36 | [21/Apr/2009:11:39:55 -0700] conn=14 fd=700 slot=700 connection from 207.1.153.51 to 192.18.122.139 37 | [21/Apr/2009:11:39:55 -0700] conn=14 op=0 BIND dn="" method=sasl version=3 mech=DIGEST-MD5 38 | [21/Apr/2009:11:39:55 -0700] conn=14 op=0 RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress 39 | [21/Apr/2009:11:39:55 -0700] conn=14 op=1 BIND dn="uid=jdoe,dc=example,dc=com" method=sasl version=3 mech=DIGEST-MD5 40 | [21/Apr/2009:11:39:55 -0700] conn=14 op=1 RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=jdoe,dc=example,dc=com" 41 | ``` 42 | 43 | The JSON output would look like: 44 | ```json 45 | {"time":"21/Apr/2009:11:39:55 -0700","client":"207.1.153.51","server":"192.18.122.139","connection":14,"ssl":false,"operation":0,"authenticateddn":"__anonymous__","action":"BIND","requests":["BIND dn=\"\" method=sasl version=3 mech=DIGEST-MD5"],"responses":["RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress"]} 46 | {"time":"21/Apr/2009:11:39:55 -0700","client":"207.1.153.51","server":"192.18.122.139","connection":14,"ssl":false,"operation":1,"authenticateddn":"uid=jdoe,dc=example,dc=com","action":"BIND","requests":["BIND dn=\"uid=jdoe,dc=example,dc=com\" method=sasl version=3 mech=DIGEST-MD5"],"responses":["RESULT err=0 tag=97 nentries=0 etime=0 dn=\"uid=jdoe,dc=example,dc=com\""]} 47 | ``` 48 | 49 | and XML output would look like: 50 | ```xml 51 | 52 | 21/Apr/2009:11:39:55 -0700 53 | 207.1.153.51 54 | 192.18.122.139 55 | 14 56 | false 57 | 0 58 | __anonymous__ 59 | BIND 60 | 61 | BIND dn="" method=sasl version=3 mech=DIGEST-MD5 62 | 63 | 64 | RESULT err=14 tag=97 nentries=0 etime=0, SASL bind in progress 65 | 66 | 67 | 68 | 21/Apr/2009:11:39:55 -0700 69 | 207.1.153.51 70 | 192.18.122.139 71 | 14 72 | false 73 | 1 74 | uid=jdoe,dc=example,dc=com 75 | BIND 76 | 77 | BIND dn="uid=jdoe,dc=example,dc=com" method=sasl version=3 mech=DIGEST-MD5 78 | 79 | 80 | RESULT err=0 tag=97 nentries=0 etime=0 dn="uid=jdoe,dc=example,dc=com" 81 | 82 | 83 | ``` 84 | 85 | TODO/Notes 86 | ----- 87 | - If a connection was initialized before the log monitoring began, the events associated with that connection number will be skipped. This should probably be a configurable option. 88 | - Connections that disconnect without any operations do not get outputted as an event. Is this okay? 89 | -------------------------------------------------------------------------------- /lap.go: -------------------------------------------------------------------------------- 1 | package main 2 | 3 | import ( 4 | "encoding/json" 5 | "encoding/xml" 6 | "flag" 7 | "fmt" 8 | "io" 9 | "os" 10 | "regexp" 11 | "strconv" 12 | "time" 13 | 14 | "github.com/hpcloud/tail" 15 | ) 16 | 17 | const ( 18 | timeFormat = "02/Jan/2006:03:04:05 -0700" 19 | version = "v1.0" 20 | ) 21 | 22 | //Event (request(s) paired with response) to be output in either 23 | //JSON or XML depending on user preference. 24 | type Event struct { 25 | OppTime string `json:"time" xml:"DateTime"` 26 | Client string `json:"client" xml:"Client"` 27 | Server string `json:"server" xml:"Server"` 28 | Connection int `json:"connection" xml:"Connection"` 29 | SSL bool `json:"ssl" xml:"SSL"` 30 | SSLCipher string `json:"sslcipher,omitempty" xml:"SSLCipher,omitempty"` 31 | SSLStrength string `json:"sslstrength,omitempty" xml:"SSLStrength,omitempty"` 32 | Operation int `json:"operation" xml:"Operation"` 33 | AuthenticatedDN string `json:"authenticateddn,omitempty" xml:"AuthenticatedDN,omitempty"` 34 | Action string `json:"action" xml:"Action"` 35 | Requests []string `json:"requests" xml:"Requests>Request"` 36 | Responses []string `json:"responses" xml:"Responses>Response"` 37 | Duration int `json:"duration,omitempty" xml:"Duration,omitempty"` 38 | ConnTime string `json:"-" xml:"-"` 39 | } 40 | 41 | type config struct { 42 | Version *bool 43 | TailFile *bool 44 | OutputFormat *string 45 | LogFiles *[]string 46 | Output io.Writer 47 | } 48 | 49 | //holds config 50 | var c config 51 | 52 | //regexes to extract relevant fields from log lines 53 | var lineMatch = `^\[(?P