└── README.md /README.md: -------------------------------------------------------------------------------- 1 | # How to configure FileBeat and Logstash with SSL mutual authentication. 2 | 3 | How to configure SSL for FileBeat and Logstash step by step with OpenSSL (Create CA, CSRs, Certificates, etc). 4 | 5 | The Elasticsearch documentation "[Securing Communication With Logstash by Using SSL](https://www.elastic.co/guide/en/beats/filebeat/current/configuring-ssl-logstash.html)" does not show how to create with openssl the necessary keys and certificates to have the mutual authentication between FileBeat (output) and Logstash (input). It is not a difficult task but it can be very tedious if you are not familiar with the use of openssl. 6 | 7 | These are some errors that can be found in the FileBeat and Logstash logs when SSL is not properly configured. 8 | ``` 9 | # FileBeat. 10 | 11 | ERR Failed to publish events caused by: EOF 12 | ERR Connecting error publishing events (retrying): remote error: tls: handshake failure 13 | ERR Failed to publish events caused by: read tcp X.X.X.X:XXXXX->X.X.X.X:XXXX: i/o timeout 14 | 15 | 16 | # Logstash. 17 | 18 | Exception: javax.net.ssl.SSLHandshakeException: error:100000b8:SSL routines:OPENSSL_internal:NO_SHARED_CIPHER 19 | [ERROR][logstash.inputs.beats ] Looks like you either have an invalid key or your private key was not in PKCS8 format. {:exception=>java.lang.IllegalArgumentException: File does not contain valid private key: /XXXX/XXXX.key} 20 | [INFO ][org.logstash.beats.BeatsHandler] Exception: javax.net.ssl.SSLHandshakeException: error:10000418:SSL routines:OPENSSL_internal:TLSV1_ALERT_UNKNOWN_CA 21 | [INFO ][org.logstash.beats.BeatsHandler] Exception: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem 22 | ``` 23 | 24 | These are the steps to configure Filebeat with Logstash using SSL, mutual authentication and TLS 2.0 encryption. 25 | **Tested in Logstash / Filebeat version**: 5.6 26 | 27 | ### Logstash input beat configuration (files ca.crt,server.crt, and server.key are needed). 28 | ``` 29 | input { 30 | beats { 31 | port => 5044 32 | ssl => true 33 | ssl_certificate_authorities => ["/etc/ca.crt"] 34 | ssl_certificate => "/etc/server.crt" 35 | ssl_key => "/etc/server.key" 36 | ssl_verify_mode => "force_peer" 37 | } 38 | } 39 | ``` 40 | 41 | ### Filebeat output (SSL) configuration (files ca.crt, client.crt and client.key are needed). 42 | ``` 43 | output.logstash: 44 | hosts: ["logs.mycompany.com:5044"] 45 | ssl.certificate_authorities: ["/etc/ca.crt"] 46 | ssl.certificate: "/etc/client.crt" 47 | ssl.key: "/etc/client.key" 48 | # ssl.key_passphrase: "PASSWORD" 49 | ssl.supported_protocols: "TLSv1.2" 50 | ``` 51 | 52 | ### CA (create files ca.key and ca.crt). 53 | ```bash 54 | openssl genrsa -out ca.key 2048 55 | openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 -out ca.crt 56 | ``` 57 | 58 | ### Logstasg server (create server.key and server.crt). 59 | 60 | File server.conf 61 | ``` 62 | [req] 63 | distinguished_name = req_distinguished_name 64 | req_extensions = v3_req 65 | prompt = no 66 | 67 | [req_distinguished_name] 68 | countryName = XX 69 | stateOrProvinceName = XXXXXX 70 | localityName = XXXXXX 71 | postalCode = XXXXXX 72 | organizationName = XXXXXX 73 | organizationalUnitName = XXXXXX 74 | commonName = XXXXXX 75 | emailAddress = XXXXXX 76 | 77 | [v3_req] 78 | keyUsage = keyEncipherment, dataEncipherment 79 | extendedKeyUsage = serverAuth 80 | subjectAltName = @alt_names 81 | 82 | [alt_names] 83 | DNS.1 = DOMAIN_1 84 | DNS.2 = DOMAIN_2 85 | DNS.3 = DOMAIN_3 86 | DNS.4 = DOMAIN_4 87 | ``` 88 | 89 | ```bash 90 | openssl genrsa -out server.key 2048 91 | openssl req -sha512 -new -key server.key -out server.csr -config server.conf 92 | echo "C2E9862A0DA8E970" > serial 93 | openssl x509 -days 3650 -req -sha512 -in server.csr -CAserial serial -CA ca.crt -CAkey ca.key -out server.crt -extensions v3_req -extfile server.conf 94 | mv server.key server.key.pem && openssl pkcs8 -in server.key.pem -topk8 -nocrypt -out server.key 95 | ``` 96 | 97 | ### FileBeat shipper (create files client.key and client.crt). 98 | 99 | File client.conf. 100 | ``` 101 | [req] 102 | distinguished_name = req_distinguished_name 103 | req_extensions = v3_req 104 | prompt = no 105 | 106 | [req_distinguished_name] 107 | countryName = XX 108 | stateOrProvinceName = XXXXXX 109 | localityName = XXXXXX 110 | postalCode = XXXXXX 111 | organizationName = XXXXXX 112 | organizationalUnitName = XXXXXX 113 | commonName = XXXXXX 114 | emailAddress = XXXXXX 115 | 116 | [ usr_cert ] 117 | # Extensions for server certificates (`man x509v3_config`). 118 | basicConstraints = CA:FALSE 119 | nsCertType = client, server 120 | nsComment = "OpenSSL FileBeat Server / Client Certificate" 121 | subjectKeyIdentifier = hash 122 | authorityKeyIdentifier = keyid,issuer:always 123 | keyUsage = critical, digitalSignature, keyEncipherment, keyAgreement, nonRepudiation 124 | extendedKeyUsage = serverAuth, clientAuth 125 | 126 | [v3_req] 127 | keyUsage = keyEncipherment, dataEncipherment 128 | extendedKeyUsage = serverAuth, clientAuth 129 | ``` 130 | ``` 131 | openssl genrsa -out client.key 2048 132 | openssl req -sha512 -new -key client.key -out client.csr -config client.conf 133 | openssl x509 -days 3650 -req -sha512 -in client.csr -CAserial serial -CA ca.crt -CAkey ca.key -out client.crt -extensions v3_req -extensions usr_cert -extfile client.conf 134 | ``` 135 | ``` 136 | # If the client key is not encrypted by passphrase, it can always be added later (filebeat "ssl.key_passphrase"). 137 | # openssl rsa -des -in client.key -out client4.key 138 | ``` 139 | --------------------------------------------------------------------------------