├── files ├── .gitignore ├── grok-pattern-0 ├── grok-pattern-1 └── null-output.conf ├── .projectile ├── spec ├── fixtures │ ├── artifacts │ │ └── .gitignore │ ├── grok-patterns │ │ ├── grok-pattern-0 │ │ └── grok-pattern-1 │ ├── templates │ │ └── configfile-template.erb │ └── plugins │ │ ├── logstash-output-cowsay-5.0.0.gem │ │ ├── logstash-output-cowsay-5.0.0.zip │ │ └── logstash-output-cowthink-5.0.0.gem ├── spec_helper_acceptance.rb ├── defines │ └── define_plugin_spec.rb ├── spec_helper.rb ├── acceptance │ ├── define_patternfile_spec.rb │ ├── define_configfile_spec.rb │ ├── class_plugin_spec.rb │ └── class_logstash_spec.rb └── support │ └── acceptance │ └── helpers.rb ├── templates ├── pipelines.yml.erb ├── jvm.options.erb ├── startup.options.erb └── logstash.yml.erb ├── .sync.yml ├── .msync.yml ├── .puppet-lint.rc ├── .rubocop.yml ├── .github ├── labeler.yml ├── workflows │ ├── labeler.yml │ ├── ci.yml │ ├── release.yml │ └── prepare_release.yml ├── PULL_REQUEST_TEMPLATE.md ├── ISSUE_TEMPLATE.md └── release.yml ├── .editorconfig ├── .fixtures.yml ├── .gitignore ├── CONTRIBUTORS ├── .pmtignore ├── Gemfile ├── Vagrantfile.d ├── manifests │ └── site.pp └── server.sh ├── Rakefile ├── Vagrantfile ├── manifests ├── patternfile.pp ├── config.pp ├── configfile.pp ├── plugin.pp ├── package.pp ├── service.pp └── init.pp ├── .overcommit.yml ├── metadata.json ├── HISTORY.md ├── LICENSE ├── README.md ├── CHANGELOG.md └── REFERENCE.md /files/.gitignore: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.projectile: -------------------------------------------------------------------------------- 1 | -/doc 2 | -/.vendor 3 | -------------------------------------------------------------------------------- /spec/fixtures/artifacts/.gitignore: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /files/grok-pattern-0: -------------------------------------------------------------------------------- 1 | GROK_PATTERN_0 . 2 | -------------------------------------------------------------------------------- /files/grok-pattern-1: -------------------------------------------------------------------------------- 1 | GROK_PATTERN_1 . 2 | -------------------------------------------------------------------------------- /templates/pipelines.yml.erb: -------------------------------------------------------------------------------- 1 | <%= @pipelines.to_yaml %> 2 | -------------------------------------------------------------------------------- /spec/fixtures/grok-patterns/grok-pattern-0: -------------------------------------------------------------------------------- 1 | GROK_PATTERN_0 . 2 | -------------------------------------------------------------------------------- /spec/fixtures/grok-patterns/grok-pattern-1: -------------------------------------------------------------------------------- 1 | GROK_PATTERN_1 . 2 | -------------------------------------------------------------------------------- /spec/fixtures/templates/configfile-template.erb: -------------------------------------------------------------------------------- 1 | # 2 + 2 equals <%= 2+2 -%> 2 | -------------------------------------------------------------------------------- /files/null-output.conf: -------------------------------------------------------------------------------- 1 | # Test output configuration with null output. 2 | output { 3 | null {} 4 | } 5 | -------------------------------------------------------------------------------- /.sync.yml: -------------------------------------------------------------------------------- 1 | --- 2 | spec/spec_helper_acceptance.rb: 3 | unmanaged: false 4 | 5 | .puppet-lint.rc: 6 | enabled_lint_checks: 7 | - parameter_documentation 8 | -------------------------------------------------------------------------------- /spec/fixtures/plugins/logstash-output-cowsay-5.0.0.gem: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/voxpupuli/puppet-logstash/HEAD/spec/fixtures/plugins/logstash-output-cowsay-5.0.0.gem -------------------------------------------------------------------------------- /spec/fixtures/plugins/logstash-output-cowsay-5.0.0.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/voxpupuli/puppet-logstash/HEAD/spec/fixtures/plugins/logstash-output-cowsay-5.0.0.zip -------------------------------------------------------------------------------- /spec/fixtures/plugins/logstash-output-cowthink-5.0.0.gem: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/voxpupuli/puppet-logstash/HEAD/spec/fixtures/plugins/logstash-output-cowthink-5.0.0.gem -------------------------------------------------------------------------------- /.msync.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # Managed by modulesync - DO NOT EDIT 3 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 4 | 5 | modulesync_config_version: '10.4.0' 6 | -------------------------------------------------------------------------------- /.puppet-lint.rc: -------------------------------------------------------------------------------- 1 | # Managed by modulesync - DO NOT EDIT 2 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 3 | 4 | --fail-on-warnings 5 | --no-parameter_types-check 6 | -------------------------------------------------------------------------------- /.rubocop.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # Managed by modulesync - DO NOT EDIT 3 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 4 | 5 | inherit_gem: 6 | voxpupuli-test: rubocop.yml 7 | -------------------------------------------------------------------------------- /.github/labeler.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # Managed by modulesync - DO NOT EDIT 3 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 4 | 5 | skip-changelog: 6 | - head-branch: ['^release-*', 'release'] 7 | -------------------------------------------------------------------------------- /.editorconfig: -------------------------------------------------------------------------------- 1 | # editorconfig.org 2 | 3 | # Managed by modulesync - DO NOT EDIT 4 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 5 | 6 | root = true 7 | 8 | [*] 9 | charset = utf-8 10 | end_of_line = lf 11 | indent_size = 2 12 | tab_width = 2 13 | indent_style = space 14 | insert_final_newline = true 15 | trim_trailing_whitespace = true 16 | -------------------------------------------------------------------------------- /spec/spec_helper_acceptance.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | # Managed by modulesync - DO NOT EDIT 4 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 5 | 6 | require 'voxpupuli/acceptance/spec_helper_acceptance' 7 | 8 | configure_beaker(modules: :metadata) 9 | 10 | Dir['./spec/support/acceptance/**/*.rb'].sort.each { |f| require f } 11 | -------------------------------------------------------------------------------- /.fixtures.yml: -------------------------------------------------------------------------------- 1 | --- 2 | fixtures: 3 | repositories: 4 | apt: https://github.com/puppetlabs/puppetlabs-apt.git 5 | elastic_stack: https://github.com/voxpupuli/puppet-elastic_stack.git 6 | stdlib: https://github.com/puppetlabs/puppetlabs-stdlib.git 7 | yumrepo: https://github.com/puppetlabs/puppetlabs-yumrepo_core.git 8 | zypprepo: https://github.com/voxpupuli/puppet-zypprepo.git 9 | -------------------------------------------------------------------------------- /templates/jvm.options.erb: -------------------------------------------------------------------------------- 1 | # This file is managed by Puppet -- <%= @name %> 2 | # 3 | # Set the 'jvm_options' parameter on the logstash class to change this file. 4 | <% 5 | def set_default(options, match_string, default) 6 | options.detect {|o| o.include?(match_string)} || options.push(default) 7 | end 8 | 9 | @jvm_options_defaults.each {|k,v| set_default(@jvm_options, k, v)} 10 | -%> 11 | 12 | <% @jvm_options.sort.each do |line| -%> 13 | <%= line %> 14 | <% end -%> 15 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | # Managed by modulesync - DO NOT EDIT 2 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 3 | 4 | /pkg/ 5 | /Gemfile.lock 6 | /Gemfile.local 7 | /vendor/ 8 | /.vendor/ 9 | /spec/fixtures/manifests/ 10 | /spec/fixtures/modules/ 11 | /.vagrant/ 12 | /.bundle/ 13 | /.ruby-version 14 | /coverage/ 15 | /log/ 16 | /.idea/ 17 | /.dependencies/ 18 | /.librarian/ 19 | /Puppetfile.lock 20 | *.iml 21 | .*.sw? 22 | /.yardoc/ 23 | /Guardfile 24 | bolt-debug.log 25 | .rerun.json 26 | -------------------------------------------------------------------------------- /.github/workflows/labeler.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # Managed by modulesync - DO NOT EDIT 3 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 4 | 5 | name: "Pull Request Labeler" 6 | 7 | # yamllint disable-line rule:truthy 8 | on: 9 | pull_request_target: {} 10 | 11 | permissions: 12 | contents: read 13 | pull-requests: write 14 | 15 | jobs: 16 | labeler: 17 | permissions: 18 | contents: read 19 | pull-requests: write 20 | runs-on: ubuntu-latest 21 | steps: 22 | - uses: actions/labeler@v5 23 | -------------------------------------------------------------------------------- /.github/workflows/ci.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # Managed by modulesync - DO NOT EDIT 3 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 4 | 5 | name: CI 6 | 7 | # yamllint disable-line rule:truthy 8 | on: 9 | pull_request: {} 10 | push: 11 | branches: 12 | - main 13 | - master 14 | 15 | concurrency: 16 | group: ${{ github.ref_name }} 17 | cancel-in-progress: true 18 | 19 | permissions: 20 | contents: read 21 | 22 | jobs: 23 | puppet: 24 | name: Puppet 25 | uses: voxpupuli/gha-puppet/.github/workflows/beaker.yml@v4 26 | -------------------------------------------------------------------------------- /CONTRIBUTORS: -------------------------------------------------------------------------------- 1 | The following is a list of people who have contributed ideas, code, bug 2 | reports, or in general have helped this puppet module along its way. 3 | 4 | Project Owner 5 | * Richard Pijnenburg (electrical) 6 | 7 | Contributors: 8 | * Jordan Sissel (jordansissel) 9 | * Dan (phrawzty) 10 | * Garth Kidd (garthk) 11 | * Tavis Aitken (tavisto) 12 | * pkubat 13 | * Jeff Wong (awole20) 14 | * Bob (rjw1) 15 | * Dan Carley (dcarley) 16 | * Brian Lalor (blalor) 17 | * Justin Lambert (jlambert) 18 | * Richard Peng (richardpeng) 19 | * Matthias Baur (baurmatt) 20 | -------------------------------------------------------------------------------- /.github/PULL_REQUEST_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | 9 | #### Pull Request (PR) description 10 | 13 | 14 | #### This Pull Request (PR) fixes the following issues 15 | 21 | -------------------------------------------------------------------------------- /spec/defines/define_plugin_spec.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | require 'spec_helper' 4 | 5 | describe 'logstash::plugin' do 6 | on_supported_os.each do |os, facts| 7 | context "on #{os}" do 8 | let(:facts) do 9 | facts 10 | end 11 | 12 | let(:pre_condition) do 13 | <<~PUPPET 14 | include elastic_stack::repo 15 | include logstash 16 | PUPPET 17 | end 18 | 19 | let(:title) { 'logstash-input-mysql' } 20 | 21 | it { is_expected.to compile } 22 | 23 | it { 24 | is_expected.to contain_exec("install-#{title}").with(user: 'root') 25 | } 26 | end 27 | end 28 | end 29 | -------------------------------------------------------------------------------- /.pmtignore: -------------------------------------------------------------------------------- 1 | # Managed by modulesync - DO NOT EDIT 2 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 3 | 4 | /docs/ 5 | /pkg/ 6 | /Gemfile 7 | /Gemfile.lock 8 | /Gemfile.local 9 | /vendor/ 10 | /.vendor/ 11 | /spec/ 12 | /Rakefile 13 | /.vagrant/ 14 | /.bundle/ 15 | /.ruby-version 16 | /coverage/ 17 | /log/ 18 | /.idea/ 19 | /.dependencies/ 20 | /.github/ 21 | /.librarian/ 22 | /Puppetfile.lock 23 | /Puppetfile 24 | *.iml 25 | /.editorconfig 26 | /.fixtures.yml 27 | /.gitignore 28 | /.msync.yml 29 | /.overcommit.yml 30 | /.pmtignore 31 | /.rspec 32 | /.rspec_parallel 33 | /.rubocop.yml 34 | /.sync.yml 35 | .*.sw? 36 | /.yardoc/ 37 | /.yardopts 38 | /Dockerfile 39 | /HISTORY.md 40 | -------------------------------------------------------------------------------- /templates/startup.options.erb: -------------------------------------------------------------------------------- 1 | # This file is managed by Puppet -- <%= @name %> 2 | 3 | ############################################################################### 4 | # These settings are ONLY used by $LS_HOME/bin/system-install to create a custom 5 | # startup script for Logstash. It should automagically use the init system 6 | # (systemd, upstart, sysv, etc.) that your Linux distribution uses. 7 | # 8 | # After changing anything here, you need to re-run $LS_HOME/bin/system-install 9 | # as root to push the changes to the init script. 10 | ################################################################################ 11 | 12 | <% @startup_options.sort.each do |k,v| -%> 13 | <%= k %>=<%= v %> 14 | <% end -%> 15 | -------------------------------------------------------------------------------- /.github/workflows/release.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # Managed by modulesync - DO NOT EDIT 3 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 4 | 5 | name: Release 6 | 7 | # yamllint disable-line rule:truthy 8 | on: 9 | push: 10 | tags: 11 | - '*' 12 | 13 | permissions: 14 | contents: write 15 | 16 | jobs: 17 | release: 18 | name: Release 19 | uses: voxpupuli/gha-puppet/.github/workflows/release.yml@v3 20 | with: 21 | allowed_owner: 'voxpupuli' 22 | secrets: 23 | # Configure secrets here: 24 | # https://docs.github.com/en/actions/security-guides/encrypted-secrets 25 | username: ${{ secrets.PUPPET_FORGE_USERNAME }} 26 | api_key: ${{ secrets.PUPPET_FORGE_API_KEY }} 27 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE.md: -------------------------------------------------------------------------------- 1 | 10 | 11 | ## Affected Puppet, Ruby, OS and module versions/distributions 12 | 13 | - Puppet: 14 | - Ruby: 15 | - Distribution: 16 | - Module version: 17 | 18 | ## How to reproduce (e.g Puppet code you use) 19 | 20 | ## What are you seeing 21 | 22 | ## What behaviour did you expect instead 23 | 24 | ## Output log 25 | 26 | ## Any additional information you'd like to impart 27 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | # Managed by modulesync - DO NOT EDIT 2 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 3 | 4 | source ENV['GEM_SOURCE'] || 'https://rubygems.org' 5 | 6 | group :test do 7 | gem 'voxpupuli-test', '~> 13.0', :require => false 8 | gem 'puppet_metadata', '~> 5.0', :require => false 9 | end 10 | 11 | group :development do 12 | gem 'guard-rake', :require => false 13 | gem 'overcommit', '>= 0.39.1', :require => false 14 | end 15 | 16 | group :system_tests do 17 | gem 'voxpupuli-acceptance', '~> 4.0', :require => false 18 | end 19 | 20 | group :release do 21 | gem 'voxpupuli-release', '~> 5.0', :require => false 22 | end 23 | 24 | gem 'rake', :require => false 25 | 26 | gem 'openvox', ENV.fetch('OPENVOX_GEM_VERSION', [">= 7", "< 9"]), :require => false, :groups => [:test] 27 | 28 | # vim: syntax=ruby 29 | -------------------------------------------------------------------------------- /spec/spec_helper.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | # Managed by modulesync - DO NOT EDIT 4 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 5 | 6 | # puppetlabs_spec_helper will set up coverage if the env variable is set. 7 | # We want to do this if lib exists and it hasn't been explicitly set. 8 | ENV['COVERAGE'] ||= 'yes' if Dir.exist?(File.expand_path('../lib', __dir__)) 9 | 10 | require 'voxpupuli/test/spec_helper' 11 | 12 | RSpec.configure do |c| 13 | c.facterdb_string_keys = false 14 | end 15 | 16 | add_mocked_facts! 17 | 18 | if File.exist?(File.join(__dir__, 'default_module_facts.yml')) 19 | facts = YAML.safe_load(File.read(File.join(__dir__, 'default_module_facts.yml'))) 20 | facts&.each do |name, value| 21 | add_custom_fact name.to_sym, value 22 | end 23 | end 24 | Dir['./spec/support/spec/**/*.rb'].sort.each { |f| require f } 25 | -------------------------------------------------------------------------------- /.github/workflows/prepare_release.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # Managed by modulesync - DO NOT EDIT 3 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 4 | 5 | name: 'Prepare Release' 6 | 7 | on: 8 | workflow_dispatch: 9 | inputs: 10 | version: 11 | description: 'Module version to be released. Must be a valid semver string without leading v. (1.2.3)' 12 | required: false 13 | 14 | permissions: 15 | contents: write 16 | pull-requests: write 17 | 18 | jobs: 19 | release_prep: 20 | uses: 'voxpupuli/gha-puppet/.github/workflows/prepare_release.yml@v3' 21 | with: 22 | version: ${{ github.event.inputs.version }} 23 | allowed_owner: 'voxpupuli' 24 | secrets: 25 | # Configure secrets here: 26 | # https://docs.github.com/en/actions/security-guides/encrypted-secrets 27 | github_pat: '${{ secrets.PCCI_PAT_RELEASE_PREP }}' 28 | -------------------------------------------------------------------------------- /templates/logstash.yml.erb: -------------------------------------------------------------------------------- 1 | <%# By default, Logstash sets 'path.config' in 'logstash.yml', however -%> 2 | <%# when that setting is present, 'pipelines.yml' is ignored. If the user -%> 3 | <%# specified their own piplines, we'll make sure they are honoured by -%> 4 | <%# removing the 'path.config' setting. -%> 5 | <%# -%> 6 | <%# REF: https://github.com/elastic/logstash/issues/8420 -%> 7 | <% @settings.delete('path.config') unless @pipelines.empty? -%> 8 | <%# -%> 9 | <%# Similiarly, when using centralized pipeline management, path.config -%> 10 | <%# is an invalid setting, and should be removed. -%> 11 | <%# REF: https://github.com/elastic/puppet-logstash/issues/357 -%> 12 | <% @settings.delete('path.config') if @settings['xpack.management.enabled'] -%> 13 | <% begin -%> 14 | <% @settings.delete('path.config') if @settings['xpack']['management']['enabled'] -%> 15 | <% rescue NoMethodError -%> 16 | <% end -%> 17 | <%# -%> 18 | <%= @settings.to_yaml %> 19 | -------------------------------------------------------------------------------- /Vagrantfile.d/manifests/site.pp: -------------------------------------------------------------------------------- 1 | $pipelines = [ 2 | { 3 | 'pipeline.id' => 'pipeline_zero', 4 | 'path.config' => '/tmp/pipeline_zero.conf', 5 | }, 6 | { 7 | 'pipeline.id' => 'pipeline_one', 8 | 'path.config' => '/tmp/pipeline_one.conf', 9 | }, 10 | ] 11 | 12 | class { 'elastic_stack::repo': 13 | version => 6, 14 | prerelease => false, 15 | } 16 | 17 | class { 'logstash': 18 | manage_repo => true, 19 | version => '6.5.1', 20 | pipelines => $pipelines, 21 | startup_options => { 'LS_USER' => 'root' }, 22 | } 23 | 24 | logstash::configfile { 'pipeline_zero': 25 | content => 'input { heartbeat{} } output { null {} }', 26 | path => '/tmp/pipeline_zero.conf', 27 | } 28 | 29 | logstash::configfile { 'pipeline_one': 30 | content => 'input { tcp { port => 2002 } } output { null {} }', 31 | path => '/tmp/pipeline_one.conf', 32 | } 33 | 34 | logstash::plugin { 'logstash-input-mysql': 35 | environment => ['LS_JAVA_OPTS=-Xms1g -Xmx1g'], 36 | } 37 | -------------------------------------------------------------------------------- /.github/release.yml: -------------------------------------------------------------------------------- 1 | --- 2 | # Managed by modulesync - DO NOT EDIT 3 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 4 | 5 | # https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes 6 | 7 | changelog: 8 | exclude: 9 | labels: 10 | - duplicate 11 | - invalid 12 | - modulesync 13 | - question 14 | - skip-changelog 15 | - wont-fix 16 | - wontfix 17 | 18 | categories: 19 | - title: Breaking Changes 🛠 20 | labels: 21 | - backwards-incompatible 22 | 23 | - title: New Features 🎉 24 | labels: 25 | - enhancement 26 | 27 | - title: Bug Fixes 🐛 28 | labels: 29 | - bug 30 | 31 | - title: Documentation Updates 📚 32 | labels: 33 | - documentation 34 | - docs 35 | 36 | - title: Dependency Updates ⬆️ 37 | labels: 38 | - dependencies 39 | 40 | - title: Other Changes 41 | labels: 42 | - "*" 43 | -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | # Managed by modulesync - DO NOT EDIT 2 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 3 | 4 | begin 5 | require 'voxpupuli/test/rake' 6 | rescue LoadError 7 | # only available if gem group test is installed 8 | end 9 | 10 | begin 11 | require 'voxpupuli/acceptance/rake' 12 | rescue LoadError 13 | # only available if gem group acceptance is installed 14 | end 15 | 16 | begin 17 | require 'voxpupuli/release/rake_tasks' 18 | rescue LoadError 19 | # only available if gem group releases is installed 20 | else 21 | GCGConfig.user = 'voxpupuli' 22 | GCGConfig.project = 'puppet-logstash' 23 | end 24 | 25 | desc "Run main 'test' task and report merged results to coveralls" 26 | task test_with_coveralls: [:test] do 27 | if Dir.exist?(File.expand_path('../lib', __FILE__)) 28 | require 'coveralls/rake/task' 29 | Coveralls::RakeTask.new 30 | Rake::Task['coveralls:push'].invoke 31 | else 32 | puts 'Skipping reporting to coveralls. Module has no lib dir' 33 | end 34 | end 35 | 36 | # vim: syntax=ruby 37 | -------------------------------------------------------------------------------- /Vagrantfile.d/server.sh: -------------------------------------------------------------------------------- 1 | set -euo pipefail 2 | 3 | # Install and configure puppetserver. 4 | rpm -Uvh https://yum.puppetlabs.com/puppet5/puppet5-release-el-7.noarch.rpm 5 | yum install -y puppetserver puppet-agent 6 | ln -sf /opt/puppetlabs/bin/* /usr/bin 7 | 8 | # REF: https://tickets.puppetlabs.com/browse/SERVER-528 9 | systemctl stop puppet 10 | systemctl stop puppetserver 11 | rm -rf /etc/puppetlabs/puppet/ssl/private_keys/* 12 | rm -rf /etc/puppetlabs/puppet/ssl/certs/* 13 | echo 'autosign = true' >> /etc/puppetlabs/puppet/puppet.conf 14 | systemctl start puppetserver 15 | 16 | # Puppet agent looks for the server called "puppet" by default. 17 | # In this case, we want that to be us (the loopback address). 18 | echo '127.0.0.1 localhost puppet' > /etc/hosts 19 | 20 | # Install puppet-logstash dependencies. 21 | /opt/puppetlabs/bin/puppet module install \ 22 | --target-dir=/etc/puppetlabs/code/environments/production/modules \ 23 | elastic-elastic_stack 24 | 25 | # Install Java 8 for Logstash. 26 | yum install -y java-1.8.0-openjdk-devel 27 | java -version 2>&1 28 | -------------------------------------------------------------------------------- /Vagrantfile: -------------------------------------------------------------------------------- 1 | # This Vagrant file is provided as a convenience for development and 2 | # exploratory testing of puppet-logstash. It's not used by the formal 3 | # testing framwork, it's just for hacking. 4 | # 5 | # See `CONTRIBUTING.md` for details on formal testing. 6 | puppet_code_root = '/etc/puppetlabs/code/environments/production' 7 | module_root = "#{puppet_code_root}/modules/logstash" 8 | manifest_dir = "#{puppet_code_root}/manifests" 9 | 10 | Vagrant.configure(2) do |config| 11 | # config.vm.box = 'puppetlabs/debian-8.2-64-puppet' 12 | config.vm.box = 'bento/centos-7.3' 13 | config.vm.provider 'virtualbox' do |vm| 14 | vm.memory = 4 * 1024 15 | end 16 | 17 | # Make the Logstash module available. 18 | %w(manifests templates files).each do |dir| 19 | config.vm.synced_folder(dir, "#{module_root}/#{dir}") 20 | end 21 | 22 | # Map in a Puppet manifest that can be used for experiments. 23 | config.vm.synced_folder('Vagrantfile.d/manifests', "#{puppet_code_root}/manifests") 24 | 25 | # Prepare a puppetserver install so we can test the module in a realistic 26 | # way. 'puppet apply' is cool, but in reality, most people need this to work 27 | # in a master/agent configuration. 28 | config.vm.provision('shell', path: 'Vagrantfile.d/server.sh') 29 | end 30 | -------------------------------------------------------------------------------- /manifests/patternfile.pp: -------------------------------------------------------------------------------- 1 | # This type represents a Grok pattern file for Logstash. 2 | # 3 | # @param [String] source 4 | # File source for the pattern file. eg. `puppet://[...]` or `file://[...]` 5 | # 6 | # @param [String] filename 7 | # Optionally set the destination filename. 8 | # 9 | # @example Define a pattern file. 10 | # logstash::patternfile { 'mypattern': 11 | # source => 'puppet:///path/to/my/custom/pattern' 12 | # } 13 | # 14 | # @example Define a pattern file with an explicit destination filename. 15 | # logstash::patternfile { 'mypattern': 16 | # source => 'puppet:///path/to/my/custom/pattern', 17 | # filename => 'custom-pattern-name' 18 | # } 19 | # 20 | # @author https://github.com/elastic/puppet-logstash/graphs/contributors 21 | # 22 | define logstash::patternfile ( 23 | Optional[Pattern[/^(puppet|file):\/\//]] $source = undef, 24 | Optional[String[1]] $filename = undef, 25 | ) { 26 | require logstash::config 27 | 28 | $destination = pick($filename, basename($source)) 29 | 30 | file { "${logstash::config_dir}/patterns/${destination}": 31 | ensure => file, 32 | source => $source, 33 | owner => 'root', 34 | group => $logstash::logstash_group, 35 | mode => '0640', 36 | tag => ['logstash_config'], 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /spec/acceptance/define_patternfile_spec.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | require 'spec_helper_acceptance' 4 | 5 | describe 'class patternfile' do 6 | def apply_pattern(pattern_number, extra_logstash_class_args = nil) 7 | manifest = <<-END 8 | #{install_logstash_manifest(extra_logstash_class_args)} 9 | 10 | logstash::patternfile { 'pattern': 11 | source => 'puppet:///modules/logstash/grok-pattern-#{pattern_number}', 12 | filename => 'the_only_pattern_file', 13 | } 14 | END 15 | apply_manifest(manifest) 16 | end 17 | 18 | context 'when declaring a pattern file' do 19 | before(:context) { apply_pattern(0) } 20 | 21 | describe file '/etc/logstash/patterns/the_only_pattern_file' do 22 | it { is_expected.to be_a_file } 23 | its(:content) { is_expected.to match(%r{GROK_PATTERN_0}) } 24 | end 25 | end 26 | 27 | context 'with a pattern file in place' do 28 | before { apply_pattern(0) } 29 | 30 | restart_message = 'Scheduling refresh of Service[logstash]' 31 | 32 | it 'restarts logstash when a pattern file changes' do 33 | log = apply_pattern(1).stdout 34 | expect(log).to include(restart_message) 35 | end 36 | 37 | it 'does not restart logstash if logstash::restart_on_change is false' do 38 | log = apply_pattern(1, 'restart_on_change => false').stdout 39 | expect(log).not_to include(restart_message) 40 | end 41 | end 42 | end 43 | -------------------------------------------------------------------------------- /manifests/config.pp: -------------------------------------------------------------------------------- 1 | # This class manages configuration directories for Logstash. 2 | # 3 | # @example Include this class to ensure its resources are available. 4 | # include logstash::config 5 | # 6 | # @author https://github.com/elastic/puppet-logstash/graphs/contributors 7 | # 8 | class logstash::config { 9 | require logstash::package 10 | 11 | File { 12 | owner => 'root', 13 | group => 'root', 14 | } 15 | 16 | # Configuration "fragment" directories for pipeline config and pattern files. 17 | # We'll keep these seperate since we may want to "purge" them. It's easy to 18 | # end up with orphan files when managing config fragments with Puppet. 19 | # Purging the directories resolves the problem. 20 | 21 | if($logstash::ensure == 'present') { 22 | file { $logstash::config_dir: 23 | ensure => directory, 24 | mode => '0755', 25 | } 26 | 27 | file { "${logstash::config_dir}/conf.d": 28 | ensure => directory, 29 | purge => $logstash::purge_config, 30 | recurse => $logstash::purge_config, 31 | mode => '0775', 32 | notify => Service['logstash'], 33 | } 34 | 35 | file { "${logstash::config_dir}/patterns": 36 | ensure => directory, 37 | purge => $logstash::purge_config, 38 | recurse => $logstash::purge_config, 39 | mode => '0755', 40 | } 41 | } 42 | elsif($logstash::ensure == 'absent') { 43 | # Completely remove the config directory. ie. 'rm -rf /etc/logstash' 44 | file { $logstash::config_dir: 45 | ensure => 'absent', 46 | recurse => true, 47 | force => true, 48 | } 49 | } 50 | } 51 | -------------------------------------------------------------------------------- /.overcommit.yml: -------------------------------------------------------------------------------- 1 | # Managed by modulesync - DO NOT EDIT 2 | # https://voxpupuli.org/docs/updating-files-managed-with-modulesync/ 3 | # 4 | # Hooks are only enabled if you take action. 5 | # 6 | # To enable the hooks run: 7 | # 8 | # ``` 9 | # bundle exec overcommit --install 10 | # # ensure .overcommit.yml does not harm to you and then 11 | # bundle exec overcommit --sign 12 | # ``` 13 | # 14 | # (it will manage the .git/hooks directory): 15 | # 16 | # Examples howto skip a test for a commit or push: 17 | # 18 | # ``` 19 | # SKIP=RuboCop git commit 20 | # SKIP=PuppetLint git commit 21 | # SKIP=RakeTask git push 22 | # ``` 23 | # 24 | # Don't invoke overcommit at all: 25 | # 26 | # ``` 27 | # OVERCOMMIT_DISABLE=1 git commit 28 | # ``` 29 | # 30 | # Read more about overcommit: https://github.com/brigade/overcommit 31 | # 32 | # To manage this config yourself in your module add 33 | # 34 | # ``` 35 | # .overcommit.yml: 36 | # unmanaged: true 37 | # ``` 38 | # 39 | # to your modules .sync.yml config 40 | --- 41 | PreCommit: 42 | RuboCop: 43 | enabled: true 44 | description: 'Runs rubocop on modified files only' 45 | command: ['bundle', 'exec', 'rubocop'] 46 | RakeTarget: 47 | enabled: true 48 | description: 'Runs lint on modified files only' 49 | targets: 50 | - 'lint' 51 | command: ['bundle', 'exec', 'rake'] 52 | YamlSyntax: 53 | enabled: true 54 | JsonSyntax: 55 | enabled: true 56 | TrailingWhitespace: 57 | enabled: true 58 | 59 | PrePush: 60 | RakeTarget: 61 | enabled: true 62 | description: 'Run rake targets' 63 | targets: 64 | - 'validate' 65 | - 'test' 66 | - 'rubocop' 67 | command: ['bundle', 'exec', 'rake'] 68 | -------------------------------------------------------------------------------- /metadata.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "puppet-logstash", 3 | "version": "9.0.1-rc0", 4 | "source": "https://github.com/voxpupuli/puppet-logstash", 5 | "author": "Vox Pupuli", 6 | "license": "Apache-2.0", 7 | "summary": "Module for managing and configuring Logstash", 8 | "project_page": "https://github.com/voxpupuli/puppet-logstash", 9 | "dependencies": [ 10 | { 11 | "name": "puppetlabs/stdlib", 12 | "version_requirement": ">= 3.2.0 < 10.0.0" 13 | }, 14 | { 15 | "name": "puppet/elastic_stack", 16 | "version_requirement": ">= 6.0.0 < 11.0.0" 17 | } 18 | ], 19 | "operatingsystem_support": [ 20 | { 21 | "operatingsystem": "Debian", 22 | "operatingsystemrelease": [ 23 | "11", 24 | "12" 25 | ] 26 | }, 27 | { 28 | "operatingsystem": "OpenSuSE", 29 | "operatingsystemrelease": [ 30 | "15" 31 | ] 32 | }, 33 | { 34 | "operatingsystem": "RedHat", 35 | "operatingsystemrelease": [ 36 | "8", 37 | "9" 38 | ] 39 | }, 40 | { 41 | "operatingsystem": "CentOS", 42 | "operatingsystemrelease": [ 43 | "9" 44 | ] 45 | }, 46 | { 47 | "operatingsystem": "Rocky", 48 | "operatingsystemrelease": [ 49 | "8", 50 | "9" 51 | ] 52 | }, 53 | { 54 | "operatingsystem": "AlmaLinux", 55 | "operatingsystemrelease": [ 56 | "8", 57 | "9" 58 | ] 59 | }, 60 | { 61 | "operatingsystem": "OracleLinux", 62 | "operatingsystemrelease": [ 63 | "8", 64 | "9" 65 | ] 66 | }, 67 | { 68 | "operatingsystem": "SLES", 69 | "operatingsystemrelease": [ 70 | "15" 71 | ] 72 | }, 73 | { 74 | "operatingsystem": "Ubuntu", 75 | "operatingsystemrelease": [ 76 | "22.04", 77 | "24.04" 78 | ] 79 | } 80 | ], 81 | "requirements": [ 82 | { 83 | "name": "openvox", 84 | "version_requirement": ">= 8.19.0 < 9.0.0" 85 | } 86 | ] 87 | } 88 | -------------------------------------------------------------------------------- /manifests/configfile.pp: -------------------------------------------------------------------------------- 1 | # This type represents a Logstash pipeline configuration file. 2 | # 3 | # Parameters are mutually exclusive. Only one should be specified. 4 | # 5 | # @param [String] content 6 | # Literal content to be placed in the file. 7 | # 8 | # @param [String] template 9 | # A template from which to render the file. 10 | # 11 | # @param [String] source 12 | # A file resource to be used for the file. 13 | # 14 | # @param [String] path 15 | # An optional full path at which to create the file. 16 | # 17 | # @example Create a config file content with literal content. 18 | # 19 | # logstash::configfile { 'heartbeat': 20 | # content => 'input { heartbeat {} }', 21 | # } 22 | # 23 | # @example Render a config file from a template. 24 | # 25 | # logstash::configfile { 'from-template': 26 | # template => 'site-logstash-module/pipeline-config.erb', 27 | # } 28 | # 29 | # @example Copy the config from a file source. 30 | # 31 | # logstash::configfile { 'apache': 32 | # source => 'puppet://path/to/apache.conf', 33 | # } 34 | # 35 | # @example Create a config at specific location. Good for multiple pipelines. 36 | # 37 | # logstash::configfile { 'heartbeat-2': 38 | # content => 'input { heartbeat {} }', 39 | # path => '/usr/local/etc/logstash/pipeline-2/heartbeat.conf' 40 | # } 41 | # 42 | # @author https://github.com/elastic/puppet-logstash/graphs/contributors 43 | # 44 | define logstash::configfile ( 45 | $content = undef, 46 | $source = undef, 47 | $template = undef, 48 | $path = undef, 49 | ) { 50 | include logstash 51 | 52 | $owner = 'root' 53 | $group = $logstash::logstash_group 54 | $mode = '0640' 55 | $require = Package['logstash'] # So that we have '/etc/logstash/conf.d'. 56 | $tag = ['logstash_config'] # So that we notify the service. 57 | 58 | if($template) { $config = template($template) } 59 | elsif($content) { $config = $content } 60 | else { $config = undef } 61 | 62 | if($path) { $config_file = $path } 63 | else { $config_file = "${logstash::config_dir}/conf.d/${name}" } 64 | 65 | if($config) { 66 | file { $config_file: 67 | content => $config, 68 | owner => $owner, 69 | group => $group, 70 | mode => $mode, 71 | require => $require, 72 | tag => $tag, 73 | } 74 | } 75 | elsif($source) { 76 | file { $config_file: 77 | source => $source, 78 | owner => $owner, 79 | group => $group, 80 | mode => $mode, 81 | require => $require, 82 | tag => $tag, 83 | } 84 | } 85 | } 86 | -------------------------------------------------------------------------------- /spec/acceptance/define_configfile_spec.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | require 'spec_helper_acceptance' 4 | 5 | describe 'define logstash::configfile' do 6 | context 'with explicit content' do 7 | logstash_config = 'input { heartbeat {} }' 8 | 9 | manifest = <<-END 10 | logstash::configfile { 'heartbeat-input': 11 | content => '#{logstash_config}' 12 | } 13 | END 14 | 15 | before(:context) do 16 | apply_manifest(manifest, catch_failures: true) 17 | end 18 | 19 | it 'creates a file with the given content' do 20 | result = shell('cat /etc/logstash/conf.d/heartbeat-input').stdout 21 | expect(result).to eq(logstash_config) 22 | end 23 | end 24 | 25 | context 'with a template' do 26 | manifest = <<-END 27 | logstash::configfile { 'from-template': 28 | template => 'logstash/configfile-template.erb' 29 | } 30 | END 31 | 32 | before(:context) do 33 | skip('There is no configfile-template.erb in the module ..') 34 | apply_manifest(manifest, catch_failures: true, debug: true) 35 | end 36 | 37 | it 'creates a config file from the template' do 38 | skip('There is no configfile-template.erb in the module ..') 39 | result = shell('cat /etc/logstash/conf.d/from-template').stdout 40 | expect(result).to include('2 + 2 equals 4') 41 | end 42 | end 43 | 44 | context 'with a puppet:// url as source parameter' do 45 | manifest = <<-END 46 | logstash::configfile { 'null-output': 47 | source => 'puppet:///modules/logstash/null-output.conf' 48 | } 49 | END 50 | 51 | before(:context) do 52 | apply_manifest(manifest, catch_failures: true) 53 | end 54 | 55 | it 'places the config file' do 56 | result = shell('cat /etc/logstash/conf.d/null-output').stdout 57 | expect(result).to include('Test output configuration with null output.') 58 | end 59 | end 60 | 61 | context 'with an explicit path parameter' do 62 | logstash_config = 'input { heartbeat { message => "right here"} }' 63 | path = '/tmp/explicit-path.cfg' 64 | 65 | manifest = <<-END 66 | logstash::configfile { 'heartbeat-input': 67 | content => '#{logstash_config}', 68 | path => '#{path}', 69 | } 70 | END 71 | 72 | before(:context) do 73 | apply_manifest(manifest, catch_failures: true) 74 | end 75 | 76 | it 'creates a file with the given content at the correct path' do 77 | result = shell("cat #{path}").stdout 78 | expect(result).to eq(logstash_config) 79 | end 80 | end 81 | end 82 | -------------------------------------------------------------------------------- /spec/acceptance/class_plugin_spec.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | require 'spec_helper_acceptance' 4 | 5 | describe 'class plugin' do 6 | before(:all) do 7 | remove_logstash 8 | install_logstash('status => "disabled", restart_on_change => false') 9 | end 10 | 11 | def ensure_plugin(present_absent, plugin, extra_args = nil) 12 | manifest = <<-END 13 | class { 'logstash': 14 | status => 'disabled', 15 | restart_on_change => false, 16 | } 17 | -> 18 | logstash::plugin { '#{plugin}': 19 | ensure => #{present_absent}, 20 | #{extra_args} 21 | } 22 | END 23 | apply_manifest(manifest, catch_failures: true) 24 | end 25 | 26 | def installed_plugins 27 | shell('/usr/share/logstash/bin/logstash-plugin list').stdout 28 | end 29 | 30 | def remove(plugin) 31 | shell("/usr/share/logstash/bin/logstash-plugin remove #{plugin} || true") 32 | end 33 | 34 | context 'when a plugin is not installed' do 35 | before do 36 | remove('logstash-input-sqs') 37 | end 38 | 39 | it 'will not remove it again' do 40 | log = ensure_plugin('absent', 'logstash-input-sqs').stdout 41 | expect(log).not_to contain('remove-logstash-input-sqs') 42 | end 43 | 44 | it 'can install it from rubygems' do 45 | ensure_plugin('present', 'logstash-input-sqs') 46 | expect(installed_plugins).to contain('logstash-input-sqs') 47 | end 48 | end 49 | 50 | context 'when a plugin is installed' do 51 | it 'will contain the required plugin' do 52 | expect(installed_plugins).to contain('logstash-input-file') 53 | end 54 | 55 | it 'will not install it again' do 56 | log = ensure_plugin('present', 'logstash-input-file').stdout 57 | expect(log).not_to contain('install-logstash-input-file') 58 | end 59 | 60 | it 'can remove it' do 61 | ensure_plugin('absent', 'logstash-input-file') 62 | expect(installed_plugins).not_to contain('logstash-input-file') 63 | end 64 | end 65 | 66 | if Gem::Version.new(LS_VERSION) >= Gem::Version.new('5.2.0') 67 | it 'can install x-pack from an https url' do 68 | skip('The latest x-pack release is 6.2.4 released April 17, 2018 ...') 69 | plugin = 'x-pack' 70 | source = "https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-#{LS_VERSION}.zip" 71 | ensure_plugin('present', plugin, "source => '#{source}'") 72 | expect(installed_plugins).to contain(plugin) 73 | end 74 | end 75 | 76 | it 'can install a plugin from a "puppet://" url' do 77 | skip('There is no plugins embedded in the module ...') 78 | plugin = 'logstash-output-cowthink' 79 | source = "puppet:///modules/logstash/#{plugin}-5.0.0.gem" 80 | ensure_plugin('present', plugin, "source => '#{source}'") 81 | expect(installed_plugins).to contain(plugin) 82 | end 83 | 84 | it 'can install a plugin from a local gem' do 85 | skip('No download means no local plugin available ...') 86 | plugin = 'logstash-output-cowsay' 87 | source = "/tmp/#{plugin}-5.0.0.gem" 88 | ensure_plugin('present', plugin, "source => '#{source}'") 89 | expect(installed_plugins).to contain(plugin) 90 | end 91 | 92 | it 'can install a plugin from an offline zip' do 93 | skip('There is no plugins embedded in the module ...') 94 | plugin = 'logstash-output-cowsay' 95 | source = "puppet:///modules/logstash/#{plugin}-5.0.0.zip" 96 | ensure_plugin('present', plugin, "source => '#{source}'") 97 | expect(installed_plugins).to contain(plugin) 98 | end 99 | end 100 | -------------------------------------------------------------------------------- /manifests/plugin.pp: -------------------------------------------------------------------------------- 1 | # Manage the installation of a Logstash plugin. 2 | # 3 | # By default, plugins are downloaded from RubyGems, but it is also possible 4 | # to install from a local Gem, or one stored in Puppet. 5 | # 6 | # @example Install a plugin. 7 | # logstash::plugin { 'logstash-input-stdin': } 8 | # 9 | # @example Remove a plugin. 10 | # logstash::plugin { 'logstash-input-stout': 11 | # ensure => absent, 12 | # } 13 | # 14 | # @example Install a plugin from a local file. 15 | # logstash::plugin { 'logstash-input-custom': 16 | # source => 'file:///tmp/logstash-input-custom.gem', 17 | # } 18 | # 19 | # @example Install a plugin from a Puppet module. 20 | # logstash::plugin { 'logstash-input-custom': 21 | # source => 'puppet:///modules/logstash-site-plugins/logstash-input-custom.gem', 22 | # } 23 | # 24 | # @example Install X-Pack. 25 | # logstash::plugin { 'x-pack': 26 | # source => 'https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.3.0.zip', 27 | # } 28 | # 29 | # @example Install a plugin, overriding JVM options via the environment. 30 | # logstash::plugin { 'logstash-input-jmx': 31 | # environment => ['LS_JAVA_OPTS=-Xms1g -Xmx1g'] 32 | # } 33 | # 34 | # @param ensure [String] Install or remove with `present` or `absent`. 35 | # 36 | # @param source [String] Install from this file, not from RubyGems. 37 | # 38 | # @param user User to install plugin as. 39 | # 40 | # @param environment [String] Environment used when running 'logstash-plugin' 41 | # 42 | define logstash::plugin ( 43 | $source = undef, 44 | $ensure = present, 45 | $environment = [], 46 | String $user = 'root', 47 | ) { 48 | require logstash::package 49 | $exe = "${logstash::home_dir}/bin/logstash-plugin" 50 | 51 | Exec { 52 | path => '/bin:/usr/bin', 53 | cwd => '/tmp', 54 | user => $user, 55 | timeout => 1800, 56 | environment => $environment, 57 | } 58 | 59 | case $source { # Where should we get the plugin from? 60 | undef: { 61 | # No explict source, so search Rubygems for the plugin, by name. 62 | # ie. "logstash-plugin install logstash-output-elasticsearch" 63 | $plugin = $name 64 | } 65 | 66 | /^(\/|file:)/: { 67 | # A gem file that is already available on the local filesystem. 68 | # Install from the local path. 69 | # ie. "logstash-plugin install /tmp/logtash-filter-custom.gem" or 70 | # "logstash-plugin install file:///tmp/logtash-filter-custom.gem" or 71 | $plugin = $source 72 | } 73 | 74 | /^puppet:/: { 75 | # A 'puppet:///' URL. Download the gem from Puppet, then install 76 | # the plugin from the downloaded file. 77 | $downloaded_file = sprintf('/tmp/%s', basename($source)) 78 | file { $downloaded_file: 79 | source => $source, 80 | before => Exec["install-${name}"], 81 | } 82 | 83 | case $source { 84 | /\.zip$/: { 85 | $plugin = "file://${downloaded_file}" 86 | } 87 | default: { 88 | $plugin = $downloaded_file 89 | } 90 | } 91 | } 92 | 93 | /^https?:/: { 94 | # An 'http(s):///' URL. 95 | $plugin = $source 96 | } 97 | 98 | default: { 99 | fail('"source" should be a local path, a "puppet:///" url, or undef.') 100 | } 101 | } 102 | 103 | case $ensure { 104 | 'present': { 105 | exec { "install-${name}": 106 | command => "${exe} install ${plugin}", 107 | unless => "${exe} list ^${name}$", 108 | } 109 | } 110 | 111 | /^\d+\.\d+\.\d+/: { 112 | exec { "install-${name}": 113 | command => "${exe} install --version ${ensure} ${plugin}", 114 | unless => "${exe} list --verbose ^${name}$ | grep --fixed-strings --quiet '(${ensure})'", 115 | } 116 | } 117 | 118 | 'absent': { 119 | exec { "remove-${name}": 120 | command => "${exe} remove ${name}", 121 | onlyif => "${exe} list | grep -q ^${name}$", 122 | } 123 | } 124 | 125 | default: { 126 | fail "'ensure' should be 'present', 'absent', or a version like '1.3.4'." 127 | } 128 | } 129 | } 130 | -------------------------------------------------------------------------------- /manifests/package.pp: -------------------------------------------------------------------------------- 1 | # This class manages the Logstash package. 2 | # 3 | # It is usually used only by the top-level `logstash` class. It's unlikely 4 | # that you will need to declare this class yourself. 5 | # 6 | # @param [String] package_name 7 | # The name of the Logstash package in the package manager. 8 | # 9 | # @param [String] version 10 | # Install precisely this version from the package manager. 11 | # 12 | # @param [String] package_url 13 | # Get the package from this URL, not from the package manager. 14 | # 15 | # @example Include this class to ensure its resources are available. 16 | # include logstash::package 17 | # 18 | # @author https://github.com/elastic/puppet-logstash/graphs/contributors 19 | # 20 | class logstash::package ( 21 | $package_url = $logstash::package_url, 22 | $version = $logstash::version, 23 | $package_name = $logstash::package_name, 24 | ) { 25 | Exec { 26 | path => ['/bin', '/usr/bin', '/usr/local/bin'], 27 | cwd => '/', 28 | tries => 3, 29 | try_sleep => 10, 30 | } 31 | 32 | File { 33 | ensure => file, 34 | backup => false, 35 | } 36 | 37 | if $logstash::ensure == 'present' { 38 | # Check if we want to install a specific version. 39 | if $version { 40 | if $facts['os']['family'] == 'redhat' { 41 | # Prerelease RPM packages have tildes ("~") in their version strings, 42 | # which can be quite surprising to the user. Let them say: 43 | # 6.0.0-rc2 44 | # not: 45 | # 6.0.0~rc2 46 | $package_ensure = regsubst($version, '(\d+)-(alpha|beta|rc)(\d+)$', '\1~\2\3') 47 | } 48 | else { 49 | $package_ensure = $version 50 | } 51 | } 52 | else { 53 | $package_ensure = $logstash::auto_upgrade ? { 54 | true => 'latest', 55 | false => 'present', 56 | } 57 | } 58 | 59 | if ($package_url) { 60 | $filename = basename($package_url) 61 | $extension = regsubst($filename, '.*\.', '') 62 | $protocol = regsubst($package_url, ':.*', '') 63 | $package_local_file = "/tmp/${filename}" 64 | 65 | case $protocol { 66 | 'puppet': { 67 | file { $package_local_file: 68 | source => $package_url, 69 | } 70 | } 71 | 'ftp', 'https', 'http': { 72 | exec { "download_package_logstash_${name}": 73 | command => "wget -O ${package_local_file} ${package_url} 2> /dev/null", 74 | path => ['/usr/bin', '/bin'], 75 | creates => $package_local_file, 76 | timeout => $logstash::download_timeout, 77 | } 78 | } 79 | 'file': { 80 | file { $package_local_file: 81 | source => $package_url, 82 | } 83 | } 84 | default: { 85 | fail("Protocol must be puppet, file, http, https, or ftp. Not '${protocol}'") 86 | } 87 | } 88 | 89 | case $extension { 90 | 'deb': { $package_provider = 'dpkg' } 91 | 'rpm': { $package_provider = 'rpm' } 92 | default: { fail("Unknown file extension '${extension}'.") } 93 | } 94 | 95 | $package_require = undef 96 | } 97 | else { 98 | # Use the OS packaging system to locate the package. 99 | $package_local_file = undef 100 | $package_provider = undef 101 | if $facts['os']['family'] == 'Debian' { 102 | $package_require = $logstash::manage_repo ? { 103 | true => Class['apt::update'], 104 | false => undef, 105 | } 106 | } else { 107 | $package_require = undef 108 | } 109 | } 110 | } 111 | else { # Package removal 112 | $package_local_file = undef 113 | $package_require = undef 114 | if ($facts['os']['family'] == 'Suse') { 115 | $package_provider = 'rpm' 116 | $package_ensure = 'absent' # "purged" not supported by provider 117 | } 118 | else { 119 | $package_provider = undef # ie. automatic 120 | $package_ensure = 'purged' 121 | } 122 | } 123 | 124 | package { 'logstash': 125 | ensure => $package_ensure, 126 | name => $package_name, 127 | source => $package_local_file, # undef if using package manager. 128 | provider => $package_provider, # undef if using package manager. 129 | require => $package_require, 130 | } 131 | } 132 | -------------------------------------------------------------------------------- /manifests/service.pp: -------------------------------------------------------------------------------- 1 | # This mangages the system service for Logstash. 2 | # 3 | # It is usually used only by the top-level `logstash` class. It's unlikely 4 | # that you will need to declare this class yourself. 5 | # 6 | # @example Include this class to ensure its resources are available. 7 | # include logstash::service 8 | # 9 | # @author https://github.com/elastic/puppet-logstash/graphs/contributors 10 | # 11 | class logstash::service { 12 | $default_settings = { 13 | 'path.data' => '/var/lib/logstash', 14 | 'path.config' => '/etc/logstash/conf.d', 15 | 'path.logs' => '/var/log/logstash', 16 | } 17 | 18 | $default_startup_options = { 19 | 'LS_HOME' => $logstash::home_dir, 20 | 'LS_SETTINGS_DIR' => $logstash::config_dir, 21 | 'LS_OPTS' => "--path.settings=${logstash::config_dir}", 22 | 'LS_JAVA_OPTS' => '""', 23 | 'LS_PIDFILE' => '/var/run/logstash.pid', 24 | 'LS_USER' => $logstash::logstash_user, 25 | 'LS_GROUP' => $logstash::logstash_group, 26 | 'LS_GC_LOG_FILE' => '/var/log/logstash/gc.log', 27 | 'LS_OPEN_FILES' => '16384', 28 | 'LS_NICE' => '19', 29 | 'SERVICE_NAME' => '"logstash"', 30 | 'SERVICE_DESCRIPTION' => '"logstash"', 31 | } 32 | 33 | $settings = $default_settings + $logstash::settings 34 | $startup_options = $default_startup_options + $logstash::startup_options 35 | $jvm_options = $logstash::jvm_options 36 | $jvm_options_defaults = $logstash::jvm_options_defaults 37 | $pipelines = $logstash::pipelines 38 | 39 | File { 40 | owner => 'root', 41 | group => 'root', 42 | mode => '0644', 43 | notify => Exec['logstash-system-install'], 44 | } 45 | 46 | if $logstash::ensure == 'present' { 47 | case $logstash::status { 48 | 'enabled': { 49 | $service_ensure = 'running' 50 | $service_enable = true 51 | } 52 | 'disabled': { 53 | $service_ensure = 'stopped' 54 | $service_enable = false 55 | } 56 | 'running': { 57 | $service_ensure = 'running' 58 | $service_enable = false 59 | } 60 | default: { 61 | fail("\"${logstash::status}\" is an unknown service status value") 62 | } 63 | } 64 | } else { 65 | $service_ensure = 'stopped' 66 | $service_enable = false 67 | } 68 | 69 | if $service_ensure == 'running' { 70 | # Then make sure the Logstash startup options are up to date. 71 | file { '/etc/logstash/startup.options': 72 | content => template('logstash/startup.options.erb'), 73 | } 74 | 75 | # ..and make sure the JVM options are up to date. 76 | file { '/etc/logstash/jvm.options': 77 | content => template('logstash/jvm.options.erb'), 78 | } 79 | 80 | # ..and pipelines.yml, if the user provided such. If they didn't, zero out 81 | # the file, which will default Logstash to traditional single-pipeline 82 | # behaviour. 83 | if(empty($pipelines)) { 84 | file { '/etc/logstash/pipelines.yml': 85 | content => '', 86 | } 87 | } 88 | else { 89 | file { '/etc/logstash/pipelines.yml': 90 | content => template('logstash/pipelines.yml.erb'), 91 | } 92 | } 93 | 94 | # ..and the Logstash internal settings too. 95 | file { '/etc/logstash/logstash.yml': 96 | content => template('logstash/logstash.yml.erb'), 97 | } 98 | 99 | # Invoke 'system-install', which generates startup scripts based on the 100 | # contents of the 'startup.options' file. 101 | # Only if restart_on_change is not false 102 | if $logstash::restart_on_change { 103 | exec { 'logstash-system-install': 104 | command => "${logstash::home_dir}/bin/system-install", 105 | refreshonly => true, 106 | notify => Service['logstash'], 107 | } 108 | } else { 109 | exec { 'logstash-system-install': 110 | command => "${logstash::home_dir}/bin/system-install", 111 | refreshonly => true, 112 | } 113 | } 114 | } 115 | 116 | service { 'logstash': 117 | ensure => $service_ensure, 118 | enable => $service_enable, 119 | hasstatus => true, 120 | hasrestart => true, 121 | provider => $logstash::service_provider, 122 | } 123 | 124 | # If any files tagged as config files for the service are changed, notify 125 | # the service so it restarts. 126 | if $logstash::restart_on_change { 127 | File<| tag == 'logstash_config' |> ~> Service['logstash'] 128 | Logstash::Plugin<| |> ~> Service['logstash'] 129 | } 130 | } 131 | -------------------------------------------------------------------------------- /spec/support/acceptance/helpers.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | require 'yaml' 4 | 5 | # Collect global options from the environment. 6 | LS_VERSION = ENV['LOGSTASH_VERSION'] || '7.17.7' 7 | IS_PRERELEASE = if LS_VERSION =~ %r{(alpha|beta|rc)} 8 | true 9 | else 10 | false 11 | end 12 | 13 | def expect_no_change_from_manifest(manifest) 14 | expect(apply_manifest(manifest).exit_code).to eq(0) 15 | end 16 | 17 | def http_package_url 18 | url_root = "https://artifacts.elastic.co/downloads/logstash/logstash-#{LS_VERSION}" 19 | 20 | case fact('osfamily') 21 | when 'Debian' 22 | "#{url_root}-amd64.deb" 23 | when 'RedHat', 'Suse' 24 | "#{url_root}-x86_64.rpm" 25 | end 26 | end 27 | 28 | def local_file_package_url 29 | "file:///tmp/#{logstash_package_filename}" 30 | end 31 | 32 | def puppet_fileserver_package_url 33 | "puppet:///modules/logstash/#{logstash_package_filename}" 34 | end 35 | 36 | def logstash_package_filename 37 | File.basename(http_package_url) 38 | end 39 | 40 | def logstash_package_version 41 | package_version = if LS_VERSION =~ %r{(alpha|beta|rc)} 42 | LS_VERSION.gsub('-', '~') 43 | else 44 | LS_VERSION 45 | end 46 | 47 | case fact('osfamily') # FIXME: Put this logic in the module, not the tests. 48 | when 'RedHat' 49 | "#{package_version}-1" 50 | when 'Debian', 'Suse' 51 | "1:#{package_version}-1" 52 | end 53 | end 54 | 55 | def logstash_config_manifest 56 | <<-END 57 | logstash::configfile { 'basic_config': 58 | content => 'input { tcp { port => 2000 } } output { null {} }' 59 | } 60 | END 61 | end 62 | 63 | def install_logstash_manifest(extra_args = nil) 64 | <<-END 65 | class { 'elastic_stack::repo': 66 | version => #{LS_VERSION[0]}, 67 | prerelease => #{IS_PRERELEASE}, 68 | } 69 | class { 'logstash': 70 | manage_repo => true, 71 | version => '#{logstash_package_version}', 72 | #{extra_args} 73 | } 74 | 75 | #{logstash_config_manifest} 76 | END 77 | end 78 | 79 | def include_logstash_manifest 80 | <<-END 81 | class { 'elastic_stack::repo': 82 | version => #{LS_VERSION[0]}, 83 | prerelease => #{IS_PRERELEASE}, 84 | } 85 | 86 | include logstash 87 | 88 | #{logstash_config_manifest} 89 | END 90 | end 91 | 92 | def install_logstash_from_url_manifest(url, extra_args = nil) 93 | <<-END 94 | class { 'logstash': 95 | package_url => '#{url}', 96 | #{extra_args} 97 | } 98 | 99 | #{logstash_config_manifest} 100 | END 101 | end 102 | 103 | def install_logstash_from_local_file_manifest(extra_args = nil) 104 | install_logstash_from_url_manifest(local_file_package_url, extra_args) 105 | end 106 | 107 | def remove_logstash_manifest 108 | "class { 'logstash': ensure => 'absent' }" 109 | end 110 | 111 | def stop_logstash_manifest 112 | "class { 'logstash': status => 'disabled' }" 113 | end 114 | 115 | # Provide a basic Logstash install. Useful as a testing pre-requisite. 116 | def install_logstash(extra_args = nil) 117 | result = apply_manifest(install_logstash_manifest(extra_args), catch_failures: true) 118 | sleep 5 # FIXME: This is horrible. 119 | result 120 | end 121 | 122 | def include_logstash 123 | result = apply_manifest(include_logstash_manifest, catch_failures: true) 124 | sleep 5 # FIXME: This is horrible. 125 | result 126 | end 127 | 128 | def install_logstash_from_url(url, extra_args = nil) 129 | manifest = install_logstash_from_url_manifest(url, extra_args) 130 | result = apply_manifest(manifest, catch_failures: true) 131 | sleep 5 # FIXME: This is horrible. 132 | result 133 | end 134 | 135 | def install_logstash_from_local_file(extra_args = nil) 136 | install_logstash_from_url(local_file_package_url, extra_args) 137 | end 138 | 139 | def remove_logstash 140 | result = apply_manifest(remove_logstash_manifest) 141 | sleep 5 # FIXME: This is horrible. 142 | result 143 | end 144 | 145 | def stop_logstash 146 | result = apply_manifest(stop_logstash_manifest, catch_failures: true) 147 | shell('ps -eo comm | grep java | xargs kill -9', accept_all_exit_codes: true) 148 | sleep 5 # FIXME: This is horrible. 149 | result 150 | end 151 | 152 | def logstash_process_list 153 | ps_cmd = 'ps -ww --no-headers -C java -o user,command | grep logstash' 154 | shell(ps_cmd, accept_all_exit_codes: true).stdout.split("\n") 155 | end 156 | 157 | def logstash_settings 158 | YAML.safe_load(shell('cat /etc/logstash/logstash.yml').stdout) 159 | end 160 | 161 | def expect_setting(setting, value) 162 | expect(logstash_settings[setting]).to eq(value) 163 | end 164 | 165 | def pipelines_from_yaml 166 | YAML.safe_load(shell('cat /etc/logstash/pipelines.yml').stdout) 167 | end 168 | 169 | def service_restart_message 170 | "Service[logstash]: Triggered 'refresh'" 171 | end 172 | -------------------------------------------------------------------------------- /manifests/init.pp: -------------------------------------------------------------------------------- 1 | # This class manages installation, configuration and execution of Logstash 5.x. 2 | # 3 | # @param [String] ensure 4 | # Controls if Logstash should be `present` or `absent`. 5 | # 6 | # If set to `absent`, the Logstash package will be 7 | # uninstalled. Related files will be purged as much as possible. The 8 | # exact behavior is dependant on the service provider, specifically its 9 | # support for the 'purgable' property. 10 | # 11 | # @param [Boolean] auto_upgrade 12 | # If set to `true`, Logstash will be upgraded if the package provider is 13 | # able to find a newer version. The exact behavior is dependant on the 14 | # service provider, specifically its support for the 'upgradeable' property. 15 | # 16 | # @param [String] status 17 | # The desired state of the Logstash service. Possible values: 18 | # 19 | # - `enabled`: Service running and started at boot time. 20 | # - `disabled`: Service stopped and not started at boot time. 21 | # - `running`: Service running but not be started at boot time. 22 | # - `unmanaged`: Service will not be started at boot time. Puppet 23 | # will neither stop nor start the service. 24 | # 25 | # @param [String] version 26 | # The specific version to install, if desired. 27 | # 28 | # @param [Boolean] restart_on_change 29 | # Restart the service whenever the configuration changes. 30 | # 31 | # Disabling automatic restarts on config changes may be desired in an 32 | # environment where you need to ensure restarts occur in a 33 | # controlled/rolling manner rather than during a Puppet run. 34 | # 35 | # @param [String] package_url 36 | # Explict Logstash package URL to download. 37 | # 38 | # Valid URL types are: 39 | # - `http://` 40 | # - `https://` 41 | # - `ftp://` 42 | # - `puppet://` 43 | # - `file:/` 44 | # 45 | # @param [String] package_name 46 | # The name of the Logstash package in the package manager. 47 | # 48 | # @param [Integer] download_timeout 49 | # Timeout, in seconds, for http, https, and ftp downloads. 50 | # 51 | # @param home_dir 52 | # The home directory for logstash. 53 | # 54 | # @param [String] logstash_user 55 | # The user that Logstash should run as. This also controls file ownership. 56 | # 57 | # @param [String] logstash_group 58 | # The group that Logstash should run as. This also controls file group ownership. 59 | # 60 | # @param [Boolean] purge_config 61 | # Purge the config directory of any unmanaged files, 62 | # 63 | # @param [String] service_provider 64 | # Service provider (init system) to use. By Default, the module will try to 65 | # choose the 'standard' provider for the current distribution. 66 | # 67 | # @param [Hash] settings 68 | # A collection of settings to be defined in `logstash.yml`. 69 | # 70 | # See: https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html 71 | # 72 | # @param [Hash] startup_options 73 | # A collection of settings to be defined in `startup.options`. 74 | # 75 | # See: https://www.elastic.co/guide/en/logstash/current/config-setting-files.html 76 | # 77 | # @param [Hash] jvm_options_defaults 78 | # Default set of optionname => option mappings from upstream 8.5 version 79 | # 80 | # @param [Array] jvm_options 81 | # A collection of settings to be defined in `jvm.options`. Override same settings in jvm_options_defaults 82 | # 83 | # @param [Array] pipelines 84 | # A collection of settings to be defined in `pipelines.yml`. 85 | # 86 | # @param [Boolean] manage_repo 87 | # Enable repository management. Configure the official repositories. 88 | # 89 | # @param [String] config_dir 90 | # Path containing the Logstash configuration. 91 | # 92 | # @example Install Logstash, ensure the service is running and enabled. 93 | # class { 'logstash': } 94 | # 95 | # @example Remove Logstash. 96 | # class { 'logstash': 97 | # ensure => 'absent', 98 | # } 99 | # 100 | # @example Install everything but disable the service. 101 | # class { 'logstash': 102 | # status => 'disabled', 103 | # } 104 | # 105 | # @example Configure Logstash settings. 106 | # class { 'logstash': 107 | # settings => { 108 | # 'http.port' => '9700', 109 | # } 110 | # } 111 | # 112 | # @example Configure Logstash startup options. 113 | # class { 'logstash': 114 | # startup_options => { 115 | # 'LS_USER' => 'root', 116 | # } 117 | # } 118 | # 119 | # @example Set JVM memory options. 120 | # class { 'logstash': 121 | # jvm_options => [ 122 | # '-Xms1g', 123 | # '-Xmx1g', 124 | # ] 125 | # } 126 | # 127 | # @example Configure multiple pipelines. 128 | # class { 'logstash': 129 | # pipelines => [ 130 | # { 131 | # "pipeline.id" => "my-pipeline_1", 132 | # "path.config" => "/etc/path/to/p1.config", 133 | # }, 134 | # { 135 | # "pipeline.id" => "my-other-pipeline", 136 | # "path.config" => "/etc/different/path/p2.cfg", 137 | # } 138 | # ] 139 | # } 140 | # 141 | # @author https://github.com/elastic/puppet-logstash/graphs/contributors 142 | # 143 | class logstash ( 144 | $ensure = 'present', 145 | $status = 'enabled', 146 | Boolean $restart_on_change = true, 147 | Boolean $auto_upgrade = false, 148 | $version = undef, 149 | $package_url = undef, 150 | $package_name = 'logstash', 151 | Integer $download_timeout = 600, 152 | Stdlib::Absolutepath $home_dir = '/usr/share/logstash', 153 | $logstash_user = 'logstash', 154 | $logstash_group = 'logstash', 155 | $config_dir = '/etc/logstash', 156 | Boolean $purge_config = true, 157 | Optional[String[1]] $service_provider = undef, 158 | $settings = {}, 159 | $startup_options = {}, 160 | $jvm_options_defaults = { 161 | '-Xms' => '-Xms1g', 162 | '-Xmx' => '-Xmx1g', 163 | 'UseConcMarkSweepGC' => '11-13:-XX:+UseConcMarkSweepGC', 164 | 'CMSInitiatingOccupancyFraction=' => '11-13:-XX:CMSInitiatingOccupancyFraction=75', 165 | 'UseCMSInitiatingOccupancyOnly' => '11-13:-XX:+UseCMSInitiatingOccupancyOnly', 166 | '-Djava.awt.headless=' => '-Djava.awt.headless=true', 167 | '-Dfile.encoding=' => '-Dfile.encoding=UTF-8', 168 | 'HeapDumpOnOutOfMemoryError' => '-XX:+HeapDumpOnOutOfMemoryError', 169 | '-Djava.security.egd' => '-Djava.security.egd=file:/dev/urandom', 170 | }, 171 | $jvm_options = [], 172 | Array $pipelines = [], 173 | Boolean $manage_repo = true, 174 | ) { 175 | if ! ($ensure in ['present', 'absent']) { 176 | fail("\"${ensure}\" is not a valid ensure parameter value") 177 | } 178 | 179 | if ! ($status in ['enabled', 'disabled', 'running', 'unmanaged']) { 180 | fail("\"${status}\" is not a valid status parameter value") 181 | } 182 | 183 | if ($manage_repo == true) { 184 | include elastic_stack::repo 185 | } 186 | include logstash::package 187 | include logstash::config 188 | include logstash::service 189 | } 190 | -------------------------------------------------------------------------------- /HISTORY.md: -------------------------------------------------------------------------------- 1 | ## [6.1.5](https://github.com/voxpupuli/puppet-logstash/tree/6.1.5) (2018-11-13) 2 | 3 | Honour `restart_on_change` for startup files #377 4 | 5 | 6.1.4 6 | Use Upstart for init on Oracle Linux 6 7 | 8 | 6.1.3 9 | Allow puppetlabs-stdlib 5.x 10 | 11 | 6.1.2 12 | Set cwd to "/tmp" during plugin execs. 13 | 14 | 6.1.1 15 | Update init system handling. 16 | 17 | 6.1.0 18 | Support centralized pipeline management. 19 | 20 | 6.0.1 21 | Fix explicit versions like "5.6.2-1.noarch" #353 22 | 23 | 6.0.0 24 | Puppet 3 support removed. Minimum Puppet version is now 4.6.1. 25 | Puppet 5 supported. 26 | Logstash 6.x (and 5.x) supported. 27 | File permissions for config files now match those from the Logstash package. 28 | elastic/elastic_stack module is now used to manage repositories. 29 | Logstash multiple pipelines supported. 30 | Config files resources accept an explicit target path. 31 | 32 | 5.3.0 33 | Allow setting environment for plugin resources 34 | 35 | 5.2.0 36 | Allow 'http(s):// URLs for plugin install. 37 | 38 | 5.1.0 39 | Make config files be owned by root. 40 | Allow 'file://' URLs for plugin install. 41 | Sort lines in jvm.options for file. 42 | 43 | 5.0.4 44 | Expose $logstash::home_dir 45 | 46 | 5.0.3 47 | Sort startup_options to prevent triggering unneeded restarts. 48 | 49 | 5.0.2 50 | Do not autmatically add ".conf" to pipeline config filesnames. 51 | 52 | 5.0.1 53 | Trivial README update. 54 | 55 | 5.0.0 56 | Major re-write for Logstash 5.x. 57 | Drop support for Logstash <5.0.0. 58 | 59 | 0.6.4 60 | Puppet 4 support. 61 | 62 | 0.6.3 63 | Documentation updates only. Functionally identical to 0.6.2. 64 | 65 | 0.6.2 66 | Allow electrical/file_concat version 1.x. 67 | 68 | 0.6.1 69 | Restart service on pattern file change. 70 | Remove dependency on external PGP server. 71 | Fix circular dependency on plugin installation. 72 | 0.6.0 73 | Deprecates the logstash-contrib package. 74 | Supports Gem-based Logstash plugins. 75 | Not compatible with Logstash versions < 1.5.0. 76 | 77 | 0.5.1 78 | Updated system tests to work with LS 1.4.1 79 | Increase package download timeout 80 | Add option to use stages for the repo setup instead anchors 81 | 82 | 0.5.0 83 | Move beaker testing to use docker 84 | Large module update to work with the contrib package 85 | Refactored rspec testing 86 | Fix inline docs 87 | Added Puppet 3.5.0 testing 88 | Fix typo in group name variable ( PR #147 ) 89 | Improve puppet module removal ( PR #149 ) 90 | Reverted PR #149. Caused issues with package removal. 91 | added lsbdistid = Debian to rspec facts ( PR #146 ) 92 | match other config perms with patterns ( PR #151 ) 93 | 94 | 0.4.3 95 | Lower puppetlabs-stdlib depdency from 4.0.0 to 3.2.0 96 | Documentation improvements ( PR #132 #137 ) 97 | Fixed yumrepo call to include description ( PR #138 ) 98 | Added beaker testing 99 | Fixed bug that sometimes LS starts before all configs are processed. 100 | Ensure java is installed before installing the package when using package_url 101 | Fail fast when using package_url and repo config 102 | 103 | 0.4.2 104 | Fix config directory for config files to be inline with the init file of the packages 105 | Update readme ( thanks to PR #130 from phrawzty ) 106 | Added repo management ( based on work of PR #121 from pcfens ) 107 | 108 | 0.4.1 109 | ** Important Update ** 110 | Ensure exec names are unique. This caused an issue when using the Elasticsearch Puppet module 111 | Removed a part in the package.pp that should have been removed ( missed with the rewrite ) 112 | Missed a few bits of the rewrite. 113 | Updated readme to reflect reality regarding configfile define. 114 | 115 | 0.4.0 116 | ** NOTE: This is a backwards compability breaking release !! ** 117 | Large rewrite of the entire module described below 118 | Make the core more dynamic for different service providers 119 | Add better testing and devided into different files 120 | Add different ways to install the package except from the repository ( puppet/http/https/ftp/file ) 121 | Update java class to install openjdk 1.7 122 | Add validation of templates 123 | Added more test scenario's 124 | Added puppet parser validate task for added checking 125 | Improve module removing when set to absent 126 | Updated readme 127 | Doc improvements by dan ( phrawzty ) 128 | Added define for managing pattern files 129 | Added define for managing plugins 130 | 131 | 0.3.4 132 | Fixing purging of created directories ( PR #61, #64 by Kayla Green and Jason Koppe ) 133 | Documentation fixes ( PR #65, #67 by Kristian Glass and Andreas Paul ) 134 | Making config dir configurable ( PR #70 by Justin Lambert ) 135 | Permit HTTP(s) for downloading logstash ( PR #71 by Phil Fenstermacher ) 136 | Ensure user/group is passed in the debian init file 137 | Spec cleanup ( PR #75 by Justin Lambert ) 138 | set logstash logdir perms when using custom jar provider ( PR #74 by Justin Lambert ) 139 | clean up installpath when updating jars ( PR #72 by Justin Lambert ) 140 | fix wrong creates path at jar custom provider ( PR #83 by Daniel Werdermann ) 141 | added 'in progress' for logstash version 1.2.x ( PR #87 by rtoma ) 142 | Add small input/output examples ( PR #89 by Andreas Paul ) 143 | Solving defaults file not being installed in some cases 144 | http download of jar should require $jardir ( PR #90 by Max Griffiths ) 145 | add ability to install a logstash config file ( PR #93 by Justin Lambert ) 146 | 147 | 0.3.3 148 | Enable puppet 3.2.x testing 149 | Fix issue that the config dir was missing in the init files 150 | Fix variable access deprecation warning ( PR #56 by Richard Peng ) 151 | 152 | 0.3.2 153 | Fixing issue when using jar file without multi-instance feature 154 | Added rspec tests to cover this issue 155 | 156 | 0.3.1 157 | Missed changes for enabling/disabling multi-instance feature 158 | Adding a few spec tests for the multi-instance feature 159 | 160 | 0.3.0 161 | Update defines for Logstash 1.1.12 162 | Adding license file 163 | Deleted old init file removal to avoid issues. ( Issue #50 ) 164 | Allow file owner/group to be variable ( Issue/PR #47 ) 165 | Ensure log directory exists before starting ( PR #53 by Brian Lalor ) 166 | Provide complete containment of the class ( PR #53 by Brian Lalor ) 167 | Update rspec tests for new defines 168 | 169 | 0.2.0 170 | Update defines for logstash 1.1.10 171 | New feature for plugins to automatically transfer files ( Issue #24 ) 172 | Create correct tmp dir ( Issue #35 ) 173 | Change file modes to be more secure ( Issue #36 ) 174 | Update defines for better input validation ( Issue #43 ) 175 | Adding rspec tests for plugin defines 176 | Fix tmp dir Debian init script ( PR #44 by Dan Carley ) 177 | 178 | 0.1.0 179 | Don't backup the Jar file or the symlink ( Issue #25 by Garth Kidd ) 180 | First implementation of the multi-instance feature. This will break certain functionality. 181 | 182 | 0.0.6 183 | Fix issue that the init file was overwritten 184 | Ensure we install java first before starting logstash if enabled 185 | 186 | 0.0.5 187 | Adding spec tests 188 | Update Readme ( PR #20 by rjw1 ) 189 | New feature to install java 190 | 191 | 0.0.4 192 | Rename Redhat to RedHat for init file ( PR #12 by pkubat ) 193 | Adding Amazon as Operating system ( PR #12 by pkubat ) 194 | Symlinking Jar file to generic name ( PR #12 by pkubat ) 195 | Correting symlink ( PR #14 by Jeff Wong ) 196 | 197 | 0.0.3 198 | Clarify jarfile usage and validation ( PR #6 by Garth Kidd ) 199 | Add default Debian Init script when non provided and using custom source ( PR #7 by Garth Kidd ) 200 | Add RedHat as OS type ( PR #8 by Dan ) 201 | Skip init script when status = unmanaged ( PR #9 by Tavis Aitken ) 202 | Refactored the custom provider part ( With help of Garth Kidd ) 203 | 204 | 0.0.2 205 | Adding a way to provide jar and init file instead of depending on a package 206 | 207 | 0.0.1 208 | Initial release of the module 209 | -------------------------------------------------------------------------------- /spec/acceptance/class_logstash_spec.rb: -------------------------------------------------------------------------------- 1 | # frozen_string_literal: true 2 | 3 | require 'spec_helper_acceptance' 4 | 5 | shared_examples 'a logstash installer' do 6 | it "installs logstash version #{LS_VERSION}" do 7 | expect(shell('/usr/share/logstash/bin/logstash --version').stdout).to contain("logstash #{LS_VERSION}") 8 | end 9 | 10 | case fact('osfamily') 11 | when 'RedHat', 'Suse' 12 | describe package('logstash') do 13 | it { is_expected.to be_installed } 14 | end 15 | when 'Debian' 16 | # Serverspec has been falsely reporting the package as not installed on 17 | # Debian 7, so we'll implement our own version of "should be_installed". 18 | it "installs logstash package version #{logstash_package_version}" do 19 | apt_output = shell('apt-cache policy logstash').stdout 20 | expect(apt_output).to include("Installed: #{logstash_package_version}") 21 | end 22 | end 23 | 24 | describe service('logstash') do 25 | it { is_expected.to be_running } 26 | 27 | it 'be_enableds' do 28 | skip('Serverspec seems confused about this on Centos 6.') if fact('lsbdistdescription') =~ %r{centos release 6}i 29 | is_expected.to be_enabled 30 | end 31 | end 32 | 33 | it 'spawns a single logstash process' do 34 | expect(logstash_process_list.length).to eq(1) 35 | end 36 | 37 | it 'runs logstash as the "logstash" user' do 38 | expect(logstash_process_list.pop).to match(%r{^logstash }) 39 | end 40 | end 41 | 42 | describe 'class logstash' do 43 | describe 'ensure => present' do 44 | context 'with include-like declaration' do 45 | before(:all) do 46 | remove_logstash 47 | include_logstash 48 | end 49 | 50 | it_behaves_like 'a logstash installer' 51 | 52 | it 'is idempotent' do 53 | expect_no_change_from_manifest(install_logstash_manifest) 54 | end 55 | end 56 | 57 | context 'when installing from an http url' do 58 | before(:all) do 59 | remove_logstash 60 | install_logstash_from_url(http_package_url) 61 | end 62 | 63 | it_behaves_like 'a logstash installer' 64 | end 65 | 66 | context 'when installing from a local file' do 67 | before(:all) do 68 | remove_logstash 69 | install_logstash_from_local_file 70 | end 71 | 72 | it_behaves_like 'a logstash installer' 73 | end 74 | 75 | context 'when installing from a "puppet://" url' do 76 | before(:all) do 77 | skip('There is no rpm package in the module ...') 78 | remove_logstash 79 | install_logstash_from_url(puppet_fileserver_package_url) 80 | end 81 | 82 | skip('There is no rpm package in the module ...') 83 | it_behaves_like 'a logstash installer' 84 | end 85 | end 86 | 87 | describe 'ensure => absent' do 88 | before(:context) do 89 | install_logstash_from_local_file 90 | remove_logstash 91 | end 92 | 93 | it 'is idempotent' do 94 | expect_no_change_from_manifest(remove_logstash_manifest) 95 | end 96 | 97 | describe package('logstash') do 98 | it { is_expected.not_to be_installed } 99 | end 100 | 101 | describe service('logstash') do 102 | it { is_expected.not_to be_running } 103 | 104 | it 'is not enabled' do 105 | is_expected.not_to be_enabled 106 | end 107 | end 108 | end 109 | 110 | describe 'service_provider parameter' do 111 | context "with service_provider => 'openrc'" do 112 | it 'tries to use openrc as the service provider' do 113 | log = apply_manifest('class {"logstash": service_provider => "openrc"}').stderr 114 | expect(log).to include('Provider openrc is not functional on this host') 115 | end 116 | end 117 | end 118 | 119 | describe 'settings parameter' do 120 | context 'with a flat key' do 121 | before(:context) do 122 | settings = "{ 'http.port' => '9999' }" 123 | install_logstash_from_local_file("settings => #{settings}") 124 | end 125 | 126 | it 'sets "http.port" to "9999"' do 127 | expect_setting('http.port', '9999') 128 | end 129 | 130 | it 'retains the default "path.data" setting' do 131 | expect_setting('path.data', '/var/lib/logstash') 132 | end 133 | 134 | it 'retains the default "path.config" setting' do 135 | expect_setting('path.config', '/etc/logstash/conf.d') 136 | end 137 | 138 | it 'retains the default "path.logs" setting' do 139 | expect_setting('path.logs', '/var/log/logstash') 140 | end 141 | end 142 | 143 | context 'with a hierarchical key' do 144 | before(:context) do 145 | settings = <<-END 146 | { 147 | 'pipeline' => { 148 | 'batch' => { 149 | 'size' => 99 150 | } 151 | } 152 | } 153 | END 154 | install_logstash_from_local_file("settings => #{settings}") 155 | end 156 | 157 | it 'sets pipeline batch size to 99 hierarchically' do 158 | # FIXME: Some Puppet versions put the string "99" in the rendered YAML 159 | # when it should really be the integer value 99. Logstash is OK with 160 | # either representation, so we get away with it, but it's not correct. 161 | expect(logstash_settings['pipeline']['batch']['size'].to_i).to eq(99) 162 | end 163 | end 164 | 165 | context 'when a default setting exists' do 166 | before(:context) do 167 | settings = "{ 'path.logs' => '/tmp' }" 168 | install_logstash_from_local_file("settings => #{settings}") 169 | end 170 | 171 | it 'can override the default' do 172 | expect_setting('path.logs', '/tmp') 173 | end 174 | end 175 | end 176 | 177 | describe 'startup_options parameter' do 178 | context "with 'LS_USER' => 'root'" do 179 | before do 180 | remove_logstash 181 | startup_options = "{ 'LS_USER' => 'root' }" 182 | install_logstash_from_local_file("startup_options => #{startup_options}") 183 | end 184 | 185 | it 'runs logstash as root' do 186 | expect(logstash_process_list.pop).to match(%r{^root }) 187 | end 188 | end 189 | end 190 | 191 | describe 'jvm_options parameter' do 192 | context "with '-Xms1g'" do 193 | before(:context) do 194 | jvm_options = "[ '-Xms1g' ]" 195 | install_logstash_from_local_file("jvm_options => #{jvm_options}") 196 | end 197 | 198 | it 'runs java with -Xms1g' do 199 | expect(logstash_process_list.pop).to include('-Xms1g') 200 | end 201 | 202 | it 'does not run java with the default of -Xms256m' do 203 | expect(logstash_process_list.pop).not_to include('-Xms256m') 204 | end 205 | 206 | it 'runs java with the default "expert" flags' do 207 | expert_flags = [ 208 | '-Dfile.encoding=UTF-8', 209 | '-Djava.awt.headless=true', 210 | '-XX:CMSInitiatingOccupancyFraction=75', 211 | '-XX:+HeapDumpOnOutOfMemoryError', 212 | '-XX:+UseCMSInitiatingOccupancyOnly', 213 | ] 214 | expert_flags.each do |flag| 215 | expect(logstash_process_list.pop).to include(flag) 216 | end 217 | end 218 | 219 | context 'when the option is changed' do 220 | it 'restarts logstash' do 221 | puppet_log = install_logstash_from_local_file( 222 | "jvm_options => [ '-Xms512m' ]" 223 | ).stdout 224 | expect(puppet_log).to include(service_restart_message) 225 | end 226 | 227 | context 'when restart_on_change is false' do 228 | it 'does not restart logstash' do 229 | puppet_log = install_logstash_from_local_file( 230 | "jvm_options => [ '-Xms256m' ], 231 | restart_on_change => false" 232 | ).stdout 233 | expect(puppet_log).not_to include(service_restart_message) 234 | end 235 | end 236 | end 237 | end 238 | end 239 | 240 | describe 'pipelines_parameter' do 241 | context 'with pipelines declared' do 242 | before(:context) do 243 | pipelines_puppet = <<-END 244 | [ 245 | { 246 | "pipeline.id" => "pipeline_one", 247 | "path.config" => "/etc/path/to/p1.config", 248 | }, 249 | { 250 | "pipeline.id" => "pipeline_two", 251 | "path.config" => "/etc/different/path/p2.cfg", 252 | } 253 | ] 254 | END 255 | install_logstash_from_local_file("pipelines => #{pipelines_puppet}") 256 | end 257 | 258 | it 'renders them to pipelines.yml' do 259 | expect(pipelines_from_yaml[0]['pipeline.id']).to eq('pipeline_one') 260 | expect(pipelines_from_yaml[1]['pipeline.id']).to eq('pipeline_two') 261 | end 262 | 263 | it 'removes "path.config" from "logstash.yml"' do 264 | expect(logstash_settings['path.config']).to be_nil 265 | end 266 | end 267 | end 268 | 269 | describe 'xpack_management_enabled_parameter' do 270 | context 'when set true with dotted notation' do 271 | before(:context) do 272 | settings_puppet_code = '{"xpack.management.enabled" => true}' 273 | install_logstash_from_local_file("settings => #{settings_puppet_code}") 274 | end 275 | 276 | it 'removes "path.config" from "logstash.yml"' do 277 | expect(logstash_settings['path.config']).to be_nil 278 | end 279 | end 280 | 281 | context 'when set true with hierarchical notation' do 282 | before(:context) do 283 | settings_puppet_code = <<-END 284 | { 285 | "xpack" => { 286 | "management" => { 287 | "enabled" => true 288 | } 289 | } 290 | } 291 | END 292 | install_logstash_from_local_file("settings => #{settings_puppet_code}") 293 | end 294 | 295 | it 'removes "path.config" from "logstash.yml"' do 296 | expect(logstash_settings['path.config']).to be_nil 297 | end 298 | end 299 | end 300 | end 301 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | 2 | Apache License 3 | Version 2.0, January 2004 4 | http://www.apache.org/licenses/ 5 | 6 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 7 | 8 | 1. Definitions. 9 | 10 | "License" shall mean the terms and conditions for use, reproduction, 11 | and distribution as defined by Sections 1 through 9 of this document. 12 | 13 | "Licensor" shall mean the copyright owner or entity authorized by 14 | the copyright owner that is granting the License. 15 | 16 | "Legal Entity" shall mean the union of the acting entity and all 17 | other entities that control, are controlled by, or are under common 18 | control with that entity. For the purposes of this definition, 19 | "control" means (i) the power, direct or indirect, to cause the 20 | direction or management of such entity, whether by contract or 21 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 22 | outstanding shares, or (iii) beneficial ownership of such entity. 23 | 24 | "You" (or "Your") shall mean an individual or Legal Entity 25 | exercising permissions granted by this License. 26 | 27 | "Source" form shall mean the preferred form for making modifications, 28 | including but not limited to software source code, documentation 29 | source, and configuration files. 30 | 31 | "Object" form shall mean any form resulting from mechanical 32 | transformation or translation of a Source form, including but 33 | not limited to compiled object code, generated documentation, 34 | and conversions to other media types. 35 | 36 | "Work" shall mean the work of authorship, whether in Source or 37 | Object form, made available under the License, as indicated by a 38 | copyright notice that is included in or attached to the work 39 | (an example is provided in the Appendix below). 40 | 41 | "Derivative Works" shall mean any work, whether in Source or Object 42 | form, that is based on (or derived from) the Work and for which the 43 | editorial revisions, annotations, elaborations, or other modifications 44 | represent, as a whole, an original work of authorship. For the purposes 45 | of this License, Derivative Works shall not include works that remain 46 | separable from, or merely link (or bind by name) to the interfaces of, 47 | the Work and Derivative Works thereof. 48 | 49 | "Contribution" shall mean any work of authorship, including 50 | the original version of the Work and any modifications or additions 51 | to that Work or Derivative Works thereof, that is intentionally 52 | submitted to Licensor for inclusion in the Work by the copyright owner 53 | or by an individual or Legal Entity authorized to submit on behalf of 54 | the copyright owner. For the purposes of this definition, "submitted" 55 | means any form of electronic, verbal, or written communication sent 56 | to the Licensor or its representatives, including but not limited to 57 | communication on electronic mailing lists, source code control systems, 58 | and issue tracking systems that are managed by, or on behalf of, the 59 | Licensor for the purpose of discussing and improving the Work, but 60 | excluding communication that is conspicuously marked or otherwise 61 | designated in writing by the copyright owner as "Not a Contribution." 62 | 63 | "Contributor" shall mean Licensor and any individual or Legal Entity 64 | on behalf of whom a Contribution has been received by Licensor and 65 | subsequently incorporated within the Work. 66 | 67 | 2. Grant of Copyright License. Subject to the terms and conditions of 68 | this License, each Contributor hereby grants to You a perpetual, 69 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 70 | copyright license to reproduce, prepare Derivative Works of, 71 | publicly display, publicly perform, sublicense, and distribute the 72 | Work and such Derivative Works in Source or Object form. 73 | 74 | 3. Grant of Patent License. Subject to the terms and conditions of 75 | this License, each Contributor hereby grants to You a perpetual, 76 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 77 | (except as stated in this section) patent license to make, have made, 78 | use, offer to sell, sell, import, and otherwise transfer the Work, 79 | where such license applies only to those patent claims licensable 80 | by such Contributor that are necessarily infringed by their 81 | Contribution(s) alone or by combination of their Contribution(s) 82 | with the Work to which such Contribution(s) was submitted. If You 83 | institute patent litigation against any entity (including a 84 | cross-claim or counterclaim in a lawsuit) alleging that the Work 85 | or a Contribution incorporated within the Work constitutes direct 86 | or contributory patent infringement, then any patent licenses 87 | granted to You under this License for that Work shall terminate 88 | as of the date such litigation is filed. 89 | 90 | 4. Redistribution. You may reproduce and distribute copies of the 91 | Work or Derivative Works thereof in any medium, with or without 92 | modifications, and in Source or Object form, provided that You 93 | meet the following conditions: 94 | 95 | (a) You must give any other recipients of the Work or 96 | Derivative Works a copy of this License; and 97 | 98 | (b) You must cause any modified files to carry prominent notices 99 | stating that You changed the files; and 100 | 101 | (c) You must retain, in the Source form of any Derivative Works 102 | that You distribute, all copyright, patent, trademark, and 103 | attribution notices from the Source form of the Work, 104 | excluding those notices that do not pertain to any part of 105 | the Derivative Works; and 106 | 107 | (d) If the Work includes a "NOTICE" text file as part of its 108 | distribution, then any Derivative Works that You distribute must 109 | include a readable copy of the attribution notices contained 110 | within such NOTICE file, excluding those notices that do not 111 | pertain to any part of the Derivative Works, in at least one 112 | of the following places: within a NOTICE text file distributed 113 | as part of the Derivative Works; within the Source form or 114 | documentation, if provided along with the Derivative Works; or, 115 | within a display generated by the Derivative Works, if and 116 | wherever such third-party notices normally appear. The contents 117 | of the NOTICE file are for informational purposes only and 118 | do not modify the License. You may add Your own attribution 119 | notices within Derivative Works that You distribute, alongside 120 | or as an addendum to the NOTICE text from the Work, provided 121 | that such additional attribution notices cannot be construed 122 | as modifying the License. 123 | 124 | You may add Your own copyright statement to Your modifications and 125 | may provide additional or different license terms and conditions 126 | for use, reproduction, or distribution of Your modifications, or 127 | for any such Derivative Works as a whole, provided Your use, 128 | reproduction, and distribution of the Work otherwise complies with 129 | the conditions stated in this License. 130 | 131 | 5. Submission of Contributions. Unless You explicitly state otherwise, 132 | any Contribution intentionally submitted for inclusion in the Work 133 | by You to the Licensor shall be under the terms and conditions of 134 | this License, without any additional terms or conditions. 135 | Notwithstanding the above, nothing herein shall supersede or modify 136 | the terms of any separate license agreement you may have executed 137 | with Licensor regarding such Contributions. 138 | 139 | 6. Trademarks. This License does not grant permission to use the trade 140 | names, trademarks, service marks, or product names of the Licensor, 141 | except as required for reasonable and customary use in describing the 142 | origin of the Work and reproducing the content of the NOTICE file. 143 | 144 | 7. Disclaimer of Warranty. Unless required by applicable law or 145 | agreed to in writing, Licensor provides the Work (and each 146 | Contributor provides its Contributions) on an "AS IS" BASIS, 147 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 148 | implied, including, without limitation, any warranties or conditions 149 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 150 | PARTICULAR PURPOSE. You are solely responsible for determining the 151 | appropriateness of using or redistributing the Work and assume any 152 | risks associated with Your exercise of permissions under this License. 153 | 154 | 8. Limitation of Liability. In no event and under no legal theory, 155 | whether in tort (including negligence), contract, or otherwise, 156 | unless required by applicable law (such as deliberate and grossly 157 | negligent acts) or agreed to in writing, shall any Contributor be 158 | liable to You for damages, including any direct, indirect, special, 159 | incidental, or consequential damages of any character arising as a 160 | result of this License or out of the use or inability to use thes 161 | Work (including but not limited to damages for loss of goodwill, 162 | work stoppage, computer failure or malfunction, or any and all 163 | other commercial damages or losses), even if such Contributor 164 | has been advised of the possibility of such damages. 165 | 166 | 9. Accepting Warranty or Additional Liability. While redistributing 167 | the Work or Derivative Works thereof, You may choose to offer, 168 | and charge a fee for, acceptance of support, warranty, indemnity, 169 | or other liability obligations and/or rights consistent with this 170 | License. However, in accepting such obligations, You may act only 171 | on Your own behalf and on Your sole responsibility, not on behalf 172 | of any other Contributor, and only if You agree to indemnify, 173 | defend, and hold each Contributor harmless for any liability 174 | incurred by, or claims asserted against, such Contributor by reason 175 | of your accepting any such warranty or additional liability. 176 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # logstash 2 | 3 | [![Build Status](https://github.com/voxpupuli/puppet-logstash/workflows/CI/badge.svg)](https://github.com/voxpupuli/puppet-logstash/actions?query=workflow%3ACI) 4 | [![Release](https://github.com/voxpupuli/puppet-logstash/actions/workflows/release.yml/badge.svg)](https://github.com/voxpupuli/puppet-logstash/actions/workflows/release.yml) 5 | [![Puppet Forge](https://img.shields.io/puppetforge/v/puppet/logstash.svg)](https://forge.puppetlabs.com/puppet/logstash) 6 | [![Puppet Forge - downloads](https://img.shields.io/puppetforge/dt/puppet/logstash.svg)](https://forge.puppetlabs.com/puppet/logstash) 7 | [![Puppet Forge - endorsement](https://img.shields.io/puppetforge/e/puppet/logstash.svg)](https://forge.puppetlabs.com/puppet/logstash) 8 | [![Puppet Forge - scores](https://img.shields.io/puppetforge/f/puppet/logstash.svg)](https://forge.puppetlabs.com/puppet/logstash) 9 | [![puppetmodule.info docs](http://www.puppetmodule.info/images/badge.png)](http://www.puppetmodule.info/m/puppet-logstash) 10 | [![Apache-2 License](https://img.shields.io/github/license/voxpupuli/puppet-logstash.svg)](LICENSE) 11 | [![Donated by Elastic](https://img.shields.io/badge/donated%20by-Elastic-fb7047.svg)](#transfer-notice) 12 | 13 | A Puppet module for managing and configuring [Logstash](https://www.elastic.co/logstash/). 14 | 15 | Version 7 and newer of this module are released by Vox Pupuli. They now follow semantic versioning. Previously the module was maintained by Elastic. 16 | 17 | ## Requirements 18 | 19 | * The [stdlib](https://forge.puppetlabs.com/puppetlabs/stdlib) module. 20 | * Logstash < 7.0.0 requires Java. The [puppetlabs/java](https://forge.puppetlabs.com/modules/puppetlabs/java) module is recommended 21 | for installing Java. 22 | 23 | Optional: 24 | * The [elastic_stack](https://forge.puppetlabs.com/elastic/elastic_stack) module 25 | when using automatic repository management. 26 | * The [apt](https://forge.puppetlabs.com/puppetlabs/apt) (>= 2.0.0) module when 27 | using repo management on Debian/Ubuntu. 28 | * The [zypprepo](https://forge.puppetlabs.com/darin/zypprepo) module when using 29 | repo management on SLES/SuSE. 30 | 31 | ## Quick Start 32 | 33 | This minimum viable configuration ensures that Logstash is installed, enabled, and running: 34 | 35 | ``` puppet 36 | include logstash 37 | 38 | # You must provide a valid pipeline configuration for the service to start. 39 | logstash::configfile { 'my_ls_config': 40 | content => template('path/to/config.file'), 41 | } 42 | ``` 43 | 44 | ## Package and service options 45 | ### Choosing a Logstash major version 46 | 47 | This module uses the related "elastic/elastic_stack" module to manage package 48 | repositories. Since there is a separate repository for each major version of 49 | the Elastic stack, if you don't want the default version, it's necessary 50 | to select which version to configure, like this: 51 | ``` puppet 52 | class { 'elastic_stack::repo': 53 | version => 6, 54 | } 55 | 56 | include logstash 57 | ``` 58 | 59 | ### Choosing a Logstash minor version 60 | ``` puppet 61 | class { 'logstash': 62 | version => '6.8.0', 63 | } 64 | ``` 65 | ### Manual repository management 66 | You may want to manage repositories manually. You can disable 67 | automatic repository management like this: 68 | 69 | ``` puppet 70 | class { 'logstash': 71 | manage_repo => false, 72 | } 73 | ``` 74 | 75 | ### Using an explicit package source 76 | Rather than use your distribution's repository system, you can specify an 77 | explicit package to fetch and install. 78 | 79 | #### From an HTTP/HTTPS/FTP URL 80 | ``` puppet 81 | class { 'logstash': 82 | package_url => 'https://artifacts.elastic.co/downloads/logstash/logstash-7.17.8-x86_64.rpm', 83 | } 84 | ``` 85 | 86 | #### From a 'puppet://' URL 87 | ``` puppet 88 | class { 'logstash': 89 | package_url => 'puppet:///modules/my_module/logstash-7.17.8-x86_64.rpm', 90 | } 91 | ``` 92 | 93 | #### From a local file on the agent 94 | ``` puppet 95 | class { 'logstash': 96 | package_url => 'file:///tmp/logstash-7.17.8-x86_64.rpm', 97 | } 98 | ``` 99 | 100 | ### Allow automatic point-release upgrades 101 | ``` puppet 102 | class { 'logstash': 103 | auto_upgrade => true, 104 | } 105 | ``` 106 | 107 | ### Use a different logstash home 108 | ``` puppet 109 | class { 'logstash': 110 | home_dir => '/opt/logstash', 111 | } 112 | ``` 113 | 114 | ### Do not run as a service 115 | ``` puppet 116 | class { 'logstash': 117 | status => 'disabled', 118 | } 119 | ``` 120 | 121 | ### Disable automatic restarts 122 | Under normal circumstances, changing a configuration will trigger a restart of 123 | the service. This behaviour can be disabled: 124 | ``` puppet 125 | class { 'logstash': 126 | restart_on_change => false, 127 | } 128 | ``` 129 | 130 | ### Disable and remove Logstash 131 | ``` puppet 132 | class { 'logstash': 133 | ensure => 'absent', 134 | } 135 | ``` 136 | 137 | ## Logstash config files 138 | 139 | ### Settings 140 | 141 | Logstash uses several files to define settings for the service and associated 142 | Java runtime. The settings files can be configured with class parameters. 143 | 144 | #### `logstash.yml` with flat keys 145 | ``` puppet 146 | class { 'logstash': 147 | settings => { 148 | 'pipeline.batch.size' => 25, 149 | 'pipeline.batch.delay' => 5, 150 | } 151 | } 152 | ``` 153 | 154 | #### `logstash.yml` with nested keys 155 | ``` puppet 156 | class { 'logstash': 157 | settings => { 158 | 'pipeline' => { 159 | 'batch' => { 160 | 'size' => 25, 161 | 'delay' => 5, 162 | } 163 | } 164 | } 165 | } 166 | ``` 167 | 168 | #### `jvm.options` 169 | ``` puppet 170 | class { 'logstash': 171 | jvm_options => [ 172 | '-Xms1g', 173 | '-Xmx1g', 174 | ] 175 | } 176 | ``` 177 | 178 | #### `startup.options` 179 | 180 | ``` puppet 181 | class { 'logstash': 182 | startup_options => { 183 | 'LS_NICE' => '10', 184 | } 185 | } 186 | ``` 187 | 188 | #### `pipelines.yml` 189 | 190 | ``` puppet 191 | class { 'logstash': 192 | pipelines => [ 193 | { 194 | "pipeline.id" => "pipeline_one", 195 | "path.config" => "/usr/local/etc/logstash/pipeline-1/one.conf", 196 | }, 197 | { 198 | "pipeline.id" => "pipeline_two", 199 | "path.config" => "/usr/local/etc/logstash/pipeline-2/two.conf", 200 | } 201 | ] 202 | } 203 | ``` 204 | 205 | Note that specifying `pipelines` will automatically remove the default 206 | `path.config` setting from `logstash.yml`, since this is incompatible with 207 | `pipelines.yml`. 208 | 209 | Enabling centralized pipeline management with `xpack.management.enabled` will 210 | also remove the default `path.config`. 211 | 212 | ### Pipeline Configuration 213 | Pipeline configuration files can be declared with the `logstash::configfile` 214 | type. 215 | 216 | ``` puppet 217 | logstash::configfile { 'inputs': 218 | content => template('path/to/input.conf.erb'), 219 | } 220 | ``` 221 | or 222 | ``` puppet 223 | logstash::configfile { 'filters': 224 | source => 'puppet:///path/to/filter.conf', 225 | } 226 | ``` 227 | 228 | For simple cases, it's possible to provide your Logstash config as an 229 | inline string: 230 | 231 | ``` puppet 232 | logstash::configfile { 'basic_ls_config': 233 | content => 'input { heartbeat {} } output { null {} }', 234 | } 235 | ``` 236 | 237 | You can also specify the exact path for the config file, which is 238 | particularly useful with multiple pipelines: 239 | 240 | ``` puppet 241 | logstash::configfile { 'config_for_pipeline_two': 242 | content => 'input { heartbeat {} } output { null {} }', 243 | path => '/usr/local/etc/logstash/pipeline-2/two.conf', 244 | } 245 | ``` 246 | 247 | If you want to use Hiera to specify your configs, include the following 248 | create_resources call in your manifest: 249 | 250 | ``` puppet 251 | create_resources('logstash::configfile', hiera('my_logstash_configs')) 252 | ``` 253 | ...and then create a data structure like this in Hiera: 254 | ``` yaml 255 | --- 256 | my_logstash_configs: 257 | nginx: 258 | template: site_logstash/nginx.conf.erb 259 | syslog: 260 | template: site_logstash/syslog.conf.erb 261 | ``` 262 | 263 | In this example, templates for the config files are stored in the custom, 264 | site-specific module "`site_logstash`". 265 | 266 | ### Patterns 267 | Many plugins (notably [Grok](http://logstash.net/docs/latest/filters/grok)) use *patterns*. While many are included in Logstash already, additional site-specific patterns can be managed as well. 268 | 269 | ``` puppet 270 | logstash::patternfile { 'extra_patterns': 271 | source => 'puppet:///path/to/extra_pattern', 272 | } 273 | ``` 274 | 275 | By default the resulting filename of the pattern will match that of the source. This can be over-ridden: 276 | ``` puppet 277 | logstash::patternfile { 'extra_patterns_firewall': 278 | source => 'puppet:///path/to/extra_patterns_firewall_v1', 279 | filename => 'extra_patterns_firewall', 280 | } 281 | ``` 282 | 283 | **IMPORTANT NOTE**: Using logstash::patternfile places new patterns in the correct directory, however, it does NOT cause the path to be included automatically for filters (example: grok filter). You will still need to include this path (by default, /etc/logstash/patterns/) explicitly in your configurations. 284 | 285 | Example: If using 'grok' in one of your configurations, you must include the pattern path in each filter like this: 286 | 287 | ``` 288 | # Note: this example is Logstash configuration, not a Puppet resource. 289 | # Logstash and Puppet look very similar! 290 | grok { 291 | patterns_dir => "/etc/logstash/patterns/" 292 | ... 293 | } 294 | ``` 295 | 296 | ## Plugin management 297 | 298 | ### Installing by name (from RubyGems.org) 299 | ``` puppet 300 | logstash::plugin { 'logstash-input-beats': } 301 | ``` 302 | 303 | ### Installing from a local Gem 304 | ``` puppet 305 | logstash::plugin { 'logstash-input-custom': 306 | source => '/tmp/logstash-input-custom-0.1.0.gem', 307 | } 308 | ``` 309 | 310 | ### Installing from a 'puppet://' URL 311 | ``` puppet 312 | logstash::plugin { 'logstash-filter-custom': 313 | source => 'puppet:///modules/my_ls_module/logstash-filter-custom-0.1.0.gem', 314 | } 315 | ``` 316 | 317 | ### Installing from an 'http(s)://' URL 318 | ``` puppet 319 | logstash::plugin { 'x-pack': 320 | source => 'https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.3.0.zip', 321 | } 322 | ``` 323 | 324 | ### Controling the environment for the `logstash-plugin` command 325 | ``` puppet 326 | logstash::plugin { 'logstash-input-websocket': 327 | environment => 'LS_JAVA_OPTS=-Xms1g -Xmx1g', 328 | } 329 | ``` 330 | 331 | ## Transfer Notice 332 | 333 | This module was originally authored by [Elastic](https://www.elastic.co). 334 | The maintainer preferred that Vox Pupuli take ownership of the module for future improvement and maintenance. 335 | Existing pull requests and issues were transferred over, please fork and continue to contribute here instead of Elastic. 336 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | All notable changes to this project will be documented in this file. 4 | Each new release typically also includes the latest modulesync defaults. 5 | These should not affect the functionality of the module. 6 | 7 | ## [v9.0.0](https://github.com/voxpupuli/puppet-logstash/tree/v9.0.0) (2025-06-27) 8 | 9 | [Full Changelog](https://github.com/voxpupuli/puppet-logstash/compare/v8.1.0...v9.0.0) 10 | 11 | **Breaking changes:** 12 | 13 | - Drop Ubuntu 18.04 and 20.04 [\#463](https://github.com/voxpupuli/puppet-logstash/pull/463) ([h-haaks](https://github.com/h-haaks)) 14 | - Drop CentOS 7 and 8 [\#462](https://github.com/voxpupuli/puppet-logstash/pull/462) ([h-haaks](https://github.com/h-haaks)) 15 | - Drop RedHat 7 [\#461](https://github.com/voxpupuli/puppet-logstash/pull/461) ([h-haaks](https://github.com/h-haaks)) 16 | - Drop Debian 10 [\#460](https://github.com/voxpupuli/puppet-logstash/pull/460) ([h-haaks](https://github.com/h-haaks)) 17 | - Remove EOL SLES 12 [\#457](https://github.com/voxpupuli/puppet-logstash/pull/457) ([thomas-merz](https://github.com/thomas-merz)) 18 | 19 | **Implemented enhancements:** 20 | 21 | - Add Ubuntu 24.04 [\#465](https://github.com/voxpupuli/puppet-logstash/pull/465) ([h-haaks](https://github.com/h-haaks)) 22 | - Allow puppet/elastic\_stack 10.x [\#459](https://github.com/voxpupuli/puppet-logstash/pull/459) ([h-haaks](https://github.com/h-haaks)) 23 | - metadata.json: Add OpenVox [\#455](https://github.com/voxpupuli/puppet-logstash/pull/455) ([jstraw](https://github.com/jstraw)) 24 | 25 | **Closed issues:** 26 | 27 | - module gets warnings when used with stdlib9. [\#428](https://github.com/voxpupuli/puppet-logstash/issues/428) 28 | 29 | ## [v8.1.0](https://github.com/voxpupuli/puppet-logstash/tree/v8.1.0) (2024-03-26) 30 | 31 | [Full Changelog](https://github.com/voxpupuli/puppet-logstash/compare/v8.0.0...v8.1.0) 32 | 33 | **Implemented enhancements:** 34 | 35 | - Add Rocky/AlmaLinux/OracleLinux support [\#442](https://github.com/voxpupuli/puppet-logstash/pull/442) ([bastelfreak](https://github.com/bastelfreak)) 36 | - Add Ubuntu 22.04 support [\#441](https://github.com/voxpupuli/puppet-logstash/pull/441) ([bastelfreak](https://github.com/bastelfreak)) 37 | - Add EL9 support [\#440](https://github.com/voxpupuli/puppet-logstash/pull/440) ([bastelfreak](https://github.com/bastelfreak)) 38 | - Add Debian 12 support [\#439](https://github.com/voxpupuli/puppet-logstash/pull/439) ([bastelfreak](https://github.com/bastelfreak)) 39 | - Don't hardcode the service provider [\#438](https://github.com/voxpupuli/puppet-logstash/pull/438) ([bastelfreak](https://github.com/bastelfreak)) 40 | - replace legacy `merge()` with native puppet code [\#437](https://github.com/voxpupuli/puppet-logstash/pull/437) ([bastelfreak](https://github.com/bastelfreak)) 41 | 42 | ## [v8.0.0](https://github.com/voxpupuli/puppet-logstash/tree/v8.0.0) (2024-01-30) 43 | 44 | [Full Changelog](https://github.com/voxpupuli/puppet-logstash/compare/v7.0.0...v8.0.0) 45 | 46 | **Breaking changes:** 47 | 48 | - Drop Puppet 6 support [\#424](https://github.com/voxpupuli/puppet-logstash/pull/424) ([bastelfreak](https://github.com/bastelfreak)) 49 | 50 | **Implemented enhancements:** 51 | 52 | - bump elastic\_stack max version to \<10.0.0 [\#432](https://github.com/voxpupuli/puppet-logstash/pull/432) ([sandwitch](https://github.com/sandwitch)) 53 | - Add Puppet 8 support [\#427](https://github.com/voxpupuli/puppet-logstash/pull/427) ([bastelfreak](https://github.com/bastelfreak)) 54 | - puppetlabs/stdlib: Allow 9.x [\#426](https://github.com/voxpupuli/puppet-logstash/pull/426) ([bastelfreak](https://github.com/bastelfreak)) 55 | 56 | ## [v7.0.0](https://github.com/voxpupuli/puppet-logstash/tree/v7.0.0) (2023-01-05) 57 | 58 | [Full Changelog](https://github.com/voxpupuli/puppet-logstash/compare/6.1.5...v7.0.0) 59 | 60 | **Breaking changes:** 61 | 62 | - Fix option spelling in jvm.options template [\#412](https://github.com/voxpupuli/puppet-logstash/pull/412) ([travees](https://github.com/travees)) 63 | 64 | **Implemented enhancements:** 65 | 66 | - allow overriding default jvm options [\#417](https://github.com/voxpupuli/puppet-logstash/pull/417) ([juliantaylor](https://github.com/juliantaylor)) 67 | - Add a parameter to override the plugin install user [\#414](https://github.com/voxpupuli/puppet-logstash/pull/414) ([thebeanogamer](https://github.com/thebeanogamer)) 68 | - New parameter for home directory [\#368](https://github.com/voxpupuli/puppet-logstash/pull/368) ([joernott](https://github.com/joernott)) 69 | 70 | **Closed issues:** 71 | 72 | - Cannot override JVM options for plugins [\#381](https://github.com/voxpupuli/puppet-logstash/issues/381) 73 | 74 | **Merged pull requests:** 75 | 76 | - Finalize voxpupuli transition [\#420](https://github.com/voxpupuli/puppet-logstash/pull/420) ([h-haaks](https://github.com/h-haaks)) 77 | - metadata.json: Adjust modulename/GitHub URLs [\#416](https://github.com/voxpupuli/puppet-logstash/pull/416) ([bastelfreak](https://github.com/bastelfreak)) 78 | - migrate .fixtures.yml to git repos [\#410](https://github.com/voxpupuli/puppet-logstash/pull/410) ([bastelfreak](https://github.com/bastelfreak)) 79 | - puppet-lint: autofix [\#409](https://github.com/voxpupuli/puppet-logstash/pull/409) ([bastelfreak](https://github.com/bastelfreak)) 80 | - Stop using Travis CI [\#406](https://github.com/voxpupuli/puppet-logstash/pull/406) ([jmlrt](https://github.com/jmlrt)) 81 | - bump up dependencies [\#405](https://github.com/voxpupuli/puppet-logstash/pull/405) ([anesterova](https://github.com/anesterova)) 82 | 83 | ## [6.1.5](https://github.com/voxpupuli/puppet-logstash/tree/6.1.5) (2018-11-13) 84 | 85 | Honour `restart_on_change` for startup files #377 86 | 87 | 6.1.4 88 | Use Upstart for init on Oracle Linux 6 89 | 90 | 6.1.3 91 | Allow puppetlabs-stdlib 5.x 92 | 93 | 6.1.2 94 | Set cwd to "/tmp" during plugin execs. 95 | 96 | 6.1.1 97 | Update init system handling. 98 | 99 | 6.1.0 100 | Support centralized pipeline management. 101 | 102 | 6.0.1 103 | Fix explicit versions like "5.6.2-1.noarch" #353 104 | 105 | 6.0.0 106 | Puppet 3 support removed. Minimum Puppet version is now 4.6.1. 107 | Puppet 5 supported. 108 | Logstash 6.x (and 5.x) supported. 109 | File permissions for config files now match those from the Logstash package. 110 | elastic/elastic_stack module is now used to manage repositories. 111 | Logstash multiple pipelines supported. 112 | Config files resources accept an explicit target path. 113 | 114 | 5.3.0 115 | Allow setting environment for plugin resources 116 | 117 | 5.2.0 118 | Allow 'http(s):// URLs for plugin install. 119 | 120 | 5.1.0 121 | Make config files be owned by root. 122 | Allow 'file://' URLs for plugin install. 123 | Sort lines in jvm.options for file. 124 | 125 | 5.0.4 126 | Expose $logstash::home_dir 127 | 128 | 5.0.3 129 | Sort startup_options to prevent triggering unneeded restarts. 130 | 131 | 5.0.2 132 | Do not autmatically add ".conf" to pipeline config filesnames. 133 | 134 | 5.0.1 135 | Trivial README update. 136 | 137 | 5.0.0 138 | Major re-write for Logstash 5.x. 139 | Drop support for Logstash <5.0.0. 140 | 141 | 0.6.4 142 | Puppet 4 support. 143 | 144 | 0.6.3 145 | Documentation updates only. Functionally identical to 0.6.2. 146 | 147 | 0.6.2 148 | Allow electrical/file_concat version 1.x. 149 | 150 | 0.6.1 151 | Restart service on pattern file change. 152 | Remove dependency on external PGP server. 153 | Fix circular dependency on plugin installation. 154 | 0.6.0 155 | Deprecates the logstash-contrib package. 156 | Supports Gem-based Logstash plugins. 157 | Not compatible with Logstash versions < 1.5.0. 158 | 159 | 0.5.1 160 | Updated system tests to work with LS 1.4.1 161 | Increase package download timeout 162 | Add option to use stages for the repo setup instead anchors 163 | 164 | 0.5.0 165 | Move beaker testing to use docker 166 | Large module update to work with the contrib package 167 | Refactored rspec testing 168 | Fix inline docs 169 | Added Puppet 3.5.0 testing 170 | Fix typo in group name variable ( PR #147 ) 171 | Improve puppet module removal ( PR #149 ) 172 | Reverted PR #149. Caused issues with package removal. 173 | added lsbdistid = Debian to rspec facts ( PR #146 ) 174 | match other config perms with patterns ( PR #151 ) 175 | 176 | 0.4.3 177 | Lower puppetlabs-stdlib depdency from 4.0.0 to 3.2.0 178 | Documentation improvements ( PR #132 #137 ) 179 | Fixed yumrepo call to include description ( PR #138 ) 180 | Added beaker testing 181 | Fixed bug that sometimes LS starts before all configs are processed. 182 | Ensure java is installed before installing the package when using package_url 183 | Fail fast when using package_url and repo config 184 | 185 | 0.4.2 186 | Fix config directory for config files to be inline with the init file of the packages 187 | Update readme ( thanks to PR #130 from phrawzty ) 188 | Added repo management ( based on work of PR #121 from pcfens ) 189 | 190 | 0.4.1 191 | ** Important Update ** 192 | Ensure exec names are unique. This caused an issue when using the Elasticsearch Puppet module 193 | Removed a part in the package.pp that should have been removed ( missed with the rewrite ) 194 | Missed a few bits of the rewrite. 195 | Updated readme to reflect reality regarding configfile define. 196 | 197 | 0.4.0 198 | ** NOTE: This is a backwards compability breaking release !! ** 199 | Large rewrite of the entire module described below 200 | Make the core more dynamic for different service providers 201 | Add better testing and devided into different files 202 | Add different ways to install the package except from the repository ( puppet/http/https/ftp/file ) 203 | Update java class to install openjdk 1.7 204 | Add validation of templates 205 | Added more test scenario's 206 | Added puppet parser validate task for added checking 207 | Improve module removing when set to absent 208 | Updated readme 209 | Doc improvements by dan ( phrawzty ) 210 | Added define for managing pattern files 211 | Added define for managing plugins 212 | 213 | 0.3.4 214 | Fixing purging of created directories ( PR #61, #64 by Kayla Green and Jason Koppe ) 215 | Documentation fixes ( PR #65, #67 by Kristian Glass and Andreas Paul ) 216 | Making config dir configurable ( PR #70 by Justin Lambert ) 217 | Permit HTTP(s) for downloading logstash ( PR #71 by Phil Fenstermacher ) 218 | Ensure user/group is passed in the debian init file 219 | Spec cleanup ( PR #75 by Justin Lambert ) 220 | set logstash logdir perms when using custom jar provider ( PR #74 by Justin Lambert ) 221 | clean up installpath when updating jars ( PR #72 by Justin Lambert ) 222 | fix wrong creates path at jar custom provider ( PR #83 by Daniel Werdermann ) 223 | added 'in progress' for logstash version 1.2.x ( PR #87 by rtoma ) 224 | Add small input/output examples ( PR #89 by Andreas Paul ) 225 | Solving defaults file not being installed in some cases 226 | http download of jar should require $jardir ( PR #90 by Max Griffiths ) 227 | add ability to install a logstash config file ( PR #93 by Justin Lambert ) 228 | 229 | 0.3.3 230 | Enable puppet 3.2.x testing 231 | Fix issue that the config dir was missing in the init files 232 | Fix variable access deprecation warning ( PR #56 by Richard Peng ) 233 | 234 | 0.3.2 235 | Fixing issue when using jar file without multi-instance feature 236 | Added rspec tests to cover this issue 237 | 238 | 0.3.1 239 | Missed changes for enabling/disabling multi-instance feature 240 | Adding a few spec tests for the multi-instance feature 241 | 242 | 0.3.0 243 | Update defines for Logstash 1.1.12 244 | Adding license file 245 | Deleted old init file removal to avoid issues. ( Issue #50 ) 246 | Allow file owner/group to be variable ( Issue/PR #47 ) 247 | Ensure log directory exists before starting ( PR #53 by Brian Lalor ) 248 | Provide complete containment of the class ( PR #53 by Brian Lalor ) 249 | Update rspec tests for new defines 250 | 251 | 0.2.0 252 | Update defines for logstash 1.1.10 253 | New feature for plugins to automatically transfer files ( Issue #24 ) 254 | Create correct tmp dir ( Issue #35 ) 255 | Change file modes to be more secure ( Issue #36 ) 256 | Update defines for better input validation ( Issue #43 ) 257 | Adding rspec tests for plugin defines 258 | Fix tmp dir Debian init script ( PR #44 by Dan Carley ) 259 | 260 | 0.1.0 261 | Don't backup the Jar file or the symlink ( Issue #25 by Garth Kidd ) 262 | First implementation of the multi-instance feature. This will break certain functionality. 263 | 264 | 0.0.6 265 | Fix issue that the init file was overwritten 266 | Ensure we install java first before starting logstash if enabled 267 | 268 | 0.0.5 269 | Adding spec tests 270 | Update Readme ( PR #20 by rjw1 ) 271 | New feature to install java 272 | 273 | 0.0.4 274 | Rename Redhat to RedHat for init file ( PR #12 by pkubat ) 275 | Adding Amazon as Operating system ( PR #12 by pkubat ) 276 | Symlinking Jar file to generic name ( PR #12 by pkubat ) 277 | Correting symlink ( PR #14 by Jeff Wong ) 278 | 279 | 0.0.3 280 | Clarify jarfile usage and validation ( PR #6 by Garth Kidd ) 281 | Add default Debian Init script when non provided and using custom source ( PR #7 by Garth Kidd ) 282 | Add RedHat as OS type ( PR #8 by Dan ) 283 | Skip init script when status = unmanaged ( PR #9 by Tavis Aitken ) 284 | Refactored the custom provider part ( With help of Garth Kidd ) 285 | 286 | 0.0.2 287 | Adding a way to provide jar and init file instead of depending on a package 288 | 289 | 0.0.1 290 | Initial release of the module 291 | 292 | 293 | \* *This Changelog was automatically generated by [github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)* 294 | -------------------------------------------------------------------------------- /REFERENCE.md: -------------------------------------------------------------------------------- 1 | # Reference 2 | 3 | 4 | 5 | ## Table of Contents 6 | 7 | ### Classes 8 | 9 | * [`logstash`](#logstash): This class manages installation, configuration and execution of Logstash 5.x. 10 | * [`logstash::config`](#logstash--config): This class manages configuration directories for Logstash. 11 | * [`logstash::package`](#logstash--package): This class manages the Logstash package. It is usually used only by the top-level `logstash` class. It's unlikely that you will need to decl 12 | * [`logstash::service`](#logstash--service): This mangages the system service for Logstash. It is usually used only by the top-level `logstash` class. It's unlikely that you will need t 13 | 14 | ### Defined types 15 | 16 | * [`logstash::configfile`](#logstash--configfile): This type represents a Logstash pipeline configuration file. Parameters are mutually exclusive. Only one should be specified. 17 | * [`logstash::patternfile`](#logstash--patternfile): This type represents a Grok pattern file for Logstash. 18 | * [`logstash::plugin`](#logstash--plugin): Manage the installation of a Logstash plugin. By default, plugins are downloaded from RubyGems, but it is also possible to install from a lo 19 | 20 | ## Classes 21 | 22 | ### `logstash` 23 | 24 | This class manages installation, configuration and execution of Logstash 5.x. 25 | 26 | #### Examples 27 | 28 | ##### Install Logstash, ensure the service is running and enabled. 29 | 30 | ```puppet 31 | class { 'logstash': } 32 | ``` 33 | 34 | ##### Remove Logstash. 35 | 36 | ```puppet 37 | class { 'logstash': 38 | ensure => 'absent', 39 | } 40 | ``` 41 | 42 | ##### Install everything but disable the service. 43 | 44 | ```puppet 45 | class { 'logstash': 46 | status => 'disabled', 47 | } 48 | ``` 49 | 50 | ##### Configure Logstash settings. 51 | 52 | ```puppet 53 | class { 'logstash': 54 | settings => { 55 | 'http.port' => '9700', 56 | } 57 | } 58 | ``` 59 | 60 | ##### Configure Logstash startup options. 61 | 62 | ```puppet 63 | class { 'logstash': 64 | startup_options => { 65 | 'LS_USER' => 'root', 66 | } 67 | } 68 | ``` 69 | 70 | ##### Set JVM memory options. 71 | 72 | ```puppet 73 | class { 'logstash': 74 | jvm_options => [ 75 | '-Xms1g', 76 | '-Xmx1g', 77 | ] 78 | } 79 | ``` 80 | 81 | ##### Configure multiple pipelines. 82 | 83 | ```puppet 84 | class { 'logstash': 85 | pipelines => [ 86 | { 87 | "pipeline.id" => "my-pipeline_1", 88 | "path.config" => "/etc/path/to/p1.config", 89 | }, 90 | { 91 | "pipeline.id" => "my-other-pipeline", 92 | "path.config" => "/etc/different/path/p2.cfg", 93 | } 94 | ] 95 | } 96 | ``` 97 | 98 | #### Parameters 99 | 100 | The following parameters are available in the `logstash` class: 101 | 102 | * [`ensure`](#-logstash--ensure) 103 | * [`auto_upgrade`](#-logstash--auto_upgrade) 104 | * [`status`](#-logstash--status) 105 | * [`version`](#-logstash--version) 106 | * [`restart_on_change`](#-logstash--restart_on_change) 107 | * [`package_url`](#-logstash--package_url) 108 | * [`package_name`](#-logstash--package_name) 109 | * [`download_timeout`](#-logstash--download_timeout) 110 | * [`home_dir`](#-logstash--home_dir) 111 | * [`logstash_user`](#-logstash--logstash_user) 112 | * [`logstash_group`](#-logstash--logstash_group) 113 | * [`purge_config`](#-logstash--purge_config) 114 | * [`service_provider`](#-logstash--service_provider) 115 | * [`settings`](#-logstash--settings) 116 | * [`startup_options`](#-logstash--startup_options) 117 | * [`jvm_options_defaults`](#-logstash--jvm_options_defaults) 118 | * [`jvm_options`](#-logstash--jvm_options) 119 | * [`pipelines`](#-logstash--pipelines) 120 | * [`manage_repo`](#-logstash--manage_repo) 121 | * [`config_dir`](#-logstash--config_dir) 122 | 123 | ##### `ensure` 124 | 125 | Data type: `String` 126 | 127 | Controls if Logstash should be `present` or `absent`. 128 | 129 | If set to `absent`, the Logstash package will be 130 | uninstalled. Related files will be purged as much as possible. The 131 | exact behavior is dependant on the service provider, specifically its 132 | support for the 'purgable' property. 133 | 134 | Default value: `'present'` 135 | 136 | ##### `auto_upgrade` 137 | 138 | Data type: `Boolean` 139 | 140 | If set to `true`, Logstash will be upgraded if the package provider is 141 | able to find a newer version. The exact behavior is dependant on the 142 | service provider, specifically its support for the 'upgradeable' property. 143 | 144 | Default value: `false` 145 | 146 | ##### `status` 147 | 148 | Data type: `String` 149 | 150 | The desired state of the Logstash service. Possible values: 151 | 152 | - `enabled`: Service running and started at boot time. 153 | - `disabled`: Service stopped and not started at boot time. 154 | - `running`: Service running but not be started at boot time. 155 | - `unmanaged`: Service will not be started at boot time. Puppet 156 | will neither stop nor start the service. 157 | 158 | Default value: `'enabled'` 159 | 160 | ##### `version` 161 | 162 | Data type: `String` 163 | 164 | The specific version to install, if desired. 165 | 166 | Default value: `undef` 167 | 168 | ##### `restart_on_change` 169 | 170 | Data type: `Boolean` 171 | 172 | Restart the service whenever the configuration changes. 173 | 174 | Disabling automatic restarts on config changes may be desired in an 175 | environment where you need to ensure restarts occur in a 176 | controlled/rolling manner rather than during a Puppet run. 177 | 178 | Default value: `true` 179 | 180 | ##### `package_url` 181 | 182 | Data type: `String` 183 | 184 | Explict Logstash package URL to download. 185 | 186 | Valid URL types are: 187 | - `http://` 188 | - `https://` 189 | - `ftp://` 190 | - `puppet://` 191 | - `file:/` 192 | 193 | Default value: `undef` 194 | 195 | ##### `package_name` 196 | 197 | Data type: `String` 198 | 199 | The name of the Logstash package in the package manager. 200 | 201 | Default value: `'logstash'` 202 | 203 | ##### `download_timeout` 204 | 205 | Data type: `Integer` 206 | 207 | Timeout, in seconds, for http, https, and ftp downloads. 208 | 209 | Default value: `600` 210 | 211 | ##### `home_dir` 212 | 213 | Data type: `Stdlib::Absolutepath` 214 | 215 | The home directory for logstash. 216 | 217 | Default value: `'/usr/share/logstash'` 218 | 219 | ##### `logstash_user` 220 | 221 | Data type: `String` 222 | 223 | The user that Logstash should run as. This also controls file ownership. 224 | 225 | Default value: `'logstash'` 226 | 227 | ##### `logstash_group` 228 | 229 | Data type: `String` 230 | 231 | The group that Logstash should run as. This also controls file group ownership. 232 | 233 | Default value: `'logstash'` 234 | 235 | ##### `purge_config` 236 | 237 | Data type: `Boolean` 238 | 239 | Purge the config directory of any unmanaged files, 240 | 241 | Default value: `true` 242 | 243 | ##### `service_provider` 244 | 245 | Data type: `Optional[String[1]]` 246 | 247 | Service provider (init system) to use. By Default, the module will try to 248 | choose the 'standard' provider for the current distribution. 249 | 250 | Default value: `undef` 251 | 252 | ##### `settings` 253 | 254 | Data type: `Hash` 255 | 256 | A collection of settings to be defined in `logstash.yml`. 257 | 258 | See: https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html 259 | 260 | Default value: `{}` 261 | 262 | ##### `startup_options` 263 | 264 | Data type: `Hash` 265 | 266 | A collection of settings to be defined in `startup.options`. 267 | 268 | See: https://www.elastic.co/guide/en/logstash/current/config-setting-files.html 269 | 270 | Default value: `{}` 271 | 272 | ##### `jvm_options_defaults` 273 | 274 | Data type: `Hash` 275 | 276 | Default set of optionname => option mappings from upstream 8.5 version 277 | 278 | Default value: 279 | 280 | ```puppet 281 | { 282 | '-Xms' => '-Xms1g', 283 | '-Xmx' => '-Xmx1g', 284 | 'UseConcMarkSweepGC' => '11-13:-XX:+UseConcMarkSweepGC', 285 | 'CMSInitiatingOccupancyFraction=' => '11-13:-XX:CMSInitiatingOccupancyFraction=75', 286 | 'UseCMSInitiatingOccupancyOnly' => '11-13:-XX:+UseCMSInitiatingOccupancyOnly', 287 | '-Djava.awt.headless=' => '-Djava.awt.headless=true', 288 | '-Dfile.encoding=' => '-Dfile.encoding=UTF-8', 289 | 'HeapDumpOnOutOfMemoryError' => '-XX:+HeapDumpOnOutOfMemoryError', 290 | '-Djava.security.egd' => '-Djava.security.egd=file:/dev/urandom', 291 | } 292 | ``` 293 | 294 | ##### `jvm_options` 295 | 296 | Data type: `Array` 297 | 298 | A collection of settings to be defined in `jvm.options`. Override same settings in jvm_options_defaults 299 | 300 | Default value: `[]` 301 | 302 | ##### `pipelines` 303 | 304 | Data type: `Array` 305 | 306 | A collection of settings to be defined in `pipelines.yml`. 307 | 308 | Default value: `[]` 309 | 310 | ##### `manage_repo` 311 | 312 | Data type: `Boolean` 313 | 314 | Enable repository management. Configure the official repositories. 315 | 316 | Default value: `true` 317 | 318 | ##### `config_dir` 319 | 320 | Data type: `String` 321 | 322 | Path containing the Logstash configuration. 323 | 324 | Default value: `'/etc/logstash'` 325 | 326 | ### `logstash::config` 327 | 328 | This class manages configuration directories for Logstash. 329 | 330 | #### Examples 331 | 332 | ##### Include this class to ensure its resources are available. 333 | 334 | ```puppet 335 | include logstash::config 336 | ``` 337 | 338 | ### `logstash::package` 339 | 340 | This class manages the Logstash package. 341 | 342 | It is usually used only by the top-level `logstash` class. It's unlikely 343 | that you will need to declare this class yourself. 344 | 345 | #### Examples 346 | 347 | ##### Include this class to ensure its resources are available. 348 | 349 | ```puppet 350 | include logstash::package 351 | ``` 352 | 353 | #### Parameters 354 | 355 | The following parameters are available in the `logstash::package` class: 356 | 357 | * [`package_name`](#-logstash--package--package_name) 358 | * [`version`](#-logstash--package--version) 359 | * [`package_url`](#-logstash--package--package_url) 360 | 361 | ##### `package_name` 362 | 363 | Data type: `String` 364 | 365 | The name of the Logstash package in the package manager. 366 | 367 | Default value: `$logstash::package_name` 368 | 369 | ##### `version` 370 | 371 | Data type: `String` 372 | 373 | Install precisely this version from the package manager. 374 | 375 | Default value: `$logstash::version` 376 | 377 | ##### `package_url` 378 | 379 | Data type: `String` 380 | 381 | Get the package from this URL, not from the package manager. 382 | 383 | Default value: `$logstash::package_url` 384 | 385 | ### `logstash::service` 386 | 387 | This mangages the system service for Logstash. 388 | 389 | It is usually used only by the top-level `logstash` class. It's unlikely 390 | that you will need to declare this class yourself. 391 | 392 | #### Examples 393 | 394 | ##### Include this class to ensure its resources are available. 395 | 396 | ```puppet 397 | include logstash::service 398 | ``` 399 | 400 | ## Defined types 401 | 402 | ### `logstash::configfile` 403 | 404 | This type represents a Logstash pipeline configuration file. 405 | 406 | Parameters are mutually exclusive. Only one should be specified. 407 | 408 | #### Examples 409 | 410 | ##### Create a config file content with literal content. 411 | 412 | ```puppet 413 | 414 | logstash::configfile { 'heartbeat': 415 | content => 'input { heartbeat {} }', 416 | } 417 | ``` 418 | 419 | ##### Render a config file from a template. 420 | 421 | ```puppet 422 | 423 | logstash::configfile { 'from-template': 424 | template => 'site-logstash-module/pipeline-config.erb', 425 | } 426 | ``` 427 | 428 | ##### Copy the config from a file source. 429 | 430 | ```puppet 431 | 432 | logstash::configfile { 'apache': 433 | source => 'puppet://path/to/apache.conf', 434 | } 435 | ``` 436 | 437 | ##### Create a config at specific location. Good for multiple pipelines. 438 | 439 | ```puppet 440 | 441 | logstash::configfile { 'heartbeat-2': 442 | content => 'input { heartbeat {} }', 443 | path => '/usr/local/etc/logstash/pipeline-2/heartbeat.conf' 444 | } 445 | ``` 446 | 447 | #### Parameters 448 | 449 | The following parameters are available in the `logstash::configfile` defined type: 450 | 451 | * [`content`](#-logstash--configfile--content) 452 | * [`template`](#-logstash--configfile--template) 453 | * [`source`](#-logstash--configfile--source) 454 | * [`path`](#-logstash--configfile--path) 455 | 456 | ##### `content` 457 | 458 | Data type: `String` 459 | 460 | Literal content to be placed in the file. 461 | 462 | Default value: `undef` 463 | 464 | ##### `template` 465 | 466 | Data type: `String` 467 | 468 | A template from which to render the file. 469 | 470 | Default value: `undef` 471 | 472 | ##### `source` 473 | 474 | Data type: `String` 475 | 476 | A file resource to be used for the file. 477 | 478 | Default value: `undef` 479 | 480 | ##### `path` 481 | 482 | Data type: `String` 483 | 484 | An optional full path at which to create the file. 485 | 486 | Default value: `undef` 487 | 488 | ### `logstash::patternfile` 489 | 490 | This type represents a Grok pattern file for Logstash. 491 | 492 | #### Examples 493 | 494 | ##### Define a pattern file. 495 | 496 | ```puppet 497 | logstash::patternfile { 'mypattern': 498 | source => 'puppet:///path/to/my/custom/pattern' 499 | } 500 | ``` 501 | 502 | ##### Define a pattern file with an explicit destination filename. 503 | 504 | ```puppet 505 | logstash::patternfile { 'mypattern': 506 | source => 'puppet:///path/to/my/custom/pattern', 507 | filename => 'custom-pattern-name' 508 | } 509 | ``` 510 | 511 | #### Parameters 512 | 513 | The following parameters are available in the `logstash::patternfile` defined type: 514 | 515 | * [`source`](#-logstash--patternfile--source) 516 | * [`filename`](#-logstash--patternfile--filename) 517 | 518 | ##### `source` 519 | 520 | Data type: `Optional[Pattern[/^(puppet|file):\/\//]]` 521 | 522 | File source for the pattern file. eg. `puppet://[...]` or `file://[...]` 523 | 524 | Default value: `undef` 525 | 526 | ##### `filename` 527 | 528 | Data type: `Optional[String[1]]` 529 | 530 | Optionally set the destination filename. 531 | 532 | Default value: `undef` 533 | 534 | ### `logstash::plugin` 535 | 536 | Manage the installation of a Logstash plugin. 537 | 538 | By default, plugins are downloaded from RubyGems, but it is also possible 539 | to install from a local Gem, or one stored in Puppet. 540 | 541 | #### Examples 542 | 543 | ##### Install a plugin. 544 | 545 | ```puppet 546 | logstash::plugin { 'logstash-input-stdin': } 547 | ``` 548 | 549 | ##### Remove a plugin. 550 | 551 | ```puppet 552 | logstash::plugin { 'logstash-input-stout': 553 | ensure => absent, 554 | } 555 | ``` 556 | 557 | ##### Install a plugin from a local file. 558 | 559 | ```puppet 560 | logstash::plugin { 'logstash-input-custom': 561 | source => 'file:///tmp/logstash-input-custom.gem', 562 | } 563 | ``` 564 | 565 | ##### Install a plugin from a Puppet module. 566 | 567 | ```puppet 568 | logstash::plugin { 'logstash-input-custom': 569 | source => 'puppet:///modules/logstash-site-plugins/logstash-input-custom.gem', 570 | } 571 | ``` 572 | 573 | ##### Install X-Pack. 574 | 575 | ```puppet 576 | logstash::plugin { 'x-pack': 577 | source => 'https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.3.0.zip', 578 | } 579 | ``` 580 | 581 | ##### Install a plugin, overriding JVM options via the environment. 582 | 583 | ```puppet 584 | logstash::plugin { 'logstash-input-jmx': 585 | environment => ['LS_JAVA_OPTS=-Xms1g -Xmx1g'] 586 | } 587 | ``` 588 | 589 | #### Parameters 590 | 591 | The following parameters are available in the `logstash::plugin` defined type: 592 | 593 | * [`ensure`](#-logstash--plugin--ensure) 594 | * [`source`](#-logstash--plugin--source) 595 | * [`user`](#-logstash--plugin--user) 596 | * [`environment`](#-logstash--plugin--environment) 597 | 598 | ##### `ensure` 599 | 600 | Data type: `String` 601 | 602 | Install or remove with `present` or `absent`. 603 | 604 | Default value: `present` 605 | 606 | ##### `source` 607 | 608 | Data type: `String` 609 | 610 | Install from this file, not from RubyGems. 611 | 612 | Default value: `undef` 613 | 614 | ##### `user` 615 | 616 | Data type: `String` 617 | 618 | User to install plugin as. 619 | 620 | Default value: `'root'` 621 | 622 | ##### `environment` 623 | 624 | Data type: `String` 625 | 626 | Environment used when running 'logstash-plugin' 627 | 628 | Default value: `[]` 629 | 630 | --------------------------------------------------------------------------------