├── .config └── tsaoptions.json ├── .gitignore ├── CONTRIBUTING.md ├── CredScanSuppressions.json ├── LICENSE.txt ├── OneBranch.Official.yml ├── PolicheckExclusion.xml ├── README.md ├── SECURITY.md ├── global.gdnbaselines ├── global.gdnsuppress └── src ├── CMakeLists.txt ├── Version.txt ├── buildall.sh ├── debpkg ├── Makefile ├── changelog ├── control ├── copyright └── lintian-overrides ├── fluent-plugin-mdsd ├── Gemfile ├── README.md ├── Rakefile ├── fluent-plugin-mdsd.gemspec ├── gemspec-template ├── lib │ └── fluent │ │ └── plugin │ │ └── out_mdsd.rb ├── mdsdCfg_sample.xml ├── out_mdsd_sample.conf ├── out_mdsd_sample_for_filelog.conf ├── out_mdsd_sample_for_syslog.conf └── test │ └── plugin │ └── test_out_mdsd.rb ├── outmdsd ├── BufferedLogger.cc ├── BufferedLogger.h ├── CMakeLists.txt ├── ConcurrentMap.h ├── ConcurrentQueue.h ├── DataReader.cc ├── DataReader.h ├── DataResender.cc ├── DataResender.h ├── DataSender.cc ├── DataSender.h ├── DjsonLogItem.cc ├── DjsonLogItem.h ├── EtwLogItem.h ├── Exceptions.h ├── FileTracer.cc ├── FileTracer.h ├── ITracer.h ├── IdMgr.cc ├── IdMgr.h ├── LogItem.cc ├── LogItem.h ├── LogItemPtr.h ├── SockAddr.cc ├── SockAddr.h ├── SocketClient.cc ├── SocketClient.h ├── SocketLogger.cc ├── SocketLogger.h ├── SyslogTracer.cc ├── SyslogTracer.h ├── Trace.cc ├── Trace.h └── TraceMacros.h ├── outmdsdrb ├── CMakeLists.txt ├── outmdsd_log.cxx ├── outmdsd_log.h ├── outmdsdrb.i └── wrap_memcpy.c └── test ├── CMakeLists.txt ├── outmdsd ├── CMakeLists.txt ├── CounterCV.h ├── MockServer.cc ├── MockServer.h ├── mockserver_main.cc ├── testbuflog.cc ├── testlogger.cc ├── testlogitem.cc ├── testmap.cc ├── testqueue.cc ├── testreader.cc ├── testresender.cc ├── testsender.cc ├── testsocket.cc ├── testtrace.cc ├── testutil.cc ├── testutil.h └── utmain.cc ├── outmdsdrb ├── CMakeLists.txt └── mdsdlogger.rb ├── parseglibc.py └── runtests.sh /.config/tsaoptions.json: -------------------------------------------------------------------------------- 1 | { 2 | "instanceUrl" : "https://msazure.visualstudio.com", 3 | "projectName" : "One", 4 | "areaPath" : "One\\EngSys\\Monitoring and Diagnostics\\CloudAgent\\Linux", 5 | "notificationAliases": [ 6 | "dmitmatv@microsoft.com", 7 | "glinuxagent@microsoft.com" 8 | ], 9 | "ppe" : "false", 10 | "template" : "TFSMSAzureDev", 11 | "codebaseName" : "Compute-Runtime-Tux-fluentd-plugin-mdsd" 12 | } 13 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | build 2 | builddir 3 | builddir.* 4 | .vscode 5 | *.bak 6 | src/debpkg 7 | fluent-plugin-mdsd.gemspec -------------------------------------------------------------------------------- /CONTRIBUTING.md: -------------------------------------------------------------------------------- 1 | # Contributing 2 | 3 | ## Install required software 4 | 5 | - CMake version >= 2.8.12 6 | - Boost unit test version 1.55 (libboost-test) 7 | NOTE: other version of libboost-test may not work. 8 | - Boost system library version 1.55 (libboost-system) 9 | - GCC version >= 4.8.4 10 | - Ruby 2.x with [OMS Linux Agent](https://github.com/Microsoft/OMS-Agent-for-Linux) 11 | or [Fluentd Agent](http://www.fluentd.org/download) 12 | - [SWIG](http://www.swig.org/) 3.0 for td-agent3 or SWIG 4.0.2 for td-agent4 13 | 14 | ## How to build the code 15 | 16 | Run: 17 | 18 | ```bash 19 | cd src 20 | ./buildall.sh [options] 21 | ``` 22 | 23 | ## How to run full tests 24 | 25 | Build the code following "How to build the code" section. 26 | 27 | ```bash 28 | cd src/builddir/release/tests 29 | ./runtests.sh [options] 30 | ``` 31 | 32 | Check logs in *.log if any error. 33 | 34 | ## Overview of mdsd plugin design and code structure 35 | 36 | mdsd output plugin sends events to mdsd by using UNIX domain socket. 37 | 38 | The core code that sends events to mdsd is in directory **outmdsd**. It is built into a static library file called **liboutmdsd.a**. 39 | 40 | [SWIG](http://www.swig.org/) is used to build a shim around C++ liboutmdsd library. This is in directory **outmdsdrb**. The shim is built into library called **Liboutmdsdrb.so**, which is called by Ruby code directly. 41 | 42 | The ruby code calling Liboutmdsdrb.so is in directory **fluent-plugin-mdsd**. 43 | 44 | ## How mdsd output plugin works 45 | 46 | The plugin will send dynamic schema data in JSON format (called DJSON) to mdsd using UNIX domain socket. Please refer to mdsd document on what dynamic schema protocol is supported. 47 | 48 | The data flow path is: 49 | 50 | 1. Fluentd takes input and/or filter plugin data to mdsd output plugin. The data are in JSON format. 51 | 52 | 1. Mdsd output plugin processes each of the data record. 53 | 54 | a) Find or create schema. 55 | It uses a global hash to hold all existing schemas. When a new record arrives, its schema is searched in the global hash. If found, it will be used. If not found, it will be created and stored in the hash. 56 | 57 | b) Add a unique message id to the data record. 58 | 59 | c) Construct the message by following mdsd DJSON protocol. 60 | 61 | d) Send data to mdsd socket. 62 | This part is complex. See related section on how this works. 63 | 64 | ## How mdsd plugin sends data to mdsd socket 65 | 66 | Mdsd plugin uses multiple threads to send data to mdsd process using UNIX domain socket. The #2/#3 threads are created in liboutmdsd.a. 67 | 68 | 1. Main thread. 69 | 70 | This is the plugin ruby code. The ruby code will call some send API to send a data string. If the send succeeds, the ruby code will return. If the send fails, a runtime error is raised. Fluentd will handle this error and retry later based on [buffered plugin design](http://docs.fluentd.org/articles/buffer-plugin-overview). 71 | 72 | What happens is when the ruby send API succeeds, the data record will be saved into a global concurrent cache. The cache is used for future retry. 73 | 74 | 1. Reader thread. 75 | 76 | This thread reads mdsd acknowledge response data. When mdsd receives data from socket client, it always returns some TAG information back to socket client. This TAG information can tell which message is received and what happened to the message. 77 | 78 | If plugin parameter acktimeoutms has a value greater than 0, when a TAG is received from mdsd agent, the corresponding data record will be removed from the global concurrent cache. 79 | 80 | 1. Resender thread. 81 | 82 | If plugin parameter acktimeoutms has a value greater than 0, the plugin will periodically resend the data in the global concurrent cache to mdsd agent. 83 | -------------------------------------------------------------------------------- /CredScanSuppressions.json: -------------------------------------------------------------------------------- 1 | { 2 | "tool": "Credential Scanner", 3 | "suppressions": [] 4 | } -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Azure Linux monitoring agent (mdsd) output plugin for Fluentd 2 | Copyright (c) Microsoft Corporation 3 | All rights reserved. 4 | MIT License 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the ""Software""), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /OneBranch.Official.yml: -------------------------------------------------------------------------------- 1 | schedules: 2 | - cron: "0 0 * * TUESDAY" 3 | displayName: CodeQL weekly scan 4 | branches: 5 | include: 6 | - master 7 | always: true 8 | 9 | ################################################################################# 10 | # Onebranch Pipelines - Official Linux (CDPXMigrated) # 11 | # This pipeline was created by EasyStart from a sample located at: # 12 | # https://aka.ms/obpipelines/easystart/samples # 13 | # Documentation: https://aka.ms/obpipelines # 14 | # Yaml Schema: https://aka.ms/obpipelines/yaml/schema # 15 | # Retail Tasks: https://aka.ms/obpipelines/tasks # 16 | # Support: https://aka.ms/onebranchsup # 17 | ################################################################################# 18 | 19 | 20 | trigger: 21 | - master 22 | 23 | 24 | parameters: # parameters are shown up in ADO UI in a build queue time 25 | - name: 'debug' 26 | displayName: 'Enable debug output' 27 | type: boolean 28 | default: false 29 | 30 | variables: 31 | CDP_DEFINITION_BUILD_COUNT: $[counter('', 0)] # needed for onebranch.pipeline.version task https://aka.ms/obpipelines/versioning 32 | # Docker image which is used to build the project https://aka.ms/obpipelines/containers 33 | # Image used in CDPX build: cdpxlinux.azurecr.io/global/ubuntu-1804-all:1.0 34 | LinuxContainerImage: 'mcr.microsoft.com/onebranch/cbl-mariner/build:2.0' 35 | DEBIAN_FRONTEND: noninteractive 36 | ROOT: $(Build.SourcesDirectory) 37 | REPOROOT: $(Build.SourcesDirectory) 38 | OUTPUTROOT: $(REPOROOT)\out 39 | CDP_USER_SOURCE_FOLDER_CONTAINER_PATH: $(Build.SourcesDirectory) 40 | CDP_DEFINITION_BUILD_COUNT_DAY: $[counter(format('{0:yyyyMMdd}', pipeline.startTime), 1)] 41 | CDP_DEFINITION_BUILD_COUNT_MONTH: $[counter(format('{0:yyyyMM}', pipeline.startTime), 1)] 42 | CDP_DEFINITION_BUILD_COUNT_YEAR: $[counter(format('{0:yyyy}', pipeline.startTime), 1)] 43 | 44 | resources: 45 | repositories: 46 | - repository: templates 47 | type: git 48 | name: OneBranch.Pipelines/GovernedTemplates 49 | ref: refs/heads/main 50 | 51 | extends: 52 | template: v2/OneBranch.Official.CrossPlat.yml@templates # https://aka.ms/obpipelines/templates 53 | parameters: 54 | # featureFlags: 55 | # linuxEsrpSigningPreview: true 56 | cloudvault: # https://aka.ms/obpipelines/cloudvault 57 | enabled: true # set to true to enable cloudvault 58 | runmode: stage # linux can run CloudVault upload as a separate stage 59 | dependsOn: linux_stage 60 | artifacts: 61 | - drop 62 | 63 | globalSdl: # https://aka.ms/obpipelines/sdl 64 | tsa: 65 | enabled: true # SDL results of non-official builds aren't uploaded to TSA by default. 66 | credscan: 67 | suppressionsFile: $(Build.SourcesDirectory)\CredScanSuppressions.json 68 | policheck: 69 | break: true # always break the build on policheck issues. You can disable it by setting to 'false' 70 | exclusionsFile: $(Build.SourcesDirectory)\PolicheckExclusion.xml 71 | baseline: 72 | baselineFile: $(Build.SourcesDirectory)\global.gdnbaselines 73 | suppression: 74 | suppressionFile: $(Build.SourcesDirectory)\global.gdnsuppress 75 | cg: 76 | ignoreDirectories: azure-linux-extensions/OpenCensusAgent,rsyslog-legacy 77 | codeql: 78 | enabled: true 79 | excludePathPatterns: "$(Build.SourcesDirectory)/azure-linux-extensions/TestHandlerLinux" 80 | 81 | stages: 82 | - stage: linux_stage 83 | jobs: 84 | - job: linux_job 85 | pool: 86 | type: linux 87 | 88 | variables: # More settings at https://aka.ms/obpipelines/yaml/jobs 89 | ob_outputDirectory: '$(Build.SourcesDirectory)/out' # this directory is uploaded to pipeline artifacts, reddog and cloudvault. More info at https://aka.ms/obpipelines/artifacts 90 | ob_artifactBaseName: 'drop' 91 | 92 | steps: 93 | 94 | - task: CmdLine@2 95 | displayName: 'Build' 96 | inputs: 97 | script: | 98 | tdnf -y -q upgrade 99 | tdnf -y -q install ca-certificates libyaml libedit libxcrypt util-linux tar sudo git build-essential ruby cmake boost-devel boost-static swig python3-pip cyrus-sasl 100 | update-alternatives --install /usr/bin/python python /usr/bin/python3.9 2 101 | cd src 102 | sed -i /^ParseGlibcVer$/d ./buildall.sh 103 | sed -i /^BuildWithMake\ debpkg$/d ./buildall.sh 104 | ./buildall.sh -o -t system 105 | workingDirectory: '$(Build.SourcesDirectory)' 106 | 107 | -------------------------------------------------------------------------------- /PolicheckExclusion.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Azure Linux monitoring agent (mdsd) output plugin for [Fluentd](http://fluentd.org) 2 | 3 | ## Overview 4 | 5 | This is fluentd output plugin for Azure Linux monitoring agent (mdsd). Mdsd is the Linux logging infrastructure for Azure services. It connects various log outputs to Azure monitoring service (Geneva warm path). 6 | 7 | The mdsd output plugin is a [buffered fluentd plugin](https://docs.fluentd.org/buffer#overview). 8 | 9 | ### Installation 10 | 11 | NOTE: The plugin gem must be installed using **fluent-gem**. 12 | 13 | Download the [gemfile](https://github.com/Azure/fluentd-plugin-mdsd/releases) to your computer. Assume it is saved to /path/to/gemfile. 14 | 15 | For td-agent, run 16 | 17 | $sudo bash 18 | $umask 22 19 | $/opt/td-agent/embedded/bin/fluent-gem install /path/to/gemfile 20 | 21 | For oms-agent, run 22 | 23 | $sudo bash 24 | $umask 22 25 | $/opt/microsoft/omsagent/ruby/bin/fluent-gem install /path/to/gemfile 26 | 27 | 28 | ### Expected Data Formats 29 | 30 | The data records sent to this plugin are expected to be in a flat structure key-value pairs, where each value are primitive types like number, string, boolean, or time. Value types including Array, Hash, and Range will be dumped as strings. 31 | 32 | If this behavior doesn't meet your requirements, the recommended way is to create filter plugins for your data records. 33 | 34 | ### Configuration 35 | 36 | [See configuration sample](./src/fluent-plugin-mdsd/out_mdsd_sample.conf) 37 | 38 | The mdsd output plugin is a buffered fluentd plugin. Besides supporting all the fluentd buffered plugin parameters, it supports the following required parameters 39 | 40 | - **log_level**: this is [standard fluentd per plugin log level](http://docs.fluentd.org/v0.12/articles/logging#per-plugin-log). Valid values are: fatal, error, warn, info, debug, trace. 41 | 42 | - **djsonsocket**: this is the full path to mdsd dynamic json socket file. 43 | 44 | - **acktimeoutms**: max time in milliseconds to wait for mdsd acknowledge response. Before timeout, mdsd plugin will retry periodically to resend the events to mdsd. After timeout, the events holding in mdsd plugin memory will be dropped. If acktimeoutms is 0, the plugin won't do any failure retry if it cannot receives acknowledge from mdsd. 45 | 46 | - **mdsd_tag_regex_patterns**: (Optional) An array of regex patterns for mdsd source name unification purpose. The passed will be matched against each regex, and if there's a match, the matched substring will be used as the resulting mdsd source name. For example, if the tag is `mdsd.ext_syslog.user.info` and the regex is `^mdsd\.ext_syslog\.\w+`, then `mdsd.ext_syslog.user` will be the mdsd source name. If this parameter is not specified, or for tags not matching any regexes in this array parameter, the original fluentd tag will be used as the mdsd source name. Default: `[]`. 47 | 48 | - **resend_interval_ms**: the interval in milliseconds that failed messages are resent to mdsd by this plugin. Default: 30,000. 49 | 50 | - **conn_retry_timeout_ms**: the timeout in milliseconds to do network connection retry when connecting to mdsd process failed. Default: 60,000. 51 | 52 | - **emit_timestamp_name**: the field name for the event emit time stamp. Default: "FluentdIngestTimestamp". 53 | 54 | - **use_source_timestamp**: use the timestamp in the record source. If false, sets the time to Time.now Default: true. 55 | 56 | ### Usage 57 | 58 | 1. Install and configure mdsd. mdsd is a separate component. Please refer to related document. 59 | 60 | 1. Install the mdsd output plugin following Installation section. 61 | 62 | 1. Configure fluentd using mdsd as the output plugin. Configure fluentd using whatever input plugin you want to use. 63 | 64 | 1. Start (or restart) fluentd daemon. 65 | -------------------------------------------------------------------------------- /SECURITY.md: -------------------------------------------------------------------------------- 1 | 2 | 3 | ## Security 4 | 5 | Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/). 6 | 7 | If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below. 8 | 9 | ## Reporting Security Issues 10 | 11 | **Please do not report security vulnerabilities through public GitHub issues.** 12 | 13 | Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report). 14 | 15 | If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey). 16 | 17 | You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc). 18 | 19 | Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue: 20 | 21 | * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.) 22 | * Full paths of source file(s) related to the manifestation of the issue 23 | * The location of the affected source code (tag/branch/commit or direct URL) 24 | * Any special configuration required to reproduce the issue 25 | * Step-by-step instructions to reproduce the issue 26 | * Proof-of-concept or exploit code (if possible) 27 | * Impact of the issue, including how an attacker might exploit the issue 28 | 29 | This information will help us triage your report more quickly. 30 | 31 | If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs. 32 | 33 | ## Preferred Languages 34 | 35 | We prefer all communications to be in English. 36 | 37 | ## Policy 38 | 39 | Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd). 40 | 41 | 42 | -------------------------------------------------------------------------------- /global.gdnbaselines: -------------------------------------------------------------------------------- 1 | { 2 | "version": "latest", 3 | "baselines": { 4 | "default": { 5 | "name": "default", 6 | "createdDate": "2022-11-18 13:51:04Z", 7 | "lastUpdatedDate": "2022-11-18 13:51:04Z" 8 | } 9 | }, 10 | "results": { 11 | 12 | } 13 | } -------------------------------------------------------------------------------- /global.gdnsuppress: -------------------------------------------------------------------------------- 1 | { 2 | "version": "latest", 3 | "suppressionSets": { 4 | "default": { 5 | "name": "default", 6 | "createdDate": "2023-05-12 02:15:33Z", 7 | "lastUpdatedDate": "2023-05-12 02:15:33Z" 8 | } 9 | }, 10 | "results": { } 11 | } -------------------------------------------------------------------------------- /src/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | cmake_minimum_required(VERSION 3.0) 2 | project(fluentd-plugin-mdsd) 3 | 4 | set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/cmake/Modules/") 5 | 6 | # Platform (not compiler) specific settings 7 | if(UNIX) # This includes Linux 8 | message("Build for Unix/Linux OS") 9 | else() 10 | message("-- Unsupported Build Platform.") 11 | endif() 12 | 13 | # Compiler (not platform) specific settings 14 | if("${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang") 15 | message("-- Clang compiler is not supported") 16 | elseif("${CMAKE_CXX_COMPILER_ID}" MATCHES "GNU") 17 | message("-- Setting gcc options") 18 | set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-literal-suffix") 19 | else() 20 | message("-- Unknown compiler, success is doubtful.") 21 | endif() 22 | 23 | set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11") 24 | 25 | set(COMM_FLAGS "${COMM_FLAGS} -fstack-protector-all") 26 | set(COMM_FLAGS "${COMM_FLAGS} -fPIC") 27 | set(COMM_FLAGS "${COMM_FLAGS} -D_FORTIFY_SOURCE=2") 28 | set(COMM_FLAGS "${COMM_FLAGS} -ffunction-sections") 29 | 30 | set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${COMM_FLAGS}") 31 | 32 | set(WARNINGS "${WARNINGS} -Wall") 33 | set(WARNINGS "${WARNINGS} -Wextra") 34 | set(WARNINGS "${WARNINGS} -Wno-missing-field-initializers") 35 | set(WARNINGS "${WARNINGS} -Wno-unused-parameter") 36 | 37 | set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${WARNINGS}") 38 | set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -ggdb") 39 | 40 | set(LINKER_FLAGS "-Wl,-z,relro -Wl,-z,now") 41 | set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} ${LINKER_FLAGS}") 42 | set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${LINKER_FLAGS}") 43 | 44 | option(BUILD_SHARED_LIBS "Build shared Libraries." ON) 45 | 46 | add_subdirectory(outmdsd) 47 | add_subdirectory(outmdsdrb) 48 | add_subdirectory(test) 49 | -------------------------------------------------------------------------------- /src/Version.txt: -------------------------------------------------------------------------------- 1 | 0.2.5 2 | -------------------------------------------------------------------------------- /src/buildall.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # This will build all source code 4 | # Usage: see Usage() 5 | # 6 | 7 | TotalErrors=0 8 | 9 | BuildType= 10 | CCompiler=gcc 11 | CXXCompiler=g++ 12 | BUILDDIR=builddir 13 | BuildName=dev 14 | # Valid ${Target} values are: td (for treasure data), oms (for OMSAgent) 15 | Target=td 16 | 17 | # The full path directory containing ruby headers (e.g. 'ruby.h') 18 | RUBY_INC_PATH= 19 | # The full path directory containing ruby bin (e.g. 'ruby', 'gem', etc) 20 | RUBY_BIN_PATH= 21 | 22 | Usage() 23 | { 24 | echo "Usage: $0 [-b buildname] <-d|-o> [-h]" 25 | echo " -b: use buildname. Default: dev." 26 | echo " -d: build debug build." 27 | echo " -h: help." 28 | echo " -o: build optimized(release) build." 29 | echo " -t: fluentd target ('system', 'td' or 'oms')" 30 | } 31 | 32 | if [ "$#" == "0" ]; then 33 | Usage 34 | exit 1 35 | fi 36 | 37 | SetBuildType() 38 | { 39 | if [ -z "${BuildType}" ]; then 40 | BuildType=$1 41 | else 42 | echo "Error: build type is already set to be ${BuildType}." 43 | exit 1 44 | fi 45 | } 46 | 47 | args=`getopt b:dhot: $*` 48 | if [ $? != 0 ]; then 49 | Usage 50 | exit 1 51 | fi 52 | set -- $args 53 | 54 | for i; do 55 | case "$i" in 56 | -b) 57 | BuildName=$2 58 | shift ; shift ;; 59 | -d) 60 | SetBuildType Debug 61 | shift ;; 62 | -h) 63 | Usage 64 | exit 0 65 | shift ;; 66 | -o) 67 | SetBuildType Release 68 | shift ;; 69 | -t) 70 | Target=$2 71 | shift; shift ;; 72 | --) shift; break ;; 73 | esac 74 | done 75 | 76 | if [ "${Target}" != "system" ] && [ "${Target}" != "td" ] && [ "${Target}" != "oms" ]; then 77 | echo "Error: invalid -t value. Expected: 'system', 'td' or 'oms'" 78 | exit 1 79 | fi 80 | 81 | if [ -z "${BuildType}" ]; then 82 | echo "Error: missing build type" 83 | exit 1 84 | fi 85 | 86 | FindRubyPath() 87 | { 88 | # The full path directory containing ruby bin (e.g. 'ruby', 'gem', etc) 89 | if [ "${Target}" == "td" ]; then 90 | if [ -d "/opt/td-agent/embedded/bin" ]; then 91 | RubyBaseDir="/opt/td-agent/embedded/include/ruby-" 92 | RUBY_BIN_PATH=/opt/td-agent/embedded/bin 93 | else 94 | RubyBaseDir="/opt/td-agent/include/ruby-" 95 | RUBY_BIN_PATH=/opt/td-agent/bin 96 | fi 97 | elif [ "${Target}" == "oms" ]; then 98 | RubyBaseDir="/opt/microsoft/omsagent/ruby/include/ruby-" 99 | RUBY_BIN_PATH=/opt/microsoft/omsagent/ruby/bin 100 | elif [ "${Target}" == "system" ]; then 101 | RUBY_BIN_PATH=/usr/bin 102 | RUBY_INC_PATH=/usr/include 103 | else 104 | echo "FindRubyPath() error: unexpected target ${Target}." 105 | exit 1 106 | fi 107 | 108 | if [ -z "${RUBY_INC_PATH}" ]; then 109 | for diritem in "${RubyBaseDir}"*; do 110 | [ -d "${diritem}" ] && RUBY_INC_PATH="${diritem}" && break 111 | done 112 | fi 113 | 114 | if [ -z "${RUBY_INC_PATH}" ]; then 115 | echo "Error: failed to get value for RUBY_INC_PATH." 116 | else 117 | echo "Ruby include dir: ${RUBY_INC_PATH}." 118 | fi 119 | } 120 | 121 | BuildWithCMake() 122 | { 123 | echo 124 | echo Start to build source code. BuildType=${BuildType} ... 125 | BinDropDir=${BUILDDIR}.${BuildType}.${Target} 126 | rm -rf ${BUILDDIR} ${BinDropDir} 127 | mkdir ${BinDropDir} 128 | ln -s ${BinDropDir} ${BUILDDIR} 129 | 130 | pushd ${BinDropDir} 131 | 132 | cmake -DCMAKE_C_COMPILER=${CCompiler} -DCMAKE_CXX_COMPILER=${CXXCompiler} \ 133 | -DRUBY_INC_PATH=${RUBY_INC_PATH} \ 134 | -DFLUENTD_TARGET=${Target} \ 135 | -DCMAKE_BUILD_TYPE=${BuildType} ../ 136 | 137 | if [ $? != 0 ]; then 138 | let TotalErrors+=1 139 | echo Error: cmake failed. 140 | exit ${TotalErrors} 141 | fi 142 | 143 | make -j4 144 | if [ $? != 0 ]; then 145 | let TotalErrors+=1 146 | echo Error: make failed. 147 | exit ${TotalErrors} 148 | fi 149 | make install 150 | if [ $? != 0 ]; then 151 | let TotalErrors+=1 152 | echo Error: \'make install\' failed. 153 | exit ${TotalErrors} 154 | fi 155 | tar czf release.tar.gz release 156 | popd 157 | } 158 | 159 | # usage: BuildWithMake 160 | # are optional 161 | BuildWithMake() 162 | { 163 | echo 164 | echo Start to build: directory=$1 $2 ... 165 | make -C $1 clean 166 | make LABEL=build.${BuildName} -C $1 $2 167 | 168 | if [ $? != 0 ]; then 169 | let TotalErrors+=1 170 | echo Error: build $1 $2 failed 171 | exit ${TotalErrors} 172 | else 173 | echo Finished built successfully: directory=$1 174 | fi 175 | } 176 | 177 | ParseGlibcVer() 178 | { 179 | glibcver=2.9 # max GLIBC version to support 180 | dirname=./${BUILDDIR}/release/lib 181 | echo 182 | echo python ./test/parseglibc.py -d ${dirname} -v ${glibcver} 183 | python ./test/parseglibc.py -d ${dirname} -v ${glibcver} 184 | if [ $? != 0 ]; then 185 | let TotalErrors+=1 186 | echo Error: ParseGlibcVer failed: maximum supported GLIBC version is ${glibcver}. 187 | exit ${TotalErrors} 188 | fi 189 | } 190 | 191 | CreateGemFile() 192 | { 193 | echo CreateGemFile ... 194 | Label=build.${BuildName} 195 | Version=$(cat ./Version.txt)-${Label} 196 | 197 | pushd fluent-plugin-mdsd 198 | 199 | cp -f ../builddir/release/lib/Liboutmdsdrb.so ./lib/fluent/plugin/Liboutmdsdrb.so 200 | cp -f ../../LICENSE.txt ../../README.md . 201 | 202 | sed "s/GEMVERSION/${Version}/g" gemspec-template > fluent-plugin-mdsd.gemspec 203 | 204 | # If Target is 'system', then use gem to build the gem file. Otherwise, use fluent-gem. 205 | GEM_BIN=${RUBY_BIN_PATH}/fluent-gem 206 | if [ "${Target}" == "system" ]; then 207 | GEM_BIN=gem 208 | fi 209 | 210 | echo ${GEM_BIN} build fluent-plugin-mdsd.gemspec 211 | ${GEM_BIN} build fluent-plugin-mdsd.gemspec 212 | if [ $? != 0 ]; then 213 | let TotalErrors+=1 214 | echo Error: CreateGemFile failed 215 | exit ${TotalErrors} 216 | fi 217 | 218 | popd 219 | } 220 | 221 | ReleaseGemFile() 222 | { 223 | GemReleaseDir=${BUILDDIR}/release/gem 224 | 225 | pushd fluent-plugin-mdsd 226 | for f in $(ls *.gem); do 227 | mv $f ${f%.gem}-${Target}.amd64.gem 228 | done 229 | popd 230 | mkdir -p ${GemReleaseDir} 231 | mv fluent-plugin-mdsd/*.gem ${GemReleaseDir} 232 | } 233 | 234 | echo Start build at `date`. BuildType=${BuildType} CC=${CCompiler} Target=${Target} ... 235 | 236 | FindRubyPath 237 | BuildWithCMake 238 | if [ "${Target}" != "system" ]; then 239 | ParseGlibcVer 240 | fi 241 | CreateGemFile 242 | ReleaseGemFile 243 | 244 | BuildWithMake debpkg 245 | 246 | echo 247 | echo Finished all builds at `date`. error = ${TotalErrors} 248 | exit ${TotalErrors} 249 | -------------------------------------------------------------------------------- /src/debpkg/Makefile: -------------------------------------------------------------------------------- 1 | LABEL?=~dev 2 | VERSION=$(shell cat ../Version.txt)-$(LABEL) 3 | DEPENDS= 4 | 5 | PACKAGE=liboutmdsd-dev 6 | BINDIR=../builddir/release/lib 7 | FAKEROOT=./data-root 8 | DOCDIR=$(FAKEROOT)/usr/share/doc/$(PACKAGE) 9 | LINTIANOVERRIDES=$(FAKEROOT)/usr/share/lintian/overrides/$(PACKAGE) 10 | INCLUDEDIR=$(FAKEROOT)/usr/include/outmdsd/ 11 | LIBDIR=$(FAKEROOT)/usr/lib/x86_64-linux-gnu 12 | DEB=$(PACKAGE)_$(VERSION)_amd64.deb 13 | 14 | package: $(DEB) 15 | 16 | signed-package: _gpgorigin $(DEB) 17 | ar r $(DEB) $< 18 | 19 | _gpgorigin: $(DEB) 20 | -rm -f $@ 21 | ar p $(DEB) debian-binary control.tar.gz data.tar.gz | gpg -abs -o _gpgorigin 22 | 23 | $(DEB): tarballs debian-binary 24 | -rm -f $@ 25 | ar rc $@ debian-binary control.tar.gz data.tar.gz 26 | lintian $@ 27 | 28 | $(DOCDIR): 29 | mkdir -p $@ 30 | 31 | $(DOCDIR)/changelog.Debian.gz: changelog $(DOCDIR) 32 | cat $< | sed "s/PACKAGE/$(PACKAGE)/g" | gzip -9 > $@ 33 | chmod 644 $(@) 34 | 35 | $(DOCDIR)/copyright: copyright $(DOCDIR) 36 | cp $< $@ 37 | chmod 644 $@ 38 | 39 | $(LINTIANOVERRIDES): lintian-overrides 40 | mkdir -p $(@D) 41 | cat $< | sed "s/PACKAGE/$(PACKAGE)/g" > $@ 42 | chmod 644 $(@) 43 | 44 | debian-binary: 45 | echo 2.0 > debian-binary 46 | 47 | tarballs: data.tar.gz control.tar.gz 48 | 49 | control.tar.gz: md5sums control 50 | -rm -rf control-root 51 | -mkdir -p control-root 52 | cp control md5sums control-root 53 | sed -i '/^Package:/c Package: $(PACKAGE)' control-root/control 54 | sed -i '/^Version:/c Version: $(VERSION)' control-root/control 55 | sed -i '/^Depends:/c Depends: $(DEPENDS)' control-root/control 56 | chmod 644 control-root/* 57 | cd control-root && tar -czf ../$@ --owner=root --group=root . 58 | 59 | md5sums: install-deps 60 | (cd $(FAKEROOT) && md5sum `find -type f`) > $@ 61 | chmod 0644 $@ 62 | 63 | data.tar.gz: install-deps \ 64 | $(DOCDIR)/changelog.Debian.gz \ 65 | $(DOCDIR)/copyright \ 66 | $(LINTIANOVERRIDES) 67 | find $(FAKEROOT) -type d | xargs chmod 0755 68 | cd $(FAKEROOT) && tar -czf ../$@ --owner=root --group=root --mode=go-w * 69 | 70 | .PHONY: clean install-clean install-deps 71 | 72 | clean: install-clean 73 | -rm -rf control-root 74 | -rm -f debian-binary *.tar.gz _gpgorigin md5sums 75 | -rm -f lib*deb 76 | 77 | install-clean: 78 | -rm -rf $(FAKEROOT) 79 | 80 | install-deps: install-clean install-lib install-header 81 | 82 | install-header: 83 | mkdir -p $(INCLUDEDIR) 84 | cp -f -r -P ../outmdsd/*.h $(INCLUDEDIR) 85 | find $(INCLUDEDIR) -type f | xargs chmod 644 86 | 87 | install-lib: 88 | mkdir -p $(LIBDIR) 89 | cp -f -P $(BINDIR)/liboutmdsd.a $(LIBDIR)/ 90 | chmod 0644 $(LIBDIR)/lib* 91 | -------------------------------------------------------------------------------- /src/debpkg/changelog: -------------------------------------------------------------------------------- 1 | PACKAGE (0.1.2) unstable; urgency=low 2 | * Add support for both Treasure data fluentd agent and OMSAgent. 3 | * Bug fix: remove '-' => '.' when creating mdsd 'source' from plugin tag. 4 | 5 | -- Azure Linux Team Tue, 14 Mar 2017 17:00:00 +0800 6 | 7 | PACKAGE (0.1.1) unstable; urgency=low 8 | * Separate tracing impl class and tracing interface class. 9 | * Implement SyslogTracer, FileTracer classes. 10 | * Bug fix: diff order schema fields should be diff schemas. 11 | 12 | -- Azure Linux Team Wed, 25 Jan 2017 22:00:00 +0800 13 | 14 | PACKAGE (0.1.0) unstable; urgency=low 15 | 16 | * Initial release 17 | 18 | -- Azure Linux Team Thu, 12 Jan 2017 18:00:00 +0800 19 | -------------------------------------------------------------------------------- /src/debpkg/control: -------------------------------------------------------------------------------- 1 | Package: PACKAGE 2 | Version: VERSION 3 | Section: libdevel 4 | Priority: optional 5 | Architecture: amd64 6 | Depends: DEPENDS 7 | Maintainer: Azure Linux Team 8 | Description: Azure Linux monitoring agent (mdsd) output plugin for fluentd 9 | This is fluentd output plugin for Azure Linux monitoring agent (mdsd). 10 | Mdsd is the Linux logging infrastructure for Azure services. It connects 11 | various log outputs to Azure monitoring service (Geneva warm path). 12 | -------------------------------------------------------------------------------- /src/debpkg/copyright: -------------------------------------------------------------------------------- 1 | Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ 2 | Source: https://github.com/Azure/fluentd-plugin-mdsd/ 3 | 4 | Files: * 5 | Copyright (c) 2017 Microsoft Corporation 6 | License: https://github.com/Azure/fluentd-plugin-mdsd/blob/master/LICENSE.txt 7 | -------------------------------------------------------------------------------- /src/debpkg/lintian-overrides: -------------------------------------------------------------------------------- 1 | # Internal package, no itp bugs involved 2 | -------------------------------------------------------------------------------- /src/fluent-plugin-mdsd/Gemfile: -------------------------------------------------------------------------------- 1 | source "http://rubygems.org" 2 | 3 | gemspec 4 | -------------------------------------------------------------------------------- /src/fluent-plugin-mdsd/README.md: -------------------------------------------------------------------------------- 1 | # Azure Linux monitoring agent (mdsd) output plugin for [Fluentd](http://fluentd.org) 2 | 3 | ## Overview 4 | 5 | This is fluentd output plugin for Azure Linux monitoring agent (mdsd). Mdsd is the Linux logging infrastructure for Azure services. It connects various log outputs to Azure monitoring service (Geneva warm path). 6 | 7 | The mdsd output plugin is a [buffered fluentd plugin](http://docs.fluentd.org/articles/buffer-plugin-overview). 8 | 9 | ### Installation 10 | 11 | NOTE: The plugin gem must be installed using **fluent-gem**. 12 | 13 | Download the [gemfile](https://github.com/Azure/fluentd-plugin-mdsd/releases) to your computer. Assume it is saved to /path/to/gemfile. 14 | 15 | For td-agent, run 16 | 17 | $sudo bash 18 | $umask 22 19 | $/opt/td-agent/embedded/bin/fluent-gem install /path/to/gemfile 20 | 21 | For oms-agent, run 22 | 23 | $sudo bash 24 | $umask 22 25 | $/opt/microsoft/omsagent/ruby/bin/fluent-gem install /path/to/gemfile 26 | 27 | 28 | ### Expected Data Formats 29 | 30 | The data records sent to this plugin are expected to be in a flat structure key-value pairs, where each value are primitive types like number, string, boolean, or time. Value types including Array, Hash, and Range will be dumped as strings. 31 | 32 | If this behavior doesn't meet your requirements, the recommended way is to create filter plugins for your data records. 33 | 34 | ### Configuration 35 | 36 | [See configuration sample](./src/fluent-plugin-mdsd/out_mdsd_sample.conf) 37 | 38 | The mdsd output plugin is a buffered fluentd plugin. Besides supporting all the fluentd buffered plugin parameters, it supports the following required parameters 39 | 40 | - **log_level**: this is [standard fluentd per plugin log level](http://docs.fluentd.org/v0.12/articles/logging#per-plugin-log). Valid values are: fatal, error, warn, info, debug, trace. 41 | 42 | - **djsonsocket**: this is the full path to mdsd dynamic json socket file. 43 | 44 | - **acktimeoutms**: max time in milliseconds to wait for mdsd acknowledge response. Before timeout, mdsd plugin will retry periodically to resend the events to mdsd. After timeout, the events holding in mdsd plugin memory will be dropped. If acktimeoutms is 0, the plugin won't do any failure retry if it cannot receives acknowledge from mdsd. 45 | 46 | - **mdsd_tag_regex_patterns**: (Optional) An array of regex patterns for mdsd source name unification purpose. The passed will be matched against each regex, and if there's a match, the matched substring will be used as the resulting mdsd source name. For example, if the tag is `mdsd.ext_syslog.user.info` and the regex is `^mdsd\.ext_syslog\.\w+`, then `mdsd.ext_syslog.user` will be the mdsd source name. If this parameter is not specified, or for tags not matching any regexes in this array parameter, the original fluentd tag will be used as the mdsd source name. Default: `[]`. 47 | 48 | - **resend_interval_ms**: the interval in milliseconds that failed messages are resent to mdsd by this plugin. Default: 30,000. 49 | 50 | - **conn_retry_timeout_ms**: the timeout in milliseconds to do network connection retry when connecting to mdsd process failed. Default: 60,000. 51 | 52 | - **emit_timestamp_name**: the field name for the event emit time stamp. Default: "FluentdIngestTimestamp". 53 | 54 | - **use_source_timestamp**: use the timestamp in the record source. If false, sets the time to Time.now Default: true. 55 | 56 | - **convert_hash_to_json**: if this is set to true, then plugin will convert hash string to proper json before sending to mdsd. 57 | Input record with hash {key => {"X"=>"Y"}} will be tranformed to {key => {"X":"Y"}} 58 | 59 | ### Usage 60 | 61 | 1. Install and configure mdsd. mdsd is a separate component. Please refer to related document. 62 | 63 | 1. Install the mdsd output plugin following Installation section. 64 | 65 | 1. Configure fluentd using mdsd as the output plugin. Configure fluentd using whatever input plugin you want to use. 66 | 67 | 1. Start (or restart) fluentd daemon. 68 | -------------------------------------------------------------------------------- /src/fluent-plugin-mdsd/Rakefile: -------------------------------------------------------------------------------- 1 | require "bundler/gem_tasks" 2 | require 'rake/testtask' 3 | 4 | Rake::TestTask.new(:test) do |test| 5 | test.libs << 'lib' << 'test' 6 | test.pattern = 'test/**/test_*.rb' 7 | test.verbose = true 8 | end 9 | 10 | task :default => :test 11 | -------------------------------------------------------------------------------- /src/fluent-plugin-mdsd/fluent-plugin-mdsd.gemspec: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/Azure/fluentd-plugin-mdsd/1d2db1b073e2763c17604b2d74b84bb0dd210a4f/src/fluent-plugin-mdsd/fluent-plugin-mdsd.gemspec -------------------------------------------------------------------------------- /src/fluent-plugin-mdsd/gemspec-template: -------------------------------------------------------------------------------- 1 | Gem::Specification.new do |s| 2 | s.name = "fluent-plugin-mdsd" 3 | s.version = "GEMVERSION" 4 | s.summary = "Fluentd output plugin for Linux monitoring agent (mdsd)" 5 | s.description = "Fluentd output plugin for Linux monitoring agent(mdsd), which sends data to Azure Geneva service." 6 | s.authors = [""] 7 | s.email = [""] 8 | s.extra_rdoc_files = [ 9 | "LICENSE.txt", 10 | "README.md" 11 | ] 12 | s.files = [ 13 | "Gemfile", 14 | "LICENSE.txt", 15 | "README.md", 16 | "Rakefile", 17 | "lib/fluent/plugin/out_mdsd.rb", 18 | "lib/fluent/plugin/Liboutmdsdrb.so", 19 | "fluent-plugin-mdsd.gemspec", 20 | "out_mdsd_sample.conf", 21 | "test/plugin/test_out_mdsd.rb" 22 | ] 23 | s.homepage = "https://github.com/Azure/fluentd-plugin-mdsd/" 24 | s.license = "MIT" 25 | 26 | s.add_dependency "fluentd" 27 | s.add_development_dependency "rake" 28 | s.add_development_dependency "test-unit", "~> 3.0.9" 29 | end 30 | -------------------------------------------------------------------------------- /src/fluent-plugin-mdsd/mdsdCfg_sample.xml: -------------------------------------------------------------------------------- 1 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | xxxxxxxxxxxxxxxxxxx 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | -------------------------------------------------------------------------------- /src/fluent-plugin-mdsd/out_mdsd_sample.conf: -------------------------------------------------------------------------------- 1 | # Assume up-stream use mdsdlog tag 2 | 3 | type mdsd 4 | log_level info 5 | djsonsocket /var/run/mdsd/default_djson.socket # Full path to mdsd dynamic json socket file 6 | acktimeoutms 5000 # max time in milliseconds to wait for mdsd acknowledge response. If 0, no wait. 7 | mdsd_tag_regex_patterns [ "^mdsd\\.syslog", "^mdsd\\.ext_syslog\\.\\w+" ] # fluentd tag patterns whose match will be used as mdsd source name 8 | num_threads 1 9 | buffer_chunk_limit 1000k 10 | buffer_type file 11 | buffer_path /var/opt/microsoft/omsagent/state/out_mdsd*.buffer 12 | # buffer_path /var/log/td-agent/buffer/out_mdsd*.buffer 13 | buffer_queue_limit 128 14 | flush_interval 10s 15 | retry_limit 3 16 | retry_wait 10s 17 | # convert hash value to valid json. 18 | convert_hash_to_json true 19 | 20 | -------------------------------------------------------------------------------- /src/fluent-plugin-mdsd/out_mdsd_sample_for_filelog.conf: -------------------------------------------------------------------------------- 1 | # For all monitored files... 2 | 3 | @type tail 4 | path /tmp/monitored_file1,/tmp/monitored_file2 5 | tag mdsd.filelog.* 6 | format none 7 | message_key Msg # LAD uses "Msg" as the field name 8 | 9 | 10 | # Add FileTag field (existing LAD behavior) 11 | 12 | @type record_transformer 13 | 14 | FileTag ${tag_suffix[2]} 15 | 16 | 17 | 18 | # Output to mdsd 19 | 20 | type mdsd 21 | log_level warn #trace 22 | djsonsocket /var/run/mdsd/default_djson.socket # Full path to mdsd dynamic json socket file 23 | acktimeoutms 5000 # max time in milliseconds to wait for mdsd acknowledge response. If 0, no wait. 24 | #mdsd_tag_regex_patterns [ "^mdsd\\.filelog" ] # fluentd tag patterns whose match will be used as mdsd source name. Uncomment this line if want to unify the different tags to "mdsd.filelog". 25 | num_threads 1 26 | buffer_chunk_limit 1000k 27 | buffer_type file 28 | buffer_path /var/opt/microsoft/omsagent/state/out_mdsd*.buffer 29 | buffer_queue_limit 128 30 | flush_interval 10s 31 | retry_limit 3 32 | retry_wait 10s 33 | 34 | -------------------------------------------------------------------------------- /src/fluent-plugin-mdsd/out_mdsd_sample_for_syslog.conf: -------------------------------------------------------------------------------- 1 | # Configure syslog source (for LAD-syslog basic config) 2 | 3 | type syslog 4 | port 25224 5 | bind 127.0.0.1 6 | protocol_type udp 7 | include_source_host true # This should be set for SendingHost field. 8 | tag mdsd.syslog 9 | 10 | 11 | # Configure syslog source (for LAD-syslog extended config) 12 | 13 | type syslog 14 | port 25225 15 | bind 127.0.0.1 16 | protocol_type udp 17 | include_source_host true # This should be set for SendingHost field. 18 | tag mdsd.ext_syslog 19 | 20 | 21 | # Generate fields expected for existing mdsd syslog collection schema 22 | 23 | @type record_transformer 24 | enable_ruby 25 | 26 | # Fields expected by mdsd for syslog messages 27 | Ignore "syslog" 28 | Facility ${tag_parts[2]} 29 | Severity ${tag_parts[3]} 30 | EventTime ${time.strftime('%Y-%m-%dT%H:%M:%S%z')} 31 | SendingHost ${record["source_host"]} 32 | Msg ${record["message"]} 33 | 34 | remove_keys host,ident,pid,message,source_host # No need of these fields for mdsd so remove 35 | 36 | 37 | # Output to mdsd 38 | 39 | type mdsd 40 | log_level warn #trace 41 | djsonsocket /var/run/mdsd/default_djson.socket # Full path to mdsd dynamic json socket file 42 | acktimeoutms 5000 # max time in milliseconds to wait for mdsd acknowledge response. If 0, no wait. 43 | mdsd_tag_regex_patterns [ "^mdsd\\.syslog", "^mdsd\\.ext_syslog\\.\\w+" ] # fluentd tag patterns whose match will be used as mdsd source name 44 | num_threads 1 45 | buffer_chunk_limit 1000k 46 | buffer_type file 47 | buffer_path /var/opt/microsoft/omsagent/state/out_mdsd*.buffer 48 | buffer_queue_limit 128 49 | flush_interval 10s 50 | retry_limit 3 51 | retry_wait 10s 52 | 53 | -------------------------------------------------------------------------------- /src/outmdsd/BufferedLogger.cc: -------------------------------------------------------------------------------- 1 | #include "ConcurrentMap.h" 2 | #include "ConcurrentQueue.h" 3 | #include "BufferedLogger.h" 4 | #include "Trace.h" 5 | #include "TraceMacros.h" 6 | #include "SocketClient.h" 7 | #include "DataReader.h" 8 | #include "DataResender.h" 9 | #include "DataSender.h" 10 | 11 | using namespace EndpointLog; 12 | 13 | BufferedLogger::BufferedLogger( 14 | const std::string& socketFile, 15 | unsigned int ackTimeoutMS, 16 | unsigned int resendIntervalMS, 17 | unsigned int connRetryTimeoutMS, 18 | size_t bufferLimit 19 | ): 20 | m_sockClient(std::make_shared(socketFile, connRetryTimeoutMS)), 21 | m_dataCache(ackTimeoutMS? std::make_shared>() : nullptr), 22 | m_incomingQueue(std::make_shared>(bufferLimit)), 23 | m_sockReader(new DataReader(m_sockClient, m_dataCache)), 24 | m_dataResender(ackTimeoutMS? new DataResender(m_sockClient, m_dataCache, 25 | ackTimeoutMS, resendIntervalMS): nullptr), 26 | m_dataSender(new DataSender(m_sockClient, m_dataCache, m_incomingQueue)) 27 | { 28 | } 29 | 30 | BufferedLogger::~BufferedLogger() 31 | { 32 | try { 33 | ADD_INFO_TRACE; 34 | 35 | m_sockClient->Stop(); 36 | m_incomingQueue->stop_once_empty(); 37 | 38 | m_dataSender->Stop(); 39 | if (m_dataResender) { 40 | m_dataResender->Stop(); 41 | } 42 | m_sockReader->Stop(); 43 | 44 | if (m_senderTask.valid()) { 45 | m_senderTask.get(); 46 | } 47 | if (m_resenderTask.valid()) { 48 | m_resenderTask.get(); 49 | } 50 | if (m_readerTask.valid()) { 51 | m_readerTask.get(); 52 | } 53 | } 54 | catch(const std::exception& ex) 55 | { 56 | Log(TraceLevel::Error, "unexpected exception: " << ex.what()); 57 | } 58 | catch(...) 59 | {} // no exception thrown from destructor 60 | } 61 | 62 | void 63 | BufferedLogger::StartWorkers() 64 | { 65 | m_senderTask = std::async(std::launch::async, [this] { m_dataSender->Run(); }); 66 | m_readerTask = std::async(std::launch::async, [this] { m_sockReader->Run(); }); 67 | if (m_dataResender) { 68 | m_resenderTask = std::async(std::launch::async, [this] { m_dataResender->Run(); }); 69 | } 70 | } 71 | 72 | void 73 | BufferedLogger::AddData( 74 | LogItemPtr item 75 | ) 76 | { 77 | if (!item) { 78 | throw std::invalid_argument("AddData(): unexpected NULL in input parameter."); 79 | } 80 | std::call_once(m_initOnceFlag, &BufferedLogger::StartWorkers, this); 81 | m_incomingQueue->push(std::move(item)); 82 | } 83 | 84 | bool 85 | BufferedLogger::WaitUntilAllSend( 86 | uint32_t timeoutMS 87 | ) 88 | { 89 | ADD_DEBUG_TRACE; 90 | m_incomingQueue->stop_once_empty(); 91 | auto status = m_senderTask.wait_for(std::chrono::milliseconds(timeoutMS)); 92 | return (std::future_status::ready == status); 93 | } 94 | 95 | 96 | size_t 97 | BufferedLogger::GetNumTagsRead() const 98 | { 99 | return m_sockReader->GetNumTagsRead(); 100 | } 101 | 102 | size_t 103 | BufferedLogger::GetTotalSend() const 104 | { 105 | return m_dataSender->GetNumSend() + GetTotalResend(); 106 | } 107 | 108 | size_t 109 | BufferedLogger::GetTotalSendSuccess() const 110 | { 111 | return m_dataSender->GetNumSuccess(); 112 | } 113 | 114 | size_t 115 | BufferedLogger::GetTotalResend() const 116 | { 117 | return (m_dataResender? m_dataResender->GetTotalSendTimes() : 0); 118 | } 119 | 120 | size_t 121 | BufferedLogger::GetNumItemsInCache() const 122 | { 123 | return (m_dataCache? m_dataCache->Size() : 0); 124 | } 125 | -------------------------------------------------------------------------------- /src/outmdsd/BufferedLogger.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #ifndef __ENDPOINTLOGGER_H__ 4 | #define __ENDPOINTLOGGER_H__ 5 | 6 | #include 7 | #include 8 | #include "LogItemPtr.h" 9 | 10 | namespace EndpointLog { 11 | 12 | template class ConcurrentQueue; 13 | template class ConcurrentMap; 14 | class SocketClient; 15 | class DataReader; 16 | class DataResender; 17 | class DataSender; 18 | 19 | // This class implements a data logger to a socket server with retry when socket 20 | // failure occurs. It uses multiple threads internally: 21 | // - The main thread will add the data to a shared, concurrent queue then move to 22 | // next data item. 23 | // - A sender thread will pop and send data from the queue to the socket server. 24 | // - A reader thread will read and process ack data from the socket server. 25 | // - An optional resender thread will resend data that failed to be sent before. 26 | // 27 | class BufferedLogger 28 | { 29 | public: 30 | /// 31 | /// Construct a logger that'll send data to a Unix domain socket. 32 | /// 33 | /// full path to socket file 34 | /// max milliseconds to wait for ack from socket server. 35 | /// After timeout, record will be dropped from cache. If this parameter's value 36 | /// is 0, no data will be cached. 37 | /// message resend interval in milliseconds. 38 | /// number of milliseconds to timeout socket 39 | /// connect() retry 40 | /// max LogItem to buffer. 0 means no limit. 41 | BufferedLogger( 42 | const std::string& socketFile, 43 | unsigned int ackTimeoutMS, 44 | unsigned int resendIntervalMS, 45 | unsigned int connRetryTimeoutMS, 46 | size_t bufferLimit 47 | ); 48 | 49 | ~BufferedLogger(); 50 | 51 | BufferedLogger(const BufferedLogger&) = delete; 52 | BufferedLogger& operator=(const BufferedLogger &) = delete; 53 | 54 | BufferedLogger(BufferedLogger&& h) = delete; 55 | BufferedLogger& operator=(BufferedLogger&& h) = delete; 56 | 57 | /// 58 | /// Add new data item to logger. 59 | /// Throw exception for any error. 60 | /// 61 | /// A new logger item. 62 | void AddData(LogItemPtr item); 63 | 64 | /// Wait until all the items are sent by the sender thread or timed out. 65 | /// Return true if all the items are sent out, false if timed out. 66 | bool WaitUntilAllSend(uint32_t timeoutMS); 67 | 68 | /// Return total number of ack tags processed by reader thread. 69 | size_t GetNumTagsRead() const; 70 | 71 | /// Return total number of Send() called, including sender thread and resender thread. 72 | size_t GetTotalSend() const; 73 | 74 | /// Return total number of successful Send() by sender thread. 75 | size_t GetTotalSendSuccess() const; 76 | 77 | /// Return total number of Send() called by data resender 78 | size_t GetTotalResend() const; 79 | 80 | /// Return number of items in the backup cache, either not resent yet, 81 | /// or not acknowledged yet 82 | size_t GetNumItemsInCache() const; 83 | 84 | private: 85 | void StartWorkers(); 86 | 87 | private: 88 | std::shared_ptr m_sockClient; 89 | std::shared_ptr> m_dataCache; // to store 90 | std::shared_ptr> m_incomingQueue; // to store incoming data item. 91 | 92 | std::future m_senderTask; 93 | std::future m_readerTask; 94 | std::future m_resenderTask; 95 | 96 | std::unique_ptr m_sockReader; // to read ack from socket server. 97 | std::unique_ptr m_dataResender; // to resend failed data to socket server. 98 | std::unique_ptr m_dataSender; // to send data to socket server. 99 | 100 | std::once_flag m_initOnceFlag; // a flag to make sure something is called exactly once. 101 | }; 102 | 103 | } // namespace 104 | 105 | #endif // __ENDPOINTLOGGER_H__ 106 | -------------------------------------------------------------------------------- /src/outmdsd/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | include_directories( 2 | ) 3 | 4 | set(SOURCES 5 | BufferedLogger.cc 6 | DataReader.cc 7 | DataResender.cc 8 | DataSender.cc 9 | DjsonLogItem.cc 10 | FileTracer.cc 11 | IdMgr.cc 12 | LogItem.cc 13 | SockAddr.cc 14 | SocketClient.cc 15 | SocketLogger.cc 16 | SyslogTracer.cc 17 | Trace.cc 18 | ) 19 | 20 | # static lib only 21 | add_library(outmdsd STATIC ${SOURCES}) 22 | 23 | install(TARGETS 24 | outmdsd 25 | ARCHIVE DESTINATION ${CMAKE_BINARY_DIR}/release/lib) 26 | -------------------------------------------------------------------------------- /src/outmdsd/ConcurrentMap.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef __ENDPOINTLOG_CONCURRENTMAP_H__ 3 | #define __ENDPOINTLOG_CONCURRENTMAP_H__ 4 | 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | 13 | namespace EndpointLog { 14 | 15 | /// This class implements thread-safe add/remove items from a hash cache. 16 | /// It is not designed to be a generic map class. It uses specific 17 | /// key/value pairs and only implements necessary APIs used in this project. 18 | 19 | template 20 | class ConcurrentMap { 21 | public: 22 | ConcurrentMap() = default; 23 | ~ConcurrentMap() = default; 24 | 25 | ConcurrentMap(const ConcurrentMap& other) 26 | { 27 | std::lock_guard lk(other.m_cacheMutex); 28 | m_cache = other.m_cache; 29 | } 30 | 31 | ConcurrentMap(ConcurrentMap&& other) 32 | { 33 | std::lock_guard lk(other.m_cacheMutex); 34 | m_cache = std::move(other.m_cache); 35 | } 36 | 37 | ConcurrentMap & operator=(const ConcurrentMap& other) 38 | { 39 | if (this != &other) { 40 | std::unique_lock lhs_lk(m_cacheMutex, std::defer_lock); 41 | std::unique_lock rhs_lk(other.m_cacheMutex, std::defer_lock); 42 | std::lock(lhs_lk, rhs_lk); 43 | m_cache = other.m_cache; 44 | } 45 | return *this; 46 | } 47 | 48 | ConcurrentMap & operator=(ConcurrentMap&& other) 49 | { 50 | if (this != &other) { 51 | std::unique_lock lhs_lk(m_cacheMutex, std::defer_lock); 52 | std::unique_lock rhs_lk(other.m_cacheMutex, std::defer_lock); 53 | std::lock(lhs_lk, rhs_lk); 54 | m_cache = std::move(other.m_cache); 55 | } 56 | return *this; 57 | } 58 | 59 | /// Add new key, value pair 60 | /// If key exists, old entry will be replaced. 61 | void Add(const std::string & key, ValueType value) 62 | { 63 | if (key.empty()) { 64 | throw std::invalid_argument("Invalid empty string for map key."); 65 | } 66 | 67 | std::lock_guard lk(m_cacheMutex); 68 | m_cache[key] = std::move(value); 69 | } 70 | 71 | /// Erase an item with given key 72 | /// Return 1 if erased, 0 if nothing is erased. 73 | size_t Erase(const std::string & key) 74 | { 75 | if (key.empty()) { 76 | return 0; 77 | } 78 | 79 | std::lock_guard lk(m_cacheMutex); 80 | auto nErased = m_cache.erase(key); 81 | return nErased; 82 | } 83 | 84 | /// Erase a list of items given their keys 85 | /// Return number of items erased. 86 | size_t Erase(const std::vector& keylist) 87 | { 88 | if (keylist.empty()) { 89 | return 0; 90 | } 91 | 92 | size_t nTotal = 0; 93 | std::lock_guard lk(m_cacheMutex); 94 | for(const auto & key : keylist) { 95 | auto nErased = m_cache.erase(key); 96 | nTotal += nErased; 97 | } 98 | 99 | return nTotal; 100 | } 101 | 102 | /// Return all the keys such that fn(value) == true. 103 | std::vector FilterEach(const std::function& fn) 104 | { 105 | std::vector keylist; 106 | std::lock_guard lk(m_cacheMutex); 107 | 108 | for(const auto & item : m_cache) { 109 | if(fn(item.second)) { 110 | keylist.push_back(item.first); 111 | } 112 | } 113 | return keylist; 114 | } 115 | 116 | /// Apply fn on each value of the cache. 117 | void ForEach(const std::function& fn) 118 | { 119 | std::lock_guard lk(m_cacheMutex); 120 | ForEachUnsafe(fn); 121 | } 122 | 123 | /// Apply fn on each value of the cache without locking the mutex. 124 | /// This is not designed to be thread-safe. 125 | void ForEachUnsafe(const std::function& fn) 126 | { 127 | std::for_each(m_cache.begin(), m_cache.end(), 128 | [fn](typename decltype(m_cache)::value_type & item) { fn(item.second); }); 129 | } 130 | 131 | /// Get an entry with the key. 132 | /// If the key doesn't exist, throw std::out_of_range exception. 133 | ValueType Get(const std::string & key) 134 | { 135 | if (key.empty()) { 136 | throw std::invalid_argument("Invalid empty string for map key."); 137 | } 138 | std::lock_guard lk(m_cacheMutex); 139 | auto item = m_cache.find(key); 140 | if (item == m_cache.end()) { 141 | throw std::out_of_range("ConcurrentMap::Get(): key not found " + key); 142 | } 143 | return item->second; 144 | } 145 | 146 | size_t Size() const 147 | { 148 | std::lock_guard lk(m_cacheMutex); 149 | return m_cache.size(); 150 | } 151 | 152 | private: 153 | std::unordered_map m_cache; 154 | mutable std::mutex m_cacheMutex; 155 | }; 156 | 157 | } // namespace 158 | #endif // __ENDPOINTLOG_CONCURRENTMAP_H__ 159 | -------------------------------------------------------------------------------- /src/outmdsd/ConcurrentQueue.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef _CONCURRENT_QUEUE_H__ 3 | #define _CONCURRENT_QUEUE_H__ 4 | 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | 11 | namespace EndpointLog { 12 | 13 | /// This class implements a thread-safe concurrent queue with an optional max size limit. 14 | /// When a max size is set, after max size is reached, before pushing new element, oldest element 15 | /// will be popped and dropped. 16 | /// 17 | /// Most of the code are from "C++ Concurrency In Action" book 18 | /// source code listing 4.5. 19 | /// https://manning-content.s3.amazonaws.com/download/0/78f6c43-a41b-4eb0-82f2-44c24eba51ad/CCiA_SourceCode.zip 20 | template 21 | class ConcurrentQueue 22 | { 23 | private: 24 | mutable std::mutex mut; // mutex to lock the queue 25 | std::queue data_queue; // underling queue 26 | std::condition_variable data_cond; // CV for queue synchronization 27 | std::atomic stopOnceEmpty{false}; // if true, notify CV to stop any further waiting once queue is empty 28 | size_t m_maxSize; // max number of items to hold in the queue. 0 means it will limited by std::queue's max size. 29 | 30 | public: 31 | ConcurrentQueue(size_t maxSize = 0) : m_maxSize(maxSize) 32 | { 33 | } 34 | 35 | ConcurrentQueue(const ConcurrentQueue& other) 36 | { 37 | std::lock_guard lk(other.mut); 38 | data_queue = other.data_queue; 39 | m_maxSize = other.m_maxSize; 40 | } 41 | 42 | ConcurrentQueue(ConcurrentQueue&& other) 43 | { 44 | std::lock_guard lk(other.mut); 45 | data_queue = std::move(other.data_queue); 46 | data_cond = std::move(other.data_cond); 47 | m_maxSize = other.m_maxSize; 48 | } 49 | 50 | ConcurrentQueue& operator=(const ConcurrentQueue&) = delete; 51 | ConcurrentQueue& operator=(ConcurrentQueue&&) = delete; 52 | 53 | ~ConcurrentQueue() 54 | { 55 | stop_once_empty(); 56 | } 57 | 58 | void push(const T & new_value) 59 | { 60 | std::lock_guard lk(mut); 61 | if (is_full()) { 62 | data_queue.pop(); 63 | } 64 | data_queue.push(new_value); 65 | data_cond.notify_one(); 66 | } 67 | 68 | void push(T && new_value) 69 | { 70 | std::lock_guard lk(mut); 71 | if (is_full()) { 72 | data_queue.pop(); 73 | } 74 | data_queue.emplace(std::move(new_value)); 75 | data_cond.notify_one(); 76 | } 77 | 78 | /// Wait until any item is available in the queue, then pop it 79 | void wait_and_pop(T& value) 80 | { 81 | std::unique_lock lk(mut); 82 | data_cond.wait(lk,[this]{ return (!data_queue.empty() || stopOnceEmpty);}); 83 | if (data_queue.empty() && stopOnceEmpty) { 84 | return; 85 | } 86 | value = data_queue.front(); 87 | data_queue.pop(); 88 | } 89 | 90 | /// Wait until any item is available in the queue, then pop it 91 | std::shared_ptr wait_and_pop() 92 | { 93 | std::unique_lock lk(mut); 94 | data_cond.wait(lk,[this]{return (!data_queue.empty() || stopOnceEmpty);}); 95 | if (data_queue.empty() && stopOnceEmpty) { 96 | return nullptr; 97 | } 98 | std::shared_ptr res(std::make_shared(data_queue.front())); 99 | data_queue.pop(); 100 | return res; 101 | } 102 | 103 | /// If queue is empty, pop nothing, return false. 104 | /// If queue is not empty, pop and return the popped item. 105 | bool try_pop(T& value) 106 | { 107 | std::lock_guard lk(mut); 108 | if(data_queue.empty()) { 109 | return false; 110 | } 111 | value = data_queue.front(); 112 | data_queue.pop(); 113 | } 114 | 115 | /// If queue is empty, pop nothing, return false. 116 | /// If queue is not empty, pop and return the popped item. 117 | std::shared_ptr try_pop() 118 | { 119 | std::lock_guard lk(mut); 120 | if(data_queue.empty()) { 121 | return std::shared_ptr(); 122 | } 123 | std::shared_ptr res(std::make_shared(data_queue.front())); 124 | data_queue.pop(); 125 | return res; 126 | } 127 | 128 | bool empty() const 129 | { 130 | std::lock_guard lk(mut); 131 | return data_queue.empty(); 132 | } 133 | 134 | size_t size() const 135 | { 136 | std::lock_guard lk(mut); 137 | return data_queue.size(); 138 | } 139 | 140 | /// Notify queue to stop any further waiting once it is empty. 141 | /// If queue is not empty, continue. 142 | void stop_once_empty() 143 | { 144 | std::lock_guard lk(mut); 145 | stopOnceEmpty = true; 146 | data_cond.notify_all(); 147 | } 148 | private: 149 | bool is_full() const { 150 | return (0 < m_maxSize && data_queue.size() == m_maxSize); 151 | } 152 | }; 153 | 154 | } // namespace 155 | 156 | #endif // _CONCURRENTQUEUE_H__ 157 | -------------------------------------------------------------------------------- /src/outmdsd/DataReader.cc: -------------------------------------------------------------------------------- 1 | extern "C" { 2 | #include 3 | } 4 | #include 5 | 6 | #include "ConcurrentMap.h" 7 | #include "DataReader.h" 8 | #include "Exceptions.h" 9 | #include "Trace.h" 10 | #include "TraceMacros.h" 11 | #include "SocketClient.h" 12 | 13 | using namespace EndpointLog; 14 | 15 | DataReader::DataReader( 16 | const std::shared_ptr & sockClient, 17 | const std::shared_ptr> & dataCache 18 | ) : 19 | m_socketClient(sockClient), 20 | m_dataCache(dataCache) 21 | { 22 | assert(m_socketClient); 23 | } 24 | 25 | DataReader::~DataReader() 26 | { 27 | try { 28 | Stop(); 29 | } 30 | catch(const std::exception & ex) { 31 | Log(TraceLevel::Error, "~DataReader() exception: " << ex.what()); 32 | } 33 | catch(...) { 34 | } // no exception thrown from destructor 35 | } 36 | 37 | void 38 | DataReader::Stop() 39 | { 40 | ADD_INFO_TRACE; 41 | m_stopRead = true; 42 | } 43 | 44 | void 45 | DataReader::Run() 46 | { 47 | ADD_INFO_TRACE; 48 | 49 | try { 50 | std::string partialData; 51 | while(true) { 52 | if (!DoRead(partialData)) { 53 | break; 54 | } 55 | } 56 | } 57 | catch(const ReaderInterruptException&) { 58 | Log(TraceLevel::Info, "DataReader is interrupted. Abort reader thread."); 59 | } 60 | catch(const std::exception & ex) { 61 | Log(TraceLevel::Error, "DataReader unexpected exception: " << ex.what()); 62 | } 63 | } 64 | 65 | bool 66 | DataReader::DoRead( 67 | std::string & partialData 68 | ) 69 | { 70 | ADD_DEBUG_TRACE; 71 | 72 | char buf[512]; 73 | try { 74 | InterruptPoint(); 75 | auto readRtn = m_socketClient->Read(buf, sizeof(buf)-1); 76 | if (-1 == readRtn) { 77 | Log(TraceLevel::Debug, "SocketClient is stopped. Abort read."); 78 | return false; 79 | } 80 | 81 | if (readRtn > 0) { 82 | InterruptPoint(); 83 | buf[readRtn] = '\0'; 84 | partialData = ProcessData(partialData+buf); 85 | Log(TraceLevel::Debug, "DoRead partialData='" << partialData << "'."); 86 | } 87 | } 88 | catch(const SocketException & ex) { 89 | Log(TraceLevel::Info, "SocketException " << ex.what()); 90 | } 91 | return true; 92 | } 93 | 94 | void 95 | DataReader::InterruptPoint() const 96 | { 97 | if (m_stopRead) { 98 | throw ReaderInterruptException(); 99 | } 100 | } 101 | 102 | std::string 103 | DataReader::ProcessData( 104 | const std::string & str 105 | ) 106 | { 107 | ADD_DEBUG_TRACE; 108 | 109 | Log(TraceLevel::Debug, "ProcessData: '" << str << "'."); 110 | 111 | if (str.empty()) { 112 | return std::string(); 113 | } 114 | 115 | auto dPos = str.find_last_of('\n'); 116 | if (std::string::npos == dPos) { 117 | return str; 118 | } 119 | 120 | std::istringstream iss(str); 121 | std::string item; 122 | while(std::getline(iss, item, '\n')) { 123 | if (!iss.eof()) { 124 | ProcessItem(item); 125 | } 126 | } 127 | 128 | if (!item.empty() && dPos == (str.size()-1)) { 129 | ProcessItem(item); 130 | return std::string(); 131 | } 132 | 133 | return str.substr(dPos+1); 134 | } 135 | 136 | void 137 | DataReader::ProcessItem( 138 | const std::string& item 139 | ) 140 | { 141 | ADD_DEBUG_TRACE; 142 | 143 | if (item.empty()) { 144 | Log(TraceLevel::Warning, "unexpected empty ack item found."); 145 | return; 146 | } 147 | 148 | m_nTagsRead++; 149 | Log(TraceLevel::Debug, "Got item='" << item << "'"); 150 | 151 | auto p = item.find(':'); 152 | if (p == std::string::npos) { 153 | ProcessTag(item); 154 | } 155 | else { 156 | auto tag = item.substr(0, p); 157 | auto ackStatus = item.substr(p+1); 158 | ProcessTag(tag, ackStatus); 159 | } 160 | } 161 | 162 | void 163 | DataReader::ProcessTag( 164 | const std::string & tag 165 | ) 166 | { 167 | if (tag.empty()) { 168 | Log(TraceLevel::Warning, "unexpected empty tag found."); 169 | return; 170 | } 171 | 172 | if (m_dataCache) { 173 | if (1 != m_dataCache->Erase(tag)) { 174 | Log(TraceLevel::Warning, "tag '" << tag << "' is not found in backup cache"); 175 | } 176 | } 177 | } 178 | 179 | static std::unordered_map & 180 | GetAckStatusMap() 181 | { 182 | static std::unordered_map m = 183 | { 184 | { "0", "ACK_SUCCESS" }, 185 | { "1", "ACK_FAILED" }, 186 | { "2", "ACK_UNKNOWN_SCHEMA_ID" }, 187 | { "3", "ACK_DECODE_ERROR" }, 188 | { "4", "ACK_INVALID_SOURCE" }, 189 | { "5", "ACK_DUPLICATE_SCHEMA_ID" } 190 | }; 191 | return m; 192 | } 193 | 194 | static std::string 195 | GetAckStatusStr( 196 | const std::string & ackCode 197 | ) 198 | { 199 | auto m = GetAckStatusMap(); 200 | auto item = m.find(ackCode); 201 | if (item == m.end()) { 202 | return "Unknown-ACK-CODE"; 203 | } 204 | return item->second; 205 | } 206 | 207 | void 208 | DataReader::ProcessTag( 209 | const std::string & tag, 210 | const std::string & ackStatus 211 | ) 212 | { 213 | if (tag.empty()) { 214 | Log(TraceLevel::Warning, "unexpected empty tag found"); 215 | return; 216 | } 217 | if (ackStatus.empty()) { 218 | Log(TraceLevel::Warning, "unexpected empty ack status string found"); 219 | return; 220 | } 221 | 222 | if ("0" != ackStatus) { 223 | auto statusStr = GetAckStatusStr(ackStatus); 224 | Log(TraceLevel::Error, "unexpected mdsd ack status: " << statusStr << ", tag '" << tag << "'" ); 225 | } 226 | else { 227 | // Only remove item from cache if ack status is 0 (Success) 228 | if (m_dataCache) { 229 | if (1 != m_dataCache->Erase(tag)) { 230 | Log(TraceLevel::Warning, "tag '" << tag << "' is not found in backup cache"); 231 | } 232 | } 233 | } 234 | } 235 | -------------------------------------------------------------------------------- /src/outmdsd/DataReader.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef __ENDPOINT_DATAREADER_H__ 3 | #define __ENDPOINT_DATAREADER_H__ 4 | 5 | #include 6 | #include 7 | #include 8 | #include "LogItemPtr.h" 9 | 10 | namespace EndpointLog { 11 | 12 | template class ConcurrentMap; 13 | class SocketClient; 14 | 15 | /// This class implements a socket data reader. The data to be read 16 | /// are expected to be a series of either '\n' or ':\n'. 17 | /// 18 | /// If a shared cache is given, the item whose key equals to will be removed 19 | /// from dataCache. 20 | /// 21 | /// Once started, DataReader will run in an infinite loop until told to stop. 22 | /// 23 | class DataReader { 24 | public: 25 | /// Constructor 26 | /// socket client 27 | /// shared cache for backup data. Can be NULL. 28 | DataReader(const std::shared_ptr & sockClient, 29 | const std::shared_ptr> & dataCache); 30 | 31 | ~DataReader(); 32 | 33 | // DataReader is not copyable but movable 34 | DataReader(const DataReader& other) = delete; 35 | DataReader& operator=(const DataReader& other) = delete; 36 | 37 | DataReader(DataReader&& other) = default; 38 | DataReader& operator=(DataReader&& other) = default; 39 | 40 | /// Run the reading process in an infinite loop until told to stop. 41 | void Run(); 42 | 43 | /// Notify the reader to stop. 44 | void Stop(); 45 | 46 | /// Get total number of tags read. For testability. 47 | size_t GetNumTagsRead() const { return m_nTagsRead; } 48 | 49 | private: 50 | /// Try to read data from the socket. Process the data if any. 51 | /// To save partial data that are not processed yet 52 | /// Return true if valid data (including 0-byte) are read, return false otherwise. 53 | bool DoRead(std::string& partialData); 54 | 55 | /// Define interruption point in the loop. 56 | void InterruptPoint() const; 57 | 58 | /// Process data read from the socket. 59 | /// data string to be processed 60 | /// Return the partial unprocessed string. 61 | std::string ProcessData(const std::string & str); 62 | 63 | /// Process each item of the read data. An item is a substring delimited by '\n' 64 | /// in the read data. The item doesn't contain the '\n'. 65 | /// Each item is in either of the following formats 66 | /// - 67 | /// - : 68 | void ProcessItem(const std::string& item); 69 | 70 | void ProcessTag(const std::string & tag); 71 | void ProcessTag(const std::string & tag, const std::string & ackStatus); 72 | 73 | private: 74 | std::shared_ptr m_socketClient; 75 | std::shared_ptr> m_dataCache; 76 | 77 | std::atomic m_stopRead{false}; /// flag to stop further reading. 78 | 79 | std::atomic m_nTagsRead{0}; /// number of tags read. for testability. 80 | }; 81 | 82 | } // namespace 83 | 84 | #endif // __ENDPOINT_DATAREADER_H__ 85 | -------------------------------------------------------------------------------- /src/outmdsd/DataResender.cc: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | #include "ConcurrentMap.h" 4 | #include "DataResender.h" 5 | #include "SocketClient.h" 6 | #include "Exceptions.h" 7 | #include "Trace.h" 8 | #include "TraceMacros.h" 9 | #include "LogItem.h" 10 | 11 | using namespace EndpointLog; 12 | 13 | DataResender::DataResender( 14 | const std::shared_ptr & sockClient, 15 | const std::shared_ptr> & dataCache, 16 | unsigned int ackTimeoutMS, 17 | unsigned int resendIntervalMS 18 | ) : 19 | m_socketClient(sockClient), 20 | m_dataCache(dataCache), 21 | m_ackTimeoutMS(ackTimeoutMS), 22 | m_resendIntervalMS(resendIntervalMS) 23 | { 24 | assert(m_socketClient); 25 | assert(m_dataCache); 26 | 27 | if (0 == ackTimeoutMS) { 28 | throw std::invalid_argument("DataResender: ack timeout must be a positive integer."); 29 | } 30 | 31 | if (0 == m_resendIntervalMS) { 32 | throw std::invalid_argument("DataResender: resend interval must be a positive integer."); 33 | } 34 | 35 | } 36 | 37 | DataResender::~DataResender() 38 | { 39 | try { 40 | Stop(); 41 | } 42 | catch(const std::exception & ex) { 43 | Log(TraceLevel::Error, "~DataResender() exception: " << ex.what()); 44 | } 45 | catch(...) { 46 | } // no exception thrown from destructor 47 | } 48 | 49 | size_t 50 | DataResender::Run() 51 | { 52 | ADD_INFO_TRACE; 53 | 54 | size_t counter = 0; 55 | 56 | while(!m_stopMe) { 57 | WaitForNextResend(); 58 | if (m_stopMe) { 59 | break; 60 | } 61 | 62 | ResendOnce(); 63 | counter++; 64 | } 65 | 66 | Log(TraceLevel::Debug, "DataResender finished: total resend round: " << counter << "."); 67 | return counter; 68 | } 69 | 70 | void 71 | DataResender::Stop() 72 | { 73 | ADD_INFO_TRACE; 74 | 75 | std::lock_guard lck(m_timerMutex); 76 | m_stopMe = true; 77 | m_timerCV.notify_one(); 78 | } 79 | 80 | void 81 | DataResender::WaitForNextResend() 82 | { 83 | ADD_TRACE_TRACE; 84 | std::unique_lock lck(m_timerMutex); 85 | 86 | // Wait for m_resendIntervalMS until it is told to abort by m_stopMe 87 | m_timerCV.wait_for(lck, std::chrono::milliseconds(m_resendIntervalMS), [this] { 88 | return m_stopMe.load(); 89 | }); 90 | } 91 | 92 | void 93 | DataResender::ResendOnce() 94 | { 95 | ADD_TRACE_TRACE; 96 | 97 | try { 98 | if (m_dataCache->Size() > 0) { 99 | ResendData(); 100 | } 101 | } 102 | catch(const std::exception& ex) { 103 | Log(TraceLevel::Error, "DataResender send failed: " << ex.what()); 104 | } 105 | } 106 | 107 | void 108 | DataResender::ResendData() 109 | { 110 | ADD_TRACE_TRACE; 111 | 112 | // Check whether any cached items need to be dropped 113 | std::function CheckItemAge = [this](LogItemPtr itemPtr) 114 | { 115 | if (itemPtr && static_cast(itemPtr->GetLastTouchMilliSeconds()) > m_ackTimeoutMS) { 116 | return true; 117 | } 118 | return false; 119 | }; 120 | 121 | // To minimize locking on m_dataCache, copy it first. Then run filter function 122 | // on the copied cache. 123 | auto cacheBak = *m_dataCache; 124 | auto keysToRemove = cacheBak.FilterEach(CheckItemAge); 125 | 126 | // The Erase() and FilterEach() are not protected together. It is OK that 127 | // some key is gone in the data cache when Erase() is called. 128 | m_dataCache->Erase(keysToRemove); 129 | 130 | for (const auto & key : keysToRemove) { 131 | Log(TraceLevel::Trace, "obsolete key erased: '" << key << "'."); 132 | } 133 | 134 | // To minimize locking on m_dataCache, copy it first. Then send data in the 135 | // copied cache. 136 | auto cacheCopy = *m_dataCache; 137 | 138 | try { 139 | std::function SendItem = [this](LogItemPtr itemPtr) 140 | { 141 | if (itemPtr) { 142 | m_socketClient->Send(itemPtr->GetData()); 143 | m_totalSend++; 144 | } 145 | }; 146 | 147 | cacheCopy.ForEachUnsafe(SendItem); 148 | Log(TraceLevel::Trace, "ResendData(): m_totalSend=" << m_totalSend); 149 | } 150 | catch(const SocketException & ex) { 151 | Log(TraceLevel::Info, "SocketException: " << ex.what()); 152 | } 153 | } 154 | -------------------------------------------------------------------------------- /src/outmdsd/DataResender.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef __ENDPOINT_DATARESENDER_H__ 3 | #define __ENDPOINT_DATARESENDER_H__ 4 | 5 | #include 6 | #include 7 | #include 8 | #include 9 | 10 | #include "LogItemPtr.h" 11 | 12 | namespace EndpointLog { 13 | 14 | template class ConcurrentMap; 15 | class SocketClient; 16 | 17 | /// This class will resend data in a shared cache to a socket server 18 | /// in a multi-thread system, while other threads keep on inserting data to 19 | /// the same shared cache. 20 | /// 21 | /// It will run in a resend-sleep loop until it is told to stop. 22 | /// 23 | /// It reads data and removes obsolete data from the shared cache. It doesn't 24 | /// add data to the cache. 25 | /// 26 | class DataResender { 27 | public: 28 | /// Constructor. 29 | /// shared socket client object 30 | /// shared cache. DataResender reads and removes data from it. 31 | /// Other threads will add items to it. 32 | /// max milliseconds to wait for socket server acknowledge 33 | /// before removing data from cache. 34 | /// milliseconds to do resending 35 | DataResender( 36 | const std::shared_ptr & sockClient, 37 | const std::shared_ptr> & dataCache, 38 | unsigned int ackTimeoutMS, 39 | unsigned int resendIntervalMS 40 | ); 41 | 42 | ~DataResender(); 43 | 44 | // not copyable, not movable 45 | DataResender(const DataResender& other) = delete; 46 | DataResender& operator=(const DataResender& other) = delete; 47 | 48 | DataResender(DataResender&& other) = delete; 49 | DataResender& operator=(DataResender&& other) = delete; 50 | 51 | /// 52 | /// Run the resend-sleep loop. It will loop forever until it is told to stop. 53 | /// Return the number of timers triggered. 54 | /// 55 | size_t Run(); 56 | 57 | /// 58 | /// Told the resending loop to stop. Typically called in a thread different 59 | /// from the one calls Run(). 60 | /// 61 | void Stop(); 62 | 63 | size_t GetTotalSendTimes() const { return m_totalSend; } 64 | 65 | private: 66 | /// Wait for m_resendIntervalMS before next resending turn. 67 | void WaitForNextResend(); 68 | 69 | /// Resend all valid data and handle exceptions. 70 | void ResendOnce(); 71 | 72 | /// 73 | /// Resend all valid data in the cache to the socket server. 74 | /// Obsolete data based on ack timeout will be removed before being resent. 75 | /// 76 | void ResendData(); 77 | 78 | private: 79 | std::shared_ptr m_socketClient; 80 | std::shared_ptr> m_dataCache; 81 | 82 | unsigned int m_ackTimeoutMS; // if ack is not received in this time, item is removed from cache. 83 | unsigned int m_resendIntervalMS; // cached items resending interval in milliseconds. 84 | 85 | std::atomic m_stopMe { false }; // A flag used to stop resending loop. 86 | 87 | /// m_timerMutex and m_timerCV are used to create an interruptible blocking wait. 88 | std::mutex m_timerMutex; 89 | std::condition_variable m_timerCV; 90 | 91 | std::atomic m_totalSend {0}; // total Send() is called on socket. for testability 92 | }; 93 | 94 | } // namespace 95 | 96 | #endif // __ENDPOINT_DATARESENDER_H__ 97 | -------------------------------------------------------------------------------- /src/outmdsd/DataSender.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include "ConcurrentMap.h" 3 | #include "ConcurrentQueue.h" 4 | #include "DataSender.h" 5 | #include "SocketClient.h" 6 | #include "Exceptions.h" 7 | #include "Trace.h" 8 | #include "TraceMacros.h" 9 | #include "LogItem.h" 10 | 11 | using namespace EndpointLog; 12 | 13 | class InterruptException {}; 14 | 15 | DataSender::DataSender( 16 | const std::shared_ptr & sockClient, 17 | const std::shared_ptr> & dataCache, 18 | const std::shared_ptr> & incomingQueue 19 | ) : 20 | m_socketClient(sockClient), 21 | m_dataCache(dataCache), 22 | m_incomingQueue(incomingQueue) 23 | { 24 | assert(m_socketClient); 25 | assert(m_incomingQueue); 26 | } 27 | 28 | DataSender::~DataSender() 29 | { 30 | try { 31 | Stop(); 32 | } 33 | catch(const std::exception & ex) { 34 | Log(TraceLevel::Error, "~DataSender() exception: " << ex.what()); 35 | } 36 | catch(...) { 37 | } // no exception thrown from destructor 38 | } 39 | 40 | void 41 | DataSender::Stop() 42 | { 43 | ADD_INFO_TRACE; 44 | m_stopSender = true; 45 | } 46 | 47 | void 48 | DataSender::InterruptPoint() const 49 | { 50 | if (m_stopSender) { 51 | throw InterruptException(); 52 | } 53 | } 54 | 55 | void 56 | DataSender::Run() 57 | { 58 | try { 59 | ADD_INFO_TRACE; 60 | 61 | while(!m_stopSender) { 62 | LogItemPtr item; 63 | m_incomingQueue->wait_and_pop(item); 64 | 65 | // When ConcurrentQueue is empty and stopped, wait_and_pop() do nothing, 66 | // so item is still pointing to nullptr 67 | if (!item) { 68 | assert(0 == m_incomingQueue->size()); 69 | Log(TraceLevel::Info, "Abort Run() because data queue is aborted."); 70 | break; 71 | } 72 | 73 | InterruptPoint(); 74 | 75 | if (!m_dataCache) { 76 | // If no caching, send it out immediately 77 | Send(item->GetData()); 78 | } 79 | else { 80 | // Move item to cache first before sending it out. 81 | // This makes sure that the cache has the tag in the thread 82 | // where response is received and handled. 83 | item->Touch(); 84 | auto tag = item->GetTag(); 85 | m_dataCache->Add(tag, std::move(item)); 86 | 87 | auto dataItem = m_dataCache->Get(tag); 88 | if (dataItem) { 89 | InterruptPoint(); 90 | Send(m_dataCache->Get(tag)->GetData()); 91 | } 92 | } 93 | 94 | InterruptPoint(); 95 | } 96 | } 97 | catch(const InterruptException&) { 98 | Log(TraceLevel::Debug, "DataSender is interrupted. Abort now."); 99 | } 100 | catch(const std::exception & ex) { 101 | Log(TraceLevel::Error, "DataSender hits unexpected exception: " << ex.what()); 102 | } 103 | } 104 | 105 | // Send data and catch SocketException. 106 | // Because DataResender can keep on resending the failed data until timed out, 107 | // it shouldn't abort further DataSender::Run(), and also log this as an information. 108 | void 109 | DataSender::Send(const char* data) 110 | { 111 | ADD_TRACE_TRACE; 112 | try { 113 | m_numSend++; 114 | m_socketClient->Send(data); 115 | m_numSuccess++; 116 | Log(TraceLevel::Trace, "m_numSend=" << m_numSend << "; m_numSuccess=" << m_numSuccess); 117 | } 118 | catch(const SocketException & ex) { 119 | Log(TraceLevel::Info, "DataSender Send() SocketException: " << ex.what()); 120 | } 121 | } 122 | -------------------------------------------------------------------------------- /src/outmdsd/DataSender.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef __ENDPOINT_DATASENDER_H__ 3 | #define __ENDPOINT_DATASENDER_H__ 4 | 5 | #include 6 | #include 7 | 8 | #include "LogItemPtr.h" 9 | 10 | namespace EndpointLog { 11 | 12 | template class ConcurrentQueue; 13 | template class ConcurrentMap; 14 | class SocketClient; 15 | 16 | /// This class will keep on sending incoming data in a shared queue to a 17 | /// socket server in a multi-thread system. Other threads will keep on 18 | /// pushing new data to the same shared queue (see BufferedLogger class). 19 | /// 20 | /// DataSender will run in an infinite loop to pop and send each item 21 | /// from the shared queue to socket server. If no item to pop, it will 22 | /// wait until there is item in the queue. 23 | /// 24 | /// To avoid message loss, each item can be optionally saved to a cache for 25 | /// future resend (see DataResender class). 26 | /// 27 | class DataSender { 28 | public: 29 | /// 30 | /// Constructor. 31 | /// socket client object 32 | /// cache for data backup if not NULL. If NULL, no backup 33 | /// data to be sent. DataSender will pop each item in the queue 34 | /// and send it to socket server. If no data to pop, it will wait until there is data to pop. 35 | /// 36 | DataSender( 37 | const std::shared_ptr & sockClient, 38 | const std::shared_ptr> & dataCache, 39 | const std::shared_ptr> & incomingQueue 40 | ); 41 | 42 | ~DataSender(); 43 | 44 | // DataSender is not copyable but movable. 45 | DataSender(const DataSender& other) = delete; 46 | DataSender& operator=(const DataSender& other) = delete; 47 | 48 | DataSender(DataSender&& other) = default; 49 | DataSender& operator=(DataSender&& other) = default; 50 | 51 | /// 52 | /// Run sending process. It will run in an infinite loop until it is told to stop. 53 | /// It will pop and process each item in the incoming queue. If queue is empty, 54 | /// it will wait until queue has item. 55 | /// 56 | void Run(); 57 | 58 | /// 59 | /// Notify sender to stop. This is typically called in a thread different from 60 | /// the Run() thread. 61 | /// 62 | void Stop(); 63 | 64 | /// Return total number of send. including fails and successes. 65 | size_t GetNumSend() const { return m_numSend; } 66 | 67 | /// Return total number of successful send. 68 | size_t GetNumSuccess() const { return m_numSuccess; } 69 | 70 | private: 71 | /// Define interruption point for Run() loop. 72 | void InterruptPoint() const; 73 | 74 | /// Send a data string. It will send all chars until it hits terminal NUL. 75 | /// NUL is not sent. 76 | void Send(const char* data); 77 | 78 | private: 79 | std::shared_ptr m_socketClient; 80 | std::shared_ptr> m_dataCache; // for data backup 81 | std::shared_ptr> m_incomingQueue; // incoming data queue 82 | 83 | std::atomic m_stopSender{false}; // a flag to notify sender loop to stop. 84 | 85 | size_t m_numSend = 0; // number of items trying to be sent. This includes fails and successes. 86 | size_t m_numSuccess = 0; // number of success send. 87 | }; 88 | 89 | } // namespace 90 | 91 | #endif // __ENDPOINT_DATASENDER_H__ 92 | -------------------------------------------------------------------------------- /src/outmdsd/DjsonLogItem.cc: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | #include "DjsonLogItem.h" 4 | #include "IdMgr.h" 5 | 6 | using namespace EndpointLog; 7 | 8 | const char* 9 | DjsonLogItem::GetData() 10 | { 11 | if (m_djsonData.empty()) { 12 | if (m_schemaAndData.empty()) { 13 | ComposeSchemaAndData(); 14 | } 15 | ComposeFullData(); 16 | } 17 | 18 | return m_djsonData.c_str(); 19 | } 20 | 21 | IdMgr& 22 | DjsonLogItem::GetIdMgr() 23 | { 24 | static IdMgr m; 25 | return m; 26 | } 27 | 28 | void 29 | DjsonLogItem::ComposeSchemaAndData() 30 | { 31 | std::ostringstream strm; 32 | 33 | ComposeSchema(strm); 34 | ComposeDataValue(strm); 35 | 36 | // free m_svlist capacity 37 | std::vector tmpv; 38 | tmpv.swap(m_svlist); 39 | 40 | m_schemaAndData = strm.str(); 41 | } 42 | 43 | void 44 | DjsonLogItem::ComposeSchema( 45 | std::ostringstream& strm 46 | ) 47 | { 48 | IdMgr::value_type_t cachedInfo; 49 | 50 | // The fields order are preserved. So schema with same names/types 51 | // but in different order will be treated different schemas. 52 | auto key = GetSchemaCacheKey(); 53 | if (GetIdMgr().GetItem(key, cachedInfo)) { 54 | strm << cachedInfo.first << "," << cachedInfo.second; 55 | } 56 | else { 57 | auto schemaArray = ComposeSchemaArray(); 58 | auto schemaId = GetIdMgr().FindOrInsert(key, schemaArray); 59 | strm << schemaId << "," << schemaArray; 60 | } 61 | } 62 | 63 | std::string 64 | DjsonLogItem::GetSchemaCacheKey() const 65 | { 66 | std::string tmpstr; 67 | for(const auto & item : m_svlist) { 68 | tmpstr.append(item.name).append(item.type); 69 | } 70 | return tmpstr; 71 | } 72 | 73 | std::string 74 | DjsonLogItem::ComposeSchemaArray() const 75 | { 76 | std::ostringstream strm; 77 | strm << "["; 78 | for (size_t i = 0; i < m_svlist.size(); i++) { 79 | strm << "[\"" << m_svlist[i].name << "\",\"" << m_svlist[i].type << "\"]"; 80 | if (i != (m_svlist.size()-1)) { 81 | strm << ","; 82 | } 83 | } 84 | strm << "],"; 85 | return strm.str(); 86 | } 87 | 88 | void 89 | DjsonLogItem::ComposeDataValue( 90 | std::ostringstream& strm 91 | ) const 92 | { 93 | strm << "["; 94 | for (size_t i = 0; i < m_svlist.size(); i++) { 95 | strm << m_svlist[i].value; 96 | if (i != (m_svlist.size()-1)) { 97 | strm << ","; 98 | } 99 | } 100 | strm << "]"; 101 | } 102 | 103 | void 104 | DjsonLogItem::ComposeFullData() 105 | { 106 | auto tag = GetTag(); 107 | size_t len = 2 + m_source.size() + 2 + tag.size() + 1 + m_schemaAndData.size() + 1; 108 | auto lenstr = std::to_string(len); 109 | 110 | m_djsonData.reserve(len + lenstr.size() + 1); 111 | m_djsonData = lenstr; 112 | m_djsonData.append("\n[\"").append(m_source).append("\",").append(tag).append(",").append(m_schemaAndData).append("]"); 113 | } 114 | -------------------------------------------------------------------------------- /src/outmdsd/DjsonLogItem.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #ifndef __ENDPOINT_DJSONLOGITEM_H__ 4 | #define __ENDPOINT_DJSONLOGITEM_H__ 5 | 6 | #include 7 | #include 8 | #include 9 | #include "LogItem.h" 10 | 11 | namespace EndpointLog { 12 | 13 | class IdMgr; 14 | 15 | // Each DjsonLogItem contains a DJSON-formatted item. 16 | // The item includes 17 | // - message length in bytes 18 | // - new line char 19 | // - a string containing the JSON payload 20 | // The JSON payload includes 5 elements: 21 | // - source, a string 22 | // - message id, uint64_t. 23 | // - schema id, uint64_t. 24 | // - schema array. 25 | // - data array. 26 | // 27 | // Example item: 28 | // 110 29 | // ["syslog",53,3,[["timestamp","FT_TIME"],["message","FT_STRING"]],[[1475129808,541868180],"This is a message"]] 30 | // 31 | class DjsonLogItem : public LogItem 32 | { 33 | private: 34 | struct ItemInfo 35 | { 36 | std::string name; 37 | std::string type; 38 | std::string value; 39 | 40 | ItemInfo(std::string n, std::string t, std::string v) : 41 | name(std::move(n)), 42 | type(std::move(t)), 43 | value(std::move(v)) 44 | {} 45 | }; 46 | 47 | struct CompItemInfo 48 | { 49 | bool operator()(const ItemInfo & x, const ItemInfo & y) { return x.name < y.name; } 50 | }; 51 | 52 | public: 53 | DjsonLogItem(std::string source) 54 | :LogItem(), 55 | m_source(std::move(source)) 56 | { 57 | } 58 | 59 | /// Construct a new object. 60 | /// source: source of the DJSON item 61 | /// schemaAndData: include schema id, schema array, data array. 62 | DjsonLogItem(std::string source, std::string schemaAndData) 63 | : LogItem(), 64 | m_source(std::move(source)), 65 | m_schemaAndData(std::move(schemaAndData)) 66 | { 67 | } 68 | 69 | ~DjsonLogItem() {} 70 | 71 | DjsonLogItem(const DjsonLogItem & other) = default; 72 | DjsonLogItem(DjsonLogItem&& other) = default; 73 | 74 | DjsonLogItem& operator=(const DjsonLogItem & other) = default; 75 | DjsonLogItem& operator=(DjsonLogItem&& other) = default; 76 | 77 | // Return full DJSON-formatted string 78 | const char* GetData() override; 79 | 80 | void AddData(std::string name, bool value) 81 | { 82 | m_svlist.emplace_back(std::move(name), "FT_BOOL", value? "true" : "false"); 83 | } 84 | 85 | void AddData(std::string name, int32_t value) 86 | { 87 | m_svlist.emplace_back(std::move(name), "FT_INT32", std::to_string(value)); 88 | } 89 | 90 | void AddData(std::string name, uint32_t value) 91 | { 92 | m_svlist.emplace_back(std::move(name), "FT_INT64", std::to_string(value)); 93 | } 94 | 95 | void AddData(std::string name, int64_t value) 96 | { 97 | m_svlist.emplace_back(std::move(name), "FT_INT64", std::to_string(value)); 98 | } 99 | 100 | void AddData(std::string name, double value) 101 | { 102 | // because std::to_string(double) only prints 6 deciman digits, don't use it 103 | std::ostringstream strm; 104 | strm << value; 105 | m_svlist.emplace_back(std::move(name), "FT_DOUBLE", strm.str()); 106 | } 107 | 108 | void AddData(std::string name, uint64_t seconds, uint32_t nanoseconds) 109 | { 110 | std::ostringstream strm; 111 | strm << "[" << seconds << "," << nanoseconds << "]"; 112 | m_svlist.emplace_back(name, "FT_TIME", strm.str()); 113 | } 114 | 115 | void AddData(std::string name, const char* value) 116 | { 117 | if (!value) { 118 | throw std::invalid_argument("DjsonLogItem::AddData: unexpected NULl for const char* parameter."); 119 | } 120 | AddData(name, std::string(value)); 121 | } 122 | 123 | void AddData(std::string name, std::string value) 124 | { 125 | std::string tmpstr; 126 | tmpstr.reserve(2 + value.size()); 127 | tmpstr.append(1, '"').append(value).append(1, '"'); 128 | m_svlist.emplace_back(std::move(name), "FT_STRING", std::move(tmpstr)); 129 | } 130 | 131 | private: 132 | static IdMgr& GetIdMgr(); 133 | 134 | void ComposeSchemaAndData(); 135 | 136 | void ComposeSchema(std::ostringstream& strm); 137 | std::string GetSchemaCacheKey() const; 138 | std::string ComposeSchemaArray() const; 139 | void ComposeDataValue(std::ostringstream& strm) const; 140 | 141 | void ComposeFullData(); 142 | 143 | private: 144 | std::string m_source; 145 | std::string m_schemaAndData; 146 | std::vector m_svlist; // contain schema and value info 147 | std::string m_djsonData; 148 | }; 149 | 150 | } // namespace 151 | 152 | #endif // __ENDPOINT_DJSONLOGITEM_H__ 153 | -------------------------------------------------------------------------------- /src/outmdsd/EtwLogItem.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef __ENDPOINT_ETWLOGITEM_H__ 3 | #define __ENDPOINT_ETWLOGITEM_H__ 4 | 5 | #include 6 | 7 | #include "DjsonLogItem.h" 8 | 9 | namespace EndpointLog { 10 | 11 | // This class contains ETW data sent to mdsd DJSON socket (e.g. from MDM MetricsExtension) 12 | class EtwLogItem : public DjsonLogItem 13 | { 14 | public: 15 | EtwLogItem( 16 | std::string source, 17 | std::string guid, 18 | int32_t eventId) : DjsonLogItem(std::move(source)) 19 | { 20 | AddData("GUID", std::move(guid)); 21 | AddData("EventId", eventId); 22 | } 23 | }; 24 | 25 | } // namespace 26 | 27 | #endif // __ENDPOINT_ETWLOGITEM_H__ 28 | -------------------------------------------------------------------------------- /src/outmdsd/Exceptions.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #ifndef __ENDPOINT_EXCEPTIONS__H__ 4 | #define __ENDPOINT_EXCEPTIONS__H__ 5 | 6 | #include 7 | #include 8 | 9 | namespace EndpointLog { 10 | 11 | class SocketException : public std::system_error { 12 | public: 13 | SocketException(int errnum, const std::string & msg) : 14 | std::system_error(errnum, std::system_category(), msg) {} 15 | 16 | SocketException(int errnum, const char * msg) : 17 | std::system_error(errnum, std::system_category(), msg) {} 18 | }; 19 | 20 | class ReaderInterruptException {}; 21 | 22 | } 23 | 24 | #endif // __ENDPOINT_EXCEPTIONS__H__ 25 | -------------------------------------------------------------------------------- /src/outmdsd/FileTracer.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | extern "C" { 5 | #include 6 | #include 7 | #include 8 | #include 9 | } 10 | 11 | #include "FileTracer.h" 12 | 13 | namespace EndpointLog { 14 | 15 | 16 | FileTracer::FileTracer( 17 | const std::string& filepath, 18 | bool createIfNotExists 19 | ) : 20 | m_filepath(filepath), 21 | m_fd(-1) 22 | { 23 | if (filepath.empty()) { 24 | throw std::invalid_argument("FileTracer: invalid empty filepath parameter"); 25 | } 26 | 27 | int rtn = 0; 28 | if (createIfNotExists) { 29 | rtn = open(filepath.c_str(), O_WRONLY | O_APPEND | O_CREAT, 0644); 30 | } 31 | else { 32 | rtn = open(filepath.c_str(), O_WRONLY | O_APPEND); 33 | } 34 | 35 | if (-1 == rtn) { 36 | throw std::system_error(errno, std::system_category(), "open " + filepath + " failed"); 37 | } 38 | m_fd = rtn; 39 | } 40 | 41 | void 42 | FileTracer::WriteLog( 43 | const std::string& msg 44 | ) 45 | { 46 | auto bytesleft = msg.size(); 47 | while(bytesleft) { 48 | auto rtn = write(m_fd, msg.c_str()+msg.size()-bytesleft, bytesleft); 49 | if (-1 == rtn) { 50 | if (EINTR != errno) { 51 | throw std::system_error(errno, std::system_category(), 52 | "write() " + m_filepath + " failed"); 53 | } 54 | } 55 | else { 56 | bytesleft -= rtn; 57 | } 58 | } 59 | } 60 | 61 | 62 | } // namespace 63 | -------------------------------------------------------------------------------- /src/outmdsd/FileTracer.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #ifndef __FILETRACER_H__ 4 | #define __FILETRACER_H__ 5 | 6 | #include 7 | #include "ITracer.h" 8 | 9 | namespace EndpointLog { 10 | 11 | class FileTracer : public ITracer { 12 | public: 13 | FileTracer(const std::string& filepath, bool createIfNotExists); 14 | ~FileTracer() = default; 15 | 16 | void WriteLog(const std::string& msg); 17 | 18 | private: 19 | std::string m_filepath; 20 | int m_fd; 21 | }; 22 | 23 | } // namespace 24 | 25 | #endif // __FILETRACER_H__ 26 | -------------------------------------------------------------------------------- /src/outmdsd/ITracer.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #ifndef __ITRACER_H__ 4 | #define __ITRACER_H__ 5 | 6 | #include 7 | 8 | namespace EndpointLog { 9 | 10 | class ITracer { 11 | public: 12 | virtual ~ITracer() = default; 13 | 14 | virtual void WriteLog(const std::string & msg) = 0; 15 | }; 16 | 17 | } // namespace 18 | 19 | #endif // __ITRACER_H__ 20 | -------------------------------------------------------------------------------- /src/outmdsd/IdMgr.cc: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | #include "IdMgr.h" 4 | 5 | namespace EndpointLog { 6 | 7 | bool 8 | IdMgr::GetItem( 9 | const std::string & key, 10 | value_type_t& result 11 | ) 12 | { 13 | if (key.empty()) { 14 | throw std::invalid_argument("GetItem(): invalid empty string for 'key' parameter."); 15 | } 16 | 17 | std::lock_guard lck(m_mutex); 18 | 19 | auto iter = m_cache.find(key); 20 | if (iter == m_cache.end()) { 21 | return false; 22 | } 23 | else { 24 | result = iter->second; 25 | return true; 26 | } 27 | } 28 | 29 | uint64_t 30 | IdMgr::FindOrInsert( 31 | const std::string & key, 32 | const std::string & data 33 | ) 34 | { 35 | if (key.empty()) { 36 | throw std::invalid_argument("FindOrInsert(): invalid empty string for 'key' parameter."); 37 | } 38 | if (data.empty()) { 39 | throw std::invalid_argument("FindOrInsert(): invalid empty string for 'data' parameter."); 40 | } 41 | 42 | std::lock_guard lck(m_mutex); 43 | auto iter = m_cache.find(key); 44 | if (iter == m_cache.end()) { 45 | auto id = static_cast(m_cache.size()+1); 46 | m_cache[key] = std::make_pair(id, data); 47 | return id; 48 | } 49 | else { 50 | if (data != iter->second.second) { 51 | throw std::runtime_error("FindOrInsert(): same key has diff values: expected=" + 52 | data + "; actual=" + iter->second.second); 53 | } 54 | return iter->second.first; 55 | } 56 | } 57 | 58 | void 59 | IdMgr::Insert( 60 | const std::string & key, 61 | const value_type_t& value 62 | ) 63 | { 64 | if (key.empty()) { 65 | throw std::invalid_argument("Insert(): invalid empty string for 'key' parameter."); 66 | } 67 | if (value.second.empty()) { 68 | throw std::invalid_argument("Insert(): invalid empty string for 'value' parameter."); 69 | } 70 | 71 | std::lock_guard lck(m_mutex); 72 | auto iter = m_cache.find(key); 73 | if (iter == m_cache.end()) { 74 | m_cache[key] = value; 75 | } 76 | } 77 | 78 | } // namespace 79 | -------------------------------------------------------------------------------- /src/outmdsd/IdMgr.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef __ENDPOINTLOG_IDMGR_H__ 3 | #define __ENDPOINTLOG_IDMGR_H__ 4 | 5 | #include 6 | #include 7 | #include 8 | 9 | namespace EndpointLog { 10 | 11 | // This class manages a cache for concurrent access. Cache key is 12 | // is std::string, cache value is 'value_type_t', a . 13 | // 14 | class IdMgr { 15 | public: 16 | using value_type_t = std::pair; 17 | 18 | /// Get an item given key. Return the item from 'result'. 19 | /// Return true if the key is found and returned by 'result', 20 | /// Return false if the key is not found. 21 | /// Throw exception if key is a empty string. 22 | bool GetItem(const std::string & key, value_type_t& result); 23 | 24 | /// Find or insert an item with given key. 25 | /// - If the key already exists, return the id part of value item; 26 | /// - If the key doesn't exist, get a unique id and add 27 | /// to the cache. Then return the new id. 28 | /// Throw exception if 'key' or 'data' is empty. 29 | uint64_t FindOrInsert(const std::string & key, const std::string & data); 30 | 31 | /// Insert an item with given key such that cache[key] = value if key doesn't exist. 32 | /// If the key already exists, do nothing. 33 | /// Throw exception if 'key' is empty, or value contains empty string. 34 | void Insert(const std::string & key, const value_type_t& value); 35 | 36 | private: 37 | std::unordered_map m_cache; 38 | std::mutex m_mutex; 39 | }; 40 | 41 | } // namespace 42 | 43 | #endif // __ENDPOINTLOG_IDMGR_H__ 44 | -------------------------------------------------------------------------------- /src/outmdsd/LogItem.cc: -------------------------------------------------------------------------------- 1 | #include "LogItem.h" 2 | 3 | using namespace EndpointLog; 4 | 5 | std::atomic LogItem::s_counter{0}; 6 | -------------------------------------------------------------------------------- /src/outmdsd/LogItem.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #ifndef __ENDPOINT_LOGITEM_H__ 4 | #define __ENDPOINT_LOGITEM_H__ 5 | 6 | #include 7 | #include 8 | #include 9 | 10 | namespace EndpointLog { 11 | 12 | /// This class contains all the data to send to mdsd socket. 13 | /// It has two parts: a tag and raw string data. 14 | /// 15 | /// Because a tag must be unique during a certain amount of time (1-hour) 16 | /// for current mdsd design, use a 64-bit counter to make it unique. 17 | /// 18 | /// This is an abstract class. Its subclass will implement the details for 19 | /// the real data operations. 20 | class LogItem 21 | { 22 | public: 23 | LogItem() : 24 | m_tag(std::to_string(++LogItem::s_counter)), 25 | m_touchTime(std::chrono::steady_clock::now()) 26 | { 27 | } 28 | 29 | virtual ~LogItem() {} 30 | 31 | LogItem(const LogItem & other) = default; 32 | LogItem(LogItem&& other) = default; 33 | 34 | LogItem& operator=(const LogItem & other) = default; 35 | LogItem& operator=(LogItem&& other) = default; 36 | 37 | virtual std::string GetTag() const { return m_tag; } 38 | 39 | virtual const char* GetData() = 0; 40 | 41 | void Touch() { 42 | m_touchTime = std::chrono::steady_clock::now(); 43 | } 44 | 45 | /// Return number of milliseconds passed since the item is last touched. 46 | /// If never touched before, it will count from creation time. 47 | int GetLastTouchMilliSeconds() const 48 | { 49 | auto now = std::chrono::steady_clock::now(); 50 | return (now - m_touchTime) / std::chrono::milliseconds(1); 51 | } 52 | 53 | private: 54 | std::string m_tag; // Tag to the log item. 55 | std::chrono::steady_clock::time_point m_touchTime; // last touch time 56 | 57 | static std::atomic s_counter; // counter of number of logItem created. 58 | }; 59 | 60 | } // namespace EndpointLog 61 | 62 | #endif // __ENDPOINT_LOGITEM_H__ 63 | -------------------------------------------------------------------------------- /src/outmdsd/LogItemPtr.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #ifndef __ENDPOINT_LOGITEMPTR_H__ 4 | #define __ENDPOINT_LOGITEMPTR_H__ 5 | 6 | #include 7 | 8 | namespace EndpointLog { 9 | 10 | class LogItem; 11 | using LogItemPtr = std::shared_ptr; 12 | 13 | } 14 | 15 | #endif // __ENDPOINT_LOGITEMPTR_H__ 16 | -------------------------------------------------------------------------------- /src/outmdsd/SockAddr.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | extern "C" { 5 | #include 6 | } 7 | 8 | #include "SockAddr.h" 9 | #include "TraceMacros.h" 10 | #include "Trace.h" 11 | 12 | using namespace EndpointLog; 13 | 14 | UnixSockAddr::UnixSockAddr( 15 | const std::string & socketfile 16 | ) 17 | { 18 | if (socketfile.empty()) { 19 | throw std::invalid_argument("UnixSockAddr: unexpected empty socketfile parameter."); 20 | } 21 | 22 | // maxLength: maximum permitted length of a path of a Unix domain socket 23 | constexpr auto maxLength = sizeof(addr.sun_path); 24 | if (socketfile.size() > maxLength) { 25 | throw std::invalid_argument("UnixSockAddr: socketfile '" + socketfile + 26 | "' exceeds max allowed length " + std::to_string(maxLength)); 27 | } 28 | 29 | memset(&addr, 0, sizeof(addr)); 30 | addr.sun_family = AF_UNIX; 31 | strncpy(addr.sun_path, socketfile.c_str(), maxLength); 32 | Log(TraceLevel::Info, "Create UNIX socket with '" << socketfile << "'"); 33 | } 34 | 35 | TcpSockAddr::TcpSockAddr( 36 | int port 37 | ) 38 | { 39 | if (port <= 0) { 40 | throw std::invalid_argument("TcpSockAddr: invalid port " + std::to_string(port)); 41 | } 42 | memset(&addr, 0, sizeof(addr)); 43 | addr.sin_family = AF_INET; 44 | addr.sin_port = htons(port); 45 | addr.sin_addr.s_addr = htonl(INADDR_LOOPBACK); 46 | Log(TraceLevel::Info, "Create IP socket with port=" << port); 47 | } 48 | -------------------------------------------------------------------------------- /src/outmdsd/SockAddr.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef __ENDPOINT_SOCKADDR_H__ 3 | #define __ENDPOINT_SOCKADDR_H__ 4 | 5 | extern "C" { 6 | #include 7 | #include 8 | #include 9 | #include 10 | } 11 | #include 12 | 13 | namespace EndpointLog { 14 | 15 | // This class implements APIs for socket() programming. 16 | class SockAddr { 17 | public: 18 | virtual ~SockAddr() {} 19 | 20 | /// Return socket communication domain used in socket(). 21 | virtual int GetDomain() const = 0; 22 | 23 | /// Return socket address used in connect(). 24 | virtual struct sockaddr* GetAddress() const = 0; 25 | 26 | /// Return size of a socket address used in connect(). 27 | virtual int GetAddrLen() const = 0; 28 | }; 29 | 30 | class UnixSockAddr : public SockAddr { 31 | public: 32 | UnixSockAddr(const std::string & socketfile); 33 | 34 | int GetDomain() const { return AF_UNIX; } 35 | 36 | struct sockaddr* GetAddress() const { 37 | return (struct sockaddr*)(&addr); 38 | } 39 | 40 | int GetAddrLen() const { return sizeof(addr); } 41 | 42 | private: 43 | struct sockaddr_un addr; 44 | }; 45 | 46 | class TcpSockAddr : public SockAddr { 47 | public: 48 | TcpSockAddr(int port); 49 | 50 | int GetDomain() const { return AF_INET; } 51 | 52 | struct sockaddr* GetAddress() const { 53 | return (struct sockaddr*)(&addr); 54 | } 55 | 56 | int GetAddrLen() const { return sizeof(addr); } 57 | 58 | private: 59 | struct sockaddr_in addr; 60 | }; 61 | 62 | } // namespace 63 | 64 | #endif // __ENDPOINT_SOCKADDR_H__ -------------------------------------------------------------------------------- /src/outmdsd/SocketClient.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef __ENDPOINT_SOCKETCLIENT_H__ 3 | #define __ENDPOINT_SOCKETCLIENT_H__ 4 | 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include 11 | #include 12 | 13 | namespace EndpointLog { 14 | 15 | class SockAddr; 16 | 17 | /// This is a specialized class to do socket send/read for the following scenario: 18 | /// - The socket server side may lose connection at any time (e.g. server process reboots). 19 | /// - Multiple threads sharing same connection can do send and read concurrently. 20 | /// - All the threads can be stopped immediately if needed when they are calling this class. 21 | /// 22 | /// It handles connection failure before each send(). If the connection 23 | /// is invalid, it will try to create new connection with certain delay 24 | /// between retries. The read() operation will block until timed out if there is 25 | /// connection failure before actual read() is performed. 26 | /// 27 | class SocketClient { 28 | public: 29 | /// 30 | /// Construct a new object using Unix domain socket file. 31 | /// 32 | /// unix domain socket file 33 | /// number of milliseconds to timeout socket 34 | /// connect() retry 35 | SocketClient(const std::string & socketfile, unsigned int connRetryTimeoutMS=60*1000); 36 | 37 | /// 38 | /// Construct a new object using TCP/IP port. 39 | /// 40 | /// port number 41 | /// number of milliseconds to timeout socket 42 | /// connect() retry 43 | SocketClient(int port, unsigned int connRetryTimeoutMS=60*1000); 44 | 45 | ~SocketClient(); 46 | 47 | // not copyable, not movable 48 | SocketClient(const SocketClient& other) = delete; 49 | SocketClient& operator=(const SocketClient& other) = delete; 50 | 51 | SocketClient(SocketClient&& other) = delete; 52 | SocketClient& operator=(SocketClient&& other) = delete; 53 | 54 | /// Notice SocketClient to stop further operation 55 | void Stop(); 56 | 57 | /// 58 | /// Create new socket and connect to it if the sock fd is invalid. 59 | /// If connection fails, socket will be closed. 60 | /// Throw exception if any error. 61 | /// 62 | void Connect(); 63 | 64 | /// Get number of connect() is called. It is for testing only 65 | size_t GetNumReConnect() const { return m_numConnect; } 66 | 67 | /// 68 | /// Read up to bufsize from the socket fd and save data to given buffer. 69 | /// If the socket fd is not valid, wait until timeout. 70 | /// Socket will be closed if any socket error is found, or EOF is reached. 71 | /// Throw exception for any error. 72 | /// Return number of bytes read. or -1 if the SocketClient is already stopped. 73 | /// 74 | /// buffer where data will be saved. The caller must make sure 75 | /// that 'buf' is valid and it has at least bufsize space. 76 | /// max number of bytes to read 77 | /// milliseconds to wait before timeout if socket fd 78 | /// is invalid. 79 | ssize_t Read(void* buf, size_t count, int timeoutMS = 60*1000); 80 | 81 | /// 82 | /// Send a data buffer to the socket. If 'len' is 0, do nothing. 83 | /// The caller must make sure 'buf' is valid. 84 | /// 85 | /// If socket is not connected, Send() will try in a loop to set up connection. 86 | /// The max time trying to set up the connection is connRetryTimeoutMS. 87 | /// 88 | /// if Send() fails at runtime, socket will be closed. 89 | /// Throw exception for any error. 90 | /// 91 | void Send(const void* buf, size_t len); 92 | 93 | /// 94 | /// Send data string to the socket. Send all the data until a terminal NUL is hit. 95 | /// The NUL char is not sent. 96 | /// If string is empty, do nothing. 97 | /// Throw exception for any error. 98 | /// 99 | void Send(const char* data); 100 | 101 | /// close socket fd. 102 | void Close(); 103 | 104 | private: 105 | void SetupSocketConnect(); 106 | 107 | /// Return true if the time from 'startTime' to 'now' is bigger or equal to 108 | /// m_connRetryTimeoutMS. Return false otherwise. 109 | bool IsRetryTimeout(const std::chrono::steady_clock::time_point & startTime) const; 110 | 111 | /// Wait some time using exponential delay policy before next connect() retry. 112 | /// max milliseconds to wait 113 | void WaitBeforeReConnect(int maxWaitMS); 114 | 115 | /// Wait until socket fd is a valid number, or until timed out. 116 | /// max milliseconds to wait 117 | void WaitForSocketToBeReady(int timeoutMS); 118 | 119 | /// send data to the socket server. 120 | /// data buffer. The caller must make sure it is valid and has 121 | /// dataLen size. 122 | /// number of bytes to send 123 | void SendData(const void* data, size_t dataLen); 124 | 125 | /// 126 | /// poll() on the sock fd for I/O. 127 | /// Use a abortPollFd to abort waiting poll() when needed. 128 | /// Throw exception for any error. 129 | /// 130 | /// poll() events value. e.g. POLLIN, POLLOUT, etc 131 | void PollSocket(short pollMode); 132 | 133 | private: 134 | constexpr static int INVALID_SOCKET = -1; 135 | std::shared_ptr m_sockaddr; 136 | unsigned int m_connRetryTimeoutMS = 0; // milliseconds to timeout connect() retry. 137 | 138 | std::atomic m_sockfd{INVALID_SOCKET}; 139 | std::mutex m_fdMutex; // protect sockfd at socket creation/close time. 140 | std::mutex m_sendMutex; // avoid interleaved message in multi Send() at the same time. 141 | 142 | // SocketClient has several APIs that'll block for certain things to be ready. For example, 143 | // poll() for I/O, and Read() for valid socket fd, m_stopClient is used 144 | // to make sure these blocking APIs can be interrupted immediately. 145 | std::atomic m_stopClient{false}; // atomic flag to stop further operations. 146 | 147 | // signal when sock fd is ready to use. This is used when some thread(s) are waiting 148 | // for the sock fd to be ready (e.g. Read() API). 149 | std::condition_variable m_connCV; 150 | 151 | size_t m_numConnect = 0; // number of times to create a new socket. 152 | 153 | std::default_random_engine m_randGen; 154 | std::uniform_real_distribution m_randDist; 155 | }; 156 | 157 | } // namespace 158 | 159 | #endif // __ENDPOINT_SOCKETCLIENT_H__ 160 | -------------------------------------------------------------------------------- /src/outmdsd/SocketLogger.cc: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | #include "ConcurrentMap.h" 4 | #include "SocketLogger.h" 5 | #include "Trace.h" 6 | #include "TraceMacros.h" 7 | #include "SocketClient.h" 8 | #include "DataReader.h" 9 | #include "DataResender.h" 10 | #include "DjsonLogItem.h" 11 | #include "Exceptions.h" 12 | 13 | using namespace EndpointLog; 14 | 15 | SocketLogger::SocketLogger( 16 | const std::string& socketFile, 17 | unsigned int ackTimeoutMS, 18 | unsigned int resendIntervalMS, 19 | unsigned int connRetryTimeoutMS 20 | ): 21 | m_socketClient(std::make_shared(socketFile, connRetryTimeoutMS)), 22 | m_dataCache(ackTimeoutMS? std::make_shared>() : nullptr), 23 | m_sockReader(new DataReader(m_socketClient, m_dataCache)), 24 | m_dataResender(ackTimeoutMS? 25 | new DataResender(m_socketClient, m_dataCache, ackTimeoutMS, resendIntervalMS) : nullptr) 26 | { 27 | } 28 | 29 | SocketLogger::~SocketLogger() 30 | { 31 | try { 32 | ADD_INFO_TRACE; 33 | 34 | m_socketClient->Stop(); 35 | 36 | if (m_dataResender) { 37 | m_dataResender->Stop(); 38 | } 39 | m_sockReader->Stop(); 40 | 41 | for(auto & task : m_workerTasks) { 42 | if (task.valid()) { 43 | task.get(); 44 | } 45 | } 46 | } 47 | catch(const std::exception& ex) 48 | { 49 | Log(TraceLevel::Error, "unexpected exception: " << ex.what()); 50 | } 51 | catch(...) 52 | {} // no exception thrown from destructor 53 | } 54 | 55 | void 56 | SocketLogger::StartWorkers() 57 | { 58 | m_workerTasks.push_back(std::async(std::launch::async, [this] { m_sockReader->Run(); })); 59 | if (m_dataResender) { 60 | m_workerTasks.push_back(std::async(std::launch::async, [this] { m_dataResender->Run(); })); 61 | } 62 | } 63 | 64 | void 65 | SocketLogger::SendData( 66 | LogItemPtr item 67 | ) 68 | { 69 | ADD_DEBUG_TRACE; 70 | 71 | if (!item) { 72 | throw std::invalid_argument("SendData(): unexpected NULL in input parameter."); 73 | } 74 | 75 | std::call_once(m_initOnceFlag, &SocketLogger::StartWorkers, this); 76 | 77 | if (!m_dataCache) { 78 | // If no caching, send it out immediately 79 | m_socketClient->Send(item->GetData()); 80 | m_totalSend++; 81 | } 82 | else { 83 | // Move item to cache first before sending it out. 84 | // This makes sure that the cache has the tag in the thread 85 | // where response is received and handled. 86 | item->Touch(); 87 | auto tag = item->GetTag(); 88 | m_dataCache->Add(tag, std::move(item)); 89 | 90 | auto dataItem = m_dataCache->Get(tag); 91 | assert(dataItem); 92 | try { 93 | m_socketClient->Send(m_dataCache->Get(tag)->GetData()); 94 | m_totalSend++; 95 | } 96 | catch(...) { 97 | // if Send() fails, the caller of SocketLogger is expected to 98 | // retry, so remove it from cache. 99 | auto nErased = m_dataCache->Erase(tag); 100 | Log(TraceLevel::Trace, "Send() failed on msgid='" << tag << "'; Try to erase. nErased=" << nErased); 101 | throw; 102 | } 103 | } 104 | } 105 | 106 | bool 107 | SocketLogger::SendDjson( 108 | const std::string & sourceName, 109 | const std::string & schemaAndData 110 | ) 111 | { 112 | ADD_DEBUG_TRACE; 113 | 114 | if (sourceName.empty()) { 115 | Log(TraceLevel::Error, "SendDjson: unexpected empty source name."); 116 | return false; 117 | } 118 | if (schemaAndData.empty()) { 119 | Log(TraceLevel::Error, "SendDjson: unexpected empty schemaAndData string."); 120 | return false; 121 | } 122 | try { 123 | LogItemPtr item(new DjsonLogItem(sourceName, schemaAndData)); 124 | SendData(std::move(item)); 125 | return true; 126 | } 127 | catch(const SocketException & ex) { 128 | Log(TraceLevel::Error, "SendDjson SocketException: " << ex.what()); 129 | } 130 | catch(const std::exception& ex) { 131 | Log(TraceLevel::Error, "SendDjson exception: " << ex.what()); 132 | } 133 | catch(...) { 134 | Log(TraceLevel::Error, "SendDjson hit unknown exception"); 135 | } 136 | return false; 137 | } 138 | 139 | size_t 140 | SocketLogger::GetNumTagsRead() const 141 | { 142 | return m_sockReader->GetNumTagsRead(); 143 | } 144 | 145 | size_t 146 | SocketLogger::GetTotalResend() const 147 | { 148 | return (m_dataResender? m_dataResender->GetTotalSendTimes() : 0); 149 | } 150 | 151 | size_t 152 | SocketLogger::GetNumItemsInCache() const 153 | { 154 | return (m_dataCache? m_dataCache->Size() : 0); 155 | } 156 | 157 | -------------------------------------------------------------------------------- /src/outmdsd/SocketLogger.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #ifndef __SOCKETLOGGER_H__ 4 | #define __SOCKETLOGGER_H__ 5 | 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include "LogItemPtr.h" 11 | 12 | namespace EndpointLog { 13 | 14 | template class ConcurrentMap; 15 | class SocketClient; 16 | class DataReader; 17 | class DataResender; 18 | 19 | class SocketLogger 20 | { 21 | public: 22 | /// 23 | /// Construct a logger that'll send data to a Unix domain socket. 24 | /// 25 | /// full path to socket file 26 | /// max milliseconds to wait for ack from socket server. 27 | /// After timeout, record will be dropped from cache. If this parameter's value 28 | /// is 0, do no caching. 29 | /// message resend interval in milliseconds 30 | /// from this logger to the targeted endpoint. 31 | /// 32 | SocketLogger( 33 | const std::string& socketFile, 34 | unsigned int ackTimeoutMS, 35 | unsigned int resendIntervalMS, 36 | unsigned int connRetryTimeoutMS = 60*1000 37 | ); 38 | 39 | ~SocketLogger(); 40 | 41 | // not copyable, not movable 42 | SocketLogger(const SocketLogger&) = delete; 43 | SocketLogger& operator=(const SocketLogger &) = delete; 44 | 45 | SocketLogger(SocketLogger&& h) = delete; 46 | SocketLogger& operator=(SocketLogger&& h) = delete; 47 | 48 | /// Send a dynamic json data to mdsd socket. 49 | /// sourceName: source name of the event. 50 | /// schemaAndData: a string containing schema info and actual data values. 51 | /// Return true if success, false if any error. 52 | bool SendDjson(const std::string & sourceName, const std::string & schemaAndData); 53 | 54 | /// Return total number of ack tags processed by reader thread. 55 | size_t GetNumTagsRead() const; 56 | 57 | /// Return total Send() called, including main thread and resender thread. 58 | size_t GetTotalSend() const { 59 | return m_totalSend + GetTotalResend(); 60 | } 61 | 62 | /// Return total number of Send() called by data resender 63 | size_t GetTotalResend() const; 64 | 65 | /// Return number of items in the backup cache, either not resent yet, 66 | /// or not acknowledged yet 67 | size_t GetNumItemsInCache() const; 68 | 69 | private: 70 | void StartWorkers(); 71 | 72 | /// 73 | /// Send new data item to socket. 74 | /// Throw exception for any error. 75 | /// 76 | /// A new logger item. 77 | void SendData(LogItemPtr item); 78 | 79 | private: 80 | std::shared_ptr m_socketClient; 81 | std::shared_ptr> m_dataCache; // to store 82 | 83 | std::vector> m_workerTasks; // store all the worker tasks (reader, resender) 84 | 85 | std::unique_ptr m_sockReader; 86 | std::unique_ptr m_dataResender; 87 | 88 | std::once_flag m_initOnceFlag; 89 | 90 | std::atomic m_totalSend{0}; // a counter. it includes main thread Send() to socket only 91 | }; 92 | 93 | } // namespace 94 | 95 | #endif // __SOCKETLOGGER_H__ 96 | -------------------------------------------------------------------------------- /src/outmdsd/SyslogTracer.cc: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | extern "C" { 4 | #include 5 | } 6 | 7 | #include "SyslogTracer.h" 8 | #include "Trace.h" 9 | 10 | namespace EndpointLog { 11 | 12 | static std::unordered_map& 13 | GetLevelTable() 14 | { 15 | static std::unordered_map levelTable = { 16 | { TraceLevel::Trace, LOG_DEBUG }, 17 | { TraceLevel::Debug, LOG_DEBUG }, 18 | { TraceLevel::Info, LOG_INFO }, 19 | { TraceLevel::Warning, LOG_WARNING }, 20 | { TraceLevel::Error, LOG_ERR }, 21 | { TraceLevel::Fatal, LOG_CRIT } 22 | }; 23 | return levelTable; 24 | } 25 | 26 | static int 27 | TraceLevel2SyslogLevel() 28 | { 29 | auto traceLevel = Trace::GetTraceLevel(); 30 | auto levelTable = GetLevelTable(); 31 | auto iter = levelTable.find(traceLevel); 32 | if (iter != levelTable.end()) { 33 | return iter->second; 34 | } 35 | return LOG_DEBUG; 36 | } 37 | 38 | SyslogTracer::SyslogTracer( 39 | int option, 40 | int facility 41 | ) : 42 | m_logLevel(TraceLevel2SyslogLevel()) 43 | { 44 | openlog(NULL, option, facility); 45 | } 46 | 47 | SyslogTracer::~SyslogTracer() 48 | { 49 | closelog(); 50 | } 51 | 52 | void 53 | SyslogTracer::WriteLog( 54 | const std::string& msg 55 | ) 56 | { 57 | syslog(m_logLevel, "%s", msg.c_str()); 58 | } 59 | 60 | 61 | } // namespace 62 | -------------------------------------------------------------------------------- /src/outmdsd/SyslogTracer.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #ifndef __SYSLOGTRACER_H__ 4 | #define __SYSLOGTRACER_H__ 5 | 6 | #include 7 | #include "ITracer.h" 8 | 9 | namespace EndpointLog { 10 | 11 | class SyslogTracer : public ITracer { 12 | public: 13 | /// This class implements logging using syslog. 14 | /// It uses openlog(NULL, option, facility) for the logging. 15 | SyslogTracer(int option, int facility); 16 | ~SyslogTracer(); 17 | 18 | void WriteLog(const std::string& msg); 19 | 20 | private: 21 | int m_logLevel; 22 | }; 23 | 24 | } // namespace 25 | 26 | #endif // __SYSLOGTRACER_H__ 27 | -------------------------------------------------------------------------------- /src/outmdsd/Trace.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | extern "C" { 6 | #include 7 | } 8 | 9 | #include "Trace.h" 10 | #include "ITracer.h" 11 | #include "Exceptions.h" 12 | 13 | using namespace EndpointLog; 14 | 15 | TraceLevel Trace::s_minLevel = TraceLevel::Info; 16 | ITracer* Trace::s_logger = nullptr; 17 | 18 | static 19 | std::string GetFileBasename( 20 | const std::string & filepath 21 | ) 22 | { 23 | size_t p = filepath.find_last_of('/'); 24 | if (p == std::string::npos) { 25 | return filepath; 26 | } 27 | return filepath.substr(p+1); 28 | } 29 | 30 | void 31 | Trace::SetTracer( 32 | ITracer* tracerObj 33 | ) 34 | { 35 | if (!tracerObj) { 36 | throw std::invalid_argument("SetTracer: TracerObj cannot be NULL"); 37 | } 38 | GetLevelStrTable(); 39 | 40 | if (s_logger) { 41 | delete s_logger; 42 | } 43 | s_logger = tracerObj; 44 | } 45 | 46 | std::string 47 | GetTimeNow() 48 | { 49 | struct timeval tv; 50 | (void) gettimeofday(&tv, 0); 51 | 52 | struct tm zulu; 53 | (void) gmtime_r(&(tv.tv_sec), &zulu); 54 | 55 | char buf[128]; 56 | auto rtn = strftime(buf, sizeof(buf), "%Y-%m-%dT%H:%M:%S", &zulu); 57 | if (0 == rtn) { 58 | throw std::runtime_error("strftime() failed"); 59 | } 60 | auto usec = static_cast(tv.tv_usec); 61 | std::ostringstream strm; 62 | strm << buf << "." << std::setfill('0') << std::setw(6) << usec << "0Z"; 63 | return strm.str(); 64 | } 65 | 66 | 67 | void 68 | Trace::WriteLog( 69 | TraceLevel traceLevel, 70 | const std::string & msg, 71 | const char * filename, 72 | int lineNumber 73 | ) 74 | { 75 | if (traceLevel < s_minLevel) { 76 | return; 77 | } 78 | 79 | try { 80 | std::string levelStr = TraceLevel2Str(traceLevel); 81 | std::string basename = GetFileBasename(filename); 82 | 83 | auto now = GetTimeNow(); 84 | 85 | char buf[1024]; 86 | snprintf( 87 | buf, 88 | sizeof(buf), 89 | "%s: %s %s:%d %s\n", 90 | now.c_str(), 91 | levelStr.c_str(), 92 | basename.c_str(), 93 | lineNumber, 94 | msg.c_str()); 95 | 96 | s_logger->WriteLog(buf); 97 | } 98 | catch(const std::exception & ex) { 99 | std::cout << "Error: Trace::WriteLog() failed: " << ex.what() << std::endl; 100 | } 101 | } 102 | 103 | std::string 104 | Trace::TraceLevel2Str(TraceLevel level) noexcept 105 | { 106 | auto levelTable = GetLevelStrTable(); 107 | auto iter = levelTable.find(level); 108 | if (iter != levelTable.end()) { 109 | return iter->second; 110 | } 111 | return std::string("TraceLevel_" + std::to_string(static_cast(level))); 112 | } 113 | 114 | TraceLevel 115 | Trace::TraceLevelFromStr(const std::string & level) noexcept 116 | { 117 | auto t = GetStr2LevelTable(); 118 | auto iter = t.find(level); 119 | if (iter != t.end()) { 120 | return iter->second; 121 | } 122 | return TraceLevel::Warning; 123 | } 124 | 125 | std::unordered_map& 126 | Trace::GetLevelStrTable() 127 | { 128 | static std::unordered_map levelTable = { 129 | { TraceLevel::Trace, "trace" }, 130 | { TraceLevel::Debug, "debug" }, 131 | { TraceLevel::Info, "info" }, 132 | { TraceLevel::Warning, "warn" }, 133 | { TraceLevel::Error, "error" }, 134 | { TraceLevel::Fatal, "fatal" } 135 | }; 136 | return levelTable; 137 | } 138 | 139 | // Get string to level enum table 140 | std::unordered_map& 141 | Trace::GetStr2LevelTable() 142 | { 143 | static std::unordered_map t = { 144 | { "trace", TraceLevel::Trace }, 145 | { "debug", TraceLevel::Debug }, 146 | { "info", TraceLevel::Info }, 147 | { "warn", TraceLevel::Warning }, 148 | { "error", TraceLevel::Error }, 149 | { "fatal", TraceLevel::Fatal } 150 | }; 151 | return t; 152 | } 153 | -------------------------------------------------------------------------------- /src/outmdsd/Trace.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | 3 | #ifndef __ENDPOINT_TRACE_H__ 4 | #define __ENDPOINT_TRACE_H__ 5 | 6 | #include 7 | #include 8 | #include 9 | 10 | namespace EndpointLog { 11 | class ITracer; 12 | 13 | enum class TraceLevel { 14 | Trace, 15 | Debug, 16 | Info, 17 | Warning, 18 | Error, 19 | Fatal 20 | }; 21 | 22 | struct EnumClassHash { 23 | template 24 | size_t operator()(T t) const { 25 | return static_cast(t); 26 | } 27 | }; 28 | 29 | class Trace { 30 | public: 31 | Trace( 32 | TraceLevel level, 33 | const std::string & func, 34 | const char* srcFilename, 35 | int lineNumber) : 36 | m_level(level), 37 | m_func(func), 38 | m_srcFilename(srcFilename), 39 | m_lineNumber(lineNumber) 40 | { 41 | if (level >= s_minLevel) { 42 | std::ostringstream ss; 43 | ss << "Entering " << m_func; 44 | WriteLog(level, ss.str(), srcFilename, lineNumber); 45 | } 46 | } 47 | 48 | Trace( 49 | TraceLevel level, 50 | std::string && func, 51 | const char* srcFilename, 52 | int lineNumber) : 53 | m_level(level), 54 | m_func(std::move(func)), 55 | m_srcFilename(srcFilename), 56 | m_lineNumber(lineNumber) 57 | { 58 | if (level >= s_minLevel) { 59 | std::ostringstream ss; 60 | ss << "Entering " << m_func; 61 | WriteLog(level, ss.str(), srcFilename, lineNumber); 62 | } 63 | } 64 | 65 | ~Trace() 66 | { 67 | if (m_level >= s_minLevel) { 68 | std::ostringstream ss; 69 | ss << "Leaving " << m_func; 70 | WriteLog(m_level, ss.str(), m_srcFilename, m_lineNumber); 71 | } 72 | } 73 | 74 | /// Set tracer object which implements the real logging. 75 | /// NOTE: This must be called before doing any tracing. 76 | /// Throw exception if any error. 77 | static void SetTracer(ITracer* tracerObj); 78 | 79 | static void SetTraceLevel(TraceLevel level) 80 | { 81 | s_minLevel = level; 82 | } 83 | 84 | static void SetTraceLevel(const std::string & level) 85 | { 86 | s_minLevel = TraceLevelFromStr(level); 87 | } 88 | 89 | static TraceLevel GetTraceLevel() { 90 | return s_minLevel; 91 | } 92 | 93 | static void WriteLog(TraceLevel level, const std::string & msg, 94 | const char* filename, int lineNumber); 95 | 96 | private: 97 | TraceLevel m_level; 98 | std::string m_func; 99 | const char* m_srcFilename; 100 | int m_lineNumber; 101 | 102 | static TraceLevel s_minLevel; 103 | // Use a static pointer to avoid static object deinitialization order issue 104 | static ITracer* s_logger; 105 | 106 | static std::unordered_map& GetLevelStrTable(); 107 | static std::unordered_map& GetStr2LevelTable(); 108 | 109 | static std::string TraceLevel2Str(TraceLevel level) noexcept; 110 | static TraceLevel TraceLevelFromStr(const std::string & level) noexcept; 111 | }; 112 | 113 | } 114 | 115 | #endif // __ENDPOINT_TRACE_H__ 116 | -------------------------------------------------------------------------------- /src/outmdsd/TraceMacros.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef __TRACEMACROS_H__ 3 | #define __TRACEMACROS_H__ 4 | 5 | #define ADD_INFO_TRACE \ 6 | Trace _trace(TraceLevel::Info, __func__, __FILE__, __LINE__) 7 | /**/ 8 | #define ADD_DEBUG_TRACE \ 9 | Trace _trace(TraceLevel::Debug, __func__, __FILE__, __LINE__) 10 | /**/ 11 | #define ADD_TRACE_TRACE \ 12 | Trace _trace(TraceLevel::Trace, __func__, __FILE__, __LINE__) 13 | /**/ 14 | #define Log(level, message) \ 15 | if (level >= Trace::GetTraceLevel()) { \ 16 | std::ostringstream _ss; \ 17 | _ss << message; \ 18 | Trace::WriteLog(level, _ss.str(), __FILE__, __LINE__); \ 19 | } 20 | /**/ 21 | 22 | #endif 23 | -------------------------------------------------------------------------------- /src/outmdsdrb/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | if(NOT "${FLUENTD_TARGET}" STREQUAL "system") 2 | # To force using memcpy@GLIBC_2.2.5 (for old Linux distro versions) 3 | set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl,--wrap=memcpy") 4 | # To avoid GLIBCXX_3.4.(high) version issue, statically link to libstdc++ & libgcc 5 | set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -static-libgcc -static-libstdc++") 6 | endif() 7 | 8 | set(CMAKE_CXX_STANDARD 11) 9 | set(CMAKE_CXX_STANDARD_REQUIRED ON) 10 | set(LIB_NAME outmdsdrb) 11 | 12 | find_package(SWIG REQUIRED) 13 | include(${SWIG_USE_FILE}) 14 | 15 | include_directories( 16 | ${CMAKE_SOURCE_DIR}/outmdsd 17 | ${CMAKE_SOURCE_DIR}/outmdsdrb 18 | ${RUBY_INC_PATH} 19 | ${RUBY_INC_PATH}/${CMAKE_SYSTEM_PROCESSOR}-linux 20 | ) 21 | 22 | set(SOURCES 23 | outmdsd_log.cxx 24 | ) 25 | if(NOT "${FLUENTD_TARGET}" STREQUAL "system") 26 | list(APPEND SOURCES 27 | wrap_memcpy.c 28 | ) 29 | endif() 30 | 31 | set_property(SOURCE outmdsdrb.i PROPERTY CPLUSPLUS ON) 32 | swig_add_library(${LIB_NAME} 33 | TYPE MODULE 34 | LANGUAGE ruby 35 | SOURCES outmdsdrb.i ${SOURCES} 36 | ) 37 | SET_TARGET_PROPERTIES(${LIB_NAME} PROPERTIES PREFIX "Lib") 38 | target_link_libraries(${LIB_NAME} 39 | outmdsd 40 | # Required not to use clock_gettime@GLIBC_2.17 41 | -Wl,--no-as-needed 42 | rt 43 | ) 44 | 45 | install(TARGETS ${LIB_NAME} 46 | LIBRARY DESTINATION ${CMAKE_BINARY_DIR}/release/lib 47 | ) 48 | -------------------------------------------------------------------------------- /src/outmdsdrb/outmdsd_log.cxx: -------------------------------------------------------------------------------- 1 | #include "outmdsd_log.h" 2 | #include "Trace.h" 3 | #include "FileTracer.h" 4 | 5 | void 6 | InitLogger( 7 | const std::string& logFilePath, 8 | bool createIfNotExist 9 | ) 10 | { 11 | EndpointLog::Trace::SetTracer(new EndpointLog::FileTracer(logFilePath, createIfNotExist)); 12 | } 13 | 14 | void 15 | SetLogLevel( 16 | const std::string & level 17 | ) 18 | { 19 | EndpointLog::Trace::SetTraceLevel(level); 20 | } 21 | 22 | -------------------------------------------------------------------------------- /src/outmdsdrb/outmdsd_log.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef __OUTMDSD_LOG_H__ 3 | #define __OUTMDSD_LOG_H__ 4 | 5 | #include 6 | 7 | void InitLogger(const std::string& logFilePath, bool createIfNotExist); 8 | 9 | void SetLogLevel(const std::string & level); 10 | 11 | 12 | #endif // __OUTMDSD_LOG_H__ 13 | -------------------------------------------------------------------------------- /src/outmdsdrb/outmdsdrb.i: -------------------------------------------------------------------------------- 1 | %module Liboutmdsdrb 2 | 3 | %{ 4 | #include "../outmdsd/SocketLogger.h" 5 | #include "outmdsd_log.h" 6 | %} 7 | %include "stdint.i" 8 | %include "std_string.i" 9 | %include "../outmdsd/SocketLogger.h" 10 | %include "outmdsd_log.h" 11 | -------------------------------------------------------------------------------- /src/outmdsdrb/wrap_memcpy.c: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | void *__memcpy_glibc_2_2_5(void *, const void *, size_t); 4 | 5 | asm(".symver __memcpy_glibc_2_2_5, memcpy@GLIBC_2.2.5"); 6 | void *__wrap_memcpy(void *dest, const void *src, size_t n) 7 | { 8 | return __memcpy_glibc_2_2_5(dest, src, n); 9 | } 10 | -------------------------------------------------------------------------------- /src/test/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | file(COPY 2 | runtests.sh 3 | DESTINATION ${CMAKE_BINARY_DIR}/release/tests) 4 | 5 | add_subdirectory(outmdsd) 6 | add_subdirectory(outmdsdrb) 7 | -------------------------------------------------------------------------------- /src/test/outmdsd/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -pthread -fno-strict-aliasing") 2 | set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -pthread") 3 | 4 | set(Boost_USE_STATIC_LIBS ON) # only find static libs 5 | set(Boost_USE_DEBUG_LIBS OFF) # ignore debug libs and 6 | set(Boost_USE_RELEASE_LIBS ON) # only find release libs 7 | # Find Boost 1.56 or higher 8 | find_package(Boost 1.56 REQUIRED COMPONENTS system unit_test_framework) 9 | 10 | include_directories( 11 | ${CMAKE_SOURCE_DIR}/outmdsd 12 | ) 13 | 14 | add_executable( 15 | ut_outmdsd 16 | MockServer.cc 17 | testbuflog.cc 18 | testlogger.cc 19 | testlogitem.cc 20 | testmap.cc 21 | testqueue.cc 22 | testreader.cc 23 | testresender.cc 24 | testsender.cc 25 | testsocket.cc 26 | testtrace.cc 27 | testutil.cc 28 | utmain.cc 29 | ) 30 | 31 | target_link_libraries( 32 | ut_outmdsd 33 | outmdsd 34 | Boost::unit_test_framework 35 | Boost::system 36 | ) 37 | 38 | add_executable( 39 | mockserver 40 | MockServer.cc 41 | mockserver_main.cc 42 | testutil.cc 43 | ) 44 | 45 | install(TARGETS 46 | ut_outmdsd 47 | mockserver 48 | RUNTIME DESTINATION ${CMAKE_BINARY_DIR}/release/tests 49 | ) 50 | -------------------------------------------------------------------------------- /src/test/outmdsd/CounterCV.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef __COUNTERCV_H__ 3 | #define __COUNTERCV_H__ 4 | 5 | #include 6 | #include 7 | #include 8 | #include 9 | 10 | 11 | // This class implements a synchronization class that can wait for 12 | // a counter reachs 0. It is similar to semaphore but it is more 13 | // specific. 14 | class CounterCV 15 | { 16 | public: 17 | // constructor 18 | CounterCV(uint32_t count) : m_count(count) {} 19 | 20 | ~CounterCV() = default; 21 | 22 | // decrease counter by 1 and notify one 23 | void notify_one() 24 | { 25 | std::lock_guard lck(m_mutex); 26 | if (m_count > 0) { 27 | m_count--; 28 | m_cv.notify_one(); 29 | } 30 | } 31 | 32 | // decrease counter by 1 and notify all 33 | void notify_all() 34 | { 35 | std::lock_guard lck(m_mutex); 36 | if (m_count > 0) { 37 | m_count--; 38 | m_cv.notify_all(); 39 | } 40 | } 41 | 42 | // wait for counter to be 0 or timeout. 43 | // Return true if counter is 0 at exit, return false otherwise. 44 | bool wait_for(uint32_t timeoutMS) 45 | { 46 | std::unique_lock lck(m_mutex); 47 | return m_cv.wait_for(lck, std::chrono::milliseconds(timeoutMS), [this] () 48 | { 49 | return (0 == m_count); 50 | }); 51 | } 52 | 53 | void wait() 54 | { 55 | std::unique_lock lck(m_mutex); 56 | m_cv.wait(lck, [this] () { return (0 == m_count); }); 57 | } 58 | 59 | uint32_t GetId() const 60 | { 61 | return m_count; 62 | } 63 | 64 | private: 65 | std::mutex m_mutex; 66 | std::condition_variable m_cv; 67 | uint32_t m_count; 68 | }; 69 | 70 | // This wrapper class will decrease counter and notify all waiting CV 71 | // in its destructor. It can be used as RAII to make sure CV is notified. 72 | class CounterCVWrap 73 | { 74 | public: 75 | CounterCVWrap(const std::shared_ptr& cv) : m_cv(cv) {} 76 | 77 | ~CounterCVWrap() 78 | { 79 | m_cv->notify_all(); 80 | } 81 | 82 | uint32_t GetId() const 83 | { 84 | return m_cv->GetId(); 85 | } 86 | 87 | private: 88 | std::shared_ptr m_cv; 89 | }; 90 | 91 | 92 | #endif // __COUNTERCV_H__ -------------------------------------------------------------------------------- /src/test/outmdsd/MockServer.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef __MOCKSERVER_H__ 3 | #define __MOCKSERVER_H__ 4 | 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | 11 | namespace TestUtil 12 | { 13 | 14 | /// This is a simple socket server for testing. It simulates design in mdsd to 15 | /// listen for connection, set up connection, read data, and sent back TAG\n in 16 | /// acknowldge. 17 | /// 18 | class MockServer 19 | { 20 | public: 21 | MockServer(const std::string & socketFile, bool parseReadData = true) : 22 | m_socketFile(socketFile), 23 | m_parseReadData(parseReadData) 24 | {} 25 | 26 | ~MockServer(); 27 | 28 | void Init(); 29 | 30 | // start and run read loop. For each read(), write back a response string. 31 | // Return total bytes read. 32 | void Run(); 33 | 34 | // Wait until all tests are done or timeout. 35 | // return true if success, false if timeout. 36 | bool WaitForTestsDone(uint32_t timeoutMS); 37 | 38 | // notify MockServer to stop 39 | void Stop(); 40 | 41 | // Close connection and sockets. Wait for N milliseconds. Then 42 | // create new socket and connections. 43 | // This is to mock socket or connection failures. 44 | void DisconnectAndRun(uint32_t disconnectTimeMS); 45 | 46 | size_t GetTotalBytesRead() const { return m_totalBytesRead; } 47 | 48 | size_t GetTotalTags() const { return m_totalTags; } 49 | 50 | std::unordered_set GetUniqDataRead() const { return m_dataSet; } 51 | 52 | private: 53 | // poll socket server for new socket connection. 54 | // Return true if poll() returns successfully, return false if any error. 55 | bool PollConnection() const; 56 | void ResetStopFlags(); 57 | void CloseServer(); 58 | void CloseClients(); 59 | void ShutdownClients(); 60 | 61 | void RunEchoTask(int connfd); 62 | void StartReadLoop(int connfd); 63 | void Log(const std::string & msg) const; 64 | void WriteBack(int fd, const std::string & msg); 65 | 66 | // Process the data read from socket. 67 | // Return true if end of test is received, return false otherwise. 68 | bool ProcessReadData(int connfd, const std::string & bufstr, 69 | std::string & lastStrForMsgId, std::string & lastStrForData); 70 | 71 | void GetMsgIdInfo(int connfd, const std::string & bufstr, std::string & lastStrForMsgId); 72 | 73 | // Process the data read from socket and get data value info. 74 | // Return true if end of test is received, return false otherwise. 75 | bool GetMsgDataInfo(const std::string & bufstr, std::string & lastStrForData); 76 | 77 | std::string ParseMsgIds(const std::string & msg, std::string& leftover); 78 | std::string ProcessMsgId(const std::string & msgid); 79 | 80 | void ParseData(const std::string & msg, std::string & leftover); 81 | bool GetMsgData(const std::string & msg, size_t & lastPos); 82 | 83 | bool WaitForRunLoopToFinish(uint32_t timeoutMS); 84 | 85 | // When MockServer receives end-of-test msg, it marks the test is done. 86 | // Return true if done, return false otherwise. 87 | bool IsTestDone(const std::string & str); 88 | void MarkTestDone(); 89 | 90 | private: 91 | std::string m_socketFile; 92 | bool m_parseReadData = true; // if true, parse details from read data; if false, don't parse. 93 | 94 | std::atomic m_stopFlag{false}; // a flag to tell server to stop 95 | std::atomic m_stopAccept{false}; // stop accept any new connection 96 | 97 | std::atomic m_listenfd{-1}; 98 | 99 | std::atomic m_totalBytesRead{0}; // total nbytes read by the server 100 | 101 | // a set for all unique data read in server. This only contains the 'data' part, 102 | // which is everything excluding the msgid. example 103 | // ["testSource",,ABC]' 104 | // Because of possible msg resent from client, this is used to remove duplicates. 105 | std::unordered_set m_dataSet; 106 | 107 | // total number of tags read by the server. can have duplicates. 108 | std::atomic m_totalTags{0}; 109 | 110 | std::unordered_set m_connfdSet; // To save all connect FDs. 111 | std::mutex m_connMutex; // to lock m_connfdSet 112 | 113 | // synchronize to finish all the tests 114 | std::mutex m_finishMutex; 115 | std::condition_variable m_finishCV; 116 | 117 | // synchronize after Run() is done for failure testing 118 | std::atomic m_runFinished{false}; 119 | std::mutex m_runMutex; 120 | std::condition_variable m_runCV; 121 | }; 122 | 123 | } // namespace 124 | 125 | #endif // __MOCKSERVER_H__ 126 | -------------------------------------------------------------------------------- /src/test/outmdsd/mockserver_main.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | extern "C" { 5 | #include 6 | } 7 | 8 | #include "MockServer.h" 9 | 10 | struct CmdArgs { 11 | uint32_t timeBeforeDisconnect; // milliseconds 12 | uint32_t timeToDisconnect; // milliseconds 13 | std::string socketFile; 14 | }; 15 | 16 | void Usage(const std::string & progname) 17 | { 18 | std::cout << "Usage:" << std::endl; 19 | std::cout << std::string(" ") + progname + " [options]" << std::endl; 20 | std::cout << " -b : Wait for milliseconds before disconnect socket." << std::endl; 21 | std::cout << " -d : Disconnect socket after milliseconds." << std::endl; 22 | std::cout << " -u : Listen to a Unix socket file. Create it if not exists." << std::endl; 23 | } 24 | 25 | CmdArgs 26 | ParseCmdLine(int argc, char** argv) 27 | { 28 | if (argc <= 1) { 29 | Usage(argv[0]); 30 | exit(1); 31 | } 32 | 33 | CmdArgs cmdargs; 34 | int opt = 0; 35 | while((opt = getopt(argc, argv, "b:d:u:")) != -1) { 36 | switch(opt) { 37 | case 'b': 38 | cmdargs.timeBeforeDisconnect = atoi(optarg); 39 | break; 40 | case 'd': 41 | cmdargs.timeToDisconnect = atoi(optarg); 42 | break; 43 | case 'u': 44 | cmdargs.socketFile = optarg; 45 | break; 46 | default: 47 | std::cout << "Error: unexpected cmd option: " << opt << std::endl; 48 | Usage(argv[0]); 49 | exit(1); 50 | } 51 | } 52 | return cmdargs; 53 | } 54 | 55 | bool 56 | RunMockServer( 57 | const CmdArgs& cmdargs 58 | ) 59 | { 60 | try { 61 | TestUtil::MockServer mockServer(cmdargs.socketFile); 62 | 63 | auto restartTask = std::async(std::launch::async, [&mockServer, cmdargs] () { 64 | usleep(cmdargs.timeBeforeDisconnect*1000); 65 | mockServer.DisconnectAndRun(cmdargs.timeToDisconnect); 66 | }); 67 | 68 | mockServer.Init(); 69 | mockServer.Run(); 70 | restartTask.wait(); 71 | 72 | return true; 73 | } 74 | catch(const std::exception & ex) { 75 | std::cout << "Error: RunMockServer exception: " << ex.what() << std::endl; 76 | } 77 | return false; 78 | } 79 | 80 | int main(int argc, char** argv) 81 | { 82 | auto cmdargs = ParseCmdLine(argc, argv); 83 | 84 | if (!RunMockServer(cmdargs)) { 85 | return 1; 86 | } 87 | 88 | return 0; 89 | } 90 | -------------------------------------------------------------------------------- /src/test/outmdsd/testbuflog.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include "BufferedLogger.h" 3 | #include "DjsonLogItem.h" 4 | #include "testutil.h" 5 | #include "MockServer.h" 6 | 7 | using namespace EndpointLog; 8 | 9 | BOOST_AUTO_TEST_SUITE(testbuflog) 10 | 11 | BOOST_AUTO_TEST_CASE(Test_BufferedLogger_Cstor_cache) 12 | { 13 | try { 14 | BufferedLogger b("/tmp/nosuchfile", 1, 1, 1, 1); 15 | } 16 | catch(const std::exception & ex) { 17 | BOOST_FAIL("Test exception " << ex.what()); 18 | } 19 | } 20 | 21 | BOOST_AUTO_TEST_CASE(Test_BufferedLogger_Cstor_nocache) 22 | { 23 | try { 24 | BufferedLogger b("/tmp/nosuchfile", 0, 1, 1, 1); 25 | } 26 | catch(const std::exception & ex) { 27 | BOOST_FAIL("Test exception " << ex.what()); 28 | } 29 | } 30 | 31 | BOOST_AUTO_TEST_CASE(Test_BufferedLogger_AddData_Null) 32 | { 33 | try { 34 | BufferedLogger b("/tmp/nosuchfile", 1, 1, 1, 1); 35 | BOOST_CHECK_THROW(b.AddData(nullptr), std::invalid_argument); 36 | } 37 | catch(const std::exception & ex) { 38 | BOOST_FAIL("Test exception " << ex.what()); 39 | } 40 | } 41 | 42 | // Validate BufferedLogger when socket server is down 43 | static void 44 | TestWhenSockServerIsDown( 45 | size_t nitems 46 | ) 47 | { 48 | int connRetryTimeoutMS = 1; 49 | BufferedLogger b("/tmp/nosuchfile", 100000, 100, connRetryTimeoutMS, nitems*10); 50 | 51 | for (size_t i = 0; i < nitems; i++) { 52 | LogItemPtr item(new DjsonLogItem("testsource", "testvalue-" + std::to_string(i+1))); 53 | b.AddData(item); 54 | } 55 | 56 | // wait timeout depends on nitems and connection retry, plus some runtime 57 | BOOST_CHECK(b.WaitUntilAllSend((connRetryTimeoutMS+10)*nitems+100)); 58 | 59 | BOOST_CHECK_EQUAL(0, b.GetNumTagsRead()); 60 | BOOST_CHECK_EQUAL(0, b.GetTotalSendSuccess()); 61 | BOOST_CHECK_EQUAL(0, b.GetTotalResend()); 62 | 63 | BOOST_CHECK_EQUAL(nitems, b.GetTotalSend()); 64 | BOOST_CHECK_EQUAL(nitems, b.GetNumItemsInCache()); 65 | } 66 | 67 | BOOST_AUTO_TEST_CASE(Test_BufferedLogger_ServerFailure_1) 68 | { 69 | try { 70 | TestWhenSockServerIsDown(1); 71 | } 72 | catch(const std::exception & ex) { 73 | BOOST_FAIL("Test exception " << ex.what()); 74 | } 75 | } 76 | 77 | BOOST_AUTO_TEST_CASE(Test_BufferedLogger_ServerFailure_100) 78 | { 79 | try { 80 | TestWhenSockServerIsDown(100); 81 | } 82 | catch(const std::exception & ex) { 83 | BOOST_FAIL("Test exception " << ex.what()); 84 | } 85 | } 86 | 87 | // Wait until client has nothing in cache to resend. 88 | // Return true if success, false if timeout. 89 | bool 90 | WaitForClientCacheEmpty( 91 | BufferedLogger& bLogger, 92 | int timeoutMS 93 | ) 94 | { 95 | for (int i = 0; i < timeoutMS; i++) { 96 | if (0 == bLogger.GetNumItemsInCache()) { 97 | return true; 98 | } 99 | usleep(1000); 100 | } 101 | return false; 102 | } 103 | 104 | // Validate that server receives total nitems. 105 | // And each msg is formated like CreateMsg(i) 106 | static void 107 | ValidateServerResults( 108 | const std::unordered_set& serverDataSet, 109 | size_t nitems 110 | ) 111 | { 112 | BOOST_CHECK_EQUAL(1, serverDataSet.count(TestUtil::EndOfTest())); 113 | 114 | BOOST_CHECK_EQUAL(nitems+1, serverDataSet.size()); // end of test is an extra 115 | if (nitems+1 == serverDataSet.size()) { 116 | for (size_t i = 0; i < nitems; i++) { 117 | auto item = TestUtil::CreateMsg(i); 118 | BOOST_CHECK_MESSAGE(1 == serverDataSet.count(item), "Not found '" << item << "'"); 119 | } 120 | } 121 | } 122 | 123 | static void 124 | RunE2ETest(size_t nitems) 125 | { 126 | const std::string sockfile = TestUtil::GetCurrDir() + "/buflog-e2e"; 127 | auto mockServer = std::make_shared(sockfile, true); 128 | mockServer->Init(); 129 | 130 | auto serverTask = std::async(std::launch::async, [mockServer]() { mockServer->Run(); }); 131 | 132 | BufferedLogger bLogger(sockfile, 1000000, 100, 100, nitems*2); 133 | 134 | size_t totalSend = 0; 135 | for (size_t i = 0; i < nitems; i++) { 136 | auto testdata = TestUtil::CreateMsg(i); 137 | LogItemPtr item(new DjsonLogItem("testsource", testdata)); 138 | bLogger.AddData(item); 139 | totalSend += testdata.size(); 140 | } 141 | 142 | LogItemPtr eot(new DjsonLogItem("testsource", TestUtil::EndOfTest())); 143 | bLogger.AddData(eot); 144 | totalSend += TestUtil::EndOfTest().size(); 145 | 146 | BOOST_CHECK(bLogger.WaitUntilAllSend(1000)); 147 | BOOST_CHECK(mockServer->WaitForTestsDone(1000)); 148 | BOOST_CHECK(WaitForClientCacheEmpty(bLogger, 1000)); 149 | 150 | mockServer->Stop(); 151 | serverTask.get(); 152 | 153 | ValidateServerResults(mockServer->GetUniqDataRead(), nitems); 154 | 155 | BOOST_CHECK_LT(0, mockServer->GetTotalTags()); 156 | 157 | BOOST_CHECK_GT(mockServer->GetTotalBytesRead(), totalSend); 158 | 159 | BOOST_CHECK_LE(nitems+1, bLogger.GetNumTagsRead()); 160 | BOOST_CHECK_EQUAL(nitems+1, bLogger.GetTotalSendSuccess()); 161 | 162 | BOOST_CHECK_LE(nitems+1, bLogger.GetTotalSend()); 163 | BOOST_CHECK_EQUAL(0, bLogger.GetNumItemsInCache()); 164 | } 165 | 166 | BOOST_AUTO_TEST_CASE(Test_BufferedLogger_E2E_1) 167 | { 168 | try { 169 | RunE2ETest(1); 170 | } 171 | catch(const std::exception & ex) { 172 | BOOST_FAIL("Test exception " << ex.what()); 173 | } 174 | } 175 | 176 | BOOST_AUTO_TEST_CASE(Test_BufferedLogger_E2E_1000) 177 | { 178 | try { 179 | RunE2ETest(1000); 180 | } 181 | catch(const std::exception & ex) { 182 | BOOST_FAIL("Test exception " << ex.what()); 183 | } 184 | } 185 | 186 | BOOST_AUTO_TEST_SUITE_END() 187 | -------------------------------------------------------------------------------- /src/test/outmdsd/testlogger.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | #include "MockServer.h" 5 | #include "SocketLogger.h" 6 | #include "SocketClient.h" 7 | #include "DataReader.h" 8 | #include "testutil.h" 9 | 10 | using namespace EndpointLog; 11 | 12 | BOOST_AUTO_TEST_SUITE(testeplog) 13 | 14 | // Test that SocketLogger constructor and destructor 15 | // work properly 16 | BOOST_AUTO_TEST_CASE(Test_SocketLogger_Constructor) 17 | { 18 | try { 19 | SocketLogger eplog("/tmp/unknownfile", 100, 1000); 20 | } 21 | catch(const std::exception & ex) { 22 | BOOST_FAIL("Test exception " << ex.what()); 23 | } 24 | } 25 | 26 | BOOST_AUTO_TEST_CASE(Test_SocketLogger_Error) 27 | { 28 | try { 29 | SocketLogger eplog("/tmp/unknownfile", 100, 1000, 1); 30 | 31 | for (int i = 0; i < 5; i++) { 32 | bool sendOK = eplog.SendDjson("testSource", "testSchemaAndData-" + std::to_string(i+1)); 33 | BOOST_CHECK(!sendOK); 34 | } 35 | } 36 | catch(const std::exception & ex) { 37 | BOOST_FAIL("Test exception " << ex.what()); 38 | } 39 | } 40 | 41 | // Return number of bytes sent. 42 | // If SendDjson() fails, return 0. 43 | // Save the index of each failed send to 'failedMsgList' for future resend. 44 | static size_t 45 | SendOnce( 46 | SocketLogger& eplog, 47 | int msgIndex, 48 | int sendDelayMS, 49 | std::vector& failedMsgList 50 | ) 51 | { 52 | auto data = TestUtil::CreateMsg(msgIndex); 53 | 54 | if (!eplog.SendDjson("testSource", data)) { 55 | failedMsgList.push_back(msgIndex); 56 | return 0; 57 | } 58 | 59 | usleep(sendDelayMS * 1000); // add some delay for test purpose 60 | return data.size(); 61 | } 62 | 63 | static size_t 64 | SendEndOfTestToServer( 65 | SocketLogger& eplog 66 | ) 67 | { 68 | bool sendOK = eplog.SendDjson("testSource", TestUtil::EndOfTest()); 69 | if (!sendOK) { 70 | return 0; 71 | } 72 | return TestUtil::EndOfTest().size(); 73 | } 74 | 75 | // Wait until client has nothing in cache to resend. 76 | // Return true if success, false if timeout. 77 | bool 78 | WaitForClientCacheEmpty( 79 | SocketLogger& eplog, 80 | int timeoutMS 81 | ) 82 | { 83 | for (int i = 0; i < timeoutMS; i++) { 84 | if (0 == eplog.GetNumItemsInCache()) { 85 | return true; 86 | } 87 | usleep(1000); 88 | } 89 | return false; 90 | } 91 | 92 | // Validate that server receives total nmsgs. 93 | // And each msg is formated like CreateMsg(i) 94 | static void 95 | ValidateServerResults( 96 | const std::unordered_set& serverDataSet, 97 | size_t nmsgs 98 | ) 99 | { 100 | BOOST_CHECK_EQUAL(1, serverDataSet.count(TestUtil::EndOfTest())); 101 | 102 | BOOST_CHECK_EQUAL(nmsgs+1, serverDataSet.size()); // end of test is an extra 103 | if (nmsgs+1 == serverDataSet.size()) { 104 | for (size_t i = 0; i < nmsgs; i++) { 105 | auto item = TestUtil::CreateMsg(i); 106 | BOOST_CHECK_MESSAGE(1 == serverDataSet.count(item), "Not found '" << item << "'"); 107 | } 108 | } 109 | } 110 | 111 | // This is to test that the logger can send and receive data reliably. 112 | // If the socket server has failures, data should be resent properly. 113 | // 114 | // sendDelayMS: milliseconds to delay between each Send(). 115 | // mockServerRestart: if true, restart MockServer in the middle to simulate mdsd failures. 116 | static size_t 117 | SendDataToServer( 118 | const std::shared_ptr& mockServer, 119 | const std::string & sockfile, 120 | int nmsgs, 121 | int sendDelayMS, 122 | bool mockServerRestart 123 | ) 124 | { 125 | BOOST_TEST_MESSAGE("Start to SendDataToServer: nmsgs=" << nmsgs << "; RestartServer=" << mockServerRestart); 126 | 127 | size_t totalSend = 0; 128 | int testRuntimeMS = 400; 129 | 130 | // Make ackTimeout long so that it never timeout. 131 | // Make resentTimeout short so that it will resend when failed 132 | SocketLogger eplog(sockfile, testRuntimeMS*10, testRuntimeMS/4, testRuntimeMS/4); 133 | 134 | auto batchSize = nmsgs / 3; 135 | std::vector failedMsgList; 136 | 137 | for (int i = 0; i < batchSize; i++) { 138 | totalSend += SendOnce(eplog, i, sendDelayMS, failedMsgList); 139 | } 140 | 141 | std::future serverTask; 142 | 143 | if (mockServerRestart) { 144 | std::promise thReady; 145 | 146 | auto disconnectMS = sendDelayMS * batchSize; 147 | serverTask = std::async(std::launch::async, [mockServer, &thReady, disconnectMS] () { 148 | thReady.set_value(); 149 | mockServer->DisconnectAndRun(disconnectMS); 150 | }); 151 | 152 | thReady.get_future().wait(); // wait until mockServer starts to call DisconnectAndRun() 153 | } 154 | 155 | for (int i = batchSize; i < nmsgs; i++) { 156 | totalSend += SendOnce(eplog, i, sendDelayMS, failedMsgList); 157 | } 158 | 159 | // resend failed cases 160 | std::vector resendFailList; 161 | for (const auto & i : failedMsgList) { 162 | totalSend += SendOnce(eplog, i, 0, resendFailList); 163 | } 164 | BOOST_CHECK_EQUAL(0, resendFailList.size()); 165 | 166 | WaitForClientCacheEmpty(eplog, testRuntimeMS); 167 | 168 | totalSend += SendEndOfTestToServer(eplog); 169 | 170 | bool mockServerDone = mockServer->WaitForTestsDone(testRuntimeMS); 171 | BOOST_CHECK(mockServerDone); 172 | 173 | mockServer->Stop(); 174 | 175 | if (serverTask.valid()) { 176 | serverTask.get(); 177 | } 178 | 179 | ValidateServerResults(mockServer->GetUniqDataRead(), nmsgs); 180 | 181 | return totalSend; 182 | } 183 | 184 | // Send messages to MockServer from SocketLogger. 185 | // sendDelayMS: milliseconds to delay between each Send(). 186 | // mockServerRestart: if true, restart MockServer in the middle to simulate mdsd failures. 187 | // validate: E2E works 188 | static void 189 | TestClientServerE2E( 190 | int nmsgs, 191 | int sendDelayMS, 192 | bool mockServerRestart 193 | ) 194 | { 195 | try { 196 | const std::string sockfile = TestUtil::GetCurrDir() + "/eplog-bvt"; 197 | auto mockServer = std::make_shared(sockfile); 198 | mockServer->Init(); 199 | 200 | // start server in a thread 201 | auto serverTask = std::async(std::launch::async, [mockServer]() { 202 | mockServer->Run(); 203 | }); 204 | 205 | auto totalSend = SendDataToServer(mockServer, sockfile, nmsgs, sendDelayMS, mockServerRestart); 206 | 207 | // wait for mockServer to finish 208 | serverTask.get(); 209 | 210 | auto totalReceived = mockServer->GetTotalBytesRead(); 211 | 212 | // because EndpoingLogger will add msg id, schema id etc, the totalSend and totalReceived 213 | // won't be equal 214 | BOOST_TEST_MESSAGE("TotalSend=" << totalSend << "; TotalReceived=" << totalReceived); 215 | BOOST_CHECK_LT(totalSend, totalReceived); 216 | } 217 | catch(const std::exception & ex) { 218 | BOOST_FAIL("Test exception: " << ex.what()); 219 | } 220 | } 221 | 222 | BOOST_AUTO_TEST_CASE(Test_SocketLogger_Msg_1) 223 | { 224 | TestClientServerE2E(1, 0, false); 225 | } 226 | 227 | BOOST_AUTO_TEST_CASE(Test_SocketLogger_Msg_100) 228 | { 229 | TestClientServerE2E(100, 0, false); 230 | } 231 | 232 | BOOST_AUTO_TEST_CASE(Test_SocketLogger_Failure_1) 233 | { 234 | TestClientServerE2E(1, 10, true); 235 | } 236 | 237 | BOOST_AUTO_TEST_CASE(Test_SocketLogger_Failure_12) 238 | { 239 | TestClientServerE2E(12, 10, true); 240 | } 241 | 242 | BOOST_AUTO_TEST_CASE(Test_SocketLogger_Failure_1000) 243 | { 244 | TestClientServerE2E(1000, 1, true); 245 | } 246 | 247 | BOOST_AUTO_TEST_SUITE_END() 248 | 249 | -------------------------------------------------------------------------------- /src/test/outmdsd/testlogitem.cc: -------------------------------------------------------------------------------- 1 | #include 2 | 3 | #include "LogItem.h" 4 | #include "DjsonLogItem.h" 5 | #include "EtwLogItem.h" 6 | #include "IdMgr.h" 7 | #include "testutil.h" 8 | 9 | using namespace EndpointLog; 10 | 11 | BOOST_AUTO_TEST_SUITE(logitem) 12 | 13 | BOOST_AUTO_TEST_CASE(Test_LogItem_Data) 14 | { 15 | const std::string source = "testSource"; 16 | const std::string schemaAndData = "testSchemaAndData"; 17 | 18 | try { 19 | DjsonLogItem item(source, schemaAndData); 20 | auto tag = item.GetTag(); 21 | auto data = item.GetData(); 22 | BOOST_REQUIRE_GT(tag.size(), 0); 23 | 24 | std::ostringstream expected; 25 | auto len = source.size() + schemaAndData.size() + tag.size() + 6; 26 | expected << len << "\n[\"" << source << "\"," << tag << "," << schemaAndData << "]"; 27 | BOOST_CHECK_EQUAL(expected.str(), data); 28 | } 29 | catch(const std::exception & ex) { 30 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 31 | } 32 | } 33 | 34 | BOOST_AUTO_TEST_CASE(Test_LogItem_CacheTime) 35 | { 36 | const std::string source = "testSource"; 37 | const std::string schemaAndData = "testSchemaAndData"; 38 | 39 | try { 40 | DjsonLogItem item(source, schemaAndData); 41 | item.Touch(); 42 | int nexpectedMS = 20; 43 | 44 | // Only compare the time when usleep() succeeds. 45 | if (0 == usleep(nexpectedMS*1000)) { 46 | auto actualMS = item.GetLastTouchMilliSeconds(); 47 | // sleep given time is never accurate. So allow some buffer. 48 | BOOST_CHECK_MESSAGE(abs(actualMS - nexpectedMS) <= 3, 49 | "actualMS=" << actualMS << "; nexpectedMS=" << nexpectedMS); 50 | } 51 | } 52 | catch(const std::exception & ex) { 53 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 54 | } 55 | } 56 | 57 | BOOST_AUTO_TEST_CASE(Test_IdMgr_BVT) 58 | { 59 | try { 60 | IdMgr m; 61 | 62 | IdMgr::value_type_t emptyResult; 63 | BOOST_CHECK_THROW(m.GetItem("", emptyResult), std::invalid_argument); 64 | 65 | const std::string key = "abc"; 66 | const std::string testValue = "testvalue"; 67 | 68 | IdMgr::value_type_t value = std::make_pair(123, testValue); 69 | auto valueCopy = value; 70 | 71 | BOOST_CHECK(!m.GetItem(key, value)); 72 | BOOST_CHECK(valueCopy == value); 73 | 74 | // SaveItem won't create new id if key already exists 75 | for (int i = 0; i < 3; i++) { 76 | auto id1 = m.FindOrInsert(key, testValue); 77 | BOOST_CHECK_EQUAL(1, id1); 78 | } 79 | 80 | IdMgr::value_type_t newValue; 81 | BOOST_CHECK(m.GetItem(key, newValue)); 82 | BOOST_CHECK_EQUAL(1, newValue.first); 83 | BOOST_CHECK_EQUAL(testValue, newValue.second); 84 | 85 | const std::string key2 = "def"; 86 | m.Insert(key2, newValue); 87 | 88 | IdMgr::value_type_t setValue; 89 | BOOST_CHECK(m.GetItem(key2, setValue)); 90 | BOOST_CHECK(setValue == newValue); 91 | } 92 | catch(const std::exception & ex) { 93 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 94 | } 95 | } 96 | 97 | static void 98 | TestEtwLogItem() 99 | { 100 | EtwLogItem etwlog("testsource", "testguid", 123); 101 | 102 | etwlog.AddData("bool_true", true); 103 | etwlog.AddData("bool_false", false); 104 | 105 | etwlog.AddData("int32_data", 1); 106 | int64_t n = ((int64_t)1) << 40; 107 | etwlog.AddData("int64_data", n); 108 | 109 | etwlog.AddData("double_data", 0.0000004); 110 | etwlog.AddData("time_data", 11, 22); 111 | 112 | etwlog.AddData("charstr_data", "charstr"); 113 | std::string str = "std::string"; 114 | etwlog.AddData("stdstr_data", str); 115 | 116 | const std::string expectedData = R"([["GUID","FT_STRING"],["EventId","FT_INT32"],["bool_true","FT_BOOL"],["bool_false","FT_BOOL"],["int32_data","FT_INT32"],["int64_data","FT_INT64"],["double_data","FT_DOUBLE"],["time_data","FT_TIME"],["charstr_data","FT_STRING"],["stdstr_data","FT_STRING"]],["testguid",123,true,false,1,1099511627776,4e-07,[11,22],"charstr","std::string"]])"; 117 | 118 | std::string actualData = etwlog.GetData(); 119 | 120 | BOOST_CHECK_MESSAGE(actualData.find(expectedData) != std::string::npos, 121 | "Actual='" << actualData << "'; Expected='" << expectedData << "'"); 122 | } 123 | 124 | BOOST_AUTO_TEST_CASE(Test_EtwLogItem_BVT) 125 | { 126 | try { 127 | TestEtwLogItem(); 128 | } 129 | catch(const std::exception & ex) { 130 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 131 | } 132 | } 133 | 134 | // Extract schema id and return it as a string. 135 | // Example input: '134\n["testsource",,,[[... 136 | // 137 | static std::string 138 | GetSchemaId( 139 | const std::string & str 140 | ) 141 | { 142 | auto p1 = str.find_first_of(','); 143 | if (p1 != std::string::npos) { 144 | auto p2 = str.find_first_of(',', p1+1); 145 | if (p2 != std::string::npos) { 146 | auto p3 = str.find_first_of(',', p2+1); 147 | return str.substr(p2+1, p3-p2); 148 | } 149 | } 150 | BOOST_FAIL("Invalid message str '" << str << "'"); 151 | return std::string(); 152 | } 153 | 154 | // Test that data name order shouldn't matter when creating 155 | // new schema string. 156 | static void 157 | TestEtwLogNameOrder() 158 | { 159 | EtwLogItem item1("testsource", "testguid", 123); 160 | item1.AddData("int32_data", 1); 161 | item1.AddData("bool", true); 162 | 163 | std::string data1 = item1.GetData(); 164 | auto schemaId = GetSchemaId(data1); 165 | 166 | // Repeat same name order to item1. 167 | // Validate: It should have same schema ID with item1. 168 | EtwLogItem item2("testsource", "testguid", 123); 169 | item2.AddData("int32_data", 2); 170 | item2.AddData("bool", false); 171 | 172 | // NOTE: GUID is added first, then EventId 173 | const std::string expectedData2 = schemaId + 174 | R"([["GUID","FT_STRING"],["EventId","FT_INT32"],["int32_data","FT_INT32"],["bool","FT_BOOL"]],["testguid",123,2,false]])"; 175 | 176 | std::string actualData2 = item2.GetData(); 177 | bool isMatch2 = actualData2.find(expectedData2) != std::string::npos; 178 | BOOST_CHECK_MESSAGE(isMatch2, "Item2='" << item2.GetData() << "'"); 179 | 180 | // Change name order from the previous items 181 | // Validate: it should not have same schema ID with item1. 182 | EtwLogItem item3("testsource", "testguid", 123); 183 | item3.AddData("bool", false); 184 | item3.AddData("int32_data", 3); 185 | 186 | std::string actualData3 = item3.GetData(); 187 | auto schemaId3 = GetSchemaId(actualData3); 188 | BOOST_CHECK_NE(schemaId, schemaId3); 189 | 190 | const std::string expectedData3 = schemaId3 + 191 | R"([["GUID","FT_STRING"],["EventId","FT_INT32"],["bool","FT_BOOL"],["int32_data","FT_INT32"]],["testguid",123,false,3]])"; 192 | 193 | bool isMatch3 = actualData3.find(expectedData3) != std::string::npos; 194 | BOOST_CHECK_MESSAGE(isMatch3, "Item3: Actual='" << actualData3 195 | << "'; ExpectedContain='" << expectedData3 << "'"); 196 | } 197 | 198 | BOOST_AUTO_TEST_CASE(Test_EtwLogItem_NameOrder) 199 | { 200 | try { 201 | TestEtwLogNameOrder(); 202 | } 203 | catch(const std::exception & ex) { 204 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 205 | } 206 | } 207 | 208 | static void 209 | CreateEtwLogItems(size_t nitems) 210 | { 211 | for (size_t i = 0; i < nitems; i++) { 212 | EtwLogItem etwlog("testsource", "testguid", 123); 213 | 214 | etwlog.AddData("bool_true", true); 215 | etwlog.AddData("bool_false", false); 216 | 217 | etwlog.AddData("int32_data", (int64_t) i); 218 | etwlog.AddData("int64_data", (int64_t) i); 219 | 220 | etwlog.AddData("double_data", 0.0000004); 221 | etwlog.AddData("time_data", 11, 22); 222 | 223 | etwlog.AddData("charstr_data", "charstr"); 224 | std::string str = "std::string " + std::to_string(i); 225 | etwlog.AddData("stdstr_data", str); 226 | 227 | (void)etwlog.GetData(); 228 | } 229 | } 230 | 231 | #if 0 232 | // Used to tune EtwLogItem related perf 233 | BOOST_AUTO_TEST_CASE(Test_EtwLogItem_Perf) 234 | { 235 | try { 236 | CreateEtwLogItems(10000); 237 | } 238 | catch(const std::exception & ex) { 239 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 240 | } 241 | } 242 | #endif 243 | 244 | BOOST_AUTO_TEST_SUITE_END() 245 | -------------------------------------------------------------------------------- /src/test/outmdsd/testmap.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include "ConcurrentMap.h" 3 | #include "DjsonLogItem.h" 4 | #include "LogItemPtr.h" 5 | #include "testutil.h" 6 | #include "CounterCV.h" 7 | 8 | using namespace EndpointLog; 9 | 10 | BOOST_AUTO_TEST_SUITE(testmap) 11 | 12 | bool filterFunc(LogItemPtr) 13 | { 14 | return true; 15 | } 16 | 17 | BOOST_AUTO_TEST_CASE(Test_ConcurrentMap_AddGet) 18 | { 19 | try { 20 | ConcurrentMap m; 21 | BOOST_CHECK_EQUAL(0, m.Size()); 22 | 23 | const int testVal = 123; 24 | const std::string testKey = "testkey"; 25 | m.Add(testKey, testVal); 26 | 27 | BOOST_CHECK_EQUAL(1, m.Size()); 28 | BOOST_CHECK_EQUAL(testVal, m.Get(testKey)); 29 | 30 | // add existing key should overwrite 31 | const int testVal2 = testVal + 1; 32 | m.Add(testKey, testVal2); 33 | BOOST_CHECK_EQUAL(1, m.Size()); 34 | BOOST_CHECK_EQUAL(testVal2, m.Get(testKey)); 35 | 36 | // not exist key should throw 37 | BOOST_CHECK_THROW(m.Get("nosuchkey"), std::out_of_range); 38 | } 39 | catch(const std::exception & ex) { 40 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 41 | } 42 | } 43 | 44 | BOOST_AUTO_TEST_CASE(Test_ConcurrentMap_Filter) 45 | { 46 | try { 47 | ConcurrentMap m; 48 | 49 | auto listForEmptyMap = m.FilterEach(filterFunc); 50 | BOOST_CHECK_EQUAL(0, listForEmptyMap.size()); 51 | 52 | LogItemPtr p(new DjsonLogItem("testSource", "testSchemaData")); 53 | const std::string testKey = "testkey"; 54 | m.Add(testKey, p); 55 | 56 | auto listForMap1 = m.FilterEach(filterFunc); 57 | BOOST_CHECK_EQUAL(1, listForMap1.size()); 58 | } 59 | catch(const std::exception & ex) { 60 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 61 | } 62 | } 63 | 64 | BOOST_AUTO_TEST_CASE(Test_ConcurrentMap_Erase) 65 | { 66 | try { 67 | ConcurrentMap m; 68 | 69 | const std::string p = "testVal"; 70 | const char* testkey = "testkey"; 71 | m.Add(testkey, p); 72 | 73 | BOOST_CHECK_EQUAL(1, m.Size()); 74 | BOOST_CHECK_EQUAL(0, m.Erase("badkey")); 75 | BOOST_CHECK_EQUAL(1, m.Size()); 76 | 77 | BOOST_CHECK_EQUAL(1, m.Erase(testkey)); 78 | BOOST_CHECK_EQUAL(0, m.Size()); 79 | 80 | std::vector keylist = {"key1", "key2", "key3"}; 81 | for (const auto & key : keylist) { 82 | m.Add(key, p); 83 | } 84 | auto nGoodKeys = keylist.size(); 85 | keylist.push_back("notExistKey"); 86 | 87 | BOOST_CHECK_EQUAL(nGoodKeys, m.Size()); 88 | BOOST_CHECK_EQUAL(nGoodKeys, m.Erase(keylist)); 89 | BOOST_CHECK_EQUAL(0, m.Size()); 90 | } 91 | catch(const std::exception & ex) { 92 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 93 | } 94 | } 95 | 96 | void 97 | AddToMap( 98 | const std::shared_future & masterReady, 99 | std::promise& thReady, 100 | const std::string & keyprefix, 101 | ConcurrentMap& m, 102 | int nitems, 103 | const std::shared_ptr& cv 104 | ) 105 | { 106 | CounterCVWrap cvwrap(cv); 107 | thReady.set_value(); 108 | masterReady.wait(); // wait for all threads to be ready 109 | 110 | for(int i = 0; i < nitems; i++) { 111 | m.Add(keyprefix + std::to_string(i), i); 112 | } 113 | } 114 | 115 | void 116 | EraseFromMap( 117 | const std::shared_future & masterReady, 118 | std::promise& thReady, 119 | const std::string & keyprefix, 120 | ConcurrentMap& m, 121 | int nitems 122 | ) 123 | { 124 | thReady.set_value(); 125 | masterReady.wait(); // wait for all threads to be ready 126 | 127 | for(int i = 0; i < nitems; i++) { 128 | m.Erase(keyprefix + std::to_string(i)); 129 | } 130 | } 131 | 132 | // Run multiple adder and eraser threads. This is to make sure that ConcurrentMap 133 | // won't have unexpected hang or exception. It doesn't validate the exact values 134 | // are added/erased. 135 | // To make sure that all threads are run concurrently, use std::promise and 136 | // std::shared_future to coordinate the threads. 137 | void 138 | MultiAddEraseTest( 139 | int nAddThreads, 140 | int nEraseThreads, 141 | int nitems 142 | ) 143 | { 144 | // use a promise and shared_future to sync all threads to start at the same time 145 | std::promise masterPromise; 146 | std::shared_future masterReady(masterPromise.get_future()); 147 | 148 | std::vector> thReadyList; 149 | for (int i = 0; i < nAddThreads+nEraseThreads; i++) { 150 | thReadyList.push_back(std::promise()); 151 | } 152 | 153 | ConcurrentMap m; 154 | std::vector> taskList; 155 | 156 | auto taskCV = std::make_shared(nAddThreads); 157 | 158 | const std::string keybase = "key"; 159 | 160 | for (int i = 0; i < nAddThreads; i++) { 161 | auto keyprefix = keybase + std::to_string(i); 162 | auto t = std::async(std::launch::async, AddToMap, masterReady, 163 | std::ref(thReadyList[i]), keyprefix, std::ref(m), nitems, taskCV); 164 | taskList.push_back(std::move(t)); 165 | } 166 | 167 | // if nEraseThread != nAddThread, not all keys are matched between Erase an Add. This is 168 | // OK because this can test code to handle not-exist keys. 169 | for (int i = 0; i < nEraseThreads; i++) { 170 | auto keyprefix = keybase + std::to_string(i); 171 | auto t = std::async(std::launch::async, EraseFromMap, masterReady, 172 | std::ref(thReadyList[i+nAddThreads]), keyprefix, std::ref(m), nitems); 173 | taskList.push_back(std::move(t)); 174 | } 175 | 176 | // wait for all threads to be ready 177 | for (auto & t : thReadyList) { 178 | t.get_future().wait(); 179 | } 180 | 181 | // start all threads 182 | masterPromise.set_value(); 183 | 184 | BOOST_CHECK(taskCV->wait_for(400)); 185 | } 186 | 187 | 188 | BOOST_AUTO_TEST_CASE(Test_ConcurrentMap_MultiThreads) 189 | { 190 | try { 191 | MultiAddEraseTest(6, 4, 10000); 192 | } 193 | catch(const std::exception & ex) { 194 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 195 | } 196 | } 197 | 198 | BOOST_AUTO_TEST_SUITE_END() 199 | -------------------------------------------------------------------------------- /src/test/outmdsd/testqueue.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include "testutil.h" 6 | #include "CounterCV.h" 7 | 8 | using namespace EndpointLog; 9 | 10 | BOOST_AUTO_TEST_SUITE(testqueue) 11 | 12 | BOOST_AUTO_TEST_CASE(Test_ConcurrentQueue_BVT) 13 | { 14 | try { 15 | ConcurrentQueue q; 16 | const int expected = 1234; 17 | q.push(expected); 18 | int actual = 0; 19 | q.wait_and_pop(actual); 20 | 21 | BOOST_CHECK_EQUAL(expected, actual); 22 | } 23 | catch(const std::exception & ex) { 24 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 25 | } 26 | } 27 | 28 | BOOST_AUTO_TEST_CASE(Test_ConcurrentQueue_MaxSize) 29 | { 30 | try { 31 | const int maxSize = 2; 32 | ConcurrentQueue q(maxSize); 33 | 34 | for (int i = 0; i < maxSize; i++) { 35 | q.push(i); 36 | BOOST_CHECK_MESSAGE(i+1, q.size()); 37 | } 38 | 39 | for (int i = 0; i < 10; i++) { 40 | q.push(i); 41 | BOOST_CHECK_MESSAGE(maxSize == q.size(), "pushed items=" << (maxSize+i+1)); 42 | } 43 | } 44 | catch(const std::exception & ex) { 45 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 46 | } 47 | } 48 | 49 | 50 | // This function should wait until element is ready. 51 | // Validate that 52 | // - it actual waits until some element is pushed to the queue. 53 | void 54 | WaitQueueItem( 55 | std::promise & masterReady, 56 | std::promise & threadReady, 57 | ConcurrentQueue* q, 58 | int expected, 59 | bool& pushedToQueue 60 | ) 61 | { 62 | threadReady.set_value(); 63 | masterReady.get_future().wait(); 64 | 65 | int actual = 0; 66 | q->wait_and_pop(actual); 67 | BOOST_REQUIRE_EQUAL(expected, actual); 68 | // Test that when wait_and_pop() is done, pushedToQueue must be set to be true. 69 | BOOST_CHECK(pushedToQueue); 70 | } 71 | 72 | BOOST_AUTO_TEST_CASE(Test_ConcurrentQueue_Wait) 73 | { 74 | try { 75 | // use a promise and shared_future to sync thread startup 76 | std::promise masterReady; 77 | std::promise threadReady; 78 | bool pushedToQueue = false; 79 | 80 | ConcurrentQueue q; 81 | const uint32_t minRunTimeMS = 10; 82 | const int expected = 1234; 83 | auto task = std::async(std::launch::async, WaitQueueItem, 84 | std::ref(masterReady), std::ref(threadReady), &q, expected, std::ref(pushedToQueue)); 85 | 86 | threadReady.get_future().wait(); 87 | masterReady.set_value(); 88 | 89 | std::this_thread::sleep_for(std::chrono::milliseconds(minRunTimeMS)); 90 | pushedToQueue = true; 91 | q.push(expected); 92 | 93 | // Validate that once an item is pushed to the queue, it shouldn't take 94 | // more than N milliseconds for wait_and_pop() thread to get the item and 95 | // break waiting. There is no exact value for N. The test uses some 96 | // reasonable small number. 97 | BOOST_CHECK(TestUtil::WaitForTask(task, 5)); 98 | } 99 | catch(const std::exception & ex) { 100 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 101 | } 102 | } 103 | 104 | // this function should wait until element is ready 105 | // validate that 106 | // - it actual waits 107 | // - it will stop when stop_once_empty() is called 108 | void 109 | WaitEmptyQueue( 110 | std::promise& masterReady, 111 | std::promise& threadReady, 112 | ConcurrentQueue* q, 113 | bool& stopQueue) 114 | { 115 | threadReady.set_value(); 116 | masterReady.get_future().wait(); 117 | 118 | const int origVal = -123; 119 | int actual = origVal; 120 | q->wait_and_pop(actual); 121 | 122 | // nothing should be popped, so value shouldn't be changed 123 | BOOST_CHECK_EQUAL(origVal, actual); 124 | // Test that when wait_and_pop() is done, stopQueue must be set to be true. 125 | BOOST_CHECK(stopQueue); 126 | } 127 | 128 | BOOST_AUTO_TEST_CASE(Test_ConcurrentQueue_StopWait) 129 | { 130 | try { 131 | // use a promise and shared_future to sync thread startup 132 | std::promise masterReady; 133 | std::promise threadReady; 134 | bool stopQueue = false; 135 | 136 | ConcurrentQueue q; 137 | const uint32_t minRunTimeMS = 10; 138 | 139 | auto task = std::async(std::launch::async, WaitEmptyQueue, 140 | std::ref(masterReady), std::ref(threadReady), &q, std::ref(stopQueue)); 141 | 142 | threadReady.get_future().wait(); 143 | masterReady.set_value(); 144 | 145 | std::this_thread::sleep_for(std::chrono::milliseconds(minRunTimeMS)); 146 | stopQueue = true; 147 | q.stop_once_empty(); 148 | 149 | // Validate that once stop_once_empty() is called, it shouldn't take 150 | // more than N milliseconds for wait_and_pop() thread to break waiting. 151 | // The value of N is implementation-dependent. The test uses 152 | // some reasonable small number. 153 | BOOST_CHECK(TestUtil::WaitForTask(task, 5)); 154 | } 155 | catch(const std::exception & ex) { 156 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 157 | } 158 | } 159 | 160 | void 161 | PushToQueue( 162 | const std::shared_future & masterReady, 163 | std::promise& thReady, 164 | ConcurrentQueue& q, 165 | int nitems, 166 | const std::shared_ptr& cv 167 | ) 168 | { 169 | CounterCVWrap cvwrap(cv); 170 | 171 | thReady.set_value(); 172 | masterReady.wait(); 173 | for (int i = 0; i < nitems; i++) { 174 | q.push(i); 175 | } 176 | } 177 | 178 | void 179 | PopFromQueue( 180 | const std::shared_future & masterReady, 181 | std::promise& thReady, 182 | ConcurrentQueue& q, 183 | const std::shared_ptr& cv 184 | ) 185 | { 186 | CounterCVWrap cvwrap(cv); 187 | 188 | thReady.set_value(); 189 | masterReady.wait(); 190 | while(true) { 191 | auto v = q.wait_and_pop(); 192 | if (nullptr == v) { 193 | break; 194 | } 195 | } 196 | } 197 | 198 | // To make sure that all threads are run concurrently, use std::promise and 199 | // std::shared_future to coordinate the threads. 200 | void 201 | MultiPushPopTest( 202 | int nPushThreads, 203 | int nPopThreads, 204 | int nitems 205 | ) 206 | { 207 | // use a promise and shared_future to sync all threads to start at the same time 208 | std::promise masterPromise; 209 | std::shared_future masterReady(masterPromise.get_future()); 210 | 211 | std::vector> thReadyList; 212 | for (int i = 0; i < (nPushThreads+nPopThreads); i++) { 213 | thReadyList.push_back(std::promise()); 214 | } 215 | 216 | ConcurrentQueue q; 217 | 218 | auto pushCV = std::make_shared(nPushThreads); 219 | auto popCV = std::make_shared(nPopThreads); 220 | 221 | std::vector> pushTasks; 222 | for (int i = 0; i < nPushThreads; i++) { 223 | auto t = std::async(std::launch::async, PushToQueue, masterReady, 224 | std::ref(thReadyList[i]), std::ref(q), nitems, pushCV); 225 | pushTasks.push_back(std::move(t)); 226 | } 227 | 228 | std::vector> popTasks; 229 | for (int i = 0; i < nPopThreads; i++) { 230 | auto t = std::async(std::launch::async, PopFromQueue, masterReady, 231 | std::ref(thReadyList[i+nPushThreads]), std::ref(q), popCV); 232 | popTasks.push_back(std::move(t)); 233 | } 234 | 235 | // wait for all threads to be ready 236 | for (auto & t: thReadyList) { 237 | t.get_future().wait(); 238 | } 239 | 240 | // start all threads 241 | masterPromise.set_value(); 242 | 243 | BOOST_CHECK(pushCV->wait_for(400)); 244 | q.stop_once_empty(); 245 | BOOST_CHECK(popCV->wait_for(400)); 246 | } 247 | 248 | BOOST_AUTO_TEST_CASE(Test_ConcurrentQueue_MultiThreads) 249 | { 250 | try { 251 | MultiPushPopTest(6, 4, 10000); 252 | } 253 | catch(const std::exception & ex) { 254 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 255 | } 256 | } 257 | 258 | BOOST_AUTO_TEST_SUITE_END() 259 | -------------------------------------------------------------------------------- /src/test/outmdsd/testreader.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | #include "MockServer.h" 5 | #include "SocketClient.h" 6 | #include "Exceptions.h" 7 | #include "DataReader.h" 8 | #include "testutil.h" 9 | #include "LogItemPtr.h" 10 | #include "ConcurrentMap.h" 11 | 12 | using namespace EndpointLog; 13 | 14 | BOOST_AUTO_TEST_SUITE(testreader) 15 | 16 | 17 | void 18 | StartSocketReader( 19 | std::promise& threadReady, 20 | const std::shared_ptr& sr, 21 | bool& stopRunLoop 22 | ) 23 | { 24 | threadReady.set_value(); 25 | sr->Run(); 26 | // Test that when Run() is done, stopRunLoop must be set to be true. 27 | BOOST_CHECK(stopRunLoop); 28 | } 29 | 30 | // This test will do 31 | // - start socket server 32 | // - start a socket client to send data to the server. 33 | // - start a socket reader to listen for data from server to client. 34 | // - the server will read data from client and send response back. 35 | // - validate reader can read the response data. 36 | static void 37 | SendDataToServer(int nmsgs) 38 | { 39 | try { 40 | const std::string sockfile = TestUtil::GetCurrDir() + "/sockreader-bvt"; 41 | auto mockServer = std::make_shared(sockfile); 42 | mockServer->Init(); 43 | 44 | // start server in a thread 45 | auto serverTask = std::async(std::launch::async, [mockServer]() { 46 | mockServer->Run(); 47 | return mockServer->GetTotalBytesRead(); 48 | }); 49 | 50 | auto sockClient = std::make_shared(sockfile, 1); 51 | auto dataCache = std::make_shared>(); 52 | 53 | // start reader in a thread 54 | auto sockReader = std::make_shared(sockClient, dataCache); 55 | auto readerTask = std::async(std::launch::async, [sockReader]() { sockReader->Run(); }); 56 | 57 | size_t totalSend = 0; 58 | 59 | for (int i = 0; i < nmsgs; i++) { 60 | auto testData = "SocketClient test data " + std::to_string(i+1); 61 | sockClient->Send(testData.c_str()); 62 | totalSend += testData.size(); 63 | } 64 | // end of test 65 | sockClient->Send(TestUtil::EndOfTest().c_str()); 66 | totalSend += TestUtil::EndOfTest().size(); 67 | 68 | bool mockServerDone = mockServer->WaitForTestsDone(500); 69 | BOOST_CHECK(mockServerDone); 70 | 71 | sockClient->Stop(); 72 | sockClient->Close(); 73 | sockReader->Stop(); 74 | readerTask.get(); 75 | 76 | mockServer->Stop(); 77 | 78 | auto nTagsRead = sockReader->GetNumTagsRead(); 79 | auto nTagsWrite = mockServer->GetTotalTags(); 80 | BOOST_CHECK_EQUAL(nTagsWrite, nTagsRead); 81 | 82 | auto totalReceived = serverTask.get(); 83 | BOOST_CHECK_EQUAL(totalSend, totalReceived); 84 | } 85 | catch(const std::exception & ex) { 86 | BOOST_FAIL("Test exception " << ex.what()); 87 | } 88 | } 89 | 90 | 91 | BOOST_AUTO_TEST_CASE(Test_SocketReader_Error) 92 | { 93 | try { 94 | const std::string socketfile = "/tmp/nosuchfile"; 95 | auto sockClient = std::make_shared(socketfile, 1); 96 | auto dataCache = std::make_shared>(); 97 | 98 | std::promise threadReady; 99 | bool stopRunLoop = false; 100 | 101 | auto sockReader = std::make_shared(sockClient, dataCache); 102 | auto readerTask = std::async(std::launch::async, StartSocketReader, 103 | std::ref(threadReady),sockReader, std::ref(stopRunLoop)); 104 | 105 | // wait until StartSocketReader thread starts 106 | threadReady.get_future().wait(); 107 | 108 | usleep(100*1000); 109 | 110 | sockClient->Stop(); 111 | // sockReader->Stop() should break sockReader's Run() loop. 112 | stopRunLoop = true; 113 | sockReader->Stop(); 114 | 115 | // Validate that once DataReader::Stop() is called, it shouldn't take 116 | // more than N milliseconds for DataReader::Run() thread to break. 117 | // There is no exact value for N. The test uses some reasonable small number. 118 | BOOST_CHECK(TestUtil::WaitForTask(readerTask, 5)); 119 | } 120 | catch(const std::exception & ex) { 121 | BOOST_FAIL("Test exception " << ex.what()); 122 | } 123 | } 124 | 125 | BOOST_AUTO_TEST_CASE(Test_SocketReader_Msg_1) 126 | { 127 | SendDataToServer(1); 128 | } 129 | 130 | BOOST_AUTO_TEST_CASE(Test_SocketReader_Msg_100) 131 | { 132 | SendDataToServer(100); 133 | } 134 | 135 | BOOST_AUTO_TEST_SUITE_END() 136 | -------------------------------------------------------------------------------- /src/test/outmdsd/testresender.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include "ConcurrentMap.h" 4 | #include "SocketClient.h" 5 | #include "DataResender.h" 6 | #include "DjsonLogItem.h" 7 | #include "testutil.h" 8 | 9 | using namespace EndpointLog; 10 | 11 | BOOST_AUTO_TEST_SUITE(testresender) 12 | 13 | 14 | size_t 15 | StartDataResender( 16 | std::promise& threadReady, 17 | const std::shared_ptr& resender, 18 | bool& stopRunLoop 19 | ) 20 | { 21 | threadReady.set_value(); 22 | 23 | auto nTimerTriggered = resender->Run(); 24 | // When resender Run() is done, the stopRunLoop must be set to be true. 25 | BOOST_CHECK(stopRunLoop); 26 | return nTimerTriggered; 27 | } 28 | 29 | std::shared_ptr 30 | CreateDataResender( 31 | const std::shared_ptr& sockClient, 32 | std::shared_ptr> dataCache, 33 | size_t cacheSize, 34 | uint32_t retryMS 35 | ) 36 | { 37 | for (size_t i = 0; i < cacheSize; i++) { 38 | auto indexStr = std::to_string(i+1); 39 | auto key = "testkey-" + indexStr; 40 | LogItemPtr value(new DjsonLogItem("testsource", "testvalue-" + indexStr)); 41 | dataCache->Add(key, value); 42 | } 43 | 44 | return std::make_shared(sockClient, dataCache, 1, retryMS); 45 | } 46 | 47 | BOOST_AUTO_TEST_CASE(Test_DataResender_EmptyCache) 48 | { 49 | try { 50 | const uint32_t retryMS = 200; 51 | const int nExpectedTimers = 2; 52 | const uint32_t minRunTimeMS = retryMS * nExpectedTimers + retryMS/2; 53 | 54 | // To sync between main thread and StartDataResender thread 55 | std::promise threadReady; 56 | bool stopRunLoop = false; 57 | 58 | auto sockClient = std::make_shared("/tmp/nosuchfile", 1); 59 | auto resender = CreateDataResender(sockClient, std::make_shared>(), 0, retryMS); 60 | auto task = std::async(std::launch::async, StartDataResender, std::ref(threadReady), resender, std::ref(stopRunLoop)); 61 | 62 | // Wait until StartDataResender is ready before starting timer 63 | threadReady.get_future().wait(); 64 | usleep(minRunTimeMS*1000); 65 | 66 | sockClient->Stop(); 67 | stopRunLoop = true; 68 | resender->Stop(); 69 | 70 | // Validate that once DataResender::Stop() is called, it shouldn't take 71 | // more than N milliseconds for DataResender::Run() thread to break. 72 | // There is no exact value for N. The test uses some reasonable small number. 73 | BOOST_CHECK(TestUtil::WaitForTask(task, 5)); 74 | 75 | auto nTimerTriggered = task.get(); 76 | BOOST_CHECK_EQUAL(nExpectedTimers, nTimerTriggered); 77 | } 78 | catch(const std::exception & ex) { 79 | BOOST_FAIL("Test exception " << ex.what()); 80 | } 81 | } 82 | 83 | BOOST_AUTO_TEST_CASE(Test_DataResender_OneItem) 84 | { 85 | try { 86 | auto retryMS = 50; 87 | auto minRunTimeMS = retryMS+retryMS/10; 88 | 89 | std::promise threadReady; 90 | bool stopRunLoop = false; 91 | 92 | auto sockClient = std::make_shared("/tmp/nosuchfile", 1); 93 | auto dataCache = std::make_shared>(); 94 | auto resender = CreateDataResender(sockClient, dataCache, 10, retryMS); 95 | auto task = std::async(std::launch::async, StartDataResender, std::ref(threadReady), resender, std::ref(stopRunLoop)); 96 | 97 | // Wait until StartDataResender is ready before starting timer 98 | threadReady.get_future().wait(); 99 | usleep(minRunTimeMS*1000); 100 | 101 | sockClient->Stop(); 102 | // resender->Stop() should stop the Run() loop of resender. 103 | stopRunLoop = true; 104 | resender->Stop(); 105 | 106 | // Validate that once DataResender::Stop() is called, it shouldn't take 107 | // more than N milliseconds for DataResender::Run() thread to break. 108 | // There is no exact value for N. The test uses some reasonable small number. 109 | BOOST_CHECK(TestUtil::WaitForTask(task, 5)); 110 | 111 | auto nTimerTriggered = task.get(); 112 | BOOST_CHECK_EQUAL(1, nTimerTriggered); 113 | 114 | // validate data items are erased 115 | BOOST_CHECK_EQUAL(0, dataCache->Size()); 116 | } 117 | catch(const std::exception & ex) { 118 | BOOST_FAIL("Test exception " << ex.what()); 119 | } 120 | } 121 | 122 | 123 | BOOST_AUTO_TEST_SUITE_END() 124 | -------------------------------------------------------------------------------- /src/test/outmdsd/testsender.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | 4 | #include "ConcurrentMap.h" 5 | #include "ConcurrentQueue.h" 6 | #include "DataSender.h" 7 | #include "SocketClient.h" 8 | #include "DjsonLogItem.h" 9 | #include "testutil.h" 10 | #include "MockServer.h" 11 | 12 | using namespace EndpointLog; 13 | 14 | BOOST_AUTO_TEST_SUITE(testsender) 15 | 16 | static void 17 | RunSender( 18 | std::promise& threadReady, 19 | DataSender& sender, 20 | bool& stopSender 21 | ) 22 | { 23 | threadReady.set_value(); 24 | sender.Run(); 25 | // When sender Run() is done, the stopSender must be set to be true. 26 | BOOST_CHECK(stopSender); 27 | } 28 | 29 | static void 30 | AddItemsToQueue( 31 | const std::shared_ptr>& q, 32 | size_t nitems, 33 | size_t nbytesPerItem, 34 | int delayMicroSeconds // number of micro-seconds to delay between send 35 | ) 36 | { 37 | auto testdata = std::string(nbytesPerItem, 'A'); 38 | 39 | for (size_t i = 0; i < nitems; i++) { 40 | LogItemPtr p(new DjsonLogItem("testsource", testdata)); 41 | q->push(std::move(p)); 42 | 43 | if (delayMicroSeconds > 0) { 44 | usleep(delayMicroSeconds); 45 | } 46 | } 47 | } 48 | 49 | static void 50 | AddEndOfTestToQueue( 51 | const std::shared_ptr>& q 52 | ) 53 | { 54 | LogItemPtr p(new DjsonLogItem("testsource", TestUtil::EndOfTest())); 55 | q->push(std::move(p)); 56 | } 57 | 58 | // This test validates DataSender APIs only. It doesn't validate the data flow 59 | // from socket client to the socket server. 60 | static void 61 | RunAPITest( 62 | bool useDataCache 63 | ) 64 | { 65 | try { 66 | std::promise threadReady; 67 | bool stopSender = false; 68 | 69 | const std::string socketfile = "/tmp/nosuchfile"; 70 | auto sockClient = std::make_shared(socketfile, 1); 71 | auto q = std::make_shared>(); 72 | 73 | std::shared_ptr> cache; 74 | if (useDataCache) { 75 | cache = std::make_shared>(); 76 | } 77 | 78 | const size_t nitems = 2; 79 | AddItemsToQueue(q, nitems, 10, 0); 80 | 81 | DataSender sender(sockClient, cache, q); 82 | auto senderTask = std::async(std::launch::async, RunSender, std::ref(threadReady), std::ref(sender), 83 | std::ref(stopSender)); 84 | 85 | threadReady.get_future().wait(); 86 | usleep(10*1000); 87 | stopSender = true; 88 | sender.Stop(); 89 | q->stop_once_empty(); 90 | 91 | BOOST_CHECK_EQUAL(nitems, sender.GetNumSend()); 92 | BOOST_CHECK_EQUAL(0, q->size()); 93 | BOOST_CHECK_EQUAL(0, sender.GetNumSuccess()); 94 | 95 | // Validate that once DataSender::Stop() is called, it shouldn't take 96 | // more than N milliseconds for DataSender::Run() thread to break. 97 | // There is no exact value for N. The test uses some reasonable small number. 98 | BOOST_CHECK(TestUtil::WaitForTask(senderTask, 5)); 99 | 100 | if (cache) { 101 | BOOST_CHECK_EQUAL(nitems, cache->Size()); 102 | } 103 | } 104 | catch(const std::exception & ex) { 105 | BOOST_FAIL("Test failed with exception " << ex.what()); 106 | } 107 | } 108 | 109 | // Test DataSender with invalid socket 110 | // - all data sent should fail. 111 | // - DataSender::Run() should continue until it is told to stop. 112 | // - no unexpected exception 113 | // 114 | BOOST_AUTO_TEST_CASE(Test_DataSender_InvalidSocket) 115 | { 116 | RunAPITest(false); 117 | } 118 | 119 | // Test that all data are moved to the cache after DataSender::Run() 120 | BOOST_AUTO_TEST_CASE(Test_DataSender_Cache) 121 | { 122 | RunAPITest(true); 123 | } 124 | 125 | void 126 | RunE2ETest(size_t nitems, size_t nbytesPerItem) 127 | { 128 | const std::string sockfile = TestUtil::GetCurrDir() + "/datasender-bvt"; 129 | auto mockServer = std::make_shared(sockfile, false); 130 | mockServer->Init(); 131 | 132 | auto serverTask = std::async(std::launch::async, [mockServer]() { mockServer->Run(); }); 133 | 134 | auto sockClient = std::make_shared(sockfile, 20); 135 | auto incomingQueue = std::make_shared>(); 136 | auto dataCache = std::make_shared>(); 137 | 138 | std::promise threadReady; 139 | DataSender sender(sockClient, dataCache, incomingQueue); 140 | auto senderTask = std::async(std::launch::async, [&sender]() { sender.Run(); }); 141 | 142 | AddItemsToQueue(incomingQueue, nitems, nbytesPerItem, 10); 143 | AddEndOfTestToQueue(incomingQueue); 144 | nitems++; // end of test message 145 | 146 | // The time it takes the mockServer to proces all msgs can vary depending on 147 | // test machine perf. Use a little bigger value to make sure all msgs are 148 | // received. 149 | bool mockServerDone = mockServer->WaitForTestsDone(3000); 150 | BOOST_CHECK(mockServerDone); 151 | 152 | mockServer->Stop(); 153 | sender.Stop(); 154 | incomingQueue->stop_once_empty(); 155 | // Wait until sender task is finished or timed out. 156 | // Without this, senderTask could be in the middle of sending, such that the sender 157 | // counters (e.g. GetNumSend(), GetNumSuccess(), etc) can be invalid. 158 | BOOST_CHECK(TestUtil::WaitForTask(senderTask, 500)); 159 | 160 | BOOST_CHECK_EQUAL(nitems, sender.GetNumSend()); 161 | BOOST_CHECK_EQUAL(0, incomingQueue->size()); 162 | 163 | BOOST_CHECK_EQUAL(sender.GetNumSend(), sender.GetNumSuccess()); 164 | BOOST_CHECK_EQUAL(sender.GetNumSend(), dataCache->Size()); 165 | BOOST_CHECK_EQUAL(0, mockServer->GetTotalTags()); // no tag parsing, so GetTotalTags() should be 0 166 | 167 | // actual data are more than minSendBytes because extra data added in DJSON 168 | auto minSendBytes = (nitems-1) * nbytesPerItem + TestUtil::EndOfTest().size(); 169 | BOOST_CHECK_GT(mockServer->GetTotalBytesRead(), minSendBytes); 170 | } 171 | 172 | BOOST_AUTO_TEST_CASE(Test_DataSender_BVT) 173 | { 174 | try { 175 | RunE2ETest(1, 100); 176 | } 177 | catch(const std::exception & ex) { 178 | BOOST_FAIL("Test failed with exception " << ex.what()); 179 | } 180 | } 181 | 182 | BOOST_AUTO_TEST_CASE(Test_DataSender_Stress) 183 | { 184 | try { 185 | RunE2ETest(5000, 100); 186 | } 187 | catch(const std::exception & ex) { 188 | BOOST_FAIL("Test failed with exception " << ex.what()); 189 | } 190 | } 191 | 192 | BOOST_AUTO_TEST_SUITE_END() 193 | -------------------------------------------------------------------------------- /src/test/outmdsd/testtrace.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | 5 | extern "C" { 6 | #include 7 | } 8 | 9 | #include "Trace.h" 10 | #include "SyslogTracer.h" 11 | #include "TraceMacros.h" 12 | #include "testutil.h" 13 | 14 | using namespace EndpointLog; 15 | 16 | BOOST_AUTO_TEST_SUITE(testtrace) 17 | 18 | // Use a high resolution clock to create a test string. 19 | // It should be unique across tests 20 | static std::string 21 | GetTestStr(int level) 22 | { 23 | static int counter = 0; 24 | static auto timens = std::chrono::steady_clock::now().time_since_epoch() / std::chrono::nanoseconds(1); 25 | 26 | counter++; 27 | std::ostringstream strm; 28 | strm << "TraceLevel_" << level << " " << timens << "-" << counter; 29 | return strm.str(); 30 | } 31 | 32 | // Search a set of strings from /var/log/syslog. 33 | // Because it takes undetermined time for a msg to show up in 34 | // /var/log/syslog, search N times with incremental delay. 35 | // Return true if all items are found, return false if not all are found. 36 | static bool 37 | SearchInSyslog( 38 | std::streampos startPos, 39 | std::unordered_set & searchSet 40 | ) 41 | { 42 | int ntimes = 10; 43 | 44 | for (int i = 0; i < ntimes; i++) { 45 | auto foundAll = TestUtil::SearchStrings("/var/log/syslog", startPos, searchSet); 46 | if (foundAll) { 47 | return true; 48 | } 49 | if (i != (ntimes-1)) { 50 | usleep((i+1)*1000); // do incremental delay 51 | } 52 | } 53 | return false; 54 | } 55 | 56 | static void 57 | ValidateInSyslog( 58 | std::streampos startPos, 59 | std::unordered_set & searchSet 60 | ) 61 | { 62 | auto foundAll = SearchInSyslog(startPos, searchSet); 63 | 64 | BOOST_CHECK(foundAll); 65 | if (!foundAll) { 66 | for (const auto & str : searchSet) { 67 | BOOST_TEST_MESSAGE("Error: item '" << str << "' is not found in syslog."); 68 | } 69 | } 70 | } 71 | 72 | static void 73 | ValidateNotInSyslog( 74 | std::streampos startPos, 75 | std::unordered_set & searchSet 76 | ) 77 | { 78 | auto sizeCopy = searchSet.size(); 79 | auto foundAll = SearchInSyslog(startPos, searchSet); 80 | BOOST_CHECK(!foundAll); 81 | BOOST_CHECK_EQUAL(sizeCopy, searchSet.size()); 82 | } 83 | 84 | static void 85 | RunTraceTest(TraceLevel minLevel) 86 | { 87 | Trace::SetTracer(new SyslogTracer(LOG_CONS, LOG_SYSLOG)); 88 | Trace::SetTraceLevel(minLevel); 89 | 90 | auto origFileSz = TestUtil::GetFileSize("/var/log/syslog"); 91 | 92 | auto start = static_cast(TraceLevel::Trace); 93 | auto end = static_cast(TraceLevel::Fatal); 94 | 95 | std::unordered_set expectedSet; 96 | std::unordered_set unexpectedSet; 97 | 98 | for (int i = start; i <= end; i++) { 99 | auto msg = GetTestStr(i); 100 | Log(static_cast(i), msg); 101 | 102 | if (i >= static_cast(minLevel)) { 103 | expectedSet.insert(msg); 104 | } 105 | else { 106 | unexpectedSet.insert(msg); 107 | } 108 | } 109 | 110 | if (!expectedSet.empty()) { 111 | ValidateInSyslog(origFileSz, expectedSet); 112 | } 113 | if (!unexpectedSet.empty()) { 114 | ValidateNotInSyslog(origFileSz, unexpectedSet); 115 | } 116 | } 117 | 118 | // Test basic logging to syslog 119 | // NOTE: make sure that some syslog daemon (like rsyslogd, syslog-ng, etc) 120 | // is running before tests are started. Because no way to tell which syslog 121 | // daemon is going to run, this is not asserted in the test. 122 | BOOST_AUTO_TEST_CASE(Test_Syslog_Basic) 123 | { 124 | try { 125 | RunTraceTest(TraceLevel::Trace); 126 | } 127 | catch(const std::exception & ex) { 128 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 129 | } 130 | } 131 | 132 | // Test that only levels >= SetTraceLevel are logged 133 | BOOST_AUTO_TEST_CASE(Test_Syslog_Level) 134 | { 135 | try { 136 | RunTraceTest(TraceLevel::Warning); 137 | } 138 | catch(const std::exception & ex) { 139 | BOOST_FAIL("Test failed with unexpected exception: " << ex.what()); 140 | } 141 | } 142 | 143 | 144 | BOOST_AUTO_TEST_SUITE_END() 145 | -------------------------------------------------------------------------------- /src/test/outmdsd/testutil.cc: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | #include 6 | 7 | extern "C" { 8 | #include 9 | #include 10 | #include 11 | #include 12 | #include 13 | } 14 | 15 | #include "testutil.h" 16 | 17 | std::string 18 | TestUtil::GetCurrDir() 19 | { 20 | char buf[1024]; 21 | if (!getcwd(buf, sizeof(buf))) { 22 | throw std::system_error(errno, std::system_category(), "getcwd failed"); 23 | } 24 | return buf; 25 | } 26 | 27 | bool 28 | TestUtil::IsFileExists( 29 | const std::string & filepath 30 | ) 31 | { 32 | if (filepath.empty()) { 33 | throw std::invalid_argument("IsFileExists(): invalid, empty file path is given."); 34 | } 35 | 36 | struct stat sb; 37 | auto rtn = stat(filepath.c_str(), &sb); 38 | return (0 == rtn); 39 | } 40 | 41 | bool 42 | TestUtil::RemoveFileIfExists( 43 | const std::string & filepath 44 | ) 45 | { 46 | if (!TestUtil::IsFileExists(filepath)) { 47 | return false; 48 | } 49 | if (unlink(filepath.c_str())) { 50 | throw std::system_error(errno, std::system_category(), "unlink(" + filepath + ")"); 51 | } 52 | return true; 53 | } 54 | 55 | std::string 56 | TestUtil::GetTimeNow() 57 | { 58 | struct timeval tv; 59 | (void) gettimeofday(&tv, 0); 60 | 61 | struct tm zulu; 62 | (void) gmtime_r(&(tv.tv_sec), &zulu); 63 | 64 | char buf[128]; 65 | auto rtn = strftime(buf, sizeof(buf), "%Y-%m-%dT%H:%M:%S", &zulu); 66 | if (0 == rtn) { 67 | throw std::runtime_error("strftime() failed"); 68 | } 69 | auto usec = static_cast(tv.tv_usec); 70 | std::ostringstream strm; 71 | strm << buf << "." << std::setfill('0') << std::setw(6) << usec << "0Z"; 72 | return strm.str(); 73 | } 74 | 75 | std::string 76 | TestUtil::GetErrnoStr(int errnum) 77 | { 78 | char errorstr[256]; 79 | char* errRC = strerror_r(errnum, errorstr, sizeof(errorstr)); 80 | return std::string(errRC); 81 | } 82 | 83 | size_t 84 | TestUtil::GetFileSize( 85 | const std::string & filename 86 | ) 87 | { 88 | if (filename.empty()) { 89 | throw std::invalid_argument("filename arg cannot be empty string"); 90 | } 91 | 92 | std::ifstream fin(filename); 93 | if (!fin.is_open()) { 94 | throw std::runtime_error("cannot open file '" + filename + "'"); 95 | } 96 | fin.seekg(0, fin.end); 97 | auto sz = fin.tellg(); 98 | fin.close(); 99 | return sz; 100 | } 101 | 102 | bool 103 | TestUtil::SearchStrings( 104 | const std::string& filename, 105 | std::streampos startPos, 106 | std::unordered_set& searchSet 107 | ) 108 | { 109 | if (filename.empty()) { 110 | throw std::invalid_argument("SearchStrings: filename arg cannot be empty string"); 111 | } 112 | if (searchSet.empty()) { 113 | throw std::invalid_argument("SearchStrings: searchSet cannot be empty"); 114 | } 115 | 116 | std::ifstream fin(filename); 117 | if (!fin.is_open()) { 118 | throw std::runtime_error("SearchStrings: cannot open file '" + filename + "'"); 119 | } 120 | 121 | fin.seekg(startPos); 122 | 123 | std::string line; 124 | while(getline(fin, line)) { 125 | std::string foundStr; 126 | bool found = false; 127 | for (const auto & str : searchSet) { 128 | if (line.find(str) != std::string::npos) { 129 | foundStr = str; 130 | found = true; 131 | break; 132 | } 133 | } 134 | 135 | if (found) { 136 | searchSet.erase(foundStr); 137 | } 138 | } 139 | fin.close(); 140 | 141 | return (searchSet.empty() == true); 142 | } 143 | -------------------------------------------------------------------------------- /src/test/outmdsd/testutil.h: -------------------------------------------------------------------------------- 1 | #pragma once 2 | #ifndef __TESTUTIL_H__ 3 | #define __TESTUTIL_H__ 4 | 5 | #include 6 | #include 7 | #include 8 | #include 9 | #include 10 | #include 11 | 12 | namespace TestUtil 13 | { 14 | inline std::string EndOfTest() { return "ENDOFTEST"; } 15 | 16 | inline std::string CreateMsg(int index) 17 | { 18 | return std::string("TestMsg-") + std::to_string(index); 19 | } 20 | 21 | std::string GetCurrDir(); 22 | 23 | bool IsFileExists(const std::string & filepath); 24 | bool RemoveFileIfExists(const std::string & filepath); 25 | 26 | std::string GetTimeNow(); 27 | std::string GetErrnoStr(int errnum); 28 | 29 | // wait for a task for a max amount of time. then check its status 30 | // Return true if finished within timeoutMS, false if otherwise. 31 | template 32 | bool WaitForTask(std::future& task, uint32_t timeoutMS) 33 | { 34 | auto status = task.wait_for(std::chrono::milliseconds(timeoutMS)); 35 | return (std::future_status::ready == status); 36 | } 37 | 38 | // Get the size of a given file 39 | // Throw exception if filename is empty string, or failed to open file. 40 | size_t GetFileSize(const std::string & filename); 41 | 42 | // Search a set of strings from a text file starting from 'startPos' 43 | // Throw exception if filename is empty string, or if searchSet if empty, 44 | // or if given file cannot be opened. 45 | // Return true if all strings are found, return false if any of them not found. 46 | // The items found are removed from searchSet. 47 | bool SearchStrings(const std::string& filename, 48 | std::streampos startPos, 49 | std::unordered_set& searchSet); 50 | } 51 | 52 | #endif // __TESTUTIL_H__ 53 | -------------------------------------------------------------------------------- /src/test/outmdsd/utmain.cc: -------------------------------------------------------------------------------- 1 | // NOTE: #define BOOST_TEST_MODULE xxx 2 | // must be defined before you include unit_test.hpp. 3 | #define BOOST_TEST_MODULE "outmdsdut" 4 | #include 5 | #include "Trace.h" 6 | #include "FileTracer.h" 7 | #include "TraceMacros.h" 8 | #include "testutil.h" 9 | 10 | using namespace EndpointLog; 11 | 12 | 13 | class TestInitializer { 14 | public: 15 | TestInitializer() 16 | { 17 | std::ostringstream strm; 18 | strm << TestUtil::GetCurrDir() << "/outmdsd.log"; 19 | auto logfile = strm.str(); 20 | Trace::SetTracer(new FileTracer(logfile, true)); 21 | Trace::SetTraceLevel(TraceLevel::Trace); 22 | Log(TraceLevel::Info, "\n\n============= start new outmdsd test =========="); 23 | } 24 | ~TestInitializer() = default; 25 | }; 26 | 27 | BOOST_GLOBAL_FIXTURE(TestInitializer); 28 | -------------------------------------------------------------------------------- /src/test/outmdsdrb/CMakeLists.txt: -------------------------------------------------------------------------------- 1 | file(GLOB RUBYFILES "*.rb") 2 | 3 | file(COPY 4 | ${RUBYFILES} 5 | DESTINATION 6 | ${CMAKE_BINARY_DIR}/release/tests) 7 | -------------------------------------------------------------------------------- /src/test/outmdsdrb/mdsdlogger.rb: -------------------------------------------------------------------------------- 1 | #!/opt/microsoft/omsagent/ruby/bin/ruby 2 | 3 | require '../lib/Liboutmdsdrb' 4 | 5 | def Usage() 6 | progname = __FILE__ 7 | puts "#{progname} " 8 | end 9 | 10 | def RemoveFile(filename) 11 | if File.exists?(filename) 12 | File.delete(filename) 13 | end 14 | end 15 | 16 | def GetSchemaVal(n) 17 | str = '1,[["timestr","FT_STRING"],["intnum","FT_INT64"],["string","FT_STRING"],["floatnum","FT_DOUBLE"]],["2016-11-04 09:17:01 -0700",' 18 | str << n.to_s << ',"costco",1.234]' 19 | return str 20 | end 21 | 22 | def GetLogger(logfile, socketFile) 23 | RemoveFile(logfile) 24 | 25 | Liboutmdsdrb::InitLogger(logfile, true) 26 | Liboutmdsdrb::SetLogLevel("trace") 27 | eplog = Liboutmdsdrb::SocketLogger.new(socketFile, 100, 1000) 28 | return eplog 29 | end 30 | 31 | def RunBasicTest(nmsgs) 32 | eplog = GetLogger("/tmp/outmdsd.log", "/tmp/default_djson.socket") 33 | 34 | nmsgs.times { |k| 35 | eplog.SendDjson("testsource", "test schemadata " + (k+1).to_s) 36 | sleep(0.01) 37 | } 38 | nTags = eplog.GetNumTagsRead() 39 | puts "Total ack tags read: #{nTags}" 40 | end 41 | 42 | def RunMdsdTest(nmsgs) 43 | eplog = GetLogger("/tmp/outmdsd.log", "/var/run/mdsd/default_djson.socket") 44 | nmsgs.times { |k| 45 | msg = GetSchemaVal(k+1) 46 | puts msg 47 | eplog.SendDjson("fluentdtest", msg) 48 | sleep(0.01) 49 | } 50 | nTags = eplog.GetNumTagsRead() 51 | puts "Total ack tags read: #{nTags}" 52 | end 53 | 54 | if __FILE__ == $0 55 | unless ARGV.length == 1 56 | Usage() 57 | exit 58 | end 59 | 60 | nmsgs = ARGV[0].to_i 61 | #RunBasicTest(nmsgs) 62 | RunMdsdTest(nmsgs) 63 | end 64 | -------------------------------------------------------------------------------- /src/test/parseglibc.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | # This script is to parse glibc-based binary files and print out 4 | # symbols whose GLIBC versions are higher than given version. 5 | # Report error if any such symbol is found. 6 | 7 | import argparse 8 | import glob 9 | import os 10 | import sys 11 | import time 12 | 13 | totalErrors = 0 14 | 15 | 16 | def LogError(msg): 17 | global totalErrors 18 | totalErrors = totalErrors + 1 19 | msg2 = "%s: Error: %s" % (sys.argv[0], msg) 20 | print(msg2) 21 | 22 | 23 | def LogInfo(msg): 24 | print(msg) 25 | 26 | 27 | def ParseCmdLine(): 28 | parser = argparse.ArgumentParser(sys.argv[0]) 29 | parser.add_argument("-d", "--dir", type=str, required=False, 30 | help="directory where all its files are parsed.") 31 | parser.add_argument("-f", "--filepath", type=str, required=False, help="binary filepath.") 32 | 33 | parser.add_argument("-v", "--glibcver", type=str, required=True, help="max GLIBC ver. ex: 2.14") 34 | args = parser.parse_args() 35 | 36 | if not args.dir and not args.filepath: 37 | LogError("either '-d' or '-f' is required.") 38 | 39 | return args 40 | 41 | 42 | def GetFilesToParse(filepath, dirname): 43 | files = [] 44 | if filepath: 45 | if not os.path.isfile(filepath): 46 | LogError("%s is not a regular file." % (filepath)) 47 | else: 48 | files.append(filepath) 49 | 50 | elif dirname: 51 | if not os.path.isdir(dirname): 52 | LogError("%s is not a directory." % (dirname)) 53 | else: 54 | files = GetAllFiles(dirname) 55 | 56 | return files 57 | 58 | 59 | # Get all files in a directory. This doesn't include subdirectories and symbolic links. 60 | def GetAllFiles(dirname): 61 | if not dirname: 62 | return [] 63 | 64 | filepattern = dirname + "/*" 65 | filedirs = glob.glob(filepattern) 66 | files = [] 67 | for f in filedirs: 68 | if os.path.isfile(f) and (not os.path.islink(f)): 69 | files.append(f) 70 | return files 71 | 72 | 73 | # Get symbol file by running 'nm' 74 | def GetSymbols(filepath): 75 | outputfile = "testfile-" + str(time.time()) + ".txt" 76 | cmdline = "nm " + filepath + " 1>" + outputfile + " 2>&1" 77 | errCode = os.system(cmdline) 78 | if errCode != 0: 79 | LogError("cmd: '%s' failed with error %d" % (cmdline, errCode)) 80 | return "" 81 | return outputfile 82 | 83 | 84 | # Parse symbol file created by 'nm' 85 | def ParseSymbols(symbolfile, glibcver): 86 | with open(symbolfile, "r") as fh: 87 | lines = fh.readlines() 88 | 89 | for line in lines: 90 | if "@GLIBC_" in line: 91 | line = line.strip() 92 | ParseLine(line, glibcver) 93 | 94 | 95 | # Parse one line to check for higher GLIBC version. 96 | # Report error if found 97 | def ParseLine(line, glibcver): 98 | global totalErrors 99 | items = line.split("GLIBC_") 100 | if len(items) != 2: 101 | LogError("unexpected symbol: %s" % (line)) 102 | else: 103 | if CompareVer(items[1], glibcver): 104 | totalErrors = totalErrors + 1 105 | LogInfo(line) 106 | 107 | 108 | # Return True if ver1 > ver2. 109 | # Return False otherwise. 110 | def CompareVer(ver1, ver2): 111 | v1list = ver1.split(".") 112 | v2list = ver2.split(".") 113 | n = min(len(v1list), len(v2list)) 114 | for i in range(n): 115 | x = int(v1list[i]) 116 | y = int(v2list[i]) 117 | if x > y: 118 | return True 119 | elif x < y: 120 | return False 121 | 122 | if len(v1list) > len(v2list): 123 | return True 124 | 125 | return False 126 | 127 | 128 | def RunTest(filepath, dirname, glibcver): 129 | LogInfo("Parse GLIBC versions ...") 130 | files = GetFilesToParse(filepath, dirname) 131 | 132 | if len(files) == 0: 133 | LogError("no file to parse. Abort.") 134 | return 135 | 136 | for binfile in files: 137 | LogInfo("\nStart to parse file '%s' ..." % (binfile)) 138 | symbolfile = GetSymbols(binfile) 139 | if symbolfile: 140 | ParseSymbols(symbolfile, glibcver) 141 | os.remove(symbolfile) 142 | 143 | if totalErrors == 0: 144 | LogInfo("\nNo error is found. Test passed successfully.") 145 | else: 146 | LogInfo("\nTest failed. Total errors found: %d" % (totalErrors)) 147 | 148 | 149 | if __name__ == "__main__": 150 | args = ParseCmdLine() 151 | RunTest(args.filepath, args.dir, args.glibcver) 152 | sys.exit(totalErrors) 153 | -------------------------------------------------------------------------------- /src/test/runtests.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # This script will run all tests. 4 | update-alternatives --install /usr/bin/ld ld /usr/bin/ld.gold 60 5 | TotalErrors=0 6 | 7 | # Valid ${Target} values are: td (for treasure data), oms (for OMSAgent) 8 | Target=td 9 | 10 | # The bin directory where ruby locates. 11 | RubyDir= 12 | 13 | Usage() 14 | { 15 | echo "Usage: $0 [-t target] [-h]" 16 | echo " -h: help." 17 | echo " -t: fluentd target ('td' or 'oms')" 18 | } 19 | 20 | FindRubyPath() 21 | { 22 | if [ ${Target} == "td" ]; then 23 | RubyDir=/opt/td-agent/embedded/bin 24 | elif [ ${Target} == "oms" ]; then 25 | RubyDir=/opt/microsoft/omsagent/ruby/bin 26 | fi 27 | } 28 | 29 | # Syslog is required for some ut_outmdsd tests. 30 | # In docker image, rsyslogd is not started and may not 31 | # configured properly. So handle them in this function. 32 | # 33 | # In non-docker machine, assume the proper syslog is 34 | # configured. If not, user may get test error, and test 35 | # error should contain details on what's wrong. 36 | ConfigSyslog() 37 | { 38 | grep docker /proc/self/cgroup > /dev/null 39 | if [ $? == 0 ]; then 40 | echo Configure syslog ... 41 | sudo chown syslog.syslog /var/log/ 42 | sudo touch /var/log/syslog 43 | sudo chmod a+r /var/log/syslog 44 | sudo chown syslog.adm /var/log/syslog 45 | sudo rsyslogd > /dev/null 2>&1 46 | fi 47 | } 48 | 49 | RunOutmdsdTest() 50 | { 51 | ConfigSyslog 52 | 53 | echo Start ut_outmdsd unittest ... 54 | ./ut_outmdsd --log_level=all --report_format=HRF --report_level=detailed --report_sink=outmdsd_report.log > outmdsd_test.log 2>&1 55 | if [ $? != 0 ]; then 56 | let TotalErrors+=1 57 | echo Error: ut_outmdsd unittest failed 58 | fi 59 | } 60 | 61 | # The ruby code part of the plugin 62 | RunPluginRubyTest() 63 | { 64 | ${RubyDir}/gem unpack ../gem/fluent-plugin-mdsd*.gem 65 | pushd fluent-plugin-mdsd* 66 | 67 | echo RunPluginRubyTest: ${RubyDir}/ruby ${RubyDir}/rake test 68 | ${RubyDir}/ruby ${RubyDir}/rake test 69 | if [ $? != 0 ]; then 70 | let TotalErrors+=1 71 | echo Error: RunPluginRubyTest failed 72 | fi 73 | 74 | popd 75 | } 76 | 77 | RunGemInstallTest() 78 | { 79 | echo RunGemInstallTest ... 80 | sudo ${RubyDir}/fluent-gem install ../gem/fluent-plugin-mdsd*.gem 81 | if [ $? != 0 ]; then 82 | let TotalErrors+=1 83 | echo Error: RunGemInstallTest failed 84 | fi 85 | } 86 | 87 | # zip all related testlog files for debugging 88 | ArchiveTestLogs() 89 | { 90 | echo ArchiveTestLogs ... 91 | rm -f testresults.tar.gz 92 | tar --ignore-failed-read -czf testresults.tar.gz *.log 93 | } 94 | 95 | 96 | if [ "$#" == "0" ]; then 97 | Usage 98 | exit 1 99 | fi 100 | 101 | args=`getopt ht: $*` 102 | 103 | if [ $? != 0 ]; then 104 | Usage 105 | exit 1 106 | fi 107 | set -- $args 108 | 109 | for i; do 110 | case "$i" in 111 | -h) 112 | Usage 113 | exit 0 114 | shift ;; 115 | -t) 116 | Target=$2 117 | shift; shift ;; 118 | --) shift; break ;; 119 | esac 120 | done 121 | 122 | if [ "${Target}" != "td" ] && [ "${Target}" != "oms" ]; then 123 | echo "Error: invalid -t value. Expected: 'td' or 'oms'" 124 | exit 1 125 | fi 126 | 127 | FindRubyPath 128 | 129 | RunOutmdsdTest 130 | RunPluginRubyTest 131 | RunGemInstallTest 132 | 133 | ArchiveTestLogs 134 | 135 | echo Finished all tests at `date`. Total failed test suites = ${TotalErrors} 136 | 137 | if [ ${TotalErrors} != 0 ]; then 138 | exit 1 139 | else 140 | exit 0 141 | fi 142 | --------------------------------------------------------------------------------