├── .filenesting.json ├── .gitattributes ├── .github └── workflows │ ├── OneSTools.EventLog (.NET 5).yml │ └── OneSTools.EventLog.Exporter (.NET 5).yml ├── .gitignore ├── .vscode ├── launch.json └── tasks.json ├── LICENSE ├── Nuget └── onestools_icon_nuget.png ├── OneSTools.EventLog.Exporter.Core ├── ClickHouse │ └── ClickHouseStorage.cs ├── ElasticSearch │ ├── AuthenticationType.cs │ ├── ElasticSearchHost.cs │ ├── ElasticSearchStorage.cs │ └── ElasticSearchStorageSettings.cs ├── EventLogExporter.cs ├── EventLogExporterSettings.cs ├── EventLogPosition.cs ├── FileLogger.cs ├── FileLoggerProvider.cs ├── IEventLogStorage.cs ├── ILoggerBuilderExtensions.cs ├── OneSTools.EventLog.Exporter.Core.csproj └── StorageType.cs ├── OneSTools.EventLog.Exporter.Manager ├── ClstEventArgs.cs ├── ClstFolder.cs ├── ClstWatcher.cs ├── ExportersManager.cs ├── OneSTools.EventLog.Exporter.Manager.csproj ├── Program.cs ├── Properties │ └── launchSettings.json ├── TemplateItem.cs ├── appsettings.Development.json └── appsettings.json ├── OneSTools.EventLog.Exporter ├── EventLogExporterService.cs ├── OneSTools.EventLog.Exporter.csproj ├── Program.cs ├── Properties │ └── launchSettings.json ├── appsettings.Development.json └── appsettings.json ├── OneSTools.EventLog.sln ├── OneSTools.EventLog ├── DateTimeZoneExtensions.cs ├── EventLogItem.cs ├── EventLogReader.cs ├── EventLogReaderSettings.cs ├── EventLogReaderTimeoutException.cs ├── IEventLogItem.cs ├── LgfReader.cs ├── LgpReader.cs ├── ObjectType.cs ├── OneSTools.EventLog.csproj ├── README.md └── StreamReaderExtensions.cs └── README.md /.filenesting.json: -------------------------------------------------------------------------------- 1 | { 2 | "help":"https://go.microsoft.com/fwlink/?linkid=866610" 3 | } -------------------------------------------------------------------------------- /.gitattributes: -------------------------------------------------------------------------------- 1 | ############################################################################### 2 | # Set default behavior to automatically normalize line endings. 3 | ############################################################################### 4 | * text=auto 5 | 6 | ############################################################################### 7 | # Set default behavior for command prompt diff. 8 | # 9 | # This is need for earlier builds of msysgit that does not have it on by 10 | # default for csharp files. 11 | # Note: This is only used by command line 12 | ############################################################################### 13 | #*.cs diff=csharp 14 | 15 | ############################################################################### 16 | # Set the merge driver for project and solution files 17 | # 18 | # Merging from the command prompt will add diff markers to the files if there 19 | # are conflicts (Merging from VS is not affected by the settings below, in VS 20 | # the diff markers are never inserted). Diff markers may cause the following 21 | # file extensions to fail to load in VS. An alternative would be to treat 22 | # these files as binary and thus will always conflict and require user 23 | # intervention with every merge. To do so, just uncomment the entries below 24 | ############################################################################### 25 | #*.sln merge=binary 26 | #*.csproj merge=binary 27 | #*.vbproj merge=binary 28 | #*.vcxproj merge=binary 29 | #*.vcproj merge=binary 30 | #*.dbproj merge=binary 31 | #*.fsproj merge=binary 32 | #*.lsproj merge=binary 33 | #*.wixproj merge=binary 34 | #*.modelproj merge=binary 35 | #*.sqlproj merge=binary 36 | #*.wwaproj merge=binary 37 | 38 | ############################################################################### 39 | # behavior for image files 40 | # 41 | # image files are treated as binary by default. 42 | ############################################################################### 43 | #*.jpg binary 44 | #*.png binary 45 | #*.gif binary 46 | 47 | ############################################################################### 48 | # diff behavior for common document formats 49 | # 50 | # Convert binary document formats to text before diffing them. This feature 51 | # is only available from the command line. Turn it on by uncommenting the 52 | # entries below. 53 | ############################################################################### 54 | #*.doc diff=astextplain 55 | #*.DOC diff=astextplain 56 | #*.docx diff=astextplain 57 | #*.DOCX diff=astextplain 58 | #*.dot diff=astextplain 59 | #*.DOT diff=astextplain 60 | #*.pdf diff=astextplain 61 | #*.PDF diff=astextplain 62 | #*.rtf diff=astextplain 63 | #*.RTF diff=astextplain 64 | -------------------------------------------------------------------------------- /.github/workflows/OneSTools.EventLog (.NET 5).yml: -------------------------------------------------------------------------------- 1 | name: EventLog .NET 5 2 | 3 | on: 4 | push: 5 | branches: [ master ] 6 | pull_request: 7 | branches: [ master ] 8 | 9 | jobs: 10 | build: 11 | 12 | runs-on: ubuntu-latest 13 | 14 | steps: 15 | - uses: actions/checkout@v2 16 | - name: Setup .NET 17 | uses: actions/setup-dotnet@v1 18 | with: 19 | dotnet-version: 5.0.100 20 | - name: Install dependencies 21 | working-directory: ./OneSTools.EventLog 22 | run: dotnet restore 23 | - name: Build 24 | working-directory: ./OneSTools.EventLog 25 | run: dotnet build --configuration Release --no-restore 26 | - name: Test 27 | working-directory: ./OneSTools.EventLog 28 | run: dotnet test --no-restore --verbosity normal 29 | -------------------------------------------------------------------------------- /.github/workflows/OneSTools.EventLog.Exporter (.NET 5).yml: -------------------------------------------------------------------------------- 1 | name: EventLogExporter .NET 5 2 | 3 | on: 4 | push: 5 | branches: [ master ] 6 | pull_request: 7 | branches: [ master ] 8 | 9 | jobs: 10 | build: 11 | 12 | runs-on: ubuntu-latest 13 | 14 | steps: 15 | - uses: actions/checkout@v2 16 | - name: Setup .NET 17 | uses: actions/setup-dotnet@v1 18 | with: 19 | dotnet-version: 5.0.100 20 | - name: Install dependencies 21 | working-directory: ./OneSTools.EventLog.Exporter 22 | run: dotnet restore 23 | - name: Build 24 | working-directory: ./OneSTools.EventLog.Exporter 25 | run: dotnet build --configuration Release --no-restore 26 | - name: Test 27 | working-directory: ./OneSTools.EventLog.Exporter 28 | run: dotnet test --no-restore --verbosity normal 29 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore 5 | 6 | .vscode/ 7 | 8 | # Project-specific files 9 | log.txt 10 | 11 | # User-specific files 12 | *.rsuser 13 | *.suo 14 | *.user 15 | *.userosscache 16 | *.sln.docstates 17 | 18 | # User-specific files (MonoDevelop/Xamarin Studio) 19 | *.userprefs 20 | 21 | # Build results 22 | [Dd]ebug/ 23 | [Dd]ebugPublic/ 24 | [Rr]elease/ 25 | [Rr]eleases/ 26 | x64/ 27 | x86/ 28 | [Aa][Rr][Mm]/ 29 | [Aa][Rr][Mm]64/ 30 | bld/ 31 | [Bb]in/ 32 | [Oo]bj/ 33 | [Ll]og/ 34 | 35 | # Visual Studio 2015/2017 cache/options directory 36 | .vs/ 37 | # Uncomment if you have tasks that create the project's static files in wwwroot 38 | #wwwroot/ 39 | 40 | # Visual Studio 2017 auto generated files 41 | Generated\ Files/ 42 | 43 | # MSTest test Results 44 | [Tt]est[Rr]esult*/ 45 | [Bb]uild[Ll]og.* 46 | 47 | # NUNIT 48 | *.VisualState.xml 49 | TestResult.xml 50 | 51 | # Build Results of an ATL Project 52 | [Dd]ebugPS/ 53 | [Rr]eleasePS/ 54 | dlldata.c 55 | 56 | # Benchmark Results 57 | BenchmarkDotNet.Artifacts/ 58 | 59 | # .NET Core 60 | project.lock.json 61 | project.fragment.lock.json 62 | artifacts/ 63 | 64 | # StyleCop 65 | StyleCopReport.xml 66 | 67 | # Files built by Visual Studio 68 | *_i.c 69 | *_p.c 70 | *_h.h 71 | *.ilk 72 | *.meta 73 | *.obj 74 | *.iobj 75 | *.pch 76 | *.pdb 77 | *.ipdb 78 | *.pgc 79 | *.pgd 80 | *.rsp 81 | *.sbr 82 | *.tlb 83 | *.tli 84 | *.tlh 85 | *.tmp 86 | *.tmp_proj 87 | *_wpftmp.csproj 88 | *.log 89 | *.vspscc 90 | *.vssscc 91 | .builds 92 | *.pidb 93 | *.svclog 94 | *.scc 95 | 96 | # Chutzpah Test files 97 | _Chutzpah* 98 | 99 | # Visual C++ cache files 100 | ipch/ 101 | *.aps 102 | *.ncb 103 | *.opendb 104 | *.opensdf 105 | *.sdf 106 | *.cachefile 107 | *.VC.db 108 | *.VC.VC.opendb 109 | 110 | # Visual Studio profiler 111 | *.psess 112 | *.vsp 113 | *.vspx 114 | *.sap 115 | 116 | # Visual Studio Trace Files 117 | *.e2e 118 | 119 | # TFS 2012 Local Workspace 120 | $tf/ 121 | 122 | # Guidance Automation Toolkit 123 | *.gpState 124 | 125 | # ReSharper is a .NET coding add-in 126 | _ReSharper*/ 127 | *.[Rr]e[Ss]harper 128 | *.DotSettings.user 129 | 130 | # JustCode is a .NET coding add-in 131 | .JustCode 132 | 133 | # TeamCity is a build add-in 134 | _TeamCity* 135 | 136 | # DotCover is a Code Coverage Tool 137 | *.dotCover 138 | 139 | # AxoCover is a Code Coverage Tool 140 | .axoCover/* 141 | !.axoCover/settings.json 142 | 143 | # Visual Studio code coverage results 144 | *.coverage 145 | *.coveragexml 146 | 147 | # NCrunch 148 | _NCrunch_* 149 | .*crunch*.local.xml 150 | nCrunchTemp_* 151 | 152 | # MightyMoose 153 | *.mm.* 154 | AutoTest.Net/ 155 | 156 | # Web workbench (sass) 157 | .sass-cache/ 158 | 159 | # Installshield output folder 160 | [Ee]xpress/ 161 | 162 | # DocProject is a documentation generator add-in 163 | DocProject/buildhelp/ 164 | DocProject/Help/*.HxT 165 | DocProject/Help/*.HxC 166 | DocProject/Help/*.hhc 167 | DocProject/Help/*.hhk 168 | DocProject/Help/*.hhp 169 | DocProject/Help/Html2 170 | DocProject/Help/html 171 | 172 | # Click-Once directory 173 | publish/ 174 | 175 | # Publish Web Output 176 | *.[Pp]ublish.xml 177 | *.azurePubxml 178 | # Note: Comment the next line if you want to checkin your web deploy settings, 179 | # but database connection strings (with potential passwords) will be unencrypted 180 | *.pubxml 181 | *.publishproj 182 | 183 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 184 | # checkin your Azure Web App publish settings, but sensitive information contained 185 | # in these scripts will be unencrypted 186 | PublishScripts/ 187 | 188 | # NuGet Packages 189 | *.nupkg 190 | # The packages folder can be ignored because of Package Restore 191 | **/[Pp]ackages/* 192 | # except build/, which is used as an MSBuild target. 193 | !**/[Pp]ackages/build/ 194 | # Uncomment if necessary however generally it will be regenerated when needed 195 | #!**/[Pp]ackages/repositories.config 196 | # NuGet v3's project.json files produces more ignorable files 197 | *.nuget.props 198 | *.nuget.targets 199 | 200 | # Microsoft Azure Build Output 201 | csx/ 202 | *.build.csdef 203 | 204 | # Microsoft Azure Emulator 205 | ecf/ 206 | rcf/ 207 | 208 | # Windows Store app package directories and files 209 | AppPackages/ 210 | BundleArtifacts/ 211 | Package.StoreAssociation.xml 212 | _pkginfo.txt 213 | *.appx 214 | 215 | # Visual Studio cache files 216 | # files ending in .cache can be ignored 217 | *.[Cc]ache 218 | # but keep track of directories ending in .cache 219 | !?*.[Cc]ache/ 220 | 221 | # Others 222 | ClientBin/ 223 | ~$* 224 | *~ 225 | *.dbmdl 226 | *.dbproj.schemaview 227 | *.jfm 228 | *.pfx 229 | *.publishsettings 230 | orleans.codegen.cs 231 | 232 | # Including strong name files can present a security risk 233 | # (https://github.com/github/gitignore/pull/2483#issue-259490424) 234 | #*.snk 235 | 236 | # Since there are multiple workflows, uncomment next line to ignore bower_components 237 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 238 | #bower_components/ 239 | 240 | # RIA/Silverlight projects 241 | Generated_Code/ 242 | 243 | # Backup & report files from converting an old project file 244 | # to a newer Visual Studio version. Backup files are not needed, 245 | # because we have git ;-) 246 | _UpgradeReport_Files/ 247 | Backup*/ 248 | UpgradeLog*.XML 249 | UpgradeLog*.htm 250 | ServiceFabricBackup/ 251 | *.rptproj.bak 252 | 253 | # SQL Server files 254 | *.mdf 255 | *.ldf 256 | *.ndf 257 | 258 | # Business Intelligence projects 259 | *.rdl.data 260 | *.bim.layout 261 | *.bim_*.settings 262 | *.rptproj.rsuser 263 | *- Backup*.rdl 264 | 265 | # Microsoft Fakes 266 | FakesAssemblies/ 267 | 268 | # GhostDoc plugin setting file 269 | *.GhostDoc.xml 270 | 271 | # Node.js Tools for Visual Studio 272 | .ntvs_analysis.dat 273 | node_modules/ 274 | 275 | # Visual Studio 6 build log 276 | *.plg 277 | 278 | # Visual Studio 6 workspace options file 279 | *.opt 280 | 281 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 282 | *.vbw 283 | 284 | # Visual Studio LightSwitch build output 285 | **/*.HTMLClient/GeneratedArtifacts 286 | **/*.DesktopClient/GeneratedArtifacts 287 | **/*.DesktopClient/ModelManifest.xml 288 | **/*.Server/GeneratedArtifacts 289 | **/*.Server/ModelManifest.xml 290 | _Pvt_Extensions 291 | 292 | # Paket dependency manager 293 | .paket/paket.exe 294 | paket-files/ 295 | 296 | # FAKE - F# Make 297 | .fake/ 298 | 299 | # JetBrains Rider 300 | .idea/ 301 | *.sln.iml 302 | 303 | # CodeRush personal settings 304 | .cr/personal 305 | 306 | # Python Tools for Visual Studio (PTVS) 307 | __pycache__/ 308 | *.pyc 309 | 310 | # Cake - Uncomment if you are using it 311 | # tools/** 312 | # !tools/packages.config 313 | 314 | # Tabs Studio 315 | *.tss 316 | 317 | # Telerik's JustMock configuration file 318 | *.jmconfig 319 | 320 | # BizTalk build output 321 | *.btp.cs 322 | *.btm.cs 323 | *.odx.cs 324 | *.xsd.cs 325 | 326 | # OpenCover UI analysis results 327 | OpenCover/ 328 | 329 | # Azure Stream Analytics local run output 330 | ASALocalRun/ 331 | 332 | # MSBuild Binary and Structured Log 333 | *.binlog 334 | 335 | # NVidia Nsight GPU debugger configuration file 336 | *.nvuser 337 | 338 | # MFractors (Xamarin productivity tool) working folder 339 | .mfractor/ 340 | 341 | # Local History for Visual Studio 342 | .localhistory/ 343 | 344 | # BeatPulse healthcheck temp database 345 | healthchecksdb 346 | -------------------------------------------------------------------------------- /.vscode/launch.json: -------------------------------------------------------------------------------- 1 | { 2 | // Use IntelliSense to learn about possible attributes. 3 | // Hover to view descriptions of existing attributes. 4 | // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 5 | "version": "0.2.0", 6 | "configurations": [ 7 | { 8 | "name": "Launch Exporters Manager", 9 | "type": "coreclr", 10 | "request": "launch", 11 | "preLaunchTask": "build-manager", 12 | "program": "${workspaceFolder}/Build/Debug/net5.0/EventLogExportersManager.dll", 13 | "args": [], 14 | "cwd": "${workspaceFolder}/Build/Debug/net5.0", 15 | "console": "internalConsole", 16 | "stopAtEntry": false 17 | }, 18 | { 19 | "name": "Launch Exporter", 20 | "type": "coreclr", 21 | "request": "launch", 22 | "preLaunchTask": "build-exporter", 23 | "program": "${workspaceFolder}/Build/Debug/net5.0/EventLogExporter.dll", 24 | "args": [], 25 | "cwd": "${workspaceFolder}/Build/Debug/net5.0", 26 | "console": "internalConsole", 27 | "stopAtEntry": false 28 | }, 29 | { 30 | "name": ".NET Core Attach", 31 | "type": "coreclr", 32 | "request": "attach" 33 | } 34 | ] 35 | } -------------------------------------------------------------------------------- /.vscode/tasks.json: -------------------------------------------------------------------------------- 1 | { 2 | "version": "2.0.0", 3 | "tasks": [ 4 | { 5 | "label": "build-manager", 6 | "command": "dotnet", 7 | "type": "process", 8 | "args": [ 9 | "build", 10 | "${workspaceFolder}/OneSTools.EventLog.Exporter.Manager/OneSTools.EventLog.Exporter.Manager.csproj", 11 | "/property:GenerateFullPaths=true", 12 | "/consoleloggerparameters:NoSummary" 13 | ], 14 | "problemMatcher": "$msCompile", 15 | "dependsOn": [ 16 | "build-exporter-core" 17 | ], 18 | "group": { 19 | "kind": "build", 20 | "isDefault": true 21 | } 22 | }, 23 | { 24 | "label": "build-exporter", 25 | "command": "dotnet", 26 | "type": "process", 27 | "args": [ 28 | "build", 29 | "${workspaceFolder}/OneSTools.EventLog.Exporter/OneSTools.EventLog.Exporter.csproj", 30 | "/property:GenerateFullPaths=true", 31 | "/consoleloggerparameters:NoSummary" 32 | ], 33 | "problemMatcher": "$msCompile", 34 | "dependsOn": [ 35 | "build-exporter-core" 36 | ], 37 | "group": { 38 | "kind": "build", 39 | } 40 | }, 41 | { 42 | "label": "build-exporter-core", 43 | "command": "dotnet", 44 | "type": "process", 45 | "args": [ 46 | "build", 47 | "${workspaceFolder}/OneSTools.EventLog.Exporter.Core/OneSTools.EventLog.Exporter.Core.csproj", 48 | "/property:GenerateFullPaths=true", 49 | "/consoleloggerparameters:NoSummary" 50 | ], 51 | "problemMatcher": "$msCompile" 52 | } 53 | ] 54 | } -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2020 Akpaev Evgeny 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /Nuget/onestools_icon_nuget.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/akpaevj/OneSTools.EventLog/c95a2bfbc7d9f42eb3f902ba3b55c56455669ec5/Nuget/onestools_icon_nuget.png -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/ClickHouse/ClickHouseStorage.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Text.RegularExpressions; 5 | using System.Threading; 6 | using System.Threading.Tasks; 7 | using ClickHouse.Client.ADO; 8 | using ClickHouse.Client.Copy; 9 | using Microsoft.Extensions.Configuration; 10 | using Microsoft.Extensions.Logging; 11 | 12 | namespace OneSTools.EventLog.Exporter.Core.ClickHouse 13 | { 14 | public class ClickHouseStorage : IEventLogStorage 15 | { 16 | private const string TableName = "EventLogItems"; 17 | private readonly ILogger _logger; 18 | private ClickHouseConnection _connection; 19 | private string _connectionString; 20 | private string _databaseName; 21 | 22 | public ClickHouseStorage(string connectionsString, ILogger logger = null) 23 | { 24 | _logger = logger; 25 | _connectionString = connectionsString; 26 | 27 | Init(); 28 | } 29 | 30 | public ClickHouseStorage(ILogger logger, IConfiguration configuration) 31 | { 32 | _logger = logger; 33 | _connectionString = configuration.GetValue("ClickHouse:ConnectionString", ""); 34 | 35 | Init(); 36 | } 37 | 38 | public async Task ReadEventLogPositionAsync(CancellationToken cancellationToken = default) 39 | { 40 | await CreateConnectionAsync(cancellationToken); 41 | 42 | var commandText = 43 | $"SELECT TOP 1 FileName, EndPosition, LgfEndPosition, Id FROM {TableName} ORDER BY DateTime DESC, EndPosition DESC"; 44 | 45 | await using var cmd = _connection.CreateCommand(); 46 | cmd.CommandText = commandText; 47 | 48 | await using var reader = await cmd.ExecuteReaderAsync(cancellationToken); 49 | 50 | if (await reader.ReadAsync(cancellationToken)) 51 | return new EventLogPosition(reader.GetString(0), reader.GetInt64(1), reader.GetInt64(2), 52 | reader.GetInt64(3)); 53 | return null; 54 | } 55 | 56 | public async Task WriteEventLogDataAsync(List entities, 57 | CancellationToken cancellationToken = default) 58 | { 59 | await CreateConnectionAsync(cancellationToken); 60 | 61 | using var copy = new ClickHouseBulkCopy(_connection) 62 | { 63 | DestinationTableName = TableName, 64 | BatchSize = entities.Count 65 | }; 66 | 67 | var data = entities.Select(item => new object[] 68 | { 69 | item.FileName ?? "", 70 | item.EndPosition, 71 | item.LgfEndPosition, 72 | item.Id, 73 | item.DateTime, 74 | item.TransactionStatus ?? "", 75 | item.TransactionDateTime == DateTime.MinValue ? new DateTime(1970, 1, 1) : item.TransactionDateTime, 76 | item.TransactionNumber, 77 | item.UserUuid ?? "", 78 | item.User ?? "", 79 | item.Computer ?? "", 80 | item.Application ?? "", 81 | item.Connection, 82 | item.Event ?? "", 83 | item.Severity ?? "", 84 | item.Comment ?? "", 85 | item.MetadataUuid ?? "", 86 | item.Metadata ?? "", 87 | item.Data ?? "", 88 | item.DataPresentation ?? "", 89 | item.Server ?? "", 90 | item.MainPort, 91 | item.AddPort, 92 | item.Session 93 | }).AsEnumerable(); 94 | 95 | try 96 | { 97 | await copy.WriteToServerAsync(data, cancellationToken); 98 | } 99 | catch (Exception ex) 100 | { 101 | _logger?.LogError(ex, $"Failed to write data to {_databaseName}"); 102 | throw; 103 | } 104 | 105 | _logger?.LogDebug($"{entities.Count} items were being written to {_databaseName}"); 106 | } 107 | 108 | public void Dispose() 109 | { 110 | _connection?.Dispose(); 111 | } 112 | 113 | private void Init() 114 | { 115 | if (_connectionString == string.Empty) 116 | throw new Exception("Connection string is not specified"); 117 | 118 | _databaseName = Regex.Match(_connectionString, "(?<=Database=).*?(?=(;|$))", RegexOptions.IgnoreCase).Value; 119 | _connectionString = Regex.Replace(_connectionString, "Database=.*?(;|$)", ""); 120 | 121 | if (string.IsNullOrWhiteSpace(_databaseName)) 122 | throw new Exception("Database name is not specified"); 123 | else 124 | _databaseName = FixDatabaseName(_databaseName); 125 | } 126 | 127 | private static string FixDatabaseName(string name) 128 | => Regex.Replace(name, @"(?:\W|-)", "_", RegexOptions.Compiled); 129 | 130 | private async Task CreateConnectionAsync(CancellationToken cancellationToken = default) 131 | { 132 | if (_connection is null) 133 | { 134 | _connection = new ClickHouseConnection(_connectionString); 135 | await _connection.OpenAsync(cancellationToken); 136 | 137 | await CreateEventLogItemsDatabaseAsync(cancellationToken); 138 | } 139 | } 140 | 141 | private async Task CreateEventLogItemsDatabaseAsync(CancellationToken cancellationToken = default) 142 | { 143 | var commandDbText = $@"CREATE DATABASE IF NOT EXISTS {_databaseName}"; 144 | 145 | await using var cmdDb = _connection.CreateCommand(); 146 | cmdDb.CommandText = commandDbText; 147 | await cmdDb.ExecuteNonQueryAsync(cancellationToken); 148 | 149 | await _connection.ChangeDatabaseAsync(_databaseName, cancellationToken); 150 | 151 | var commandText = 152 | $@"CREATE TABLE IF NOT EXISTS {TableName} 153 | ( 154 | FileName LowCardinality(String), 155 | EndPosition Int64 Codec(DoubleDelta, LZ4), 156 | LgfEndPosition Int64 Codec(DoubleDelta, LZ4), 157 | Id Int64 Codec(DoubleDelta, LZ4), 158 | DateTime DateTime('UTC') Codec(Delta, LZ4), 159 | TransactionStatus LowCardinality(String), 160 | TransactionDate DateTime('UTC') Codec(Delta, LZ4), 161 | TransactionNumber Int64 Codec(DoubleDelta, LZ4), 162 | UserUuid LowCardinality(String), 163 | User LowCardinality(String), 164 | Computer LowCardinality(String), 165 | Application LowCardinality(String), 166 | Connection Int64 Codec(DoubleDelta, LZ4), 167 | Event LowCardinality(String), 168 | Severity LowCardinality(String), 169 | Comment String Codec(ZSTD), 170 | MetadataUuid String Codec(ZSTD), 171 | Metadata LowCardinality(String), 172 | Data String Codec(ZSTD), 173 | DataPresentation String Codec(ZSTD), 174 | Server LowCardinality(String), 175 | MainPort Int32 Codec(DoubleDelta, LZ4), 176 | AddPort Int32 Codec(DoubleDelta, LZ4), 177 | Session Int64 Codec(DoubleDelta, LZ4) 178 | ) 179 | engine = MergeTree() 180 | PARTITION BY (toYYYYMM(DateTime)) 181 | ORDER BY (DateTime, EndPosition) 182 | SETTINGS index_granularity = 8192;"; 183 | 184 | await using var cmd = _connection.CreateCommand(); 185 | cmd.CommandText = commandText; 186 | await cmd.ExecuteNonQueryAsync(cancellationToken); 187 | } 188 | } 189 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/ElasticSearch/AuthenticationType.cs: -------------------------------------------------------------------------------- 1 | namespace OneSTools.EventLog.Exporter.Core.ElasticSearch 2 | { 3 | public enum AuthenticationType 4 | { 5 | None = 0, 6 | Basic = 1, 7 | ApiKey = 2 8 | } 9 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/ElasticSearch/ElasticSearchHost.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | 4 | namespace OneSTools.EventLog.Exporter.Core.ElasticSearch 5 | { 6 | public class ElasticSearchNode 7 | { 8 | public string Host { get; set; } 9 | public AuthenticationType AuthenticationType { get; set; } 10 | public string Id { get; set; } 11 | public string ApiKey { get; set; } 12 | public string UserName { get; set; } 13 | public string Password { get; set; } 14 | 15 | public override bool Equals(object obj) 16 | { 17 | return obj is ElasticSearchNode host && 18 | Host == host.Host; 19 | } 20 | 21 | public override int GetHashCode() 22 | { 23 | return HashCode.Combine(Host); 24 | } 25 | 26 | public static bool operator ==(ElasticSearchNode left, ElasticSearchNode right) 27 | { 28 | return EqualityComparer.Default.Equals(left, right); 29 | } 30 | 31 | public static bool operator !=(ElasticSearchNode left, ElasticSearchNode right) 32 | { 33 | return !(left == right); 34 | } 35 | } 36 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/ElasticSearch/ElasticSearchStorage.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Linq; 4 | using System.Text.RegularExpressions; 5 | using System.Threading; 6 | using System.Threading.Tasks; 7 | using Elasticsearch.Net; 8 | using Microsoft.Extensions.Configuration; 9 | using Microsoft.Extensions.Logging; 10 | using Nest; 11 | 12 | namespace OneSTools.EventLog.Exporter.Core.ElasticSearch 13 | { 14 | public class ElasticSearchStorage : IEventLogStorage 15 | { 16 | public static int DefaultMaximumRetries = 2; 17 | public static int DefaultMaxRetryTimeoutSec = 30; 18 | private string _eventLogItemsIndex; 19 | 20 | private readonly ILogger _logger; 21 | private readonly int _maximumRetries; 22 | private readonly TimeSpan _maxRetryTimeout; 23 | private readonly List _nodes = new List(); 24 | private readonly string _separation; 25 | private ElasticClient _client; 26 | private ElasticSearchNode _currentNode; 27 | 28 | public ElasticSearchStorage(ElasticSearchStorageSettings settings, ILogger logger = null) 29 | { 30 | _logger = logger; 31 | 32 | _nodes.AddRange(settings.Nodes); 33 | _eventLogItemsIndex = settings.Index.ToLower(); 34 | _separation = settings.Separation; 35 | _maximumRetries = settings.MaximumRetries; 36 | _maxRetryTimeout = settings.MaxRetryTimeout; 37 | 38 | CheckSettings(); 39 | } 40 | 41 | public ElasticSearchStorage(ILogger logger, IConfiguration configuration) 42 | { 43 | _logger = logger; 44 | 45 | _nodes = configuration.GetSection("ElasticSearch:Nodes").Get>(); 46 | _eventLogItemsIndex = configuration.GetValue("ElasticSearch:Index", ""); 47 | _separation = configuration.GetValue("ElasticSearch:Separation", "H"); 48 | _maximumRetries = configuration.GetValue("ElasticSearch:MaximumRetries", DefaultMaximumRetries); 49 | _maxRetryTimeout = 50 | TimeSpan.FromSeconds(configuration.GetValue("ElasticSearch:MaxRetryTimeout", 51 | DefaultMaxRetryTimeoutSec)); 52 | 53 | CheckSettings(); 54 | } 55 | 56 | public async Task ReadEventLogPositionAsync(CancellationToken cancellationToken = default) 57 | { 58 | if (_client is null) 59 | await ConnectAsync(cancellationToken); 60 | 61 | while (true) 62 | { 63 | var response = await _client.SearchAsync(sd => sd 64 | .Index($"{_eventLogItemsIndex}-*") 65 | .Sort(ss => 66 | ss.Descending(c => c.Id)) 67 | .Size(1) 68 | , cancellationToken); 69 | 70 | if (response.IsValid) 71 | { 72 | var item = response.Documents.FirstOrDefault(); 73 | 74 | if (item is null) 75 | return null; 76 | return new EventLogPosition(item.FileName, item.EndPosition, item.LgfEndPosition, item.Id); 77 | } 78 | 79 | if (response.OriginalException is TaskCanceledException) 80 | throw response.OriginalException; 81 | 82 | _logger?.LogError( 83 | $"Failed to get last file's position ({_eventLogItemsIndex}): {response.OriginalException.Message}"); 84 | 85 | var currentNodeHost = _currentNode.Host; 86 | 87 | await ConnectAsync(cancellationToken); 88 | 89 | // If it's the same node then wait while MaxRetryTimeout occurs, otherwise it'll be a too often request's loop 90 | if (_currentNode.Host.Equals(currentNodeHost)) 91 | await Task.Delay(_maxRetryTimeout, cancellationToken); 92 | } 93 | } 94 | 95 | public async Task WriteEventLogDataAsync(List entities, 96 | CancellationToken cancellationToken = default) 97 | { 98 | if (_client is null) 99 | await ConnectAsync(cancellationToken); 100 | 101 | var data = GetGroupedData(entities); 102 | 103 | for (var i = 0; i < data.Count; i++) 104 | { 105 | if (cancellationToken.IsCancellationRequested) 106 | return; 107 | 108 | var item = data[i]; 109 | 110 | var responseItems = await _client.IndexManyAsync(item.Entities, item.IndexName, cancellationToken); 111 | 112 | if (!responseItems.ApiCall.Success) 113 | { 114 | if (responseItems.OriginalException is TaskCanceledException) 115 | throw responseItems.OriginalException; 116 | 117 | if (responseItems.Errors) 118 | { 119 | foreach (var itemWithError in responseItems.ItemsWithErrors) 120 | _logger?.LogError( 121 | $"Failed to index document {itemWithError.Id} in {item.IndexName}: {itemWithError.Error}"); 122 | 123 | throw new Exception( 124 | $"Failed to write items to {item.IndexName}: {responseItems.OriginalException.Message}"); 125 | } 126 | 127 | _logger?.LogError( 128 | $"Failed to write items to {item.IndexName}: {responseItems.OriginalException.Message}"); 129 | 130 | await ConnectAsync(cancellationToken); 131 | 132 | i--; 133 | } 134 | else 135 | { 136 | if (responseItems.Errors) 137 | { 138 | foreach (var itemWithError in responseItems.ItemsWithErrors) 139 | _logger?.LogError( 140 | $"Failed to index document {itemWithError.Id} in {item.IndexName}: {itemWithError.Error}"); 141 | 142 | throw new Exception( 143 | $"Failed to write items to {item.IndexName}: {responseItems.OriginalException.Message}"); 144 | } 145 | 146 | _logger?.LogDebug($"{item.Entities.Count} items were being written to {item.IndexName}"); 147 | } 148 | } 149 | } 150 | 151 | public void Dispose() 152 | { 153 | } 154 | 155 | private void CheckSettings() 156 | { 157 | if (_nodes.Count == 0) 158 | throw new Exception("ElasticSearch hosts is not specified"); 159 | 160 | if (_eventLogItemsIndex == string.Empty) 161 | throw new Exception("ElasticSearch index name is not specified"); 162 | else 163 | _eventLogItemsIndex = FixIndexName(_eventLogItemsIndex); 164 | } 165 | 166 | private static string FixIndexName(string name) 167 | { 168 | var result = Regex.Replace(name, @"[\\,/,\*,\?,"",<,>,\|, ]", "_", RegexOptions.Compiled); 169 | return Regex.Replace(result, "^[-,+,_]*", "", RegexOptions.Compiled); 170 | } 171 | 172 | private async Task ConnectAsync(CancellationToken cancellationToken = default) 173 | { 174 | while (!cancellationToken.IsCancellationRequested) 175 | { 176 | var connected = await SwitchToNextNodeAsync(cancellationToken); 177 | 178 | if (connected) 179 | { 180 | await CreateIndexTemplateAsync(cancellationToken); 181 | 182 | break; 183 | } 184 | } 185 | } 186 | 187 | private async Task CreateIndexTemplateAsync(CancellationToken cancellationToken = default) 188 | { 189 | var indexTemplateName = "oneslogs"; 190 | 191 | var getItResponse = await _client.LowLevel.DoRequestAsync(HttpMethod.GET, 192 | $"_index_template/{indexTemplateName}", cancellationToken); 193 | 194 | // if it exists then skip creating 195 | if (!getItResponse.Success) 196 | throw getItResponse.OriginalException; 197 | if (getItResponse.HttpStatusCode != 404) 198 | return; 199 | 200 | var cmd = 201 | @"{ 202 | ""index_patterns"": ""*-el-*"", 203 | ""template"": { 204 | ""settings"": { 205 | ""index.codec"": ""best_compression"" 206 | }, 207 | ""mappings"": { 208 | ""properties"": { 209 | ""fileName"": { ""type"": ""keyword"" }, 210 | ""endPosition"": { ""type"": ""long"" }, 211 | ""lgfEndPosition"": { ""type"": ""long"" }, 212 | ""Id"": { ""type"": ""long"" }, 213 | ""dateTime"": { ""type"": ""date"" }, 214 | ""severity"": { ""type"": ""keyword"" }, 215 | ""server"": { ""type"": ""keyword"" }, 216 | ""metadata"": { ""type"": ""keyword"" }, 217 | ""data"": { ""type"": ""text"" }, 218 | ""transactionDateTime"": { ""type"": ""date"" }, 219 | ""transactionStatus"": { ""type"": ""keyword"" }, 220 | ""session"": { ""type"": ""long"" }, 221 | ""mainPort"": { ""type"": ""integer"" }, 222 | ""transactionNumber"": { ""type"": ""long"" }, 223 | ""addPort"": { ""type"": ""integer"" }, 224 | ""computer"": { ""type"": ""keyword"" }, 225 | ""application"": { ""type"": ""keyword"" }, 226 | ""userUuid"": { ""type"": ""keyword"" }, 227 | ""comment"": { ""type"": ""text"" }, 228 | ""connection"": { ""type"": ""long"" }, 229 | ""event"": { ""type"": ""keyword"" }, 230 | ""metadataUuid"": { ""type"": ""keyword"" }, 231 | ""dataPresentation"": { ""type"": ""text"" }, 232 | ""user"": { ""type"": ""keyword"" } 233 | } 234 | } 235 | } 236 | }"; 237 | 238 | var response = await _client.LowLevel.DoRequestAsync(HttpMethod.PUT, 239 | $"_index_template/{indexTemplateName}", cancellationToken, PostData.String(cmd)); 240 | 241 | if (!response.Success) 242 | throw response.OriginalException; 243 | } 244 | 245 | private async Task SwitchToNextNodeAsync(CancellationToken cancellationToken = default) 246 | { 247 | if (_currentNode == null) 248 | { 249 | _currentNode = _nodes[0]; 250 | } 251 | else 252 | { 253 | var currentIndex = _nodes.IndexOf(_currentNode); 254 | 255 | _currentNode = currentIndex == _nodes.Count - 1 ? _nodes[0] : _nodes[currentIndex + 1]; 256 | } 257 | 258 | var uri = new Uri(_currentNode.Host); 259 | 260 | var settings = new ConnectionSettings(uri); 261 | settings.EnableHttpCompression(); 262 | settings.MaximumRetries(_maximumRetries); 263 | settings.MaxRetryTimeout(_maxRetryTimeout); 264 | 265 | switch (_currentNode.AuthenticationType) 266 | { 267 | case AuthenticationType.Basic: 268 | settings.BasicAuthentication(_currentNode.UserName, _currentNode.Password); 269 | break; 270 | case AuthenticationType.ApiKey: 271 | settings.ApiKeyAuthentication(_currentNode.Id, _currentNode.ApiKey); 272 | break; 273 | } 274 | 275 | _client = new ElasticClient(settings); 276 | 277 | _logger?.LogInformation($"Trying to connect to {uri} ({_eventLogItemsIndex})"); 278 | 279 | var response = await _client.PingAsync(pd => pd, cancellationToken); 280 | 281 | if (!(response.OriginalException is TaskCanceledException)) 282 | { 283 | if (!response.IsValid) 284 | _logger?.LogWarning( 285 | $"Failed to connect to {uri} ({_eventLogItemsIndex}): {response.OriginalException.Message}"); 286 | else 287 | _logger?.LogInformation($"Successfully connected to {uri} ({_eventLogItemsIndex})"); 288 | } 289 | 290 | return response.IsValid; 291 | } 292 | 293 | private List<(string IndexName, List Entities)> GetGroupedData(List entities) 294 | { 295 | var data = new List<(string IndexName, List Entities)>(); 296 | 297 | switch (_separation) 298 | { 299 | case "H": 300 | var groups = entities.GroupBy(c => c.DateTime.ToString("yyyyMMddhh")).OrderBy(c => c.Key); 301 | foreach (var item in groups) 302 | data.Add(($"{_eventLogItemsIndex}-{item.Key}", item.ToList())); 303 | break; 304 | case "D": 305 | groups = entities.GroupBy(c => c.DateTime.ToString("yyyyMMdd")).OrderBy(c => c.Key); 306 | foreach (var item in groups) 307 | data.Add(($"{_eventLogItemsIndex}-{item.Key}", item.ToList())); 308 | break; 309 | case "M": 310 | groups = entities.GroupBy(c => c.DateTime.ToString("yyyyMM")).OrderBy(c => c.Key); 311 | foreach (var item in groups) 312 | data.Add(($"{_eventLogItemsIndex}-{item.Key}", item.ToList())); 313 | break; 314 | default: 315 | data.Add(($"{_eventLogItemsIndex}-all", entities)); 316 | break; 317 | } 318 | 319 | return data; 320 | } 321 | } 322 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/ElasticSearch/ElasticSearchStorageSettings.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | 4 | namespace OneSTools.EventLog.Exporter.Core.ElasticSearch 5 | { 6 | public class ElasticSearchStorageSettings 7 | { 8 | public List Nodes { get; } = new List(); 9 | public string Index { get; set; } = ""; 10 | public string Separation { get; set; } = ""; 11 | public int MaximumRetries { get; set; } = ElasticSearchStorage.DefaultMaximumRetries; 12 | 13 | public TimeSpan MaxRetryTimeout { get; set; } = 14 | TimeSpan.FromSeconds(ElasticSearchStorage.DefaultMaxRetryTimeoutSec); 15 | } 16 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/EventLogExporter.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.IO; 3 | using System.Linq; 4 | using System.Threading; 5 | using System.Threading.Tasks; 6 | using System.Threading.Tasks.Dataflow; 7 | using Microsoft.Extensions.Configuration; 8 | using Microsoft.Extensions.Logging; 9 | using NodaTime; 10 | 11 | namespace OneSTools.EventLog.Exporter.Core 12 | { 13 | public class EventLogExporter : IDisposable 14 | { 15 | private readonly int _collectedFactor; 16 | private readonly bool _loadArchive; 17 | 18 | // Exporter settings 19 | private readonly string _logFolder; 20 | private readonly ILogger _logger; 21 | private readonly int _portion; 22 | private readonly int _readingTimeout; 23 | private readonly IEventLogStorage _storage; 24 | private readonly DateTimeZone _timeZone = DateTimeZoneProviders.Tzdb.GetSystemDefault(); 25 | private readonly int _writingMaxDop; 26 | private BatchBlock _batchBlock; 27 | private readonly DateTime _skipEventsBeforeDate; 28 | 29 | private string _currentLgpFile; 30 | 31 | private bool _disposedValue; 32 | 33 | // DataFlow blocks 34 | private EventLogReader _eventLogReader; 35 | private ActionBlock _writeBlock; 36 | 37 | public EventLogExporter(EventLogExporterSettings settings, IEventLogStorage storage, 38 | ILogger logger = null) 39 | { 40 | _logger = logger; 41 | _storage = storage; 42 | 43 | _logFolder = settings.LogFolder; 44 | _portion = settings.Portion; 45 | _writingMaxDop = settings.WritingMaxDop; 46 | _collectedFactor = settings.CollectedFactor; 47 | _loadArchive = settings.LoadArchive; 48 | _timeZone = settings.TimeZone; 49 | _readingTimeout = settings.ReadingTimeout; 50 | _skipEventsBeforeDate = settings.SkipEventsBeforeDate; 51 | 52 | CheckSettings(); 53 | } 54 | 55 | public EventLogExporter(ILogger logger, IConfiguration configuration, 56 | IEventLogStorage storage) 57 | { 58 | _logger = logger; 59 | _storage = storage; 60 | 61 | _logFolder = configuration.GetValue("Exporter:LogFolder", ""); 62 | _portion = configuration.GetValue("Exporter:Portion", 10000); 63 | _writingMaxDop = configuration.GetValue("Exporter:WritingMaxDegreeOfParallelism", 1); 64 | _collectedFactor = configuration.GetValue("Exporter:CollectedFactor", 2); 65 | _loadArchive = configuration.GetValue("Exporter:LoadArchive", false); 66 | _readingTimeout = configuration.GetValue("Exporter:ReadingTimeout", 1); 67 | _skipEventsBeforeDate = configuration.GetValue("Exporter:SkipEventsBeforeDate", DateTime.MinValue); 68 | 69 | var timeZone = configuration.GetValue("Exporter:TimeZone", ""); 70 | 71 | if (!string.IsNullOrWhiteSpace(timeZone)) 72 | { 73 | var zone = DateTimeZoneProviders.Tzdb.GetZoneOrNull(timeZone); 74 | 75 | _timeZone = zone ?? throw new Exception($"\"{timeZone}\" is unknown time zone"); 76 | } 77 | 78 | CheckSettings(); 79 | } 80 | 81 | private void CheckSettings() 82 | { 83 | if (_logFolder == string.Empty) 84 | throw new Exception("Event log folder is not specified"); 85 | 86 | if (!Directory.Exists(_logFolder)) 87 | throw new Exception($"Event log folder ({_logFolder}) doesn't exist"); 88 | 89 | if (_writingMaxDop <= 0) 90 | throw new Exception("WritingMaxDegreeOfParallelism cannot be equal to or less than 0"); 91 | 92 | if (_collectedFactor <= 0) 93 | throw new Exception("CollectedFactor cannot be equal to or less than 0"); 94 | } 95 | 96 | public async Task StartAsync(CancellationToken cancellationToken = default) 97 | { 98 | _logger?.LogInformation($"Log folder: {_logFolder}"); 99 | 100 | if (_loadArchive) 101 | _logger?.LogWarning("\"Load archive\" mode enabled"); 102 | 103 | _logger?.LogInformation($"Portion per request: {_portion}"); 104 | 105 | InitializeDataflow(cancellationToken); 106 | 107 | try 108 | { 109 | var settings = await GetReaderSettingsAsync(cancellationToken); 110 | _eventLogReader = new EventLogReader(settings); 111 | 112 | while (!cancellationToken.IsCancellationRequested && !_writeBlock.Completion.IsCompleted) 113 | { 114 | var forceSending = false; 115 | 116 | EventLogItem item = null; 117 | 118 | try 119 | { 120 | item = _eventLogReader.ReadNextEventLogItem(cancellationToken); 121 | } 122 | catch (EventLogReaderTimeoutException) 123 | { 124 | forceSending = true; 125 | } 126 | catch (Exception) 127 | { 128 | _batchBlock.Complete(); 129 | throw; 130 | } 131 | 132 | if (item != null) 133 | { 134 | await SendAsync(_batchBlock, item, cancellationToken); 135 | 136 | if (!string.IsNullOrEmpty(_eventLogReader.LgpFileName) && 137 | _currentLgpFile != _eventLogReader.LgpFileName) 138 | { 139 | _logger?.LogInformation($"Reader started reading {_eventLogReader.LgpFileName}"); 140 | 141 | _currentLgpFile = _eventLogReader.LgpFileName; 142 | } 143 | } 144 | else if (!settings.LiveMode) 145 | { 146 | forceSending = true; 147 | } 148 | 149 | if (forceSending) 150 | _batchBlock.TriggerBatch(); 151 | } 152 | } 153 | catch (TaskCanceledException) 154 | { 155 | } 156 | } 157 | 158 | private void InitializeDataflow(CancellationToken cancellationToken = default) 159 | { 160 | var writeBlockSettings = new ExecutionDataflowBlockOptions 161 | { 162 | CancellationToken = cancellationToken, 163 | BoundedCapacity = _collectedFactor, 164 | MaxDegreeOfParallelism = _writingMaxDop 165 | }; 166 | 167 | var batchBlockSettings = new GroupingDataflowBlockOptions 168 | { 169 | CancellationToken = cancellationToken, 170 | BoundedCapacity = _portion * _collectedFactor 171 | }; 172 | 173 | _writeBlock = new ActionBlock(async c => 174 | { 175 | try 176 | { 177 | await _storage.WriteEventLogDataAsync(c.ToList(), cancellationToken); 178 | } 179 | catch (Exception) 180 | { 181 | _batchBlock.Complete(); 182 | _writeBlock.Complete(); 183 | throw; 184 | } 185 | }, 186 | writeBlockSettings); 187 | 188 | _batchBlock = new BatchBlock(_portion, batchBlockSettings); 189 | _batchBlock.LinkTo(_writeBlock, new DataflowLinkOptions { PropagateCompletion = true }); 190 | } 191 | 192 | private async Task GetReaderSettingsAsync(CancellationToken cancellationToken = default) 193 | { 194 | var eventLogReaderSettings = new EventLogReaderSettings 195 | { 196 | LogFolder = _logFolder, 197 | LiveMode = true, 198 | ReadingTimeout = _readingTimeout * 1000, 199 | TimeZone = _timeZone, 200 | SkipEventsBeforeDate = _skipEventsBeforeDate 201 | }; 202 | 203 | if (!_loadArchive) 204 | { 205 | var position = await _storage.ReadEventLogPositionAsync(cancellationToken); 206 | 207 | if (position != null) 208 | { 209 | var lgpFilePath = Path.Combine(_logFolder, position.FileName); 210 | 211 | if (!File.Exists(lgpFilePath)) 212 | { 213 | _logger?.LogWarning( 214 | $"Lgp file ({lgpFilePath}) doesn't exist. The reading will be started from the first found file"); 215 | } 216 | else 217 | { 218 | eventLogReaderSettings.LgpFileName = position.FileName; 219 | eventLogReaderSettings.LgpStartPosition = position.EndPosition; 220 | eventLogReaderSettings.LgfStartPosition = position.LgfEndPosition; 221 | eventLogReaderSettings.ItemId = position.Id; 222 | 223 | _logger?.LogInformation( 224 | $"File {position.FileName} will be read from {position.EndPosition} position, LGF file will be read from {position.LgfEndPosition} position"); 225 | } 226 | } 227 | else 228 | { 229 | _logger?.LogInformation( 230 | "There're no log items in the database, first found log file will be read from 0 position"); 231 | } 232 | } 233 | else 234 | { 235 | _logger?.LogWarning("LoadArchive parameter is true. Live mode will not be used"); 236 | 237 | eventLogReaderSettings.LiveMode = false; 238 | } 239 | 240 | return eventLogReaderSettings; 241 | } 242 | 243 | private static async Task SendAsync(ITargetBlock nextBlock, EventLogItem item, 244 | CancellationToken stoppingToken = default) 245 | { 246 | while (!stoppingToken.IsCancellationRequested && !nextBlock.Completion.IsCompleted) 247 | if (await nextBlock.SendAsync(item, stoppingToken)) 248 | break; 249 | } 250 | 251 | protected virtual void Dispose(bool disposing) 252 | { 253 | if (_disposedValue) 254 | return; 255 | 256 | if (disposing) _storage?.Dispose(); 257 | 258 | _eventLogReader?.Dispose(); 259 | 260 | _disposedValue = true; 261 | } 262 | 263 | ~EventLogExporter() 264 | { 265 | Dispose(false); 266 | } 267 | 268 | public void Dispose() 269 | { 270 | Dispose(true); 271 | GC.SuppressFinalize(this); 272 | } 273 | } 274 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/EventLogExporterSettings.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using NodaTime; 3 | 4 | namespace OneSTools.EventLog.Exporter.Core 5 | { 6 | public class EventLogExporterSettings 7 | { 8 | public string LogFolder { get; set; } = ""; 9 | public int Portion { get; set; } = 10000; 10 | public DateTimeZone TimeZone { get; set; } = DateTimeZoneProviders.Tzdb.GetSystemDefault(); 11 | public int WritingMaxDop { get; set; } = 1; 12 | public int CollectedFactor { get; set; } = 2; 13 | public int ReadingTimeout { get; set; } = 1; 14 | public bool LoadArchive { get; set; } = false; 15 | public DateTime SkipEventsBeforeDate { get; set; } 16 | } 17 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/EventLogPosition.cs: -------------------------------------------------------------------------------- 1 | namespace OneSTools.EventLog.Exporter.Core 2 | { 3 | public class EventLogPosition 4 | { 5 | public EventLogPosition(string fileName, long endPosition, long lgfEndPosition, long id) 6 | { 7 | FileName = fileName; 8 | EndPosition = endPosition; 9 | LgfEndPosition = lgfEndPosition; 10 | Id = id; 11 | } 12 | 13 | public string FileName { get; } 14 | public long EndPosition { get; } 15 | public long LgfEndPosition { get; } 16 | public long Id { get; } 17 | } 18 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/FileLogger.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.IO; 3 | using Microsoft.Extensions.Logging; 4 | 5 | namespace OneSTools.EventLog.Exporter.Core 6 | { 7 | public class FileLogger : ILogger 8 | { 9 | private static readonly object Locker = new object(); 10 | private readonly string _categoryName; 11 | private readonly string _path; 12 | 13 | public FileLogger(string path, string categoryName) 14 | { 15 | _path = path; 16 | _categoryName = categoryName; 17 | } 18 | 19 | public IDisposable BeginScope(TState state) 20 | { 21 | return null; 22 | } 23 | 24 | public bool IsEnabled(LogLevel logLevel) 25 | { 26 | return true; 27 | } 28 | 29 | public void Log(LogLevel logLevel, EventId eventId, TState state, Exception exception, 30 | Func formatter) 31 | { 32 | var levelName = Enum.GetName(typeof(LogLevel), logLevel); 33 | var message = $"{DateTime.Now:yyyy-MM-dd HH:mm:ss.fff} | {levelName} | {_categoryName}[{eventId.Id}]\n\t{ formatter(state, exception)}"; 34 | 35 | lock (Locker) 36 | { 37 | File.AppendAllText(_path, message + Environment.NewLine); 38 | } 39 | } 40 | } 41 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/FileLoggerProvider.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.Logging; 2 | 3 | namespace OneSTools.EventLog.Exporter.Core 4 | { 5 | public class FileLoggerProvider : ILoggerProvider 6 | { 7 | private readonly string _path; 8 | 9 | public FileLoggerProvider(string path) 10 | { 11 | _path = path; 12 | } 13 | 14 | public ILogger CreateLogger(string categoryName) 15 | { 16 | return new FileLogger(_path, categoryName); 17 | } 18 | 19 | public void Dispose() 20 | { 21 | } 22 | } 23 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/IEventLogStorage.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Threading; 4 | using System.Threading.Tasks; 5 | 6 | namespace OneSTools.EventLog.Exporter.Core 7 | { 8 | public interface IEventLogStorage : IDisposable 9 | { 10 | Task ReadEventLogPositionAsync(CancellationToken cancellationToken = default); 11 | Task WriteEventLogDataAsync(List entities, CancellationToken cancellationToken = default); 12 | } 13 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/ILoggerBuilderExtensions.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.Extensions.Logging; 2 | 3 | namespace OneSTools.EventLog.Exporter.Core 4 | { 5 | public static class LoggerBuilderExtensions 6 | { 7 | public static ILoggingBuilder AddFile(this ILoggingBuilder loggingBuilder, string path) 8 | { 9 | var provider = new FileLoggerProvider(path); 10 | 11 | return loggingBuilder.AddProvider(provider); 12 | } 13 | } 14 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/OneSTools.EventLog.Exporter.Core.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | netstandard2.1 5 | OneSTools.EventLog.Exporter.Core 6 | OneSTools.EventLog.Exporter.Core 7 | Akpaev Evgeny 8 | Базовый пакет экспортеров ЖР 9 | Akpaev Evgeny 10 | 1.1.1 11 | 12 | 13 | 14 | ..\Build\Release 15 | 16 | 17 | 18 | ..\Build\Debug 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Core/StorageType.cs: -------------------------------------------------------------------------------- 1 | namespace OneSTools.EventLog.Exporter.Core 2 | { 3 | public enum StorageType 4 | { 5 | None = 0, 6 | ClickHouse, 7 | ElasticSearch 8 | } 9 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Manager/ClstEventArgs.cs: -------------------------------------------------------------------------------- 1 | namespace OneSTools.EventLog.Exporter.Manager 2 | { 3 | public class ClstEventArgs 4 | { 5 | internal ClstEventArgs(string path, string name, string databaseName) 6 | { 7 | Path = path; 8 | Name = name; 9 | DataBaseName = databaseName; 10 | } 11 | 12 | public string Path { get; } 13 | public string Name { get; } 14 | public string DataBaseName { get; } 15 | } 16 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Manager/ClstFolder.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Generic; 2 | 3 | namespace OneSTools.EventLog.Exporter.Manager 4 | { 5 | public class ClstFolder 6 | { 7 | public string Folder { get; set; } = ""; 8 | public List Templates { get; set; } = new(); 9 | } 10 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Manager/ClstWatcher.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.Collections.ObjectModel; 4 | using System.IO; 5 | using System.Linq; 6 | using System.Text.RegularExpressions; 7 | using OneSTools.BracketsFile; 8 | 9 | namespace OneSTools.EventLog.Exporter.Manager 10 | { 11 | public class ClstWatcher : IDisposable 12 | { 13 | public delegate void InfoBaseAddedHandler(object sender, ClstEventArgs args); 14 | 15 | public delegate void InfoBaseDeletedHandler(object sender, ClstEventArgs args); 16 | 17 | private readonly string _folder; 18 | private readonly string _path; 19 | private readonly List _templates; 20 | private FileSystemWatcher _clstWatcher; 21 | private Dictionary _infoBases; 22 | 23 | public ClstWatcher(string folder, List templates) 24 | { 25 | _folder = folder; 26 | _templates = templates; 27 | _path = Path.Combine(_folder, "1CV8Clst.lst"); 28 | if (!File.Exists(_path)) 29 | throw new Exception("Couldn't find LST \"1CV8Clst.lst\" file"); 30 | 31 | _infoBases = ReadInfoBases(); 32 | InitializeWatcher(); 33 | } 34 | 35 | public ReadOnlyDictionary InfoBases => new(_infoBases); 36 | 37 | public void Dispose() 38 | { 39 | _clstWatcher?.Dispose(); 40 | GC.SuppressFinalize(this); 41 | } 42 | 43 | public event InfoBaseAddedHandler InfoBasesAdded; 44 | public event InfoBaseDeletedHandler InfoBasesDeleted; 45 | 46 | private Dictionary ReadInfoBases() 47 | { 48 | var items = new Dictionary(); 49 | 50 | var fileData = File.ReadAllText(_path); 51 | var parsedData = BracketsParser.ParseBlock(fileData); 52 | 53 | var infoBasesNode = parsedData[2]; 54 | int count = infoBasesNode[0]; 55 | 56 | if (count > 0) 57 | for (var i = 1; i <= count; i++) 58 | { 59 | var infoBaseNode = infoBasesNode[i]; 60 | 61 | var elPath = Path.Combine(_folder, infoBaseNode[0]); 62 | string name = infoBaseNode[5]; 63 | 64 | foreach (var template in _templates) 65 | if (Regex.IsMatch(name, template.Mask)) 66 | { 67 | var dataBaseName = template.Template.Replace("[IBNAME]", name); 68 | items.Add(elPath, (name, dataBaseName)); 69 | 70 | break; 71 | } 72 | } 73 | 74 | return items; 75 | } 76 | 77 | private void ReadInfoBasesAndRaiseEvents() 78 | { 79 | var newInfoBases = ReadInfoBases(); 80 | 81 | var added = newInfoBases.Except(_infoBases); 82 | foreach (var (key, (item1, item2)) in added) 83 | InfoBasesAdded?.Invoke(this, new ClstEventArgs(key, item1, item2)); 84 | 85 | var deleted = _infoBases.Except(newInfoBases); 86 | foreach (var (key, (item1, item2)) in deleted) 87 | InfoBasesDeleted?.Invoke(this, new ClstEventArgs(key, item1, item2)); 88 | 89 | _infoBases = newInfoBases; 90 | } 91 | 92 | private void InitializeWatcher() 93 | { 94 | _clstWatcher = new FileSystemWatcher(_folder, "1CV8Clst.lst") 95 | { 96 | NotifyFilter = NotifyFilters.CreationTime | NotifyFilters.LastWrite 97 | }; 98 | _clstWatcher.Changed += ClstWatcher_Changed; 99 | _clstWatcher.EnableRaisingEvents = true; 100 | } 101 | 102 | private void ClstWatcher_Changed(object sender, FileSystemEventArgs e) 103 | { 104 | lock (InfoBases) 105 | { 106 | ReadInfoBasesAndRaiseEvents(); 107 | } 108 | } 109 | } 110 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Manager/ExportersManager.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.IO; 4 | using System.Linq; 5 | using System.Threading; 6 | using System.Threading.Tasks; 7 | using Microsoft.Extensions.Configuration; 8 | using Microsoft.Extensions.Hosting; 9 | using Microsoft.Extensions.Logging; 10 | using NodaTime; 11 | using OneSTools.EventLog.Exporter.Core; 12 | using OneSTools.EventLog.Exporter.Core.ClickHouse; 13 | using OneSTools.EventLog.Exporter.Core.ElasticSearch; 14 | 15 | namespace OneSTools.EventLog.Exporter.Manager 16 | { 17 | public class ExportersManager : BackgroundService 18 | { 19 | private readonly List _clstFolders; 20 | private readonly List _clstWatchers = new(); 21 | 22 | private readonly int _collectedFactor; 23 | 24 | // ClickHouse 25 | private readonly string _connectionString; 26 | private readonly bool _loadArchive; 27 | private readonly ILogger _logger; 28 | private readonly int _maximumRetries; 29 | 30 | private readonly TimeSpan _maxRetryTimeout; 31 | 32 | // ElasticSearch 33 | private readonly List _nodes; 34 | private readonly int _portion; 35 | private readonly int _readingTimeout; 36 | private readonly Dictionary _runExporters = new(); 37 | private readonly string _separation; 38 | 39 | private readonly IServiceProvider _serviceProvider; 40 | 41 | // Common settings 42 | private readonly StorageType _storageType; 43 | private readonly DateTimeZone _timeZone = DateTimeZoneProviders.Tzdb.GetSystemDefault(); 44 | private readonly int _writingMaxDop; 45 | private readonly DateTime _skipEventsBeforeDate; 46 | 47 | public ExportersManager(ILogger logger, IServiceProvider serviceProvider, 48 | IConfiguration configuration) 49 | { 50 | _logger = logger; 51 | _serviceProvider = serviceProvider; 52 | 53 | _clstFolders = configuration.GetSection("Manager:ClstFolders").Get>(); 54 | _storageType = configuration.GetValue("Exporter:StorageType", StorageType.None); 55 | _portion = configuration.GetValue("Exporter:Portion", 10000); 56 | _writingMaxDop = configuration.GetValue("Exporter:WritingMaxDegreeOfParallelism", 1); 57 | _collectedFactor = configuration.GetValue("Exporter:CollectedFactor", 2); 58 | _loadArchive = configuration.GetValue("Exporter:LoadArchive", false); 59 | _readingTimeout = configuration.GetValue("Exporter:ReadingTimeout", 1); 60 | _skipEventsBeforeDate = configuration.GetValue("Exporter:SkipEventsBeforeDate", DateTime.MinValue); 61 | 62 | var timeZone = configuration.GetValue("Exporter:TimeZone", ""); 63 | 64 | if (!string.IsNullOrWhiteSpace(timeZone)) 65 | _timeZone = DateTimeZoneProviders.Tzdb.GetZoneOrNull(timeZone) ?? 66 | throw new Exception($"\"{timeZone}\" is unknown time zone"); 67 | 68 | CheckSettings(); 69 | 70 | switch (_storageType) 71 | { 72 | case StorageType.ClickHouse: 73 | { 74 | _connectionString = configuration.GetValue("ClickHouse:ConnectionString", ""); 75 | if (_connectionString == string.Empty) 76 | throw new Exception("Connection string is not specified"); 77 | break; 78 | } 79 | case StorageType.ElasticSearch: 80 | { 81 | _nodes = configuration.GetSection("ElasticSearch:Nodes").Get>(); 82 | if (_nodes == null) 83 | throw new Exception("ElasticSearch nodes are not specified"); 84 | 85 | _separation = configuration.GetValue("ElasticSearch:Separation", "H"); 86 | _maximumRetries = configuration.GetValue("ElasticSearch:MaximumRetries", 87 | ElasticSearchStorage.DefaultMaximumRetries); 88 | _maxRetryTimeout = TimeSpan.FromSeconds(configuration.GetValue("ElasticSearch:MaxRetryTimeout", 89 | ElasticSearchStorage.DefaultMaxRetryTimeoutSec)); 90 | break; 91 | } 92 | } 93 | } 94 | 95 | private void CheckSettings() 96 | { 97 | if (_clstFolders == null || _clstFolders.Count == 0) 98 | throw new Exception("\"ClstFolders\" is not specified"); 99 | 100 | foreach (var clstFolder in _clstFolders.Where(clstFolder => !Directory.Exists(clstFolder.Folder))) 101 | throw new Exception($"Clst folder ({clstFolder.Folder}) doesn't exist"); 102 | 103 | if (_writingMaxDop <= 0) 104 | throw new Exception("WritingMaxDegreeOfParallelism cannot be equal to or less than 0"); 105 | 106 | if (_collectedFactor <= 0) 107 | throw new Exception("CollectedFactor cannot be equal to or less than 0"); 108 | } 109 | 110 | protected override async Task ExecuteAsync(CancellationToken stoppingToken) 111 | { 112 | stoppingToken.Register(() => 113 | { 114 | lock (_runExporters) 115 | { 116 | foreach (var ib in _runExporters) 117 | ib.Value.Cancel(); 118 | } 119 | }); 120 | 121 | foreach (var clstFolder in _clstFolders) 122 | { 123 | var clstWatcher = new ClstWatcher(clstFolder.Folder, clstFolder.Templates); 124 | 125 | foreach (var (key, (name, dataBaseName)) in clstWatcher.InfoBases) 126 | StartExporter(key, name, dataBaseName); 127 | 128 | clstWatcher.InfoBasesAdded += ClstWatcher_InfoBasesAdded; 129 | clstWatcher.InfoBasesDeleted += ClstWatcher_InfoBasesDeleted; 130 | 131 | _clstWatchers.Add(clstWatcher); 132 | } 133 | 134 | await Task.Factory.StartNew(stoppingToken.WaitHandle.WaitOne, stoppingToken); 135 | } 136 | 137 | private void ClstWatcher_InfoBasesDeleted(object sender, ClstEventArgs args) 138 | { 139 | StopExporter(args.Path, args.Name); 140 | } 141 | 142 | private void ClstWatcher_InfoBasesAdded(object sender, ClstEventArgs args) 143 | { 144 | StartExporter(args.Path, args.Name, args.DataBaseName); 145 | } 146 | 147 | private void StartExporter(string path, string name, string dataBaseName) 148 | { 149 | var logFolder = Path.Combine(path, "1Cv8Log"); 150 | 151 | // Check this is an old event log format 152 | var lgfPath = Path.Combine(logFolder, "1Cv8.lgf"); 153 | 154 | var needStart = File.Exists(lgfPath); 155 | 156 | if (needStart) 157 | { 158 | lock (_runExporters) 159 | { 160 | if (!_runExporters.ContainsKey(path)) 161 | { 162 | var cts = new CancellationTokenSource(); 163 | var logger = 164 | (ILogger)_serviceProvider.GetService(typeof(ILogger)); 165 | 166 | var settings = new EventLogExporterSettings 167 | { 168 | LogFolder = logFolder, 169 | CollectedFactor = _collectedFactor, 170 | LoadArchive = _loadArchive, 171 | Portion = _portion, 172 | ReadingTimeout = _readingTimeout, 173 | TimeZone = _timeZone, 174 | WritingMaxDop = _writingMaxDop, 175 | SkipEventsBeforeDate = _skipEventsBeforeDate 176 | }; 177 | 178 | Task.Factory.StartNew(async () => 179 | { 180 | while (!cts.Token.IsCancellationRequested) 181 | { 182 | try 183 | { 184 | using var storage = GetStorage(dataBaseName); 185 | using var exporter = new EventLogExporter(settings, storage, logger); 186 | await exporter.StartAsync(cts.Token); 187 | } 188 | catch (TaskCanceledException) 189 | { 190 | } 191 | catch (Exception ex) 192 | { 193 | _logger?.LogCritical(ex, "Failed to execute EventLogExporter"); 194 | } 195 | await Task.Delay(5000); 196 | } 197 | }, cts.Token); 198 | _runExporters.Add(path, cts); 199 | 200 | _logger?.LogInformation( 201 | $"Event log exporter for \"{name}\" information base to \"{dataBaseName}\" is started"); 202 | } 203 | } 204 | } 205 | else 206 | { 207 | _logger?.LogInformation( 208 | $"Event log of \"{name}\" information base is in \"new\" format, it won't be handled"); 209 | } 210 | } 211 | 212 | private void StopExporter(string id, string name) 213 | { 214 | lock (_runExporters) 215 | { 216 | if (_runExporters.TryGetValue(id, out var cts)) 217 | { 218 | cts.Cancel(); 219 | _logger?.LogInformation($"Event log exporter for \"{name}\" information base is stopped"); 220 | } 221 | } 222 | } 223 | 224 | private IEventLogStorage GetStorage(string dataBaseName) 225 | { 226 | switch (_storageType) 227 | { 228 | case StorageType.ClickHouse: 229 | { 230 | var logger = 231 | (ILogger)_serviceProvider.GetService(typeof(ILogger)); 232 | var connectionString = $"{_connectionString}Database={dataBaseName};"; 233 | 234 | return new ClickHouseStorage(connectionString, logger); 235 | } 236 | case StorageType.ElasticSearch: 237 | { 238 | var logger = 239 | (ILogger)_serviceProvider.GetService( 240 | typeof(ILogger)); 241 | 242 | var settings = new ElasticSearchStorageSettings 243 | { 244 | Index = dataBaseName, 245 | Separation = _separation, 246 | MaximumRetries = _maximumRetries, 247 | MaxRetryTimeout = _maxRetryTimeout 248 | }; 249 | settings.Nodes.AddRange(_nodes); 250 | 251 | return new ElasticSearchStorage(settings, logger); 252 | } 253 | case StorageType.None: 254 | throw new Exception("StorageType parameter is not specified"); 255 | default: 256 | throw new Exception("Try to get a storage for unknown StorageType value"); 257 | } 258 | } 259 | 260 | public override void Dispose() 261 | { 262 | base.Dispose(); 263 | 264 | foreach (var clstWatcher in _clstWatchers) 265 | clstWatcher?.Dispose(); 266 | 267 | GC.SuppressFinalize(this); 268 | } 269 | } 270 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Manager/OneSTools.EventLog.Exporter.Manager.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | net5.0 5 | dotnet-OneSTools.EventLog.Exporter.Manager-436AD68C-DBEB-42E3-8010-E66614FE89B6 6 | EventLogExportersManager 7 | 0.0.5 8 | Akpaev Evgeny 9 | Akpaev Evgeny 10 | Менеджер служб экспорта журналов регистрации 11 | 12 | 13 | 14 | ..\Build\Debug 15 | 16 | 17 | 18 | ..\Build\Release 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Manager/Program.cs: -------------------------------------------------------------------------------- 1 | using System.IO; 2 | using Microsoft.Extensions.DependencyInjection; 3 | using Microsoft.Extensions.Hosting; 4 | using Microsoft.Extensions.Logging; 5 | using OneSTools.EventLog.Exporter.Core; 6 | 7 | namespace OneSTools.EventLog.Exporter.Manager 8 | { 9 | public class Program 10 | { 11 | public static void Main(string[] args) 12 | { 13 | CreateHostBuilder(args).Build().Run(); 14 | } 15 | 16 | public static IHostBuilder CreateHostBuilder(string[] args) 17 | { 18 | return Host.CreateDefaultBuilder(args) 19 | .UseWindowsService() 20 | .UseSystemd() 21 | .ConfigureLogging((hostingContext, logging) => 22 | { 23 | var logPath = Path.Combine(hostingContext.HostingEnvironment.ContentRootPath, "log.txt"); 24 | logging.AddFile(logPath); 25 | logging.AddConfiguration(hostingContext.Configuration.GetSection("Logging")); 26 | }) 27 | .ConfigureServices((_, services) => { services.AddHostedService(); }); 28 | } 29 | } 30 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Manager/Properties/launchSettings.json: -------------------------------------------------------------------------------- 1 | { 2 | "profiles": { 3 | "OneSTools.EventLog.Exporter.Manager": { 4 | "commandName": "Project", 5 | "environmentVariables": { 6 | "DOTNET_ENVIRONMENT": "Development" 7 | } 8 | } 9 | } 10 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Manager/TemplateItem.cs: -------------------------------------------------------------------------------- 1 | namespace OneSTools.EventLog.Exporter.Manager 2 | { 3 | public class TemplateItem 4 | { 5 | public string Mask { get; set; } = ""; 6 | public string Template { get; set; } = ""; 7 | } 8 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Manager/appsettings.Development.json: -------------------------------------------------------------------------------- 1 | { 2 | "Logging": { 3 | "LogLevel": { 4 | "Default": "Debug", 5 | "Microsoft": "Warning", 6 | "Microsoft.Hosting.Lifetime": "Information" 7 | } 8 | } 9 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter.Manager/appsettings.json: -------------------------------------------------------------------------------- 1 | { 2 | "Logging": { 3 | "LogLevel": { 4 | "Default": "Debug", 5 | "Microsoft": "Warning", 6 | "Microsoft.Hosting.Lifetime": "Information" 7 | } 8 | }, 9 | "Manager": { 10 | "ClstFolders": [ 11 | { 12 | "Folder": "\\\\s01\\c$\\Program Files\\1cv8\\srvinfo\\reg_1541", 13 | "Templates": [ 14 | { 15 | "Mask": "upp_main", 16 | "Template": "[IBNAME]_el" 17 | } 18 | ] 19 | } 20 | ] 21 | }, 22 | "Exporter": { 23 | "StorageType": 1, 24 | "Portion": 10000, 25 | "TimeZone": "Europe/Moscow", 26 | "WritingMaxDegreeOfParallelism": 1, 27 | "CollectedFactor": 3, 28 | "ReadingTimeout": 1, 29 | "LoadArchive": false, 30 | "SkipEventsBeforeDate": "2022-04-01T00:00:00" 31 | }, 32 | "ClickHouse": { 33 | "ConnectionString": "Host=172.19.149.61;Port=8123;Username=default;password=;" 34 | }, 35 | "ElasticSearch": { 36 | "Nodes": [ 37 | { 38 | "Host": "http://192.168.0.95:9200", 39 | "AuthenticationType": "0" 40 | } 41 | ], 42 | "Separation": "M", 43 | "MaximumRetries": 2, 44 | "MaxRetryTimeout": 30 45 | } 46 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter/EventLogExporterService.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Threading; 3 | using System.Threading.Tasks; 4 | using Microsoft.Extensions.DependencyInjection; 5 | using Microsoft.Extensions.Hosting; 6 | using Microsoft.Extensions.Logging; 7 | using OneSTools.EventLog.Exporter.Core; 8 | 9 | namespace OneSTools.EventLog.Exporter 10 | { 11 | public class EventLogExporterService : BackgroundService 12 | { 13 | private readonly IServiceProvider _serviceProvider; 14 | private readonly ILogger _logger; 15 | private bool _disposedValue; 16 | 17 | public EventLogExporterService(IServiceProvider serviceProvider, ILogger logger) 18 | { 19 | _serviceProvider = serviceProvider; 20 | _logger = logger; 21 | } 22 | 23 | protected override async Task ExecuteAsync(CancellationToken stoppingToken) 24 | { 25 | while (!stoppingToken.IsCancellationRequested) 26 | { 27 | try 28 | { 29 | using var exporter = _serviceProvider.GetService(); 30 | await exporter.StartAsync(stoppingToken); 31 | } 32 | catch (TaskCanceledException) 33 | { 34 | } 35 | catch (Exception ex) 36 | { 37 | _logger?.LogCritical(ex, "Failed to execute EventLogExporter"); 38 | } 39 | await Task.Delay(5000); 40 | } 41 | } 42 | 43 | protected virtual void Dispose(bool disposing) 44 | { 45 | if (!_disposedValue) 46 | { 47 | if (disposing) 48 | { 49 | } 50 | 51 | _disposedValue = true; 52 | } 53 | } 54 | 55 | ~EventLogExporterService() 56 | { 57 | Dispose(false); 58 | } 59 | 60 | public override void Dispose() 61 | { 62 | Dispose(true); 63 | GC.SuppressFinalize(this); 64 | } 65 | } 66 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter/OneSTools.EventLog.Exporter.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | net5.0 5 | dotnet-EventLogExporterElasticSearch-E480DCDC-5542-443A-A879-763D915661EB 6 | Akpaev Evgeny 7 | Akpaev Evgeny 8 | 9 | Служба для экспорта журнала регистрации 1С в ClickHouse и ElasticSearch 10 | 1.0.5 11 | EventLogExporter 12 | OneSTools.EventLog.Exporter 13 | 14 | 15 | 16 | none 17 | false 18 | ..\Build\Release 19 | 20 | 21 | 22 | portable 23 | true 24 | ..\Build\Debug 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter/Program.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.IO; 3 | using Microsoft.Extensions.Configuration; 4 | using Microsoft.Extensions.DependencyInjection; 5 | using Microsoft.Extensions.Hosting; 6 | using Microsoft.Extensions.Logging; 7 | using OneSTools.EventLog.Exporter.Core; 8 | using OneSTools.EventLog.Exporter.Core.ClickHouse; 9 | using OneSTools.EventLog.Exporter.Core.ElasticSearch; 10 | 11 | namespace OneSTools.EventLog.Exporter 12 | { 13 | public class Program 14 | { 15 | public static void Main(string[] args) 16 | { 17 | CreateHostBuilder(args).Build().Run(); 18 | } 19 | 20 | public static IHostBuilder CreateHostBuilder(string[] args) 21 | { 22 | return Host.CreateDefaultBuilder(args) 23 | //.ConfigureAppConfiguration(c => { 24 | // c.SetBasePath(Path.GetDirectoryName(Assembly.GetEntryAssembly().Location)); 25 | // c.AddJsonFile("appsettings.json"); 26 | //}) 27 | .UseWindowsService() 28 | .UseSystemd() 29 | .ConfigureLogging((hostingContext, logging) => 30 | { 31 | var logPath = Path.Combine(hostingContext.HostingEnvironment.ContentRootPath, "log.txt"); 32 | logging.AddFile(logPath); 33 | logging.AddConfiguration(hostingContext.Configuration.GetSection("Logging")); 34 | }) 35 | .ConfigureServices((hostContext, services) => 36 | { 37 | var storageType = hostContext.Configuration.GetValue("Exporter:StorageType", StorageType.None); 38 | 39 | switch (storageType) 40 | { 41 | case StorageType.ClickHouse: 42 | services.AddTransient(); 43 | break; 44 | case StorageType.ElasticSearch: 45 | services.AddTransient(); 46 | break; 47 | case StorageType.None: 48 | throw new Exception("You must set StorageType parameter before starting the exporter"); 49 | default: 50 | throw new Exception($"{storageType} is not available value of StorageType enum"); 51 | } 52 | 53 | services.AddTransient(); 54 | services.AddHostedService(); 55 | }); 56 | } 57 | } 58 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter/Properties/launchSettings.json: -------------------------------------------------------------------------------- 1 | { 2 | "profiles": { 3 | "OneSTools.EventLog.Exporter": { 4 | "commandName": "Project", 5 | "environmentVariables": { 6 | "DOTNET_ENVIRONMENT": "Development" 7 | } 8 | } 9 | } 10 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter/appsettings.Development.json: -------------------------------------------------------------------------------- 1 | { 2 | "Logging": { 3 | "LogLevel": { 4 | "Default": "Debug", 5 | "Microsoft": "Warning", 6 | "Microsoft.Hosting.Lifetime": "Information" 7 | } 8 | }, 9 | "Exporter": { 10 | "StorageType": 2, 11 | "LogFolder": "C:\\Users\\akpaev.e.ENTERPRISE\\Desktop\\test", 12 | "Portion": 10000, 13 | "TimeZone": "Europe/Moscow", 14 | "WritingMaxDegreeOfParallelism": 8, 15 | "CollectedFactor": 8, 16 | "ReadingTimeout": 1, 17 | "LoadArchive": false 18 | }, 19 | "ClickHouse": { 20 | "ConnectionString": "Host=192.168.0.93;Port=8123;Database=zup3_united_el;Username=default;password=;" 21 | }, 22 | "ElasticSearch": { 23 | "Nodes": [ 24 | { 25 | "Host": "http://192.168.0.95:9200", 26 | "AuthenticationType": "0" // 0 - None, 1 - Basic, 2 - ApiKey 27 | } 28 | ], 29 | "Index": "test", 30 | "Separation": "M", 31 | "MaximumRetries": 2, 32 | "MaxRetryTimeout": 30 33 | } 34 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.Exporter/appsettings.json: -------------------------------------------------------------------------------- 1 | { 2 | "Logging": { 3 | "LogLevel": { 4 | "Default": "Debug", 5 | "Microsoft": "Warning", 6 | "Microsoft.Hosting.Lifetime": "Information" 7 | } 8 | }, 9 | "Exporter": { 10 | "StorageType": 0, 11 | "LogFolder": "", 12 | "Portion": 10000, 13 | "TimeZone": "Europe/Moscow", 14 | "WritingMaxDegreeOfParallelism": 1, 15 | "CollectedFactor": 3, 16 | "ReadingTimeout": 1, 17 | "LoadArchive": false, 18 | "SkipEventsBeforeDate": "2022-04-01T00:00:00" 19 | }, 20 | "ClickHouse": { 21 | "ConnectionString": "Host=CH_HOST_NAME;Port=8123;Database=DATABASE_NAME;Username=USER_NAME;password=PASSWORD;" 22 | }, 23 | "ElasticSearch": { 24 | "Nodes": [ 25 | { 26 | "Host": "", // address:port 27 | "AuthenticationType": "0" // 0 - None, 1 - Basic, 2 - ApiKey 28 | } 29 | ], 30 | "Index": "upp-main-el", // index prefix 31 | "Separation": "M", 32 | "MaximumRetries": 2, 33 | "MaxRetryTimeout": 30 34 | } 35 | } -------------------------------------------------------------------------------- /OneSTools.EventLog.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio Version 16 4 | VisualStudioVersion = 16.0.30611.23 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "OneSTools.EventLog", "OneSTools.EventLog\OneSTools.EventLog.csproj", "{0A35F7ED-8BF9-464B-BF77-2015E8EBA815}" 7 | EndProject 8 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "OneSTools.EventLog.Exporter.Core", "OneSTools.EventLog.Exporter.Core\OneSTools.EventLog.Exporter.Core.csproj", "{52D856AB-3545-43E2-AA6A-9EF2105A3049}" 9 | ProjectSection(ProjectDependencies) = postProject 10 | {0A35F7ED-8BF9-464B-BF77-2015E8EBA815} = {0A35F7ED-8BF9-464B-BF77-2015E8EBA815} 11 | EndProjectSection 12 | EndProject 13 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "OneSTools.EventLog.Exporter", "OneSTools.EventLog.Exporter\OneSTools.EventLog.Exporter.csproj", "{3268B883-CF2B-44A3-8C20-73A633B1831C}" 14 | EndProject 15 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "OneSTools.EventLog.Exporter.Manager", "OneSTools.EventLog.Exporter.Manager\OneSTools.EventLog.Exporter.Manager.csproj", "{853EB61D-AF7C-4105-A917-7B6195DB4843}" 16 | EndProject 17 | Global 18 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 19 | Debug|Any CPU = Debug|Any CPU 20 | Release|Any CPU = Release|Any CPU 21 | EndGlobalSection 22 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 23 | {0A35F7ED-8BF9-464B-BF77-2015E8EBA815}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 24 | {0A35F7ED-8BF9-464B-BF77-2015E8EBA815}.Debug|Any CPU.Build.0 = Debug|Any CPU 25 | {0A35F7ED-8BF9-464B-BF77-2015E8EBA815}.Release|Any CPU.ActiveCfg = Release|Any CPU 26 | {0A35F7ED-8BF9-464B-BF77-2015E8EBA815}.Release|Any CPU.Build.0 = Release|Any CPU 27 | {52D856AB-3545-43E2-AA6A-9EF2105A3049}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 28 | {52D856AB-3545-43E2-AA6A-9EF2105A3049}.Debug|Any CPU.Build.0 = Debug|Any CPU 29 | {52D856AB-3545-43E2-AA6A-9EF2105A3049}.Release|Any CPU.ActiveCfg = Release|Any CPU 30 | {52D856AB-3545-43E2-AA6A-9EF2105A3049}.Release|Any CPU.Build.0 = Release|Any CPU 31 | {3268B883-CF2B-44A3-8C20-73A633B1831C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 32 | {3268B883-CF2B-44A3-8C20-73A633B1831C}.Debug|Any CPU.Build.0 = Debug|Any CPU 33 | {3268B883-CF2B-44A3-8C20-73A633B1831C}.Release|Any CPU.ActiveCfg = Release|Any CPU 34 | {3268B883-CF2B-44A3-8C20-73A633B1831C}.Release|Any CPU.Build.0 = Release|Any CPU 35 | {853EB61D-AF7C-4105-A917-7B6195DB4843}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 36 | {853EB61D-AF7C-4105-A917-7B6195DB4843}.Debug|Any CPU.Build.0 = Debug|Any CPU 37 | {853EB61D-AF7C-4105-A917-7B6195DB4843}.Release|Any CPU.ActiveCfg = Release|Any CPU 38 | {853EB61D-AF7C-4105-A917-7B6195DB4843}.Release|Any CPU.Build.0 = Release|Any CPU 39 | EndGlobalSection 40 | GlobalSection(SolutionProperties) = preSolution 41 | HideSolutionNode = FALSE 42 | EndGlobalSection 43 | GlobalSection(ExtensibilityGlobals) = postSolution 44 | SolutionGuid = {41A23FC9-9A36-4199-8757-F764ADDD44E3} 45 | EndGlobalSection 46 | EndGlobal 47 | -------------------------------------------------------------------------------- /OneSTools.EventLog/DateTimeZoneExtensions.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using NodaTime; 3 | 4 | namespace OneSTools.EventLog 5 | { 6 | internal static class DateTimeZoneExtensions 7 | { 8 | public static DateTime ToUtc(this DateTimeZone dateTimeZone, DateTime dateTime) 9 | { 10 | return LocalDateTime.FromDateTime(dateTime).InZoneLeniently(dateTimeZone).ToDateTimeUtc(); 11 | } 12 | } 13 | } -------------------------------------------------------------------------------- /OneSTools.EventLog/EventLogItem.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | 3 | namespace OneSTools.EventLog 4 | { 5 | public class EventLogItem 6 | { 7 | public long Id { get; set; } = 0; 8 | public virtual string FileName { get; set; } = ""; 9 | public virtual long EndPosition { get; set; } = 0; 10 | public virtual long LgfEndPosition { get; set; } = 0; 11 | public virtual DateTime DateTime { get; set; } = DateTime.MinValue; 12 | public virtual string TransactionStatus { get; set; } = ""; 13 | public virtual DateTime TransactionDateTime { get; set; } = new DateTime(1970, 1, 1); 14 | public virtual long TransactionNumber { get; set; } = 0; 15 | public virtual string UserUuid { get; set; } = ""; 16 | public virtual string User { get; set; } = ""; 17 | public virtual string Computer { get; set; } = ""; 18 | public virtual string Application { get; set; } = ""; 19 | public virtual long Connection { get; set; } = 0; 20 | public virtual string Event { get; set; } = ""; 21 | public virtual string Severity { get; set; } = ""; 22 | public virtual string Comment { get; set; } = ""; 23 | public virtual string MetadataUuid { get; set; } = ""; 24 | public virtual string Metadata { get; set; } = ""; 25 | public virtual string Data { get; set; } = ""; 26 | public virtual string DataPresentation { get; set; } = ""; 27 | public virtual string Server { get; set; } = ""; 28 | public virtual int MainPort { get; set; } = 0; 29 | public virtual int AddPort { get; set; } = 0; 30 | public virtual long Session { get; set; } = 0; 31 | } 32 | } -------------------------------------------------------------------------------- /OneSTools.EventLog/EventLogReader.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.IO; 4 | using System.Linq; 5 | using System.Threading; 6 | 7 | namespace OneSTools.EventLog 8 | { 9 | /// 10 | /// Presents methods for reading 1C event log 11 | /// 12 | public class EventLogReader : IDisposable 13 | { 14 | private readonly EventLogReaderSettings _settings; 15 | private bool _disposedValue; 16 | private LgfReader _lgfReader; 17 | private ManualResetEvent _lgpChangedCreated; 18 | private FileSystemWatcher _lgpFilesWatcher; 19 | private LgpReader _lgpReader; 20 | 21 | public EventLogReader(EventLogReaderSettings settings) 22 | { 23 | _settings = settings; 24 | 25 | _lgfReader = new LgfReader(Path.Combine(_settings.LogFolder, "1Cv8.lgf")); 26 | _lgfReader.SetPosition(settings.LgfStartPosition); 27 | 28 | if (settings.LgpFileName != string.Empty) 29 | { 30 | var file = Path.Combine(_settings.LogFolder, settings.LgpFileName); 31 | 32 | _lgpReader = new LgpReader(file, settings.TimeZone, _lgfReader, settings.SkipEventsBeforeDate); 33 | _lgpReader.SetPosition(settings.LgpStartPosition); 34 | } 35 | } 36 | 37 | /// 38 | /// Current reader's "lgp" file name 39 | /// 40 | public string LgpFileName => _lgpReader.LgpFileName; 41 | 42 | public void Dispose() 43 | { 44 | // Не изменяйте этот код. Разместите код очистки в методе "Dispose(bool disposing)". 45 | Dispose(true); 46 | GC.SuppressFinalize(this); 47 | } 48 | 49 | /// 50 | /// The behaviour of the method depends on the mode of the reader. In the "live" mode it'll be waiting for an appearing 51 | /// of the new event item, otherwise It'll just return null 52 | /// 53 | /// Token for interrupting of the reader 54 | /// 55 | public EventLogItem ReadNextEventLogItem(CancellationToken cancellationToken = default) 56 | { 57 | if (_lgpReader == null) 58 | SetNextLgpReader(); 59 | 60 | if (_settings.LiveMode && _lgpFilesWatcher == null) 61 | StartLgpFilesWatcher(); 62 | 63 | EventLogItem item = null; 64 | 65 | while (!cancellationToken.IsCancellationRequested) 66 | { 67 | try 68 | { 69 | item = _lgpReader.ReadNextEventLogItem(cancellationToken); 70 | } 71 | catch (ObjectDisposedException) 72 | { 73 | item = null; 74 | _lgpReader = null; 75 | break; 76 | } 77 | 78 | if (item == null) 79 | { 80 | var newReader = SetNextLgpReader(); 81 | 82 | if (_settings.LiveMode) 83 | { 84 | if (!newReader) 85 | { 86 | _lgpChangedCreated.Reset(); 87 | 88 | var waitHandle = WaitHandle.WaitAny( 89 | new[] {_lgpChangedCreated, cancellationToken.WaitHandle}, _settings.ReadingTimeout); 90 | 91 | if (_settings.ReadingTimeout != Timeout.Infinite && waitHandle == WaitHandle.WaitTimeout) 92 | throw new EventLogReaderTimeoutException(); 93 | 94 | _lgpChangedCreated.Reset(); 95 | } 96 | } 97 | else 98 | { 99 | if (!newReader) 100 | break; 101 | } 102 | } 103 | else 104 | { 105 | _settings.ItemId++; 106 | item.Id = _settings.ItemId; 107 | 108 | break; 109 | } 110 | } 111 | 112 | return item; 113 | } 114 | 115 | private bool SetNextLgpReader() 116 | { 117 | var currentReaderLastWriteDateTime = DateTime.MinValue; 118 | 119 | if (_lgpReader != null) 120 | currentReaderLastWriteDateTime = new FileInfo(_lgpReader.LgpPath).LastWriteTime; 121 | else if(_settings.SkipEventsBeforeDate != DateTime.MinValue) 122 | currentReaderLastWriteDateTime = _settings.SkipEventsBeforeDate.AddSeconds(-1); 123 | 124 | var filesDateTime = new List<(string, DateTime)>(); 125 | 126 | var files = Directory.GetFiles(_settings.LogFolder, "*.lgp"); 127 | 128 | foreach (var file in files) 129 | if (_lgpReader != null) 130 | { 131 | if (_lgpReader.LgpPath != file) 132 | filesDateTime.Add((file, new FileInfo(file).LastWriteTime)); 133 | } 134 | else 135 | { 136 | filesDateTime.Add((file, new FileInfo(file).LastWriteTime)); 137 | } 138 | 139 | var orderedFiles = filesDateTime.OrderBy(c => c.Item2).ToList(); 140 | 141 | var (item1, _) = orderedFiles.FirstOrDefault(c => c.Item2 > currentReaderLastWriteDateTime); 142 | 143 | if (string.IsNullOrEmpty(item1)) 144 | { 145 | return false; 146 | } 147 | 148 | _lgpReader?.Dispose(); 149 | _lgpReader = null; 150 | 151 | _lgpReader = new LgpReader(item1, _settings.TimeZone, _lgfReader, _settings.SkipEventsBeforeDate); 152 | 153 | return true; 154 | } 155 | 156 | private void StartLgpFilesWatcher() 157 | { 158 | _lgpChangedCreated = new ManualResetEvent(false); 159 | 160 | _lgpFilesWatcher = new FileSystemWatcher(_settings.LogFolder, "*.lgp") 161 | { 162 | NotifyFilter = NotifyFilters.CreationTime | NotifyFilters.LastWrite | NotifyFilters.Size 163 | }; 164 | _lgpFilesWatcher.Changed += LgpFilesWatcher_Event; 165 | _lgpFilesWatcher.Created += LgpFilesWatcher_Event; 166 | _lgpFilesWatcher.EnableRaisingEvents = true; 167 | } 168 | 169 | private void LgpFilesWatcher_Event(object sender, FileSystemEventArgs e) 170 | { 171 | if (e.ChangeType == WatcherChangeTypes.Created || e.ChangeType == WatcherChangeTypes.Changed) 172 | _lgpChangedCreated.Set(); 173 | } 174 | 175 | protected virtual void Dispose(bool disposing) 176 | { 177 | if (!_disposedValue) 178 | { 179 | _lgpFilesWatcher?.Dispose(); 180 | _lgpFilesWatcher = null; 181 | _lgpChangedCreated?.Dispose(); 182 | _lgpChangedCreated = null; 183 | _lgfReader?.Dispose(); 184 | _lgfReader = null; 185 | _lgpReader?.Dispose(); 186 | _lgpReader = null; 187 | 188 | _disposedValue = true; 189 | } 190 | } 191 | 192 | ~EventLogReader() 193 | { 194 | Dispose(false); 195 | } 196 | } 197 | } -------------------------------------------------------------------------------- /OneSTools.EventLog/EventLogReaderSettings.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Threading; 3 | using NodaTime; 4 | 5 | namespace OneSTools.EventLog 6 | { 7 | public class EventLogReaderSettings 8 | { 9 | public string LogFolder { get; set; } = ""; 10 | public bool LiveMode { get; set; } = true; 11 | public string LgpFileName { get; set; } = ""; 12 | public long LgpStartPosition { get; set; } = 0; 13 | public long LgfStartPosition { get; set; } = 0; 14 | public long ItemId { get; set; } = 0; 15 | public int ReadingTimeout { get; set; } = Timeout.Infinite; 16 | public DateTimeZone TimeZone { get; set; } = DateTimeZoneProviders.Tzdb.GetSystemDefault(); 17 | public DateTime SkipEventsBeforeDate { get; set; } 18 | } 19 | } -------------------------------------------------------------------------------- /OneSTools.EventLog/EventLogReaderTimeoutException.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | 3 | namespace OneSTools.EventLog 4 | { 5 | public class EventLogReaderTimeoutException : Exception 6 | { 7 | } 8 | } -------------------------------------------------------------------------------- /OneSTools.EventLog/IEventLogItem.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | 3 | namespace OneSTools.EventLog 4 | { 5 | public interface IEventLogItem 6 | { 7 | int AddPort { get; set; } 8 | string Application { get; set; } 9 | string Comment { get; set; } 10 | string Computer { get; set; } 11 | long Connection { get; set; } 12 | string Data { get; set; } 13 | string DataPresentation { get; set; } 14 | DateTime DateTime { get; set; } 15 | long EndPosition { get; set; } 16 | long LgfEndPosition { get; set; } 17 | string Event { get; set; } 18 | string FileName { get; set; } 19 | int MainPort { get; set; } 20 | string Metadata { get; set; } 21 | string MetadataUuid { get; set; } 22 | string Server { get; set; } 23 | long Session { get; set; } 24 | string Severity { get; set; } 25 | DateTime TransactionDateTime { get; set; } 26 | long TransactionNumber { get; set; } 27 | string TransactionStatus { get; set; } 28 | string User { get; set; } 29 | string UserUuid { get; set; } 30 | } 31 | } -------------------------------------------------------------------------------- /OneSTools.EventLog/LgfReader.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Collections.Generic; 3 | using System.IO; 4 | using System.Threading; 5 | using OneSTools.BracketsFile; 6 | 7 | namespace OneSTools.EventLog 8 | { 9 | internal class LgfReader : IDisposable 10 | { 11 | private BracketsListReader _bracketsReader; 12 | private bool _disposedValue; 13 | private FileStream _fileStream; 14 | 15 | /// 16 | /// Key tuple - first value is an object type, second value is an object number in the array 17 | /// 18 | private Dictionary<(ObjectType, int), string> _objects = new Dictionary<(ObjectType, int), string>(); 19 | 20 | /// 21 | /// Key tuple - first value is an object type, second value is an object number in the array 22 | /// Value tuple - first value is an object value, second value is a guid of the object value 23 | /// 24 | private Dictionary<(ObjectType, int), (string, string)> _referencedObjects = 25 | new Dictionary<(ObjectType, int), (string, string)>(); 26 | 27 | public LgfReader(string lgfPath) 28 | { 29 | LgfPath = lgfPath; 30 | } 31 | 32 | public string LgfPath { get; } 33 | 34 | public void Dispose() 35 | { 36 | Dispose(true); 37 | GC.SuppressFinalize(this); 38 | } 39 | 40 | private void ReadTill(ObjectType objectType, int number, long position, 41 | CancellationToken cancellationToken = default) 42 | { 43 | InitializeStreams(); 44 | 45 | var stop = false; 46 | 47 | while (!stop && !_bracketsReader.EndOfStream && !cancellationToken.IsCancellationRequested) 48 | { 49 | var itemData = _bracketsReader.NextNode(); 50 | 51 | var ot = (ObjectType) (int) itemData[0]; 52 | 53 | // Skip unknown object types 54 | if (ot >= ObjectType.Unknown) 55 | continue; 56 | 57 | switch (ot) 58 | { 59 | case ObjectType.Users: 60 | case ObjectType.Metadata: 61 | var key = (ot, (int) itemData[3]); 62 | var value = ((string) itemData[2], (string) itemData[1]); 63 | 64 | if (_referencedObjects.ContainsKey(key)) 65 | _referencedObjects.Remove(key); 66 | 67 | _referencedObjects.Add(key, value); 68 | 69 | if (objectType == ObjectType.None && GetPosition() >= position) 70 | { 71 | stop = true; 72 | break; 73 | } 74 | 75 | if (ot == objectType && key.Item2 == number) 76 | stop = true; 77 | 78 | break; 79 | default: 80 | var key1 = (ot, (int) itemData[2]); 81 | var value1 = (string) itemData[1]; 82 | 83 | if (_objects.ContainsKey(key1)) 84 | _objects.Remove(key1); 85 | 86 | _objects.Add(key1, value1); 87 | 88 | if (objectType == ObjectType.None && GetPosition() >= position) 89 | { 90 | stop = true; 91 | break; 92 | } 93 | 94 | if (ot == objectType && key1.Item2 == number) 95 | stop = true; 96 | 97 | break; 98 | } 99 | } 100 | } 101 | 102 | public string GetObjectValue(ObjectType objectType, int number, CancellationToken cancellationToken = default) 103 | { 104 | if (number == 0) 105 | return ""; 106 | 107 | if (_objects.TryGetValue((objectType, number), out var value)) 108 | return value; 109 | ReadTill(objectType, number, 0, cancellationToken); 110 | 111 | if (_objects.TryGetValue((objectType, number), out value)) 112 | return value; 113 | 114 | return null; 115 | } 116 | 117 | public (string Value, string Uuid) GetReferencedObjectValue(ObjectType objectType, int number, 118 | CancellationToken cancellationToken = default) 119 | { 120 | if (number == 0) 121 | return ("", ""); 122 | 123 | if (_referencedObjects.TryGetValue((objectType, number), out var value)) 124 | return value; 125 | ReadTill(objectType, number, 0, cancellationToken); 126 | 127 | if (_referencedObjects.TryGetValue((objectType, number), out value)) 128 | return value; 129 | 130 | return (null, null); 131 | } 132 | 133 | private void InitializeStreams() 134 | { 135 | if (_fileStream is null) 136 | { 137 | if (!File.Exists(LgfPath)) 138 | throw new Exception("Cannot find \"1Cv8.lgf\""); 139 | 140 | _fileStream = new FileStream(LgfPath, FileMode.Open, FileAccess.Read, 141 | FileShare.ReadWrite | FileShare.Delete); 142 | _bracketsReader = new BracketsListReader(_fileStream); 143 | } 144 | } 145 | 146 | public void SetPosition(long position, CancellationToken cancellationToken = default) 147 | { 148 | ReadTill(ObjectType.None, 0, position, cancellationToken); 149 | } 150 | 151 | public long GetPosition() 152 | { 153 | InitializeStreams(); 154 | 155 | return _bracketsReader.Position; 156 | } 157 | 158 | protected virtual void Dispose(bool disposing) 159 | { 160 | if (!_disposedValue) 161 | { 162 | if (disposing) 163 | { 164 | _objects = null; 165 | _referencedObjects = null; 166 | } 167 | 168 | _bracketsReader?.Dispose(); 169 | _bracketsReader = null; 170 | _fileStream?.Dispose(); 171 | _fileStream = null; 172 | 173 | _disposedValue = true; 174 | } 175 | } 176 | 177 | ~LgfReader() 178 | { 179 | Dispose(false); 180 | } 181 | } 182 | } -------------------------------------------------------------------------------- /OneSTools.EventLog/LgpReader.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.Globalization; 3 | using System.IO; 4 | using System.Text; 5 | using System.Threading; 6 | using NodaTime; 7 | using OneSTools.BracketsFile; 8 | 9 | namespace OneSTools.EventLog 10 | { 11 | internal class LgpReader : IDisposable 12 | { 13 | private readonly DateTimeZone _timeZone; 14 | private BracketsListReader _bracketsReader; 15 | private bool _disposedValue; 16 | private FileStream _fileStream; 17 | private LgfReader _lgfReader; 18 | private FileSystemWatcher _lgpFileWatcher; 19 | private DateTime _skipEventsBeforeDate; 20 | 21 | public LgpReader(string lgpPath, DateTimeZone timeZone, LgfReader lgfReader, DateTime skipEventsBeforeDate) 22 | { 23 | LgpPath = lgpPath; 24 | _timeZone = timeZone; 25 | _lgfReader = lgfReader; 26 | _skipEventsBeforeDate = skipEventsBeforeDate; 27 | } 28 | 29 | public string LgpPath { get; } 30 | public string LgpFileName => Path.GetFileName(LgpPath); 31 | 32 | public void Dispose() 33 | { 34 | Dispose(true); 35 | GC.SuppressFinalize(this); 36 | } 37 | 38 | public EventLogItem ReadNextEventLogItem(CancellationToken cancellationToken = default) 39 | { 40 | if (_disposedValue) 41 | throw new ObjectDisposedException(nameof(LgpReader)); 42 | 43 | InitializeStreams(); 44 | 45 | return ReadEventLogItemData(cancellationToken); 46 | } 47 | 48 | public void SetPosition(long position) 49 | { 50 | InitializeStreams(); 51 | 52 | _bracketsReader.Position = position; 53 | } 54 | 55 | private void InitializeStreams() 56 | { 57 | if (_fileStream is null) 58 | { 59 | if (!File.Exists(LgpPath)) 60 | throw new Exception($"Cannot find lgp file by path {LgpPath}"); 61 | 62 | _lgpFileWatcher = new FileSystemWatcher(Path.GetDirectoryName(LgpPath)!, "*.lgp") 63 | { 64 | NotifyFilter = NotifyFilters.CreationTime | NotifyFilters.LastWrite | NotifyFilters.FileName | 65 | NotifyFilters.Attributes 66 | }; 67 | _lgpFileWatcher.Deleted += LgpFileWatcher_Deleted; 68 | _lgpFileWatcher.EnableRaisingEvents = true; 69 | 70 | _fileStream = new FileStream(LgpPath, FileMode.Open, FileAccess.Read, 71 | FileShare.ReadWrite | FileShare.Delete); 72 | _bracketsReader = new BracketsListReader(_fileStream); 73 | } 74 | } 75 | 76 | private void LgpFileWatcher_Deleted(object sender, FileSystemEventArgs e) 77 | { 78 | if (e.ChangeType == WatcherChangeTypes.Deleted && LgpPath == e.FullPath) Dispose(); 79 | } 80 | 81 | private (StringBuilder Data, long EndPosition) ReadNextEventLogItemData() 82 | { 83 | var data = _bracketsReader.NextNodeAsStringBuilder(); 84 | 85 | return (data, _bracketsReader.Position); 86 | } 87 | 88 | private EventLogItem ReadEventLogItemData(CancellationToken cancellationToken = default) 89 | { 90 | while (true) 91 | { 92 | var (data, endPosition) = ReadNextEventLogItemData(); 93 | if (data.Length == 0) 94 | return null; 95 | 96 | var eventLogItem = ParseEventLogItemData(data, endPosition, cancellationToken); 97 | if (eventLogItem != null) 98 | return eventLogItem; 99 | } 100 | } 101 | 102 | private EventLogItem ParseEventLogItemData(StringBuilder eventLogItemData, long endPosition, 103 | CancellationToken cancellationToken = default) 104 | { 105 | var parsedData = BracketsParser.ParseBlock(eventLogItemData); 106 | 107 | DateTime dateTime = default; 108 | try 109 | { 110 | dateTime = _timeZone.ToUtc(DateTime.ParseExact(parsedData[0], "yyyyMMddHHmmss", 111 | CultureInfo.InvariantCulture)); 112 | } 113 | catch 114 | { 115 | dateTime = DateTime.MinValue; 116 | } 117 | 118 | if (dateTime < _skipEventsBeforeDate) 119 | return null; 120 | 121 | var eventLogItem = new EventLogItem 122 | { 123 | DateTime = dateTime, 124 | TransactionStatus = GetTransactionPresentation(parsedData[1]), 125 | FileName = LgpFileName, 126 | EndPosition = endPosition, 127 | LgfEndPosition = _lgfReader.GetPosition() 128 | }; 129 | 130 | var transactionData = parsedData[2]; 131 | eventLogItem.TransactionNumber = Convert.ToInt64(transactionData[1], 16); 132 | 133 | var transactionDate = new DateTime().AddSeconds(Convert.ToInt64(transactionData[0], 16) / 10000); 134 | eventLogItem.TransactionDateTime = transactionDate == DateTime.MinValue 135 | ? transactionDate 136 | : _timeZone.ToUtc(transactionDate); 137 | 138 | var (value, uuid) = _lgfReader.GetReferencedObjectValue(ObjectType.Users, parsedData[3], cancellationToken); 139 | eventLogItem.UserUuid = uuid; 140 | eventLogItem.User = value; 141 | 142 | eventLogItem.Computer = _lgfReader.GetObjectValue(ObjectType.Computers, parsedData[4], cancellationToken); 143 | 144 | var application = _lgfReader.GetObjectValue(ObjectType.Applications, parsedData[5], cancellationToken); 145 | eventLogItem.Application = GetApplicationPresentation(application); 146 | 147 | eventLogItem.Connection = parsedData[6]; 148 | 149 | var ev = _lgfReader.GetObjectValue(ObjectType.Events, parsedData[7], cancellationToken); 150 | eventLogItem.Event = GetEventPresentation(ev); 151 | 152 | var severity = (string)parsedData[8]; 153 | eventLogItem.Severity = GetSeverityPresentation(severity); 154 | 155 | eventLogItem.Comment = parsedData[9]; 156 | 157 | (value, uuid) = _lgfReader.GetReferencedObjectValue(ObjectType.Metadata, parsedData[10], cancellationToken); 158 | eventLogItem.MetadataUuid = uuid; 159 | eventLogItem.Metadata = value; 160 | 161 | eventLogItem.Data = GetData(parsedData[11]).Trim(); 162 | eventLogItem.DataPresentation = parsedData[12]; 163 | eventLogItem.Server = _lgfReader.GetObjectValue(ObjectType.Servers, parsedData[13], cancellationToken); 164 | 165 | var mainPort = _lgfReader.GetObjectValue(ObjectType.MainPorts, parsedData[14], cancellationToken); 166 | if (mainPort != "") 167 | eventLogItem.MainPort = int.Parse(mainPort); 168 | 169 | var addPort = _lgfReader.GetObjectValue(ObjectType.AddPorts, parsedData[15], cancellationToken); 170 | if (addPort != "") 171 | eventLogItem.AddPort = int.Parse(addPort); 172 | 173 | eventLogItem.Session = parsedData[16]; 174 | 175 | return eventLogItem; 176 | } 177 | 178 | private static string GetData(BracketsNode node) 179 | { 180 | var dataType = (string)node[0]; 181 | 182 | switch (dataType) 183 | { 184 | case "R": // Reference 185 | return node[1]; 186 | case "U": // Undefined 187 | return ""; 188 | case "S": // String 189 | return node[1]; 190 | case "B": // Boolean 191 | return (string)node[1] == "0" ? "false" : "true"; 192 | case "P": // Complex data 193 | var str = new StringBuilder(); 194 | 195 | var subDataNode = node[1]; 196 | 197 | //var subDataType = (int)subDataNode[0]; 198 | // What's known (subDataNode): 199 | // 1 - additional data of "Authentication (Windows auth) in thin or thick client" 200 | // 2 - additional data of "Authentication in COM connection" event 201 | // 6 - additional data of "Authentication in thin or thick client" event 202 | // 11 - additional data of "Access denied" event 203 | 204 | // I hope this is temporarily method 205 | var subDataCount = subDataNode.Count - 1; 206 | 207 | if (subDataCount > 0) 208 | for (var i = 1; i <= subDataCount; i++) 209 | { 210 | var value = GetData(subDataNode[i]); 211 | 212 | if (value != string.Empty) 213 | str.AppendLine($"Item {i}: {value}"); 214 | } 215 | 216 | return str.ToString(); 217 | default: 218 | return ""; 219 | } 220 | } 221 | 222 | private static string GetTransactionPresentation(string str) 223 | { 224 | return str switch 225 | { 226 | "U" => "Зафиксирована", 227 | "C" => "Отменена", 228 | "R" => "Не завершена", 229 | "N" => "Нет транзакции", 230 | _ => "" 231 | }; 232 | } 233 | 234 | private static string GetSeverityPresentation(string str) 235 | { 236 | return str switch 237 | { 238 | "I" => "Информация", 239 | "E" => "Ошибка", 240 | "W" => "Предупреждение", 241 | "N" => "Примечание", 242 | _ => "" 243 | }; 244 | } 245 | 246 | private static string GetApplicationPresentation(string str) 247 | { 248 | return str switch 249 | { 250 | "1CV8" => "Толстый клиент", 251 | "1CV8C" => "Тонкий клиент", 252 | "WebClient" => "Веб-клиент", 253 | "Designer" => "Конфигуратор", 254 | "COMConnection" => "Внешнее соединение (COM, обычное)", 255 | "WSConnection" => "Сессия web-сервиса", 256 | "BackgroundJob" => "Фоновое задание", 257 | "SystemBackgroundJob" => "Системное фоновое задание", 258 | "SrvrConsole" => "Консоль кластера", 259 | "COMConsole" => "Внешнее соединение (COM, административное)", 260 | "JobScheduler" => "Планировщик заданий", 261 | "Debugger" => "Отладчик", 262 | "RAS" => "Сервер администрирования", 263 | _ => str 264 | }; 265 | } 266 | 267 | private static string GetEventPresentation(string str) 268 | { 269 | return str switch 270 | { 271 | "_$Access$_.Access" => "Доступ.Доступ", 272 | "_$Access$_.AccessDenied" => "Доступ.Отказ в доступе", 273 | "_$Data$_.Delete" => "Данные.Удаление", 274 | "_$Data$_.DeletePredefinedData" => " Данные.Удаление предопределенных данных", 275 | "_$Data$_.DeleteVersions" => "Данные.Удаление версий", 276 | "_$Data$_.New" => "Данные.Добавление", 277 | "_$Data$_.NewPredefinedData" => "Данные.Добавление предопределенных данных", 278 | "_$Data$_.NewVersion" => "Данные.Добавление версии", 279 | "_$Data$_.Pos" => "Данные.Проведение", 280 | "_$Data$_.PredefinedDataInitialization" => "Данные.Инициализация предопределенных данных", 281 | "_$Data$_.PredefinedDataInitializationDataNotFound" => 282 | "Данные.Инициализация предопределенных данных.Данные не найдены", 283 | "_$Data$_.SetPredefinedDataInitialization" => "Данные.Установка инициализации предопределенных данных", 284 | "_$Data$_.SetStandardODataInterfaceContent" => "Данные.Изменение состава стандартного интерфейса OData", 285 | "_$Data$_.TotalsMaxPeriodUpdate" => "Данные.Изменение максимального периода рассчитанных итогов", 286 | "_$Data$_.TotalsMinPeriodUpdate" => "Данные.Изменение минимального периода рассчитанных итогов", 287 | "_$Data$_.Post" => "Данные.Проведение", 288 | "_$Data$_.Unpost" => "Данные.Отмена проведения", 289 | "_$Data$_.Update" => "Данные.Изменение", 290 | "_$Data$_.UpdatePredefinedData" => "Данные.Изменение предопределенных данных", 291 | "_$Data$_.VersionCommentUpdate" => "Данные.Изменение комментария версии", 292 | "_$InfoBase$_.ConfigExtensionUpdate" => "Информационная база.Изменение расширения конфигурации", 293 | "_$InfoBase$_.ConfigUpdate" => "Информационная база.Изменение конфигурации", 294 | "_$InfoBase$_.DBConfigBackgroundUpdateCancel" => "Информационная база.Отмена фонового обновления", 295 | "_$InfoBase$_.DBConfigBackgroundUpdateFinish" => "Информационная база.Завершение фонового обновления", 296 | "_$InfoBase$_.DBConfigBackgroundUpdateResume" => 297 | "Информационная база.Продолжение (после приостановки) процесса фонового обновления", 298 | "_$InfoBase$_.DBConfigBackgroundUpdateStart" => "Информационная база.Запуск фонового обновления", 299 | "_$InfoBase$_.DBConfigBackgroundUpdateSuspend" => 300 | "Информационная база.Приостановка (пауза) процесса фонового обновления", 301 | "_$InfoBase$_.DBConfigExtensionUpdate" => "Информационная база.Изменение расширения конфигурации", 302 | "_$InfoBase$_.DBConfigExtensionUpdateError" => 303 | "Информационная база.Ошибка изменения расширения конфигурации", 304 | "_$InfoBase$_.DBConfigUpdate" => "Информационная база.Изменение конфигурации базы данных", 305 | "_$InfoBase$_.DBConfigUpdateStart" => "Информационная база.Запуск обновления конфигурации базы данных", 306 | "_$InfoBase$_.DumpError" => "Информационная база.Ошибка выгрузки в файл", 307 | "_$InfoBase$_.DumpFinish" => "Информационная база.Окончание выгрузки в файл", 308 | "_$InfoBase$_.DumpStart" => "Информационная база.Начало выгрузки в файл", 309 | "_$InfoBase$_.EraseData" => " Информационная база.Удаление данных информационной баз", 310 | "_$InfoBase$_.EventLogReduce" => "Информационная база.Сокращение журнала регистрации", 311 | "_$InfoBase$_.EventLogReduceError" => "Информационная база.Ошибка сокращения журнала регистрации", 312 | "_$InfoBase$_.EventLogSettingsUpdate" => "Информационная база.Изменение параметров журнала регистрации", 313 | "_$InfoBase$_.EventLogSettingsUpdateError" => 314 | "Информационная база.Ошибка при изменение настроек журнала регистрации", 315 | "_$InfoBase$_.InfoBaseAdmParamsUpdate" => 316 | "Информационная база.Изменение параметров информационной базы", 317 | "_$InfoBase$_.InfoBaseAdmParamsUpdateError" => 318 | "Информационная база.Ошибка изменения параметров информационной базы", 319 | "_$InfoBase$_.IntegrationServiceActiveUpdate" => 320 | "Информационная база.Изменение активности сервиса интеграции", 321 | "_$InfoBase$_.IntegrationServiceSettingsUpdate" => 322 | "Информационная база.Изменение настроек сервиса интеграции", 323 | "_$InfoBase$_.MasterNodeUpdate" => "Информационная база.Изменение главного узла", 324 | "_$InfoBase$_.PredefinedDataUpdate" => "Информационная база.Обновление предопределенных данных", 325 | "_$InfoBase$_.RegionalSettingsUpdate" => "Информационная база.Изменение региональных установок", 326 | "_$InfoBase$_.RestoreError" => "Информационная база.Ошибка загрузки из файла", 327 | "_$InfoBase$_.RestoreFinish" => "Информационная база.Окончание загрузки из файла", 328 | "_$InfoBase$_.RestoreStart" => "Информационная база.Начало загрузки из файла", 329 | "_$InfoBase$_.SecondFactorAuthTemplateDelete" => 330 | "Информационная база.Удаление шаблона вторго фактора аутентификации", 331 | "_$InfoBase$_.SecondFactorAuthTemplateNew" => 332 | "Информационная база.Добавление шаблона вторго фактора аутентификации", 333 | "_$InfoBase$_.SecondFactorAuthTemplateUpdate" => 334 | "Информационная база.Изменение шаблона вторго фактора аутентификации", 335 | "_$InfoBase$_.SetPredefinedDataUpdate" => 336 | "Информационная база.Установить обновление предопределенных данных", 337 | "_$InfoBase$_.TARImportant" => "Тестирование и исправление.Ошибка", 338 | "_$InfoBase$_.TARInfo" => "Тестирование и исправление.Сообщение", 339 | "_$InfoBase$_.TARMess" => "Тестирование и исправление.Предупреждение", 340 | "_$Job$_.Cancel" => "Фоновое задание.Отмена", 341 | "_$Job$_.Fail" => "Фоновое задание.Ошибка выполнения", 342 | "_$Job$_.Start" => "Фоновое задание.Запуск", 343 | "_$Job$_.Succeed" => "Фоновое задание.Успешное завершение", 344 | "_$Job$_.Terminate" => "Фоновое задание.Принудительное завершение", 345 | "_$OpenIDProvider$_.NegativeAssertion" => "Провайдер OpenID.Отклонено", 346 | "_$OpenIDProvider$_.PositiveAssertion" => "Провайдер OpenID.Подтверждено", 347 | "_$PerformError$_" => "Ошибка выполнения", 348 | "_$Session$_.Authentication" => "Сеанс.Аутентификация", 349 | "_$Session$_.AuthenticationError" => "Сеанс.Ошибка аутентификации", 350 | "_$Session$_.AuthenticationFirstFactor" => "Сеанс.Аутентификация первый фактор", 351 | "_$Session$_.ConfigExtensionApplyError" => "Сеанс.Ошибка применения расширения конфигурации", 352 | "_$Session$_.Finish" => "Сеанс.Завершение", 353 | "_$Session$_.Start" => "Сеанс.Начало", 354 | "_$Transaction$_.Begin" => "Транзакция.Начало", 355 | "_$Transaction$_.Commit" => "Транзакция.Фиксация", 356 | "_$Transaction$_.Rollback" => "Транзакция.Отмена", 357 | "_$User$_.AuthenticationLock" => "Пользователи.Блокировка аутентификации", 358 | "_$User$_.AuthenticationUnlock" => "Пользователи.Разблокировка аутентификации", 359 | "_$User$_.AuthenticationUnlockError " => "Пользователи.Ошибка разблокировки аутентификации", 360 | "_$User$_.Delete" => "Пользователи.Удаление", 361 | "_$User$_.DeleteError" => "Пользователи.Ошибка удаления", 362 | "_$User$_.New" => "Пользователи.Добавление", 363 | "_$User$_.NewError" => "Пользователи.Ошибка добавления", 364 | "_$User$_.Update" => "Пользователи.Изменение", 365 | "_$User$_.UpdateError" => "Пользователи. Ошибка изменения", 366 | _ => str 367 | }; 368 | } 369 | 370 | protected virtual void Dispose(bool disposing) 371 | { 372 | if (_disposedValue) return; 373 | 374 | _bracketsReader?.Dispose(); 375 | _bracketsReader = null; 376 | _fileStream = null; 377 | 378 | _lgpFileWatcher?.Dispose(); 379 | _lgpFileWatcher = null; 380 | 381 | _lgfReader = null; 382 | 383 | _disposedValue = true; 384 | } 385 | 386 | ~LgpReader() 387 | { 388 | Dispose(false); 389 | } 390 | } 391 | } -------------------------------------------------------------------------------- /OneSTools.EventLog/ObjectType.cs: -------------------------------------------------------------------------------- 1 | namespace OneSTools.EventLog 2 | { 3 | internal enum ObjectType 4 | { 5 | None = 0, 6 | Users = 1, 7 | Computers = 2, 8 | Applications = 3, 9 | Events = 4, 10 | Metadata = 5, 11 | Servers = 6, 12 | MainPorts = 7, 13 | AddPorts = 8, 14 | Unknown = 9 15 | } 16 | } -------------------------------------------------------------------------------- /OneSTools.EventLog/OneSTools.EventLog.csproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | netstandard2.1 5 | 8.0 6 | OneSTools.EventLog 7 | OneSTools.EventLog 8 | onestools_icon_nuget.png 9 | true 10 | Библиотека для чтения и парсинга журнала регистрации 1С 11 | Akpaev Evgeny 12 | Akpaev Evgeny 13 | Akpaev Evgeny 14 | https://github.com/akpaevj/OneSTools.EventLog 15 | 16 | https://github.com/akpaevj/OneSTools.EventLog 17 | LICENSE 18 | 1.2.6 19 | 20 | 21 | 22 | portable 23 | true 24 | 25 | ..\Build\Debug 26 | 27 | 28 | 29 | full 30 | true 31 | ..\Build\Release 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | True 43 | 44 | 45 | 46 | True 47 | 48 | 49 | 50 | 51 | 52 | -------------------------------------------------------------------------------- /OneSTools.EventLog/README.md: -------------------------------------------------------------------------------- 1 | # OneSTools.EventLog 2 | [![Nuget](https://img.shields.io/nuget/v/OneSTools.EventLog)](https://www.nuget.org/packages/OneSTools.EventLog)
3 | Библиотека для чтения и парсинга данных журнала регистрации 1С 4 | 5 | Предоставляет легкий интерфейс для чтения журнала регистрации. Позволяет работать в "live" режиме, либо считывать только доступные данные журнала. 6 | 7 | Чтение журнала регистрации осуществляется с помощью класса **EventLogReader**. Конструктор класса принимает в качестве аргумента экземпляр класса **EventLogReaderSettings**, содержащий свойства: 8 | 1. *LogFolder* - путь к каталогу журнала регистрации 1С. 9 | 2. *LiveMode* - если свойство выставлено в true, то при достижении конца файла метод **ReadNextEventLogItem** не будет возвращать null, а будет ожидать появления в файле новой порции данных. Если false - то при достижении конца файла, вместо экземпляра класса **EventLogItem** будет возвращено null. 10 | 3. *ReadingTimeout* - Количество миллисекунд, сколько класс будет ожидать появления новых данных при достижении конца файла. Имеет эффект только если свойство **LiveMode** выставлено в true. По истечении указанного времени будет вызвано исключение **EventLogReaderTimeoutException**. 11 | 4. *LgpFileName* - наименование **lgp** файла, с которого требуется начать чтение журнала. 12 | 5. *LgpStartPosition* - позиция (в байтах) **lgp** файла, с которой требуется начать чтение журнала. 13 | 6. *LgfStartPosition* - позиция (в байтах) **lgf** файла, с которой требуется начать чтение журнала. 14 | 7. *ItemId* - номер, с которого будет начинаться нумерация считываемых событий (свойство **Id** класса **EventLogItem**). 15 | 8. *TimeZone* - часовой пояс (в формате IANA Time Zone Database), в котором записан журнал регистрации. По умолчанию - часовой пояс системы. 16 | 17 | *Параметры 4-7 нужны, если предполагается использование библиотеки в качестве ядра службы, работа которой может быть приостановлена и возобновлена в любое время. Предполагается что служба может возобновить состояние ридера с последующим продолжением чтения данных с последней сохраненной (во внешней системе) позиции.* 18 | 19 | Пример использования библиотеки: 20 | ```csharp 21 | var eventLogReaderSettings = new EventLogReaderSettings 22 | { 23 | LogFolder = "C:\\LogFolder", 24 | LiveMode = true, 25 | ReadingTimeout = 2 * 1000 26 | }; 27 | 28 | var eventLogReader = new EventLogReader(settings); 29 | 30 | try 31 | { 32 | // or wait for a cancellationToken cancel request 33 | while (true) 34 | { 35 | var item = eventLogReader.ReadNextEventLogItem(); 36 | 37 | if (item is null) 38 | break; 39 | else 40 | { 41 | // to do something with the event data 42 | } 43 | } 44 | } 45 | catch (EventLogReaderTimeoutException) 46 | { 47 | // timeout occurred, it must catch it if "LiveMode" is enabled 48 | } 49 | catch (Exception ex) 50 | { 51 | // something went wrong 52 | } 53 | ``` 54 | -------------------------------------------------------------------------------- /OneSTools.EventLog/StreamReaderExtensions.cs: -------------------------------------------------------------------------------- 1 | using System; 2 | using System.IO; 3 | using System.Reflection; 4 | 5 | namespace OneSTools.EventLog 6 | { 7 | internal static class StreamReaderExtensions 8 | { 9 | private static readonly FieldInfo charPosField = typeof(StreamReader).GetField("_charPos", 10 | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.DeclaredOnly); 11 | 12 | private static readonly FieldInfo byteLenField = typeof(StreamReader).GetField("_byteLen", 13 | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.DeclaredOnly); 14 | 15 | private static readonly FieldInfo charBufferField = typeof(StreamReader).GetField("_charBuffer", 16 | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.DeclaredOnly); 17 | 18 | public static long GetPosition(this StreamReader reader) 19 | { 20 | // shift position back from BaseStream.Position by the number of bytes read 21 | // into internal buffer. 22 | var byteLen = (int) byteLenField.GetValue(reader); 23 | var position = reader.BaseStream.Position - byteLen; 24 | 25 | // if we have consumed chars from the buffer we need to calculate how many 26 | // bytes they represent in the current encoding and add that to the position. 27 | var charPos = (int) charPosField.GetValue(reader); 28 | if (charPos > 0) 29 | { 30 | var charBuffer = (char[]) charBufferField.GetValue(reader); 31 | var encoding = reader.CurrentEncoding; 32 | var bytesConsumed = encoding.GetBytes(charBuffer, 0, charPos).Length; 33 | position += bytesConsumed; 34 | } 35 | 36 | return position; 37 | } 38 | 39 | public static void SetPosition(this StreamReader reader, long position) 40 | { 41 | reader.DiscardBufferedData(); 42 | reader.BaseStream.Seek(position, SeekOrigin.Begin); 43 | 44 | if (reader.BaseStream.Position != position) 45 | throw new Exception("Couldn't set the stream position"); 46 | } 47 | } 48 | } -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Инструменты для чтения и экспорта журнала регистрации 1С 2 | Репозиторий содержит как библиотеки так и готовые инструменты для чтения и экспорта журнала регистрации 1С в ClickHouse и ElasticSearch. В основе служб экспорта находится pipeline (TPL Dataflow) обработка данных, за счет чего достигается высокая скорость экспорта с возможностью параметризации потребления ресурсов CPU<->RAM. 3 | 4 | Обзор инструмента и использование в Docker (от [Андрея Овсянкина](https://github.com/EvilBeaver)): 5 | [Сверхбыстрый Журнал Регистрации 1C с помощью Yandex Clickhouse](https://youtu.be/HnZ0Of-YpW0) 6 | 7 | ## Состав: 8 | |Наименование|Описание|Actions/Nuget| 9 | |:-----------|:-------|:------------:| 10 | |[OneSTools.EventLog](https://github.com/akpaevj/OneSTools.EventLog/tree/master/OneSTools.EventLog)|Библиотека для чтения журнала регистрации (старый формат, LGF и LGP файлы). Позволяет выполнять как разовое чтение данных, так и запуск в "live" режиме|[![Nuget](https://img.shields.io/nuget/v/OneSTools.EventLog)](https://www.nuget.org/packages/OneSTools.EventLog) ![EventLog .NET 5](https://github.com/akpaevj/OneSTools.EventLog/workflows/EventLog%20.NET%205/badge.svg)| 11 | |[OneSTools.EventLog.Exporter.Core](https://github.com/akpaevj/OneSTools.EventLog/tree/master/OneSTools.EventLog.Exporter.Core)|Библиотека-ядро для инструментов экспорта журнала регистрации|| 12 | |[EventLogExporter](https://github.com/akpaevj/OneSTools.EventLog/tree/master/OneSTools.EventLog.Exporter)|Служба для экспорта журнала регистрации в [ClickHouse](https://clickhouse.tech/) и [ElasticSearch](https://www.elastic.co/)|![EventLogExporter .NET 5](https://github.com/akpaevj/OneSTools.EventLog/workflows/EventLogExporter%20.NET%205/badge.svg)| 13 | |[EventLogExportersManager](https://github.com/akpaevj/OneSTools.EventLog/tree/master/OneSTools.EventLog.Exporter.Manager)|Служба, выполняющая роль менеджера и наблюдающая каталоги серверов на предмет появления/удаления информационных баз с автоматическим подключением/отключением экспорта их журналов регистраций|| 14 | 15 | ## Get started: 16 | 17 | Инструмент поставляется с возможностью использования в 2 режимах: 18 | 1. С использованием менеджера экспортеров - это полноценный сервис наблюдения за кластером серверов 1С, автоматизирующий подключение/отключение экспорта ЖР информационных баз **(EventLogExportersManager.exe/dll)** 19 | 2. Без использования менеджера - отдельный процесс, обслуживающий ЖР конкретно указанной информационной базы **(EventLogExporter.exe/dll)** 20 | 21 | В случае использования приложения без менеджера, нет необходимости заполнять секцию *Manager* конфигурационного файла 22 | 23 | ### Конфигурация: 24 | Файл конфигурации (appsettings.json) разбит на несколько секций, каждая из которых отвечает за функциональность определенной части приложения. 25 | 26 | **Manager:** 27 | Секция настроек менеджера служб экспорта. 28 | ```json 29 | "Manager": { 30 | "ClstFolders": [ 31 | { 32 | "Folder": "C:\\\\Program Files\\1cv8\\srvinfo\\reg_1541", 33 | "Templates": [ 34 | { 35 | "Mask": "upp_main", 36 | "Template": "[IBNAME]_el" 37 | } 38 | ] 39 | }, 40 | { 41 | "Folder": "C:\\\\Program Files\\1cv8\\srvinfo\\reg_1542", 42 | "Templates": [ 43 | { 44 | "Mask": "bp3_.*", 45 | "Template": "[IBNAME]_el" 46 | } 47 | ] 48 | } 49 | ] 50 | } 51 | ``` 52 | где: 53 | 1. *ClstFolders* - массив описаний каталогов рабочих серверов (reg_*) 54 | 2. *Folder* - Путь к каталогу рабочего сервера (reg_*) 55 | 3. *Templates* - шаблоны обрабатываемых наименований информационных баз и правила наименования баз данных логов 56 | *Mask* - регулярное выражение, применяемое к имени информационной базы 57 | *Template* - шаблон имени базы данных хранения логов журнала. Обязательно должен содержать в себе переменную [IBNAME] 58 | 59 | Информационные базы, которые не попали ни под одну из масок будут пропущены, экспорт их журналов выполняться не будет 60 | 61 | **При использовании менеджера изменяются настройки для СУБД. При использовании ClickHouse из строки подключения нужно удалить параметр Database, менеджер автоматически создаст базу данных для каждой экспортируемой информационной базы с именем, которое будет определено в зависимости от выбранного шаблона наименования. При использовании ElasticSearch имя индекса будет определено таким же способом, поэтому параметр Index будет просто проигнорирован** 62 | 63 | **Exporter:** 64 | В этой секции размещены общие параметры экспортера, не зависящие от СУБД. 65 | ```json 66 | "Exporter": { 67 | "StorageType": 2, 68 | "LogFolder": "C:\\Users\\akpaev.e.ENTERPRISE\\Desktop\\1Cv8Log", 69 | "Portion": 10000, 70 | "TimeZone": "Europe/Moscow", 71 | "WritingMaxDegreeOfParallelism": 8, 72 | "CollectedFactor": 8, 73 | "ReadingTimeout": 1, 74 | "LoadArchive": false 75 | } 76 | ``` 77 | где: 78 | 1. *StorageType* - тип хранилища журнала регистрации. Может принимать значения: 79 | *1* - Clickhouse 80 | *2* - ElasticSearch 81 | 2. *LogFolder* - путь к каталогу журнала регистрации 1С 82 | 3. *Portion* - Размер порции, записываемый в БД за одну итерацию (10000 по умолчанию) 83 | 4. *TimeZone* - часовой пояс (в формате IANA Time Zone Database), в котором записан журнал регистрации. По умолчанию - часовой пояс системы 84 | 5. *WritingMaxDegreeOfParallelism* - количество потоков записи в СУБД. Т.к. в ClickHouse не поддерживаются одновременные BULK операции, то параметр имеет смысл только для ElasticSearch. По умолчанию - 1 85 | 6. *CollectedFactor* - коэффициент количества элементов, которые могут быть помещены в очередь записи. Предельное количество элементов равно Portion * CollectedFactor. По умолчанию - 2 86 | 7. *ReadingTimeout* - таймаут сброса данных при достижении конца файла (в секундах). По умолчанию - 1 сек. 87 | 8. *LoadArchive* - Специальный параметр, предназначенный для первоначальной загрузки архивных данных. При установке параметра в true, отключается "live" режим и не выполняется запрос последнего обработанного файла из БД 88 | 89 | **ClickHouse:** 90 | ```json 91 | "ConnectionStrings": { 92 | "Default": "Host=localhost;Port=8123;Username=default;password=;Database=database_name;" 93 | } 94 | ``` 95 | **ElasticSearch:** 96 | ```json 97 | "ElasticSearch": { 98 | "Nodes": [ 99 | { 100 | "Host": "http://192.168.0.95:9200", 101 | "AuthenticationType": "0" 102 | }, 103 | { 104 | "Host": "http://192.168.0.93:9200", 105 | "AuthenticationType": "1", 106 | "UserName": "", 107 | "Password": "" 108 | } 109 | { 110 | "Host": "http://192.168.0.94:9200", 111 | "AuthenticationType": "2", 112 | "Id": "", 113 | "ApiKey": "" 114 | } 115 | ], 116 | "Index": "upp-main-el", 117 | "Separation": "M", 118 | "MaximumRetries": 2, 119 | "MaxRetryTimeout": 30 120 | } 121 | ``` 122 | где:
123 | 1. *Nodes* - узел, содержащий хосты кластера ElasticSearch, либо один узел при работе с одной нодой. При недоступности текущего узла будет происходить переключение на следующий узел списка. Для узлов доступны 3 типа аутентификации:
124 | *0* - без аутентификации
125 | *1* - Basic
126 | *2* - ApiKey
127 | 2. *Index* - префикс названия индекса, конечное название будет определено в зависимости от значения параметра Separation.
128 | 3. *Separation* - метод разделения данных по индексам. Может принимать значения:
129 | *H* (Hour) - делить индексы по часам. Пример конечного названия индекса: index-name-el-2020010113
130 | *D* (Day) - делить индексы по дням. Пример конечного названия индекса: index-name-el-20200101
131 | *M* (Month) - делить индексы по месяцам. Пример конечного названия индекса: index-name-el-202001
132 | При указании любого другого (либо не указании вовсе) значения, разделения индекса не будет и конечное название индекса будет выглядеть так: index-name-el-all
133 | 4. *MaximumRetries* - количество попыток переподключения к очередному узлу 134 | 5. *MaxRetryTimeout* - таймаут попытки подключения 135 | 136 | Так-же при первом подключении к узлу приложение проверяет наличие шаблона индекса (Index template) с именем "oneslogs" и при отсутствии - создает. Если шаблон уже создан, то его перезапись происходить не будет, так как предполагается возможная ручная модификация первично созданного шаблона. 137 | 138 | **Пример файла кофигурации, содержащий секции для всех поддерживаемых СУБД:** 139 | ```json 140 | { 141 | "Logging": { 142 | "LogLevel": { 143 | "Default": "Debug", 144 | "Microsoft": "Warning", 145 | "Microsoft.Hosting.Lifetime": "Information" 146 | } 147 | }, 148 | "Manager": { 149 | "ClstFolders": [ 150 | { 151 | "Folder": "C:\\\\Program Files\\1cv8\\srvinfo\\reg_1541", 152 | "Templates": [ 153 | { 154 | "Mask": "upp_main", 155 | "Template": "[IBNAME]_el" 156 | } 157 | ] 158 | } 159 | ] 160 | }, 161 | "Exporter": { 162 | "StorageType": 2, 163 | "LogFolder": "C:\\Users\\akpaev.e.ENTERPRISE\\Desktop\\1Cv8Log", 164 | "Portion": 10000, 165 | "TimeZone": "Europe/Moscow", 166 | "WritingMaxDegreeOfParallelism": 1, 167 | "CollectedFactor": 2, 168 | "ReadingTimeout": 1, 169 | "LoadArchive": false 170 | }, 171 | "ClickHouse": { 172 | "ConnectionString": "Host=192.168.0.93;Port=8123;Database=upp_main_el;Username=default;password=;" 173 | }, 174 | "ElasticSearch": { 175 | "Nodes": [ 176 | { 177 | "Host": "http://192.168.0.95:9200", 178 | "AuthenticationType": "0" 179 | } 180 | ], 181 | "Index": "upp-main-el", 182 | "Separation": "M", 183 | "MaximumRetries": 2, 184 | "MaxRetryTimeout": 30 185 | } 186 | } 187 | ``` 188 | 189 | ### Использование: 190 | Все приложения могут быть запущены в 2 режимах: как обычное приложение, либо как служба Windows/Linux. Для теста в Вашей среде, достаточно просто выполнить конфигурацию приложения в файле *appsettings.json*, установить runtime .net 5 (при его отсутствии) и запустить exe/dll. Базы данных в СУБД вручную создавать не нужно, они будут созданы автоматически. 191 | 192 | Для запуска приложения как службы необходимо (название службы и путь к исполняемому файлу подставить свои):
193 | 194 | **Windows:**
195 | Поместить файлы приложения в каталог и выполнить в консоли команду: 196 | ``` 197 | sc create EventLogExporter binPath= "C:\elexporter\EventLogExporter.exe" 198 | ``` 199 | и запустить службу командой: 200 | ``` 201 | sc start EventLogExporter 202 | ``` 203 | **Linux: (на примере Ubuntu 20.04.1 LTS)**:
204 | *В этом примере файлы приложения были помещены в каталог /opt/EventLogExporter*
205 | В /etc/systemd/system создать файл eventlogexporter.service с содержимым: 206 | ``` 207 | [Service] 208 | Type=notify 209 | WorkingDirectory=/opt/EventLogExporter 210 | ExecStart=/usr/bin/dotnet /opt/EventLogExporter/EventLogExporter.dll 211 | 212 | [Install] 213 | WantedBy=multi-user.target 214 | ``` 215 | Применить изменения командой: 216 | ``` 217 | systemctl daemon-reload 218 | ``` 219 | и запустить службу: 220 | ``` 221 | systemctl start eventlogexporter.service 222 | ``` 223 | 224 | ### Результаты тестирования: 225 | Для теста был использован сервер с Intel Xeon E5-2643 3.40 GHz x2, 128 GB RAM и SAS дисками (Windows Server 2016). Экземпляр ElasticSearch установлен на хосте, экземпляр ClickHouse развернут на нем же в виртуальной машине (Hyper-V) с 4096 MiB RAM. Размер загружаемого журнала регистрации - 945 MiB.
226 | 227 | |СУБД |Порция|Время загрузки |Потребляемая память |Событий/сек |MiB/сек |Итоговый размер таблицы| 228 | |:-----------:|:----:|:--------------:|:-------------------:|:-----------:|:-------:|:---------------------:| 229 | |ClickHouse |10000 |1 мин. 41 сек. | ~ 60 MiB |71032 |9.13 |56.66 MiB | 230 | |ElasticSearch|5000 |2 мин. 35 сек. | ~ 100 MiB |45968 |6.09 |1106.7 MiB | 231 | 232 | ClickHouse использовался as is, но к колонкам (в зависимости от типа и состава данных) применены кодеки. Для шаблона индекса ElasticSearch были выставлены параметры number_of_shards = 6, number_of_replicas = 0, index.codec = best_compression и использовалось 4 потока записи. 233 | --------------------------------------------------------------------------------