├── .gitignore ├── OutboxScaling ├── .dockerignore ├── Messaging.Contracts │ ├── AssemblyReference.cs │ ├── Messaging.Contracts.csproj │ └── OrderCreatedIntegrationEvent.cs ├── OutboxProcessing │ ├── DatabaseInitializer.cs │ ├── Outbox │ │ ├── OutboxBackgroundService.cs │ │ ├── OutboxLoggers.cs │ │ ├── OutboxMessage.cs │ │ ├── OutboxProcessor.cs │ │ └── OutboxProcessor_Old.cs │ ├── OutboxProcessing.csproj │ ├── Program.cs │ ├── Properties │ │ └── launchSettings.json │ ├── appsettings.Development.json │ ├── appsettings.json │ └── create_database.sql └── OutboxScaling.sln └── README.md /.gitignore: -------------------------------------------------------------------------------- 1 | ## Ignore Visual Studio temporary files, build results, and 2 | ## files generated by popular Visual Studio add-ons. 3 | ## 4 | ## Get latest from https://github.com/github/gitignore/blob/main/VisualStudio.gitignore 5 | 6 | # User-specific files 7 | *.rsuser 8 | *.suo 9 | *.user 10 | *.userosscache 11 | *.sln.docstates 12 | 13 | # User-specific files (MonoDevelop/Xamarin Studio) 14 | *.userprefs 15 | 16 | # Mono auto generated files 17 | mono_crash.* 18 | 19 | # Build results 20 | [Dd]ebug/ 21 | [Dd]ebugPublic/ 22 | [Rr]elease/ 23 | [Rr]eleases/ 24 | x64/ 25 | x86/ 26 | [Ww][Ii][Nn]32/ 27 | [Aa][Rr][Mm]/ 28 | [Aa][Rr][Mm]64/ 29 | bld/ 30 | [Bb]in/ 31 | [Oo]bj/ 32 | [Ll]og/ 33 | [Ll]ogs/ 34 | 35 | # Visual Studio 2015/2017 cache/options directory 36 | .vs/ 37 | # Uncomment if you have tasks that create the project's static files in wwwroot 38 | #wwwroot/ 39 | 40 | # Visual Studio 2017 auto generated files 41 | Generated\ Files/ 42 | 43 | # MSTest test Results 44 | [Tt]est[Rr]esult*/ 45 | [Bb]uild[Ll]og.* 46 | 47 | # NUnit 48 | *.VisualState.xml 49 | TestResult.xml 50 | nunit-*.xml 51 | 52 | # Build Results of an ATL Project 53 | [Dd]ebugPS/ 54 | [Rr]eleasePS/ 55 | dlldata.c 56 | 57 | # Benchmark Results 58 | BenchmarkDotNet.Artifacts/ 59 | 60 | # .NET Core 61 | project.lock.json 62 | project.fragment.lock.json 63 | artifacts/ 64 | 65 | # ASP.NET Scaffolding 66 | ScaffoldingReadMe.txt 67 | 68 | # StyleCop 69 | StyleCopReport.xml 70 | 71 | # Files built by Visual Studio 72 | *_i.c 73 | *_p.c 74 | *_h.h 75 | *.ilk 76 | *.meta 77 | *.obj 78 | *.iobj 79 | *.pch 80 | *.pdb 81 | *.ipdb 82 | *.pgc 83 | *.pgd 84 | *.rsp 85 | *.sbr 86 | *.tlb 87 | *.tli 88 | *.tlh 89 | *.tmp 90 | *.tmp_proj 91 | *_wpftmp.csproj 92 | *.log 93 | *.tlog 94 | *.vspscc 95 | *.vssscc 96 | .builds 97 | *.pidb 98 | *.svclog 99 | *.scc 100 | 101 | # Chutzpah Test files 102 | _Chutzpah* 103 | 104 | # Visual C++ cache files 105 | ipch/ 106 | *.aps 107 | *.ncb 108 | *.opendb 109 | *.opensdf 110 | *.sdf 111 | *.cachefile 112 | *.VC.db 113 | *.VC.VC.opendb 114 | 115 | # Visual Studio profiler 116 | *.psess 117 | *.vsp 118 | *.vspx 119 | *.sap 120 | 121 | # Visual Studio Trace Files 122 | *.e2e 123 | 124 | # TFS 2012 Local Workspace 125 | $tf/ 126 | 127 | # Guidance Automation Toolkit 128 | *.gpState 129 | 130 | # ReSharper is a .NET coding add-in 131 | _ReSharper*/ 132 | *.[Rr]e[Ss]harper 133 | *.DotSettings.user 134 | 135 | # TeamCity is a build add-in 136 | _TeamCity* 137 | 138 | # DotCover is a Code Coverage Tool 139 | *.dotCover 140 | 141 | # AxoCover is a Code Coverage Tool 142 | .axoCover/* 143 | !.axoCover/settings.json 144 | 145 | # Coverlet is a free, cross platform Code Coverage Tool 146 | coverage*.json 147 | coverage*.xml 148 | coverage*.info 149 | 150 | # Visual Studio code coverage results 151 | *.coverage 152 | *.coveragexml 153 | 154 | # NCrunch 155 | _NCrunch_* 156 | .*crunch*.local.xml 157 | nCrunchTemp_* 158 | 159 | # MightyMoose 160 | *.mm.* 161 | AutoTest.Net/ 162 | 163 | # Web workbench (sass) 164 | .sass-cache/ 165 | 166 | # Installshield output folder 167 | [Ee]xpress/ 168 | 169 | # DocProject is a documentation generator add-in 170 | DocProject/buildhelp/ 171 | DocProject/Help/*.HxT 172 | DocProject/Help/*.HxC 173 | DocProject/Help/*.hhc 174 | DocProject/Help/*.hhk 175 | DocProject/Help/*.hhp 176 | DocProject/Help/Html2 177 | DocProject/Help/html 178 | 179 | # Click-Once directory 180 | publish/ 181 | 182 | # Publish Web Output 183 | *.[Pp]ublish.xml 184 | *.azurePubxml 185 | # Note: Comment the next line if you want to checkin your web deploy settings, 186 | # but database connection strings (with potential passwords) will be unencrypted 187 | *.pubxml 188 | *.publishproj 189 | 190 | # Microsoft Azure Web App publish settings. Comment the next line if you want to 191 | # checkin your Azure Web App publish settings, but sensitive information contained 192 | # in these scripts will be unencrypted 193 | PublishScripts/ 194 | 195 | # NuGet Packages 196 | *.nupkg 197 | # NuGet Symbol Packages 198 | *.snupkg 199 | # The packages folder can be ignored because of Package Restore 200 | **/[Pp]ackages/* 201 | # except build/, which is used as an MSBuild target. 202 | !**/[Pp]ackages/build/ 203 | # Uncomment if necessary however generally it will be regenerated when needed 204 | #!**/[Pp]ackages/repositories.config 205 | # NuGet v3's project.json files produces more ignorable files 206 | *.nuget.props 207 | *.nuget.targets 208 | 209 | # Microsoft Azure Build Output 210 | csx/ 211 | *.build.csdef 212 | 213 | # Microsoft Azure Emulator 214 | ecf/ 215 | rcf/ 216 | 217 | # Windows Store app package directories and files 218 | AppPackages/ 219 | BundleArtifacts/ 220 | Package.StoreAssociation.xml 221 | _pkginfo.txt 222 | *.appx 223 | *.appxbundle 224 | *.appxupload 225 | 226 | # Visual Studio cache files 227 | # files ending in .cache can be ignored 228 | *.[Cc]ache 229 | # but keep track of directories ending in .cache 230 | !?*.[Cc]ache/ 231 | 232 | # Others 233 | ClientBin/ 234 | ~$* 235 | *~ 236 | *.dbmdl 237 | *.dbproj.schemaview 238 | *.jfm 239 | *.pfx 240 | *.publishsettings 241 | orleans.codegen.cs 242 | 243 | # Including strong name files can present a security risk 244 | # (https://github.com/github/gitignore/pull/2483#issue-259490424) 245 | #*.snk 246 | 247 | # Since there are multiple workflows, uncomment next line to ignore bower_components 248 | # (https://github.com/github/gitignore/pull/1529#issuecomment-104372622) 249 | #bower_components/ 250 | 251 | # RIA/Silverlight projects 252 | Generated_Code/ 253 | 254 | # Backup & report files from converting an old project file 255 | # to a newer Visual Studio version. Backup files are not needed, 256 | # because we have git ;-) 257 | _UpgradeReport_Files/ 258 | Backup*/ 259 | UpgradeLog*.XML 260 | UpgradeLog*.htm 261 | ServiceFabricBackup/ 262 | *.rptproj.bak 263 | 264 | # SQL Server files 265 | *.mdf 266 | *.ldf 267 | *.ndf 268 | 269 | # Business Intelligence projects 270 | *.rdl.data 271 | *.bim.layout 272 | *.bim_*.settings 273 | *.rptproj.rsuser 274 | *- [Bb]ackup.rdl 275 | *- [Bb]ackup ([0-9]).rdl 276 | *- [Bb]ackup ([0-9][0-9]).rdl 277 | 278 | # Microsoft Fakes 279 | FakesAssemblies/ 280 | 281 | # GhostDoc plugin setting file 282 | *.GhostDoc.xml 283 | 284 | # Node.js Tools for Visual Studio 285 | .ntvs_analysis.dat 286 | node_modules/ 287 | 288 | # Visual Studio 6 build log 289 | *.plg 290 | 291 | # Visual Studio 6 workspace options file 292 | *.opt 293 | 294 | # Visual Studio 6 auto-generated workspace file (contains which files were open etc.) 295 | *.vbw 296 | 297 | # Visual Studio 6 auto-generated project file (contains which files were open etc.) 298 | *.vbp 299 | 300 | # Visual Studio 6 workspace and project file (working project files containing files to include in project) 301 | *.dsw 302 | *.dsp 303 | 304 | # Visual Studio 6 technical files 305 | *.ncb 306 | *.aps 307 | 308 | # Visual Studio LightSwitch build output 309 | **/*.HTMLClient/GeneratedArtifacts 310 | **/*.DesktopClient/GeneratedArtifacts 311 | **/*.DesktopClient/ModelManifest.xml 312 | **/*.Server/GeneratedArtifacts 313 | **/*.Server/ModelManifest.xml 314 | _Pvt_Extensions 315 | 316 | # Paket dependency manager 317 | .paket/paket.exe 318 | paket-files/ 319 | 320 | # FAKE - F# Make 321 | .fake/ 322 | 323 | # CodeRush personal settings 324 | .cr/personal 325 | 326 | # Python Tools for Visual Studio (PTVS) 327 | __pycache__/ 328 | *.pyc 329 | 330 | # Cake - Uncomment if you are using it 331 | # tools/** 332 | # !tools/packages.config 333 | 334 | # Tabs Studio 335 | *.tss 336 | 337 | # Telerik's JustMock configuration file 338 | *.jmconfig 339 | 340 | # BizTalk build output 341 | *.btp.cs 342 | *.btm.cs 343 | *.odx.cs 344 | *.xsd.cs 345 | 346 | # OpenCover UI analysis results 347 | OpenCover/ 348 | 349 | # Azure Stream Analytics local run output 350 | ASALocalRun/ 351 | 352 | # MSBuild Binary and Structured Log 353 | *.binlog 354 | 355 | # NVidia Nsight GPU debugger configuration file 356 | *.nvuser 357 | 358 | # MFractors (Xamarin productivity tool) working folder 359 | .mfractor/ 360 | 361 | # Local History for Visual Studio 362 | .localhistory/ 363 | 364 | # Visual Studio History (VSHistory) files 365 | .vshistory/ 366 | 367 | # BeatPulse healthcheck temp database 368 | healthchecksdb 369 | 370 | # Backup folder for Package Reference Convert tool in Visual Studio 2017 371 | MigrationBackup/ 372 | 373 | # Ionide (cross platform F# VS Code tools) working folder 374 | .ionide/ 375 | 376 | # Fody - auto-generated XML schema 377 | FodyWeavers.xsd 378 | 379 | # VS Code files for those working on multiple tools 380 | .vscode/* 381 | !.vscode/settings.json 382 | !.vscode/tasks.json 383 | !.vscode/launch.json 384 | !.vscode/extensions.json 385 | *.code-workspace 386 | 387 | # Local History for Visual Studio Code 388 | .history/ 389 | 390 | # Windows Installer files from build outputs 391 | *.cab 392 | *.msi 393 | *.msix 394 | *.msm 395 | *.msp 396 | 397 | # JetBrains Rider 398 | *.sln.iml 399 | -------------------------------------------------------------------------------- /OutboxScaling/.dockerignore: -------------------------------------------------------------------------------- 1 | **/.classpath 2 | **/.dockerignore 3 | **/.env 4 | **/.git 5 | **/.gitignore 6 | **/.project 7 | **/.settings 8 | **/.toolstarget 9 | **/.vs 10 | **/.vscode 11 | **/*.*proj.user 12 | **/*.dbmdl 13 | **/*.jfm 14 | **/azds.yaml 15 | **/bin 16 | **/charts 17 | **/docker-compose* 18 | **/Dockerfile* 19 | **/node_modules 20 | **/npm-debug.log 21 | **/obj 22 | **/secrets.dev.yaml 23 | **/values.dev.yaml 24 | LICENSE 25 | README.md 26 | !**/.gitignore 27 | !.git/HEAD 28 | !.git/config 29 | !.git/packed-refs 30 | !.git/refs/heads/** -------------------------------------------------------------------------------- /OutboxScaling/Messaging.Contracts/AssemblyReference.cs: -------------------------------------------------------------------------------- 1 | using System.Reflection; 2 | 3 | namespace Messaging.Contracts; 4 | 5 | public static class AssemblyReference 6 | { 7 | public static readonly Assembly Assembly = typeof(AssemblyReference).Assembly; 8 | } 9 | -------------------------------------------------------------------------------- /OutboxScaling/Messaging.Contracts/Messaging.Contracts.csproj: -------------------------------------------------------------------------------- 1 |  2 | 3 | 4 | net8.0 5 | enable 6 | enable 7 | 8 | 9 | 10 | -------------------------------------------------------------------------------- /OutboxScaling/Messaging.Contracts/OrderCreatedIntegrationEvent.cs: -------------------------------------------------------------------------------- 1 | namespace Messaging.Contracts; 2 | 3 | public sealed record OrderCreatedIntegrationEvent(Guid OrderId); 4 | -------------------------------------------------------------------------------- /OutboxScaling/OutboxProcessing/DatabaseInitializer.cs: -------------------------------------------------------------------------------- 1 | using System.Text.Json; 2 | using Dapper; 3 | using Messaging.Contracts; 4 | using Npgsql; 5 | using NpgsqlTypes; 6 | 7 | namespace OutboxProcessing; 8 | 9 | internal sealed class DatabaseInitializer( 10 | NpgsqlDataSource dataSource, 11 | IConfiguration configuration, 12 | ILogger logger) 13 | { 14 | public async Task Execute(CancellationToken stoppingToken = default) 15 | { 16 | try 17 | { 18 | logger.LogInformation("Starting database initialization."); 19 | 20 | await EnsureDatabaseExists(); 21 | await InitializeDatabase(); 22 | await SeedInitialData(); 23 | 24 | logger.LogInformation("Database initialization completed successfully."); 25 | } 26 | catch (Exception ex) 27 | { 28 | logger.LogError(ex, "An error occurred while initializing the database."); 29 | } 30 | } 31 | 32 | private async Task EnsureDatabaseExists() 33 | { 34 | string connectionString = configuration.GetConnectionString("Database")!; 35 | var builder = new NpgsqlConnectionStringBuilder(connectionString); 36 | string? databaseName = builder.Database; 37 | builder.Database = "postgres"; // Connect to the default 'postgres' database 38 | 39 | using var connection = new NpgsqlConnection(builder.ToString()); 40 | await connection.OpenAsync(); 41 | 42 | bool databaseExists = await connection.ExecuteScalarAsync( 43 | "SELECT EXISTS(SELECT 1 FROM pg_database WHERE datname = @databaseName)", 44 | new { databaseName }); 45 | 46 | if (!databaseExists) 47 | { 48 | logger.LogInformation("Creating database {DatabaseName}", databaseName); 49 | await connection.ExecuteAsync($"CREATE DATABASE {databaseName}"); 50 | } 51 | } 52 | 53 | private async Task InitializeDatabase() 54 | { 55 | const string sql = 56 | """ 57 | -- Create outbox_messages table if it doesn't exist 58 | CREATE TABLE IF NOT EXISTS public.outbox_messages ( 59 | id UUID PRIMARY KEY, 60 | type VARCHAR(255) NOT NULL, 61 | content JSONB NOT NULL, 62 | occurred_on_utc TIMESTAMP WITH TIME ZONE NOT NULL, 63 | processed_on_utc TIMESTAMP WITH TIME ZONE NULL, 64 | error TEXT NULL 65 | ); 66 | 67 | -- Create a filtered index on unprocessed messages, including all necessary columns 68 | CREATE INDEX IF NOT EXISTS idx_outbox_messages_unprocessed 69 | ON public.outbox_messages (occurred_on_utc, processed_on_utc) 70 | INCLUDE (id, type, content) 71 | WHERE processed_on_utc IS NULL; 72 | """; 73 | using NpgsqlConnection connection = await dataSource.OpenConnectionAsync(); 74 | await connection.ExecuteAsync(sql); 75 | } 76 | 77 | private async Task SeedInitialData() 78 | { 79 | using NpgsqlConnection connection = await dataSource.OpenConnectionAsync(); 80 | 81 | logger.LogInformation("Deleting existing records from outbox_messages table."); 82 | await connection.ExecuteAsync("TRUNCATE TABLE public.outbox_messages"); 83 | 84 | logger.LogInformation("Seeding 2 million records to outbox_messages table."); 85 | 86 | const int batchSize = 500_000; 87 | const int totalRecords = 2_000_000; 88 | 89 | await using var writer = await connection.BeginBinaryImportAsync( 90 | "COPY public.outbox_messages (id, type, content, occurred_on_utc) FROM STDIN (FORMAT BINARY)"); 91 | 92 | for (int i = 0; i < totalRecords; i++) 93 | { 94 | await writer.StartRowAsync(); 95 | await writer.WriteAsync(Guid.NewGuid(), NpgsqlDbType.Uuid); 96 | await writer.WriteAsync(typeof(OrderCreatedIntegrationEvent).FullName, NpgsqlDbType.Varchar); 97 | await writer.WriteAsync( 98 | JsonSerializer.Serialize(new OrderCreatedIntegrationEvent(Guid.NewGuid())), NpgsqlDbType.Jsonb); 99 | await writer.WriteAsync(DateTime.UtcNow, NpgsqlDbType.TimestampTz); 100 | 101 | if ((i + 1) % batchSize == 0) 102 | { 103 | logger.LogInformation("Inserted {Count} records", i + 1); 104 | } 105 | } 106 | 107 | await writer.CompleteAsync(); 108 | 109 | logger.LogInformation("Finished seeding 2 million records to outbox_messages table."); 110 | } 111 | } 112 | -------------------------------------------------------------------------------- /OutboxScaling/OutboxProcessing/Outbox/OutboxBackgroundService.cs: -------------------------------------------------------------------------------- 1 | namespace OutboxProcessing.Outbox; 2 | 3 | internal class OutboxBackgroundService( 4 | IServiceScopeFactory serviceScopeFactory, 5 | ILogger logger) : BackgroundService 6 | { 7 | private const int OutboxProcessorFrequency = 5; 8 | private readonly int _maxParallelism = 5; 9 | private int _totalIterations = 0; 10 | private int _totalProcessedMessage = 0; 11 | 12 | protected override async Task ExecuteAsync(CancellationToken stoppingToken) 13 | { 14 | OutboxLoggers.LogStarting(logger); 15 | 16 | using var cts = new CancellationTokenSource(TimeSpan.FromMinutes(1)); 17 | using var linkedCts = CancellationTokenSource.CreateLinkedTokenSource(cts.Token, stoppingToken); 18 | 19 | var parallelOptions = new ParallelOptions 20 | { 21 | MaxDegreeOfParallelism = _maxParallelism, 22 | CancellationToken = linkedCts.Token 23 | }; 24 | 25 | try 26 | { 27 | await Parallel.ForEachAsync( 28 | Enumerable.Range(0, _maxParallelism), 29 | parallelOptions, 30 | async (_, token) => 31 | { 32 | await ProcessOutboxMessages(token); 33 | }); 34 | } 35 | catch (OperationCanceledException) 36 | { 37 | OutboxLoggers.LogOperationCancelled(logger); 38 | } 39 | catch (Exception ex) 40 | { 41 | OutboxLoggers.LogError(logger, ex); 42 | } 43 | finally 44 | { 45 | OutboxLoggers.LogFinished(logger, _totalIterations, _totalProcessedMessage); 46 | } 47 | } 48 | 49 | private async Task ProcessOutboxMessages(CancellationToken cancellationToken) 50 | { 51 | using var scope = serviceScopeFactory.CreateScope(); 52 | var outboxProcessor = scope.ServiceProvider.GetRequiredService(); 53 | 54 | while (!cancellationToken.IsCancellationRequested) 55 | { 56 | var iterationCount = Interlocked.Increment(ref _totalIterations); 57 | OutboxLoggers.LogStartingIteration(logger, iterationCount); 58 | 59 | int processedMessages = await outboxProcessor.Execute(cancellationToken); 60 | var totalProcessedMessages = Interlocked.Add(ref _totalProcessedMessage, processedMessages); 61 | 62 | OutboxLoggers.LogIterationCompleted(logger, iterationCount, processedMessages, totalProcessedMessages); 63 | 64 | // Simulate running Outbox processing every N seconds 65 | //await Task.Delay(TimeSpan.FromSeconds(OutboxProcessorFrequency), cancellationToken); 66 | } 67 | } 68 | } -------------------------------------------------------------------------------- /OutboxScaling/OutboxProcessing/Outbox/OutboxLoggers.cs: -------------------------------------------------------------------------------- 1 | namespace OutboxProcessing.Outbox; 2 | 3 | internal static partial class OutboxLoggers 4 | { 5 | [LoggerMessage(Level = LogLevel.Information, Message = "OutboxBackgroundService starting...")] 6 | internal static partial void LogStarting(ILogger logger); 7 | 8 | [LoggerMessage(Level = LogLevel.Information, Message = "Starting iteration {IterationCount}")] 9 | internal static partial void LogStartingIteration(ILogger logger, int iterationCount); 10 | 11 | [LoggerMessage(Level = LogLevel.Information, Message = "Iteration {IterationCount} completed. Processed {ProcessedMessages} messages. Total processed: {TotalProcessedMessages}")] 12 | internal static partial void LogIterationCompleted(ILogger logger, int iterationCount, int processedMessages, int totalProcessedMessages); 13 | 14 | [LoggerMessage(Level = LogLevel.Information, Message = "OutboxBackgroundService operation cancelled.")] 15 | internal static partial void LogOperationCancelled(ILogger logger); 16 | 17 | [LoggerMessage(Level = LogLevel.Error, Message = "An error occurred in OutboxBackgroundService")] 18 | internal static partial void LogError(ILogger logger, Exception exception); 19 | 20 | [LoggerMessage(Level = LogLevel.Information, Message = "OutboxBackgroundService finished. Total iterations: {IterationCount}, Total processed messages: {TotalProcessedMessages}")] 21 | internal static partial void LogFinished(ILogger logger, int iterationCount, int totalProcessedMessages); 22 | 23 | [LoggerMessage(Level = LogLevel.Information, Message = "Outbox processing completed. Total time: {TotalTime}ms, Query time: {QueryTime}ms, Publish time: {PublishTime}ms, Update time: {UpdateTime}ms, Messages processed: {MessageCount}")] 24 | internal static partial void LogProcessingPerformance(ILogger logger, long totalTime, long queryTime, long publishTime, long updateTime, int messageCount); 25 | } -------------------------------------------------------------------------------- /OutboxScaling/OutboxProcessing/Outbox/OutboxMessage.cs: -------------------------------------------------------------------------------- 1 | namespace OutboxProcessing.Outbox; 2 | 3 | public sealed class OutboxMessage 4 | { 5 | public Guid Id { get; init; } 6 | public required string Type { get; init; } 7 | public required string Content { get; init; } 8 | public DateTime OccurredOnUtc { get; init; } 9 | public DateTime? ProcessedOnUtc { get; init; } 10 | public string? Error { get; init; } 11 | } 12 | -------------------------------------------------------------------------------- /OutboxScaling/OutboxProcessing/Outbox/OutboxProcessor.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Concurrent; 2 | using System.Diagnostics; 3 | using Dapper; 4 | using MassTransit; 5 | using Npgsql; 6 | using System.Text.Json; 7 | 8 | namespace OutboxProcessing.Outbox; 9 | 10 | internal sealed class OutboxProcessor( 11 | NpgsqlDataSource dataSource, 12 | IPublishEndpoint publishEndpoint, 13 | ILogger logger) 14 | { 15 | private const int BatchSize = 1000; 16 | private static readonly ConcurrentDictionary TypeCache = new(); 17 | 18 | public async Task Execute(CancellationToken cancellationToken = default) 19 | { 20 | var totalStopwatch = Stopwatch.StartNew(); 21 | var stepStopwatch = new Stopwatch(); 22 | 23 | await using var connection = await dataSource.OpenConnectionAsync(cancellationToken); 24 | await using var transaction = await connection.BeginTransactionAsync(cancellationToken); 25 | 26 | stepStopwatch.Restart(); 27 | var messages = (await connection.QueryAsync( 28 | """ 29 | SELECT id AS Id, type AS Type, content AS Content 30 | FROM outbox_messages 31 | WHERE processed_on_utc IS NULL 32 | ORDER BY occurred_on_utc LIMIT @BatchSize 33 | FOR UPDATE SKIP LOCKED 34 | """, 35 | new { BatchSize }, 36 | transaction: transaction)).AsList(); 37 | var queryTime = stepStopwatch.ElapsedMilliseconds; 38 | 39 | var updateQueue = new ConcurrentQueue(); 40 | 41 | stepStopwatch.Restart(); 42 | var publishTasks = messages 43 | .Select(message => PublishMessage(message, updateQueue, publishEndpoint, cancellationToken)) 44 | .ToList(); 45 | 46 | await Task.WhenAll(publishTasks); 47 | var publishTime = stepStopwatch.ElapsedMilliseconds; 48 | 49 | stepStopwatch.Restart(); 50 | if (!updateQueue.IsEmpty) 51 | { 52 | var updateSql = 53 | """ 54 | UPDATE outbox_messages 55 | SET processed_on_utc = v.processed_on_utc, 56 | error = v.error 57 | FROM (VALUES 58 | {0} 59 | ) AS v(id, processed_on_utc, error) 60 | WHERE outbox_messages.id = v.id::uuid 61 | """; 62 | 63 | var updates = updateQueue.ToList(); 64 | var valuesList = string.Join(",", 65 | updateQueue.Select((_, i) => $"(@Id{i}, @ProcessedOn{i}, @Error{i})")); 66 | 67 | var parameters = new DynamicParameters(); 68 | 69 | for (int i = 0; i < updateQueue.Count; i++) 70 | { 71 | parameters.Add($"Id{i}", updates[i].Id.ToString()); 72 | parameters.Add($"ProcessedOn{i}", updates[i].ProcessedOnUtc); 73 | parameters.Add($"Error{i}", updates[i].Error); 74 | } 75 | 76 | var formattedSql = string.Format(updateSql, valuesList); 77 | 78 | await connection.ExecuteAsync(formattedSql, parameters, transaction: transaction); 79 | } 80 | var updateTime = stepStopwatch.ElapsedMilliseconds; 81 | 82 | await transaction.CommitAsync(cancellationToken); 83 | 84 | totalStopwatch.Stop(); 85 | var totalTime = totalStopwatch.ElapsedMilliseconds; 86 | 87 | OutboxLoggers.LogProcessingPerformance(logger, totalTime, queryTime, publishTime, updateTime, messages.Count); 88 | 89 | return messages.Count; 90 | } 91 | 92 | private static async Task PublishMessage( 93 | OutboxMessage message, 94 | ConcurrentQueue updateQueue, 95 | IPublishEndpoint publishEndpoint, 96 | CancellationToken cancellationToken) 97 | { 98 | try 99 | { 100 | var messageType = GetOrAddMessageType(message.Type); 101 | var deserializedMessage = JsonSerializer.Deserialize(message.Content, messageType)!; 102 | 103 | await publishEndpoint.Publish(deserializedMessage, cancellationToken); 104 | 105 | updateQueue.Enqueue(new OutboxUpdate { Id = message.Id, ProcessedOnUtc = DateTime.UtcNow }); 106 | } 107 | catch (Exception ex) 108 | { 109 | updateQueue.Enqueue( 110 | new OutboxUpdate { Id = message.Id, ProcessedOnUtc = DateTime.UtcNow, Error = ex.ToString() }); 111 | } 112 | } 113 | 114 | private static Type GetOrAddMessageType(string typename) 115 | { 116 | return TypeCache.GetOrAdd(typename, name => Messaging.Contracts.AssemblyReference.Assembly.GetType(name)!); 117 | } 118 | 119 | private struct OutboxUpdate 120 | { 121 | public Guid Id { get; init; } 122 | public DateTime ProcessedOnUtc { get; init; } 123 | public string? Error { get; init; } 124 | } 125 | } 126 | -------------------------------------------------------------------------------- /OutboxScaling/OutboxProcessing/Outbox/OutboxProcessor_Old.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Concurrent; 2 | using System.Diagnostics; 3 | using Dapper; 4 | using MassTransit; 5 | using Npgsql; 6 | using System.Text.Json; 7 | 8 | namespace OutboxProcessing.Outbox; 9 | 10 | internal sealed class OutboxProcessor_Old( 11 | NpgsqlDataSource dataSource, 12 | IPublishEndpoint publishEndpoint, 13 | ILogger logger) 14 | { 15 | private const int BatchSize = 1000; 16 | private static readonly ConcurrentDictionary TypeCache = new(); 17 | 18 | public async Task Execute(CancellationToken cancellationToken = default) 19 | { 20 | var totalStopwatch = Stopwatch.StartNew(); 21 | var stepStopwatch = new Stopwatch(); 22 | 23 | await using var connection = await dataSource.OpenConnectionAsync(cancellationToken); 24 | await using var transaction = await connection.BeginTransactionAsync(cancellationToken); 25 | 26 | stepStopwatch.Restart(); 27 | var messages = (await connection.QueryAsync( 28 | """ 29 | SELECT * 30 | FROM outbox_messages 31 | WHERE processed_on_utc IS NULL 32 | ORDER BY occurred_on_utc LIMIT @BatchSize 33 | """, 34 | new { BatchSize }, 35 | transaction: transaction)).AsList(); 36 | var queryTime = stepStopwatch.ElapsedMilliseconds; 37 | 38 | var updateQueue = new ConcurrentQueue(); 39 | 40 | stepStopwatch.Restart(); 41 | foreach (var message in messages) 42 | { 43 | try 44 | { 45 | var messageType = Messaging.Contracts.AssemblyReference.Assembly.GetType(message.Type); 46 | var deserializedMessage = JsonSerializer.Deserialize(message.Content, messageType); 47 | 48 | await publishEndpoint.Publish(deserializedMessage, messageType, cancellationToken); 49 | 50 | updateQueue.Enqueue(new OutboxUpdate 51 | { 52 | Id = message.Id, 53 | ProcessedOnUtc = DateTime.UtcNow 54 | }); 55 | } 56 | catch (Exception ex) 57 | { 58 | updateQueue.Enqueue(new OutboxUpdate 59 | { 60 | Id = message.Id, 61 | ProcessedOnUtc = DateTime.UtcNow, 62 | Error = ex.ToString() 63 | }); 64 | } 65 | } 66 | var publishTime = stepStopwatch.ElapsedMilliseconds; 67 | 68 | stepStopwatch.Restart(); 69 | foreach (var outboxUpdate in updateQueue) 70 | { 71 | await connection.ExecuteAsync( 72 | """ 73 | UPDATE outbox_messages 74 | SET processed_on_utc = @ProcessedOnUtc, error = @Error 75 | WHERE id = @Id 76 | """, 77 | outboxUpdate, 78 | transaction: transaction); 79 | } 80 | var updateTime = stepStopwatch.ElapsedMilliseconds; 81 | 82 | await transaction.CommitAsync(cancellationToken); 83 | 84 | totalStopwatch.Stop(); 85 | var totalTime = totalStopwatch.ElapsedMilliseconds; 86 | 87 | OutboxLoggers.LogProcessingPerformance(logger, totalTime, queryTime, publishTime, updateTime, messages.Count); 88 | 89 | return messages.Count; 90 | } 91 | 92 | private struct OutboxUpdate 93 | { 94 | public Guid Id { get; init; } 95 | public DateTime ProcessedOnUtc { get; init; } 96 | public string? Error { get; init; } 97 | } 98 | } 99 | -------------------------------------------------------------------------------- /OutboxScaling/OutboxProcessing/OutboxProcessing.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | net8.0 5 | enable 6 | enable 7 | dotnet-OutboxProcessor-0791200e-7d0a-4a0b-97fd-01f00aabd870 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | -------------------------------------------------------------------------------- /OutboxScaling/OutboxProcessing/Program.cs: -------------------------------------------------------------------------------- 1 | using MassTransit; 2 | using Npgsql; 3 | using OutboxProcessing; 4 | using OutboxProcessing.Outbox; 5 | 6 | var builder = Host.CreateApplicationBuilder(args); 7 | 8 | builder.Services.AddScoped(); 9 | builder.Services.AddSingleton(); 10 | 11 | builder.Services.AddSingleton(_ => 12 | { 13 | var connectionString = builder.Configuration.GetConnectionString("Database"); 14 | return new NpgsqlDataSourceBuilder(connectionString).Build(); 15 | }); 16 | 17 | builder.Services.AddMassTransit(x => 18 | { 19 | x.UsingRabbitMq((context, cfg) => 20 | { 21 | cfg.Host(builder.Configuration.GetConnectionString("Queue"), hostCfg => 22 | { 23 | hostCfg.ConfigureBatchPublish(batch => 24 | { 25 | batch.Enabled = true; 26 | }); 27 | }); 28 | 29 | cfg.ConfigureEndpoints(context); 30 | }); 31 | }); 32 | 33 | builder.Services.AddHostedService(); 34 | 35 | var host = builder.Build(); 36 | 37 | await host.Services.GetRequiredService().Execute(); 38 | 39 | host.Run(); 40 | -------------------------------------------------------------------------------- /OutboxScaling/OutboxProcessing/Properties/launchSettings.json: -------------------------------------------------------------------------------- 1 | { 2 | "$schema": "http://json.schemastore.org/launchsettings.json", 3 | "profiles": { 4 | "OutboxProcessor": { 5 | "commandName": "Project", 6 | "dotnetRunMessages": true, 7 | "environmentVariables": { 8 | "DOTNET_ENVIRONMENT": "Development" 9 | } 10 | } 11 | } 12 | } 13 | -------------------------------------------------------------------------------- /OutboxScaling/OutboxProcessing/appsettings.Development.json: -------------------------------------------------------------------------------- 1 | { 2 | "Logging": { 3 | "LogLevel": { 4 | "Default": "Information", 5 | "Microsoft.Hosting.Lifetime": "Information" 6 | } 7 | } 8 | } 9 | -------------------------------------------------------------------------------- /OutboxScaling/OutboxProcessing/appsettings.json: -------------------------------------------------------------------------------- 1 | { 2 | "ConnectionStrings": { 3 | "Database": "Host=localhost;Port=5432;Database=outbox_scaling;Username=postgres;Password=postgres", 4 | "Queue": "amqp://guest:guest@localhost:5672" 5 | }, 6 | "Logging": { 7 | "LogLevel": { 8 | "Default": "Information", 9 | "Microsoft.Hosting.Lifetime": "Information" 10 | } 11 | } 12 | } 13 | -------------------------------------------------------------------------------- /OutboxScaling/OutboxProcessing/create_database.sql: -------------------------------------------------------------------------------- 1 | CREATE TABLE IF NOT EXISTS outbox_messages ( 2 | id UUID PRIMARY KEY, 3 | type VARCHAR(255) NOT NULL, 4 | content JSONB NOT NULL, 5 | occurred_on_utc TIMESTAMP WITH TIME ZONE NOT NULL, 6 | processed_on_utc TIMESTAMP WITH TIME ZONE NULL, 7 | error TEXT NULL 8 | ); 9 | 10 | -- Create a filtered index on unprocessed messages, including all necessary columns 11 | CREATE INDEX IF NOT EXISTS idx_outbox_messages_unprocessed 12 | ON public.outbox_messages (occurred_on_utc, processed_on_utc) 13 | INCLUDE (id, type, content) 14 | WHERE processed_on_utc IS NULL; -------------------------------------------------------------------------------- /OutboxScaling/OutboxScaling.sln: -------------------------------------------------------------------------------- 1 |  2 | Microsoft Visual Studio Solution File, Format Version 12.00 3 | # Visual Studio Version 17 4 | VisualStudioVersion = 17.11.35312.102 5 | MinimumVisualStudioVersion = 10.0.40219.1 6 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "OutboxProcessing", "OutboxProcessing\OutboxProcessing.csproj", "{0D96F937-0CD2-4612-A6D4-AC561340D2E0}" 7 | EndProject 8 | Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Messaging.Contracts", "Messaging.Contracts\Messaging.Contracts.csproj", "{A8AF6974-CE05-4452-9EC8-311193358AC8}" 9 | EndProject 10 | Global 11 | GlobalSection(SolutionConfigurationPlatforms) = preSolution 12 | Debug|Any CPU = Debug|Any CPU 13 | Release|Any CPU = Release|Any CPU 14 | EndGlobalSection 15 | GlobalSection(ProjectConfigurationPlatforms) = postSolution 16 | {0D96F937-0CD2-4612-A6D4-AC561340D2E0}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 17 | {0D96F937-0CD2-4612-A6D4-AC561340D2E0}.Debug|Any CPU.Build.0 = Debug|Any CPU 18 | {0D96F937-0CD2-4612-A6D4-AC561340D2E0}.Release|Any CPU.ActiveCfg = Release|Any CPU 19 | {0D96F937-0CD2-4612-A6D4-AC561340D2E0}.Release|Any CPU.Build.0 = Release|Any CPU 20 | {A8AF6974-CE05-4452-9EC8-311193358AC8}.Debug|Any CPU.ActiveCfg = Debug|Any CPU 21 | {A8AF6974-CE05-4452-9EC8-311193358AC8}.Debug|Any CPU.Build.0 = Debug|Any CPU 22 | {A8AF6974-CE05-4452-9EC8-311193358AC8}.Release|Any CPU.ActiveCfg = Release|Any CPU 23 | {A8AF6974-CE05-4452-9EC8-311193358AC8}.Release|Any CPU.Build.0 = Release|Any CPU 24 | EndGlobalSection 25 | GlobalSection(SolutionProperties) = preSolution 26 | HideSolutionNode = FALSE 27 | EndGlobalSection 28 | GlobalSection(ExtensibilityGlobals) = postSolution 29 | SolutionGuid = {D4B2EAEE-0A68-41D9-B512-2B0B714CD109} 30 | EndGlobalSection 31 | EndGlobal 32 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Scaling the Outbox Pattern 2 | 3 | Demonstrating how to scale the Outbox Pattern to 2B+ messages per day (~30,000 messages per second). 4 | 5 | Here's a detailed breakdown of the performance optimizations: [Scaling the Outbox Pattern](https://www.milanjovanovic.tech/blog/scaling-the-outbox-pattern) 6 | 7 | P.S. If you want to level up you software architecture skills [go here](https://www.milanjovanovic.tech/courses). --------------------------------------------------------------------------------