├── charts └── replicator │ ├── .gitignore │ ├── Chart.yaml │ ├── templates │ ├── NOTES.txt │ ├── configmap.yaml │ ├── pvc.yaml │ ├── configmap-transform.yaml │ ├── configmap-partitioner.yaml │ ├── podmonitor.yaml │ ├── service.yaml │ ├── statefulset.yaml │ └── _helpers.tpl │ └── values.yaml ├── src ├── replicator │ ├── ClientApp │ │ ├── .browserslistrc │ │ ├── .env │ │ ├── .env.development │ │ ├── public │ │ │ └── favicon.ico │ │ ├── src │ │ │ ├── assets │ │ │ │ └── logo.png │ │ │ ├── views │ │ │ │ ├── About.vue │ │ │ │ └── Home.vue │ │ │ ├── plugins │ │ │ │ └── element.js │ │ │ ├── shims-vue.d.ts │ │ │ ├── main.ts │ │ │ ├── App.vue │ │ │ ├── components │ │ │ │ ├── Gauge.vue │ │ │ │ ├── ChannelSize.vue │ │ │ │ └── Dashboard.vue │ │ │ └── router │ │ │ │ └── index.ts │ │ ├── .gitignore │ │ ├── README.md │ │ ├── vite.config.js │ │ ├── package.json │ │ ├── tsconfig.json │ │ └── index.html │ ├── appsettings.json │ ├── config │ │ ├── route.js │ │ ├── appsettings.yaml │ │ ├── appsettings-kafka-simple.yaml │ │ ├── appsettings-adv.yaml │ │ └── transform.js │ ├── HttpApi │ │ ├── Health.cs │ │ ├── CountersKeep.cs │ │ └── Counters.cs │ ├── Measurements.cs │ ├── Settings │ │ ├── Filters.cs │ │ ├── EnvConfigProvider.cs │ │ ├── Transformers.cs │ │ └── ReplicatorSettings.cs │ ├── ReplicatorService.cs │ ├── Program.cs │ └── replicator.csproj ├── Kurrent.Replicator.EventStore │ ├── Predefined.cs │ ├── Kurrent.Replicator.EventStore.csproj │ ├── TcpClientLogger.cs │ ├── Configurator.cs │ ├── ConnectionExtensions.cs │ ├── EventFilters.cs │ ├── Realtime.cs │ ├── StreamMetaCache.cs │ └── TcpEventWriter.cs ├── Kurrent.Replicator.KurrentDb │ ├── Predefined.cs │ ├── Internals │ │ ├── MetaSerialization.cs │ │ ├── SystemMetadata.cs │ │ └── StreamAclJsonConverter.cs │ ├── Kurrent.Replicator.KurrentDb.csproj │ ├── Configurator.cs │ ├── ConnectionExtensions.cs │ ├── EventFilters.cs │ ├── Realtime.cs │ ├── StreamMetaCache.cs │ └── GrpcEventWriter.cs ├── Kurrent.Replicator.Shared │ ├── ICheckpointSeeder.cs │ ├── Contracts │ │ ├── TracingMetadata.cs │ │ ├── StreamMetadata.cs │ │ ├── StreamAcl.cs │ │ ├── EventDetails.cs │ │ ├── EventMetadata.cs │ │ ├── ProposedEvent.cs │ │ └── OriginalEvent.cs │ ├── LogPosition.cs │ ├── IEventWriter.cs │ ├── IConfigurator.cs │ ├── Ensure.cs │ ├── Extensions │ │ ├── RegexExtensions.cs │ │ └── DictionaryExtensions.cs │ ├── ICheckpointStore.cs │ ├── Kurrent.Replicator.Shared.csproj │ ├── IEventReader.cs │ ├── Observe │ │ └── ReplicationStatus.cs │ └── Pipeline │ │ ├── Filters.cs │ │ └── Transforms.cs ├── Kurrent.Replicator │ ├── ReplicatorOptions.cs │ ├── IEventDetailsContext.cs │ ├── Sink │ │ ├── SinkPipelineOptions.cs │ │ ├── SinkContext.cs │ │ └── SinkPipe.cs │ ├── NoCheckpointSeeder.cs │ ├── Partitioning │ │ ├── KeyProviders.cs │ │ ├── JsKeyProvider.cs │ │ ├── PartitionChannel.cs │ │ ├── ValuePartitioner.cs │ │ └── HashPartitioner.cs │ ├── Prepare │ │ ├── PreparePipelineOptions.cs │ │ ├── PrepareContext.cs │ │ ├── PreparePipe.cs │ │ ├── EventFilterFilter.cs │ │ └── TransformFilter.cs │ ├── Kurrent.Replicator.csproj │ ├── Factories.cs │ ├── Observers │ │ └── LoggingRetryObserver.cs │ ├── ChaserCheckpointSeeder.cs │ ├── Logging.cs │ ├── Read │ │ └── ReaderPipe.cs │ └── FileCheckpointStore.cs ├── Kurrent.Replicator.Js │ ├── Extensions.cs │ ├── Kurrent.Replicator.Js.csproj │ ├── FunctionLoader.cs │ ├── JsFunction.cs │ └── JsTransform.cs ├── Kurrent.Replicator.Http │ ├── Kurrent.Replicator.Http.csproj │ └── HttpTransform.cs ├── Kurrent.Replicator.Mongo │ ├── Kurrent.Replicator.Mongo.csproj │ └── MongoCheckpointStore.cs ├── Kurrent.Replicator.Kafka │ ├── DefaultRouters.cs │ ├── Kurrent.Replicator.Kafka.csproj │ ├── Configurator.cs │ ├── KafkaJsMessageRouter.cs │ └── KafkaWriter.cs └── Directory.Build.props ├── docs ├── public │ └── favicon.ico ├── src │ ├── assets │ │ ├── replicator.png │ │ ├── replicator1.png │ │ └── kurrent-logo-white.svg │ ├── fonts │ │ ├── Solina-Bold.woff2 │ │ ├── Solina-Light.woff2 │ │ ├── Solina-Medium.woff2 │ │ ├── Solina-Regular.woff2 │ │ └── font-face.css │ ├── content │ │ └── docs │ │ │ ├── deployment │ │ │ ├── images │ │ │ │ └── grafana.png │ │ │ └── docker.mdx │ │ │ ├── features │ │ │ ├── scavenge.md │ │ │ ├── metadata.mdx │ │ │ ├── filters.mdx │ │ │ └── readers.mdx │ │ │ ├── index.mdx │ │ │ └── intro │ │ │ ├── overview.md │ │ │ ├── limitations.mdx │ │ │ └── concepts.mdx │ ├── content.config.ts │ └── styles │ │ ├── global.css │ │ ├── custom.css │ │ └── buttons.css ├── tsconfig.json ├── .gitignore └── package.json ├── test └── Kurrent.Replicator.Tests │ ├── partition.js │ ├── Timing.cs │ ├── Fakes │ └── CheckpointStore.cs │ ├── Logging │ ├── TUnitSink.cs │ └── SerilogExtensions.cs │ ├── Kurrent.Replicator.Tests.csproj │ ├── Fixtures │ └── ContainerFixture.cs │ └── ChaserCheckpointSeedingTests.cs ├── compose ├── grafana │ ├── dashboards.yml │ ├── datasources.yml │ └── __inputs.json ├── prometheus.yml ├── replicator.yml ├── transform.js └── docker-compose.yml ├── .github ├── dependabot.yml ├── workflows │ ├── pr-check.yml │ ├── docker-build.yml │ ├── helm-publish.yml │ ├── pr-build-test.yml │ └── docker-publish.yml └── ISSUE_TEMPLATE │ ├── feature_request.md │ └── bug_report.md ├── example ├── values.yml └── transform.js ├── CHANGELOG.md ├── Kurrent.Replicator.slnx ├── README.md ├── .gitattributes ├── Dockerfile ├── docker-compose.yml └── LICENSE.md /charts/replicator/.gitignore: -------------------------------------------------------------------------------- 1 | charts/* -------------------------------------------------------------------------------- /src/replicator/ClientApp/.browserslistrc: -------------------------------------------------------------------------------- 1 | > 1% 2 | last 2 versions 3 | not dead 4 | -------------------------------------------------------------------------------- /src/replicator/ClientApp/.env: -------------------------------------------------------------------------------- 1 | VUE_APP_NAME="Replicator" 2 | VUE_APP_API_URL=/api 3 | -------------------------------------------------------------------------------- /src/replicator/ClientApp/.env.development: -------------------------------------------------------------------------------- 1 | VUE_APP_API_URL=http://localhost:5000/api 2 | -------------------------------------------------------------------------------- /docs/public/favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kurrent-io/replicator/HEAD/docs/public/favicon.ico -------------------------------------------------------------------------------- /src/replicator/appsettings.json: -------------------------------------------------------------------------------- 1 | { 2 | "AllowedHosts": "*", 3 | "Urls": "http://*:5098" 4 | } 5 | -------------------------------------------------------------------------------- /test/Kurrent.Replicator.Tests/partition.js: -------------------------------------------------------------------------------- 1 | function partition(event) { 2 | return event.data.Tenant; 3 | } -------------------------------------------------------------------------------- /docs/src/assets/replicator.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kurrent-io/replicator/HEAD/docs/src/assets/replicator.png -------------------------------------------------------------------------------- /docs/src/assets/replicator1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kurrent-io/replicator/HEAD/docs/src/assets/replicator1.png -------------------------------------------------------------------------------- /docs/src/fonts/Solina-Bold.woff2: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kurrent-io/replicator/HEAD/docs/src/fonts/Solina-Bold.woff2 -------------------------------------------------------------------------------- /docs/src/fonts/Solina-Light.woff2: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kurrent-io/replicator/HEAD/docs/src/fonts/Solina-Light.woff2 -------------------------------------------------------------------------------- /docs/src/fonts/Solina-Medium.woff2: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kurrent-io/replicator/HEAD/docs/src/fonts/Solina-Medium.woff2 -------------------------------------------------------------------------------- /docs/src/fonts/Solina-Regular.woff2: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kurrent-io/replicator/HEAD/docs/src/fonts/Solina-Regular.woff2 -------------------------------------------------------------------------------- /src/replicator/ClientApp/public/favicon.ico: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kurrent-io/replicator/HEAD/src/replicator/ClientApp/public/favicon.ico -------------------------------------------------------------------------------- /docs/tsconfig.json: -------------------------------------------------------------------------------- 1 | { 2 | "extends": "astro/tsconfigs/strict", 3 | "include": [".astro/types.d.ts", "**/*"], 4 | "exclude": ["dist"] 5 | } 6 | -------------------------------------------------------------------------------- /src/replicator/ClientApp/src/assets/logo.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kurrent-io/replicator/HEAD/src/replicator/ClientApp/src/assets/logo.png -------------------------------------------------------------------------------- /docs/src/content/docs/deployment/images/grafana.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/kurrent-io/replicator/HEAD/docs/src/content/docs/deployment/images/grafana.png -------------------------------------------------------------------------------- /src/replicator/config/route.js: -------------------------------------------------------------------------------- 1 | function route(stream, eventType, data, meta) { 2 | return { 3 | topic: "myTopic", 4 | partitionKey: stream 5 | } 6 | } -------------------------------------------------------------------------------- /src/replicator/ClientApp/src/views/About.vue: -------------------------------------------------------------------------------- 1 | 4 | 5 | 9 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.EventStore/Predefined.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.EventStore; 2 | 3 | public static class Predefined { 4 | public const string MetadataEventType = "$metadata"; 5 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator.KurrentDb/Predefined.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.KurrentDb; 2 | 3 | public static class Predefined { 4 | public const string MetadataEventType = "$metadata"; 5 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/ICheckpointSeeder.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.Shared; 2 | 3 | public interface ICheckpointSeeder { 4 | ValueTask Seed(CancellationToken cancellationToken); 5 | } -------------------------------------------------------------------------------- /src/replicator/ClientApp/src/plugins/element.js: -------------------------------------------------------------------------------- 1 | import ElementPlus from "element-plus"; 2 | import "element-plus/dist/index.css"; 3 | 4 | export default (app) => { 5 | app.use(ElementPlus); 6 | } 7 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator/ReplicatorOptions.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator; 2 | 3 | public record ReplicatorOptions(bool RestartOnFailure, bool RunContinuously, TimeSpan RestartDelay, TimeSpan ReportMetricsFrequency); -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Contracts/TracingMetadata.cs: -------------------------------------------------------------------------------- 1 | using System.Diagnostics; 2 | 3 | namespace Kurrent.Replicator.Shared.Contracts; 4 | 5 | public record TracingMetadata(ActivityTraceId TraceId, ActivitySpanId SpanId); -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/LogPosition.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.Shared; 2 | 3 | public record LogPosition(long EventNumber, ulong EventPosition) { 4 | public static readonly LogPosition Start = new(0, 0); 5 | } -------------------------------------------------------------------------------- /compose/grafana/dashboards.yml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | 3 | providers: 4 | - name: 'replicator' 5 | orgId: 1 6 | folder: '' 7 | type: file 8 | disableDeletion: true 9 | options: 10 | path: /dashboards -------------------------------------------------------------------------------- /src/Kurrent.Replicator/IEventDetailsContext.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | 3 | namespace Kurrent.Replicator; 4 | 5 | public interface IEventDetailsContext { 6 | public EventDetails EventDetails { get; } 7 | } -------------------------------------------------------------------------------- /src/replicator/ClientApp/src/shims-vue.d.ts: -------------------------------------------------------------------------------- 1 | /* eslint-disable */ 2 | declare module '*.vue' { 3 | import type { DefineComponent } from 'vue' 4 | const component: DefineComponent<{}, {}, any> 5 | export default component 6 | } 7 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Js/Extensions.cs: -------------------------------------------------------------------------------- 1 | using System.Text; 2 | 3 | namespace Kurrent.Replicator.Js; 4 | 5 | public static class Extensions { 6 | public static string AsUtf8String(this byte[] data) => Encoding.UTF8.GetString(data); 7 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Http/Kurrent.Replicator.Http.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Sink/SinkPipelineOptions.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.Sink; 2 | 3 | public record SinkPipeOptions( 4 | int PartitionCount = 1, 5 | int BufferSize = 1000, 6 | string? Partitioner = null 7 | ); -------------------------------------------------------------------------------- /compose/grafana/datasources.yml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | 3 | datasources: 4 | - name: prometheus 5 | type: prometheus 6 | access: proxy 7 | orgId: 1 8 | url: http://prometheus:9090 9 | isDefault: true 10 | version: 1 11 | editable: false -------------------------------------------------------------------------------- /charts/replicator/Chart.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | description: Kurrent Replicator Helm chart 3 | name: replicator 4 | version: 0.5.1 5 | icon: https://github.com/kurrent-io/EventStore/blob/db2de074fcf1d4a9b29cfa4baac560706e087386/ouro.png 6 | home: https://replicator.kurrent.io -------------------------------------------------------------------------------- /src/Kurrent.Replicator/NoCheckpointSeeder.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared; 2 | 3 | namespace Kurrent.Replicator; 4 | 5 | public class NoCheckpointSeeder : ICheckpointSeeder { 6 | public ValueTask Seed(CancellationToken cancellationToken) => ValueTask.CompletedTask; 7 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Partitioning/KeyProviders.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | 3 | namespace Kurrent.Replicator.Partitioning; 4 | 5 | public static class KeyProvider { 6 | public static string ByStreamName(BaseProposedEvent evt) => evt.EventDetails.Stream; 7 | } -------------------------------------------------------------------------------- /compose/prometheus.yml: -------------------------------------------------------------------------------- 1 | global: 2 | scrape_interval: 5s 3 | 4 | scrape_configs: 5 | - job_name: 'prometheus' 6 | static_configs: 7 | - targets: ['localhost:9090'] 8 | - job_name: 'replicator' 9 | static_configs: 10 | - targets: 11 | - 'replicator:5000' -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Contracts/StreamMetadata.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.Shared.Contracts; 2 | 3 | public record StreamMetadata( 4 | int? MaxCount, 5 | TimeSpan? MaxAge, 6 | long? TruncateBefore, 7 | TimeSpan? CacheControl, 8 | StreamAcl? StreamAcl 9 | ); -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/IEventWriter.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | 3 | namespace Kurrent.Replicator.Shared; 4 | 5 | public interface IEventWriter { 6 | Task Start(); 7 | Task WriteEvent(BaseProposedEvent proposedEvent, CancellationToken cancellationToken); 8 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/IConfigurator.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.Shared; 2 | 3 | public interface IConfigurator { 4 | string Protocol { get; } 5 | IEventReader ConfigureReader(string connectionString); 6 | IEventWriter ConfigureWriter(string connectionString); 7 | } -------------------------------------------------------------------------------- /compose/grafana/__inputs.json: -------------------------------------------------------------------------------- 1 | { 2 | "__inputs": [ 3 | { 4 | "name": "DS_PROMETHEUS", 5 | "label": "prometheus", 6 | "description": "Default data source", 7 | "type": "datasource", 8 | "pluginId": "prometheus", 9 | "pluginName": "Prometheus" 10 | } 11 | ] 12 | } -------------------------------------------------------------------------------- /docs/src/content.config.ts: -------------------------------------------------------------------------------- 1 | import { defineCollection } from 'astro:content'; 2 | import { docsLoader } from '@astrojs/starlight/loaders'; 3 | import { docsSchema } from '@astrojs/starlight/schema'; 4 | 5 | export const collections = { 6 | docs: defineCollection({ loader: docsLoader(), schema: docsSchema() }), 7 | }; 8 | -------------------------------------------------------------------------------- /docs/src/styles/global.css: -------------------------------------------------------------------------------- 1 | @layer base, starlight, theme, components, utilities; 2 | 3 | @import '@astrojs/starlight-tailwind'; 4 | @import 'tailwindcss/theme.css' layer(theme); 5 | @import 'tailwindcss/utilities.css' layer(utilities); 6 | @import "./tailwind-theme.css"; 7 | @import "./buttons.css" layer(components); 8 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Ensure.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.Shared; 2 | 3 | public static class Ensure { 4 | public static string NotEmpty(string? value, string parameter) 5 | => string.IsNullOrWhiteSpace(value) 6 | ? throw new ArgumentNullException(parameter) 7 | : value; 8 | } -------------------------------------------------------------------------------- /docs/.gitignore: -------------------------------------------------------------------------------- 1 | # build output 2 | dist/ 3 | # generated types 4 | .astro/ 5 | 6 | # dependencies 7 | node_modules/ 8 | 9 | # logs 10 | npm-debug.log* 11 | yarn-debug.log* 12 | yarn-error.log* 13 | pnpm-debug.log* 14 | 15 | 16 | # environment variables 17 | .env 18 | .env.production 19 | 20 | # macOS-specific files 21 | .DS_Store 22 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Js/Kurrent.Replicator.Js.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Contracts/StreamAcl.cs: -------------------------------------------------------------------------------- 1 | // ReSharper disable SuggestBaseTypeForParameter 2 | 3 | namespace Kurrent.Replicator.Shared.Contracts; 4 | 5 | public record StreamAcl( 6 | string[]? ReadRoles, 7 | string[]? WriteRoles, 8 | string[]? DeleteRoles, 9 | string[]? MetaReadRoles, 10 | string[]? MetaWriteRoles 11 | ); -------------------------------------------------------------------------------- /.github/dependabot.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | updates: 3 | - package-ecosystem: npm 4 | directory: "/docs" 5 | schedule: 6 | interval: daily 7 | time: '20:00' 8 | open-pull-requests-limit: 10 9 | - package-ecosystem: bundler 10 | directory: "/docs" 11 | schedule: 12 | interval: daily 13 | time: '20:00' 14 | open-pull-requests-limit: 10 15 | -------------------------------------------------------------------------------- /src/replicator/HttpApi/Health.cs: -------------------------------------------------------------------------------- 1 | using Microsoft.AspNetCore.Mvc; 2 | 3 | namespace replicator.HttpApi; 4 | 5 | [Route("")] 6 | public class Health : ControllerBase { 7 | [HttpGet] 8 | [Route("/ping")] 9 | public string Ping() => "Pong"; 10 | 11 | [HttpGet] 12 | [Route("/health")] 13 | public string Healthy() => "OK"; 14 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Mongo/Kurrent.Replicator.Mongo.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Contracts/EventDetails.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.Shared.Contracts; 2 | 3 | public record EventDetails(string Stream, Guid EventId, string EventType, string ContentType); 4 | 5 | public static class ContentTypes { 6 | public const string Json = "application/json"; 7 | public const string Binary = "application/octet-stream"; 8 | } 9 | -------------------------------------------------------------------------------- /src/replicator/ClientApp/.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | node_modules 3 | /dist 4 | 5 | 6 | # local env files 7 | .env.local 8 | .env.*.local 9 | 10 | # Log files 11 | npm-debug.log* 12 | yarn-debug.log* 13 | yarn-error.log* 14 | pnpm-debug.log* 15 | 16 | # Editor directories and files 17 | .idea 18 | .vscode 19 | *.suo 20 | *.ntvs* 21 | *.njsproj 22 | *.sln 23 | *.sw? 24 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Kafka/DefaultRouters.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.Kafka; 2 | 3 | public static class DefaultRouters { 4 | internal static MessageRoute RouteByCategory(string stream) { 5 | var catIndex = stream.IndexOf('-'); 6 | 7 | var topic = catIndex >= 0 ? stream[..catIndex] : stream; 8 | return new(topic, stream); 9 | } 10 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Prepare/PreparePipelineOptions.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Pipeline; 2 | 3 | namespace Kurrent.Replicator.Prepare; 4 | 5 | public record PreparePipelineOptions( 6 | FilterEvent? Filter, 7 | TransformEvent? Transform, 8 | int TransformConcurrencyLevel = 1, 9 | int BufferSize = 1000 10 | ); -------------------------------------------------------------------------------- /src/replicator/ClientApp/src/main.ts: -------------------------------------------------------------------------------- 1 | import { createApp } from "vue"; 2 | import App from "./App.vue"; 3 | import router from "./router"; 4 | import {store, key} from "./store"; 5 | // @ts-ignore 6 | import installElementPlus from "./plugins/element.js"; 7 | 8 | const app = createApp(App); 9 | installElementPlus(app); 10 | app.use(store, key).use(router).mount("#app"); 11 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.KurrentDb/Internals/MetaSerialization.cs: -------------------------------------------------------------------------------- 1 | using System.Text.Json; 2 | 3 | namespace Kurrent.Replicator.KurrentDb.Internals; 4 | 5 | public static class MetaSerialization { 6 | internal static readonly JsonSerializerOptions StreamMetadataJsonSerializerOptions = new() { 7 | Converters = { StreamMetadataJsonConverter.Instance }, 8 | }; 9 | } 10 | -------------------------------------------------------------------------------- /src/replicator/ClientApp/README.md: -------------------------------------------------------------------------------- 1 | # hello-world 2 | 3 | ## Project setup 4 | ``` 5 | yarn install 6 | ``` 7 | 8 | ### Compiles and hot-reloads for development 9 | ``` 10 | yarn serve 11 | ``` 12 | 13 | ### Compiles and minifies for production 14 | ``` 15 | yarn build 16 | ``` 17 | 18 | ### Customize configuration 19 | See [Configuration Reference](https://cli.vuejs.org/config/). 20 | -------------------------------------------------------------------------------- /charts/replicator/templates/NOTES.txt: -------------------------------------------------------------------------------- 1 | The Kurrent Replicator has been deployed. It might take some time before the pod starts. 2 | 3 | Open the Replicator UI using port forwarding. 4 | 5 | 1. Establish the proxy connection to the service: 6 | 7 | kubectl port-forward --namespace {{ .Release.Namespace }} svc/{{ template "replicator.fullname" . }} 5000 8 | 9 | 2. Open http://localhost:5000 10 | -------------------------------------------------------------------------------- /example/values.yml: -------------------------------------------------------------------------------- 1 | replicator: 2 | reader: 3 | connectionString: "ConnectTo=localhost:2113;UseSslConnection=false;" 4 | pageSize: 2048 5 | sink: 6 | connectionString: "esdb://admin:changeit@c1etr0lo0aeu6ojco770.mesdb.eventstore.cloud:2113" 7 | partitionCount: 1 8 | transform: 9 | type: js 10 | config: transform.js 11 | prometheus: 12 | metrics: true 13 | operator: true 14 | -------------------------------------------------------------------------------- /src/replicator/config/appsettings.yaml: -------------------------------------------------------------------------------- 1 | replicator: 2 | reader: 3 | connectionString: "esdb+discover://admin:zja1cgz1jmy-qjg1VYE@cscn41rtv1lqpuo3qi9g-1.mesdb.eventstore.cloud:2113" 4 | protocol: "grpc" 5 | pageSize: 2048 6 | sink: 7 | connectionString: "esdb://localhost:2113?tls=false" 8 | partitionCount: 10 9 | protocol: "grpc" 10 | checkpoint: 11 | path: "./checkpoint" 12 | -------------------------------------------------------------------------------- /src/replicator/ClientApp/src/views/Home.vue: -------------------------------------------------------------------------------- 1 | 6 | 7 | 18 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Extensions/RegexExtensions.cs: -------------------------------------------------------------------------------- 1 | using System.Text.RegularExpressions; 2 | 3 | namespace Kurrent.Replicator.Shared.Extensions; 4 | 5 | static class RegexExtensions { 6 | public static bool IsNullOrMatch(this Regex? regex, string value) => regex == null || regex.IsMatch(value); 7 | 8 | public static bool IsNullOrDoesntMatch(this Regex? regex, string value) => regex == null || !regex.IsMatch(value); 9 | } -------------------------------------------------------------------------------- /charts/replicator/templates/configmap.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | data: 3 | "appsettings.yaml": | 4 | replicator: 5 | {{ toYaml .Values.replicator | indent 6 }} 6 | kind: ConfigMap 7 | metadata: 8 | name: {{ template "replicator.fullname" . }} 9 | labels: 10 | app: {{ template "replicator.name" . }} 11 | chart: {{ template "replicator.chart" . }} 12 | release: {{ .Release.Name }} 13 | heritage: {{ .Release.Service }} -------------------------------------------------------------------------------- /src/replicator/ClientApp/vite.config.js: -------------------------------------------------------------------------------- 1 | import { defineConfig } from 'vite' 2 | import vue from '@vitejs/plugin-vue' 3 | import path from "path"; 4 | 5 | // https://vitejs.dev/config/ 6 | export default defineConfig({ 7 | plugins: [vue()], 8 | resolve: { 9 | alias: { 10 | "@": path.resolve(__dirname, "./src"), 11 | }, 12 | extensions: ['.mjs', '.js', '.ts', '.jsx', '.tsx', '.json', '.vue'], 13 | }, 14 | }) -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Kurrent.Replicator.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | -------------------------------------------------------------------------------- /src/replicator/config/appsettings-kafka-simple.yaml: -------------------------------------------------------------------------------- 1 | replicator: 2 | reader: 3 | connectionString: ConnectTo=tcp://admin:changeit@localhost:1113; HeartBeatTimeout=500; UseSslConnection=false; 4 | protocol: tcp 5 | sink: 6 | connectionString: bootstrap.servers=localhost:9092 7 | protocol: kafka 8 | partitionCount: 1 9 | router: ./config/route.js 10 | scavenge: false 11 | filters: [] 12 | checkpoint: 13 | path: "./checkpoint" 14 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/ICheckpointStore.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.Shared; 2 | 3 | public interface ICheckpointStore { 4 | ValueTask HasStoredCheckpoint(CancellationToken cancellationToken); 5 | 6 | ValueTask LoadCheckpoint(CancellationToken cancellationToken); 7 | 8 | ValueTask StoreCheckpoint(LogPosition logPosition, CancellationToken cancellationToken); 9 | 10 | ValueTask Flush(CancellationToken cancellationToken); 11 | } -------------------------------------------------------------------------------- /test/Kurrent.Replicator.Tests/Timing.cs: -------------------------------------------------------------------------------- 1 | using System.Diagnostics; 2 | using Serilog; 3 | 4 | namespace Kurrent.Replicator.Tests; 5 | 6 | public static class Timing { 7 | public static async Task Measure(string what, Task task) { 8 | var watch = new Stopwatch(); 9 | watch.Start(); 10 | await task; 11 | watch.Stop(); 12 | Log.Information("{What} took {Time}", what, TimeSpan.FromMilliseconds(watch.ElapsedMilliseconds)); 13 | } 14 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Sink/SinkContext.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | using GreenPipes; 3 | 4 | namespace Kurrent.Replicator.Sink; 5 | 6 | public class SinkContext(BaseProposedEvent proposedEvent, CancellationToken cancellationToken) : BasePipeContext(cancellationToken), PipeContext, IEventDetailsContext { 7 | public BaseProposedEvent ProposedEvent { get; } = proposedEvent; 8 | public EventDetails EventDetails => ProposedEvent.EventDetails; 9 | } 10 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Js/FunctionLoader.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.Js; 2 | 3 | public static class FunctionLoader { 4 | public static string? LoadFile(string? name, string description) { 5 | if (name == null) { 6 | return null; 7 | } 8 | 9 | if (!File.Exists(name)) { 10 | throw new ArgumentException($"{description} function file doesn't exist", nameof(name)); 11 | } 12 | 13 | return File.ReadAllText(name); 14 | } 15 | } 16 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Kurrent.Replicator.Shared.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | LIBLOG_PUBLIC 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/IEventReader.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | 3 | namespace Kurrent.Replicator.Shared; 4 | 5 | public interface IEventReader { 6 | string Protocol { get; } 7 | 8 | Task ReadEvents(LogPosition fromLogPosition, Func next, CancellationToken cancellationToken); 9 | 10 | Task GetLastPosition(CancellationToken cancellationToken); 11 | 12 | ValueTask Filter(BaseOriginalEvent originalEvent); 13 | } -------------------------------------------------------------------------------- /compose/replicator.yml: -------------------------------------------------------------------------------- 1 | replicator: 2 | reader: 3 | connectionString: ConnectTo=tcp://admin:changeit@10.211.55.3:1113; HeartBeatTimeout=500; UseSslConnection=false; 4 | protocol: tcp 5 | sink: 6 | connectionString: esdb://10.211.55.3:2114?tls=false 7 | protocol: grpc 8 | partitionCount: 1 9 | bufferSize: 1000 10 | scavenge: false 11 | transform: 12 | type: js 13 | config: ./transform.js 14 | bufferSize: 1000 15 | filters: [] 16 | checkpoint: 17 | path: "./checkpoint" 18 | -------------------------------------------------------------------------------- /docs/src/styles/custom.css: -------------------------------------------------------------------------------- 1 | :root { 2 | --sl-font: 'Solina', serif; 3 | --sl-color-text: var(--color-base-100); 4 | --sl-color-text-accent: var(--color-plum-200); 5 | --sl-color-gray-3: var(--color-base-300); 6 | --sl-color-bg: #000000; 7 | --sl-text-xs: 0.875rem 8 | } 9 | 10 | :root[data-theme='light'] { 11 | --sl-color-bg: var(--color-base-50); 12 | --sl-color-text: var(--color-base-900); 13 | --sl-color-text-accent: var(--color-plum-500); 14 | --sl-color-gray-3: var(--color-base-600); 15 | } -------------------------------------------------------------------------------- /src/replicator/config/appsettings-adv.yaml: -------------------------------------------------------------------------------- 1 | replicator: 2 | reader: 3 | connectionString: ConnectTo=tcp://admin:changeit@localhost:1113; HeartBeatTimeout=500; UseSslConnection=false; 4 | protocol: tcp 5 | sink: 6 | connectionString: bootstrap.servers=localhost:9092 7 | protocol: kafka 8 | partitionCount: 1 9 | router: ./config/route.js 10 | scavenge: false 11 | transform: 12 | type: js 13 | config: ./config/transform.js 14 | filters: [] 15 | checkpoint: 16 | path: "./checkpoint" 17 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Extensions/DictionaryExtensions.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Concurrent; 2 | 3 | namespace Kurrent.Replicator.Shared.Extensions; 4 | 5 | public static class DictionaryExtensions { 6 | public static async Task GetOrAddAsync(this ConcurrentDictionary dict, string key, Func> get) { 7 | if (dict.TryGetValue(key, out var value)) return value; 8 | 9 | var newValue = await get(); 10 | dict.TryAdd(key, newValue); 11 | 12 | return newValue; 13 | } 14 | } 15 | -------------------------------------------------------------------------------- /charts/replicator/templates/pvc.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: PersistentVolumeClaim 3 | metadata: 4 | name: {{ template "replicator.fullname" . }} 5 | labels: 6 | app: {{ template "replicator.name" . }} 7 | chart: {{ template "replicator.chart" . }} 8 | release: {{ .Release.Name }} 9 | heritage: {{ .Release.Service }} 10 | spec: 11 | accessModes: 12 | - ReadWriteOnce 13 | resources: 14 | requests: 15 | storage: 1Mi 16 | {{ if .Values.pvc.storageClass }} 17 | storageClassName: {{ .Values.pvc.storageClass }} 18 | {{ end }} -------------------------------------------------------------------------------- /charts/replicator/templates/configmap-transform.yaml: -------------------------------------------------------------------------------- 1 | {{- if eq (include "replicator.shouldUseJavascriptTransform" .) "true" -}} 2 | apiVersion: v1 3 | data: 4 | {{ include "replicator.transform.filename" . | indent 2 }}: |- 5 | {{ .Values.transformJs | indent 4 }} 6 | 7 | kind: ConfigMap 8 | metadata: 9 | name: {{ template "replicator.fullname" . }}-transform-js 10 | labels: 11 | app: {{ template "replicator.name" . }} 12 | chart: {{ template "replicator.chart" . }} 13 | release: {{ .Release.Name }} 14 | heritage: {{ .Release.Service }} 15 | {{- end }} -------------------------------------------------------------------------------- /charts/replicator/templates/configmap-partitioner.yaml: -------------------------------------------------------------------------------- 1 | {{- if eq (include "replicator.shouldUseCustomPartitioner" .) "true" -}} 2 | apiVersion: v1 3 | data: 4 | {{ include "replicator.sink.partitioner.filename" . | indent 2 }}: |- 5 | {{ .Values.partitionerJs | indent 4 }} 6 | 7 | kind: ConfigMap 8 | metadata: 9 | name: {{ template "replicator.fullname" . }}-partitioner-js 10 | labels: 11 | app: {{ template "replicator.name" . }} 12 | chart: {{ template "replicator.chart" . }} 13 | release: {{ .Release.Name }} 14 | heritage: {{ .Release.Service }} 15 | {{- end }} -------------------------------------------------------------------------------- /compose/transform.js: -------------------------------------------------------------------------------- 1 | function transform(original) { 2 | log.debug("Transforming event {Type} from {Stream}", original.EventType, original.Stream); 3 | 4 | if (original.Stream.length > 7) return undefined; 5 | 6 | const newEvent = { 7 | ...original.Data, 8 | Data1: `new${original.Data.Data1}`, 9 | NewProp: `${original.Data.Id} - ${original.Data.Data2}` 10 | }; 11 | return { 12 | Stream: `transformed${original.Stream}`, 13 | EventType: `V2.${original.EventType}`, 14 | Data: newEvent, 15 | Meta: original.Meta 16 | } 17 | } -------------------------------------------------------------------------------- /example/transform.js: -------------------------------------------------------------------------------- 1 | function transform(original) { 2 | log.debug("Transforming event {Type} from {Stream}", original.EventType, original.Stream); 3 | 4 | if (original.Stream.length > 7) return undefined; 5 | 6 | const newEvent = { 7 | ...original.Data, 8 | Data1: `new${original.Data.Data1}`, 9 | NewProp: `${original.Data.Id} - ${original.Data.Data2}` 10 | }; 11 | return { 12 | Stream: `transformed${original.Stream}`, 13 | EventType: `V2.${original.EventType}`, 14 | Data: newEvent, 15 | Meta: original.Meta 16 | } 17 | } -------------------------------------------------------------------------------- /src/replicator/ClientApp/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "replicator", 3 | "version": "0.1.0", 4 | "private": true, 5 | "scripts": { 6 | "dev": "vite", 7 | "build": "vite build", 8 | "serve": "vite preview" 9 | }, 10 | "dependencies": { 11 | "axios": "^1.6.7", 12 | "element-plus": "^2.5.5", 13 | "vue": "^3.4.15", 14 | "vue-router": "^4.2.5", 15 | "vuex": "^4.1.0" 16 | }, 17 | "devDependencies": { 18 | "@vitejs/plugin-vue": "^1.6.1", 19 | "vite": "^2.5.4", 20 | "sass": "^1.62.1", 21 | "stylus": "^0.59.0", 22 | "typescript": "~5.0.4" 23 | } 24 | } 25 | -------------------------------------------------------------------------------- /src/replicator/config/transform.js: -------------------------------------------------------------------------------- 1 | function transform(original) { 2 | log.debug("Transforming event {Type} from {Stream}", original.EventType, original.Stream); 3 | 4 | if (original.Stream.length > 7) return undefined; 5 | 6 | const newEvent = { 7 | ...original.Data, 8 | Data1: `new${original.Data.Data1}`, 9 | NewProp: `${original.Data.Id} - ${original.Data.Data2}` 10 | }; 11 | return { 12 | Stream: `transformed${original.Stream}`, 13 | EventType: `V2.${original.EventType}`, 14 | Data: newEvent, 15 | Meta: original.Meta 16 | } 17 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator.KurrentDb/Kurrent.Replicator.KurrentDb.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Observe/ReplicationStatus.cs: -------------------------------------------------------------------------------- 1 | using System.Diagnostics; 2 | 3 | namespace Kurrent.Replicator.Shared.Observe; 4 | 5 | public static class ReplicationStatus { 6 | static Stopwatch Watch { get; } = new(); 7 | 8 | public static void Start() { 9 | Watch.Restart(); 10 | ReaderRunning = true; 11 | } 12 | 13 | public static void Stop() { 14 | Watch.Stop(); 15 | ReaderRunning = false; 16 | } 17 | 18 | public static double ElapsedSeconds => Watch.Elapsed.TotalSeconds; 19 | 20 | public static bool ReaderRunning { get; set; } 21 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Kafka/Kurrent.Replicator.Kafka.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | -------------------------------------------------------------------------------- /.github/workflows/pr-check.yml: -------------------------------------------------------------------------------- 1 | name: Pull Request Check 2 | on: 3 | pull_request: 4 | paths-ignore: 5 | - "test/**" 6 | - "example/**" 7 | - "docs/**" 8 | - "**.md" 9 | - "**.mdx" 10 | - ".github/**" 11 | - ".gitignore" 12 | - ".gitattributes" 13 | - ".editorconfig" 14 | types: [opened, edited] 15 | jobs: 16 | checkPullRequest: 17 | name: Pull Request check 18 | runs-on: ubuntu-latest 19 | steps: 20 | - 21 | name: Checkout 22 | uses: actions/checkout@v4 23 | - 24 | name: Check pull requests 25 | uses: EventStore/Automations/pr-check@master -------------------------------------------------------------------------------- /src/replicator/ClientApp/src/App.vue: -------------------------------------------------------------------------------- 1 | 20 | 21 | -------------------------------------------------------------------------------- /src/Directory.Build.props: -------------------------------------------------------------------------------- 1 | 2 | 3 | $([System.IO.Directory]::GetParent($(MSBuildThisFileDirectory)).Parent.FullName) 4 | net9.0 5 | latest 6 | false 7 | $(NoWarn);CS1591;CS0618; 8 | enable 9 | enable 10 | 11 | 12 | 13 | 14 | 15 | 16 | -------------------------------------------------------------------------------- /.github/workflows/docker-build.yml: -------------------------------------------------------------------------------- 1 | name: Replicator / Build Docker Image 2 | on: 3 | push: 4 | paths: 5 | - 'src/**' 6 | branches: 7 | - 'master' 8 | pull_request: 9 | paths: 10 | - 'src/**' 11 | branches: 12 | - 'master' 13 | 14 | jobs: 15 | docker: 16 | runs-on: ubuntu-latest 17 | steps: 18 | - 19 | name: Checkout 20 | uses: actions/checkout@v4 21 | - 22 | name: Set up Docker Buildx 23 | uses: docker/setup-buildx-action@v3 24 | - 25 | name: Build 26 | uses: docker/build-push-action@v6 27 | with: 28 | context: . 29 | push: false 30 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/feature_request.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Feature request 3 | about: Suggest an idea for this project 4 | title: '' 5 | labels: kind/enhancement 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Is your feature request related to a problem? Please describe.** 11 | A clear and concise description of what the problem is. 12 | 13 | **Describe the solution you'd like** 14 | A clear and concise description of what you want to happen. 15 | 16 | **Describe alternatives you've considered** 17 | A clear and concise description of any alternative solutions or features you've considered. 18 | 19 | **Additional context** 20 | Add any other context or screenshots about the feature request here. -------------------------------------------------------------------------------- /src/replicator/ClientApp/src/components/Gauge.vue: -------------------------------------------------------------------------------- 1 | 7 | 8 | 17 | 18 | 31 | -------------------------------------------------------------------------------- /test/Kurrent.Replicator.Tests/Fakes/CheckpointStore.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared; 2 | 3 | namespace Kurrent.Replicator.Tests.Fakes; 4 | 5 | public class CheckpointStore : ICheckpointStore { 6 | public ValueTask HasStoredCheckpoint(CancellationToken cancellationToken) 7 | => ValueTask.FromResult(true); 8 | 9 | public ValueTask LoadCheckpoint(CancellationToken cancellationToken) 10 | => ValueTask.FromResult(LogPosition.Start); 11 | 12 | public ValueTask StoreCheckpoint(LogPosition logPosition, CancellationToken cancellationToken) 13 | => ValueTask.CompletedTask; 14 | 15 | public ValueTask Flush(CancellationToken cancellationToken) => ValueTask.CompletedTask; 16 | } -------------------------------------------------------------------------------- /src/replicator/ClientApp/src/router/index.ts: -------------------------------------------------------------------------------- 1 | import { createRouter, createWebHistory, RouteRecordRaw } from 'vue-router' 2 | import Home from '../views/Home.vue' 3 | 4 | const routes: Array = [ 5 | { 6 | path: '/', 7 | name: 'Home', 8 | component: Home 9 | }, 10 | { 11 | path: '/about', 12 | name: 'About', 13 | // route level code-splitting 14 | // this generates a separate chunk (about.[hash].js) for this route 15 | // which is lazy-loaded when the route is visited. 16 | component: () => import('../views/About.vue') 17 | } 18 | ] 19 | 20 | const router = createRouter({ 21 | history: createWebHistory(process.env.BASE_URL), 22 | routes 23 | }) 24 | 25 | export default router 26 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.EventStore/Kurrent.Replicator.EventStore.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | -------------------------------------------------------------------------------- /test/Kurrent.Replicator.Tests/Logging/TUnitSink.cs: -------------------------------------------------------------------------------- 1 | using Serilog.Core; 2 | using Serilog.Events; 3 | using Serilog.Formatting; 4 | 5 | namespace Kurrent.Replicator.Tests.Logging; 6 | 7 | public class TestOutputSink(ITextFormatter textFormatter) : ILogEventSink { 8 | private readonly ITextFormatter _textFormatter = textFormatter ?? throw new ArgumentNullException(nameof(textFormatter)); 9 | 10 | public void Emit(LogEvent logEvent) { 11 | ArgumentNullException.ThrowIfNull(logEvent); 12 | 13 | var renderSpace = new StringWriter(); 14 | _textFormatter.Format(logEvent, renderSpace); 15 | var message = renderSpace.ToString().Trim(); 16 | TestContext.Current?.OutputWriter.WriteLine(message); 17 | } 18 | } 19 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Contracts/EventMetadata.cs: -------------------------------------------------------------------------------- 1 | using System.Text.Json.Serialization; 2 | 3 | namespace Kurrent.Replicator.Shared.Contracts; 4 | 5 | public record EventMetadata { 6 | public const string EventNumberPropertyName = "$originalEventNumber"; 7 | public const string PositionPropertyName = "$originalPosition"; 8 | public const string CreatedDate = "$originalCreatedDate"; 9 | 10 | [JsonPropertyName(EventNumberPropertyName)] 11 | public long OriginalEventNumber { get; init; } 12 | 13 | [JsonPropertyName(PositionPropertyName)] 14 | public ulong OriginalPosition { get; init; } 15 | 16 | [JsonPropertyName(CreatedDate)] 17 | public DateTimeOffset OriginalCreatedDate { get; init; } 18 | } -------------------------------------------------------------------------------- /docs/src/content/docs/features/scavenge.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Scavenge" 3 | sidebar: 4 | order: 1 5 | description: > 6 | Filter out events from deleted streams, expired by age or count, but not yet scavenged. 7 | --- 8 | 9 | By default, Replicator would watch for events, which should not be present in the source cluster as they are considered as deleted, but not removed from the database due to lack of scavenge. 10 | 11 | Example cases: 12 | - The database wasn't scavenged for a while 13 | - A stream was deleted, but not scavenged yet 14 | - A stream was truncated either by max age or max count, but not scavenged yet 15 | 16 | All those events will not be replicated. 17 | 18 | This feature can be disabled by setting the `replicator.scavenge` option to `false`. 19 | -------------------------------------------------------------------------------- /docs/src/fonts/font-face.css: -------------------------------------------------------------------------------- 1 | @font-face { 2 | font-family: "Solina"; 3 | src: 4 | url("./Solina-Light.woff2") format("woff2"); 5 | font-weight: 400; 6 | font-style: normal; 7 | } 8 | 9 | @font-face { 10 | font-family: "Solina"; 11 | src: 12 | url("./Solina-Regular.woff2") format("woff2"); 13 | font-weight: 450; 14 | font-style: normal; 15 | } 16 | 17 | @font-face { 18 | font-family: "Solina"; 19 | src: 20 | url("./Solina-Medium.woff2") format("woff2"); 21 | font-weight: 500; 22 | font-style: normal; 23 | } 24 | 25 | @font-face { 26 | font-family: "Solina"; 27 | src: 28 | url("./Solina-Bold.woff2") format("woff2"); 29 | font-weight: 700; 30 | font-style: normal; 31 | } 32 | -------------------------------------------------------------------------------- /charts/replicator/templates/podmonitor.yaml: -------------------------------------------------------------------------------- 1 | {{- if and .Values.prometheus.metrics .Values.prometheus.operator }} 2 | apiVersion: monitoring.coreos.com/v1 3 | kind: PodMonitor 4 | metadata: 5 | name: {{ template "replicator.fullname" . }} 6 | labels: 7 | app: {{ template "replicator.name" . }} 8 | chart: {{ template "replicator.chart" . }} 9 | release: {{ .Release.Name }} 10 | heritage: {{ .Release.Service }} 11 | tier: web 12 | spec: 13 | podMetricsEndpoints: 14 | - interval: 15s 15 | path: /metrics 16 | port: web 17 | namespaceSelector: 18 | matchNames: 19 | - {{ .Release.Namespace }} 20 | selector: 21 | matchLabels: 22 | app: {{ template "replicator.name" . }} 23 | release: {{ .Release.Name }} 24 | {{- end }} -------------------------------------------------------------------------------- /.github/workflows/helm-publish.yml: -------------------------------------------------------------------------------- 1 | name: Replicator / Publish Helm Chart 2 | on: 3 | push: 4 | tags: 5 | - '*.*.*' 6 | 7 | jobs: 8 | helm: 9 | runs-on: ubuntu-latest 10 | steps: 11 | - 12 | name: Checkout 13 | uses: actions/checkout@v4 14 | with: 15 | fetch-depth: 0 16 | - 17 | name: Install Helm 18 | uses: azure/setup-helm@v4 19 | - 20 | name: Configure Git 21 | run: | 22 | git config user.name "$GITHUB_ACTOR" 23 | git config user.email "$GITHUB_ACTOR@users.noreply.github.com" 24 | - 25 | name: Publish Helm Chart 26 | uses: helm/chart-releaser-action@v1.7.0 27 | env: 28 | CR_TOKEN: "${{ secrets.GITHUB_TOKEN }}" -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | All notable changes to this project will be documented in this files. 3 | 4 | The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/). 5 | 6 | ## [Unreleased] 7 | ### Added 8 | - Replicator docs (hugo site) to repository. [replicator#90](https://github.com/EventStore/replicator/pull/90) 9 | - Replicator docs workflows. [replicator#90](https://github.com/EventStore/replicator/pull/90) 10 | 11 | ### Changed 12 | - Updated workflows to add clear naming separation between replicator (app) and replicator (docs) workflows. [replicator#90](https://github.com/EventStore/replicator/pull/90) 13 | - Updated hugo site to use hugo modules, removed blog content. [replicator#90](https://github.com/EventStore/replicator/pull/90) 14 | 15 | 16 | -------------------------------------------------------------------------------- /charts/replicator/templates/service.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: v1 2 | kind: Service 3 | metadata: 4 | annotations: 5 | {{- if and .Values.prometheus.metrics (not .Values.prometheus.operator) }} 6 | prometheus.io/scrape: "true" 7 | prometheus.io/port: "5000" 8 | {{- end }} 9 | name: {{ template "replicator.fullname" . }} 10 | labels: 11 | app: {{ template "replicator.name" . }} 12 | chart: {{ template "replicator.chart" . }} 13 | release: {{ .Release.Name }} 14 | heritage: {{ .Release.Service }} 15 | tier: web 16 | spec: 17 | ports: 18 | - name: web 19 | port: 5000 20 | protocol: TCP 21 | targetPort: 5000 22 | selector: 23 | app: {{ template "replicator.name" . }} 24 | release: {{ .Release.Name }} 25 | tier: web 26 | type: ClusterIP 27 | -------------------------------------------------------------------------------- /docs/package.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "docs", 3 | "type": "module", 4 | "version": "0.0.1", 5 | "scripts": { 6 | "dev": "astro dev", 7 | "start": "astro dev", 8 | "build": "astro build", 9 | "preview": "astro preview", 10 | "astro": "astro" 11 | }, 12 | "dependencies": { 13 | "@astrojs/cloudflare": "^12.5.0", 14 | "@astrojs/starlight": "^0.34.1", 15 | "@astrojs/starlight-tailwind": "^4.0.1", 16 | "@tailwindcss/vite": "^4.1.4", 17 | "astro": "^5.6.1", 18 | "astro-rehype-relative-markdown-links": "^0.18.1", 19 | "sharp": "^0.32.5", 20 | "tailwindcss": "^4.1.4" 21 | }, 22 | "packageManager": "pnpm@10.10.0+sha512.d615db246fe70f25dcfea6d8d73dee782ce23e2245e3c4f6f888249fb568149318637dca73c2c5c8ef2a4ca0d5657fb9567188bfab47f566d1ee6ce987815c39" 23 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Factories.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared; 2 | 3 | namespace Kurrent.Replicator; 4 | 5 | public class Factory(IEnumerable configurators) { 6 | readonly List _configurators = configurators.ToList(); 7 | 8 | public IEventReader GetReader(string protocol, string connectionString) 9 | => GetConfigurator(protocol).ConfigureReader(connectionString); 10 | 11 | public IEventWriter GetWriter(string protocol, string connectionString) 12 | => GetConfigurator(protocol).ConfigureWriter(connectionString); 13 | 14 | IConfigurator GetConfigurator(string protocol) { 15 | var configurator = _configurators.FirstOrDefault(x => x.Protocol == protocol); 16 | 17 | return configurator ?? throw new NotSupportedException($"Unsupported protocol: {protocol}"); 18 | } 19 | } -------------------------------------------------------------------------------- /charts/replicator/values.yaml: -------------------------------------------------------------------------------- 1 | prometheus: 2 | metrics: false 3 | operator: false 4 | pvc: 5 | storageClass: null 6 | terminationGracePeriodSeconds: 300 7 | jsConfigMaps: [] 8 | resources: 9 | limits: 10 | cpu: 1 11 | memory: 1Gi 12 | requests: 13 | cpu: 250m 14 | memory: 512Mi 15 | image: 16 | registry: docker.io 17 | repository: eventstore/replicator 18 | tag: 0.4.7 19 | pullPolicy: IfNotPresent 20 | replicator: 21 | reader: 22 | connectionString: 23 | protocol: tcp 24 | sink: 25 | connectionString: 26 | protocol: grpc 27 | partitionCount: 1 28 | partitioner: 29 | bufferSize: 1000 30 | scavenge: true 31 | filters: [] 32 | transform: null 33 | checkpoint: 34 | type: file 35 | path: ./checkpoint 36 | checkpointAfter: 1000 37 | # restartOnFailure: false 38 | # runContinuously: true 39 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.KurrentDb/Configurator.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.KurrentDb; 2 | 3 | public class GrpcConfigurator : IConfigurator { 4 | public string Protocol => "grpc"; 5 | 6 | public IEventReader ConfigureReader(string connectionString) 7 | => new GrpcEventReader(ConfigureEventStoreGrpc(connectionString, true)); 8 | 9 | public IEventWriter ConfigureWriter(string connectionString) 10 | => new GrpcEventWriter(ConfigureEventStoreGrpc(connectionString, false)); 11 | 12 | static EventStoreClient ConfigureEventStoreGrpc(string connectionString, bool follower) { 13 | var settings = EventStoreClientSettings.Create(connectionString); 14 | 15 | if (follower) { 16 | settings.ConnectivitySettings.NodePreference = NodePreference.Follower; 17 | } 18 | 19 | return new(settings); 20 | } 21 | } -------------------------------------------------------------------------------- /src/replicator/ClientApp/tsconfig.json: -------------------------------------------------------------------------------- 1 | { 2 | "compilerOptions": { 3 | "target": "esnext", 4 | "module": "esnext", 5 | "strict": true, 6 | "jsx": "preserve", 7 | "importHelpers": true, 8 | "moduleResolution": "node", 9 | "skipLibCheck": true, 10 | "esModuleInterop": true, 11 | "allowSyntheticDefaultImports": true, 12 | "sourceMap": true, 13 | "baseUrl": ".", 14 | "types": [ 15 | "webpack-env" 16 | ], 17 | "paths": { 18 | "@/*": [ 19 | "src/*" 20 | ] 21 | }, 22 | "lib": [ 23 | "esnext", 24 | "dom", 25 | "dom.iterable", 26 | "scripthost" 27 | ] 28 | }, 29 | "include": [ 30 | "src/**/*.ts", 31 | "src/**/*.tsx", 32 | "src/**/*.vue", 33 | "tests/**/*.ts", 34 | "tests/**/*.tsx" 35 | ], 36 | "exclude": [ 37 | "node_modules" 38 | ] 39 | } 40 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.EventStore/TcpClientLogger.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Logging; 2 | 3 | namespace Kurrent.Replicator.EventStore; 4 | 5 | public class TcpClientLogger : ILogger { 6 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 7 | 8 | public void Error(string format, params object[] args) => Log.Error(format, args); 9 | 10 | public void Error(Exception ex, string format, params object[] args) => Log.Error(ex, format, args); 11 | 12 | public void Info(string format, params object[] args) => Log.Info(format, args); 13 | 14 | public void Info(Exception ex, string format, params object[] args) => Log.Info(ex, format, args); 15 | 16 | public void Debug(string format, params object[] args) => Log.Debug(format, args); 17 | 18 | public void Debug(Exception ex, string format, params object[] args) => Log.Debug(ex, format, args); 19 | } -------------------------------------------------------------------------------- /src/replicator/Measurements.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Observe; 2 | using Ubiquitous.Metrics; 3 | using Ubiquitous.Metrics.InMemory; 4 | using Ubiquitous.Metrics.Labels; 5 | using Ubiquitous.Metrics.Prometheus; 6 | 7 | namespace replicator; 8 | 9 | public static class Measurements { 10 | static readonly InMemoryMetricsProvider InMemory = new(); 11 | 12 | public static void ConfigureMetrics(string environment) { 13 | var metrics = Metrics.CreateUsing( 14 | new PrometheusConfigurator(new Label("app", "replicator"), new Label("environment", environment)), 15 | InMemory 16 | ); 17 | ReplicationMetrics.Configure(metrics); 18 | } 19 | 20 | public static InMemoryGauge GetGauge(string name) => InMemory.GetGauge(name)!; 21 | 22 | public static InMemoryHistogram GetHistogram(string name) => InMemory.GetHistogram(name)!; 23 | } 24 | -------------------------------------------------------------------------------- /.github/ISSUE_TEMPLATE/bug_report.md: -------------------------------------------------------------------------------- 1 | --- 2 | name: Bug report 3 | about: Create a report to help us improve 4 | title: '' 5 | labels: kind/bug 6 | assignees: '' 7 | 8 | --- 9 | 10 | **Describe the bug** 11 | A clear and concise description of what the bug is. 12 | 13 | **To Reproduce** 14 | Steps to reproduce the behavior: 15 | 1. 16 | 2. 17 | 3. 18 | 4. 19 | 20 | **Expected behavior** 21 | A clear and concise description of what you expected to happen. 22 | 23 | **Actual behavior** 24 | A clear and concise description of what actually happened. 25 | 26 | **Config/Logs/Screenshots** 27 | If applicable, please attach your node configuration, logs or any screenshots. 28 | 29 | **EventStore details** 30 | - EventStore server version: 31 | 32 | - Operating system: 33 | 34 | - EventStore client version (if applicable): 35 | 36 | **Additional context** 37 | Add any other context about the problem here. -------------------------------------------------------------------------------- /Kurrent.Replicator.slnx: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Prepare/PrepareContext.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | using GreenPipes; 3 | 4 | namespace Kurrent.Replicator.Prepare; 5 | 6 | public class PrepareContext : BasePipeContext, PipeContext, IEventDetailsContext { 7 | public PrepareContext(BaseOriginalEvent originalEvent, CancellationToken cancellationToken) 8 | : base( cancellationToken ) 9 | => OriginalEvent = originalEvent; 10 | 11 | public BaseOriginalEvent OriginalEvent { get; private set; } 12 | 13 | public void IgnoreEvent() 14 | => OriginalEvent = new IgnoredOriginalEvent( 15 | OriginalEvent.Created, 16 | OriginalEvent.EventDetails, 17 | OriginalEvent.LogPosition, 18 | OriginalEvent.SequenceNumber, 19 | OriginalEvent.TracingMetadata 20 | ); 21 | 22 | public EventDetails EventDetails => OriginalEvent.EventDetails; 23 | } -------------------------------------------------------------------------------- /compose/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3.7' 2 | 3 | services: 4 | 5 | replicator: 6 | container_name: repl-replicator 7 | image: eventstore/replicator:latest 8 | ports: 9 | - "5000:5000" 10 | volumes: 11 | - ./replicator.yml:/app/config/appsettings.yaml 12 | - ./transform.js:/app/transform.js 13 | environment: 14 | REPLICATOR_DEBUG: 1 15 | 16 | prometheus: 17 | container_name: repl-prometheus 18 | image: prom/prometheus:v2.17.1 19 | ports: 20 | - "9090:9090" 21 | volumes: 22 | - ./prometheus.yml:/etc/prometheus/prometheus.yml 23 | 24 | grafana: 25 | container_name: repl-grafana 26 | image: grafana/grafana:6.7.2 27 | ports: 28 | - "3000:3000" 29 | volumes: 30 | - ./grafana/dashboards.yml:/etc/grafana/provisioning/dashboards/rabbitmq.yaml 31 | - ./grafana/datasources.yml:/etc/grafana/provisioning/datasources/prometheus.yaml 32 | - ./grafana/dashboards:/dashboards 33 | -------------------------------------------------------------------------------- /src/replicator/Settings/Filters.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared; 2 | using Kurrent.Replicator.Shared.Pipeline; 3 | 4 | namespace replicator.Settings; 5 | 6 | public static class EventFilters { 7 | public static FilterEvent? GetFilter(Replicator settings, IEventReader reader) { 8 | var filters = (settings.Filters ?? Array.Empty()).Select(Configure).ToList(); 9 | if (settings.Scavenge) filters.Add(reader.Filter); 10 | 11 | return filters.Count > 1 12 | ? x => Filters.CombinedFilter(x, filters.ToArray()) 13 | : filters.Count == 0 ? null : filters[0]; 14 | 15 | static FilterEvent Configure(Filter cfg) => cfg.Type switch { 16 | "eventType" => new Filters.EventTypeFilter(cfg.Include, cfg.Exclude).Filter, 17 | "streamName" => new Filters.StreamNameFilter(cfg.Include, cfg.Exclude).Filter, 18 | _ => throw new ArgumentException($"Unknown filter: {cfg.Type}") 19 | }; 20 | } 21 | } -------------------------------------------------------------------------------- /src/replicator/ReplicatorService.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator; 2 | using Kurrent.Replicator.Prepare; 3 | using Kurrent.Replicator.Shared; 4 | using Kurrent.Replicator.Sink; 5 | 6 | namespace replicator; 7 | 8 | public class ReplicatorService( 9 | IEventReader reader, 10 | IEventWriter writer, 11 | SinkPipeOptions sinkOptions, 12 | PreparePipelineOptions prepareOptions, 13 | ReplicatorOptions replicatorOptions, 14 | ICheckpointSeeder checkpointSeeder, 15 | ICheckpointStore checkpointStore 16 | ) 17 | : BackgroundService { 18 | protected override Task ExecuteAsync(CancellationToken stoppingToken) 19 | => Replicator.Replicate( 20 | reader, 21 | writer, 22 | sinkOptions, 23 | prepareOptions, 24 | checkpointSeeder, 25 | checkpointStore, 26 | replicatorOptions, 27 | stoppingToken 28 | ); 29 | } -------------------------------------------------------------------------------- /src/replicator/ClientApp/src/components/ChannelSize.vue: -------------------------------------------------------------------------------- 1 | 8 | 9 | 27 | 28 | 40 | -------------------------------------------------------------------------------- /src/replicator/ClientApp/index.html: -------------------------------------------------------------------------------- 1 | 2 | 3 | 12 | 13 | 14 | 15 | 16 | 17 | Event Store Replicator 18 | 19 | 20 | 24 |
25 | 26 | 27 | 28 | 29 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.EventStore/Configurator.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.EventStore; 2 | 3 | public class TcpConfigurator(int pageSize) : IConfigurator { 4 | public string Protocol => "tcp"; 5 | 6 | public IEventReader ConfigureReader(string connectionString) 7 | => new TcpEventReader(ConfigureEventStoreTcp(connectionString, true), pageSize); 8 | 9 | public IEventWriter ConfigureWriter(string connectionString) 10 | => new TcpEventWriter(ConfigureEventStoreTcp(connectionString, false)); 11 | 12 | IEventStoreConnection ConfigureEventStoreTcp(string connectionString, bool follower) { 13 | var builder = ConnectionSettings.Create() 14 | .UseCustomLogger(new TcpClientLogger()) 15 | .KeepReconnecting() 16 | .KeepRetrying(); 17 | 18 | if (follower) { 19 | builder = builder.PreferFollowerNode(); 20 | } 21 | 22 | var connection = EventStoreConnection.Create(connectionString, builder); 23 | 24 | return connection; 25 | } 26 | } 27 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Kafka/Configurator.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Js; 2 | 3 | namespace Kurrent.Replicator.Kafka; 4 | 5 | public class KafkaConfigurator(string router) : IConfigurator { 6 | public string Protocol => "kafka"; 7 | 8 | public IEventReader ConfigureReader(string connectionString) { 9 | throw new NotImplementedException("Kafka reader is not supported"); 10 | } 11 | 12 | public IEventWriter ConfigureWriter(string connectionString) 13 | => new KafkaWriter(ParseKafkaConnection(connectionString), FunctionLoader.LoadFile(router, "Router")); 14 | 15 | static ProducerConfig ParseKafkaConnection(string connectionString) { 16 | var settings = connectionString.Split(';'); 17 | var dict = settings.Select(ParsePair).ToDictionary(x => x.Key, x => x.Value); 18 | 19 | return new(dict); 20 | } 21 | 22 | static KeyValuePair ParsePair(string s) { 23 | var split = s.Split('='); 24 | 25 | return new(split[0].Trim(), split[1].Trim()); 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.KurrentDb/ConnectionExtensions.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.KurrentDb; 2 | 3 | static class ConnectionExtensions { 4 | public static async Task GetStreamSize(this EventStoreClient client, string stream) { 5 | var read = client.ReadStreamAsync(Direction.Backwards, stream, StreamPosition.End, 1); 6 | var last = await read.ToArrayAsync().ConfigureAwait(false); 7 | 8 | return new(last[0].OriginalEventNumber.ToInt64()); 9 | } 10 | 11 | public static async Task GetStreamMeta(this EventStoreClient client, string stream) { 12 | var streamMeta = await client.GetStreamMetadataAsync(stream).ConfigureAwait(false); 13 | 14 | var streamDeleted = streamMeta.StreamDeleted || streamMeta.Metadata.TruncateBefore == StreamPosition.End; 15 | 16 | return new( 17 | streamDeleted, 18 | streamMeta.Metadata.MaxAge, 19 | streamMeta.Metadata.MaxCount, 20 | streamMeta.MetastreamRevision!.Value.ToInt64() 21 | ); 22 | } 23 | } -------------------------------------------------------------------------------- /docs/src/content/docs/features/metadata.mdx: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Metadata" 3 | description: > 4 | Additional metadata fields for the target, which include the original event number, position, and creation date. 5 | sidebar: 6 | order: 6 7 | --- 8 | import { Aside } from '@astrojs/starlight/components'; 9 | 10 | In addition to copying the events, Replicator will also copy streams metadata. Therefore, changes in ACLs, truncations, stream deletions, setting max count and max age will all be propagated properly to the target cluster. 11 | 12 | However, the max age won't be working properly for events, which are going to exceed the age in the source cluster. That's because in the target cluster all the events will appear as recently added. 13 | 14 | To mitigate the issue, Replication will add the following metadata to all the copied events: 15 | 16 | - `$originalCreatedDate` 17 | - `$originalEventNumber` 18 | - `$originalEventPosition` 19 | 20 | 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Kurrent Replicator 2 | 3 | Kurrent Replicator allows live copying events from one EventStoreDB and KurrentDB instance or cluster, to another. 4 | 5 | Additional features: 6 | - Filter out (drop) events 7 | - Transform events 8 | - Propagate streams metadata 9 | - Propagate streams deletion 10 | 11 | Implemented readers and writers: 12 | - KurrentDB 13 | - EventStoreDB gRPC (v20+) 14 | - EventStore TCP (v4+) 15 | 16 | ## Build 17 | 18 | ```sh 19 | docker build . 20 | ``` 21 | 22 | The default target architecture is amd64 (x86_64). 23 | 24 | You can build targeting arm64 (e.g. to execute on Apple Silicon) like so: 25 | 26 | ```sh 27 | docker build --build-arg RUNTIME=linux-arm64 . 28 | ``` 29 | 30 | ## Documentation 31 | 32 | Find out the details, including deployment scenarios, in the [documentation](https://replicator.kurrent.io). 33 | 34 | ## Support 35 | 36 | Kurrent Replicator is provided as-is, without any warranty, and is not covered by Kurrent support contract. 37 | 38 | If you experience an issue when using Replicator, or you'd like to suggest a new feature, please open an issue in this GitHub project. 39 | -------------------------------------------------------------------------------- /.gitattributes: -------------------------------------------------------------------------------- 1 | # Explicitly declare text files we want to always be normalized and converted 2 | # to native line endings on checkout. 3 | 4 | *.pdf binary 5 | *.sch binary 6 | *.isch binary 7 | *.ist binary 8 | 9 | # Explicitly declare text files we want to always be normalized and converted 10 | # to native line endings on checkout. 11 | *.c text 12 | *.h text 13 | *.cs text 14 | *.config text 15 | *.xml text 16 | *.manifest text 17 | *.bat text 18 | *.cmd text 19 | *.sh text 20 | *.txt text 21 | *.dat text 22 | *.rc text 23 | *.ps1 text 24 | *.psm1 text 25 | *.js text 26 | *.css text 27 | *.html text 28 | *.sln text 29 | *.DotSettings text 30 | *.csproj text 31 | *.ncrunchproject text 32 | *.fs text 33 | *.fsproj text 34 | *.liquid text 35 | *.boo text 36 | *.pp text 37 | *.targets text 38 | *.markdown text 39 | *.md text 40 | *.bat text 41 | *.xslt text 42 | 43 | # Declare files that will always have CRLF line endings on checkout. 44 | 45 | # Denote all files that are truly binary and should not be modified. 46 | *.ico binary 47 | *.gif binary 48 | *.png binary 49 | *.jpg binary 50 | *.dll binary 51 | *.exe binary 52 | *.pdb binary 53 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.EventStore/ConnectionExtensions.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Observe; 2 | using Ubiquitous.Metrics; 3 | 4 | namespace Kurrent.Replicator.EventStore; 5 | 6 | static class ConnectionExtensions { 7 | public static async Task GetStreamSize(this IEventStoreConnection connection, string stream) { 8 | var last = await connection.ReadStreamEventsBackwardAsync(stream, StreamPosition.End, 1, false).ConfigureAwait(false); 9 | 10 | return new(last.LastEventNumber); 11 | } 12 | 13 | public static async Task GetStreamMeta(this IEventStoreConnection connection, string stream) { 14 | var streamMeta = await Metrics.Measure(() => connection.GetStreamMetadataAsync(stream), ReplicationMetrics.MetaReadsHistogram).ConfigureAwait(false); 15 | 16 | return new( 17 | streamMeta.IsStreamDeleted, 18 | default, 19 | streamMeta.StreamMetadata.MaxAge, 20 | streamMeta.StreamMetadata.MaxCount, 21 | streamMeta.StreamMetadata.TruncateBefore, 22 | streamMeta.MetastreamVersion 23 | ); 24 | } 25 | } 26 | -------------------------------------------------------------------------------- /.github/workflows/pr-build-test.yml: -------------------------------------------------------------------------------- 1 | name: "PR Build and test" 2 | on: 3 | workflow_dispatch: 4 | pull_request: 5 | paths-ignore: 6 | - docs/** 7 | 8 | jobs: 9 | event_file: 10 | name: "Event File" 11 | runs-on: ubuntu-latest 12 | steps: 13 | - name: Upload 14 | uses: actions/upload-artifact@v4 15 | with: 16 | name: Event File 17 | path: ${{ github.event_path }} 18 | build-and-test: 19 | name: "Build and test" 20 | runs-on: ubuntu-latest 21 | steps: 22 | - 23 | name: Checkout 24 | uses: actions/checkout@v4 25 | - 26 | name: Setup .NET 27 | uses: actions/setup-dotnet@v4 28 | with: 29 | dotnet-version: "9.0" 30 | - 31 | name: Run tests 32 | run: | 33 | dotnet test 34 | - 35 | name: Upload Test Results 36 | if: always() 37 | uses: actions/upload-artifact@v4 38 | with: 39 | name: Test Results 40 | path: | 41 | test-results/**/*.xml 42 | test-results/**/*.trx 43 | test-results/**/*.json -------------------------------------------------------------------------------- /src/replicator/HttpApi/CountersKeep.cs: -------------------------------------------------------------------------------- 1 | using Ubiquitous.Metrics.InMemory; 2 | using static replicator.Measurements; 3 | using static Kurrent.Replicator.Shared.Observe.ReplicationMetrics; 4 | 5 | namespace replicator.HttpApi; 6 | 7 | public class CountersKeep { 8 | public InMemoryGauge SinkChannelGauge { get; } = GetGauge(SinkChannelSizeName); 9 | public InMemoryGauge SourcePositionGauge { get; } = GetGauge(LastSourcePositionGaugeName); 10 | public InMemoryGauge ProcessedPositionGauge { get; } = GetGauge(ProcessedPositionGaugeName); 11 | public InMemoryGauge SinkPositionGauge { get; } = GetGauge(SinkPositionGaugeName); 12 | public InMemoryGauge ReadPositionGauge { get; } = GetGauge(ReadingPositionGaugeName); 13 | public InMemoryHistogram PrepareHistogram { get; } = GetHistogram(PrepareHistogramName); 14 | public InMemoryHistogram SinkHistogram { get; } = GetHistogram(WritesHistogramName); 15 | public InMemoryHistogram ReadsHistogram { get; } = GetHistogram(ReadsHistogramName); 16 | public InMemoryGauge PrepareChannelGauge { get; } = GetGauge(PrepareChannelSizeName); 17 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator.KurrentDb/EventFilters.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | 3 | namespace Kurrent.Replicator.KurrentDb; 4 | 5 | class ScavengedEventsFilter(EventStoreClient client, StreamMetaCache cache) { 6 | public async ValueTask Filter(BaseOriginalEvent originalEvent) { 7 | var meta = await cache.GetOrAddStreamMeta(originalEvent.EventDetails.Stream, client.GetStreamMeta).ConfigureAwait(false); 8 | 9 | return meta == null || !meta.IsDeleted && !TtlExpired() && !await OverMaxCount().ConfigureAwait(false); 10 | 11 | bool TtlExpired() => meta.MaxAge.HasValue && originalEvent.Created < DateTime.Now - meta.MaxAge; 12 | 13 | // add the check timestamp, so we can check again if we get newer events (edge case) 14 | async Task OverMaxCount() { 15 | if (!meta.MaxCount.HasValue) 16 | return false; 17 | 18 | var streamSize = await cache.GetOrAddStreamSize(originalEvent.EventDetails.Stream, client.GetStreamSize).ConfigureAwait(false); 19 | 20 | return originalEvent.LogPosition.EventNumber < streamSize.LastEventNumber - meta.MaxCount; 21 | } 22 | } 23 | } 24 | -------------------------------------------------------------------------------- /Dockerfile: -------------------------------------------------------------------------------- 1 | ARG RUNNER_IMG 2 | 3 | FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:9.0 AS builder 4 | ARG TARGETARCH 5 | 6 | WORKDIR /app 7 | 8 | ARG RUNTIME 9 | RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \ 10 | && apt-get install -y --no-install-recommends nodejs \ 11 | && npm install -g yarn 12 | 13 | COPY ./src/Directory.Build.props ./src/*/*.csproj ./src/ 14 | RUN for file in $(ls src/*.csproj); do mkdir -p ./${file%.*}/ && mv $file ./${file%.*}/; done 15 | RUN dotnet restore ./src/replicator -nowarn:msb3202,nu1503 -a $TARGETARCH 16 | 17 | COPY ./src/replicator/ClientApp/package.json ./src/replicator/ClientApp/ 18 | COPY ./src/replicator/ClientApp/yarn.lock ./src/replicator/ClientApp/ 19 | RUN cd ./src/replicator/ClientApp && yarn install 20 | 21 | FROM builder AS publish 22 | ARG TARGETARCH 23 | COPY ./src ./src 24 | RUN dotnet publish ./src/replicator -c Release -a $TARGETARCH -clp:NoSummary --no-self-contained -o /app/publish 25 | 26 | FROM mcr.microsoft.com/dotnet/aspnet:9.0 AS runner 27 | 28 | WORKDIR /app 29 | COPY --from=publish /app/publish . 30 | 31 | ENV ALLOWED_HOSTS "*" 32 | ENV ASPNETCORE_URLS "http://*:5000" 33 | 34 | EXPOSE 5000 35 | ENTRYPOINT ["dotnet", "replicator.dll"] 36 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Observers/LoggingRetryObserver.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Logging; 2 | using GreenPipes; 3 | 4 | namespace Kurrent.Replicator.Observers; 5 | 6 | public class LoggingRetryObserver : IRetryObserver { 7 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 8 | 9 | public Task PostCreate(RetryPolicyContext context) where T : class, PipeContext => Task.CompletedTask; 10 | 11 | public Task PostFault(RetryContext context) where T : class, PipeContext { 12 | Log.Error(context.Exception, "Error: {Error}, will retry", context.Exception.Message); 13 | return Task.CompletedTask; 14 | } 15 | 16 | public Task PreRetry(RetryContext context) where T : class, PipeContext => Task.CompletedTask; 17 | 18 | public Task RetryFault(RetryContext context) where T : class, PipeContext { 19 | Log.Error(context.Exception, "Error: {Error}, will fail", context.Exception.Message); 20 | return Task.CompletedTask; 21 | } 22 | 23 | public Task RetryComplete(RetryContext context) where T : class, PipeContext { 24 | Log.Info("Error recovered: {Error}", context.Exception.Message); 25 | return Task.CompletedTask; 26 | } 27 | } -------------------------------------------------------------------------------- /src/replicator/Settings/EnvConfigProvider.cs: -------------------------------------------------------------------------------- 1 | using System.Collections; 2 | 3 | namespace replicator.Settings; 4 | 5 | public class EnvConfigSource : IConfigurationSource { 6 | public IConfigurationProvider Build(IConfigurationBuilder builder) => new EnvConfigProvider(); 7 | } 8 | 9 | public class EnvConfigProvider : ConfigurationProvider { 10 | public override void Load() { 11 | var envVars = Environment.GetEnvironmentVariables(); 12 | 13 | var vars = envVars.Cast() 14 | .Select(x => new EnvVar(x.Key.ToString()!, x.Value?.ToString())) 15 | .Where(x => x.Key.StartsWith("REPLICATOR_") && x.Value != null); 16 | 17 | Data = vars.ToDictionary(x => x.ConfigKey, x => x.Value, StringComparer.OrdinalIgnoreCase); 18 | } 19 | 20 | record EnvVar(string Key, string? Value) { 21 | public string ConfigKey { 22 | get { 23 | var newKey = Key.Replace("_", ":"); 24 | Console.WriteLine($"{newKey} = {Value}"); 25 | return newKey; 26 | } 27 | } 28 | } 29 | } 30 | 31 | public static class ConfigurationExtensions { 32 | public static IConfigurationBuilder AndEnvConfig(this IConfigurationBuilder builder) 33 | => builder.Add(new EnvConfigSource()); 34 | } -------------------------------------------------------------------------------- /test/Kurrent.Replicator.Tests/Logging/SerilogExtensions.cs: -------------------------------------------------------------------------------- 1 | // Copyright (C) Eventuous HQ OÜ.All rights reserved 2 | // Licensed under the Apache License, Version 2.0. 3 | 4 | using Serilog; 5 | using Serilog.Configuration; 6 | using Serilog.Core; 7 | using Serilog.Events; 8 | using Serilog.Formatting.Display; 9 | 10 | namespace Kurrent.Replicator.Tests.Logging; 11 | 12 | public static class SerilogExtensions { 13 | const string DefaultConsoleOutputTemplate = "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj}{NewLine}{Exception}"; 14 | 15 | public static LoggerConfiguration TestOutput( 16 | this LoggerSinkConfiguration sinkConfiguration, 17 | LogEventLevel restrictedToMinimumLevel = LevelAlias.Minimum, 18 | string outputTemplate = DefaultConsoleOutputTemplate, 19 | IFormatProvider formatProvider = null, 20 | LoggingLevelSwitch levelSwitch = null 21 | ) { 22 | ArgumentNullException.ThrowIfNull(sinkConfiguration); 23 | 24 | var formatter = new MessageTemplateTextFormatter(outputTemplate, formatProvider); 25 | 26 | return sinkConfiguration.Sink(new TestOutputSink(formatter), restrictedToMinimumLevel, levelSwitch); 27 | } 28 | } 29 | -------------------------------------------------------------------------------- /src/replicator/Settings/Transformers.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Http; 2 | using Kurrent.Replicator.Js; 3 | using Kurrent.Replicator.Shared; 4 | using Kurrent.Replicator.Shared.Pipeline; 5 | 6 | namespace replicator.Settings; 7 | 8 | public static class Transformers { 9 | public static TransformEvent GetTransformer(Replicator settings) { 10 | var type = settings.Transform?.Type; 11 | 12 | return type switch { 13 | "default" => Transforms.DefaultWithExtraMeta, 14 | "http" => GetHttpTransform().Transform, 15 | "js" => GetJsTransform().Transform, 16 | _ => Transforms.DefaultWithExtraMeta, 17 | }; 18 | 19 | HttpTransform GetHttpTransform() { 20 | Ensure.NotEmpty(settings.Transform?.Config, "Transform config"); 21 | 22 | return new(settings.Transform!.Config); 23 | } 24 | 25 | JsTransform GetJsTransform() { 26 | Ensure.NotEmpty(settings.Transform?.Config, "Transform config"); 27 | 28 | var fileName = settings.Transform!.Config; 29 | 30 | if (!File.Exists(fileName)) 31 | throw new ArgumentException($"JavaScript file {fileName} not found"); 32 | 33 | var js = File.ReadAllText(fileName); 34 | 35 | return new(js); 36 | } 37 | } 38 | } 39 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator/ChaserCheckpointSeeder.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared; 2 | using Kurrent.Replicator.Shared.Logging; 3 | 4 | namespace Kurrent.Replicator; 5 | 6 | public class ChaserCheckpointSeeder(string filePath, ICheckpointStore checkpointStore) : ICheckpointSeeder { 7 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 8 | 9 | public async ValueTask Seed(CancellationToken cancellationToken) { 10 | if (await checkpointStore.HasStoredCheckpoint(cancellationToken)) { 11 | Log.Info("Checkpoint already present in store, skipping seeding"); 12 | 13 | return; 14 | } 15 | 16 | if (!File.Exists(filePath)) { 17 | Log.Warn("Seeding failed because the file at {FilePath} does not exist", filePath); 18 | 19 | return; 20 | } 21 | 22 | await using var fileStream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite); 23 | 24 | if (fileStream.Length != 8) { 25 | Log.Warn("Seeding failed because the file at {FilePath} does not appear to be an 8-byte position file", filePath); 26 | 27 | return; 28 | } 29 | 30 | using var reader = new BinaryReader(fileStream); 31 | var position = reader.ReadInt64(); 32 | await checkpointStore.StoreCheckpoint(new LogPosition(0L, (ulong)position), cancellationToken); 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Partitioning/JsKeyProvider.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | using Jint; 3 | using Jint.Native; 4 | using Jint.Native.Json; 5 | using Kurrent.Replicator.Js; 6 | 7 | namespace Kurrent.Replicator.Partitioning; 8 | 9 | public class JsKeyProvider(string partitionFunction) { 10 | readonly TypedJsFunction _function = new(partitionFunction, "partition", AsPartition); 11 | 12 | static string? AsPartition(JsValue? result, PartitionEvent evt) 13 | => result == null || result.IsUndefined() || !result.IsString() ? null : result.ToString(); 14 | 15 | public string GetPartitionKey(BaseProposedEvent original) { 16 | if (original is not ProposedEvent evt) { 17 | return Default(); 18 | } 19 | 20 | var parser = new JsonParser(_function.Engine); 21 | 22 | var result = _function.Execute( 23 | new PartitionEvent( 24 | original.EventDetails.Stream, 25 | original.EventDetails.EventType, 26 | parser.Parse(evt.Data.AsUtf8String()), 27 | evt.Metadata != null ? parser.Parse(evt.Metadata.AsUtf8String()) : null 28 | ) 29 | ); 30 | 31 | return result ?? Default(); 32 | 33 | string Default() => KeyProvider.ByStreamName(original); 34 | } 35 | } 36 | 37 | record PartitionEvent(string Stream, string EventType, object Data, object? Meta); 38 | -------------------------------------------------------------------------------- /.github/workflows/docker-publish.yml: -------------------------------------------------------------------------------- 1 | name: Replicator / Publish Docker Image 2 | on: 3 | push: 4 | paths: 5 | - 'src/**' 6 | branches: 7 | - 'master' 8 | tags: 9 | - '*.*.*' 10 | 11 | jobs: 12 | docker: 13 | runs-on: ubuntu-latest 14 | steps: 15 | - 16 | name: Checkout 17 | uses: actions/checkout@v4 18 | - 19 | name: Set up QEMU 20 | uses: docker/setup-qemu-action@v3 21 | - 22 | name: Set up Docker metadata 23 | id: meta 24 | uses: docker/metadata-action@v5 25 | with: 26 | images: | 27 | eventstore/replicator 28 | tags: | 29 | type=semver,pattern={{major}} 30 | type=semver,pattern={{major}}.{{minor}} 31 | type=semver,pattern={{version}} 32 | - 33 | name: Set up Docker Buildx 34 | uses: docker/setup-buildx-action@v3 35 | - 36 | name: Login to DockerHub 37 | uses: docker/login-action@v3 38 | with: 39 | username: ${{ secrets.DOCKERHUB_USERNAME }} 40 | password: ${{ secrets.DOCKERHUB_TOKEN }} 41 | - 42 | name: Publish 43 | uses: docker/build-push-action@v6 44 | with: 45 | context: . 46 | push: true 47 | tags: ${{ steps.meta.outputs.tags }} 48 | labels: ${{ steps.meta.outputs.labels }} 49 | platforms: linux/amd64,linux/arm64 50 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Contracts/ProposedEvent.cs: -------------------------------------------------------------------------------- 1 | // ReSharper disable SuggestBaseTypeForParameter 2 | 3 | namespace Kurrent.Replicator.Shared.Contracts; 4 | 5 | public abstract record BaseProposedEvent(EventDetails EventDetails, LogPosition SourceLogPosition, long SequenceNumber); 6 | 7 | public record ProposedEvent( 8 | EventDetails EventDetails, 9 | byte[] Data, 10 | byte[]? Metadata, 11 | LogPosition SourceLogPosition, 12 | long SequenceNumber 13 | ) : BaseProposedEvent(EventDetails, SourceLogPosition, SequenceNumber); 14 | 15 | public record ProposedMetaEvent( 16 | EventDetails EventDetails, 17 | StreamMetadata Data, 18 | LogPosition SourceLogPosition, 19 | long SequenceNumber 20 | ) : BaseProposedEvent(EventDetails, SourceLogPosition, SequenceNumber); 21 | 22 | public record ProposedDeleteStream(EventDetails EventDetails, LogPosition SourceLogPosition, long SequenceNumber) 23 | : BaseProposedEvent(EventDetails, SourceLogPosition, SequenceNumber); 24 | 25 | public record IgnoredEvent(EventDetails EventDetails, LogPosition SourceLogPosition, long SequenceNumber) 26 | : BaseProposedEvent(EventDetails, SourceLogPosition, SequenceNumber); 27 | 28 | public record NoEvent(EventDetails EventDetails, LogPosition SourceLogPosition, long SequenceNumber) 29 | : BaseProposedEvent(EventDetails, SourceLogPosition, SequenceNumber); 30 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.EventStore/EventFilters.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | 3 | namespace Kurrent.Replicator.EventStore; 4 | 5 | class ScavengedEventsFilter(IEventStoreConnection connection, StreamMetaCache cache) { 6 | public async ValueTask Filter(BaseOriginalEvent originalEvent) { 7 | if (originalEvent is not OriginalEvent) return true; 8 | 9 | var meta = (await cache.GetOrAddStreamMeta(originalEvent.EventDetails.Stream, connection.GetStreamMeta).ConfigureAwait(false))!; 10 | var isDeleted = meta.IsDeleted && meta.DeletedAt > originalEvent.Created; 11 | 12 | return !isDeleted && !Truncated() && !TtlExpired() && !await OverMaxCount().ConfigureAwait(false); 13 | 14 | bool TtlExpired() => meta.MaxAge.HasValue && originalEvent.Created < DateTime.Now - meta.MaxAge; 15 | 16 | bool Truncated() => meta.TruncateBefore.HasValue && originalEvent.LogPosition.EventNumber < meta.TruncateBefore; 17 | 18 | // add the check timestamp, so we can check again if we get newer events (edge case) 19 | async Task OverMaxCount() { 20 | if (!meta.MaxCount.HasValue) return false; 21 | 22 | var streamSize = await cache.GetOrAddStreamSize(originalEvent.EventDetails.Stream, connection.GetStreamSize).ConfigureAwait(false); 23 | 24 | return originalEvent.LogPosition.EventNumber < streamSize.LastEventNumber - meta.MaxCount; 25 | } 26 | } 27 | } 28 | -------------------------------------------------------------------------------- /docs/src/styles/buttons.css: -------------------------------------------------------------------------------- 1 | .sl-link-button { 2 | @apply flex items-center justify-center rounded-xl px-5 py-2.5 font-medium tracking-tight transition-(--button-transition-properties) duration-300; 3 | @apply focus:outline-none; 4 | @apply disabled:pointer-events-none; 5 | } 6 | 7 | .sl-link-button.primary { 8 | @apply border-primary-600 from-primary-700 to-primary-700 relative border bg-gradient-to-t text-white; 9 | @apply before:bg-primary-500 before:absolute before:-inset-0.5 before:-z-10 before:rounded-full before:opacity-0 before:blur-sm before:transition before:duration-300; 10 | @apply dark:hover:border-primary-300 hover:before:opacity-100; 11 | @apply focus-visible:ring-primary-300 focus-visible:ring-4; 12 | } 13 | 14 | .sl-link-button.secondary { 15 | @apply border-base-300 bg-base-50 text-base-800 border; 16 | @apply hover:bg-base-100; 17 | @apply focus-visible:ring-primary-500 focus-visible:ring-4; 18 | } 19 | 20 | .button--outline { 21 | @apply border-base-400 text-base-900 dark:border-base-600 dark:text-base-100 border bg-transparent; 22 | @apply hover:bg-base-800 dark:hover:border-base-100 dark:hover:bg-base-100 dark:hover:text-base-900 hover:text-white; 23 | @apply focus-visible:ring-primary-500 focus-visible:ring-4; 24 | } 25 | 26 | .sl-link-button.minimal { 27 | @apply text-foreground/80 bg-transparent; 28 | @apply hover:bg-base-200 hover:text-foreground dark:hover:bg-base-900; 29 | @apply focus-visible:ring-primary-500 focus-visible:ring-4; 30 | } 31 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Kafka/KafkaJsMessageRouter.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Js; 2 | using Kurrent.Replicator.Shared.Contracts; 3 | using Jint; 4 | using Jint.Native; 5 | using Jint.Native.Json; 6 | 7 | // ReSharper disable NotAccessedPositionalProperty.Global 8 | 9 | namespace Kurrent.Replicator.Kafka; 10 | 11 | public class KafkaJsMessageRouter(string routingFunction) { 12 | readonly TypedJsFunction _function = new(routingFunction, "route", AsRoute); 13 | 14 | static MessageRoute AsRoute(JsValue? result, RouteEvent evt) { 15 | if (result == null || result.IsUndefined() || !result.IsObject()) 16 | return DefaultRouters.RouteByCategory(evt.Stream); 17 | 18 | var obj = result.AsObject(); 19 | 20 | return new(obj.Get("topic").AsString(), obj.Get("partitionKey").AsString()); 21 | } 22 | 23 | public MessageRoute Route(ProposedEvent evt) { 24 | var parser = new JsonParser(_function.Engine); 25 | 26 | return _function.Execute( 27 | new( 28 | evt.EventDetails.Stream, 29 | evt.EventDetails.EventType, 30 | parser.Parse(evt.Data.AsUtf8String()), 31 | evt.Metadata == null ? null : parser.Parse(evt.Metadata.AsUtf8String()) 32 | ) 33 | ); 34 | } 35 | } 36 | 37 | record RouteEvent(string Stream, string EventType, object Data, object? Meta); 38 | 39 | public record MessageRoute(string Topic, string PartitionKey); 40 | -------------------------------------------------------------------------------- /docs/src/content/docs/index.mdx: -------------------------------------------------------------------------------- 1 | --- 2 | title: Kurrent Replicator 3 | description: Replicate data between KurrentDB clusters with ease. 4 | template: splash 5 | hero: 6 | tagline: | 7 | Migrate KurrentDB on-prem to Kurrent Cloud by replicating events. 8 | Keep your data safe by replicating events to a backup cluster. 9 | One-time or continuous replication, filtering events, schema and data transformations. 10 | image: 11 | file: ../../assets/replicator1.png 12 | actions: 13 | - text: Read the docs 14 | link: /intro/overview/ 15 | icon: right-arrow 16 | - text: Kurrent Home 17 | link: https://kurrentdb.com 18 | icon: external 19 | variant: minimal 20 | --- 21 | 22 | import { Card, CardGrid } from '@astrojs/starlight/components'; 23 | 24 | 25 | 26 | Read and write to KurrentDB and EventStoreDB v5 and v20+, using legacy TCP or the latest client protocol. 27 | Both secure and unsecure connections are supported. 28 | 29 | 30 | Filter out obsolete and deleted events with scavenge, stream name, and event type filter. You can even use transformations for advanced filtering! 31 | 32 | 33 | Upgrade your event schema, remove or add fields, change values, enrich events with information from external sources. 34 | 35 | 36 | Deploy Replicator as a Docker container, or use Helm chart to deploy it on Kubernetes. 37 | 38 | 39 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Prepare/PreparePipe.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | using Kurrent.Replicator.Shared.Pipeline; 3 | using GreenPipes; 4 | using Kurrent.Replicator.Observers; 5 | using Kurrent.Replicator.Sink; 6 | 7 | namespace Kurrent.Replicator.Prepare; 8 | 9 | public class PreparePipe { 10 | readonly IPipe _pipe; 11 | 12 | public PreparePipe(FilterEvent? filter, TransformEvent? transform, Func send) 13 | => _pipe = Pipe.New(cfg => { 14 | cfg.UseRetry(r => { 15 | r.Incremental(10, TimeSpan.Zero, TimeSpan.FromMilliseconds(10)); 16 | r.ConnectRetryObserver(new LoggingRetryObserver()); 17 | } 18 | ); 19 | cfg.UseLog(); 20 | cfg.UseConcurrencyLimit(10); 21 | 22 | cfg.UseEventFilter(Filters.EmptyDataFilter); 23 | cfg.UseEventFilter(filter ?? Filters.EmptyFilter); 24 | 25 | cfg.UseEventTransform(transform ?? Transforms.DefaultWithExtraMeta); 26 | 27 | cfg.UseExecuteAsync(async ctx => { 28 | var proposedEvent = ctx.GetPayload(); 29 | 30 | try { 31 | await send(new(proposedEvent, CancellationToken.None)).ConfigureAwait(false); 32 | } catch (OperationCanceledException) { } 33 | } 34 | ); 35 | } 36 | ); 37 | 38 | public Task Send(PrepareContext context) => _pipe.Send(context); 39 | } 40 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.KurrentDb/Realtime.cs: -------------------------------------------------------------------------------- 1 | using System.Text.Json; 2 | using Kurrent.Replicator.KurrentDb.Internals; 3 | 4 | namespace Kurrent.Replicator.KurrentDb; 5 | 6 | class Realtime(EventStoreClient client, StreamMetaCache metaCache) { 7 | bool _started; 8 | 9 | public Task Start() { 10 | if (_started) return Task.CompletedTask; 11 | 12 | _started = true; 13 | 14 | return client.SubscribeToAllAsync(FromAll.End, (_, evt, _) => HandleEvent(evt), subscriptionDropped: HandleDrop); 15 | } 16 | 17 | void HandleDrop(StreamSubscription subscription, SubscriptionDroppedReason reason, Exception? exception) { 18 | if (reason == SubscriptionDroppedReason.Disposed) return; 19 | 20 | _started = false; 21 | Task.Run(Start); 22 | } 23 | 24 | Task HandleEvent(ResolvedEvent re) { 25 | if (IsSystemEvent()) 26 | return Task.CompletedTask; 27 | 28 | if (IsMetadataUpdate()) { 29 | var stream = re.OriginalStreamId[2..]; 30 | var meta = JsonSerializer.Deserialize(re.Event.Data.Span, MetaSerialization.StreamMetadataJsonSerializerOptions); 31 | metaCache.UpdateStreamMeta(stream, meta, re.OriginalEventNumber.ToInt64()); 32 | } 33 | else { 34 | metaCache.UpdateStreamLastEventNumber(re.OriginalStreamId, re.OriginalEventNumber.ToInt64()); 35 | } 36 | 37 | return Task.CompletedTask; 38 | 39 | bool IsSystemEvent() => re.Event.EventType.StartsWith('$') && re.Event.EventType != Predefined.MetadataEventType; 40 | 41 | bool IsMetadataUpdate() => re.Event.EventType == Predefined.MetadataEventType; 42 | } 43 | } 44 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Prepare/EventFilterFilter.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Observe; 2 | using Kurrent.Replicator.Shared.Pipeline; 3 | using GreenPipes; 4 | using Ubiquitous.Metrics; 5 | 6 | namespace Kurrent.Replicator.Prepare; 7 | 8 | public class EventFilterFilter(FilterEvent filter) : IFilter { 9 | public async Task Send(PrepareContext context, IPipe next) { 10 | // ReSharper disable once ConditionIsAlwaysTrueOrFalse 11 | // ReSharper disable once ConditionIsAlwaysTrueOrFalseAccordingToNullableAPIContract 12 | if (context.OriginalEvent == null) return; 13 | 14 | var accept = await Metrics 15 | .MeasureValueTask(() => filter(context.OriginalEvent), ReplicationMetrics.PrepareHistogram) 16 | .ConfigureAwait(false); 17 | 18 | if (!accept) context.IgnoreEvent(); 19 | await next.Send(context).ConfigureAwait(false); 20 | } 21 | 22 | public void Probe(ProbeContext context) { } 23 | } 24 | 25 | public class EventFilterSpecification(FilterEvent? filter) : IPipeSpecification { 26 | public void Apply(IPipeBuilder builder) => builder.AddFilter(new EventFilterFilter(filter!)); 27 | 28 | public IEnumerable Validate() { 29 | if (filter == null) { 30 | yield return this.Failure("validationFilterPipe", "Event filter is missing"); 31 | } 32 | } 33 | } 34 | 35 | public static class EventFilterPipeExtensions { 36 | public static void UseEventFilter(this IPipeConfigurator configurator, FilterEvent filter) 37 | => configurator.AddPipeSpecification(new EventFilterSpecification(filter)); 38 | } 39 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Logging.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Logging; 2 | using GreenPipes; 3 | 4 | namespace Kurrent.Replicator; 5 | 6 | public class LoggingFilter : IFilter where T : class, PipeContext { 7 | // ReSharper disable once StaticMemberInGenericType 8 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 9 | 10 | public async Task Send(T context, IPipe next) { 11 | try { 12 | await next.Send(context).ConfigureAwait(false); 13 | } 14 | catch (Exception e) { 15 | if (context is IEventDetailsContext eventDetailsContext) { 16 | Log.Error( 17 | e, 18 | "Error occured in the {Type} pipe {@Event}: {Message}", 19 | typeof(T).Name, 20 | eventDetailsContext.EventDetails, 21 | e.Message 22 | ); 23 | } 24 | else { 25 | Log.Error(e, "Error occured in the {Type} pipe: {Message}", typeof(T).Name, e.Message); 26 | } 27 | 28 | throw; 29 | } 30 | } 31 | 32 | public void Probe(ProbeContext context) { } 33 | } 34 | 35 | public class LoggingFilterSpecification : IPipeSpecification where T : class, PipeContext { 36 | public void Apply(IPipeBuilder builder) => builder.AddFilter(new LoggingFilter()); 37 | 38 | public IEnumerable Validate() { 39 | yield return this.Success("Logging is good"); 40 | } 41 | } 42 | 43 | public static class LoggingFilterExtensions { 44 | public static void UseLog(this IPipeConfigurator cfg) where T : class, PipeContext 45 | => cfg.AddPipeSpecification(new LoggingFilterSpecification()); 46 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Pipeline/Filters.cs: -------------------------------------------------------------------------------- 1 | using System.Text.RegularExpressions; 2 | using Kurrent.Replicator.Shared.Extensions; 3 | using Kurrent.Replicator.Shared.Contracts; 4 | 5 | namespace Kurrent.Replicator.Shared.Pipeline; 6 | 7 | public delegate ValueTask FilterEvent(BaseOriginalEvent originalEvent); 8 | 9 | public static class Filters { 10 | public static async ValueTask CombinedFilter(BaseOriginalEvent originalEvent, params FilterEvent[] filters) { 11 | foreach (var filter in filters) { 12 | if (!await filter(originalEvent)) return false; 13 | } 14 | 15 | return true; 16 | } 17 | 18 | public static ValueTask EmptyFilter(BaseOriginalEvent originalEvent) => new(true); 19 | 20 | public abstract class RegExFilter(string? include, string? exclude, Func getProp) { 21 | readonly Regex? _include = include != null ? new Regex(include) : null; 22 | readonly Regex? _exclude = exclude != null ? new Regex(exclude) : null; 23 | 24 | public ValueTask Filter(BaseOriginalEvent originalEvent) { 25 | var propValue = getProp(originalEvent); 26 | var pass = _include.IsNullOrMatch(propValue) && _exclude.IsNullOrDoesntMatch(propValue); 27 | 28 | return new(pass); 29 | } 30 | } 31 | 32 | public class EventTypeFilter(string? include, string? exclude) : RegExFilter(include, exclude, x => x.EventDetails.EventType); 33 | 34 | public class StreamNameFilter(string? include, string? exclude) : RegExFilter(include, exclude, x => x.EventDetails.Stream); 35 | 36 | public static ValueTask EmptyDataFilter(BaseOriginalEvent originalEvent) 37 | => new(originalEvent is OriginalEvent { Data.Length: > 0 }); 38 | } 39 | -------------------------------------------------------------------------------- /docs/src/content/docs/features/filters.mdx: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Filters" 3 | sidebar: 4 | order: 2 5 | description: > 6 | Filter out events by stream name or event type, using regular expressions. Positive and negative matches are supported. 7 | --- 8 | 9 | import { Aside } from '@astrojs/starlight/components'; 10 | 11 | You may want to prevent some events from being replicated. Those could be obsolete or "wrong" events, which your system doesn't need. 12 | 13 | Replicator provides two filters in addition to the special [scavenge filter](../scavenge/): 14 | - Event type filter 15 | - Stream name filter 16 | 17 | Filter options: 18 | 19 | | Option | Values | Description | 20 | |:----------|:----------------------------|:---------------------------------------------------| 21 | | `type` | `eventType` or `streamName` | One of the available filters | 22 | | `include` | Regular expression | Filter for allowing events to be replicated | 23 | | `exclude` | Regular expression | Filter for preventing events from being replicated | 24 | 25 | For example: 26 | 27 | ```yaml 28 | replicator: 29 | filters: 30 | - type: eventType 31 | include: "." 32 | exclude: "((Bad|Wrong)\w+Event)" 33 | ``` 34 | 35 | You can configure zero or more filters. Scavenge filter is enabled by the `scavenge` setting and doesn't need to be present in the `filter` list. You can specify either `include` or `exclude` regular expression, or both. When both `include` and `exclude` regular expressions are configured, the filter will check both, so the event must match the inclusion expression and not match the exclusion expression. 36 | 37 | 40 | 41 | -------------------------------------------------------------------------------- /docs/src/content/docs/features/readers.mdx: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Sources" 3 | description: > 4 | Sources where Replicator can read events from. 5 | sidebar: 6 | order: 0 7 | --- 8 | import { Badge } from '@astrojs/starlight/components'; 9 | 10 | A source is a where Replicate reads events that are replicated to a [sink](../sinks/). 11 | Replicator supports the following sources: 12 | 13 | ## KurrentDB 14 | 15 | When replicating events from EventStoreDB v20+ or KurrentDB, use the KurrentDB source. 16 | 17 | You need to specify two configurations options for it: 18 | 19 | - `replicator.reader.protocol` - set to `grpc` 20 | - `replicator.reader.connectionString` - use the source cluster [connection string](https://docs.kurrent.io/clients/grpc/getting-started.html#connection-string), which you'd use for the regular client. 21 | 22 | For example, for a Kurrent Cloud cluster the connection string would look like: 23 | 24 | `esdb+discover://:@.mesdb.eventstore.cloud`. 25 | 26 | Using gRPC gives you more predictable write operation time. For example, on a C4-size instance in Google Cloud Platform, one write would take 4-5 ms, and this number allows you to calculate the replication process throughput, as it doesn't change much when the database size grows. 27 | 28 | ## EventStoreDB TCP 29 | 30 | The TCP source should only be used when migrating from an older version cluster, for example Event Store v5 or earlier. 31 | 32 | For the TCP sink, you need to specify two configurations options for it: 33 | 34 | - `replicator.reader.protocol` - set to `tcp` 35 | - `replicator.reader.connectionString` - use the target cluster connection string, which you'd use for the TCP client. 36 | 37 | Check the connection string format and options in the [TCP client documentation](https://docs.kurrent.io/clients/tcp/dotnet/21.2/connecting.html#connection-string). 38 | -------------------------------------------------------------------------------- /src/replicator/Program.cs: -------------------------------------------------------------------------------- 1 | using System.Diagnostics; 2 | using System.Reflection; 3 | using Serilog; 4 | using Serilog.Events; 5 | using Serilog.Formatting.Compact; 6 | using Kurrent.Replicator; 7 | using replicator; 8 | using replicator.Settings; 9 | 10 | var builder = WebApplication.CreateBuilder(args); 11 | 12 | var isDebug = Environment.GetEnvironmentVariable("REPLICATOR_DEBUG") != null; 13 | var logConfig = new LoggerConfiguration(); 14 | logConfig = isDebug ? logConfig.MinimumLevel.Debug() : logConfig.MinimumLevel.Information(); 15 | 16 | logConfig = logConfig 17 | .MinimumLevel.Override("Microsoft", LogEventLevel.Information) 18 | .MinimumLevel.Override("Microsoft.AspNetCore", LogEventLevel.Warning) 19 | .MinimumLevel.Override("Grpc", LogEventLevel.Error) 20 | .Enrich.FromLogContext(); 21 | 22 | logConfig = builder.Environment.IsDevelopment() 23 | ? logConfig.WriteTo.Console(outputTemplate: "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj} ;{NewLine}{Exception}") 24 | : logConfig.WriteTo.Console(new RenderedCompactJsonFormatter()); 25 | Log.Logger = logConfig.CreateLogger(); 26 | var fileInfo = FileVersionInfo.GetVersionInfo(Assembly.GetExecutingAssembly().Location); 27 | Log.Information("Starting replicator {Version}", fileInfo.ProductVersion); 28 | 29 | builder.Host.UseSerilog(); 30 | builder.Configuration.AddYamlFile("./config/appsettings.yaml", false, true).AndEnvConfig(); 31 | Startup.ConfigureServices(builder); 32 | 33 | var app = builder.Build(); 34 | Startup.Configure(app); 35 | 36 | var restartOnFailure = app.Services.GetService()?.RestartOnFailure == true; 37 | 38 | try { 39 | app.Run(); 40 | 41 | return 0; 42 | } catch (Exception ex) { 43 | Log.Fatal(ex, "Host terminated unexpectedly"); 44 | 45 | if (restartOnFailure) return -1; 46 | 47 | while (true) { 48 | await Task.Delay(5000); 49 | } 50 | } finally { 51 | Log.CloseAndFlush(); 52 | } 53 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Contracts/OriginalEvent.cs: -------------------------------------------------------------------------------- 1 | // ReSharper disable SuggestBaseTypeForParameter 2 | 3 | namespace Kurrent.Replicator.Shared.Contracts; 4 | 5 | public abstract record BaseOriginalEvent( 6 | DateTimeOffset Created, 7 | EventDetails EventDetails, 8 | LogPosition LogPosition, 9 | long SequenceNumber, 10 | TracingMetadata TracingMetadata 11 | ); 12 | 13 | public record OriginalEvent( 14 | DateTimeOffset Created, 15 | EventDetails EventDetails, 16 | byte[] Data, 17 | byte[]? Metadata, 18 | LogPosition LogPosition, 19 | long SequenceNumber, 20 | TracingMetadata TracingMetadata 21 | ) : BaseOriginalEvent(Created, EventDetails, LogPosition, SequenceNumber, TracingMetadata); 22 | 23 | public record StreamMetadataOriginalEvent( 24 | DateTimeOffset Created, 25 | EventDetails EventDetails, 26 | StreamMetadata Data, 27 | LogPosition LogPosition, 28 | long SequenceNumber, 29 | TracingMetadata TracingMetadata 30 | ) : BaseOriginalEvent(Created, EventDetails, LogPosition, SequenceNumber, TracingMetadata); 31 | 32 | public record StreamDeletedOriginalEvent( 33 | DateTimeOffset Created, 34 | EventDetails EventDetails, 35 | LogPosition LogPosition, 36 | long SequenceNumber, 37 | TracingMetadata TracingMetadata 38 | ) : BaseOriginalEvent(Created, EventDetails, LogPosition, SequenceNumber, TracingMetadata); 39 | 40 | public record IgnoredOriginalEvent( 41 | DateTimeOffset Created, 42 | EventDetails EventDetails, 43 | LogPosition LogPosition, 44 | long SequenceNumber, 45 | TracingMetadata TracingMetadata 46 | ) 47 | : BaseOriginalEvent(Created, EventDetails, LogPosition, SequenceNumber, TracingMetadata); 48 | -------------------------------------------------------------------------------- /docs/src/content/docs/deployment/docker.mdx: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Docker" 3 | description: Running Replicator with Docker Compose 4 | --- 5 | import { Aside } from '@astrojs/starlight/components'; 6 | 7 | You can run Replicator using [Docker Compose](https://docs.docker.com/compose/), on any machine, which has Docker installed. 8 | 9 | We prepared a complete set of files for this scenario. You find those files in the [Replicator repository](https://github.com/kurrent-io/replicator/tree/master/compose). 10 | 11 | The Compose file includes the following components: 12 | - Replicator itself 13 | - Prometheus, pre-configured to scrape Replicator metrics endpoint 14 | - Grafana, pre-configured to use Prometheus, with the Replicator dashboard included 15 | 16 | ## Configuration 17 | 18 | Before spinning up this setup, you need to change the `replicator.yml` file. Find out about Replicator settings on the [Configuration](../configuration/) page. Check the [sample](https://github.com/kurrent-io/replicator/blob/02736f6e3dd18e41d5536f26ca4f9497733d5f3f/compose/replicator.yml) configuration file to the repository. 19 | 20 | 23 | 24 | The sample configuration enables verbose logging using the `REPLICATOR_DEBUG` environment variable. For production deployments, you should remove it from the configuration. 25 | 26 | ## Monitoring 27 | 28 | When you start all the component using `docker-compose up`, you'd be able to check the Replicator web UI by visiting [http://localhost:5000](http://localhost:5000), as well as Grafana at [http://localhost:3000](http://localhost:3000). Use `admin`/`admin` default credentials for Grafana. The Replicator dashboard is included in the deployment, so you can find it in the [dashboards list](http://localhost:3000/dashboards). 29 | 30 | 33 | 34 | 35 | -------------------------------------------------------------------------------- /src/replicator/HttpApi/Counters.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Observe; 2 | using Microsoft.AspNetCore.Mvc; 3 | using Ubiquitous.Metrics.InMemory; 4 | using static Kurrent.Replicator.Shared.Observe.ReplicationMetrics; 5 | 6 | namespace replicator.HttpApi; 7 | 8 | [Route("api/counters")] 9 | public class Counters(CountersKeep keep) : ControllerBase { 10 | CountersKeep Keep { get; } = keep; 11 | 12 | [HttpGet] 13 | public MetricsResponse GetCounters() { 14 | var prepareCount = (int) Keep.PrepareChannelGauge.Value; 15 | return new( 16 | Keep.ReadsHistogram.Count, 17 | Keep.SinkHistogram.Count, 18 | (long) Keep.ReadPositionGauge.Value, 19 | (long) Keep.ProcessedPositionGauge.Value, 20 | prepareCount, 21 | ReadRate(), 22 | GetRate(Keep.SinkHistogram), 23 | (long) Keep.SourcePositionGauge.Value, 24 | (long) Keep.SinkPositionGauge.Value, 25 | ReplicationStatus.ReaderRunning, 26 | new(PrepareChannelCapacity, prepareCount), 27 | new(SinkChannelCapacity, (int) Keep.SinkChannelGauge.Value), 28 | GetRate(Keep.PrepareHistogram) 29 | ); 30 | 31 | static Rate GetRate(InMemoryHistogram metric) => new(metric.Sum, metric.Count); 32 | 33 | Rate ReadRate() => ReplicationStatus.ReaderRunning 34 | ? new(ReplicationStatus.ElapsedSeconds, Keep.ReadsHistogram.Count) 35 | : new(0, 0); 36 | } 37 | 38 | public record MetricsResponse( 39 | long ReadEvents, 40 | long ProcessedEvents, 41 | long ReadPosition, 42 | long WritePosition, 43 | long InFlightSink, 44 | Rate ReadRate, 45 | Rate WriteRate, 46 | long LastSourcePosition, 47 | long SinkPosition, 48 | bool ReaderRunning, 49 | Channel PrepareChannel, 50 | Channel SinkChannel, 51 | Rate PrepareRate 52 | ); 53 | 54 | public record Rate(double Sum, long Count); 55 | 56 | public record Channel(int Capacity, int Size); 57 | } -------------------------------------------------------------------------------- /docs/src/content/docs/intro/overview.md: -------------------------------------------------------------------------------- 1 | --- 2 | title: Overview 3 | description: What is Replicator and why you might want to use it. 4 | sidebar: 5 | order: 1 6 | --- 7 | 8 | ## What is it? 9 | 10 | Event Store Replicator, as the name suggests, aims to address the need to keep one KurrentDB cluster in sync with another. It does so by establishing connections to both source and target clusters, then reading all the events from the source, and propagating them to the target. During the replication process, the tool can apply some rules, which allow you to exclude some events from being propagated to the target. 11 | 12 | ## Why would you use it? 13 | 14 | Common replication scenarios include: 15 | 16 | * **Cloud migration**: When migrating from a self-hosted cluster to Kurrent Cloud, you might want to take the data with you. Replicator can help you with that, but it has some limitations on how fast it can copy the historical data. 17 | 18 | * **Store migration**: Migrating the whole store when your event schema changes severely, or you need to get rid of some obsolete data, you can do it using Replicator too. You can transform events from one contract to another, and filter out obsolete events. It allows you also to overcome the limitation of not being able to delete events in the middle of the stream. Greg Young promotes a complete store migration with transformation as part of the release cycle, to avoid event versioning issues. You can, for example, [listen about it here](https://youtu.be/FKFu78ZEIi8?t=856). 19 | 20 | * **Backup**: You can also replicate data between two clusters, so in case of catastrophic failure, you will have a working cluster with recent data. 21 | 22 | * **What is it *not yet* good for?**: Replicator uses client protocols (TCP and gRPC), with all the limitations. For example, to keep the global event order intact, you must use a single writer. As the transaction scope is limited to one stream, you get sequential writes of one event at a time, which doesn't deliver exceptional speed. Relaxing ordering guarantees helps to increase the performance, but for large databases (hundreds of millions events and more) and guaranteed global order, it might not be the tool for you. 23 | 24 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3.7' 2 | 3 | services: 4 | 5 | eventstore: 6 | container_name: repl-eventstore 7 | image: eventstore/eventstore:release-5.0.8 8 | ports: 9 | - '2113:2113' 10 | - '1113:1113' 11 | environment: 12 | EVENTSTORE_CLUSTER_SIZE: 1 13 | EVENTSTORE_EXT_TCP_PORT: 1113 14 | EVENTSTORE_EXT_HTTP_PORT: 2113 15 | EVENTSTORE_RUN_PROJECTIONS: all 16 | EVENTSTORE_START_STANDARD_PROJECTIONS: "true" 17 | 18 | esdb: 19 | container_name: repl-kurrentdb 20 | image: kurrentplatform/kurrentdb:25.0 21 | ports: 22 | - '2114:2114' 23 | environment: 24 | KURRENTDB_INSECURE: 'true' 25 | KURRENTDB_CLUSTER_SIZE: 1 26 | KURRENTDB_HTTP_PORT: 2114 27 | KURRENTDB_RUN_PROJECTIONS: all 28 | KURRENTDB_ENABLE_ATOM_PUB_OVER_HTTP: "true" 29 | 30 | mongo: 31 | container_name: repl-mongo 32 | image: mongo 33 | ports: 34 | - '27017:27017' 35 | environment: 36 | MONGO_INITDB_ROOT_USERNAME: mongoadmin 37 | MONGO_INITDB_ROOT_PASSWORD: secret 38 | 39 | zookeeper: 40 | image: confluentinc/cp-zookeeper:6.1.0 41 | hostname: zookeeper 42 | container_name: repl-zookeeper 43 | ports: 44 | - '2181:2181' 45 | environment: 46 | ZOOKEEPER_CLIENT_PORT: 2181 47 | ZOOKEEPER_TICK_TIME: 2000 48 | 49 | kafka: 50 | image: confluentinc/cp-kafka:6.1.0 51 | hostname: kafka 52 | container_name: repl-kafka 53 | depends_on: 54 | - zookeeper 55 | ports: 56 | - '29092:29092' 57 | - '9092:9092' 58 | - '9101:9101' 59 | environment: 60 | KAFKA_BROKER_ID: 1 61 | KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181' 62 | KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 63 | KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 64 | KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1 65 | KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0 66 | KAFKA_JMX_PORT: 9101 67 | KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092 68 | KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT 69 | KAFKA_NUM_PARTITIONS: 5 70 | 71 | networks: 72 | default: 73 | name: repl-network 74 | 75 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Shared/Pipeline/Transforms.cs: -------------------------------------------------------------------------------- 1 | using System.Text.Json; 2 | using Kurrent.Replicator.Shared.Contracts; 3 | 4 | namespace Kurrent.Replicator.Shared.Pipeline; 5 | 6 | public delegate ValueTask TransformEvent(OriginalEvent originalEvent, CancellationToken cancellationToken); 7 | 8 | public static class Transforms { 9 | public static ValueTask DefaultWithExtraMeta(OriginalEvent originalEvent, CancellationToken _) { 10 | var proposed = 11 | new ProposedEvent( 12 | originalEvent.EventDetails, 13 | originalEvent.Data, 14 | AddMeta(), 15 | originalEvent.LogPosition, 16 | originalEvent.SequenceNumber 17 | ); 18 | 19 | return new(proposed); 20 | 21 | byte[] AddMeta() { 22 | if (originalEvent.Metadata == null || originalEvent.Metadata.Length == 0) { 23 | var eventMeta = new EventMetadata { 24 | OriginalEventNumber = originalEvent.LogPosition.EventNumber, 25 | OriginalPosition = originalEvent.LogPosition.EventPosition, 26 | OriginalCreatedDate = originalEvent.Created 27 | }; 28 | 29 | return JsonSerializer.SerializeToUtf8Bytes(eventMeta); 30 | } 31 | 32 | using var stream = new MemoryStream(); 33 | using var writer = new Utf8JsonWriter(stream); 34 | using var originalMeta = JsonDocument.Parse(originalEvent.Metadata); 35 | 36 | writer.WriteStartObject(); 37 | 38 | foreach (var jsonElement in originalMeta.RootElement.EnumerateObject()) { 39 | jsonElement.WriteTo(writer); 40 | } 41 | 42 | writer.WriteNumber(EventMetadata.EventNumberPropertyName, originalEvent.LogPosition.EventNumber); 43 | writer.WriteNumber(EventMetadata.PositionPropertyName, originalEvent.LogPosition.EventPosition); 44 | writer.WriteString(EventMetadata.CreatedDate, originalEvent.Created); 45 | writer.WriteEndObject(); 46 | writer.Flush(); 47 | 48 | return stream.ToArray(); 49 | } 50 | } 51 | } 52 | -------------------------------------------------------------------------------- /test/Kurrent.Replicator.Tests/Kurrent.Replicator.Tests.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | $([System.IO.Directory]::GetParent($(MSBuildThisFileDirectory)).Parent.FullName) 5 | net9.0 6 | latest 7 | false 8 | enable 9 | true 10 | true 11 | true 12 | true 13 | --report-trx --results-directory $(RepoRoot)/test-results/$(TargetFramework) 14 | false 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.KurrentDb/Internals/SystemMetadata.cs: -------------------------------------------------------------------------------- 1 | namespace Kurrent.Replicator.KurrentDb.Internals; 2 | 3 | /// 4 | ///Constants for information in stream metadata 5 | /// 6 | static class SystemMetadata { 7 | /// 8 | ///The definition of the MaxAge value assigned to stream metadata 9 | ///Setting this allows all events older than the limit to be deleted 10 | /// 11 | public const string MaxAge = "$maxAge"; 12 | 13 | /// 14 | ///The definition of the MaxCount value assigned to stream metadata 15 | ///setting this allows all events with a sequence less than current -maxcount to be deleted 16 | /// 17 | public const string MaxCount = "$maxCount"; 18 | 19 | /// 20 | ///The definition of the Truncate Before value assigned to stream metadata 21 | ///setting this allows all events prior to the integer value to be deleted 22 | /// 23 | public const string TruncateBefore = "$tb"; 24 | 25 | /// 26 | /// Sets the cache control in seconds for the head of the stream. 27 | /// 28 | public const string CacheControl = "$cacheControl"; 29 | 30 | 31 | /// 32 | /// The acl definition in metadata 33 | /// 34 | public const string Acl = "$acl"; 35 | 36 | /// 37 | /// to read from a stream 38 | /// 39 | public const string AclRead = "$r"; 40 | 41 | /// 42 | /// to write to a stream 43 | /// 44 | public const string AclWrite = "$w"; 45 | 46 | /// 47 | /// to delete a stream 48 | /// 49 | public const string AclDelete = "$d"; 50 | 51 | /// 52 | /// to read metadata 53 | /// 54 | public const string AclMetaRead = "$mr"; 55 | 56 | /// 57 | /// to write metadata 58 | /// 59 | public const string AclMetaWrite = "$mw"; 60 | 61 | /// 62 | /// The user default acl stream 63 | /// 64 | public const string UserStreamAcl = "$userStreamAcl"; 65 | 66 | /// 67 | /// the system stream defaults acl stream 68 | /// 69 | public const string SystemStreamAcl = "$systemStreamAcl"; 70 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Partitioning/PartitionChannel.cs: -------------------------------------------------------------------------------- 1 | using System.Threading.Channels; 2 | using Kurrent.Replicator.Shared.Logging; 3 | using GreenPipes; 4 | using GreenPipes.Agents; 5 | using Kurrent.Replicator.Sink; 6 | 7 | namespace Kurrent.Replicator.Partitioning; 8 | 9 | public class PartitionChannel : Agent { 10 | readonly int _index; 11 | readonly Task _reader; 12 | readonly ChannelWriter _writer; 13 | 14 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 15 | 16 | long _writeSequence; 17 | 18 | public PartitionChannel(int index) { 19 | _index = index; 20 | var channel = Channel.CreateBounded(1); 21 | _reader = Task.Run(() => Reader(channel.Reader)); 22 | _writer = channel.Writer; 23 | } 24 | 25 | public async Task Send(T context, IPipe next) where T : class, PipeContext 26 | => await _writer.WriteAsync(new(context as SinkContext, next as IPipe)); 27 | 28 | async Task Reader(ChannelReader reader) { 29 | while (!IsStopping) { 30 | try { 31 | var (context, pipe) = await reader.ReadAsync(Stopping); 32 | 33 | if (context == null || pipe == null) 34 | throw new InvalidCastException("Wrong context type, expected SinkContext"); 35 | 36 | if (context.ProposedEvent.SequenceNumber < _writeSequence) 37 | Log.Warn("Wrong sequence for {Type}", context.ProposedEvent.EventDetails.EventType); 38 | _writeSequence = context.ProposedEvent.SequenceNumber; 39 | 40 | await pipe.Send(context); 41 | } 42 | catch (OperationCanceledException) { 43 | if (!IsStopping) throw; 44 | } 45 | } 46 | } 47 | 48 | protected override async Task StopAgent(StopContext context) { 49 | await _reader.ConfigureAwait(false); 50 | await base.StopAgent(context).ConfigureAwait(false); 51 | } 52 | 53 | public void Probe(ProbeContext context) => context.CreateScope($"partition-{_index}"); 54 | } 55 | 56 | record DelayedOperation(SinkContext? Context, IPipe? Pipe); -------------------------------------------------------------------------------- /src/Kurrent.Replicator.EventStore/Realtime.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Logging; 2 | 3 | namespace Kurrent.Replicator.EventStore; 4 | 5 | class Realtime(IEventStoreConnection connection, StreamMetaCache metaCache) { 6 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 7 | 8 | bool _started; 9 | 10 | public Task Start() { 11 | if (_started) return Task.CompletedTask; 12 | 13 | Log.Info("Starting realtime subscription for meta updates"); 14 | _started = true; 15 | 16 | return connection.SubscribeToAllAsync(false, (_, re) => HandleEvent(re), HandleDrop); 17 | } 18 | 19 | void HandleDrop(EventStoreSubscription subscription, SubscriptionDropReason reason, Exception exception) { 20 | Log.Info("Connection dropped because {Reason}", reason); 21 | 22 | if (reason is SubscriptionDropReason.UserInitiated) return; 23 | 24 | _started = false; 25 | 26 | var task = reason is SubscriptionDropReason.ConnectionClosed ? StartAfterDelay() : Start(); 27 | Task.Run(() => task); 28 | 29 | return; 30 | 31 | async Task StartAfterDelay() { 32 | await Task.Delay(5000).ConfigureAwait(false); 33 | await Start().ConfigureAwait(false); 34 | } 35 | } 36 | 37 | Task HandleEvent(ResolvedEvent re) { 38 | if (IsSystemEvent()) { 39 | return Task.CompletedTask; 40 | } 41 | 42 | if (IsMetadataUpdate()) { 43 | var stream = re.OriginalStreamId[2..]; 44 | var meta = StreamMetadata.FromJsonBytes(re.Event.Data); 45 | 46 | if (Log.IsDebugEnabled()) { 47 | Log.Debug("Real-time meta update {Stream}: {Meta}", stream, meta); 48 | } 49 | 50 | metaCache.UpdateStreamMeta(stream, meta, re.OriginalEventNumber, re.OriginalEvent.Created); 51 | } 52 | else { 53 | metaCache.UpdateStreamLastEventNumber(re.OriginalStreamId, re.OriginalEventNumber); 54 | } 55 | 56 | return Task.CompletedTask; 57 | 58 | bool IsSystemEvent() => re.Event.EventType.StartsWith('$') && re.Event.EventType != Predefined.MetadataEventType; 59 | 60 | bool IsMetadataUpdate() => re.Event.EventType == Predefined.MetadataEventType; 61 | } 62 | } 63 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Js/JsFunction.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Logging; 2 | using Jint; 3 | using Jint.Native; 4 | 5 | // ReSharper disable UnusedMember.Global 6 | // ReSharper disable InconsistentNaming 7 | 8 | namespace Kurrent.Replicator.Js; 9 | 10 | public class JsFunction { 11 | protected readonly JsValue _func; 12 | 13 | public Engine Engine { get; } 14 | 15 | protected JsFunction(string jsFunc, string name) { 16 | var jsLog = new JsLog(name); 17 | 18 | Engine = new Engine(cfg => cfg.AllowClr()).SetValue("log", jsLog); 19 | 20 | _func = Engine.Execute(jsFunc).GetValue(name); 21 | } 22 | } 23 | 24 | public class TypedJsFunction(string jsFunc, string name, Func convert) 25 | : JsFunction(jsFunc, name) 26 | where T : class { 27 | public TResult Execute(T arg) { 28 | var result = Engine.Invoke(_func, arg); 29 | 30 | return convert(result, arg); 31 | } 32 | } 33 | 34 | class JsLog(string context) { 35 | readonly ILog _log = LogProvider.GetLogger($"js-{context}"); 36 | 37 | public void info( 38 | string message, 39 | object? arg1 = null, 40 | object? arg2 = null, 41 | object? arg3 = null, 42 | object? arg4 = null, 43 | object? arg5 = null 44 | ) => _log.Info(message, arg1, arg2, arg3, arg4, arg5); 45 | 46 | public void debug( 47 | string message, 48 | object? arg1 = null, 49 | object? arg2 = null, 50 | object? arg3 = null, 51 | object? arg4 = null, 52 | object? arg5 = null 53 | ) => _log.Debug(message, arg1, arg2, arg3, arg4, arg5); 54 | 55 | public void warn( 56 | string message, 57 | object? arg1 = null, 58 | object? arg2 = null, 59 | object? arg3 = null, 60 | object? arg4 = null, 61 | object? arg5 = null 62 | ) => _log.Warn(message, arg1, arg2, arg3, arg4, arg5); 63 | 64 | public void error( 65 | string message, 66 | object? arg1 = null, 67 | object? arg2 = null, 68 | object? arg3 = null, 69 | object? arg4 = null, 70 | object? arg5 = null 71 | ) => _log.Error(message, arg1, arg2, arg3, arg4, arg5); 72 | } 73 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.KurrentDb/StreamMetaCache.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Concurrent; 2 | using Kurrent.Replicator.Shared.Extensions; 3 | using Kurrent.Replicator.Shared.Logging; 4 | 5 | namespace Kurrent.Replicator.KurrentDb; 6 | 7 | class StreamMetaCache { 8 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 9 | 10 | readonly ConcurrentDictionary _streamsSize = new(); 11 | readonly ConcurrentDictionary _streamsMeta = new(); 12 | 13 | public async Task GetOrAddStreamMeta(string stream, Func> getMeta) { 14 | try { 15 | var meta = await _streamsMeta.GetOrAddAsync(stream, () => getMeta(stream)); 16 | 17 | return meta; 18 | } catch (Exception e) { 19 | Log.Warn(e, "Unable to read metadata for stream {Stream}", stream); 20 | 21 | return null; 22 | } 23 | } 24 | 25 | public void UpdateStreamMeta(string stream, StreamMetadata streamMetadata, long version) { 26 | var isDeleted = IsStreamDeleted(streamMetadata); 27 | 28 | if (!_streamsMeta.TryGetValue(stream, out var meta)) { 29 | _streamsMeta[stream] = new(isDeleted, streamMetadata.MaxAge, streamMetadata.MaxCount, version); 30 | 31 | return; 32 | } 33 | 34 | if (meta.Version > version) 35 | return; 36 | 37 | if (streamMetadata.MaxAge.HasValue) 38 | meta = meta with { MaxAge = streamMetadata.MaxAge }; 39 | 40 | if (streamMetadata.MaxCount.HasValue) 41 | meta = meta with { MaxCount = streamMetadata.MaxCount }; 42 | 43 | if (isDeleted) 44 | meta = meta with { IsDeleted = true }; 45 | 46 | _streamsMeta[stream] = meta; 47 | } 48 | 49 | public Task GetOrAddStreamSize(string stream, Func> getSize) 50 | => _streamsSize.GetOrAddAsync(stream, () => getSize(stream)); 51 | 52 | public void UpdateStreamLastEventNumber(string stream, long lastEventNumber) { 53 | if (!_streamsSize.TryGetValue(stream, out var size) || size.LastEventNumber < lastEventNumber) { 54 | _streamsSize[stream] = new(lastEventNumber); 55 | } 56 | } 57 | 58 | static bool IsStreamDeleted(StreamMetadata meta) => meta.TruncateBefore == long.MaxValue; 59 | } 60 | 61 | record StreamSize(long LastEventNumber); 62 | 63 | record StreamMeta(bool IsDeleted, TimeSpan? MaxAge, long? MaxCount, long Version); 64 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Read/ReaderPipe.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared; 2 | using Kurrent.Replicator.Shared.Logging; 3 | using Kurrent.Replicator.Shared.Observe; 4 | using GreenPipes; 5 | using Kurrent.Replicator.Observers; 6 | using Kurrent.Replicator.Prepare; 7 | 8 | namespace Kurrent.Replicator.Read; 9 | 10 | public class ReaderPipe { 11 | readonly IPipe _pipe; 12 | 13 | public ReaderPipe(IEventReader reader, ICheckpointStore checkpointStore, Func send) { 14 | var log = LogProvider.GetCurrentClassLogger(); 15 | 16 | _pipe = Pipe.New(cfg => { 17 | cfg.UseConcurrencyLimit(1); 18 | 19 | cfg.UseRetry(retry => { 20 | retry.Incremental(50, TimeSpan.Zero, TimeSpan.FromMilliseconds(10)); 21 | retry.ConnectRetryObserver(new LoggingRetryObserver()); 22 | } 23 | ); 24 | cfg.UseLog(); 25 | cfg.UseExecuteAsync(Reader); 26 | } 27 | ); 28 | 29 | return; 30 | 31 | async Task Reader(ReaderContext ctx) { 32 | try { 33 | var start = await checkpointStore.LoadCheckpoint(ctx.CancellationToken).ConfigureAwait(false); 34 | log.Info("Reading from {Position}", start); 35 | 36 | await reader.ReadEvents( 37 | start, 38 | async read => { 39 | ReplicationMetrics.ReadingPosition.Set(read.LogPosition.EventPosition); 40 | await send(new(read, ctx.CancellationToken)).ConfigureAwait(false); 41 | }, 42 | ctx.CancellationToken 43 | ) 44 | .ConfigureAwait(false); 45 | } catch (OperationCanceledException) { 46 | // it's ok 47 | } catch (Exception e) { 48 | log.Error(e, "Reader error"); 49 | } finally { 50 | log.Info("Reader stopped"); 51 | } 52 | } 53 | } 54 | 55 | public async Task Start(CancellationToken stoppingToken) { 56 | try { 57 | await _pipe.Send(new(stoppingToken)); 58 | } catch (Exception e) { 59 | Console.WriteLine(e); 60 | 61 | throw; 62 | } 63 | } 64 | } 65 | 66 | public class ReaderContext(CancellationToken cancellationToken) : BasePipeContext(cancellationToken), PipeContext; 67 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Http/HttpTransform.cs: -------------------------------------------------------------------------------- 1 | using System.Net; 2 | using System.Text; 3 | using System.Text.Json; 4 | using Kurrent.Replicator.Shared.Contracts; 5 | 6 | namespace Kurrent.Replicator.Http; 7 | 8 | public class HttpTransform { 9 | readonly HttpClient _client; 10 | 11 | public HttpTransform(string? url) { 12 | ArgumentException.ThrowIfNullOrEmpty(url); 13 | _client = new() { BaseAddress = new(url) }; 14 | } 15 | 16 | public async ValueTask Transform(OriginalEvent originalEvent, CancellationToken cancellationToken) { 17 | var httpEvent = new HttpEvent( 18 | originalEvent.EventDetails.EventType, 19 | originalEvent.EventDetails.Stream, 20 | Encoding.UTF8.GetString(originalEvent.Data), 21 | originalEvent.Metadata == null ? null : Encoding.UTF8.GetString(originalEvent.Metadata) 22 | ); 23 | 24 | try { 25 | var response = await _client.PostAsync("", new ByteArrayContent(JsonSerializer.SerializeToUtf8Bytes(httpEvent)), cancellationToken).ConfigureAwait(false); 26 | 27 | if (!response.IsSuccessStatusCode) { 28 | throw new HttpRequestException($"Transformation request failed: {response.ReasonPhrase}"); 29 | } 30 | 31 | if (response.StatusCode == HttpStatusCode.NoContent) { 32 | return new IgnoredEvent(originalEvent.EventDetails, originalEvent.LogPosition, originalEvent.SequenceNumber); 33 | } 34 | 35 | var httpResponse = (await JsonSerializer.DeserializeAsync( 36 | await response.Content.ReadAsStreamAsync(cancellationToken).ConfigureAwait(false), 37 | cancellationToken: cancellationToken 38 | ) 39 | .ConfigureAwait(false))!; 40 | 41 | return new ProposedEvent( 42 | originalEvent.EventDetails with { 43 | EventType = httpResponse.EventType, Stream = httpResponse.StreamName 44 | }, 45 | Encoding.UTF8.GetBytes(httpResponse.Payload), 46 | httpResponse.Metadata == null ? originalEvent.Metadata : Encoding.UTF8.GetBytes(httpResponse.Metadata), 47 | originalEvent.LogPosition, 48 | originalEvent.SequenceNumber 49 | ); 50 | } catch (OperationCanceledException) { 51 | return new NoEvent(originalEvent.EventDetails, originalEvent.LogPosition, originalEvent.SequenceNumber); 52 | } 53 | } 54 | 55 | record HttpEvent(string EventType, string StreamName, string Payload, string? Metadata); 56 | } 57 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Partitioning/ValuePartitioner.cs: -------------------------------------------------------------------------------- 1 | using System.Text; 2 | using Kurrent.Replicator.Shared.Logging; 3 | using GreenPipes; 4 | using GreenPipes.Agents; 5 | using GreenPipes.Partitioning; 6 | 7 | namespace Kurrent.Replicator.Partitioning; 8 | 9 | public class ValuePartitioner : Supervisor, IPartitioner { 10 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 11 | 12 | readonly string _id = Guid.NewGuid().ToString("N"); 13 | 14 | readonly Dictionary _partitions = new(); 15 | 16 | int _partitionsCount; 17 | 18 | IPartitioner IPartitioner.GetPartitioner(PartitionKeyProvider keyProvider) 19 | => new ContextPartitioner(this, keyProvider); 20 | 21 | public void Probe(ProbeContext context) { 22 | var scope = context.CreateScope("partitioner"); 23 | scope.Add("id", _id); 24 | 25 | foreach (var t in _partitions.Values) 26 | t.Probe(scope); 27 | } 28 | 29 | async Task Send(string key, T context, IPipe next) where T : class, PipeContext { 30 | var partitionKey = key.StartsWith("$") ? "$system" : key; 31 | if (!_partitions.ContainsKey(partitionKey)) { 32 | Log.Info("Adding new partition {Partition}", partitionKey); 33 | _partitions[partitionKey] = new PartitionChannel(_partitionsCount++); 34 | } 35 | await _partitions[partitionKey].Send(context, next); 36 | } 37 | 38 | class ContextPartitioner(ValuePartitioner partitioner, PartitionKeyProvider keyProvider) 39 | : IPartitioner where TContext : class, PipeContext { 40 | public Task Send(TContext context, IPipe next) { 41 | var key = Encoding.UTF8.GetString(keyProvider(context)); 42 | 43 | if (key == null) 44 | throw new InvalidOperationException("The key cannot be null"); 45 | 46 | return partitioner.Send(key, context, next); 47 | } 48 | 49 | public void Probe(ProbeContext context) => partitioner.Probe(context); 50 | 51 | Task IAgent.Ready => partitioner.Ready; 52 | Task IAgent.Completed => partitioner.Completed; 53 | 54 | CancellationToken IAgent.Stopping => partitioner.Stopping; 55 | CancellationToken IAgent.Stopped => partitioner.Stopped; 56 | 57 | Task IAgent.Stop(StopContext context) => partitioner.Stop(context); 58 | 59 | int ISupervisor. PeakActiveCount => partitioner.PeakActiveCount; 60 | long ISupervisor.TotalCount => partitioner.TotalCount; 61 | 62 | void ISupervisor.Add(IAgent agent) => partitioner.Add(agent); 63 | } 64 | } -------------------------------------------------------------------------------- /src/Kurrent.Replicator/FileCheckpointStore.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared; 2 | using Kurrent.Replicator.Shared.Logging; 3 | 4 | namespace Kurrent.Replicator; 5 | 6 | public class FileCheckpointStore : ICheckpointStore { 7 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 8 | 9 | readonly string _fileName; 10 | readonly int _checkpointAfter; 11 | 12 | public FileCheckpointStore(string filePath, int checkpointAfter) { 13 | _fileName = filePath; 14 | _checkpointAfter = checkpointAfter; 15 | 16 | try { 17 | if (File.Exists(filePath)) { 18 | return; 19 | } 20 | 21 | File.AppendAllText(filePath, "test"); 22 | File.Delete(filePath); 23 | } catch (Exception e) { 24 | Log.Fatal(e, "Unable to write to {File}", filePath); 25 | 26 | throw; 27 | } 28 | } 29 | 30 | public ValueTask HasStoredCheckpoint(CancellationToken cancellationToken) { 31 | return ValueTask.FromResult(File.Exists(_fileName) && new FileInfo(_fileName).Length > 0); 32 | } 33 | 34 | public async ValueTask LoadCheckpoint(CancellationToken cancellationToken) { 35 | if (_lastPosition != null) { 36 | Log.Info("Starting from a previously known checkpoint {LastKnown}", _lastPosition); 37 | 38 | return _lastPosition; 39 | } 40 | 41 | if (!File.Exists(_fileName)) { 42 | Log.Info("No checkpoint file found, starting from the beginning"); 43 | 44 | return LogPosition.Start; 45 | } 46 | 47 | var content = await File.ReadAllTextAsync(_fileName, cancellationToken).ConfigureAwait(false); 48 | var numbers = content.Split(',').Select(x => Convert.ToInt64(x)).ToArray(); 49 | 50 | Log.Info("Loaded the checkpoint from file: {Checkpoint}", numbers[1]); 51 | 52 | return new LogPosition(numbers[0], (ulong)numbers[1]); 53 | } 54 | 55 | int _counter; 56 | LogPosition? _lastPosition; 57 | 58 | public async ValueTask StoreCheckpoint(LogPosition logPosition, CancellationToken cancellationToken) { 59 | _lastPosition = logPosition; 60 | 61 | Interlocked.Increment(ref _counter); 62 | 63 | if (_counter < _checkpointAfter) return; 64 | 65 | await Flush(cancellationToken).ConfigureAwait(false); 66 | 67 | Interlocked.Exchange(ref _counter, 0); 68 | } 69 | 70 | public async ValueTask Flush(CancellationToken cancellationToken) { 71 | if (_lastPosition == null) return; 72 | 73 | await File.WriteAllTextAsync(_fileName, $"{_lastPosition.EventNumber},{_lastPosition.EventPosition}", cancellationToken).ConfigureAwait(false); 74 | } 75 | } 76 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Partitioning/HashPartitioner.cs: -------------------------------------------------------------------------------- 1 | using GreenPipes; 2 | using GreenPipes.Agents; 3 | using GreenPipes.Partitioning; 4 | 5 | namespace Kurrent.Replicator.Partitioning; 6 | 7 | public class HashPartitioner : Supervisor, IPartitioner { 8 | readonly IHashGenerator _hashGenerator; 9 | readonly string _id= Guid.NewGuid().ToString("N"); 10 | readonly int _partitionCount; 11 | readonly PartitionChannel[] _partitions; 12 | 13 | public HashPartitioner(int partitionCount, IHashGenerator hashGenerator) { 14 | _partitionCount = partitionCount; 15 | _hashGenerator = hashGenerator; 16 | 17 | _partitions = Enumerable.Range(0, partitionCount) 18 | .Select(index => new PartitionChannel(index)) 19 | .ToArray(); 20 | } 21 | 22 | IPartitioner IPartitioner.GetPartitioner(PartitionKeyProvider keyProvider) 23 | => new ContextPartitioner(this, keyProvider); 24 | 25 | public void Probe(ProbeContext context) { 26 | var scope = context.CreateScope("partitioner"); 27 | scope.Add("id", _id); 28 | scope.Add("partitionCount", _partitionCount); 29 | 30 | foreach (var t in _partitions) 31 | t.Probe(scope); 32 | } 33 | 34 | Task Send(byte[] key, T context, IPipe next) where T : class, PipeContext { 35 | var hash = key.Length > 0 ? _hashGenerator.Hash(key) : 0; 36 | 37 | var partitionId = hash % _partitionCount; 38 | 39 | return _partitions[partitionId].Send(context, next); 40 | } 41 | 42 | class ContextPartitioner(HashPartitioner partitioner, PartitionKeyProvider keyProvider) 43 | : IPartitioner where TContext : class, PipeContext { 44 | public Task Send(TContext context, IPipe next) { 45 | var key = keyProvider(context); 46 | 47 | if (key == null) 48 | throw new InvalidOperationException("The key cannot be null"); 49 | 50 | return partitioner.Send(key, context, next); 51 | } 52 | 53 | public void Probe(ProbeContext context) => partitioner.Probe(context); 54 | 55 | Task IAgent.Ready => partitioner.Ready; 56 | Task IAgent.Completed => partitioner.Completed; 57 | 58 | CancellationToken IAgent.Stopping => partitioner.Stopping; 59 | CancellationToken IAgent.Stopped => partitioner.Stopped; 60 | 61 | Task IAgent.Stop(StopContext context) => partitioner.Stop(context); 62 | 63 | int ISupervisor. PeakActiveCount => partitioner.PeakActiveCount; 64 | long ISupervisor.TotalCount => partitioner.TotalCount; 65 | 66 | void ISupervisor.Add(IAgent agent) => partitioner.Add(agent); 67 | } 68 | } 69 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Kafka/KafkaWriter.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | using Kurrent.Replicator.Shared.Logging; 3 | using Kurrent.Replicator.Shared.Observe; 4 | using Ubiquitous.Metrics; 5 | 6 | namespace Kurrent.Replicator.Kafka; 7 | 8 | public class KafkaWriter : IEventWriter { 9 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 10 | 11 | readonly IProducer _producer; 12 | readonly Action? _debug; 13 | readonly Func _route; 14 | 15 | KafkaWriter(ProducerConfig config) { 16 | var producerBuilder = new ProducerBuilder(config); 17 | _producer = producerBuilder.Build(); 18 | _debug = Log.IsDebugEnabled() ? Log.Debug : null; 19 | _route = x => DefaultRouters.RouteByCategory(x.EventDetails.Stream); 20 | } 21 | 22 | public KafkaWriter(ProducerConfig config, string? routingFunction) : this(config) { 23 | if (routingFunction != null) 24 | _route = new KafkaJsMessageRouter(routingFunction).Route; 25 | } 26 | 27 | public Task Start() => Task.CompletedTask; 28 | 29 | public Task WriteEvent(BaseProposedEvent proposedEvent, CancellationToken cancellationToken) { 30 | var task = proposedEvent switch { 31 | ProposedEvent p => Append(p), 32 | ProposedMetaEvent => NoOp(), 33 | ProposedDeleteStream => NoOp(), 34 | IgnoredEvent => NoOp(), 35 | _ => throw new InvalidOperationException("Unknown proposed event type") 36 | }; 37 | 38 | return Metrics.Measure(() => task, ReplicationMetrics.WritesHistogram, ReplicationMetrics.WriteErrorsCount); 39 | 40 | async Task Append(ProposedEvent p) { 41 | var (topic, partitionKey) = _route(p); 42 | 43 | _debug?.Invoke( 44 | "Kafka: Write event with id {Id} of type {Type} to {Stream} with original position {Position}", 45 | [ 46 | proposedEvent.EventDetails.EventId, 47 | proposedEvent.EventDetails.EventType, 48 | topic, 49 | proposedEvent.SourceLogPosition.EventPosition 50 | ] 51 | ); 52 | 53 | // TODO: Map meta to headers, but only for JSON 54 | var message = new Message { 55 | Key = partitionKey, 56 | Value = p.Data 57 | }; 58 | 59 | var result = await _producer.ProduceAsync(topic, message, cancellationToken).ConfigureAwait(false); 60 | 61 | return result.Offset.Value; 62 | } 63 | 64 | static Task NoOp() => Task.FromResult(-1L); 65 | } 66 | } 67 | -------------------------------------------------------------------------------- /src/replicator/Settings/ReplicatorSettings.cs: -------------------------------------------------------------------------------- 1 | // ReSharper disable UnusedAutoPropertyAccessor.Global 2 | 3 | #nullable disable 4 | namespace replicator.Settings; 5 | 6 | public record EsdbSettings { 7 | public string ConnectionString { get; init; } 8 | public string Protocol { get; init; } 9 | public int PageSize { get; init; } = 1024; 10 | } 11 | 12 | public record CheckpointSeeder { 13 | public string Path { get; init; } 14 | 15 | public string Type { get; init; } = "none"; // "chaser" 16 | } 17 | 18 | public record Checkpoint { 19 | public string Path { get; init; } 20 | public string Type { get; init; } = "file"; 21 | public int CheckpointAfter { get; init; } = 1000; 22 | public string Database { get; init; } = "replicator"; 23 | public string InstanceId { get; init; } = "default"; 24 | public CheckpointSeeder Seeder { get; init; } = new(); 25 | } 26 | 27 | public record SinkSettings : EsdbSettings { 28 | public int PartitionCount { get; init; } = 1; 29 | public string Router { get; init; } 30 | public string Partitioner { get; init; } 31 | public int BufferSize { get; init; } = 1000; 32 | } 33 | 34 | public record TransformSettings { 35 | public string Type { get; init; } = "default"; 36 | public string Config { get; init; } 37 | public int BufferSize { get; init; } = 1; 38 | } 39 | 40 | public record Filter { 41 | public string Type { get; init; } 42 | public string Include { get; init; } 43 | public string Exclude { get; init; } 44 | } 45 | 46 | public record Replicator { 47 | public EsdbSettings Reader { get; init; } 48 | public SinkSettings Sink { get; init; } 49 | public bool Scavenge { get; init; } 50 | public bool RestartOnFailure { get; init; } = true; 51 | public bool RunContinuously { get; init; } = true; 52 | public int RestartDelayInSeconds { get; init; } = 5; 53 | public int ReportMetricsFrequencyInSeconds { get; init; } = 5; 54 | public Checkpoint Checkpoint { get; init; } = new(); 55 | public TransformSettings Transform { get; init; } = new(); 56 | public Filter[] Filters { get; init; } 57 | } 58 | 59 | public static class ConfigExtensions { 60 | public static T GetAs(this IConfiguration configuration) where T : new() { 61 | T result = new(); 62 | configuration.GetSection(typeof(T).Name).Bind(result); 63 | 64 | return result; 65 | } 66 | } 67 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.EventStore/StreamMetaCache.cs: -------------------------------------------------------------------------------- 1 | using System.Collections.Concurrent; 2 | using Kurrent.Replicator.Shared.Extensions; 3 | using Kurrent.Replicator.Shared.Logging; 4 | 5 | namespace Kurrent.Replicator.EventStore; 6 | 7 | class StreamMetaCache { 8 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 9 | 10 | readonly ConcurrentDictionary _streamsSize = new(); 11 | readonly ConcurrentDictionary _streamsMeta = new(); 12 | 13 | public async Task GetOrAddStreamMeta(string stream, Func> getMeta) { 14 | try { 15 | var meta = await _streamsMeta.GetOrAddAsync(stream, () => getMeta(stream)); 16 | return meta; 17 | } 18 | catch (Exception e) { 19 | Log.Warn(e, "Unable to read metadata for stream {Stream}", stream); 20 | return null; 21 | } 22 | } 23 | 24 | public void UpdateStreamMeta(string stream, StreamMetadata streamMetadata, long version, DateTime created) { 25 | var isDeleted = IsStreamDeleted(streamMetadata); 26 | 27 | if (!_streamsMeta.TryGetValue(stream, out var meta)) { 28 | _streamsMeta[stream] = 29 | new( 30 | isDeleted, 31 | isDeleted ? created : DateTime.MaxValue, 32 | streamMetadata.MaxAge, 33 | streamMetadata.MaxCount, 34 | streamMetadata.TruncateBefore, 35 | version 36 | ); 37 | return; 38 | } 39 | 40 | if (meta.Version > version) return; 41 | 42 | if (streamMetadata.MaxAge.HasValue) 43 | meta = meta with {MaxAge = streamMetadata.MaxAge}; 44 | 45 | if (streamMetadata.MaxCount.HasValue) 46 | meta = meta with {MaxCount = streamMetadata.MaxCount}; 47 | 48 | if (streamMetadata.TruncateBefore.HasValue) 49 | meta = meta with {TruncateBefore = streamMetadata.TruncateBefore}; 50 | 51 | if (isDeleted) 52 | meta = meta with {IsDeleted = true, DeletedAt = created}; 53 | 54 | _streamsMeta[stream] = meta; 55 | } 56 | 57 | public Task GetOrAddStreamSize(string stream, Func> getSize) 58 | => _streamsSize.GetOrAddAsync(stream, () => getSize(stream)); 59 | 60 | public void UpdateStreamLastEventNumber(string stream, long lastEventNumber) { 61 | if (!_streamsSize.TryGetValue(stream, out var size) || size.LastEventNumber < lastEventNumber) { 62 | _streamsSize[stream] = new StreamSize(lastEventNumber); 63 | } 64 | } 65 | 66 | static bool IsStreamDeleted(StreamMetadata meta) => meta.TruncateBefore == long.MaxValue; 67 | } 68 | 69 | record StreamSize(long LastEventNumber); 70 | 71 | record StreamMeta(bool IsDeleted, DateTime DeletedAt, TimeSpan? MaxAge, long? MaxCount, long? TruncateBefore, long Version); -------------------------------------------------------------------------------- /src/replicator/ClientApp/src/components/Dashboard.vue: -------------------------------------------------------------------------------- 1 | 58 | 59 | 85 | 86 | 99 | -------------------------------------------------------------------------------- /src/replicator/replicator.csproj: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | ClientApp\ 5 | $(DefaultItemExcludes);$(SpaRoot)node_modules\** 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | %(DistFiles.Identity) 45 | PreserveNewest 46 | true 47 | 48 | 49 | 50 | 51 | 52 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Sink/SinkPipe.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared; 2 | using Kurrent.Replicator.Shared.Observe; 3 | using GreenPipes; 4 | using GreenPipes.Partitioning; 5 | using Kurrent.Replicator.Observers; 6 | using Kurrent.Replicator.Partitioning; 7 | 8 | namespace Kurrent.Replicator.Sink; 9 | 10 | public class SinkPipe { 11 | readonly IPipe _pipe; 12 | 13 | public SinkPipe(IEventWriter writer, SinkPipeOptions options, ICheckpointStore checkpointStore) 14 | => _pipe = Pipe.New(cfg => { 15 | cfg.UseRetry(r => { 16 | r.Incremental(10, TimeSpan.Zero, TimeSpan.FromMilliseconds(10)); 17 | r.ConnectRetryObserver(new LoggingRetryObserver()); 18 | } 19 | ); 20 | 21 | var useJsPartitioner = !string.IsNullOrWhiteSpace(options.Partitioner); 22 | 23 | if (options.PartitionCount > 1) { 24 | var keyProvider = !useJsPartitioner 25 | ? x => KeyProvider.ByStreamName(x.ProposedEvent) 26 | : GetJsPartitioner(); 27 | 28 | cfg.UsePartitioner(new HashPartitioner(options.PartitionCount, new Murmur3UnsafeHashGenerator()), keyProvider); 29 | } 30 | else if (useJsPartitioner) { 31 | cfg.UsePartitioner(new ValuePartitioner(), GetJsPartitioner()); 32 | } 33 | 34 | cfg.UseLog(); 35 | 36 | cfg.UseEventWriter(writer); 37 | cfg.UseCheckpointStore(checkpointStore); 38 | 39 | cfg.UseExecute(ctx => ReplicationMetrics.ProcessedPosition.Set(ctx.ProposedEvent.SourceLogPosition.EventPosition)); 40 | 41 | return; 42 | 43 | Func GetJsPartitioner() { 44 | var jsProvider = new JsKeyProvider(options.Partitioner!); 45 | 46 | return x => jsProvider.GetPartitionKey(x.ProposedEvent); 47 | } 48 | } 49 | ); 50 | 51 | public Task Send(SinkContext context) => _pipe.Send(context); 52 | } 53 | 54 | static class SinkPipelineExtensions { 55 | public static void UseEventWriter(this IPipeConfigurator cfg, IEventWriter writer) 56 | => cfg.UseExecuteAsync(async ctx => { 57 | var position = await writer.WriteEvent(ctx.ProposedEvent, ctx.CancellationToken).ConfigureAwait(false); 58 | 59 | if (position != -1) { 60 | ReplicationMetrics.WriterPosition.Set(position); 61 | } 62 | } 63 | ); 64 | 65 | public static void UseCheckpointStore(this IPipeConfigurator cfg, ICheckpointStore checkpointStore) 66 | => cfg.UseExecuteAsync(async ctx => await checkpointStore.StoreCheckpoint( 67 | ctx.ProposedEvent.SourceLogPosition, 68 | ctx.CancellationToken 69 | ) 70 | .ConfigureAwait(false) 71 | ); 72 | } 73 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Mongo/MongoCheckpointStore.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared; 2 | using Kurrent.Replicator.Shared.Logging; 3 | using MongoDB.Driver; 4 | 5 | namespace Kurrent.Replicator.Mongo; 6 | 7 | public class MongoCheckpointStore : ICheckpointStore { 8 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 9 | 10 | readonly string _id; 11 | readonly int _checkpointAfter; 12 | readonly IMongoCollection _collection; 13 | 14 | public MongoCheckpointStore(string connectionString, string database, string id, int checkpointAfter) { 15 | _id = id; 16 | _checkpointAfter = checkpointAfter; 17 | var settings = MongoClientSettings.FromConnectionString(connectionString); 18 | var db = new MongoClient(settings).GetDatabase(database); 19 | _collection = db.GetCollection("checkpoint"); 20 | } 21 | 22 | int _counter; 23 | LogPosition? _lastPosition; 24 | 25 | public async ValueTask HasStoredCheckpoint(CancellationToken cancellationToken) { 26 | var doc = await _collection 27 | .Find(x => x.Id == _id) 28 | .Limit(1) 29 | .SingleOrDefaultAsync(cancellationToken) 30 | .ConfigureAwait(false); 31 | 32 | return doc != null; 33 | } 34 | 35 | public async ValueTask LoadCheckpoint(CancellationToken cancellationToken) { 36 | if (_lastPosition != null) { 37 | Log.Info("Starting from a previously known checkpoint {LastKnown}", _lastPosition); 38 | 39 | return _lastPosition; 40 | } 41 | 42 | var doc = await _collection 43 | .Find(x => x.Id == _id) 44 | .Limit(1) 45 | .SingleOrDefaultAsync(cancellationToken) 46 | .ConfigureAwait(false); 47 | 48 | if (doc == null) { 49 | Log.Info("No checkpoint file found, starting from the beginning"); 50 | 51 | await _collection.InsertOneAsync(new(_id, LogPosition.Start), cancellationToken: cancellationToken).ConfigureAwait(false); 52 | 53 | return LogPosition.Start; 54 | } 55 | 56 | Log.Info("Loaded the checkpoint from MongoDB: {Checkpoint}", doc.LogPosition); 57 | 58 | return doc.LogPosition; 59 | } 60 | 61 | public async ValueTask StoreCheckpoint(LogPosition logPosition, CancellationToken cancellationToken) { 62 | _lastPosition = logPosition; 63 | 64 | Interlocked.Increment(ref _counter); 65 | 66 | if (_counter < _checkpointAfter) return; 67 | 68 | await Flush(cancellationToken).ConfigureAwait(false); 69 | 70 | Interlocked.Exchange(ref _counter, 0); 71 | } 72 | 73 | public async ValueTask Flush(CancellationToken cancellationToken) { 74 | if (_lastPosition == null) return; 75 | 76 | await _collection.ReplaceOneAsync(Builders.Filter.Eq(x => x.Id, _id), new(_id, _lastPosition), cancellationToken: cancellationToken).ConfigureAwait(false); 77 | } 78 | 79 | record Checkpoint(string Id, LogPosition LogPosition); 80 | } 81 | -------------------------------------------------------------------------------- /charts/replicator/templates/statefulset.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: {{ template "deployment.apiVersion" . }} 2 | kind: StatefulSet 3 | metadata: 4 | name: {{ template "replicator.fullname" . }} 5 | labels: 6 | app: {{ template "replicator.name" . }} 7 | chart: {{ template "replicator.chart" . }} 8 | release: {{ .Release.Name }} 9 | heritage: {{ .Release.Service }} 10 | tier: web 11 | spec: 12 | serviceName: {{ template "replicator.fullname" . }} 13 | replicas: 1 14 | selector: 15 | matchLabels: 16 | app: {{ template "replicator.name" . }} 17 | release: {{ .Release.Name }} 18 | tier: web 19 | template: 20 | metadata: 21 | labels: 22 | app: {{ template "replicator.name" . }} 23 | release: {{ .Release.Name }} 24 | tier: web 25 | spec: 26 | containers: 27 | - name: {{ .Chart.Name }} 28 | image: {{ template "replicator.image" . }} 29 | imagePullPolicy: {{ .Values.image.pullPolicy | quote }} 30 | livenessProbe: 31 | httpGet: 32 | path: /health 33 | port: 5000 34 | scheme: HTTP 35 | ports: 36 | - containerPort: 5000 37 | name: web 38 | readinessProbe: 39 | httpGet: 40 | path: /ping 41 | port: 5000 42 | scheme: HTTP 43 | volumeMounts: 44 | - mountPath: /data 45 | name: {{ template "replicator.name" . }} 46 | - name: config-volume 47 | mountPath: /app/config 48 | {{- range .Values.jsConfigMaps }} 49 | - name: {{ .configMapName }} 50 | mountPath: /app/js/{{ .fileName }} 51 | subPath: {{ .fileName }} 52 | {{- end }} 53 | {{- if eq (include "replicator.shouldUseJavascriptTransform" .) "true" }} 54 | - name: transform-js 55 | mountPath: /app{{ include "replicator.transform.filepath" . }} 56 | subPath: {{ include "replicator.transform.filename" . }} 57 | {{- end }} 58 | {{- if eq (include "replicator.shouldUseCustomPartitioner" .) "true" }} 59 | - name: partitioner-js 60 | mountPath: /app{{ include "replicator.sink.partitioner.filepath" . }} 61 | subPath: {{ include "replicator.sink.partitioner.filename" . }} 62 | {{- end }} 63 | resources: 64 | {{ toYaml .Values.resources | indent 10 }} 65 | imagePullSecrets: [] 66 | terminationGracePeriodSeconds: {{ .Values.terminationGracePeriodSeconds }} 67 | volumes: 68 | - name: config-volume 69 | configMap: 70 | name: {{ template "replicator.fullname" . }} 71 | {{- range .Values.jsConfigMaps }} 72 | - name: {{ .configMapName }} 73 | configMap: 74 | name: {{ .configMapName }} 75 | {{- end }} 76 | {{- if eq (include "replicator.shouldUseJavascriptTransform" .) "true" }} 77 | - name: transform-js 78 | configMap: 79 | name: {{ template "replicator.fullname" . }}-transform-js 80 | {{- end }} 81 | {{- if eq (include "replicator.shouldUseCustomPartitioner" .) "true" }} 82 | - name: partitioner-js 83 | configMap: 84 | name: {{ template "replicator.fullname" . }}-partitioner-js 85 | {{- end }} 86 | - name: {{ template "replicator.name" . }} 87 | persistentVolumeClaim: 88 | claimName: {{ template "replicator.fullname" . }} 89 | -------------------------------------------------------------------------------- /test/Kurrent.Replicator.Tests/Fixtures/ContainerFixture.cs: -------------------------------------------------------------------------------- 1 | using DotNet.Testcontainers.Builders; 2 | using DotNet.Testcontainers.Containers; 3 | using EventStore.Client; 4 | using EventStore.ClientAPI; 5 | using Testcontainers.EventStoreDb; 6 | 7 | namespace Kurrent.Replicator.Tests.Fixtures; 8 | 9 | public class ContainerFixture { 10 | EventStoreDbContainer _kurrentDbContainer; 11 | IContainer _eventStoreContainer; 12 | public DirectoryInfo V5DataPath { get; private set; } 13 | 14 | public async Task StartContainers() { 15 | V5DataPath = Directory.CreateTempSubdirectory(); 16 | _kurrentDbContainer = BuildV23Container(); 17 | _eventStoreContainer = BuildV5Container(V5DataPath); 18 | 19 | await _kurrentDbContainer.StartAsync(); 20 | await _eventStoreContainer.StartAsync(); 21 | await Task.Delay(TimeSpan.FromSeconds(2)); // give it some time to spin up 22 | } 23 | 24 | public async Task StopContainers() { 25 | await Task.WhenAll(_kurrentDbContainer.StopAsync(), _eventStoreContainer.StopAsync()); 26 | await _eventStoreContainer.DisposeAsync(); 27 | await _kurrentDbContainer.DisposeAsync(); 28 | } 29 | 30 | public IEventStoreConnection GetV5Client() { 31 | var connectionString = $"ConnectTo=tcp://admin:changeit@localhost:{_eventStoreContainer.GetMappedPublicPort(1113)}; HeartBeatTimeout=500; UseSslConnection=false;"; 32 | var client = ConfigureEventStoreTcp(connectionString); 33 | 34 | return client; 35 | } 36 | 37 | public EventStoreClient GetKurrentClient() { 38 | var connectionString = _kurrentDbContainer.GetConnectionString(); 39 | var settings = EventStoreClientSettings.Create(connectionString); 40 | 41 | return new(settings); 42 | } 43 | 44 | static EventStoreDbContainer BuildV23Container() => new EventStoreDbBuilder() 45 | .WithImage("eventstore/eventstore:24.10") 46 | .WithEnvironment("EVENTSTORE_RUN_PROJECTIONS", "None") 47 | .WithEnvironment("EVENTSTORE_START_STANDARD_PROJECTIONS", "false") 48 | .WithEnvironment("EVENTSTORE_ENABLE_ATOM_PUB_OVER_HTTP", bool.TrueString) 49 | .Build(); 50 | 51 | const int TcpPort = 1113; 52 | const int HttpPort = 2113; 53 | 54 | static IContainer BuildV5Container(DirectoryInfo data) => new ContainerBuilder() 55 | .WithImage("eventstore/eventstore:5.0.11-bionic") 56 | .WithEnvironment("EVENTSTORE_CLUSTER_SIZE", "1") 57 | .WithEnvironment("EVENTSTORE_RUN_PROJECTIONS", "None") 58 | .WithEnvironment("EVENTSTORE_START_STANDARD_PROJECTIONS", "false") 59 | .WithEnvironment("EVENTSTORE_EXT_HTTP_PORT", "2113") 60 | .WithBindMount(data.FullName, "/var/lib/eventstore") 61 | .WithExposedPort(HttpPort) 62 | .WithExposedPort(TcpPort) 63 | .WithPortBinding(HttpPort, true) 64 | .WithPortBinding(TcpPort, true) 65 | .WithWaitStrategy(Wait.ForUnixContainer().UntilPortIsAvailable(TcpPort)) 66 | .Build(); 67 | 68 | static IEventStoreConnection ConfigureEventStoreTcp(string connectionString) { 69 | var builder = ConnectionSettings.Create() 70 | .KeepReconnecting() 71 | .KeepRetrying(); 72 | 73 | var connection = EventStoreConnection.Create(connectionString, builder); 74 | 75 | return connection; 76 | } 77 | } 78 | -------------------------------------------------------------------------------- /docs/src/content/docs/intro/limitations.mdx: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Known limitations" 3 | description: Be aware of these Replicator limitations 4 | sidebar: 5 | order: 3 6 | --- 7 | 8 | import { Aside } from '@astrojs/starlight/components'; 9 | 10 | ## Performance 11 | 12 | Replicator uses conventional client protocols: TCP and gRPC. We recommend using TCP for the source clusters connection (reading) and gRPC for the sink. 13 | 14 | When copying the data, we must ensure that the order of events in the target cluster remains the same. The level of this guarantee depends on the selected write mode (single writer or partitioned concurrent writers), but events are still written one by one, as that's the write mode supported by all the clients. 15 | 16 | These factors impact the overall write performance. Considering the normal latency of a write operation via gRPC (3-6 ms, depending on the cluster instance size and the cloud provider), a single writer can only write 150-300 events per second. Event size, unless it's very big, doesn't play much of a role for the latency figure. Partitioned writes, running concurrently, can effectively reach the speed of more than 1000 events per second. Using more than six concurrent writers would not increase the performance as the bottleneck will shift to the server. 17 | 18 | Based on the mentioned figures, we can expect to replicate around one million events per hour with a single writer, and 3.5 million events per hour when using concurrent writers. Therefore, the tool mainly aims to help customers with small to medium size databases. Replicating a multi-terabyte database with billions of events would probably never work as it won't be able to catch up with frequent writes to the source cluster. 19 | 20 | Therefore, an important indicator that replication will complete is observing the replication gap metric provided by the tool and ensure the gap is lowering. If the gap stays constant or is increasing, then the tool is not suitable for your database. 21 | 22 | ## Created date 23 | 24 | The system property, which holds the timestamp when the event was physically written to the database, won't be propagated to the target cluster as it's impossible to set this value using a conventional client. To mitigate this issue, Replicator will add a metadata field `$originalCreatedDate`, which will contain the original event creation date. 25 | 26 | 29 | 30 | ## Max age stream metadata 31 | 32 | Replicator will copy all the stream metadata. However, the max age set on a stream will not be set as expected, because all the events in the target cluster will be assigned a new date. The `$originalCreatedDate` metadata field might help to mitigate this issue. 33 | 34 | ## Replication of emitted streams 35 | 36 | By default, the Replicator replicates all emitted streams, which can lead to unintended consequences, including disruptions in target cluster projections. To resolve this: 37 | 38 | - **Apply Filters:** Use filters to specify which streams should and should not be replicated. Properly configured filters enable selective control over the replication of emitted streams, ensuring only necessary data is transferred between clusters or instances. 39 | 40 | - **Delete and Restart:** If necessary, delete the emitted streams and restart the projection. Enabling the `track emitted events` option allows for resetting the projection, triggering the re-processing and rewriting of all emitted stream events. 41 | 42 | -------------------------------------------------------------------------------- /test/Kurrent.Replicator.Tests/ChaserCheckpointSeedingTests.cs: -------------------------------------------------------------------------------- 1 | using System.Text.Json; 2 | using EventStore.Client; 3 | using EventStore.ClientAPI; 4 | using Kurrent.Replicator.Shared.Observe; 5 | using Kurrent.Replicator.EventStore; 6 | using Kurrent.Replicator.KurrentDb; 7 | using Kurrent.Replicator.Tests.Fixtures; 8 | using Kurrent.Replicator.Tests.Logging; 9 | using Serilog; 10 | using Ubiquitous.Metrics; 11 | using Ubiquitous.Metrics.NoMetrics; 12 | using EventData = EventStore.ClientAPI.EventData; 13 | using Position = EventStore.Client.Position; 14 | 15 | namespace Kurrent.Replicator.Tests; 16 | 17 | [ClassDataSource] 18 | public class ChaserCheckpointSeedingTests { 19 | readonly ContainerFixture _fixture; 20 | 21 | public ChaserCheckpointSeedingTests(ContainerFixture fixture) { 22 | _fixture = fixture; 23 | ReplicationMetrics.Configure(Metrics.CreateUsing(new NoMetricsProvider())); 24 | 25 | Log.Logger = new LoggerConfiguration() 26 | .MinimumLevel.Information() 27 | .WriteTo.TestOutput() 28 | .CreateLogger() 29 | .ForContext(); 30 | } 31 | 32 | [Before(Test)] 33 | public async Task Start() => await _fixture.StartContainers(); 34 | 35 | [After(Test)] 36 | public async Task Stop() => await _fixture.StopContainers(); 37 | 38 | [Test] 39 | public async Task Verify() { 40 | var v5DataPath = _fixture.V5DataPath; 41 | 42 | await SeedV5WithEvents("ItHappenedBefore", 1000); 43 | 44 | await Task.Delay(TimeSpan.FromSeconds(2)); // give it some time to settle 45 | 46 | // Snapshot chaser.chk 47 | var chaserChkCopy = Path.GetTempFileName(); 48 | File.Copy(Path.Combine(v5DataPath.FullName, "chaser.chk"), chaserChkCopy, true); 49 | 50 | await SeedV5WithEvents("ItHappenedAfterwards", 1001); 51 | 52 | var store = new FileCheckpointStore(Path.GetTempFileName(), 100); 53 | 54 | using var eventStoreClient = _fixture.GetV5Client(); 55 | await using var kurrentClient = _fixture.GetKurrentClient(); 56 | 57 | await 58 | Replicator.Replicate( 59 | new TcpEventReader(eventStoreClient), 60 | new GrpcEventWriter(kurrentClient), 61 | new(), 62 | new(null, null), 63 | new ChaserCheckpointSeeder(chaserChkCopy, store), 64 | store, 65 | new(false, false, TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(5)), 66 | TestContext.Current?.CancellationToken ?? CancellationToken.None 67 | ); 68 | 69 | var events = await kurrentClient.ReadAllAsync(Direction.Forwards, Position.Start) 70 | .Where(evt => !evt.Event.EventType.StartsWith('$')) 71 | .ToArrayAsync(TestContext.Current!.CancellationToken); 72 | 73 | await Assert.That(events).HasCount(1000); 74 | } 75 | 76 | async Task SeedV5WithEvents(string named, int count) { 77 | using var connection = _fixture.GetV5Client(); 78 | await connection.ConnectAsync(); 79 | 80 | var emptyBody = JsonSerializer.SerializeToUtf8Bytes("{}"); 81 | 82 | for (var index = 0; index < count; index++) { 83 | await connection.AppendToStreamAsync( 84 | $"stream-{Random.Shared.Next(0, 10)}", 85 | ExpectedVersion.Any, 86 | new EventData(Guid.NewGuid(), named, true, emptyBody, []) 87 | ); 88 | } 89 | 90 | connection.Close(); 91 | } 92 | } 93 | -------------------------------------------------------------------------------- /charts/replicator/templates/_helpers.tpl: -------------------------------------------------------------------------------- 1 | {{/* vim: set filetype=mustache: */}} 2 | {{/* 3 | Expand the name of the chart. 4 | */}} 5 | {{- define "replicator.name" -}} 6 | {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} 7 | {{- end -}} 8 | 9 | {{/* 10 | Create a default fully qualified app name. 11 | If release name contains chart name it will be used as a full name. 12 | */}} 13 | {{- define "replicator.fullname" -}} 14 | {{- $name := .Chart.Name -}} 15 | {{- if contains $name .Release.Name -}} 16 | {{- .Release.Name | trunc 63 | trimSuffix "-" -}} 17 | {{- else -}} 18 | {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} 19 | {{- end -}} 20 | {{- end -}} 21 | 22 | {{/* 23 | Create chart name and version as used by the chart label. 24 | */}} 25 | {{- define "replicator.chart" -}} 26 | {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} 27 | {{- end -}} 28 | 29 | {{/* 30 | Return the appropriate apiVersion for deployment. 31 | */}} 32 | {{- define "deployment.apiVersion" -}} 33 | {{- if semverCompare ">=1.9-0" .Capabilities.KubeVersion.GitVersion -}} 34 | {{- print "apps/v1" -}} 35 | {{- else -}} 36 | {{- print "extensions/v1beta1" -}} 37 | {{- end -}} 38 | {{- end -}} 39 | 40 | {{/* 41 | Return the proper Replicator image name 42 | */}} 43 | {{- define "replicator.image" -}} 44 | {{- $registryName := .Values.image.registry -}} 45 | {{- $repositoryName := .Values.image.repository -}} 46 | {{- $tag := .Values.image.tag | toString -}} 47 | {{- printf "%s/%s:%s" $registryName $repositoryName $tag -}} 48 | {{- end -}} 49 | 50 | {{/* 51 | Checks if replicator is configured to use javascript transform 52 | */}} 53 | {{- define "replicator.shouldUseJavascriptTransform" -}} 54 | {{- if and 55 | .Values.replicator.transform 56 | (eq .Values.replicator.transform.type "js") 57 | .Values.replicator.transform.config 58 | .Values.transformJs 59 | -}} 60 | {{- print "true" }} 61 | {{- end -}} 62 | {{- end -}} 63 | 64 | {{/* 65 | Checks if replicator is configured to use a custom partitioner 66 | */}} 67 | {{- define "replicator.shouldUseCustomPartitioner" -}} 68 | {{- if and 69 | .Values.replicator.sink.partitioner 70 | .Values.partitionerJs 71 | -}} 72 | {{- print "true" }} 73 | {{- end -}} 74 | {{- end -}} 75 | 76 | {{- define "replicator.transform.filename" -}} 77 | {{- if eq (include "replicator.shouldUseJavascriptTransform" .) "true" -}} 78 | {{ printf "%s" (include "replicator.helpers.filename" .Values.replicator.transform.config) }} 79 | {{- end -}} 80 | {{- end -}} 81 | 82 | {{- define "replicator.transform.filepath" -}} 83 | {{- if eq (include "replicator.shouldUseJavascriptTransform" .) "true" -}} 84 | {{ printf "%s" (include "replicator.helpers.cleansePath" .Values.replicator.transform.config) }} 85 | {{- end -}} 86 | {{- end -}} 87 | 88 | {{- define "replicator.sink.partitioner.filename" -}} 89 | {{- if eq (include "replicator.shouldUseCustomPartitioner" .) "true" -}} 90 | {{ printf "%s" (include "replicator.helpers.filename" .Values.replicator.sink.partitioner) }} 91 | {{- end -}} 92 | {{- end -}} 93 | 94 | {{- define "replicator.sink.partitioner.filepath" -}} 95 | {{- if eq (include "replicator.shouldUseCustomPartitioner" .) "true" -}} 96 | {{ printf "%s" (include "replicator.helpers.cleansePath" .Values.replicator.sink.partitioner) }} 97 | {{- end -}} 98 | {{- end -}} 99 | 100 | {{- define "replicator.helpers.filename" -}} 101 | {{- $file := . -}} 102 | {{- $filename := last (splitList "/" $file) -}} 103 | {{- $filename -}} 104 | {{- end -}} 105 | 106 | {{- define "replicator.helpers.cleansePath" -}} 107 | {{- $path := . -}} 108 | {{- $path = replace "./" "/" $path -}} 109 | {{- if not (hasPrefix "/" $path) -}} 110 | {{- $path = printf "/%s" $path -}} 111 | {{- end -}} 112 | {{- $path -}} 113 | {{- end -}} -------------------------------------------------------------------------------- /src/Kurrent.Replicator/Prepare/TransformFilter.cs: -------------------------------------------------------------------------------- 1 | using System.Diagnostics; 2 | using Kurrent.Replicator.Shared.Contracts; 3 | using Kurrent.Replicator.Shared.Logging; 4 | using Kurrent.Replicator.Shared.Observe; 5 | using Kurrent.Replicator.Shared.Pipeline; 6 | using GreenPipes; 7 | using Ubiquitous.Metrics; 8 | 9 | namespace Kurrent.Replicator.Prepare; 10 | 11 | public class TransformFilter(TransformEvent transform) : IFilter { 12 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 13 | 14 | public async Task Send(PrepareContext context, IPipe next) { 15 | var transformed = context.OriginalEvent is OriginalEvent oe 16 | ? await Metrics.Measure(() => Transform(oe), ReplicationMetrics.PrepareHistogram) 17 | .ConfigureAwait(false) 18 | : TransformMeta(context.OriginalEvent); 19 | 20 | if (transformed is NoEvent) 21 | return; 22 | 23 | context.AddOrUpdatePayload(() => transformed, _ => transformed); 24 | 25 | await next.Send(context).ConfigureAwait(false); 26 | 27 | return; 28 | 29 | async Task Transform(OriginalEvent originalEvent) { 30 | using var activity = new Activity("transform"); 31 | 32 | activity.SetParentId( 33 | context.OriginalEvent.TracingMetadata.TraceId, 34 | context.OriginalEvent.TracingMetadata.SpanId 35 | ); 36 | activity.Start(); 37 | 38 | try { 39 | var res = await transform(originalEvent, context.CancellationToken).ConfigureAwait(false); 40 | activity.SetStatus(ActivityStatusCode.Ok); 41 | 42 | return res; 43 | } catch (Exception e) { 44 | activity.SetStatus(ActivityStatusCode.Error, e.Message); 45 | 46 | Log.Error( 47 | e, 48 | "Failed to transform event from stream {Stream} of type {EventType}", 49 | context.OriginalEvent.EventDetails.Stream, 50 | context.OriginalEvent.EventDetails.EventType 51 | ); 52 | 53 | throw; 54 | } 55 | } 56 | } 57 | 58 | public void Probe(ProbeContext context) => context.Add("eventTransform", transform); 59 | 60 | static BaseProposedEvent TransformMeta(BaseOriginalEvent originalEvent) 61 | => originalEvent switch { 62 | StreamDeletedOriginalEvent deleted => 63 | new ProposedDeleteStream( 64 | deleted.EventDetails, 65 | deleted.LogPosition, 66 | deleted.SequenceNumber 67 | ), 68 | StreamMetadataOriginalEvent meta => 69 | new ProposedMetaEvent( 70 | meta.EventDetails, 71 | meta.Data, 72 | meta.LogPosition, 73 | meta.SequenceNumber 74 | ), 75 | IgnoredOriginalEvent ignored => new IgnoredEvent( 76 | ignored.EventDetails, 77 | ignored.LogPosition, 78 | ignored.SequenceNumber 79 | ), 80 | _ => throw new InvalidOperationException("Unknown original event type") 81 | }; 82 | } 83 | 84 | public class TransformSpecification(TransformEvent transform) : IPipeSpecification { 85 | public void Apply(IPipeBuilder builder) 86 | => builder.AddFilter(new TransformFilter(transform)); 87 | 88 | public IEnumerable Validate() { 89 | yield return this.Success("filter"); 90 | } 91 | } 92 | 93 | public static class TransformPipeExtensions { 94 | public static void UseEventTransform(this IPipeConfigurator configurator, TransformEvent transformEvent) 95 | => configurator.AddPipeSpecification(new TransformSpecification(transformEvent)); 96 | } 97 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.Js/JsTransform.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | using Kurrent.Replicator.Shared.Logging; 3 | using Jint; 4 | using Jint.Native; 5 | using Jint.Native.Json; 6 | using JsonSerializer = System.Text.Json.JsonSerializer; 7 | 8 | // ReSharper disable NotAccessedPositionalProperty.Local 9 | 10 | namespace Kurrent.Replicator.Js; 11 | 12 | public class JsTransform(string jsFunc) { 13 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 14 | 15 | readonly TypedJsFunction _function = new(jsFunc, "transform", HandleResponse); 16 | 17 | static TransformedEvent? HandleResponse(JsValue? result, TransformEvent original) { 18 | if (result == null || result.IsUndefined()) { 19 | Log.Debug("Got empty response, ignoring"); 20 | 21 | return null; 22 | } 23 | 24 | var obj = result.AsObject(); 25 | 26 | if (!TryGetString("Stream", true, out var stream) || 27 | string.IsNullOrWhiteSpace(stream) || 28 | !TryGetString("EventType", true, out var eventType) || 29 | string.IsNullOrWhiteSpace(eventType)) 30 | return null; 31 | 32 | var data = GetSerializedObject("Data"); 33 | 34 | if (data == null) return null; 35 | 36 | var meta = GetSerializedObject("Meta"); 37 | 38 | return new(stream, eventType, data, meta); 39 | 40 | byte[]? GetSerializedObject(string propName) { 41 | var candidate = obj.Get(propName); 42 | 43 | if (candidate == JsValue.Undefined || !candidate.IsObject()) { 44 | return null; 45 | } 46 | 47 | return JsonSerializer.SerializeToUtf8Bytes(candidate.ToObject()); 48 | } 49 | 50 | bool TryGetString(string propName, bool log, out string value) { 51 | var candidate = obj.Get(propName); 52 | 53 | if (candidate == JsValue.Undefined || !candidate.IsString()) { 54 | if (log) Log.Debug("Transformed object property {Prop} is null or not a string", propName); 55 | value = string.Empty; 56 | 57 | return false; 58 | } 59 | 60 | value = candidate.AsString(); 61 | 62 | return true; 63 | } 64 | } 65 | 66 | public ValueTask Transform(OriginalEvent original, CancellationToken cancellationToken) { 67 | var parser = new JsonParser(_function.Engine); 68 | 69 | var result = _function.Execute( 70 | new( 71 | original.Created, 72 | original.EventDetails.Stream, 73 | original.EventDetails.EventType, 74 | parser.Parse(original.Data.AsUtf8String()), 75 | ParseMeta() 76 | ) 77 | ); 78 | 79 | BaseProposedEvent evt = result == null 80 | ? new IgnoredEvent(original.EventDetails, original.LogPosition, original.SequenceNumber) 81 | : new ProposedEvent( 82 | original.EventDetails with { Stream = result.Stream, EventType = result.EventType }, 83 | result.Data, 84 | result.Meta, 85 | original.LogPosition, 86 | original.SequenceNumber 87 | ); 88 | 89 | return new(evt); 90 | 91 | JsValue? ParseMeta() { 92 | if (original.Metadata == null) return null; 93 | 94 | var metaString = original.Metadata.AsUtf8String(); 95 | 96 | try { 97 | return metaString.Length == 0 ? null : parser.Parse(metaString); 98 | } catch (Exception) { 99 | return null; 100 | } 101 | } 102 | } 103 | 104 | record TransformEvent(DateTimeOffset Created, string Stream, string EventType, JsValue? Data, JsValue? Meta); 105 | 106 | record TransformedEvent(string Stream, string EventType, byte[] Data, byte[]? Meta); 107 | } 108 | -------------------------------------------------------------------------------- /docs/src/assets/kurrent-logo-white.svg: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.KurrentDb/Internals/StreamAclJsonConverter.cs: -------------------------------------------------------------------------------- 1 | using System.Text.Json; 2 | using System.Text.Json.Serialization; 3 | 4 | namespace Kurrent.Replicator.KurrentDb.Internals; 5 | 6 | class StreamAclJsonConverter : JsonConverter { 7 | public static readonly StreamAclJsonConverter Instance = new(); 8 | 9 | public override StreamAcl Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options) { 10 | string[]? read = null, 11 | write = null, 12 | delete = null, 13 | metaRead = null, 14 | metaWrite = null; 15 | 16 | if (reader.TokenType != JsonTokenType.StartObject) { 17 | throw new InvalidOperationException(); 18 | } 19 | 20 | while (reader.Read()) { 21 | if (reader.TokenType == JsonTokenType.EndObject) { 22 | break; 23 | } 24 | 25 | if (reader.TokenType != JsonTokenType.PropertyName) { 26 | throw new InvalidOperationException(); 27 | } 28 | 29 | switch (reader.GetString()) { 30 | case SystemMetadata.AclRead: 31 | read = ReadRoles(ref reader); 32 | 33 | break; 34 | case SystemMetadata.AclWrite: 35 | write = ReadRoles(ref reader); 36 | 37 | break; 38 | case SystemMetadata.AclDelete: 39 | delete = ReadRoles(ref reader); 40 | 41 | break; 42 | case SystemMetadata.AclMetaRead: 43 | metaRead = ReadRoles(ref reader); 44 | 45 | break; 46 | case SystemMetadata.AclMetaWrite: 47 | metaWrite = ReadRoles(ref reader); 48 | 49 | break; 50 | } 51 | } 52 | 53 | return new(read, write, delete, metaRead, metaWrite); 54 | } 55 | 56 | static string[]? ReadRoles(ref Utf8JsonReader reader) { 57 | if (!reader.Read()) { 58 | throw new InvalidOperationException(); 59 | } 60 | 61 | if (reader.TokenType == JsonTokenType.Null) { 62 | return null; 63 | } 64 | 65 | if (reader.TokenType == JsonTokenType.String) { 66 | return new[] { reader.GetString()! }; 67 | } 68 | 69 | if (reader.TokenType != JsonTokenType.StartArray) { 70 | throw new InvalidOperationException(); 71 | } 72 | 73 | var roles = new List(); 74 | 75 | while (reader.Read()) { 76 | if (reader.TokenType == JsonTokenType.EndArray) { 77 | return roles.Count == 0 ? Array.Empty() : roles.ToArray(); 78 | } 79 | 80 | if (reader.TokenType != JsonTokenType.String) { 81 | throw new InvalidOperationException(); 82 | } 83 | 84 | roles.Add(reader.GetString()!); 85 | } 86 | 87 | return roles.ToArray(); 88 | } 89 | 90 | public override void Write(Utf8JsonWriter writer, StreamAcl value, JsonSerializerOptions options) { 91 | writer.WriteStartObject(); 92 | 93 | WriteRoles(writer, SystemMetadata.AclRead, value.ReadRoles); 94 | WriteRoles(writer, SystemMetadata.AclWrite, value.WriteRoles); 95 | WriteRoles(writer, SystemMetadata.AclDelete, value.DeleteRoles); 96 | WriteRoles(writer, SystemMetadata.AclMetaRead, value.MetaReadRoles); 97 | WriteRoles(writer, SystemMetadata.AclMetaWrite, value.MetaWriteRoles); 98 | 99 | writer.WriteEndObject(); 100 | } 101 | 102 | static void WriteRoles(Utf8JsonWriter writer, string name, string[]? roles) { 103 | if (roles == null) { 104 | return; 105 | } 106 | 107 | writer.WritePropertyName(name); 108 | writer.WriteStartArray(); 109 | 110 | foreach (var role in roles) { 111 | writer.WriteStringValue(role); 112 | } 113 | 114 | writer.WriteEndArray(); 115 | } 116 | } 117 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.EventStore/TcpEventWriter.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | using Kurrent.Replicator.Shared.Logging; 3 | using Kurrent.Replicator.Shared.Observe; 4 | using Ubiquitous.Metrics; 5 | using StreamMetadata = EventStore.ClientAPI.StreamMetadata; 6 | 7 | // ReSharper disable SuggestBaseTypeForParameter 8 | namespace Kurrent.Replicator.EventStore; 9 | 10 | public class TcpEventWriter(IEventStoreConnection connection) : IEventWriter { 11 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 12 | 13 | public Task Start() => connection.ConnectAsync(); 14 | 15 | public Task WriteEvent(BaseProposedEvent proposedEvent, CancellationToken cancellationToken) { 16 | var task = proposedEvent switch { 17 | ProposedEvent p => Append(p), 18 | ProposedMetaEvent meta => SetMeta(meta), 19 | ProposedDeleteStream delete => Delete(delete), 20 | IgnoredEvent _ => Task.FromResult(-1L), 21 | _ => throw new InvalidOperationException("Unknown proposed event type") 22 | }; 23 | 24 | return Metrics.Measure(() => task, ReplicationMetrics.WritesHistogram, ReplicationMetrics.WriteErrorsCount); 25 | 26 | async Task Append(ProposedEvent p) { 27 | if (Log.IsDebugEnabled()) { 28 | Log.Debug( 29 | "TCP: Write event with id {Id} of type {Type} to {Stream} with original position {Position}", 30 | proposedEvent.EventDetails.EventId, 31 | proposedEvent.EventDetails.EventType, 32 | proposedEvent.EventDetails.Stream, 33 | proposedEvent.SourceLogPosition.EventPosition 34 | ); 35 | } 36 | 37 | var result = await connection.AppendToStreamAsync(p.EventDetails.Stream, ExpectedVersion.Any, Map(p)).ConfigureAwait(false); 38 | 39 | return result.LogPosition.CommitPosition; 40 | } 41 | 42 | async Task SetMeta(ProposedMetaEvent meta) { 43 | if (Log.IsDebugEnabled()) 44 | Log.Debug( 45 | "TCP: Setting metadata to {Stream} with original position {Position}", 46 | proposedEvent.EventDetails.Stream, 47 | proposedEvent.SourceLogPosition.EventPosition 48 | ); 49 | 50 | var result = await connection.SetStreamMetadataAsync( 51 | meta.EventDetails.Stream, 52 | ExpectedVersion.Any, 53 | StreamMetadata.Create( 54 | meta.Data.MaxCount, 55 | meta.Data.MaxAge, 56 | meta.Data.TruncateBefore, 57 | meta.Data.CacheControl, 58 | new( 59 | meta.Data.StreamAcl?.ReadRoles, 60 | meta.Data.StreamAcl?.WriteRoles, 61 | meta.Data.StreamAcl?.DeleteRoles, 62 | meta.Data.StreamAcl?.MetaReadRoles, 63 | meta.Data.StreamAcl?.MetaWriteRoles 64 | ) 65 | ) 66 | ) 67 | .ConfigureAwait(false); 68 | 69 | return result.LogPosition.CommitPosition; 70 | } 71 | 72 | async Task Delete(ProposedDeleteStream delete) { 73 | if (Log.IsDebugEnabled()) { 74 | Log.Debug( 75 | "TCP: Deleting stream {Stream} with original position {Position}", 76 | proposedEvent.EventDetails.Stream, 77 | proposedEvent.SourceLogPosition.EventPosition 78 | ); 79 | } 80 | 81 | var result = await connection.DeleteStreamAsync(delete.EventDetails.Stream, ExpectedVersion.Any).ConfigureAwait(false); 82 | 83 | return result.LogPosition.CommitPosition; 84 | } 85 | 86 | static EventData Map(ProposedEvent evt) => new( 87 | evt.EventDetails.EventId, 88 | evt.EventDetails.EventType, 89 | evt.EventDetails.ContentType == ContentTypes.Json, 90 | evt.Data, 91 | evt.Metadata 92 | ); 93 | } 94 | } 95 | -------------------------------------------------------------------------------- /src/Kurrent.Replicator.KurrentDb/GrpcEventWriter.cs: -------------------------------------------------------------------------------- 1 | using Kurrent.Replicator.Shared.Contracts; 2 | using Kurrent.Replicator.Shared.Logging; 3 | using Kurrent.Replicator.Shared.Observe; 4 | using Ubiquitous.Metrics; 5 | using StreamAcl = EventStore.Client.StreamAcl; 6 | 7 | namespace Kurrent.Replicator.KurrentDb; 8 | 9 | public class GrpcEventWriter(EventStoreClient client) : IEventWriter { 10 | static readonly ILog Log = LogProvider.GetCurrentClassLogger(); 11 | 12 | public Task Start() => Task.CompletedTask; 13 | 14 | public async Task WriteEvent(BaseProposedEvent proposedEvent, CancellationToken cancellationToken) { 15 | var task = proposedEvent switch { 16 | ProposedEvent p => AppendEvent(p), 17 | ProposedDeleteStream delete => DeleteStream(delete.EventDetails.Stream), 18 | ProposedMetaEvent meta => SetStreamMeta(meta), 19 | IgnoredEvent _ => Task.FromResult(-1L), 20 | _ => throw new InvalidOperationException("Unknown proposed event type") 21 | }; 22 | 23 | return 24 | task.IsCompleted 25 | ? task.Result 26 | : await Metrics.Measure(() => task, ReplicationMetrics.WritesHistogram, ReplicationMetrics.WriteErrorsCount).ConfigureAwait(false); 27 | 28 | async Task AppendEvent(ProposedEvent p) { 29 | if (Log.IsDebugEnabled()) 30 | Log.Debug( 31 | "gRPC: Write event with id {Id} of type {Type} to {Stream} with original position {Position}", 32 | p.EventDetails.EventId, 33 | p.EventDetails.EventType, 34 | p.EventDetails.Stream, 35 | p.SourceLogPosition.EventPosition 36 | ); 37 | 38 | var result = await client.AppendToStreamAsync(proposedEvent.EventDetails.Stream, StreamState.Any, [Map(p)], cancellationToken: cancellationToken).ConfigureAwait(false); 39 | 40 | return (long)result.LogPosition.CommitPosition; 41 | } 42 | 43 | async Task DeleteStream(string stream) { 44 | if (Log.IsDebugEnabled()) 45 | Log.Debug("Deleting stream {Stream}", stream); 46 | 47 | var result = await client.DeleteAsync(stream, StreamState.Any, cancellationToken: cancellationToken).ConfigureAwait(false); 48 | 49 | return (long)result.LogPosition.CommitPosition; 50 | } 51 | 52 | async Task SetStreamMeta(ProposedMetaEvent meta) { 53 | if (Log.IsDebugEnabled()) 54 | Log.Debug("Setting meta for {Stream} to {Meta}", meta.EventDetails.Stream, meta); 55 | 56 | var result = await client.SetStreamMetadataAsync( 57 | meta.EventDetails.Stream, 58 | StreamState.Any, 59 | new( 60 | meta.Data.MaxCount, 61 | meta.Data.MaxAge, 62 | ValueOrNull(meta.Data.TruncateBefore, x => new StreamPosition((ulong)x!)), 63 | meta.Data.CacheControl, 64 | ValueOrNull( 65 | meta.Data.StreamAcl, 66 | x => 67 | new StreamAcl( 68 | x.ReadRoles, 69 | x.WriteRoles, 70 | x.DeleteRoles, 71 | x.MetaReadRoles, 72 | x.MetaWriteRoles 73 | ) 74 | ) 75 | ), 76 | cancellationToken: cancellationToken 77 | ) 78 | .ConfigureAwait(false); 79 | 80 | return (long)result.LogPosition.CommitPosition; 81 | } 82 | } 83 | 84 | static EventData Map(ProposedEvent evt) 85 | => new( 86 | Uuid.FromGuid(evt.EventDetails.EventId), 87 | evt.EventDetails.EventType, 88 | evt.Data, 89 | evt.Metadata, 90 | evt.EventDetails.ContentType 91 | ); 92 | 93 | static T? ValueOrNull(T1? source, Func transform) 94 | => source == null ? default : transform(source); 95 | } 96 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | # Kurrent License v1 2 | 3 | Copyright (c) 2011-2025, Kurrent, Inc. All rights reserved. 4 | 5 | ### Acceptance 6 | 7 | By using the software, you agree to all of the terms and conditions below. 8 | 9 | ### Copyright License 10 | 11 | The licensor grants you a non-exclusive, royalty-free, worldwide, non-sublicensable, non-transferable license to use, copy, distribute, make available, and prepare derivative works of the software, in each case subject to the limitations and conditions below. 12 | 13 | ### Limitations 14 | 15 | You may not provide the software to third parties as a hosted or managed service, where the service provides users with access to any substantial set of the features or functionality of the software. 16 | 17 | Unless authorized in writing by the licensor, you may not move, change, disable, interfere with, or circumvent the license mechanisms in the software, and you may not remove or obscure any functionality in the software that is protected by the license mechanisms. 18 | 19 | You may not alter, remove, or obscure any licensing, copyright, or other notices of the licensor in the software. Any use of the licensor’s trademarks is subject to applicable law. 20 | 21 | ### Patents 22 | 23 | The licensor grants you a license, under any patent claims the licensor can license, or becomes able to license, to make, have made, use, sell, offer for sale, import and have imported the software, in each case subject to the limitations and conditions in this license. This license does not cover any patent claims that you cause to be infringed by modifications or additions to the software. If you or your company make any written claim that the software infringes or contributes to infringement of any patent, your patent license for the software granted under these terms ends immediately. If your company makes such a claim, your patent license ends immediately for work on behalf of your company. 24 | 25 | ### Notices 26 | 27 | You must ensure that anyone who gets a copy of any part of the software from you also gets a copy of these terms. 28 | 29 | If you modify the software, you must include in any modified copies of the software prominent notices stating that you have modified the software. 30 | 31 | ### No Other Rights 32 | 33 | These terms do not imply any licenses other than those expressly granted in these terms. 34 | 35 | ### Termination 36 | 37 | If you use the software in violation of these terms, such use is not licensed, and your licenses will automatically terminate. If the licensor provides you with a notice of your violation, and you cease all violation of this license no later than 30 days after you receive that notice, your licenses will be reinstated retroactively. However, if you violate these terms after such reinstatement, any additional violation of these terms will cause your licenses to terminate automatically and permanently. 38 | 39 | ### No Liability 40 | 41 | ***As far as the law allows, the software comes as is, without any warranty or condition, and the licensor will not be liable to you for any damages arising out of these terms or the use or nature of the software, under any kind of legal claim.*** 42 | 43 | ### Definitions 44 | 45 | The **licensor** is the entity offering these terms, and the **software** is the software the licensor makes available under these terms, including any portion of it. 46 | 47 | **licensing mechanisms** refers to functionality that restricts use of the software based on whether you possess a valid license key, including functionality to validate license keys and audit usage of the software to ensure license compliance. 48 | 49 | **you** refers to the individual or entity agreeing to these terms. 50 | 51 | **your company** is any legal entity, sole proprietorship, or other kind of organization that you work for, plus all organizations that have control over, are under the control of, or are under common control with that organization. **control** means ownership of substantially all the assets of an entity, or the power to direct its management and policies by vote, contract, or otherwise. Control can be direct or indirect. 52 | 53 | **your licenses** are all the licenses granted to you for the software under these terms. 54 | 55 | **use** means anything you do with the software requiring one of your licenses. 56 | 57 | **trademark** means trademarks, service marks, and similar rights. 58 | -------------------------------------------------------------------------------- /docs/src/content/docs/intro/concepts.mdx: -------------------------------------------------------------------------------- 1 | --- 2 | title: "Concepts" 3 | description: Find out some basic concepts of Replicator. 4 | sidebar: 5 | order: 2 6 | --- 7 | 8 | import { Steps } from '@astrojs/starlight/components'; 9 | 10 | Replicator is designed to copy events from one place to another, which sounds like a relatively simple task. Still, there are some concepts you need to understand before using the tool. 11 | 12 | ## Data flow 13 | 14 | Replicator implements the following data flow: 15 | 16 | 17 | 1. Source and reader 18 | > A source is where Replicator reads events that are replicated to a sink. 19 | 2. Filter 20 | > Filters allow you to cover cases like preventing some obsolete events from being replicated, or splitting one source to two targets. 21 | 3. Transform 22 | > You might want to remove some of it using filters, but you might also want to keep the data in a different format. 23 | 4. Sink and writers 24 | > A sink is a place where Replicator writes events. The sink has one or more writers. 25 | 5. Checkpoint 26 | > Checkpointing is used to keep track of the last processed event in the source. 27 | 28 | 29 | ## Source and reader 30 | 31 | A source is a place where Replicator reads events that are replicated to a sink. Reader is an adapter for the infrastructure, where you want to copy events _from_. The _reader_ reads from a _source_. 32 | 33 | Currently, we support readers for KurrentDB/EventStoreDB, using legacy TCP and current gRPC-based protocols. Each reader type requires its own configuration, which is usually just a connection string, specific to each reader type. 34 | 35 | The reader always reads events in sequence, but all the readers support batched reads. 36 | 37 | There is only one reader per running Replicator instance. 38 | 39 | ## Sink and writers 40 | 41 | Reader is an adapter for the infrastructure, where you want to copy events _to_. The _sink_ has one or more _writers_. By using multiple writers, one sink can improve performance by parallelizing writes. 42 | 43 | When using one writer for a sink, the order of events in the target remains exactly the same as it was in the source. 44 | 45 | When using more than one writer, the global order of events in the source cannot be guaranteed. However, multiple writers also enable partitioning. The default partition key is the stream name, which guarantees the order of events in each stream. 46 | 47 | You can only have one sink per running Replicator instance, but it might have multiple writers. 48 | 49 | ## Checkpoint 50 | 51 | A running Replicator instance progresses linearly over a given stream of events, so it knows at any time, which events were already processed. As the process might be shut down for different reasons, it needs to maintain the last processed event position, so in case of restart, Replicator will start from there, and not from the very beginning. This way, you don't get duplicated events in the sink, and you can be sure that the replication process will eventually be completed. 52 | 53 | The location of the last processed event in the source is known as _checkpoint_. Replicator supports storing the checkpoint in [different stores](../../features/checkpoints/). If you want to run the replication again, from the same source, using the same Replicator instance, you need to delete the checkpoint file. 54 | 55 | ## Filters 56 | 57 | As you might want to ignore some events during replication, Replicator supports different [filters](../../features/filters/). Filters allow you to cover cases like preventing some obsolete events from being replicated, or splitting one source to two targets. In the latter case, you can run two Replicator instances with different filters, so events will be distributed to different sinks. 58 | 59 | ## Transforms 60 | 61 | After being in production for a while, most systems accumulate legacy data. You might want to remove some of it using filters, but you might also want to keep the data in a different format. Typical scenarios include evolution of event schema, missing fields, incorrect data format, oversharing (sensitive unprotected information), etc. 62 | 63 | These cases can be handler by using [transforms](../../features/transforms/), which allow you to change any part of the event that comes from the source, before writing it to the sink. 64 | 65 | 66 | --------------------------------------------------------------------------------