6 | This tutorial guides you through a distributed order management application 7 | based on eventuate by 8 | using distributed and 9 | event-sourced 10 | order-actors. It 11 | demonstrates the most important parts of such an application and how to build 12 | one yourself. The 13 | example 14 | is actually taken from the eventuate project itself 15 | and only modified slightly to allow better integration into an activator template. 16 |
17 | The application 18 | runs an OrderManager on several nodes. Each one is able to accept 19 |
20 |27 | The changes are replicated to all connected nodes. Changing the same order on 28 | multiple nodes concurrently is considered a conflict. In that case 29 | multiple versions of this order are maintained until one is picked interactively 30 | to resolve the conflict. 31 |
32 |
37 | The entry into the application is
38 | OrderLocation.
39 | It starts the actor-system and the actors
40 | OrderManager
as well as OrderView
in the
41 | akka-typical manner. What makes it an eventuate-based application that supports distributed
42 | deployment is the
43 | ReplicationEndpoint.
44 | This takes care of replicating changes to all connected ReplicationEndpoint
s.
45 | A ReplicationEndpoint
can either be setup programmatically or (as in this case) by
46 | configuration. For this on each node
47 | akka-remoting
48 | host and port (under which this RelicationEndpoint
is available) needs to be defined:
49 |
51 | akka.remote.netty.tcp.hostname = "127.0.0.1"
52 | akka.remote.netty.tcp.port=2554
53 |
54 | 55 | as well as an endpoint-id and the replication-partners on the other nodes: 56 |
57 |
58 | eventuate {
59 | endpoint.id = "C"
60 | endpoint.connections = ["127.0.0.1:2552", "127.0.0.1:2553", "127.0.0.1:2555"]
61 | }
62 |
63 |
64 | So in this case ReplicationEndpoint
C listens on 127.0.0.1:2554 and connects to
65 | ReplicationEndpoint
s on other nodes listening on
66 | 127.0.0.1:2552, 127.0.0.1:2553 and 127.0.0.1:2555.
67 |
73 | At the heart of the application is the
74 | OrderManager.
75 | It is implemented as
76 | EventsourcedView
77 | to better illustrate how orders are created in the distributed system. For the moment
78 | it is enough to assume that it is a plain actor and its onCommand
method is actually the
79 | receive
method of an actor.
80 |
81 | Its main purpose is to maintain a Map
of
82 | OrderActors
83 | by id each one representing a specific
84 | order and dispatch OrderCommand
s or a Resolve
-command to them
85 | (to modify the order) or a SaveSnapshot
-command for taking a
86 | snapshot
87 | of the internal state of an order.
88 | Additionally it allows to retrieve the current state of all orders (currently in memory).
89 |
95 | A single order is represented by an
96 | OrderActor.
97 | It is implemented as an
98 | EventsourcedActor.
99 | It represents a special case of an
100 | aggregate root
101 | and that is why it defines
102 | the member aggregateId
. This has to be an application wide unique id.
103 | Even eventsourced aggregate roots of different type must not share the same ids.
104 | In addition to this it defines the members id
and eventLog
.
105 | id
must be a globally unique id, so all other EventsourcedActor
s or even
106 | EventsourcedView
s in the entire distributed setup must not share this id. Here it
107 | is built as combination of replicaId
that
108 | identifies a node or location and orderId
. The idea is that there are several OrderActor
-instances
109 | with the same orderId
(respectively aggregateId
)
110 | at different nodes (and thus with different replicaId
) and state-changes to one of them are replicated to the
111 | others so that they are eventually all in the same state.
112 | For this an
113 | EventsourcedActor
defines the two methods:
114 |
onCommand
onEvent
120 | The purpose of onCommand
is to process commands (of type
121 | OrderCommand
) that sre sent to change
122 | the state of the order (like AddOrderItem
) by
123 |
persist(event)
)
129 | So for example if an AddOrderItem
-command can be accepted the
130 | onCommand
-handler emits an OrderItemAdded
-event.
131 | Before this event is processed by the onEvent
-handler to
132 | perform the requested state-change, it is persisted by eventuate to the
133 | event-log (represented by the member eventLog
).
134 |
136 | This reflects the core principle of eventsourced applications. Instead of
137 | persisting the current state, state-changes are persisted. When the application (or just the actor)
138 | is restarted all persisted events are replayed (i.e. the onEvent
-handler
139 | is called for each of them) to reconstruct the state as before.
140 |
142 | Note that onCommand
and onEvent
are both called during normal
143 | actor message dispatch so they can safely access and modify an actor's mutable state,
144 | however as persisting is done asynchronously they are not invoked through a single message.
145 | Nonetheless eventuate guarantees by default that no new command slips between an
146 | onCommand
-call and the onEvent
-calls for each emitted/persisted
147 | event by stashing commands that arrive in the meantime. This guarantee can be
148 | relaxed
149 | for performance reasons.
150 |
152 | eventuate provides two additional important features: 153 |
154 |onEvent
-handler of all affected eventsourced-actors on all nodes. In this
157 | case the affected eventsourced actor is an (already active) OrderActor with the same aggregate-id.onEvent
-handler
162 | before the other event168 | Another typical element of eventsourced application are so called 169 | eventsourced views. 170 | These are actors that consume events (to build up internal state) but cannot emit any. They 171 | can be used to implement the query-side of 172 | a CQRS based application. 173 |
174 |
175 | The
176 | OrderView
177 | implements a simple example of such an
178 | EventsourcedView.
179 | As it does not define aggregateId
it consumes all events that are written to its
180 | eventLog
either directly by eventsourced actors on the same node or through replication.
181 |
183 | Here the OrderView
simply counts for each order its updates
184 | (i.e. the emitted OrderEvent
s) and allows to query for this number by order-id.
185 |
190 | On the Run-tab you can start an example-setup
191 | with two locations (C and D)
192 | running in a single JVM in action (main file: sample.eventuate.OrderBot
).
193 | A bot will send commands
194 | alternating to the OrderManager
s of both locations.
195 |
197 | When you start it the first time you will see something like this in the log-output: 198 |
199 |
200 | ( 1) 15:40:16.965 CreateOrderAction$: ------- Send CreateOrder to location-D (1/9) -------
201 | ( 1) 15:40:16.966 OrderManager: [D]: Process command: CreateOrder(0762ee15-c570-4c57-8b03-bf757418df9f)
202 | ( 2) 15:40:16.966 OrderManager: [D]: Create OrderActor for 0762ee15-c570-4c57-8b03-bf757418df9f
203 | ( 3) 15:40:16.971 OrderActor: [D]: OrderCreated: [0762ee15-c570-4c57-8b03-bf757418df9f] items= cancelled=false
204 | ( 5) 15:40:16.978 OrderManager: [C]: Create OrderActor for 0762ee15-c570-4c57-8b03-bf757418df9f
205 | ( 6) 15:40:16.981 OrderActor: [C]: Initialized from Log: [0762ee15-c570-4c57-8b03-bf757418df9f] items= cancelled=false
206 | ( 7) 15:40:19.978 AddOrderItemAction$: ------- Send AddOrderItem to location-C (2/9) --------
207 | ( 8) 15:40:19.978 OrderManager: [C]: Process command: AddOrderItem(0762ee15-c570-4c57-8b03-bf757418df9f,Fqmtv)
208 | ( 9) 15:40:19.984 OrderActor: [C]: OrderItemAdded: [0762ee15-c570-4c57-8b03-bf757418df9f] items=Fqmtv cancelled=false
209 | (10) 15:40:19.990 OrderActor: [D]: OrderItemAdded: [0762ee15-c570-4c57-8b03-bf757418df9f] items=Fqmtv cancelled=false
210 |
211 | 212 | This shows that: 213 |
214 |CreateOrder
-command is sent to D.OrderManager
starts the OrderActor
-accordingly.OrderActor
emits the OrderCreated
event.OrderManager
sees it and eagerly starts a corresponding OrderActor
.OrderActor
gets initialized from C's event log
221 | that already contains the replicated event and thus ends up in the
222 | same state as D's order.AddOrderItem
-command for this order is sent to C.OrderManager
dispatches the command to the OrderActor
,OrderItemAdded
-eventOrderActor
227 | can consume the same event to bring itself into the same state as C's order.234 | Now that you have seen the example application you may wonder what happens if 235 | an item is added to the same order on both locations C and D simultaneously. 236 | By tracking causality of events eventuate can detect concurrent events 237 | and thus potentially conflicting updates. 238 |
239 |240 | You can actually try this out by commenting the sleep-statement in 241 | OrderBot 242 | and running the example again. As the activator UI gives access to a 243 | limited number of log-lines only, it makes sense to test this using 244 | sbt started from a terminal. For this execute: 245 |
246 |sbt "runMain sample.eventuate.OrderBot"
247 |
248 | Depending on your hardware you may need a couple of tries or it might even make sense
249 | to increase the total number of commands sent to the application (sample.eventuate.Action#total
),
250 | but eventually you should find something like this in the output:
251 |
253 | 16:25:59.248 OrderActor: [C]: OrderItemAdded: Conflict:
254 | - version 0: [5f789bc3-9a20-4741-a6c6-dd87cfe36042] items=XN7iM,3hZQO cancelled=false
255 | - version 1: [5f789bc3-9a20-4741-a6c6-dd87cfe36042] items=XN7iM,T6dEy cancelled=false
256 | ...
257 | 16:25:59.252 OrderActor: [D]: OrderItemAdded: Conflict:
258 | - version 0: [5f789bc3-9a20-4741-a6c6-dd87cfe36042] items=XN7iM,T6dEy cancelled=false
259 | - version 1: [5f789bc3-9a20-4741-a6c6-dd87cfe36042] items=XN7iM,3hZQO cancelled=false
260 |
261 | 262 | In this case the bot added to the order with id 5f789bc3... (which already 263 | contained item XN7iM) simultaneously the items 3hZQO and T6dEy. 264 | As the bot actually runs purely sequential, simultaneously in this case means 265 | the items were added on nodes C and D before the corresponding event was replicated 266 | to the other location. 267 |
268 |269 | By using 270 | ConcurrentVersions 271 | an actor is able to maintain a 272 | tree of conflicting versions. 273 | These conflicts can be resolved by a selecting a winner-version either 274 | automatically 275 | or 276 | interactively. 277 | See the section on resolving conflicts in the example application for an 278 | interactive example. 279 |
280 |285 | While seeing the replication and conflict detection in a log-file is nice, you 286 | would typically want to write automated tests to check correct behaviour of 287 | the application in these circumstances. 288 |
289 |290 | For this akka comes with the amazing 291 | multi-node-testing-toolkit 292 | and 293 | OrderSpec 294 | uses exactly this to verify that the application behaves as expected 295 | when distributed to two JVMs. 296 |
297 |298 | A multi-jvm test basically consists out of two parts: 299 |
300 |
307 | The MultiNodeConfig
TwoNodesReplicationConfig
defines two roles
308 | nodeA and nodeB which are equivalent
309 | to nodes in the test. Additionally it configures replication properties that
310 | ensure better timing for testing than the default ones.
311 |
313 | The actual test-code can be found in OrderSpec
. Each test uses the method
314 | withTwoOrderManagers
(implementing the
315 | loan-fixture-pattern)
316 | to get a MultiNodeSpec
-reference that sets up the replication for the two nodes
317 | (running in two JVMs!) and starts an
318 | OrderManager
(on each node). In addition this reference comes with some convenience methods that ease
319 | testing the OrderManager
. The actual test-code comes in the withTwoOrderManagers
-block and is
320 | executed on both JVMs simultaneously.
321 |
324 | The first test simply sends the CreateOder
-command to the local OrderManager
(on each node)
325 |
327 | executeCommand(CreateOrder(newOrderId))
328 |
329 |
330 | waits for both OrderCreated
-events (on each node)
331 |
333 | val emittedEvents = listener.expectMsgAllClassOf(
334 | List.fill(2)(classOf[OrderCreated]): _*)
335 |
336 |
337 | and verifies (on each node) that the OrderManager
s contain both orders
338 |
340 | allOrders shouldBe emittedEvents.map(toOrder).toSet
341 |
342 | 344 | The second test creates the order only on node A 345 |
346 |
347 | runOn(config.nodeA)(executeCommand(CreateOrder(newOrderId)))
348 |
349 | 350 | waits for the order to be created (on each node) 351 |
352 |
353 | val OrderCreated(orderId, _) = waitFor(orderCreated)
354 |
355 | 356 | and adds an item to the order only on node B 357 |
358 |
359 | runOn(config.nodeB)(executeCommand(AddOrderItem(orderId, "item")))
360 |
361 |
362 | At the end the OrderManager
(on each node) must contain the
363 | same order.
364 |
367 | The third test finally provokes a conflict by 368 |
369 |
376 | Once the connection is reestablished the added items are replicated to the other node
377 | and (since they took place concurrently) result in a conflict on each node. At the end the
378 | test verifies (on each node) that the OrderManager
contain both versions
379 | of the order.
380 |
386 | To demonstrate interactive conflict resolution one can use the 387 | OrderExample 388 | started from a terminal. For this execute in the root-directory of the project (Linux and Mac only): 389 |
390 |
391 | ./example A B C
392 |
393 | 394 | to start three nodes in three terminal windows on your machine. With 395 |
396 |
397 | create Order1
398 |
399 | 400 | in any of the windows you can create an order and see that it gets replicated to all nodes. 401 | Once that is done you can add an item in another window: 402 |
403 |
404 | add Order1 Item1
405 |
406 | 407 | Now lets create a partition by stopping node C (Ctrl-C) and add different item in A's and B's 408 | window: 409 |
410 |
411 | (in A:) add Order1 ItemA
412 | (in B:) add Order1 ItemB
413 |
414 | 415 | Restart C (Cursor Up + Enter) to re-enable replication and you should see something as 416 | follows in all windows: 417 |
418 |
419 | 14:17:46.863 OrderActor: [?]: OrderItemAdded: Conflict:
420 | - version 0: [Order1] items=Item1,ItemA cancelled=false
421 | - version 1: [Order1] items=Item1,ItemB cancelled=false
422 |
423 | 424 | The conflict can be resolved with the following command: 425 |
426 |
427 | resolve Order1 0
428 |
429 |
430 | To avoid conflicting resolutions the example application implements the rule that a conflict can
431 | only be resolved on the node that initially created the order. This command sends an
432 | Resolve(Order1, 0)
(defined in the eventuate-library) to the
433 | OrderManager
434 | which in turn forwards it to the corresponding
435 | OrderActor.
436 | When the command can be accepted a corresponding Resolved(Order1, vector-timestamp)
437 | event is emitted and consumed and processed by all nodes.
438 |