Event Sourcing the New State of Distributed Applications
Oct 20th 2020
By Robert Kirchner JR
You can think of application state as a snapshot of the values of all data objects in an application at any given time(ex: At time x, there were y active users).
Traditionally maybe this information would be stored in it's current state in an SQL database.
There are many drawbacks to the mentioned traditional approach. Imagine being in charge of inventory, and the application reports an item needed for a shipment is in a location that is actually empty. With this style of application state, it might be hard to audit the history of the item, and narrow down it's actual location.
Imagine we have an object called Item, that represents an item in an inventory system.
case class Item( id: UUID, name: String, itemNumber: String, description: String, category: Category )
Now lets imagine there is an event in which we create an item:
case class CreateItemEvent( name: String, itemNumber: String, description: String, category: Category )
Now that we've created an item, maybe we want to change it, and we can optionally update anything but the item number:
case class UpdateItemEvent( id: UUID, name: Option[String], description: Option[String], category: Option[Category] )
Instead of storing the current state of items, we could store just the events. We could then derive the state of an item at any given moment in time by replaying all events on that item up until that given moment.
Command and Query Responsibility Segregation
Command and Query Responsibility Segregation(CQRS) is a pattern in which you separate the reading of data, from the writing/updating of data.
CRUD(Create Read Update Delete) is a pattern that defines the four basic operations you can perform on data.
CQRS focuses on separating the read from the create, update and delete.
There are many ways to implement the CQRS pattern, we will focus on one of the most common approaches: separate read and write APIs
APIs are one of the most common ways to power a front-end application
An API is commonly a web service, with defined request/response contracts, that a front-end app can communicate with in order to get and manipulate data.
In this CQRS approach, we would use at least two APIs for an app, one API for fetching data, and another API for changing data.
The two APIs are connected to a common datasource(es). The user app will make requests to query data to the read API, and send requests to manipulate data to the write API.
This pattern works well with and compliments a microservices architecture.
The write API will persist events in the event store, and the read API will replay and fold events into the item state that the user queries for.
The event store is the building block that defines how we will store events, and also replay them and aggregate object state.
At the heart of most event sourcing platforms, is an event stream. You can think of the event stream as a data pipeline.
Events enter the stream commonly through a messaging-like platform, maybe even Apache Kafka. Typically there are stateful stream processing applications, that listen to the stream for particular types of events.
When the stream processor picks up events it cares about, it might write information to a database, or it might manipulate the data and make new events, for a down stream processor to pick up and persist.
Typically when events enter a stream, they are written the stream permanently. They become part of the platform history, so that the state can always be replayed from the origin of the application's platform time. General rule of thumb is that data cannot be overly redundant in a system of this type. The more the better.
Often, data footprint is enlarged in order for the pipeline to process data in the fastest ways possible. Often these patterns are applied to high volume applications that may process millions of events every second.
In conclusion, this is a very high level abstraction of event sourcing. Event sourcing is a very powerful and commonly used high volume applications.
As always, if you want to ask questions, or suggest updates or corrections, leave a comment or contact us.