#19 Microservices Data Management: 7 Important Design Patterns

What are microservices

Microservices – also known as the microservice architecture – is an architectural style that structures an application as a collection of loosely coupled services, which implement business capabilities. The microservice architecture enables the continuous delivery/deployment of large, complex applications. It also enables an organization to evolve its technology stack.

Need of design patterns for Microservices Data Management

 We have decomposed the application in Microservices but Business demand integrated data that’s where we miss monolithic architecture. Since we have addressed some of the major challenges with microservices we introduced some powerful design patterns to address the data management need in microservices architecture.

Design Patterns for Data management

Database Per Service

When we have segregated the responsibilities at the application layer we want to maintain this decoupled nature till the database layer. So, what database architecture we should follow?

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Some business use cases are not independent, they must interact with multiple services either to fetch the data or updated the data owned by other services
• Some business use cases need to join the data owned by multiple services
• Databases must sometimes be replicated and sharded in order to scale
• Every microservice has different data storage requirement. For some services, RDBMS is the best option and for some NOSQL is the right fit

Solution

“Keep each microservice’s persistent data private to that service and accessible only via its API. A service’s transactions only involve its database”

In the above diagram Data, A will be private to Service A and will not never be accessed directly by any service. There are many ways to keep a service’s persistent data private. You not necessarily need to set up a separate database service for all microservices. But you can use below for instance if you are using RDBMS

• Private-tables-per-service – each service owns a set of tables that must only be accessed by that service
• Schema-per-service – each service has a database schema that’s private to that service
• Database-server-per-service – each service has it’s own database server.

Benefits   

• Services are loosely coupled
• Each services are free to use best suited database for their need

Drawbacks

• Implementing business transactions that span multiple services is not straightforward
• Implementing queries that join data that is now in multiple databases is challenging
• Complexity of managing multiple SQL and NoSQL databases

Shared Database

There are use cases when each business cases are so interlinked that you will end up touching most of Microservices in a single call. For systems, we must use a shared database.

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Many use cases require data owned by multiple services
• Use cases need to update data owned by multiple services

Solution

Use a (single) database that is shared by multiple services. Each service freely accesses data owned by other services using local ACID transactions.

Benefits   

• Easy for developer to create a query for required data
• Easy to operate single set of data for ETL & other data management activities

Drawbacks

• Database became monolithic and we have coupled the responsibilities at data layer
• Reduced freedom to use best fit database for each microservices
• Manage the schema switch at run time will slow down system

Saga

As we talked about business cases where we require to get the data owned by multiple services also there are transactions that need to update the data owned by multiple microservices. This pattern will address the problem related to transactions that span across multiple microservices.

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Use cases need to update data owned by multiple services

Solution

Each business transaction that span across multiple services should be implemented as Saga. Saga is basically a sequence of local transactions where each transaction updates the database and trigger a event or message to execute the next transaction. In this series we also implement the fallback mechanism linked to each transaction so that if local transaction fails saga can execute sequence of compensating transaction to rollback the changes made by earlier transactions.

There are two ways of coordination sagas:

• Choreography – each local transaction publishes domain events that trigger local transactions in other services
• Orchestration – an orchestrator (object) tells the participants what local transactions to execute

Example – Choreography-based saga

Example – Orchestration-based saga

Benefits   

• It enables an application to maintain data consistency across multiple services without using distributed transactions
• Resilient Architecture with high performance

Drawbacks

• The programming model is more complex
• In order to be reliable, a service must atomically update its database and publish a message/event

API Composition

API composition is designed to address the problem that comes as an outcome of implementing a Database per service pattern for Microservices. When we implement a Database per service pattern, we can no longer write queries that join data from multiple services. API composition will provide ways to implement queries in Microservices.

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Address the use cases requesting data from multiple data sources

Solution

Implement a query by defining an API Composer, which invoking the services that own the data and performs an in-memory join of the results.

Benefits   

• It a simple way to query data in a microservice architecture

Drawbacks

• Some queries would result in inefficient, in-memory joins of large datasets.

Command Query Responsibility Segregation (CQRS)

Command Query Responsibility Segregation pattern is also designed to address the issues after implementing Database per service pattern. it is no longer straightforward to implement queries that join data from multiple services. Also, if you have applied the Event sourcing pattern then the data is no longer easily queried. CQRS provide another mechanism to implement query hat retrieves data from multiple services in a microservice architecture.

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Address the use cases requesting data from multiple data sources

Solution

In this approach, we define a view database which is the read-only replica of the actual data. This database is designed to support the query that require data from multiple data sources. CQRS define the application responsibility to keep the data up-to-date by subscribing to event published by service that own the data.

Benefits   

• Supports multiple denormalized views that are scalable and performant
• Improved separation of concerns = simpler command and query models
• Necessary in an event sourced architecture

Drawbacks

• Increased complexity
• Potential code duplication
• Replication lag/eventually consistent views

Domain Event

A service often needs to publish data/events when it updates its database. These events might be needed, for example, to update a CQRS view database. Alternatively, the service might participate in a choreography-based saga, which uses events for coordination. Domain Event pattern provide a method for a service to publish an event

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Address the use cases requesting data from multiple data sources

Solution

Organize the business logic of a service as a collection of DDD aggregates that emit domain events when they created or updated. The service publishes these domain events so that they can be consumed by other services.

Event Sourcing

As we learned, CQRS and Saga. In CQRS, a command needs to update the database and publish the message so that other services and perform the required action. So the database update and sending a message must be atomic in order to avoid data inconsistency. Event sourcing talks about how reliably and automatically update the database and publish the message/event.

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Update the database and sending message must be atomic to avoid data inconsistency

Solution

Use the event sourcing when you are updating the database and publishing a message for that event. Applications persist events in an event store, which is a database of events. The store has an API for adding and retrieving an entity’s events. The event store also behaves like a message broker. It provides an API that enables services to subscribe to events. When a service saves an event in the event store, it is delivered to all interested subscribers.

Some entities, such as a Customer, can have many events. In order to optimize loading, an application can periodically save a snapshot of an entity’s current state. To reconstruct the current state, the application finds the most recent snapshot and the events that have occurred since that snapshot. As a result, there are fewer events to replay.

Example

Taking example of Booking and Passenger service. Let’s develop it using event sourcing and CQRS.

Benefits   

• It solves one of the key problems in implementing an event-driven architecture and makes it possible to reliably publish events whenever state changes.
• Because it persists events rather than domain objects, it mostly avoids the object relational impedance mismatch problem.
• It provides a 100% reliable audit log of the changes made to a business entity
• It makes it possible to implement temporal queries that determine the state of an entity at any point in time.
• Event sourcing-based business logic consists of loosely coupled business entities that exchange events. This makes it a lot easier to migrate from a monolithic application to a microservice architecture.

Drawbacks

• It is a different and unfamiliar style of programming and so there is a learning curve.
• The event store is difficult to query since it requires typical queries to reconstruct the state of the business entities. That is likely to be complex and inefficient. As a result, the application must use Command Query Responsibility Segregation (CQRS) to implement queries. This in turn means that applications must handle eventually consistent data.

Conclusion

When we move from monolithic to Microservices architecture, we address many challenges like scalability, agility, flexibility, and many more. But when using this architecture there are numerous issues that we must address. When we work on some business cases we need to play around with the data, if our solution architecture is not planned with design patterns instead of getting the benefits of Microservices we will loop into the various issues. So, understand these design patterns and take the maximum benefits of Microservices architecture.

References

Microservice Architecture – https://microservices.io/

Explore more at Teknonauts.com