What are microservices

Microservices – also known as the microservice architecture – is an architectural style that structures an application as a collection of loosely coupled services, which implement business capabilities. The microservice architecture enables the continuous delivery/deployment of large, complex applications. It also enables an organization to evolve its technology stack.

Need of design patterns for Microservices Data Management

 We have decomposed the application in Microservices but Business demand integrated data that’s where we miss monolithic architecture. Since we have addressed some of the major challenges with microservices we introduced some powerful design patterns to address the data management need in microservices architecture.

Design Patterns for Data management

Database Per Service

When we have segregated the responsibilities at the application layer we want to maintain this decoupled nature till the database layer. So, what database architecture we should follow?

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Some business use cases are not independent, they must interact with multiple services either to fetch the data or updated the data owned by other services
• Some business use cases need to join the data owned by multiple services
• Databases must sometimes be replicated and sharded in order to scale
• Every microservice has different data storage requirement. For some services, RDBMS is the best option and for some NOSQL is the right fit

Solution

“Keep each microservice’s persistent data private to that service and accessible only via its API. A service’s transactions only involve its database”

In the above diagram Data, A will be private to Service A and will not never be accessed directly by any service. There are many ways to keep a service’s persistent data private. You not necessarily need to set up a separate database service for all microservices. But you can use below for instance if you are using RDBMS

• Private-tables-per-service – each service owns a set of tables that must only be accessed by that service
• Schema-per-service – each service has a database schema that’s private to that service
• Database-server-per-service – each service has it’s own database server.

Benefits   

• Services are loosely coupled
• Each services are free to use best suited database for their need

Drawbacks

• Implementing business transactions that span multiple services is not straightforward
• Implementing queries that join data that is now in multiple databases is challenging
• Complexity of managing multiple SQL and NoSQL databases

Shared Database

There are use cases when each business cases are so interlinked that you will end up touching most of Microservices in a single call. For systems, we must use a shared database.

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Many use cases require data owned by multiple services
• Use cases need to update data owned by multiple services

Solution

Use a (single) database that is shared by multiple services. Each service freely accesses data owned by other services using local ACID transactions.

Benefits   

• Easy for developer to create a query for required data
• Easy to operate single set of data for ETL & other data management activities

Drawbacks

• Database became monolithic and we have coupled the responsibilities at data layer
• Reduced freedom to use best fit database for each microservices
• Manage the schema switch at run time will slow down system

Saga

As we talked about business cases where we require to get the data owned by multiple services also there are transactions that need to update the data owned by multiple microservices. This pattern will address the problem related to transactions that span across multiple microservices.

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Use cases need to update data owned by multiple services

Solution

Each business transaction that span across multiple services should be implemented as Saga. Saga is basically a sequence of local transactions where each transaction updates the database and trigger a event or message to execute the next transaction. In this series we also implement the fallback mechanism linked to each transaction so that if local transaction fails saga can execute sequence of compensating transaction to rollback the changes made by earlier transactions.

There are two ways of coordination sagas:

• Choreography – each local transaction publishes domain events that trigger local transactions in other services
• Orchestration – an orchestrator (object) tells the participants what local transactions to execute

Example – Choreography-based saga

Example – Orchestration-based saga

Benefits   

• It enables an application to maintain data consistency across multiple services without using distributed transactions
• Resilient Architecture with high performance

Drawbacks

• The programming model is more complex
• In order to be reliable, a service must atomically update its database and publish a message/event

API Composition

API composition is designed to address the problem that comes as an outcome of implementing a Database per service pattern for Microservices. When we implement a Database per service pattern, we can no longer write queries that join data from multiple services. API composition will provide ways to implement queries in Microservices.

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Address the use cases requesting data from multiple data sources

Solution

Implement a query by defining an API Composer, which invoking the services that own the data and performs an in-memory join of the results.

Benefits   

• It a simple way to query data in a microservice architecture

Drawbacks

• Some queries would result in inefficient, in-memory joins of large datasets.

Command Query Responsibility Segregation (CQRS)

Command Query Responsibility Segregation pattern is also designed to address the issues after implementing Database per service pattern. it is no longer straightforward to implement queries that join data from multiple services. Also, if you have applied the Event sourcing pattern then the data is no longer easily queried. CQRS provide another mechanism to implement query hat retrieves data from multiple services in a microservice architecture.

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Address the use cases requesting data from multiple data sources

Solution

In this approach, we define a view database which is the read-only replica of the actual data. This database is designed to support the query that require data from multiple data sources. CQRS define the application responsibility to keep the data up-to-date by subscribing to event published by service that own the data.

Benefits   

• Supports multiple denormalized views that are scalable and performant
• Improved separation of concerns = simpler command and query models
• Necessary in an event sourced architecture

Drawbacks

• Increased complexity
• Potential code duplication
• Replication lag/eventually consistent views

Domain Event

A service often needs to publish data/events when it updates its database. These events might be needed, for example, to update a CQRS view database. Alternatively, the service might participate in a choreography-based saga, which uses events for coordination. Domain Event pattern provide a method for a service to publish an event

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Address the use cases requesting data from multiple data sources

Solution

Organize the business logic of a service as a collection of DDD aggregates that emit domain events when they created or updated. The service publishes these domain events so that they can be consumed by other services.

Event Sourcing

As we learned, CQRS and Saga. In CQRS, a command needs to update the database and publish the message so that other services and perform the required action. So the database update and sending a message must be atomic in order to avoid data inconsistency. Event sourcing talks about how reliably and automatically update the database and publish the message/event.

Forces

• Services must be loosely coupled so that they can be developed, deployed and scaled independently
• Update the database and sending message must be atomic to avoid data inconsistency

Solution

Use the event sourcing when you are updating the database and publishing a message for that event. Applications persist events in an event store, which is a database of events. The store has an API for adding and retrieving an entity’s events. The event store also behaves like a message broker. It provides an API that enables services to subscribe to events. When a service saves an event in the event store, it is delivered to all interested subscribers.

Some entities, such as a Customer, can have many events. In order to optimize loading, an application can periodically save a snapshot of an entity’s current state. To reconstruct the current state, the application finds the most recent snapshot and the events that have occurred since that snapshot. As a result, there are fewer events to replay.

Example

Taking example of Booking and Passenger service. Let’s develop it using event sourcing and CQRS.

Benefits   

• It solves one of the key problems in implementing an event-driven architecture and makes it possible to reliably publish events whenever state changes.
• Because it persists events rather than domain objects, it mostly avoids the object relational impedance mismatch problem.
• It provides a 100% reliable audit log of the changes made to a business entity
• It makes it possible to implement temporal queries that determine the state of an entity at any point in time.
• Event sourcing-based business logic consists of loosely coupled business entities that exchange events. This makes it a lot easier to migrate from a monolithic application to a microservice architecture.

Drawbacks

• It is a different and unfamiliar style of programming and so there is a learning curve.
• The event store is difficult to query since it requires typical queries to reconstruct the state of the business entities. That is likely to be complex and inefficient. As a result, the application must use Command Query Responsibility Segregation (CQRS) to implement queries. This in turn means that applications must handle eventually consistent data.

Conclusion

When we move from monolithic to Microservices architecture, we address many challenges like scalability, agility, flexibility, and many more. But when using this architecture there are numerous issues that we must address. When we work on some business cases we need to play around with the data, if our solution architecture is not planned with design patterns instead of getting the benefits of Microservices we will loop into the various issues. So, understand these design patterns and take the maximum benefits of Microservices architecture.

References

Microservice Architecture – https://microservices.io/

Explore more at Teknonauts.com

Awadhesh Pratap Dwivedi

Awadhesh Pratap Dwivedi is an IT industry leader with over 13+ years of experience. He is excellent at providing an easy solution to complex business problems with his tremendous problem-solving skills. Currently, he is working with Oracle as a Principal Solution Engineer.

teknonauts.com/

Helidon framework (Open sourced by Oracle ) to develop fast, lightweight and secure microservices

Microservices – also known as the microservice architecture – is an architectural style that structures an application as a collection of loosely coupled services, which implement business capabilities. The microservice architecture enables the continuous delivery/deployment of large, complex applications. It also enables an organization to evolve its technology stack. There are many framework used for writing microservices like Spring boot, Micronauts, micronaut, Quarkus and Helidon. In this article we will talk about Helidon framework and how we can write an effective, fast and lightweight microservice using Oracle Helidon.

What is Helidon ?

Helidon is a microservices framework developed by Oracle internal projects under the name J4C(Java for Cloud). Now Helidon framework has been open sourced by Oracle.

Helidon SE and Helidon MP

Currently there are two programming model supported by this framework – Helidon SE and Helidon MP. Helidon SE is designed for microservices that support reactive programming model. Helidon MP is designed to support Eclipse MicroProfile runtime that allow Jakarta EE community to run portable microservices.

Helidon architecture and how Helidon SE & Helidon MP fit together

The relationship between Helidon SE and Helidon MP is shown in the architecture diagram

Helidon SE quickstart

Prerequisite

• Maven
• Java > 11

1. Run following command from cmd/bash

mvn -U archetype:generate -DinteractiveMode=false -DarchetypeGroupId=io.helidon.archetypes -DarchetypeArtifactId=helidon-quickstart-se -DarchetypeVersion=2.2.2 -DgroupId=com.teknonauts.examples -DartifactId=helidon-example-se -Dpackage=com.teknonauts.examples.helidon.se

2. It will generate project: helidon-example-se. Import this project as an existing maven project in your workspace. Structure will look like this

3. Understand main method in Main.java

WebServer server = WebServer.builder(createRouting(config))
                .config(config.get("server"))
                .addMediaSupport(JsonpSupport.create())
                .build();

        // Try to start the server. If successful, print some info and arrange to
        // print a message at shutdown. If unsuccessful, print the exception.
        server.start()
                .thenAccept(ws -> {
                    System.out.println(
                            "WEB server is up! http://localhost:" + ws.port() + "/greet");
                    ws.whenShutdown().thenRun(()
                            -> System.out.println("WEB server is DOWN. Good bye!"));
                })
                .exceptionally(t -> {
                    System.err.println("Startup failed: " + t.getMessage());
                    t.printStackTrace(System.err);
                    return null;
                });

4. Now run the main method and check the services –

http://localhost:8080/greet

http://localhost:8080/greet/{name}

5. You can also run below command to generate jar and run the server

mvn package
java -jar target/helidon-example-se.jar

6. Build and Run Docker Image

docker build -t helidon-example-se .
docker run --rm -p 8080:8080 helidon-example-se:latest

Helidon MP quickstart

Prerequisite

• Maven,
• Java > 11

1. Run following command from cmd/bash

mvn -U archetype:generate -DinteractiveMode=false -DarchetypeGroupId=io.helidon.archetypes -DarchetypeArtifactId=helidon-quickstart-mp -DarchetypeVersion=2.2.2 -DgroupId=com.teknonauts.examples -DartifactId=helidon-example-mp -Dpackage=com.teknonauts.examples.helidon.mp

2. It will generate project: helidon-example-mp. Import this project as an existing maven project in your workspace. Structure will look like this

3. Package the project with below command

mvn package

4. Run the below command to start the server

java -jar target/helidon-example-mp.jar

5. Observe below API and related code

http://localhost:8080/greet

http://localhost:8080/greet/{name}

6. Health Matrics

Health 
curl -s -X GET http://localhost:8080/health

Matrics in Prometheus format 
curl -s -X GET http://localhost:8080/metrics 

7. Build and Run docker image

docker build -t helidon-example-mp .
docker run --rm -p 8080:8080 helidon-example-mp:latest

Conclusion Helidon SE vs Helidon MP

Helidon SE & Helidon MP example code available on our git page

https://github.com/teknonauts/helidon

Next topic to cover

In this article I have covered basics of helidon framework and how to create simple project in both categories. Below are the two topics I will cover in next post of this series

• GraalVM and its benefit. GraalVM support in Helidon framework
• Non-blocking Database client in Helidon

Reference

• Helidon.io
• Example from baeldung
• Microprofile.io — https://microprofile.io/

Explore more at Teknonauts.com

Awadhesh Pratap Dwivedi

Awadhesh Pratap Dwivedi is an IT industry leader with over 13+ years of experience. He is excellent at providing an easy solution to complex business problems with his tremendous problem-solving skills. Currently, he is working with Oracle as a Principal Solution Engineer.

teknonauts.com/

Let me start with the background of integration & need of enterprise service bus which emerged from a concept called interoperability.

What is interoperability?

Interoperability is the ability of an enterprise and its architecture domains i.e., business, data, applications and technology to share information and services, and seamlessly communicate with other architecture domains. We can achieve this through the standardization and adoption of common systems, standards and data exchange protocols.

Why interoperate?

A connected business requires strategic collaboration and coordination with data & services. Exchange of data and services enable them with many new capabilities and deliver significant business value to their customers.

Interoperability architecture viewpoints

Interoperability architecture established in terms of establishing and identifying design architectures that used as the foundational blueprint for establishing collaboration between organizations or providing a reference guideline for solution architects to implement solutions. Interoperable architecture viewpoints established with different viewpoints depending on:

Architecture viewpoint for an enterprise architect

As a part of the enterprise architecture design and the development of an organizational blueprint, it is necessary that the blueprint considers inter and intra organization transactions and establishes a mechanism to facilitate smooth and seamless exchange of information between the two entities. Establishing interoperability demands focus on identification of the transactions or business processes inter-linked or dependent on other organizations, review of the information systems required or available around the data exchange requirements and the supporting technology for the same.

The architecture domain viewpoints entail focus on the core architecture dimensions of enterprise architecture including:

• Business
• Information systems (data and applications)
• Technology

Implementation viewpoint for a solution architect

In this context, it is essential to understand the two terminologies — interoperability and integration at the beginning. While interoperable looks at a broader spectrum of compatibility of two disparate and distinct systems, integration is a sub-component of interoperability as explained below

Enterprise Service Bus

In simple term, Enterprise Service Bus (ESB) is a flexible connectivity infrastructure for integrating applications and services.

An Enterprise Service Bus performs the following:

• Matches and routes communication between services (Routing, Mediation & Transformation)
• Converts between different transport protocols
• Transforms message formats between requestor and service
• Identifies and distributes business events from disparate sources
• An Enterprise Service Bus is based on open standards (Work of principal Connecting Anything to Anything)

At the heart of the Enterprise Service Bus architecture — is the enterprise services bus, a collection of middleware services that provides integration capabilities. The bus provides the medium for messages (In Message Broker architecture) to reach their destinations.

Enterprise Service Bus provided services are themselves distributed in the sense that different components come and play their role in providing the infrastructure services promised by the Enterprise Service Bus.

In short, ESB is a centralized middleware, which replaces complex point-to-point communication and provides flexible & highly scalable solution to integrate with multiple heterogeneous systems/applications.

API Gateway

API Manager is a solution for designing and publishing APIs, creating & managing API documentation. API manager is also known as API gateway, which is, used as an entry point for all your enterprise APIs so you can monitor, secure & transform them as per the need. These use case is more relevant for microservices architecture.

API Manager perform following

• Design and Prototype APIs
• Publish and Govern API Use
• Control Access and Enforce Security
• Create a Store of all Available APIs
• Manage Developer Community
• Manage API Traffic
• Monitor API Usage and Performance

Fundamentals of Microservices Architecture

When service-oriented architecture & REST has proven their solutions and widely accepted for system-to-system communications as compared to legacy technologies like EJB, distributed computing concept has been extended for service-to-service communications that emerged as Microservice architecture.

Microservices is an architecture style that designs an application as collections of services based on functions or any other logical unit. The aim is to breakdown services from not only functions/logical unit perspective but also consider maintainability & deployment strategy that can benefit the OSGi concept.

Let me share an architecture of Microservices, I have implemented recently. We have separated core data & business functions into respective microservices.

While developing Microservices has become very easy with the frameworks like spring boot, cross-cutting concerns like security, data validation, logging, caching, monitoring, etc. also made available as a pluggable component. So ideally, you do not want to bother about these cross-cutting concerns while developing your services rather you want to focus on your business logic.

Now if we observed the need in Microservices we do not see the scope of any to any connectivity (which is a concept of ESB). All cross-cutting concerns fulfilled by API Manager/Gateway if you refer features as described before. However, another driving principle that is not present in the architecture above is “third party integrations”. Therefore, if you see features of ESB and understand the need for third-party integrations, it is worth implementing along with Microservices.

The Mobile-first approach (due to the explosion in mobile device adoption), has forced most of the applications to expose their services over HTTP as REST by default irrespective of the technology you are using for development which has limited the scope for ESB.

Most of the ESB products come with build-in API gateway feature and I have seen examples where ESB is being implemented but actually utilizing API gateway features. We should avoid and assess well before buying ESB. We should pay only for what you use

Conclusion

Recent market demand and trends have unified the approach of service interoperability, which has limited the need for any to any connectivity ( Enterprise Service Bus ). API Gateway makes more sense to implement with Microservice architecture. However, the API Manager is not a replacement for an ESB.

For more information refer – API friends and explore more on Teknonauts

Awadhesh Pratap Dwivedi

Awadhesh Pratap Dwivedi is an IT industry leader with over 13+ years of experience. He is excellent at providing an easy solution to complex business problems with his tremendous problem-solving skills. Currently, he is working with Oracle as a Principal Solution Engineer.

teknonauts.com/