Log Correlation in Microservices

 Logging is one of the most important factors to trace any issue in the system. Multiple requests will reach the system at the same time. Each request will have its own way. In Microservices architecture the complexity will be more. The same request will get forwarded to multiple systems. Each layer in each system will add some logs to trace the issue. But we need to link all the logs belonging to the request. Then only we identify how the request is behaving on each system. To Link all the logs into a single link, that is why correlation is required. There are multiple frameworks are available to correlate all the logs. Azure is providing azure app insights to link and showcase all the requests into a single pipeline. Kibana and Serilog also provide to link the logs. Here we are going to see to link the logs without any third-party components.

Setup the CorrelationId:

The request will originate from any client application or any subsystem. Each request can be identified by its request Id. So the origin system can send its request Id or we can create the Id in case the origin didn't send it.



We have created new middleware that will fetch the correlation from the httpcontext or it will create the new id and set it into the request. Middleware is added into the request pipeline. All the requests will be processed and they will be added with CorrelationId.

Setup Logs:

The request will get into multiple flows and add its logs. While adding the logs it will send the information alone to the log. While saving the logs we are fetching the CorrelationId from the request and adding it with the logs. So all the logs will get added with CorrelationId. After that, we can store the logs based on our format.



The request needs to be forwarded to multiple systems to complete it. To send the request to multiple subsystems, we need to use any protocol like HTTP or AMQP. In this case, we need to attach the CorrelationId as a header with the corresponding protocol. The corresponding subsystem will read the id and logs the stuff.  

All the log data will be get stored in the same or multiple systems. But CorrelationId will be the key to linking all the logs. Using this id we can fetch the request flow order. It will help to understand the system's flow as well as to trace any issue in the system's behavior. Adding more logs will increase the system space. We need to ensure the log type on each log also.

 

Comments

Post a Comment

Popular posts from this blog

Hybrid Federated Ocelot Gateway

Image
Ocelot:  People who have experienced with .net core distributed microservices architecture will be very familiar with an ocelot. It's an API gateway to forward the request based on the routes. It supports the HTTP for rest services and WebSocket protocol for signalR services. It's providing a lot of features like authentication, Authorization, adding custom headers, and a lot. All are configurable. We can easily orchestrate multiple services using ocelot. It's a single point of contact to the outside world. It acts as a reverse proxy. As part of microservices architecture, each service will have a single responsibility. Sometimes service needs some data from other services or needs to complete some action from other services as part of business workflow. As part of distributed microservices architecture, there are many ways to enable communication between the services.  1) Service to Service communication via HTTP or Grpc. 2) Event-Driven via pub/sub model Service to servic
Read more

.NET Core NUGET packages

 .Net core development is based on the open-source NuGet packages. NuGet will reduce work and manage the application very easily. We need to select the NuGet based on the purpose otherwise it will impact the application performance as well as application size also. As part of the development lifecycle, we need to select a different set of libraries on each stage.  While creating a new service we need to inform what kind of APIs are going to be available and what are fields will belong to that each APIs. It will provide high-level information about the service. So We need clear documentation at the first phase of development itself. We can create documentation either manually or code-wise. Manual documentation will double the work. Code wise we can generate the document by code and based on the needs we can update it later. We can create high-level API documentation using Swashbuckle at the initial stage of development. Swashbuckle: We need to share the API specs while developing a rest
Read more

API Calls Evolution(Callback vs Promise vs Observables)

Application without API calls is very less in the market. At least they will send the app analytics details to the backend to improve their application feature. Making API calls is simple but it will have a lot of complexities in the implementation. As part of javascript, making API calls is a very critical part in terms of performance. We shouldn't block the thread until we receive the data. The application needs to know when the data is returned. Then its need to execute the data processing. Execute the data processing based on the API response will be simple or complex based on the business logic. Data processing parts have been improved a lot based on the developer pain-points. I have started my developments with callbacks, then promise, and now Observer. In each phase, they have addressed a lot of problems. Callback: Calling the function once a response is received. The response will be success or failure. Based on that function will get executed. This is a simple use case.
Read more