Custom Logs Management Solution on Amazon OpenSearch
Find out how we developed a centralized log management system with effective issue detection and incident notification mechanisms.
Our Customer
Omnyfy is a platform as a service (PaaS) multi-vendor E-commerce marketplace platform catering to B2B, B2C or Service marketplaces. The platform has been adopted by some of Australia’s largest enterprises to power multi-million dollar marketplaces.
The Obstacles They Faced
Cumbersome work of manual logs searching from multiple sources as the result of the absence of a dedicated platform’s solution for logs management.
How We Helped
Keeping the whole platform’s IT infrastructure scalable, our experts developed a centralized logs processing solution, implemented effective issue detection and incident notification mechanisms, and ensured controlled access to logs.
Previously, the client’s platform lacked a specialized solution for logs gathering, searching, visualizing, and managing. This resulted in time-consuming work on the system logs: the client found themselves spending a lot of time sifting through logs across various users’ marketplaces, their AWS accounts, and instances, just to find the relevant information.
Recognizing the need for making cost-effective solution, Romexsoft’s core goal was to design and build an effective logs processing and troubleshooting solution to address the challenge.
Centralized logs monitoring and issue detection
In order to handle this issue, our suggestion was to start by establishing a Centralized Logging System (Covered area) in a management account, which is built on top of Amazon OpenSearch and OpenSearch Dashboard. Specifically for this scenario, we installed Fluent td-agent on each EC2 instance of a separate marketplace to stream all the logs accounts to stream logs into the Amazon OpenSearch management account. By leveraging Amazon OpenSearch, we successfully centralized log collection into a single location.
Then, we set up logs monitoring system based on specific phrases and words: the solution automatically sends notifications via AWS SNS to the client whenever the preset keywords (phrases and words) occur in the logs. This feature helps to detect and distinguish issues on the platform timely and appropriately.
Differentiated access to the logs for the platform’s users
Another essential request from Omnyfy was to provide access to the logs for their clients (marketplace’s owners) through the OpenSearch dashboard, however, it was crucial to ensure access management that would be tailored only to each client’s individual. Here is where Amazon Cognito & Amazon OpenSearch take the scene for ensuring required access segregation.
To fulfill this requirement in accordance with AWS security best practices, we implemented log segregation with separate tenants for each marketplace (at that moment Omnyfy owned around 30 client’s marketplaces). As a part of the solution, Amazon Cognito authentication for OpenSearch was configured granting granular access to specific indices. To ensure proper identity management, the identity provider was set up to use an IAM role that corresponds with the user’s authentication token. Users belonging to a specific marketplace were grouped together in Cognito, on the basis of their specific IAM roles, thus ensuring that logs can only be viewed from their respective marketplace, and that the access requirements are met.