Log Search and Processing Solution on Amazon OpenSearch Service
Discover how we implemented a log search and processing solution, which enhances observability and security for client’s platform.
Our Customer
Gorgany is a huge retail, wholesale & distribution company of outdoor equipment in Ukraine. The company specializes in the wholesale trade of goods for tourism, mountaineering, skiing, and active recreation.
The Obstacles They Faced
The client’s platform had difficulties and constraints in efficiently identifying and resolving issues on their log collection and processing system. It required too much of engineers manual work and impacted on the resilience and versatility of the platform as a whole.
How We Helped
By leveraging the power of a new log search and processing solution built on Amazon OpenSearch by our professionals, the client gained the ability to promptly resolve any potential breakdowns, ensuring uninterrupted performance and optimal functionality of their system.
Previously, all the logs of the client’s e-commerce platform were collected and processed via Amazon S3 buckets – cloud object storage. This solution was inconvenient and time-consuming for the identification of occurred issues with the product: the engineers had to download the files whenever an issue arose, and then to search needed data in each file, which significantly impeded the efficiency of issue resolutions. As a result, log search and processing were the weak points of the client’s platform, which in turn affected the all-encompassing observability and stability of the system.
Having conducted a thorough analysis of the challenge, our experts resolved to rely on Amazon OpenSearch Service as the technological cornerstone of a newly designed log processing solution. This was mainly motivated by two factors, namely:
- the capability of this service to collect and store logs from different sources in one place;
- the functionality of fast and easy navigation across diverse facets and data attributes.
The solution is built and configured in the following way: every ECS task definition has its own AWS ECS Fargate container, responsible for collecting and transmitting the data to the Amazon OpenSearch cluster. The same functionality of collecting and routing the data to the OpenSearch cluster is also implemented with the td-agents – an open-source and multi-platform log processor and forwarder. Then, on the side of Amazon OpenSearch, all the collected logs are stored, processed, and searched whenever required.