Home > Case studies

Case Study

Data streaming platform implementation and management

DevOps Engineers implementing a monitoring system for Kafka clusters
Industry
  • Technology
  • Company specialised in areas such as telecommunications, automation, digitalisation, engineering, Artificial Intelligence (AI), Internet of Things (IoT), cybersecurity, metaverse, transport and smart buildings

 

Challenges
  • Implement and manage data streaming platform, as well as related tools, which are used by the client’s management systems, sales and factories all over the world.
  • Develop and maintain pipelines to manage Kafka resources and Kafka’s ecosystem applications, such as Kafka Connect, Schema Registry, REST Proxy or ksqlDB.
  • Define the architecture, implement, and automate solutions to improve resource management and reduce expenses.
  • Implement complementary solutions to the data streaming platform for data use, monitoring, and governance.

Solution

The team dedicated to this project was responsible for developments such as:

  • Implementation of a GitOps model for managing Kafka resources, a repository with “Single Source Of Truth” (SSOT), and adoption of IaC (Infrastructure as Code).
  • Creation of a new Kafka cluster in Kubernetes, using Strimzi, migration of the current workload, running on OpenShift (on-premises), with the objective of reducing costs.
  • Restructuring applications and namespaces (Kubernetes) in order to reduce costs.
  • Analysis and treatment of vulnerabilities.
  • Implementation of Conduktor (data management, monitoring, and governance tool for Kafka).
  • Implementation of a monitoring system for Kafka clusters (Confluent Cloud and Strimzi), dashboards and alerts, using Prometheus and Grafana.
  • Implementation of Kafka Connect and connectors (sink/source) for data transfer between Kafka and different systems such as SAP, Snowflake, Amazon S3 and Celonis.
  • Implementation of Kafka REST Proxy instances for systems using Kafka, via REST API.
  • Creation of pipelines for customised applications developed in-house.

Methodologies

DevOps and GitOps.

Technologies

  • Apache Kafka
  • Kubernetes
  • Terraform
  • Helm
  • Amazon Web Services (AWS)
  • Red Hat OpenShift
  • Prometheus
  • Grafana
  • Confluent Cloud
  • GitHub Actions
  • GitLab CI
  • Strimzi
  • Conduktor
  • HashiCorp Vault

Timeline and resources

The project began in 2022 and the functionalities have been delivered according to the internal areas’ needs. The team currently works for approximately 15 internal clients/projects.

The team includes:

  • 1 Product Owner
  • 1 Service Owner
  • 2 DevOps Engineers

Results and customer experience

Our work was essential in continuing the implementation of the solution developed at the beginning of 2022. 

We enabled the development of several automations and documentation, as well as the implementation of new solutions, functionalities, and bug fixes.

We were also able to implement a GitOps model for the client’s data streaming platform.

Architecture

Data Streaming Platform Architecture