Lead DevOps Engineer (Kafka)

DESCRIPTION

Our client is a leading information services provider of financial, property, and consumer information, analytics, and business intelligence.

Responsibilities

• Install and provision new Kafka clusters and supporting components
• Regularly monitor the health and performance of the Kafka platform and data pipelines
• Identify and fix issues related to the platform, including data pipelines, network problems, cloud or containerization resources failures, or software bugs
• Perform regular performance tuning of Kafka platform components
• Monitor and optimize the cost and performance of Kafka clusters
• Upgrade the Kafka platform to newer versions, including planning, testing, and implementation
• Manage the security of the Kafka platform, including access control lists, encryption, and regular security reviews
• Perform regular backups and disaster recovery procedures
• Manage the capacity of the Kafka platform, including projecting future growth and scaling needs
• Document procedures, configurations, and issue resolutions, and share knowledge with the team
• Work with Confluent Support for issues that cannot be resolved in-house
• Support existing Infrastructure as Code (IaC) and Configuration Management (CM) automations/pipelines for Kafka platform management and maintenance
• Maintain and enhance onboarding automation scripts
• Support Kafka self-service automations for Topic, RBAC, Schema, Connectors management
• Support application teams in setting up and onboarding Kafka consumers, producers, connectors, and streams
• Provide support for team requests in Slack and convert to CLOUD Tickets if complexity is high
• Implement new features released by the vendor as part of their product roadmap, in coordination with the CoreLogic Platform owner team
• Implement and enhance automation scripts and processes to reduce the number of tickets via self-service

• Proven experience in the implementation and maintenance of Confluent Platform
• Relevant certifications (such as Confluent Certified Developer or Administrator for Apache Kafka) would be an advantage
• Strong knowledge of HELM
• Excellent problem-solving skills and the ability to troubleshoot complex issues
• Strong written and verbal communication skills
• Experience in automating processes and maintaining automated scripts
• Understanding of networking, and capability to coordinate with different teams including the Networking team, and CICD team, among others
• Kafka Confluent Stack - Advanced
• Terraform
• HELM Kubernetes/Containers, Docker -
• Cloud GCP (compute, networking, storage, IAM)
• Cloud AWS
• Jenkins
• Linux administration
• PagerDuty
• Uptrends

• Competitive compensation depending on experience and skills
• Individual career path
• Unlimited access to LinkedIn learning solutions
• Sick leave and regular vacation
• English classes with certified English teachers
• Flexible work hours