This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more.
Role Description
We are currently looking for a Confluent Kafka Expert who will be responsible for the architecture, implementation, and ongoing optimization of our Kafka-based systems. This role will involve collaborating with various teams to drive Kafka strategy, resolve complex issues, and enhance the overall reliability and scalability of our data streaming infrastructure.
-
Design & Architecture: Lead the design and architecture of Kafka clusters and event-driven systems to handle high-throughput, low-latency, and scalable data streaming.
-
Kafka Cluster Management: Take ownership of Kafka cluster setup, scaling, monitoring, and optimization for fault tolerance and high availability.
-
Performance Tuning: Optimize Kafka clusters and improve throughput, latency, and message delivery reliability.
-
Advanced Troubleshooting and Production Support: Resolve complex Kafka-related issues related to message delivery, partitioning, consumer lag, replication and provide production support in case of failures.
-
Kafka Integration: Guide integration of Kafka with external systems, including data lakes, relational databases, microservices, and real-time analytics platforms.
-
Capacity Planning: Assess system requirements, plan for future growth, and ensure that Kafka infrastructure can scale without sacrificing performance.
-
Monitoring & Alerting: Implement advanced monitoring, alerting, and reporting on Kafka clusters using tools like Prometheus, Grafana, Confluent Control Center, Dynatrace, etc.
-
Security & Compliance: Implement and enforce best practices for securing Kafka clusters, including SSL/TLS encryption, SASL authentication, ACLs, and data governance.
-
Collaboration: Work closely with cross-functional teams to align Kafka infrastructure with overall system architecture and operational requirements.
-
Research & Innovation: Stay up-to-date with the latest advancements in Kafka and streaming technologies and propose innovative approaches to enhance system performance and reliability.
Qualifications
-
4-year college degree + 4 years of experience in applicable field; OR Completion of advanced degree (master or higher) in applicable and relevant field + 2 years of related experience in applicable field or Experience will be considered in lieu of degree.
-
Expert-level knowledge of Confluent Kafka architecture, internals, and its ecosystem.
-
Hands-on experience in setting up, configuring, and managing Confluent Kafka clusters in a production environment.
-
Expertise in Kafka tuning and optimization (e.g., adjusting partitions, consumer groups, replication factors).
-
Strong knowledge of Kafka Streams API, Kafka Connect, and other stream processing tools.
-
Proficiency with Java and Python for building Kafka producers, consumers, and stream processing applications.
-
Experience with real-time data streaming and event-driven architecture.
-
In-depth understanding of distributed systems principles (e.g., partitioning, consistency, fault tolerance, and high availability).
-
Familiarity with cloud-based Kafka deployments (e.g., Confluent Cloud).
-
Advanced knowledge of data pipelines and integration with data lakes, relational databases, and big data ecosystems.
-
Strong DevOps experience in managing Kafka infrastructure using tools such as Docker, Kubernetes, and CI/CD pipelines.
-
Expertise in monitoring and alerting tools such as Prometheus, Grafana, Confluent Control Center, and Dynatrace.
-
Advanced understanding of security measures in Kafka, including data encryption, access control, and compliance.
-
Experience with disaster recovery, backup strategies, and ensuring the durability of data across multiple data centers.
-
Excellent problem-solving skills and ability to troubleshoot Kafka-related issues in complex, distributed systems.
-
Experience with Confluent Kafka, Confluent Schema Registry, and KSQL.
-
Knowledge of event-driven architecture and microservices using Kafka as the backbone for communication.
-
Experience with SQL and NoSQL databases.
-
Experience with Apache Flink or Apache Spark Streaming for stream processing in conjunction with Kafka will be good to have.
-
Cloud-native deployment experience (AWS, GCP, Azure) and managing Kafka in cloud environments.
-
Knowledge of data governance, compliance, and audit logging.
-
Work with agile methodology tools to monitor test plans.
-
Previous experience working with Burndown Charts.
-
Mortgage and Financial Experience is a Plus.
-
Travel for the role will be 10-15%.
Benefits
-
Comprehensive package including Remote/Hybrid workplace options
-
Health Benefits
-
Unlimited Flexible Time Off
-
Family Planning Services
-
Tuition Reimbursement
-
Paid Family Leave
-
401(k) Matching
-
Pet Insurance
-
LinkedIn Learning
-
Virtual Social Experiences
-
Career Pathing
-
Focus Time Fridays
-
Much, much more!