Back to Remote jobs >
Software Development
Kafka Architect
@Mothership
[Hiring] Kafka Architect @Mothership
Feb 25, 2025 - Mothership is hiring a remote Kafka Architect. 💸 Salary: unspecified. 📍Location: USA.
This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more.
Role Description
We are seeking a highly experienced Kafka Architect with 8-10+ years of expertise in Kafka administration. This is a strategic role, not for a developer or admin, but for someone with deep architectural knowledge of Kafka and a strong ability to communicate complex concepts effectively. The ideal candidate will have extensive experience in designing, documenting, and optimizing both platform and developer elements of Kafka, with a focus on messaging and streaming capabilities. The candidate should be well-versed in implementing Kafka in diverse environments, with the ability to generalize concepts across different Kafka implementations (including Confluent and others).
Key Responsibilities:
-
Platform Focus: Infrastructure, Deployment, Operations, and Security
- Define brokers per cluster, partition count, and replication factor based on data volume, retention policies, and throughput needs.
- Design scaling configurations (horizontal and vertical) for Kafka clusters.
- Establish optimal network configurations to ensure low-latency performance.
- Define Zookeeper configurations (if not using KRaft mode) or alternative metadata management.
-
Storage Considerations:
- Provide recommendations for optimal disk configurations, with a focus on NVMe SSDs.
-
High Availability & Fault Tolerance:
- Define optimal replication factor (RF) strategies for both production and non-production environments.
- Design broker failover strategies and leader election mechanisms.
- Develop multi-region and multi-AZ deployment strategies for Kafka.
-
Security, Audit & Compliance:
- Implement and recommend Kafka authentication strategies (SASL, Kerberos, OAuth, TLS).
- Design authorization mechanisms (ACLs, RBAC) for Kafka.
- Advise on key management strategies, key rotation, and secure storage.
- Define auditing best practices for tracking access and changes to Kafka resources.
- Ensure encryption for both data in-transit (TLS) and at-rest (disk encryption).
- Advise on compliance frameworks (e.g., SOX, CCPA) and ensure Kafka adheres to necessary standards.
-
Monitoring & Observability:
- Advise on metrics collection using tools like Prometheus, Grafana, and Confluent Control Center.
- Implement security monitoring tools to detect and respond to real-time threats.
- Provide recommendations for monitoring disk usage and log aggregation (e.g., Elasticsearch, Kibana, Splunk).
- Implement lag monitoring strategies using tools like Burrow or Kafka UI.
-
Developer Focus:
- Define log segment configurations and cleanup policies (delete vs. compact).
- Provide recommendations for Kafka compaction processes and scheduling.
- Advise on time-based vs. size-based retention policies to optimize resource usage.
- Define strategies for cross-cluster replication and cluster linking.
- Set recovery point objectives (RPO) and recovery time objectives (RTO).
- Define automated backup verification and recovery procedures.
- Develop Kafka backup strategies, including configuration and topic-level backups.
- Recommend strategies for optimizing consumer group performance.
- Define dynamic partition rebalancing and scaling strategies.
- Recommend optimal data retention policies and efficient data compression formats.
Qualifications
- 8-10+ years of experience in Kafka architecture and administration (platform-focused).
- Strong knowledge of Kafka internals, cluster design, security, and monitoring tools.
- Proven ability to document and communicate complex Kafka platform architectures.
- Deep understanding of Kafka's role in both messaging and streaming use cases.
- Experience with different Kafka implementations (e.g., Confluent, Apache Kafka, others).
- Strong knowledge of distributed systems, scaling, and capacity planning.
- Familiarity with disaster recovery strategies and backup procedures.
- Experience with security best practices, including authentication, encryption, and access control.
- Understanding of compliance regulations and how to implement them in Kafka.
- Familiarity with metrics collection, monitoring, and observability tools (e.g., Prometheus, Grafana, Burrow).
- Excellent communication skills, both written and verbal, with the ability to engage with multiple stakeholders.
Preferred Skills
- Expertise in cloud-native Kafka implementations and multi-cloud architectures.
- Experience with KRaft mode or alternative metadata management strategies.
- Familiarity with automation and orchestration tools in Kafka environments.
Similar Remote Jobs
More jobs at Mothership
-
Software Development $80 to $85 per hour USAFeb 13, 2025
More Software Development jobs
-
Software Development unspecified MoroccoApr 04, 2025
-
Software Development unspecified MexicoApr 04, 2025
More jobs in USA
-
Sales / Business unspecified USA, IndiaApr 04, 2025
-
All others 103,700 - 172,800 usd / yearly, plus bonus USAApr 04, 2025
Kickstart Your Job Search
Need advice to apply?
Join our free
Webinar « 3
Mistakes to Avoid When Looking For A Remote Startup Job (And What To Do Instead) »
Register for free
Unlock 54,777 additional remote jobs, advanced search & email notifications
Get Access Now
Too many emails? Declutter your inbox with Meco
Your home for reading newsletters
Try for free - no card required.
️
| 📍 | Be aware of the location restriction for this remote position: USA |
| ‼ | Beware of scams! When applying for jobs, you should NEVER have to pay anything. Learn more. |
Back to Remote jobs >
Software Development
Software Development
|
Salary
💸
unspecified
|
Remote
Location
USA
|
|
Job Type
full-time
|
Posted
Feb 25, 2025
|
️
| 📍 | Be aware of the location restriction for this remote position: USA |
| ‼ | Beware of scams! When applying for jobs, you should NEVER have to pay anything. Learn more. |
