[Hiring] Senior Data Engineer @Lighthouse

Role Description

As a Senior Data Engineer, you will be a key player in the end-to-end lifecycle of our large-scale ingestion and transformation engine. You will be part of the full engineering loop:

• Designing, developing, scaling and maintaining our data pipelines handling large data streams - to the order of 100 TB per day.
• Combining advanced greenfield data engineering with improving, maintaining and supporting the stability and quality of existing pipelines.

Where you will have impact:

• End-to-End Pipeline Engineering: Design, build, and deploy scalable data pipelines that ingest data from a polyglot of sources (APIs, Webhooks, SFTP, Cloud Storage).
• Design and implement complex domain modeling and transformation logic to ensure disparate data sources are unified into high-fidelity data products.
• Architectural improvements: Transition legacy processes and components into modern, scalable and stable data architectures.
• Operational Excellence: Improve the reliability and quality of our data pipelines by building and improving advanced observability, automated testing, automation and self-healing features to proactively address root causes.
• Collaborate closely with product, other engineering teams, support and business teams, customers and partners.
• Lead the investigation and resolution of complex technical issues, performing deep-dive analysis into our code and systems to identify the root cause of issues in your domain.
• Drive engineering velocity by smartly applying AI to speed up development and leveraging AI and automation to streamline complex debugging, root-cause investigations, and recurring tasks.

Qualifications

• Experience architecting, designing, implementing, testing, deploying and maintaining complex data architectures and data pipelines.
• Professional proficiency in Python for large-scale data processing and pipeline development.
• Strong knowledge of data cloud technologies, data streaming systems (like Kafka or Google Cloud Pub/Sub), and cloud database solutions (like BigQuery, Snowflake, or Databricks).
• Hands-on experience with a major cloud platform such as GCP, AWS, or Azure.
• Fluency in English, with excellent communication and stakeholder management skills.

Requirements

• A forward-thinking builder who uses AI-assisted tools to ship code with higher quality and faster.
• Eager to apply AI and automation to solve scale challenges in data engineering and day-to-day operations.

Benefits

• Flexible time off: Autonomy to manage your work-life balance.
• Alan Flex benefits: 160€/month for food or nursery.
• Flexible retribution: Optional benefits through tax-free payroll deductions for food, transportation and/or nursery.
• Wellbeing support: Subsidized ClassPass subscription.
• Comprehensive health insurance: 100% Alan coverage for you, your spouse, and dependents.
• Impactful work: Shape products relied on by 85,000+ users worldwide.
• Referral bonuses: Earn rewards for bringing in new talent.

Technologies you will work with

• Python
• Google Cloud Platform (GCP)
• BigQuery
• Spanner
• Pub/Sub
• Kubernetes