[Hiring] Senior / Staff Data Engineer @Index Exchange

Role Description

We are hiring a Senior / Staff Data Engineer to build and evolve the data processing and pipeline layer that powers reporting, billing systems, and real-time data products at Index Exchange. This role focuses on designing and operating large-scale batch and streaming data pipelines , enabling reliable, scalable, and efficient data transformation across the platform.

You will work on systems that transform raw, high-volume event data into clean, queryable, and production-grade datasets, supporting both API-driven data products and analytical workflows.

• Process billions of events per day across distributed pipelines
• Power core business datasets (reporting, billing, marketplace metrics)
• Operate across batch (Spark) and streaming (Kafka / Flink) architectures
• Require careful balancing of:
  • data correctness
  • processing efficiency
  • latency vs cost trade-offs

You will solve problems such as:

• Designing pipelines that scale without exploding compute costs
• Managing data correctness at scale (deduplication, late data, joins)
• Building systems that support both historical backfills and near real-time updates
• Evolving pipelines from centralized processing (Hadoop) toward more distributed and efficient patterns
• Streaming pipelines and Streaming DWs.

Qualifications

• Strong experience in data engineering at scale
• Deep expertise in:
  • Spark (required)
  • SQL and data modeling
• Experience with:
  • Airflow or workflow orchestration
  • Kafka or streaming systems
• Strong understanding of:
  • distributed data processing
  • data modeling (large-scale datasets)
  • performance optimization
• Ability to:
  • own pipelines end-to-end
  • debug complex data issues
  • work in high-scale, evolving environments

Requirements

• Define data processing standards and patterns across teams
• Lead large-scale pipeline and platform initiatives
• Influence data architecture and modeling decisions
• Drive improvements across:
  • reliability
  • cost efficiency
  • scalability

Benefits

• Comprehensive health, dental, and vision plans for you and your dependents
• Paid time off, health days, and personal obligation days plus flexible work schedules
• Competitive retirement matching plans
• Equity packages
• Generous parental leave available to birthing, non-birthing, and adoptive parents
• Annual well-being allowance plus fitness discounts and group wellness activities
• Commuter benefits and discounts, where available
• Employee assistance program
• Mental health first aid program that provides an in-the-moment point of contact and reassurance
• One day of volunteer time off per year and a donation-matching program
• Bi-weekly town halls and regular community-led team events
• Multiple resources and programming to support continuous learning
• A workplace that supports a diverse, equitable, and inclusive environment