[Hiring] Senior Data Engineer @Capgemini

Role Description

Senior Data Engineer with production experience designing, optimizing, and operating large-scale Spark workloads on Microsoft Fabric or Databricks.

• Strong in Delta Lake performance (liquid clustering, OPTIMIZE, file sizing, incremental merges).
• Expert in Spark UI-driven troubleshooting (skew, spill, OOM, shuffle).
• Knowledgeable in Fabric capacity economics (CU-seconds, bursting, smoothing, throttling stages, pool and node sizing).
• Builds reliable, cost-aware pipelines that stay inside capacity limits while meeting SLAs for both interactive and batch workloads.
• Develops and maintains internal Python libraries and reusable Spark modules that standardize ingestion, governance, and optimization patterns across teams.

Qualifications

• Spark load optimization & troubleshooting:
• Stage-level diagnosis (task duration histograms, shuffle skew, spill).
• AQE, broadcast vs shuffle joins, predicate pushdown, salting only when proven.
• Driver vs executor memory pressure, Python UDF cost.
• Delta Lake & Fabric Lakehouse:
• Liquid clustering vs classic partitioning + Z-order, target file size, small-file mitigation.
• Vorder for Direct Lake, statistics, VACUUM retention vs time travel.
• Idempotent incremental MERGE patterns with watermarks and quarantine.
• Fabric capacity & compute:
• CU-second accounting, throttling windows (overage protection, interactive delay, background rejection).
• Carryforward and burndown, starter vs custom vs custom live pools.
• Node sizing (Small through XX-Large) and its effect on max node count.
• Auto-scale and dynamic allocation, single node resource sharing, CU budgeting across concurrent jobs.
• Operational design:
• Separating interactive, scheduled, and heavy batch workloads.
• Hard caps on pool max nodes, capacity isolation strategies.
• Monitoring via Capacity Metrics app and Spark resource usage UI.
• Maintenance pipelines (OPTIMIZE timing, statistics).
• Python library development:
• Designs, packages, and versions internal Python libraries and Spark utility modules.
• Emphasizes production readiness - type hints, logging, error handling, configuration-driven behavior, unit/integration testing, and clear READMEs.

Requirements

• Starts with measurement (Spark UI, Capacity Metrics, file-level Delta stats) before changing infrastructure.
• Prefers code and configuration changes that reduce shuffled bytes and future CU-second debt over "make the pool bigger."
• Documents pool sizing rationale, throttling risks, and maintenance expectations.
• Comfortable explaining capacity behavior to both engineering and platform teams.
• Built and iterated production Python libraries and notebook modules used across multiple Fabric workspaces.

Benefits

• Health insurance from the first days, regardless of the probationary period.
• Christmas holidays from 25 December to 31 December.
• Support for psychological rehabilitation department of Superhumans and Veteran HUB.
• Donated over UAH 500,000 for prosthetics for three Ukrainian defenders.
• Implemented internal policy making the company friendly to military and veterans.