[Hiring] Data Engineer @In All Media

Role Description

In this role, you will drive critical data platform migration, modernization, and integration initiatives across large-scale enterprise environments with a core focus on Salesforce data flows. You will architect, refactor, and operate robust data pipelines—transitioning legacy Hive table structures to modern Apache Iceberg formats, migrating and scaling Apache Airflow DAGs, and establishing cross-regional Kafka event streams. Operating within an AWS-native cloud environment and Snowflake data warehouse, you will ensure high-throughput processing, optimal query performance, and strict data quality standards throughout the migration lifecycle.

Key Responsibilities

• Execute the end-to-end migration of mission-critical datasets, legacy data workflows, and orchestrated DAGs to modernized cloud infrastructure.
• Convert legacy Hive table architectures to Apache Iceberg format on AWS S3, improving partition evolution, query performance, and metadata management.
• Reconfigure, validate, and manage Apache Kafka connections, brokers, and topics across multiple AWS regions to support continuous data streaming.
• Adapt, model, and tune data pipelines directly in Snowflake, leveraging advanced SQL techniques (CTEs, window functions, complex joins) to optimize compute utilization and latency.
• Author, schedule, and actively debug Apache Airflow DAGs, resolving upstream failures, optimizing sensor dependencies, and preventing pipeline bottlenecks.
• Define and execute structured data reconciliation and validation strategies to guarantee zero data loss, schema consistency, and end-to-end data integrity across migrations.
• Autonomously diagnose, profile, and resolve complex performance issues across distributed Spark/PySpark execution jobs, Kafka consumer groups, and AWS storage layers.

Qualifications

• 6+ years of hands-on data engineering experience designing, deploying, and maintaining distributed, production-grade data pipelines on petabyte-scale datasets.
• High proficiency in Python (pipeline scripting, data manipulation) and advanced SQL (complex CTEs, window functions, schema design, and query optimization).
• Proven production expertise with Apache Spark / PySpark for batch and streaming transformations.
• Strong hands-on experience authoring and maintaining complex DAGs in Apache Airflow and configuring cross-regional event pipelines with Apache Kafka.
• Direct experience managing AWS S3 data lakes utilizing Apache Iceberg and Parquet formats.
• Hands-on experience with Snowflake schema modeling, warehouse configuration, and query tuning.
• Demonstrated capability to independently troubleshoot distributed pipeline execution failures, network latencies, and serialization bottlenecks.
• Must be able to operate within an enterprise AWS WorkSpace (VDI) environment and complete a mandatory AWS network health-check verification (with passing screenshot required during application).

Nice-to-Have Skills

• Hands-on experience migrating legacy on-prem or cloud data warehouses to Snowflake.
• Practical experience handling platform-level Airflow migrations (e.g., major version upgrades, executor switches to Celery/Kubernetes).
• Experience utilizing AI-assisted software development tools (e.g., GitHub Copilot, Cursor) to optimize data modeling, pipeline scripting, and debugging.
• Exposure to Salesforce data architectures, object modeling, and CDC/API ingestion mechanisms.

Time Zone & Collaboration

This role requires full-time business-hour alignment with US Central Time (CT) to ensure real-time collaboration with US engineering leadership, active participation in daily standups, and live architectural syncs.

Language

All interviews, technical documentation, architectural reviews, daily team syncs, and operational communications will be conducted in English.