[Hiring] Data Engineer @Sphere Partners

Role Description

We are looking for a Data Engineer to support a data migration initiative for our client. The project involves migrating approximately 200 existing ETL jobs from PostgreSQL to Snowflake.

The core platform is already established and operational. The primary focus will be:

• Adapting existing SQL logic and pipeline orchestration.
• Validating results against the current PostgreSQL outputs.
• Troubleshooting discrepancies to ensure data consistency.

You will work independently on migration tasks while collaborating with the existing engineering team for onboarding and technical support.

Responsibilities

• Migrate and adapt existing SQL scripts and ETL jobs from PostgreSQL to Snowflake.
• Configure and update Python-based pipeline orchestration to support the migrated jobs.
• Develop validation scripts to compare outputs between PostgreSQL and Snowflake.
• Investigate and troubleshoot data discrepancies, identifying root causes related to source data, replication delays, SQL behavior, or differences between database platforms.
• Debug and resolve migration issues while preserving existing transformation logic and expected outputs.
• Collaborate with the existing engineering team to understand current pipelines, resolve technical challenges, and ensure a reliable migration process.

Requirements

• Strong SQL skills, including writing, analyzing, and debugging complex queries.
• Hands-on experience with PostgreSQL and Snowflake.
• Practical experience developing, maintaining, or migrating ETL/ELT pipelines.
• Proficiency in Python for pipeline orchestration, scripting, and data validation.
• Experience comparing datasets across systems, investigating discrepancies, and performing data reconciliation.
• Strong troubleshooting and analytical skills, with the ability to identify root causes of data inconsistencies.
• Ability to work independently, manage assigned tasks, and communicate effectively with an existing engineering team.

Nice to Have

• Experience with Pentaho or other ETL/orchestration tools.
• Experience working with custom Python-based ETL frameworks.
• Familiarity with data replication, CDC, and replication lag.
• Previous experience migrating SQL workloads between relational databases and cloud data warehouses.

Engagement

• 3-month initial contract, full-time, with potential extension.