[Hiring] Lead Data Engineer @Emory University

Role Description

The Data Engineer Lead provides technical leadership and hands-on engineering expertise for the development and operation of Emory University’s Unified Data Platform (UDP), built on Microsoft Fabric. The role leads to the integration of data from HR, Finance, Student, Research, and other enterprise applications to support analytics, reporting, research, and institutional decision-making.

• Lead the technical delivery of source system onboarding and data integration initiatives, from source analysis and design through development, validation, deployment, and production support.
• Partner with architects to implement platform designs, reusable ingestion frameworks, and engineering standards aligned with the UDP architecture and institutional priorities.
• Design, develop, and optimize ETL and ELT pipelines using SQL, Python, PySpark, and Microsoft Fabric or comparable cloud technologies.
• Analyze source schemas, business rules, and dependencies across enterprise applications; define source-to-target mappings, incremental load strategies, and reconciliation requirements.
• Guide implementation of bronze, silver, and gold data layers, including data transformation, dimensional modeling, and curated datasets for reporting and analytics.
• Establish and review standards for coding, testing, documentation, version control, and deployment; conduct code reviews and mentor data engineers.
• Coordinate engineering tasks and technical dependencies with Business Analysts, Project Managers, Data Analysts, Architects, and source system owners; communicate delivery progress, risks, and resource needs.
• Implement data quality checks, reconciliation controls, monitoring, alerting, error handling, and recovery procedures to support reliable production processing.
• Troubleshoot complex pipeline and data issues; tune SQL, Spark workloads, and orchestration to improve performance, scalability, and efficient use of platform capacity.
• Apply institutional data governance and security standards, including access controls, metadata, lineage, and appropriate handling of sensitive data; support HIPAA requirements where applicable.
• Work with researchers and business stakeholders to translate requirements into technical specifications and explain design choices and data limitations clearly.
• Create and maintain technical documentation, operational runbooks, source mappings, and support procedures; coordinate production deployments and knowledge transfer.
• Support existing UDP and Enterprise Business Intelligence (EBI) integrations and services, balancing operational needs with project delivery priorities.
• Evaluate emerging technologies and develop focused proofs of concept that address identified business or engineering needs.

Qualifications

• Strong experience designing and maintaining data pipelines and ETL/ELT processes across multiple source systems.
• Proficiency in Python, SQL, and Apache Spark (PySpark).
• Experience with modern data platforms such as Azure Fabric (preferred), Azure Synapse Serverless, or Databricks.
• Ability to work with large, complex, and distributed datasets.
• Solid understanding of data modeling concepts and best practices.
• Familiarity with data governance, metadata management, and data quality frameworks.
• Ability to lead technical design discussions, review code, mentor engineers, and guide delivery across multiple workstreams.
• Experience with CI/CD practices and proficiency with version control systems like Git.
• Familiarity with data pipeline orchestration and monitoring tools like Airflow, Azure Data Factory or similar frameworks.
• Understanding of biomedical informatics methodologies is a plus.
• Strong analytical, problem-solving, and critical-thinking abilities.
• Strong analytical, problem-solving, communication, and collaboration skills, with the ability to explain technical concepts to diverse audiences.
• Ability to prioritize engineering work, identify delivery risks, and adapt to changing business and platform requirements.

Requirements

• A bachelor's degree in a related field and seven years of related experience, OR an equivalent combination of education, training, and experience.

Preferred Qualifications

• Experience with Microsoft Fabric Lakehouse, Warehouse, notebooks, and data pipelines.
• Experience in integrating higher education and Healthcare Enterprise HR, Finance, Student, and Research applications.

Additional Details

Tasks related to this position can be performed remotely with only occasional visits to an Emory University location. Eastern (EST) time zone business hours may apply. Emory reserves the right to change this status with notice to employee. Emory does not approve as a primary work location in the following states; NJ, AK, and HI, any U.S. Territories or outside of the United States.