[Hiring] Data Pipeline Engineer @Carnegie Mellon University

Role Description

We are seeking a Data Pipeline Engineer to join the team! As a Data Pipeline Engineer, your role is vital in ensuring the integrity and reliability of our data pipelines. This position is responsible for monitoring, troubleshooting, and conducting root cause analysis of data quality issues within our pipelines, but as a part-time team member, you will consult and assist rather than lead in these areas. Your contributions are crucial to maintaining the high standard of our epidemiological tracking and forecasting tools. This role will report directly to the Delphi Engineering Manager.

Core Responsibilities

• Monitor and maintain the health and efficiency of data pipelines.
• Troubleshoot and perform root cause analysis for data discrepancies and pipeline issues.
• Communicate with data providers to understand data discrepancies and manage changes in data delivery.
• Implement fixes and enhancements to improve data quality and pipeline performance.
• Collaborate with data scientists and analysts to understand data needs and implement effective data solutions.
• Develop strategies for data validation and quality assurance.
• Optimize data flow and collection to improve system efficiency.
• Document and manage data pipeline architectures, including maintenance and update protocols.
• Use tools such as SQL, version control and CI/CD, containerization, task schedulers, python frameworks, and cloud services for data pipeline management.
• Ensure compliance with data governance and security standards.

Qualifications

• Bachelor’s Degree required.
• Minimum one year of research computing experience required.
• Basic Linux use and administration: system layout, file permissions, shell, utilities (syslog, cron), diagnostic tools (ps, htop, grep, lsof).
• Experience in Apache Airflow, preferably version 3.0.
• Basic database use, especially in Postgres.
• Rough script programming (Python, bash).
• Team software development (git/GitHub, Jira, code reviews, agile methodologies).
• Data analysis: diagnosing and fixing runtime errors and logic bugs; performing basic growth projections to predict future problems; communicating results.
• Required technologies: Python, MySQL/Postgres, Linux, git & GitHub, Apache Airflow.
• A combination of education and proven experience from which comparable knowledge is demonstrated may be considered.

Requirements

• Successful completion of a pre-employment background check.

Benefits

• Comprehensive medical, prescription, dental, and vision insurance.
• A generous retirement savings program with employer contributions.
• Tuition benefits.
• Amply paid time off and observed holidays.
• Life and accidental death and disability insurance.
• Free Pittsburgh Regional Transit bus pass.
• Access to our Family Concierge Team to help navigate childcare needs.
• Fitness center access.
• And much more!