[Hiring] Databricks Engineer @CNX

Role Description

The Data Engineer is responsible for building, maintaining, and supporting data solutions on the Databricks Data Intelligence Platform running on Microsoft Azure, AWS or GCP. The role leverages Databricks, Python, PySpark, and SQL to deliver data ingestion, transformation, modeling, and pipeline orchestration.

Working within the standards and designs set with the Senior Data Engineer, this role independently owns the delivery of assigned pipelines from development through production support.

Responsibilities

• Build and maintain batch and streaming data pipelines for extracting, transforming, and loading (ETL/ELT) data from multiple sources into the Databricks lakehouse on Azure.
• Implement data ingestion using Auto Loader, Lakeflow Connect, and Azure Data Factory.
• Develop pipelines with Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables), including data quality expectations.
• Perform data manipulation and transformation using PySpark and Spark SQL, ensuring data is prepared for analytics and reporting.
• Configure, schedule, and monitor data workflows with Lakeflow Jobs (formerly Databricks Workflows).
• Implement data models within the lakehouse following medallion architecture (bronze/silver/gold) and team modeling standards.
• Apply Unity Catalog permissions and governance standards to the data assets you build.
• Write unit and data quality tests and deploy pipelines through CI/CD using Git folders and Declarative Automation Bundles (formerly Databricks Asset Bundles).
• Monitor, troubleshoot, and resolve pipeline failures and data quality issues in production.
• Participate in code reviews and contribute to team documentation and engineering standards.
• Collaborate with cross-functional teams, including business stakeholders and IT, to ensure data solutions meet business and performance requirements aligned to project goals.

Qualifications

• 6+ years of professional experience in data engineering, data & analytics or data warehousing.
• 3+ years of hands-on experience working with Databricks in a data engineering capacity.
• 3+ years of experience with Python, including manipulating large datasets using PySpark DataFrames, Spark SQL, and/or the pandas API on Spark.
• 2+ years of data engineering experience on Microsoft Azure (e.g., ADLS Gen2, Azure Data Factory, Key Vault).

Skills and Knowledge

• Proficiency in Python programming for data engineering tasks.
• Strong knowledge of SQL for data modeling, transformation, and querying.
• Working knowledge of Delta Lake fundamentals (e.g., MERGE, schema evolution, time travel).
• Working knowledge of Unity Catalog concepts, including catalogs, schemas, and permissions.
• Experience with Git-based version control and familiarity with CI/CD practices for data pipelines.
• Understanding of testing and deployment strategies for data pipelines, including productionizing ETL pipelines and Databricks SQL queries.

Education

• Bachelor’s degree in computer science, Information Systems, Data Engineering, or a closely related field, or equivalent professional experience.

Preferred Qualifications

• Experience implementing streaming data ingestion in Databricks (Structured Streaming, Auto Loader, Event Hubs).
• Experience with healthcare data (e.g., claims, EHR/clinical data, HL7/FHIR) and HIPAA/PHI handling.
• Exposure to Spark and Delta Lake performance tuning and compute cost optimization.
• Familiarity with dimensional modeling and SCD Type 1 and Type 2 patterns.
• Experience enabling BI consumption through Databricks SQL warehouses and tools such as Power BI.

Benefits

• Medical, dental, and vision insurance.
• Comprehensive employee assistance program.
• 401(k) retirement plan.
• Paid time off and holidays.
• Paid learning days.