[Hiring] Senior Databricks Engineer @CNX

Role Description

The Senior Data Engineer — Databricks will own the end-to-end data lifecycle, from analysis and solution design through implementation, orchestration, governance, and optimization. This role will build modern data solutions on the Databricks Data Intelligence Platform across Microsoft Azure, AWS, or GCP, using Databricks, Python, PySpark, and SQL to deliver scalable ingestion, transformation, modeling, and pipeline orchestration.

As a senior engineer, you will lead solution design, establish engineering standards, and guide other data engineers in building reliable, governed, and production-ready data pipelines that support analytics, reporting, and AI/ML use cases.

Responsibilities

• Design, develop, and maintain robust batch and streaming data pipelines for extracting, transforming, and loading data from multiple sources into the Databricks lakehouse on Azure.
• Automate and streamline data ingestion using Auto Loader, Lakeflow Connect, and Azure services such as Azure Data Factory and Event Hubs.
• Build and manage pipelines with Lakeflow Spark Declarative Pipelines, formerly Delta Live Tables, including data quality expectations and change data capture.
• Perform complex data manipulation and transformation using PySpark and Spark SQL to prepare data for advanced analytics, reporting, and AI/ML use cases.
• Design and implement workflow orchestration and scheduling using Lakeflow Jobs, formerly Databricks Workflows, to support reliable and efficient data operations.
• Create scalable lakehouse data models using medallion architecture — bronze, silver, and gold — and dimensional modeling, including SCD Type 1 and Type 2.
• Implement data governance and security in Unity Catalog, including access controls, lineage, row filters, and column masks for sensitive data.
• Optimize performance and cost across Delta tables, Spark workloads, serverless and classic compute, and SQL warehouses.
• Establish CI/CD and testing practices for data pipelines using Git folders, Declarative Automation Bundles, formerly Databricks Asset Bundles, and Azure DevOps or GitHub Actions.
• Monitor pipeline health and data quality using system tables and data quality monitoring; lead root-cause analysis and resolution of production issues.
• Lead design and code reviews, define engineering standards, and mentor data engineers across the team.
• Collaborate with cross-functional teams, including business stakeholders and IT, to ensure data solutions meet business, performance, and project requirements.

Qualifications

• 10+ years of professional experience in data engineering, data and analytics, or data warehousing.
• 5+ years of hands-on experience working with Databricks in a data engineering capacity, including production workloads.
• 5+ years of experience with Python, including manipulating large datasets using PySpark DataFrames, Spark SQL, and/or the pandas API on Spark.
• 3+ years of data engineering experience on Microsoft Azure, including services such as ADLS Gen2, Azure Data Factory, Key Vault, and Microsoft Entra ID.
• Strong experience designing and developing ETL/ELT pipelines for batch and streaming data workloads.
• Experience with Databricks lakehouse architecture, Delta tables, Unity Catalog, Lakeflow Jobs, and Lakeflow Spark Declarative Pipelines.
• Experience implementing scalable data models using medallion architecture, dimensional modeling, and slowly changing dimensions.
• Experience with CI/CD and testing practices for data pipelines using Azure DevOps or GitHub Actions.
• Ability to lead technical design discussions, conduct code reviews, define engineering standards, and mentor other engineers.
• Strong collaboration skills with the ability to work across business, IT, and technical stakeholder groups.

Requirements

• Preferred certification: Databricks Data Engineer Professional.
• The base salary for this position is $106,087 – $151,554 plus incentives that align with individual and company performance.
• Actual salaries will vary based on work location, qualifications, skills, education, experience, and competencies.
• Benefits available to eligible employees in this role include medical, dental, and vision insurance, comprehensive employee assistance program, 401(k) retirement plan, paid time off and holidays, and paid learning days.
• The deadline to apply for this position is: 11/01/2026.
• This position is for an existing, immediate vacancy. We are currently seeking to fill this role with an individual who can start as soon as possible.

Benefits

• Medical, dental, and vision insurance.
• Comprehensive employee assistance program.
• 401(k) retirement plan.
• Paid time off and holidays.
• Paid learning days.