[Hiring] Lead Databricks Engineer @Sentara Health

Role Description

Sentara Health is looking for a Remote Lead Databricks Engineer. This is a remote position available in various states.

We are looking for a Senior Data Engineer to help build and scale a modern, metadata-driven data platform on Azure. This role focuses on:

• Designing reusable data ingestion frameworks
• Enabling governance at scale
• Integrating with modern data catalog and access patterns

The ideal candidate is hands-on, thinks in terms of platforms rather than pipelines, and has strong experience with Databricks, Python, and cloud-native data architectures.

Key Responsibilities

• Design and develop scalable, reusable data ingestion frameworks using Azure Databricks and PySpark
• Build metadata-driven pipelines that decouple configuration from execution
• Implement and maintain Bronze, Silver, and Gold data layers following Medallion architecture
• Integrate data pipelines with data catalog and governance tools (e.g., DataHub, Purview)
• Implement schema enforcement, data contracts, and data quality checks within pipelines
• Work with ingestion tools (e.g., Fivetran) and file-based ingestion patterns
• Enable automation for dataset onboarding using configuration-driven approaches
• Collaborate with architecture, governance, and business teams to ensure scalable and governed data access
• Contribute to evolving toward event-driven and near real-time ingestion patterns
• Support CI/CD and DevOps practices for data platform components

Qualifications

• Bachelor's degree
• 8+ years of experience in data engineering or data platform development with degree
• 10+ years of relevant experience without a degree
• Strong hands-on experience with Azure Databricks and PySpark
• Proficiency in Python for data engineering and framework development
• Experience with Delta Lake and Lakehouse architecture
• Strong understanding of Medallion architecture (Bronze, Silver, Gold)
• Experience designing reusable and scalable data ingestion frameworks
• Experience working with data catalog / governance tools (DataHub, Purview, Collibra, etc.)
• Solid understanding of data lineage, schema evolution, data contracts, and data quality frameworks
• Experience working with cloud storage (ADLS Gen2) and distributed data processing
• Familiarity with CI/CD pipelines and Git-based workflows

Benefits

• Medical, Dental, Vision plans
• Adoption, Fertility and Surrogacy Reimbursement up to $10,000
• Paid Time Off and Sick Leave
• Paid Parental & Family Caregiver Leave
• Emergency Backup Care
• Long-Term, Short-Term Disability, and Critical Illness plans
• Life Insurance
• 401k/403B with Employer Match
• Tuition Assistance – $5,250/year and discounted educational opportunities through Guild Education
• Student Debt Pay Down – $10,000
• Reimbursement for certifications and free access to complete CEUs and professional development
• Pet Insurance
• Legal Resources Plan
• Opportunity to earn an annual discretionary bonus if established system and employee eligibility criteria is met