[Hiring] Senior Technical Lead – Azure Data Engineer @USEReady Technology

Role Description

We are looking for a Senior Technical Lead – Azure Data Engineer with 8–10 years of experience and strong hands-on expertise in Microsoft Azure, Databricks, ADF, Python, PySpark, and SQL. The ideal candidate will have strong experience designing and implementing scalable data engineering solutions, architecting Databricks environments, optimizing data pipelines and workloads, and implementing data governance and security using Unity Catalog.

The candidate should be comfortable working with large-scale distributed data platforms, mentoring engineering teams, and collaborating with business and technical stakeholders to deliver high-quality data solutions.

Key Responsibilities

• Architect and design scalable Azure Databricks environments and data engineering solutions.
• Design and implement robust, scalable, and optimized data pipelines using Databricks and Azure Data Factory (ADF).
• Develop Databricks notebooks using Python, PySpark, and SQL.
• Implement and manage Unity Catalog for data governance, security, access control, and data discovery.
• Design data models and define schema strategies, data structures, and pipeline organization.
• Build and maintain Delta Lake tables and data models supporting analytics and BI workloads.
• Optimize Databricks jobs, Spark workloads, SQL queries, and data pipelines for performance, scalability, and cost efficiency.
• Apply knowledge of Apache Spark internals to troubleshoot and tune distributed data processing workloads.
• Develop complex and optimized SQL queries, including efficient joins and processing of large datasets.
• Implement data quality, monitoring, validation, and reconciliation mechanisms.
• Work with Azure data services including Azure Data Lake Storage (ADLS), Azure Synapse Analytics, and Azure SQL Database.
• Implement data governance, security, access controls, and data lineage.
• Troubleshoot pipeline failures and performance issues and ensure reliability and uptime.
• Collaborate with stakeholders to understand business requirements and translate them into scalable data solutions.
• Identify opportunities to automate manual processes and improve overall data platform efficiency.
• Mentor junior and mid-level engineers and establish best practices across Azure and Databricks development.

Qualifications

• 8–10 years of overall experience in Data Engineering, with strong focus on Azure.
• Strong hands-on experience with Azure Databricks.
• Excellent experience with Python and PySpark.
• Strong knowledge of Apache Spark architecture, internals, and performance tuning.
• Strong SQL skills, including complex queries, query optimization, and large-scale data processing.
• Strong experience with Azure Data Factory (ADF).
• Hands-on experience with Databricks Notebooks using Python and SQL.
• Experience with Unity Catalog, data governance, security, and access control.
• Strong understanding of Delta Lake.
• Experience with Azure Data Lake Storage (ADLS).
• Good understanding of data warehousing and data modeling, including Star and Snowflake schemas.
• Experience working with large-scale distributed data processing environments.
• Strong troubleshooting and performance optimization skills.

Requirements

• Experience with Azure Synapse Analytics.
• Experience with Azure SQL Database.
• Databricks or Azure Data Engineering certifications.
• Experience implementing enterprise-level data governance and data lineage.
• Experience mentoring technical teams and driving engineering best practices.
• Strong stakeholder management and communication skills.