[Hiring] Data Engineer @Beyondsoft Consulting

Role Description

We are seeking an experienced Data Engineer in India with strong expertise in Python, SQL, and Azure Databricks to design, build, and maintain scalable cloud-based data solutions. The ideal candidate will have hands-on experience developing production-grade data pipelines, transforming and managing large datasets, and building reliable data solutions on Azure. In addition, the candidate should have a good understanding of Large Language Models (LLMs) and Generative AI, with experience or strong interest in integrating LLM capabilities into data and AI solutions.

You will work closely with engineers, analysts, and business stakeholders to build reliable data platforms and enable data-driven and AI-powered applications.

• Design, develop, and maintain scalable data pipelines using Azure Databricks, Python, and SQL.
• Develop ETL/ELT workflows for ingesting, transforming, validating, and publishing data from multiple data sources.
• Build and maintain Azure Databricks notebooks, workflows, and jobs for production data processing.
• Develop reusable Python components for data ingestion, transformation, automation, and integration.
• Write and optimize complex SQL queries for data transformation, analytics, and reporting.
• Work with large-scale structured and semi-structured datasets.
• Design and maintain reliable data models and data processing workflows.
• Monitor data pipelines and troubleshoot data quality, reliability, and performance issues.
• Implement appropriate logging, monitoring, error handling, and data quality controls.
• Explore and integrate LLM and Generative AI capabilities into data engineering solutions and applications.
• Develop data preparation and processing workflows to support LLM-based applications.
• Work with LLM APIs and services to enable use cases such as information extraction, summarization, classification, and intelligent data processing.
• Collaborate with engineering and business teams to identify opportunities where LLM technologies can improve data processing and automation.
• Participate in technical design, code reviews, testing, deployment, and production support.
• Follow engineering best practices for source control, CI/CD, security, documentation, and maintainability.

Qualifications

• Strong hands-on experience with Azure Databricks.
• Strong programming skills in Python.
• Strong proficiency in SQL, including complex queries, data transformation, and query optimization.
• Hands-on experience designing and developing production-grade ETL/ELT data pipelines.
• Experience working with large-scale structured and semi-structured datasets.
• Solid understanding of data modeling, data warehousing, and modern cloud-based data architectures.
• Experience troubleshooting and optimizing data pipelines and data processing workloads.
• Good understanding of Large Language Models (LLMs) and Generative AI concepts.
• Familiarity with integrating LLM APIs or AI services into applications or data processing workflows.
• Understanding of common LLM concepts such as prompt engineering, embeddings, vector search, and Retrieval-Augmented Generation (RAG).
• Strong problem-solving and analytical skills.
• Good communication and collaboration skills.

Requirements

• Python
• SQL
• Azure Databricks
• ETL/ELT
• Data pipeline development
• Data processing and transformation
• LLM/generative AI fundamentals
• LLM API integration
• Prompt engineering
• RAG fundamentals

Preferred Qualifications

• Experience with Azure OpenAI Service or other enterprise LLM platforms.
• Experience developing RAG-based applications or solutions.
• Familiarity with vector databases or vector search technologies.
• Experience preparing and managing enterprise data for LLM-based applications.
• Experience with Azure Data Factory for data integration and orchestration.
• Experience with Azure Data Lake Storage (ADLS).
• Experience with Azure SQL, Azure Synapse Analytics, or other cloud-based data platforms.
• Experience with REST APIs and integrating external services into data pipelines.
• Experience with CI/CD, Git, and Azure DevOps.
• Knowledge of data security, governance, access control, and data quality best practices.
• Understanding of responsible AI, data privacy, and security considerations when working with LLMs.
• Relevant Azure, Databricks, or AI certification is a plus.

Preferred Skills

• Azure OpenAI
• Embeddings/vector search
• RAG application development
• Azure Data Factory
• Azure Data Lake Storage
• Azure SQL/Azure Synapse Analytics
• REST API integration
• Git/Azure DevOps/CI/CD
• Data governance and security