[Hiring] Data Engineer @Livestock Information

Role Description

Are you an experienced Azure Data Engineer with a passion for building scalable, high-quality data solutions? Can you design and optimise data pipelines using tools like Azure Databricks and Data Factory to deliver real business impact? Do you enjoy collaborating with cross-functional teams to turn complex data into valuable insights? If so, we’d love to hear from you!

As a Data Engineer at LI, you will play a key role in delivering scalable, high-quality data solutions that support critical business needs. In this role, you’ll:

• Design and build new data services
• Lead migrations of existing solutions
• Ensure the ongoing performance and enhancement of live data platforms
• Work in a collaborative, agile environment with cross-functional teams including Product Owners, Business Analysts, Developers, and Testers
• Utilize a modern Azure data stack, including Databricks, Synapse, Data Factory, and Data Lake technologies
• Use tools like Azure DevOps to support end-to-end development and delivery

Responsibilities

• Designing & Implementing Data Storage Solutions on Azure: Select Azure data storage services appropriately and design data storage schemas, ensuring they are optimised for specific use cases.
• Building and Maintaining Data Pipelines for Data Integration and Processing: Build and maintain data pipelines for data movement and process orchestration. Collect data from various sources, transform data into a practical format and load it into appropriate data storage solutions on Azure using tools such as Azure Data Factory and Azure Databricks. Ensure data pipelines are scalable and efficient.
• Ensuring Data Quality & Accuracy Through Testing & Validation: Test and verify data at different phases of the data pipeline, from extraction through to loading.
• Optimising Data Processing Performance Through Tuning & Monitoring: Ensure data processing is efficient and scalable by tuning the data pipeline and monitoring performance; identify and resolve bottlenecks in the data pipeline and optimise data processing algorithms.
• Developing & Maintaining Data Models & Schemas: Develop and maintain data models and schemas optimised for specific use cases. Select the appropriate data modelling techniques and ensure the data schema is scalable and efficient.
• Collaborating With Other Teams to Provide Data for Analytics & Reporting: Work closely with other teams to provide data they need to perform their job functions.
• Ensuring Data Security & Privacy Prerequisites Are Followed: Ensure that data streamed and processed on Azure confirms with applicable data security and privacy requirements such as the General Data Protection Regulation (GDPR).
• Working with Large Datasets & Performing Data Analysis: Work with large datasets of hundreds of millions and billions of rows of data and perform data analysis.
• Staying updated on the latest Azure features, updates and best practices to ensure they are adopted effectively as appropriate.
• You may be required to undertake other duties assigned by your line manager as appropriate.

Qualifications

• Proven experience in Python programming for data engineering and analytics
• Strong proficiency in SQL, including the Microsoft T-SQL dialect
• Hands-on experience with Azure data storage solutions
• Practical knowledge of data integration and processing frameworks, such as Azure Databricks and Azure Data Factory
• Solid understanding of data modelling concepts and schema design best practices
• Strong analytical and problem-solving skills, with excellent attention to detail
• Effective communication skills and the ability to work collaboratively within cross-functional teams

Benefits

• A fully remote and flexible working set up
• 25 days annual leave, increasing by 1 day each year of service up to a maximum of 30 days (plus Bank Holidays)
• 1 Privilege Day
• 4 x annual salary Life Assurance
• Market leading pension scheme through Legal & General
• Enhanced Family Friendly Policies
• Excellent learning, training, and career development opportunities
• 24/7 access to our Employee Assistance Programme
• A diverse and inclusive culture where everyone is respected and valued