[Hiring] Big Data Engineer @24-MAG

Role Description

We are sharing a specialised part-time consulting opportunity for experienced Big Data Engineers to contribute technical expertise to an advanced AI development project involving large-scale data pipelines, distributed systems, databases, and modern data-engineering workflows.

Selected professionals will apply practical big-data engineering experience to complex data problems involving:

• Pipeline architecture
• Integration
• Transformation
• Distributed processing
• Storage
• System reliability

The work requires strong Python skills, deep knowledge of database technologies, and the ability to design and evaluate scalable data solutions across modern engineering environments.

Key Responsibilities

• Big Data Architecture & Pipeline Engineering
  • Design, build, and maintain scalable data pipelines and big-data architectures
  • Develop reliable workflows for ingesting, transforming, processing, and delivering large volumes of data
  • Translate technical and business requirements into robust data-engineering solutions
  • Apply appropriate architectural patterns for distributed and high-volume data systems
  • Ensure data workflows remain scalable, maintainable, and suitable for production-oriented environments
• Python, Data Integration & Processing
  • Develop data-processing, automation, and integration solutions using Python
  • Implement ETL and data-transformation workflows across structured and unstructured data sources
  • Work with distributed processing technologies and modern big-data frameworks
  • Build reproducible and efficient programmatic workflows for large-scale data operations
  • Identify opportunities to improve processing efficiency, reliability, and maintainability
• Databases, Storage & Performance
  • Develop, manage, and optimise relational, NoSQL, distributed database, and storage systems
  • Apply data-modelling and data-warehousing principles to support scalable analytical and operational workloads
  • Monitor data-system performance and identify bottlenecks, failures, and reliability issues
  • Troubleshoot performance, availability, and integration problems across distributed data environments
  • Optimise storage, queries, and processing workflows for efficiency and resilience
• Data Quality, Governance & Technical Collaboration
  • Apply appropriate data-quality, security, and governance standards across engineering workflows
  • Collaborate with technical and cross-functional stakeholders to understand data requirements and delivery priorities
  • Document architectures, data flows, implementation decisions, and technical trade-offs
  • Communicate complex engineering concepts clearly to both technical and non-technical audiences
  • Support consistent engineering standards across remote and distributed project environments

Qualifications

• Proven professional experience in big-data engineering with hands-on responsibility for large-scale data pipelines and architectures
• Advanced proficiency in Python for data processing, automation, integration, and related engineering workflows
• Deep understanding of relational and NoSQL databases, including optimisation, management, and performance considerations
• Practical experience with distributed data-processing frameworks such as Hadoop, Spark, Flink, or comparable technologies
• Strong foundation in data modelling, ETL processes, data integration, and data-warehousing principles
• Ability to troubleshoot distributed systems, database performance, pipeline failures, and data-quality issues
• Excellent written and verbal communication skills
• Ability to explain technical concepts clearly to both engineering and non-technical stakeholders
• Strong attention to detail, initiative, and ability to work independently in remote environments
• Experience supporting fast-paced, startup-like, or globally distributed teams is advantageous
• Familiarity with cloud-based data platforms such as AWS, GCP, or Azure is advantageous
• Exposure to machine-learning operations, data-science workflows, or adjacent AI infrastructure is beneficial

Engagement Details

• Part-time independent contractor engagement
• Fully remote
• Compensation: $30–$80/hour
• Work will involve big-data architecture, pipeline development, Python-based processing, database optimisation, and distributed data systems
• Collaboration may involve technical, product, data, and other cross-functional stakeholders
• Project scope, workload, timing, and duration may evolve depending on project requirements
• Work must be completed without using confidential or proprietary information belonging to any employer, client, institution, or other third party