[Hiring] Big Data Engineer @24-MAG

Role Description

We are sharing a specialised part-time consulting opportunity for experienced Big Data Engineers to contribute technical expertise to an advanced AI development project involving large-scale data pipelines, distributed systems, databases, and modern data-engineering workflows. Selected professionals will apply practical big-data engineering experience to complex data problems involving pipeline architecture, integration, transformation, distributed processing, storage, and system reliability.

The work requires strong Python skills, deep knowledge of database technologies, and the ability to design and evaluate scalable data solutions across modern engineering environments.

Key Responsibilities

• Big Data Architecture & Pipeline Engineering
  • Design, build, and maintain scalable data pipelines and big-data architectures
  • Develop reliable workflows for ingesting, transforming, processing, and delivering large volumes of data
  • Translate technical and business requirements into robust data-engineering solutions
  • Apply appropriate architectural patterns for distributed and high-volume data systems
  • Ensure data workflows remain scalable, maintainable, and suitable for production-oriented environments
• Python, Data Integration & Processing
  • Develop data-processing, automation, and integration solutions using Python
  • Implement ETL and data-transformation workflows across structured and unstructured data sources
  • Work with distributed processing technologies and modern big-data frameworks
  • Build reproducible and efficient programmatic workflows for large-scale data operations
  • Identify opportunities to improve processing efficiency, reliability, and maintainability
• Databases, Storage & Performance
  • Develop, manage, and optimise relational, NoSQL, distributed database, and storage systems
  • Apply data-modelling and data-warehousing principles to support scalable analytical and operational workloads
  • Monitor data-system performance and identify bottlenecks, failures, and reliability issues
  • Troubleshoot performance, availability, and integration problems across distributed data environments
  • Optimise storage, queries, and processing workflows for efficiency and resilience
• Data Quality, Governance & Technical Collaboration
  • Apply appropriate data-quality, security, and governance standards across engineering workflows
  • Collaborate with technical and cross-functional stakeholders to understand data requirements and delivery priorities
  • Document architectures, data flows, implementation decisions, and technical trade-offs
  • Communicate complex engineering concepts clearly to both technical and non-technical audiences
  • Support consistent engineering standards across remote and distributed project environments

Qualifications

• Proven professional experience in big-data engineering with hands-on responsibility for large-scale data pipelines and architectures
• Advanced proficiency in Python for data processing, automation, integration, and related engineering workflows
• Deep understanding of relational and NoSQL databases, including optimisation, management, and performance considerations
• Practical experience with distributed data-processing frameworks such as Hadoop, Spark, Flink, or comparable technologies
• Strong foundation in data modelling, ETL processes, data integration, and data-warehousing principles
• Ability to troubleshoot distributed systems, database performance, pipeline failures, and data-quality issues
• Excellent written and verbal communication skills
• Ability to explain technical concepts clearly to both engineering and non-technical stakeholders
• Strong attention to detail, initiative, and ability to work independently in remote environments
• Experience supporting fast-paced, startup-like, or globally distributed teams is advantageous
• Familiarity with cloud-based data platforms such as AWS, GCP, or Azure is advantageous
• Exposure to machine-learning operations, data-science workflows, or adjacent AI infrastructure is beneficial

Engagement Details

• Part-time independent contractor engagement
• Fully remote
• Compensation: $30–$80/hour
• Work will involve big-data architecture, pipeline development, Python-based processing, database optimisation, and distributed data systems
• Collaboration may involve technical, product, data, and other cross-functional stakeholders
• Project scope, workload, timing, and duration may evolve depending on project requirements
• Work must be completed without using confidential or proprietary information belonging to any employer, client, institution, or other third party