[Hiring] Senior Data Engineer @Cynet Systems

Role Description

The organization is seeking a Senior Data Engineer focused on Middle Office Data. The candidate will design, build, and optimize scalable data ingestion, transformation, and workflow orchestration pipelines within AWS environments, supporting business processes, analytics, reporting, and downstream consumption.

Key Responsibilities

• Onboard new datasets required to support Middle Office business processes, analytics, reporting, and downstream applications.
• Work with business and technology stakeholders to understand data requirements, source systems, data structures, business rules, and downstream consumption patterns.
• Design scalable data ingestion, transformation, and processing solutions using AWS and Apache Spark/PySpark.
• Develop and maintain production-grade data pipelines using services such as AWS EMR, AWS Glue, Amazon S3, and Amazon Athena.
• Design and work with large-scale data lake tables using technologies such as Apache Hudi and Apache Iceberg.
• Build workflow orchestration and scheduling solutions using Apache Airflow or equivalent orchestration technologies.
• Perform data modeling and design appropriate schemas, partitioning strategies, data structures, and transformation layers for analytical and operational use cases.
• Build pipelines with appropriate operational capabilities, including monitoring, logging, alerting, data-quality validation, failure handling, retries, and recovery mechanisms.
• Develop clean, reusable, testable, and maintainable code primarily using Python and PySpark.
• Implement and maintain automated deployment processes using CI/CD and infrastructure-as-code practices, including GitHub Actions and Terraform.
• Troubleshoot data-quality, pipeline-performance, and production issues and identify appropriate long-term solutions.
• Produce appropriate technical documentation covering data lineage, pipeline design, business rules, dependencies, and operational procedures.
• Collaborate effectively with data platform, business, Middle Office, and downstream technology teams.

Qualifications

• Strong hands-on experience with AWS data and analytics services, particularly AWS EMR, AWS Glue and Glue Data Catalog, Amazon Athena, Amazon S3, AWS IAM, and general AWS security concepts.
• Understanding of how AWS services work together when designing and operating enterprise-scale data pipelines.
• Expert-level or strong hands-on experience with Apache Spark, PySpark, distributed data processing, large-scale dataset transformation and optimization, Spark performance tuning and troubleshooting, and partitioning strategies.
• Strong experience working with modern data lake/table formats, particularly Apache Hudi and Apache Iceberg.
• Understanding of concepts such as schema evolution, partitioning, incremental processing, upserts, snapshots, and management of large analytical tables.
• Strong programming skills in Python and PySpark.
• Ability to consistently produce clean, modular, maintainable, and production-quality code, with experience in object-oriented design, testing, exception handling, logging, code reviews, and Git-based workflows.
• Strong hands-on experience with Apache Airflow or a comparable enterprise orchestration platform, including designing DAGs, workflow dependencies, scheduling, retry strategies, backfill processes, monitoring, and parameterized workflows.
• Experience with Amazon MWAA is an asset.
• Strong understanding of data modeling, data pipeline architecture, data lake architecture, schema design, data partitioning, incremental data processing, historical data management, data quality, and data lineage.
• Capability to translate business requirements into a scalable technical data model and an operationally supportable data pipeline.
• Working knowledge of modern DevOps and infrastructure-as-code practices, including Terraform, GitHub Actions, Git, CI/CD pipelines, automated testing and deployment, and environment configuration.

Preferred Qualifications

• Previous experience building data platforms or data products within financial services, asset management, capital markets, or investment management.
• Experience supporting Middle Office, investment operations, risk, portfolio management, or trading-related datasets.
• Experience onboarding datasets from external vendors, APIs, files, databases, or enterprise data platforms.
• Understanding of data governance, lineage, entitlement, security, and audit requirements in an enterprise environment.
• Experience designing pipelines that process both historical and incremental/daily datasets.
• Experience troubleshooting and optimizing large-scale Spark workloads.
• Experience operating production data pipelines with defined SLAs and operational support requirements.

Education

• Bachelor's or Master's degree in Computer Science, Computer or Electrical Engineering, Mathematics, or a related field.