[Hiring] Data Engineer @Irth Solutions

Role Description

We are looking for a Data Engineer to design, build, and maintain data ingestion and processing pipelines in Databricks, transforming high-volume external data sources into clean, structured, reliable inputs for downstream analysis and intelligence. This role will primarily support our Stakeholder Engagement offering, working closely with our Data Scientist and application teams.

Key responsibilities

• Data Pipeline Development (Primary Responsibility)
  • Design, build, and maintain ingestion pipelines from high-volume external APIs, capable of running continuously and reliably at scale.
  • Implement ingestion and transformation workflows using Databricks (Spark/PySpark, SQL, Delta Live Tables), applying medallion architecture patterns (Bronze → Silver → Gold) to move from raw ingested content to clean, structured, analysis-ready data.
  • Build the infrastructure for deduplication and relevance filtering of incoming content, implementing filtering logic and quality criteria defined in collaboration with the Data Scientist.
  • Implement schema evolution handling and data validation rules as data sources and formats change over time.
• Platform & Storage Implementation
  • Configure and manage Delta Lake storage structures, tables, partitions, and optimization routines (OPTIMIZE, Z-ORDER, VACUUM).
  • Design and evolve data schemas that balance query performance, cost, and maintainability as data volume grows.
  • Maintain clear metadata and documentation of table structures to support easy consumption by the Data Science and application teams.
• Reliability, Monitoring & Operational Support
  • Ensure pipeline reliability and observability: error handling, retries, monitoring, and alerting for a continuously running system.
  • Adapt pipelines to evolving external API contracts, rate limits, authentication changes, and new data sources.
  • Troubleshoot pipeline failures, perform recovery, and tune performance as needed.
• Orchestration & Automation
  • Build, schedule, and monitor workflows using Databricks Workflows, Delta Live Tables, or similar orchestration tools.
  • Contribute to CI/CD pipelines for code deployment, versioning, and environment management.
• Collaboration & Documentation
  • Work closely with the Data Scientist to expose clean, well-structured data feeding LLM/NLP pipelines and downstream models.
  • Participate in technical decisions around data architecture and propose structuring solutions as the team's needs evolve.
  • Document pipelines, data dictionaries, job schedules, and transformation logic.
  • Support the onboarding of new data sources and pipelines as the product expands to additional solution areas.

Qualifications

• 3 to 5 years of experience in data engineering, with solid experience building and operating production-grade data pipelines.
• Familiarity with data modeling, data quality, and schema evolution.
• Solid understanding of data pipeline reliability practices: monitoring, alerting, and handling failures gracefully in a continuously running system.
• Hands-on experience with Databricks (or an equivalent Spark-based environment): schema design, Delta Lake, performance tuning, and pipeline orchestration.
• Experience with at least one major cloud (Azure preferred; AWS/GCP also beneficial).
• Experience integrating with external APIs at scale: authentication, pagination, rate limiting, retries, error handling.
• Strong proficiency in Python and SQL.
• Comfortable working with unstructured/semi-structured text data at scale.

Requirements

• Strong preference for candidates residing in Quebec, with fluency in French (spoken and written) as a strong asset in addition to English.

Nice to have Qualifications

• LLM prompting experience and/or basic understanding of AI/NLP concepts.
• Exposure to medallion architecture or lakehouse best practices.
• Experience with orchestration frameworks (ADF, Workflows, Airflow, DBX, etc.).
• Experience with CI/CD tools and version control (Git, GitHub Actions or equivalent).
• Basic understanding of security practices: RBAC, encryption, credential management.
• Databricks certification (Data Engineer Associate or equivalent).

Compensation

The salary range for this role is CAD $75000. This range reflects the base salary only and does not include any additional compensation components. Any offer of employment is dependent on several factors, including, but not limited to, the candidate’s experience, skills, qualifications, and location.

Benefits

• Competitive Salary – A competitive compensation package based on experience and qualifications.
• Medical, Dental, and Vision Insurance – Comprehensive insurance coverage to support you and your family.
• 401(k) Plan with Company Match.
• Generous Paid Time Off (PTO) – Time off to support work-life balance and personal needs.
• Company-Paid Holidays – Paid holidays throughout the year.
• Flexible Work Options – Work-from-home opportunities are available, depending on role and business needs.
• On-Call Compensation – Additional pay for eligible on-call shifts.