[Hiring] PySpark / SQL Engineer @Veracity

Role Description

We are seeking a skilled PySpark / SQL Engineer to support our Threat Detection team in building and migrating security analytics pipelines using Databricks. This role will focus on a platform migration project, moving detection rules and associated content from a legacy system into Databricks' native detection framework. You will be responsible for:

• Creating equivalent PySpark log pipelines
• Rule configuration files
• Unit tests
• Data validation checks
• Deploying these pipelines using Terraform

A strong background in data engineering, particularly with large-scale log analytics, is essential.

Qualifications

• 5-10+ years of experience as a PySpark / SQL Engineer, with a strong focus on data engineering and analytics
• Prior experience in building threat detection or log analytics pipelines using PySpark, SQL, and Databricks
• Hands-on experience with Terraform for deploying data infrastructure
• Proficient in PySpark for large-scale data processing and transformation
• Familiarity with cloud platforms such as AWS or Azure is preferred
• Strong analytical skills and attention to detail when working with complex datasets
• Proven ability to work effectively in collaborative, cross-functional teams
• Excellent verbal and written communication skills in English

Requirements

• Design and build threat detection pipelines using PySpark, SQL, and Databricks
• Support the migration of detection rules and content from a legacy platform to Databricks
• Create and maintain PySpark log pipelines and associated rule configuration files
• Write unit tests to ensure pipeline accuracy and stability
• Perform data validation checks to ensure data integrity
• Deploy pipelines and infrastructure using Terraform
• Optimize existing data workflows and queries for performance and scalability
• Collaborate with cross-functional teams to understand data requirements and ensure alignment with detection objectives

Company Description