[Hiring] Senior Lead Data Engineer @Lighthouse

Role Description

The Senior Lead Data Engineer on the Prism team sets the technical direction for designs and delivers the core data engineering infrastructure for Lighthouse's eDiscovery AI products — including AI-enabled TAS (Technology Assisted Solutions) and Generative AI document analysis services. This engineer is the senior technical authority and hands-on lead for data engineering, owning the architecture of highly scalable, reliable, and fault-tolerant data pipelines for multi-million document corpora, operationalizing ML model workflows, and guiding the design of data systems that support the full eDiscovery AI lifecycle: from ESI ingestion and text extraction through encoder model training, document scoring, and LLM-based RAG pipelines. The role partners closely with data science, product, and engineering leaders and raises the technical bar across the team through architecture leadership, standards, and mentoring.

***This is a part-time role - 24 hours per week***

What will this person do?

• Set the technical direction and own the end-to-end architecture for data engineering across Prism eDiscovery AI products, ensuring designs are scalable, reliable, secure, and cost-effective.
• Lead the design, build, and management of data preprocessing, validation, and transformation pipelines that extract metadata and full text from heterogeneous ESI corpora at multi-million document scale.
• Lead the build and operation of Data Orchestration platforms that coordinate training data preparation, model scoring, embedding generation, and result delivery across eDiscovery AI workflows.
• Architect and implement scalable, performant data pipelines that cleanse, structure, and integrate disparate ESI datasets into optimal formats for encoder model training (BERT fine-tuning) and LLM context preparation.
• Direct the design and evolution of infrastructure for LLM/Generative AI applications: document chunking, embedding generation pipelines, vector database population, and retrieval layer optimization for RAG architectures.
• Define and drive data engineering standards, best practices, and design patterns, including code quality, testing, documentation, and design reviews.
• Establish the approach and tooling to monitor, debug, and analyze data pipeline health, data quality, and model serving data flows in production, and lead resolution of complex production issues.
• Design and govern portable, scalable data schemas for document metadata, model inputs/outputs, scoring results, and embeddings.
• Lead experiments to validate efficiency, throughput, and cost of modern data engineering stacks for legal AI workloads and translate findings into roadmap recommendations.
• Own technical recommendations regarding buy vs. build decisions for data infrastructure components and present them to engineering and product leadership.
• Partner with data science, product, and engineering stakeholders to shape requirements, plan, and sequence technical work, and align data engineering priorities with business goals.
• Mentor and provide technical guidance to data engineers, fostering skill growth and consistent engineering practices across the team.
• Perform other related duties as assigned.

Qualifications

• Degree in computer science or computer engineering with 8+ years of experience in a data engineering related field, including experience leading the technical design of large-scale data platforms (required).
• Demonstrated technical leadership: setting architecture direction, leading design reviews, and guiding engineers on complex, cross-team initiatives.
• Expertise building data pipelines for unstructured document data: text extraction, OCR processing, metadata extraction, and legal document format handling.
• Experience implementing big data workflows in cloud-native tech stacks (Azure preferred): Spark, Databricks, Airflow, Azure Data Factory, Kafka, and Kubernetes.
• Deep knowledge of data science and ML pipeline patterns: training data preparation, feature engineering, embedding generation, and MLOps operationalization.
• Experience with vector databases and semantic search infrastructure (Azure AI Search, Pinecone, or similar) for RAG pipeline support.
• Experience with relational and NoSQL databases, cloud-native storage technologies, and data serialization formats (Parquet, Delta Lake, JSON).
• Experience working with data scientists to operationalize encoder-based classification models and LLM applications in production.
• Strong communication skills, with the ability to explain complex technical trade-offs to technical and non-technical stakeholders.
• Proficiency with agile development methodologies.

Requirements

• Duties are performed in a typical office environment while at a desk or computer table.
• Duties require the ability to use a computer, communicate over the telephone, and read printed material, in a quiet and professional setting.
• Duties may require being on call periodically and working outside normal working hours (evenings and weekends).

Benefits

• This role will be eligible to participate in an annual bonus or incentive program.

Company Description

Lighthouse celebrates and thrives on diversity and is an Equal Opportunity Employer. We hire, train, and promote regardless of race, religion, color, national origin, sex, disability, age, veteran status, and other protected status as required by applicable law. We welcome any talents and contributions you can bring to the team and are deeply committed to growing an environment where everyone can feel safe, is respected, and can show up as themselves. Come as you are!

As required by applicable pay transparency laws, Lighthouse complies with compensation disclosure requirements for roles that may be hired in locations under these requirements. Factors that may be used to determine your actual salary may include a wide array of factors, including: your specific skills and experience, geographic location, or other relevant factors. The salary range for this position may be tailored to be lower or higher in different talent markets.

As a trailblazer and catalyst for change, Lighthouse rises to each opportunity to help our clients, and our people do what they do best—shine.