[Hiring] Research Crawling Engineer @True Talent Solutions Partners

Role Description

Join a data infrastructure team building large-scale web crawling systems that supply public web data for AI research and model development. You will work closely with research teams to shape data collection around their modeling needs.

• Build and maintain web crawlers across diverse domains, including social, travel, and multilingual websites.
• Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.
• Adapt collection methods to anti-bot protections, rate limits, and dynamic, JavaScript-heavy sites.
• Build pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
• Develop and maintain datasets for research and model training in collaboration with research teams.
• Monitor crawl coverage, performance, and data quality, and respond to changing web environments.
• Optimize crawling infrastructure for cost, latency, and reliability across cloud and bare-metal environments.

Qualifications

• At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, with 2 or more years considered for candidates with a PhD.
• Experience operating data pipelines at terabyte scale and building distributed or parallel systems.
• Strong programming skills in Go, Rust, Python, Java, or C++.
• Deep knowledge of HTTP, networking, and browser behavior, with experience handling anti-bot systems, rate limits, or JavaScript-heavy sites.
• Experience with headless browsers such as Playwright or Puppeteer, proxy systems, and IP rotation.
• Background in data quality evaluation, web data processing, NLP pipelines, or ML dataset curation is beneficial.
• Experience running data workloads on cloud platforms or bare-metal infrastructure.

Benefits

• Base salary: $160,000 to $250,000 USD annually.
• Visa sponsorship is not available.

Location

Remote.