[Hiring] Machine Learning/AI Research Evaluator @Lightly AG

Role Description

We’re looking for experienced ML/AI researchers with a strong publication track record at top-tier conferences (NeurIPS, ICML, ICLR) to support an advanced AI research evaluation project. This is a remote, project-based contractor opportunity with flexible working hours.

• Critically evaluate AI-generated ML/AI research papers, assessing their scientific contributions, technical correctness, methodology, experimental design, and conclusions.
• Assess scientific novelty and significance, determining whether proposed approaches represent meaningful advancements over existing state-of-the-art research.
• Evaluate experimental rigor and validity, including baseline comparisons, benchmark selection, evaluation metrics, statistical evidence, and ablation studies.
• Identify methodological flaws, unsupported claims, inconsistencies, or misleading results that undermine the scientific validity of proposed contributions.
• Assess reproducibility and technical soundness, including whether reported findings are sufficiently supported by experimental evidence and, where available, accompanying code and implementation details.
• Review and verify relevant academic literature, identifying overlooked prior work, inaccurate citations, and unsupported novelty claims.
• Evaluate whether AI-generated research meets the scientific standards of top-tier ML/AI conferences, such as NeurIPS, ICML, and ICLR.
• Provide structured, evidence-based assessments of research quality, including strengths, weaknesses, critical technical concerns, and overall scientific merit.
• Apply detailed evaluation rubrics consistently, providing clear, well-reasoned justifications for your assessments.

Qualifications

• PhD in Machine Learning, Artificial Intelligence, Computer Science, Statistics, or a closely related technical field, or equivalent demonstrated research expertise.
• Published at least 3 peer-reviewed research papers at NeurIPS, ICML, and/or ICLR (hard requirement), ideally including first-author or significant research contributions.
• Substantial hands-on ML/AI research experience, including developing novel methods, designing experiments, analyzing results, and publishing original scientific work.
• Strong understanding of scientific methodology and experimental rigor, including appropriate baselines, statistical validity, benchmark design, ablation studies, and reproducibility.
• Ability to independently assess the novelty, significance, and technical soundness of advanced ML/AI research, identifying both meaningful scientific contributions and fundamental methodological weaknesses.
• Experience reviewing research papers for top-tier ML/AI conferences or journals, ideally NeurIPS, ICML, or ICLR — strongly preferred.
• Highly familiar with state-of-the-art ML/AI research and can critically assess new methods against existing literature and established benchmarks.
• Comfortable verifying scientific claims, including reviewing technical arguments, experimental evidence, citations, and, where relevant, code implementations.
• Strong analytical and scientific writing skills and can articulate nuanced technical judgments through concise, evidence-based feedback.
• Ability to consistently apply detailed evaluation guidelines and scoring rubrics while exercising independent expert judgment.
• Exceptional attention to detail, particularly when identifying subtle methodological flaws, unsupported conclusions, or discrepancies between claims and evidence.

Benefits

• Fully remote and flexible — work from anywhere.
• Part-time contractor role with flexible hours.
• Work on the evaluation of cutting-edge autonomous AI systems capable of conducting end-to-end scientific research.
• Apply your research expertise to help assess and advance the scientific capabilities of frontier AI agents.

Company Description

Lightly AG is a Zurich-based AI company and ETH/HSG spin-off, backed by Y Combinator and top-tier investors. We're funded by YCombinator and top investors.