Now Hiring: Founding Lead Researcher – Medical AI Evaluations | Fully Remote | $100K–$300K USD

The rapidly evolving field of medical artificial intelligence is creating new opportunities for researchers who can combine advanced machine learning expertise with rigorous scientific evaluation. Sophont is hiring a Founding Lead Researcher – Medical AI Evaluations, a fully remote leadership position focused on developing the evaluation systems and standards needed to assess medical AI models.

With a salary range of $100,000–$300,000 USD per year, plus equity and benefits, the position offers an opportunity for an experienced researcher to help shape how medical foundation models are evaluated across the industry.

About Sophont

Sophont is a public-benefit corporation developing open, universal medical AI designed to understand areas including pathology, neuroimaging and clinical text. The organisation works alongside MedARC, its open-source medical AI research community, to advance transparent research and develop frontier AI models for medicine.

The company has already established a research track record, with open-source work published in respected venues including NeurIPS, ICML, CVPR and Nature Biomedical Engineering. Its approach emphasises transparency, open science and collaboration.

Sophont describes its working environment as closer to a live research laboratory than a conventional startup. Researchers and contributors work together to develop ideas, conduct experiments and build systems in the open.

About the Medical AI Evaluations Role

The Founding Lead Researcher will take ownership of the evaluation layer across Sophont’s medical AI portfolio.

This includes evaluating language models, pathology foundation models, neuroimaging models, multimodal systems, retrieval tools, agentic systems and clinical-facing products.

The successful candidate will help establish standards that can influence how researchers measure progress, how partners assess AI systems and how medical technologies are determined to be ready for real-world applications.

The role builds on projects such as Medmarks, which focuses on medical LLM evaluation, as well as Brainmarks for fMRI foundation models, Nanopath for pathology evaluation and OpenMidnight for open pathology foundation modelling.

Key Responsibilities

The researcher will be responsible for developing reusable evaluation methodologies across a wide range of medical AI applications.

Key duties include:

• Building benchmark infrastructure for public and internal leaderboards.
• Developing reproducible evaluation jobs and model comparison systems.
• Designing evaluation datasets with researchers, clinicians and community contributors.
• Establishing annotation rubrics, quality-control procedures and leakage checks.
• Measuring inter-rater reliability and data quality.
• Developing standards for source-grounded AI systems.
• Evaluating retrieval quality, evidence recall and evidence precision.
• Assessing uncertainty calibration and medical fact verification.
• Designing clinically meaningful evaluation protocols.
• Conducting statistically rigorous model comparisons.
• Performing subgroup analyses, robustness checks and ablation studies.
• Reporting negative results clearly and scientifically.
• Translating research findings into papers, technical reports, blog posts and partner materials.
• Leading MedARC contributors through reproducible research workflows.

Who Should Apply?

Sophont is seeking an exceptional researcher with experience in machine learning evaluation, benchmarking, model validation or scientific measurement infrastructure.

Applicants should have a strong understanding of statistics, experimental design, data quality analysis and scientific interpretation. Strong Python and software engineering skills are also important, particularly experience working with data pipelines, reproducible experiments and evaluation infrastructure.

Candidates should be comfortable working with systems such as LLMs, agents, computer vision models or biomedical foundation models and understand the challenges involved in evaluating complex AI systems.

The organisation is particularly interested in people who can carefully reason about ground truth in noisy biomedical datasets and develop reliable annotation protocols.

Preferred Experience

Experience with medical, biological, clinical, pathology or neuroimaging datasets would be valuable.

Additional experience with retrieval-augmented generation, hybrid search, BM25, dense retrieval, ColBERT-style retrieval, LLM-as-a-judge systems or human evaluation pipelines would be an advantage.

Sophont also values experience building public benchmarks, leaderboards, crowdsourcing platforms or annotation systems.

Candidates with experience in GPU clusters, distributed evaluation, experiment tracking or model-serving infrastructure may also stand out.

A strong publication or open-source record in machine learning, biomedical AI or evaluation methodology is desirable, as is experience leading open-source research communities.

Fully Remote Position

The role is fully remote, with employees able to largely determine their own working hours as long as they can participate in core collaboration meetings.

This flexibility makes the opportunity particularly attractive to experienced researchers who prefer distributed, asynchronous and research-driven environments.

Sophont also encourages employees to publish their work, contribute to open-source projects and participate in workshops, hackathons and conferences.

Salary and Benefits

The position offers an annual salary range of $100,000 to $300,000 USD, depending on experience and qualifications.

Benefits include:

• Meaningful equity.
• OpenAI and Anthropic subscriptions.
• 401(k) retirement plan with a 4% match.
• Medical insurance.
• Dental insurance.
• Vision insurance.
• Basic life insurance.
• Opportunities to publish and open-source research.

How to Apply

Interested candidates should prepare their CV and apply directly by email. Applicants are asked to use the subject line “[Your Full Name] CV – Medical AI Evaluations” when submitting their application.

This is a particularly strong opportunity for an experienced medical AI or machine learning researcher who wants to influence not only individual models, but also the standards used to determine whether medical AI systems are reliable, scientifically sound and clinically meaningful.

With its emphasis on open science, reproducibility and rigorous evaluation, the position offers the chance to contribute to a growing field while working with researchers and contributors across a global open-source community.

LEARN MORE & APPLY HERE

Join Our WhatsApp Community

Disclaimer: Remote Job Opportunities (RJO) is not the  organization offering this opportunity. For any inquiries, please contact the responsible organization directly. Please do not send your applications to RJO, as we are unable to process them. Due to the high volume of  emails we receive daily, we may not be able to respond to all inquiries. Thank you for your understanding.