OpenAI and Ligo today released the BioBasicsBench benchmark, a 100‑task suite designed to measure how AI agents handle routine wet‑lab analysis. The benchmark focuses on the everyday experimental reasoning scientists use when reading gels, inspecting plates, interpreting microscopy and working with instrument outputs, testing whether models can interpret raw experimental evidence, detect errors and determine when data do not support a conclusion.
BioBasicsBench preserves the raw forms of evidence researchers use at the bench and draws tasks from real experiments. Inputs in the suite include gel and blot images, colony plate photographs, microscopy stacks, sequencing reads, plate‑reader exports, instrument screens, short protocol videos, and bench and reagent photos. OpenAI provides counts for each input type: 31 tasks include gels and blots; 27 include protocols, notes and specifications; 20 include tables and instrument exports; 18 include bench and reagent photos; 14 include microscopy; 13 include instrument screens and plots; 12 include sequencing files; 7 include protocol videos; 7 include agar plates and colonies; and 21 include reference sheets to support interpretation.
Task construction and grading were intended to reflect realistic laboratory needs. Tasks were drawn from scientists’ own experiments and paired with expert‑written grading rubrics that allocate credit to individual criteria, so the benchmark can distinguish fully correct conclusions from partially correct analyses. Each task underwent multiple reviews: an internal check by a Ligo scientist who did not author the task, an external review by a PhD‑level wet‑lab researcher, and a final internal review. Answers are scored against the rubrics, and the benchmark reports performance as a mean reward — the average task score expressed as a percentage of the available credit — with 95% confidence intervals computed by resampling whole tasks while keeping five attempts per task together.
OpenAI evaluated seven models on BioBasicsBench across 37 model‑and‑reasoning‑effort settings. The models tested were GPT‑5.2, GPT‑5.4, GPT‑5.5, GPT‑5.6 Sol, GPT‑6.1 Sol, GPT‑6 Luna and GPT‑6 Astra. For each setting the evaluation ran five attempts on each of the 100 tasks, producing a total of 18,500 results. At the highest reasoning‑effort setting reported for all models (xhigh), GPT‑6 Astra achieved the strongest result, with a mean reward of 66.7% (95% CI 60.3–73.1). Other xhigh results reported include GPT‑6.1 Sol at 62.0% (95% CI 55.6–68.4), GPT‑5.6 Sol at 43.8% (95% CI 37.6–50.0), GPT‑6 Luna at 38.3% (95% CI 32.3–44.5), GPT‑5.5 at 35.4% (95% CI 29.7–41.0), GPT‑5.4 at 33.2% (95% CI 28.3–38.4) and GPT‑5.2 at 31.2% (95% CI 26.2–36.3).
OpenAI attributes a portion of GPT‑6 Astra’s gains to stronger multimodal handling, suggesting that native multimodal capability assists routine wet‑lab analysis. The company highlights three areas where Astra shows measurable progress relative to earlier models: recognizing the limits of what experimental data show, handling multimodal inputs more reliably, and maintaining experimental context across analyses. Concrete examples in the evaluation include Astra correctly rejecting a control plate that had been labeled with the wrong antibiotic, discerning when overlapping fluorescence channels could confound a dead‑cell signal, and identifying a z‑stack sequence when cells move in and out of focus.
The benchmark also exposes important failure modes that remain. Astra can allow assumptions to override experimental controls, for example selecting the wrong colonies in a colony PCR task by reversing the experimental identities attached to visual groups. It can discount observations that contradict a preferred interpretation, such as generalizing vacuole contrast from a subset of regions while omitting opposite examples visible in the same image. Astra also miscounts colonies in some images — failing to separate close neighbors, missing visible colonies and marking empty agar — producing an overcount of roughly 12% on one plate relative to the benchmark reference. OpenAI notes these types of errors would affect downstream experimental decisions, including estimates of transformation efficiency.
By focusing on everyday bench tasks, the BioBasicsBench benchmark aims to direct progress toward the practical competencies needed for routine experimental work. Accurate interpretation of gel images, plates, microscopy and instrument outputs matters because misreads and overlooked control problems can lead scientists to incorrect conclusions and wasted experimental effort. OpenAI frames the benchmark as a tool both for tracking model improvements on these scenarios and for informing considerations about biosafety and biosecurity.
BioBasicsBench provides a standardized, reviewed dataset and scoring approach that researchers and developers can use to compare capabilities and identify where models succeed or still fall short on realistic wet‑lab evidence. The benchmark’s preservation of raw input types and its multi‑stage review process are intended to keep evaluations closely tied to the kinds of decisions scientists must make at the bench. As multimodal models advance, the benchmark offers a concrete yardstick for measuring their ability to reason from the kinds of evidence that matter in laboratory workflows.
Source: https://openai.com/index/measuring-how-ai-agents-handle-wet-lab-science
Source: Read the original source
Leave a Reply