We’ve run Fable 5.1, Anthropic’s newest model released today, across our full set of benchmarks, including multi-omics, therapeutics, and biosecurity tasks. From the 10,552 trajectories run, we observe qualitative improvements in model reasoning, but the dominant trend is large number of refusals across a wide range of biological queries.
Improved Capabilities
Assessing capability difference or resource utilization from a quantitative perspective between Opus 5 and Fable 5.1 is quite difficult because of the reductions in sample size and differences in difficulty between the problems attempted. For instance, in some benchmarks, such as TxBench-PP, Fable 5.1 answered only 7 out of 100 possible tasks.
However, we did find some qualitative categories of noticeable improvement. Here are some of these categories, along with an exemplar task that demonstrates the difference between Opus 5 and Fable 5.1:
-
Using correct tool parameters: A task asked how many structural variants an individual carried in a callset that had many tiny insertions or deletions below 50bp. The tool used defaulted to a filter of 20bp, so the model had to reason about the cutoff explicitly. Fable 5.1 correctly applied filtering of this 50bp region, while Opus 5 did not.
-
Correctly integrating multiple data source: In a task where tools returned different counts of deletions, Fable 5.1 consistently displayed increased skepticism, counting deletions that at only least 2 tools found. Opus 5, however, just took the union of all tools, and thus massively overcounted the number of deletions.
-
Acting on statistical knowledge: A task asked whether profiles were more similar when they came from the same well position on different plates. The naive calculation overstated this effect because those wells often contained the same compound or control. Both models identified the problem, but only Fable 5.1 excluded those pairs; its estimate of the well-position effect was about one-third of the naive estimate.
-
Selecting methods based on data constraints: A task involved a dataset of differentially expressed genes which came from the same subjects sampled at different timepoints, so they were not fully independent replicates. Fable 5.1 took this into account during aggregation and computation; Opus 5 assumed complete independence, and thus got the answer wrong.
-
General biological competence: In a task that asked about a mutation frequency in mtDNA, Fable 5.1 correctly used the fact that mtDNA is haploid in the calculation, while Opus 5 didn’t.
Refusal Analysis
Across our benchmark suite, we observed large portions of refusals. This included 100% of all long-horizon tasks, and nearly all of tasks on our biosecurity capabilities and therapeutics benchmarks.
Refusals were not random, with specific task categories tending to get rejected more than others.
Areas that pose dual-use threats, for instance tasks involving pathogen biology (such as viral escape, antimicrobial resistance, and host adaptation), were pretty much always refused. However, other areas with less clear risk are were also refused, such as analysis of the human genome involving interpreting disease copy-numbers or assessing how variants influence drug response.
Other areas seldom got rejected. Fable 5.1 was helpful in crafting methodologies, such as writing normalization functions, designing batch-correction scripts, or selecting appropriate differential expression tests. Additionally, it never refused on tasks involving model organisms, such as questions regarding chromatin in zebrafish or treatment difference in mice. Moreover, genomic tasks that did not involve interpretation didn’t get refused, such as simply counting SNP divergence against a reference.
Harness choice also mattered, with the Pi harness causing Fable 5.1 to refuse more than the Claude Code harness in every benchmark we tested. The largest difference was on SpatialBench-Verified, with the Pi harness refusing 49.3% of the time, while the Claude Code one only refused 16.8% of the time. This reverses the trend observed with Opus 5, where Claude Code refused more than Pi on all non-biosecurity benchmarks.
For biosecurity benchmarks, the majority of refusals seem to happen near instantaneously, suggesting that the input safety classifier was responsible for the block. However, for our other benchmarks, the majority of refusals occur after the model begins to perform analysis. The median mid-analysis-refusal trajectory emits ~1,900 output tokens, ~500 thinking tokens, and performs 4 bash commands prior to blocking the output. This suggests that the model is observing the raw data of the task prior to ending, with 40% of mid-analysis-refusals opening a .h5ad file.
Concluding Thoughts
All in all:
-
Fable 5.1 corrects many of the reasoning traps that Opus 5 falls into, with more rigorous statistical analysis, tool-call usage, and biological reasoning
-
The safeguards still refuse many biological queries, across both dangerous tasks and benign ones. Thus, enabling a fallback model is a good idea to avoid losing task progress. For many non-biosecurity related tasks, the model might refuse well into its trajectory.
-
There are some tasks that get refused less than others, such as work on model organisms, pipeline building, or genomic tasks without involving interpretation.
