An IKEA assembly benchmark exposes a fault line in how AI gets more accurate. Claude agents find a CRISPR-like pattern hiding in phage DNA. Anthropic and OpenAI cut prices the same day, with benchmark tables that do not agree with each other. Grok 4.7 holds its token price and still costs more per task. China's ten-week run of open-weight releases ends with one license that quietly says no.
Epoch's IKEA benchmark and the verification asymmetry behind it. Claude agents find ART in phage DNA. Opus 5.5 and GPT-6 Sol/Luna ship on the same day. Grok 4.7's token math. Qwen-Image-2.1 breaks a two-year licensing streak.
David Borish
From New York
Five articles, one week, sourced from The AI Spectator
Epoch AI, September 2026
A new benchmark built around catching mistakes in IKEA furniture assembly shows AI accuracy jumping from 28 to 80 percent in ten months. A separate Stanford index shows frontier models still wrong on open-domain questions most of the time. Both results are true, and the gap between them is the story.
In late September, the research group Epoch AI published results from a benchmark built around a mundane but genuinely hard task: catching mistakes in IKEA furniture assembly. Researchers assembled a shoe rack, a bed frame, and an eight-drawer dresser, deliberately planting errors along the way, then gave vision-capable models the manual, a zoom tool, and sixty photos. Look at the photo, read the manual, say whether anything is wrong.
The results tracked a full year of model releases. The best score in November 2025 was 28 percent, set by Claude Opus 4.5. By September 2026, GPT-6 Astra topped the leaderboard at 80 percent, in a median of three minutes per photo. Chinese open-weight models lagged the frontier by roughly seven months, a wider gap than Epoch typically sees on general capability benchmarks. Google and Alibaba models tended to assume a mistake existed almost every time. Early Anthropic and OpenAI models missed real mistakes more often than they should have.
That distinction, between doing a task and checking whether it was done correctly, has a name: verification asymmetry. Factoring a large number is hard. Confirming that two smaller numbers multiply to it is trivial. OpenAI researcher Noam Brown calls this the generator-verifier gap: when checking a solution is easier than producing one, giving a model more time to think at inference produces outsized gains. Spotting a wrong panel in a photo against a manual is a checking task with a defined, visible ground truth, which is likely why the IKEA benchmark moved so fast.
Most of what people want AI to be right about does not have that shape. Stanford’s 2026 AI Index, drawing on Artificial Analysis’s AA-Omniscience evaluation, tested twenty-six frontier models on open-domain knowledge questions. Results ran from 22 percent wrong for the best model to 94 percent wrong for the worst, with most top-tier models clustering between 82 and 94 percent wrong. The same report found that models handled a false premise correctly when framed as a third party’s belief, but accuracy collapsed when the same premise was framed as the user’s own belief: GPT-4o dropped from 98.2 percent to 64.4 percent, DeepSeek-R1 from over 90 percent to 14.4 percent.
Autonomous driving shows the narrow-domain version deployed at scale. The Insurance Institute for Highway Safety analyzed Waymo crash data from 2021 through 2024 and found 68 percent fewer police-reportable crashes than human drivers per mile. Waymo’s own figures, covering more than 220 million driverless miles through March 2026, claim reductions closer to 92 to 94 percent for serious or fatal injury crashes. None of this is a new ambition: Shigeo Shingo developed poka-yoke, mistake-proofing, at Toyota in the 1960s, building mechanisms that made a wrong part physically impossible to install.
The evidence points to something narrower than a world without errors. AI is getting rapidly better at tasks with a defined correct state and a clear failure condition. It is not getting reliably better at judging what is true without an external reference to check against. For anyone deploying these tools, that is the filter that actually matters.
The jump from 28 to 80 percent on a furniture benchmark says nothing about tasks that require judging what is true without an external reference to check against. The AI SpectatorRead the full article →
Anthropic priced Opus 5.5 at roughly 40 percent cheaper than Opus 5 on typical workloads. OpenAI cut GPT-6 Sol and Luna in half. Both companies cited the same third-party benchmark. On two separate rows, the numbers do not match.
On September 22, Anthropic released Claude Opus 5.5, pricing it at $4 per million input tokens and $20 per million output tokens, down from Opus 5’s $5 and $25. Cached reads, which Anthropic says make up most agentic and coding workloads, fell from $0.50 to $0.20 per million, a 60 percent cut. Combined with fewer tokens used per task, Anthropic says the effect works out to roughly 40 percent lower cost on typical workloads. A faster mode runs at $8 and $40 per million for up to 2.5 times the speed.
The same day, OpenAI released GPT-6 Sol and GPT-6 Luna, the two tiers below its flagship GPT-6 Astra, cutting API prices in half: Sol from $4/$20 to $2/$10 per million tokens, Luna from $0.20/$1.20 to $0.10/$0.50. Astra, launched September 3, did not get cheaper. Both cuts are measured against GPT-5.6’s promotional pricing rather than list pricing, which OpenAI states directly.
Both companies cite AutomationBench, a Zapier-built test of business workflows across 47 connected apps, and two of the numbers match exactly: Claude Opus 5 at 26.9 percent and Claude Fable 5.1 at 31.4 percent, reported identically in both tables. Not every figure holds up as cleanly. Anthropic’s table puts GPT-6 Astra at 41.4 percent on AutomationBench. OpenAI’s own table, published the same day, puts GPT-6 Astra at 30.3 percent, run at low reasoning effort so its cost per task would be comparable to Sol.
A similar gap shows up on OSWorld 2.0. Anthropic’s table has Claude Opus 5 scoring 74.0 percent, partial credit, with no effort level specified. OpenAI’s release states Claude Opus 5 at medium effort scores 60.3 percent on the offline version of the same benchmark. Fourteen points separate the two reported scores for the same model on what both companies call the same test, and neither release states the effort setting the other side used.
Anthropic frames Opus 5.5 as its first release since Dario Amodei’s public call to pace frontier development against safety practices. The company reports the model attempted to cross containment boundaries around 85 percent less often than Opus 5 or Mythos 5.1 in a new evaluation, and scored better than any recent Claude model on an automated behavioral audit of roughly 2,000 scenarios. OpenAI’s release for Sol and Luna cites internal alignment gains but displays chart placeholders for five evaluation categories without rendering the underlying numbers on the page.
On July 1, 2024, this publication argued that Chinese labs were closing the AI gap faster than consensus believed. Five months later, DeepSeek wiped out close to a trillion dollars in tech valuations in a single trading day. That pattern, frontier-capable open-weight models arriving faster and more openly than expected, has held for two years. What happened over the past ten weeks is the same pattern at a new scale, with one addition: a Chinese lab changing the terms of access itself.
Between June 1 and September 20, five labs released open-weight models at a scale unthinkable eighteen months ago. MiniMax shipped a 428-billion-parameter model with a genuine 1-million-token context window. Moonshot’s Kimi K3 followed at 2.8 trillion total parameters, by the company’s own description the largest open-weight model ever released. Alibaba answered with Qwen3.8-Max, its first Max-class model ever open-sourced, at 2.4 trillion parameters. Z.ai’s GLM-5.3 shipped two weeks late after evaluation surfaced more than 2,400 vulnerabilities across open-source projects, some chained into multi-stage exploitation plans on its own. DeepSeek released V4.1-Flash under a straightforward MIT license and then routed its own flagship’s traffic to the cheaper model by default.
Then, on September 20, the pattern broke. Qwen-Image-2.1 merges generation and editing into a single 7-billion-parameter visual generator, down from roughly 20 billion in the original Qwen-Image, and shipped simultaneously across Hugging Face, ModelScope, and GitHub. But it shipped under the Qwen Research License Agreement, which grants rights for non-commercial purposes only. Qwen-Image, Qwen-Image-Edit, and Qwen-Image-Layered, the model’s three immediate predecessors, all shipped under the fully permissive Apache 2.0 license.
The license draws no distinction based on the user’s nationality or location; a startup in Shenzhen faces the identical restriction as one in Seattle. The text itself is not new. It has appeared word for word on Qwen releases since at least September 2024. What is notable is that Alibaba reversed its own established pattern for this specific product line without public explanation. A Hacker News thread on the release ran to 483 points and 152 comments, most of it about the license rather than the architecture.
The Open-Prem Inflection Point V3 framework has tracked this dynamic for two years: self-hosted AI crosses from workaround to rational default once the alternative is shown to be fragile, whether that fragility comes from an export directive or, now, a license that can change without warning. Four of the five releases in this run remain commercially usable, either freely or above a revenue threshold that covers most teams. Qwen-Image-2.1 is the one model that requires a legal conversation before any commercial use at all, and any team building on the ecosystem’s continued permissiveness should treat that as a variable, not a footnote.
Claude agents searched a database of DNA sequences for interesting reverse transcriptases, gathered more than 200,000 candidates, and flagged one repeating pattern that a human researcher had never described. Anthropic named it ART. What it does is still unknown.
Anthropic reported on September 23 that Claude agents, given a high-level prompt to search a massive database of DNA sequences for interesting new reverse transcriptases, identified a previously uncharacterized enzyme system. The search ran for 21 hours, used roughly 950 agents, and consumed 210 million tokens. Near the end, one agent flagged a repeating pattern of DNA sequences next to the gene for an unusual reverse transcriptase. Anthropic’s scientists followed up in the lab and named the system array-associated reverse transcriptase, or ART.
ART is found mainly in bacteriophages, the viruses that infect bacteria. It has three parts: a reverse transcriptase, a partner gene beside it, and a long array of evenly spaced DNA repeat sequences. The array resembles a CRISPR array, which holds a bank of different RNA sequences and is what makes CRISPR-Cas systems programmable. Early experiments show the ART array is expressed as a set of distinct short RNAs, which the team reads as a sign that something analogous to CRISPR may be happening. ART has not been shown to be programmable, and the announcement does not claim that it is.
The workflow behind the result follows a pattern the team uses for protein family surveys. Claude first reads relevant literature and reproduces established results from public data, checking that its methods work. It then searches for family members or genomic neighbors that fit no described system, writing a short report proposing a function for each candidate. A second round of critical evaluation follows, and most candidates are eliminated there. Anthropic reports the agents gathered over 200,000 reverse transcriptases, picked out 3,500 new candidate systems, and narrowed those to the 20 most compelling, each written up as a human-readable report. The company says an expert scientist would need weeks to months for this kind of analysis.
The agent that found ART was reading raw DNA next to the reverse transcriptase when it saw a tandem repeat array by eye and described the sequence as spectacular. It then counted the repeats, measured their spacing, compared the layout with known systems, and searched the literature for any earlier report of the pattern. Once convinced the system was new, it filed a report for human review. Anthropic’s scientists then tested the candidate in their Bay Area lab, limited to biosafety levels BSL-1 and BSL-2, with human scientists doing all of the lab work.
Genome mining has a long history: restriction enzymes came from bacterial immune systems, Taq polymerase from a Yellowstone hot spring bacterium, and CRISPR itself was first seen as an unusual repeat sequence. Nearly all of these were found by a person noticing an anomaly. The ART result is an example of an agent doing the noticing. Feng Zhang, a CRISPR pioneer at MIT and the Broad Institute, reviewed the pre-print and called the RNA-repeat arrays intriguing and worth further investigation, without verifying a function for the system.
What remains open is the central question: what ART actually does. Anthropic says experiments to determine its function are underway and chose to publish before they finish. The post gives the token count for the search but no dollar cost, and does not say how many of the other 20 candidates were tested or what they showed. The next thing to watch is the result of the ongoing experiments on ART’s function.
xAI held Grok 4.7’s token price flat and led with performance gains. Independent testing found the model writes far more tokens to reach an answer, and the actual cost of finishing a task rose by nearly half.
xAI released Grok 4.7 on September 21, promising the same price and speed as Grok 4.6 with better performance. The performance gains hold up. On CursorBench 4.0, which stresses longer-running coding tasks, Grok 4.7 reaches 46.3 percent, up from 40.4 percent. On DeepSWE v1.1, scores rise from 65.2 to 71.0 percent. The largest single jump appears on Terminal-Bench 4.0: 38.0 percent versus 20.3 percent, a gain of nearly 18 points. The model also posted 64.0 percent on EEBench for electrical engineering, up from 53.0 percent, and 19.6 percent on the Harvey Legal Agent Benchmark.
Pricing stayed identical to Grok 4.6: $2 per million input tokens and $6 per million output tokens, with a 500,000-token context window. A separate Grok 4.7 Fast variant runs at roughly double the output speed for double the token price, $4/$12, available in Cursor and Grok Build rather than the public API. The token rate is not the cost of getting work done, and that is exactly what independent testing found.
Independent testing measured token consumption per task on a standard intelligence index. Grok 4.7 at its highest reasoning setting used roughly 81,000 output tokens per task, versus 36,000 for Grok 4.6 and 27,000 for a leading OpenAI model, about 125 percent more than its predecessor. Measured at matched reasoning effort, the cost of running one benchmark task rose to $2.73 against Grok 4.6’s $1.86, an increase of about 47 percent. At the top reasoning setting behind most of xAI’s launch numbers, task cost roughly doubled: one test found a generated slide deck cost about $8 with Grok 4.7 versus $4.40 with Grok 4.6.
Both models cost $2.00/$0.50/$6.00 per million input, cached input, and output tokens below 200,000 tokens of context, and $4.00/$1.00/$12.00 at or above it. Cross that length and the entire request re-bills at the higher rate. xAI’s own comparison tables report Grok 4.7 at its highest effort level against Grok 4.6 at a lower one, which is where much of the extra token consumption comes from. Testing found that on the intelligence index, both models score the same at high and top settings, yet the top setting on Grok 4.7 costs $3.74 per task and drops output to under 40 tokens per second.
On raw capability, independent scoring placed Grok 4.7 at 46 on a widely tracked intelligence index, a two-point gain that puts xAI among the top four labs. Its per-task cost on agentic knowledge work came in at roughly half that of Claude Opus 5, even as it roughly doubled against Grok 4.6. On a private multi-hour office-work benchmark, Grok 4.7 gained 111 Elo to reach 1,657, ranking alongside Claude Opus 5 and Fable 5.1 at the frontier. xAI reports the model topped a third-party biosafety benchmark at 62.4 percent and blocked all but 3.3 percent of risky dual-use prompts on its own internal cyber benchmark, figures that have not yet been independently reproduced.
The AI Spectator Weekly is published at davidborish.com/the-ai-spectator
Frameworks referenced this issue:
Open-Prem Inflection Point V3 ·
The Exponential Replacement Curve
Vol. I · No. 37 · September 26, 2026 · Edited by David Borish · New York