@lindsay_support asked me prep a feature request on this and I passed it to Claude Code that generated the following.
Research date: August 27, 2026. Sources cited inline. Test evidence in section “What I verified”.
Title: Expose the per-message model record through the API, and add a per-agent fallback rate, so silent fallbacks are detectable at scale
Summary
The Fallback Model toggle silently substitutes a backup model when the primary fails. That is the right behavior for uptime. The problem is that nothing tells the builder it happened.
The good news, which I confirmed rather than assumed, is that most of the data already exists. Message Insights stores a per-message model, cost, and latency breakdown, and it stores them correctly. What is missing is narrower than it first appears.
Three asks, in priority order. Record the serving model and fallback reason where the per-message data already lives. Expose that record through the Studio API. Aggregate it into a per-agent fallback and failure rate that works on maximum-privacy workspaces.
The toggle is currently on for both of my workspaces, including the one behind 80+ public tools, so this is live exposure rather than a hypothetical.
Environment
- Workspace: Patron Toolbox (STUDIOM6Q5DRYYU873JXP), 91 agents (88 production tools plus 3 private staging slots), Pro plan, maximum privacy, Fallback Model on
- Second workspace used for testing: AI Thomas & Beta Readers (STUDIO60C60UFYA2ATSQQ), retains history, Fallback Model on
- Deployment: 80+ public patron-facing tools at Patron Toolbox - Author Media
- Volume example: Chapterizer alone has 1,937 lifetime runs
What exists today
The fallback feature. Settings → Agent → Enable fallback model. One toggle, applied to every agent in the workspace. The panel says any agent “that experiences a model failure will automatically retry using one of our fallback models ensuring higher reliability.” It does not name the fallback models and describes no indicator when one is used.
Announcement: Fallback Models for Everyone!
Message Insights. The ⓘ under a bot message shows token allocation, cost, a four-part latency breakdown, Knowledge Base files, memories, actions, and a “Model used” row. It is per-message only. There is no aggregate view.
Source: Message Insights: Latency Monitoring Improvements 🚀
Confirmation that silent fallback is real and hard to detect. On May 29, 2026 a user reported that their agent used Grok no matter which model they selected. Staff confirmed the backend switched to an available fallback without saying so, credited the account, and committed to labeling unavailable models in the UI.
Source: Is this a bug ，why my model always be Grok？
That fix addressed one cause of fallback. It did not add any way to observe fallback when it happens for other reasons, such as a provider outage or a rate limit.
What I verified
I tested a disposable agent, AI Thomas Testbed pickaxe-project-104, with 0 lifetime uses. I ran one completion under grok-4-1-fast-reasoning, then changed the configured model to gemini-3.5-flash without running anything, and read the same conversation back through both surfaces.
| Surface | Reports | Correct? |
|---|---|---|
| Message Insights, “Model used” | grok-4-1-fast-reasoning |
Yes |
pickaxe_history, model field |
gemini-3.5-flash |
No |
The good news. The Insights value did not move, and survived a full page reload. A per-message model is genuinely stored. So is per-message cost and a four-part latency breakdown. That is real infrastructure I can build on, and it means request 1 is smaller than I first assumed.
The gap. The API returns a live join against current configuration instead of the stored value. I have filed that separately as a bug, since it damages retrospective analysis whether or not anyone uses the fallback feature.
What I still cannot determine. Whether the stored value is the model that actually served the request, or the model that was configured at run time. My test cannot separate them, because those were the same model when the run happened. If it is the configured model, then nothing records a fallback and request 1 is a genuine new data path. If it is the serving model, request 1 is mostly a labeling change. This is question 1 and it drives the size of everything else.
The gap in practice
| Question a builder needs to answer | Available today |
|---|---|
| Which model produced this response? | Stored, but only visible one message at a time in the UI |
| Did a fallback fire on this run? | No indicator anywhere |
| What is this agent’s fallback rate this week? | No aggregate view exists |
| Which of my 88 production tools is failing most often? | No aggregate view exists |
| Can I monitor any of this programmatically? | No. See below |
What the API surfaces
pickaxe_getreturns one model field. No fallback configuration, no fallback state, no run history.run_pickaxe_completionreturns{'success': True, 'result': '<text>'}. No model attribution, no retry count, no error detail. Automated tests cannot detect that they just graded output from the wrong model.model_listreports per-model availability but no backup mapping, so I cannot determine what my agents would fall back to.studio_whoamidoes not report the workspace Fallback Model toggle state. I had to read it in the UI.pickaxe_historyreturns onlymodel, which is unreliable as described, andusedTokens, which reads 71 where Insights reports 10,405 tokens for the same conversation.
So the cost and latency data that Insights displays per message is UI-only. Even on a workspace with full history retention, none of it can be reached programmatically.
Maximum privacy makes the per-message view unreachable
This is the part I most want considered in the design.
My main workspace retains no transcripts, which is correct and intended for it. Patrons upload unpublished manuscripts, so I do not want their text stored.
But Message Insights hangs off stored messages. On a maximum-privacy workspace there is no message to click, which means every diagnostic currently offered is unavailable to exactly the workspaces handling the most sensitive content. That workspace also has the fallback toggle on and 88 production tools behind public landing pages.
A fallback counter needs no message content. A count of runs, failures, and fallbacks per agent per day carries no user text at all. Building the visibility as an aggregate counter rather than a transcript annotation makes it work on every privacy tier.
Why this matters
Model choice on my tools is a tested decision, not a default. I A/B test models against a fixed manuscript before committing. The Chapterizer runs gpt-5.6-luna at high reasoning because high was the only level with zero misses at full chapter length, and medium dropped an author’s note on a 2,400-word test. The Amazon Keyword Planner runs grok-4.3 at high because that was measured too.
A silent substitution invalidates every one of those tests for the runs it touches.
The real cost is misdiagnosis. Prompt tuning is model-specific. When a patron reports a bad output, my first move is to look at the prompt. If that run silently fell back to another model, I will find a prompt defect that is not there and rewrite a prompt that was working correctly. I cannot rule out a fallback before starting, so it becomes an invisible confound in every quality investigation I run.
Fallback also changes cost. A run served by a different model consumes a different number of credits than the one I budgeted for. Without a count, credit anomalies are unexplainable.
Scale makes manual checking impossible. Across 88 production tools, opening Insights modals one message at a time is not a monitoring strategy.
What I am asking for
1. Record the serving model and the fallback reason
If the stored per-message model is already the serving model, label it so builders can trust it, and add the requested model and reason when they differ:
Model requested: claude-sonnet-5
Model served: grok-4.6 (fallback: primary returned 529)
If it is the configured model, store the serving model alongside it. Either way the reason code matters as much as the model name, because a rate limit and a provider outage call for different responses from me.
2. Aggregate it into a per-agent fallback and failure rate
For each agent, over a selectable window: total runs, failed runs by reason, runs served by a fallback with the rate as a percentage, and which backup model served them.
Counts only, no message content, so it works on maximum-privacy workspaces. A workspace-level roll-up sorted by fallback rate would let me open one page and see which of 88 production tools is unhealthy.
3. Expose the per-message record and the counters through the Studio API
The per-message data already exists for Insights. Returning it through the API is the smallest high-value change on this list.
- Fix
pickaxe_historyto return the stored model rather than a join. Filed separately as a bug. - Add the cost and latency fields Insights already computes.
- Add the counters from request 2, as fields on
pickaxe_getor a dedicated endpoint. - Report the workspace Fallback Model toggle state in
studio_whoami. - Include the serving model in the
run_pickaxe_completionresponse.
4. Optional: an alert threshold
A notification when an agent’s fallback rate crosses a threshold I set. Lower priority than 1 through 3.
Ruled out
- Not a plan limitation. The workspace is on Pro and
model_listreports all models available. - Not solved by chat history. My main workspace retains none by design, and the API’s model field is unreliable in any case.
- Not solved by per-tool settings.
pickaxe_getexposes no fallback field on the agent record. - Not the free-trial cause from the May 29 thread. My models are all available to this account, so any fallback here would come from provider failures rather than access gating.
- Not something I can infer from output. Model fingerprinting by writing style is guesswork and does not scale to 88 tools.
Questions
- Is the “Model used” value in Message Insights the model that actually served the request, or the configured model captured at run time? This is the single most important answer, because it decides whether request 1 is a labeling change or a new data path.
- Are fallback events recorded server-side today, even if nothing surfaces them?
- What is the fallback chain? A fixed backup per provider, a single global backup, or a dynamic choice?
- What conditions trigger fallback? Overload only, or also timeouts, rate limits, content filters, and context-length errors?
- Does a fallback run bill at the primary model’s credit rate or the backup’s?
- Does fallback preserve reasoning effort settings? My tools depend on specific reasoning levels, and a backup that ignores the setting is a quality change beyond the model swap itself.
- Is the per-message Insights record kept on maximum-privacy workspaces, or discarded with the transcript? If the counters can be derived there, request 2 works everywhere. If not, they may need to be incremented at run time instead.