Visibility Into Fallback Model Usage

[thomasumstattd]

@lindsay_support asked me prep a feature request on this and I passed it to Claude Code that generated the following.

Research date: August 27, 2026. Sources cited inline. Test evidence in section “What I verified”.

---

Title: Expose the per-message model record through the API, and add a per-agent fallback rate, so silent fallbacks are detectable at scale

Summary

The Fallback Model toggle silently substitutes a backup model when the primary fails. That is the right behavior for uptime. The problem is that nothing tells the builder it happened.

The good news, which I confirmed rather than assumed, is that most of the data already exists. Message Insights stores a per-message model, cost, and latency breakdown, and it stores them correctly. What is missing is narrower than it first appears.

Three asks, in priority order. Record the serving model and fallback reason where the per-message data already lives. Expose that record through the Studio API. Aggregate it into a per-agent fallback and failure rate that works on maximum-privacy workspaces.

The toggle is currently on for both of my workspaces, including the one behind 80+ public tools, so this is live exposure rather than a hypothetical.

Environment

• Workspace: Patron Toolbox (STUDIOM6Q5DRYYU873JXP), 91 agents (88 production tools plus 3 private staging slots), Pro plan, maximum privacy, Fallback Model on
• Second workspace used for testing: AI Thomas & Beta Readers (STUDIO60C60UFYA2ATSQQ), retains history, Fallback Model on
• Deployment: 80+ public patron-facing tools at Patron Toolbox - Author Media
• Volume example: Chapterizer alone has 1,937 lifetime runs

What exists today

The fallback feature. Settings → Agent → Enable fallback model. One toggle, applied to every agent in the workspace. The panel says any agent “that experiences a model failure will automatically retry using one of our fallback models ensuring higher reliability.” It does not name the fallback models and describes no indicator when one is used.

Announcement: Fallback Models for Everyone!

Message Insights. The ⓘ under a bot message shows token allocation, cost, a four-part latency breakdown, Knowledge Base files, memories, actions, and a “Model used” row. It is per-message only. There is no aggregate view.

Source: Message Insights: Latency Monitoring Improvements 🚀

Confirmation that silent fallback is real and hard to detect. On May 29, 2026 a user reported that their agent used Grok no matter which model they selected. Staff confirmed the backend switched to an available fallback without saying so, credited the account, and committed to labeling unavailable models in the UI.

Source: Is this a bug ，why my model always be Grok？

That fix addressed one cause of fallback. It did not add any way to observe fallback when it happens for other reasons, such as a provider outage or a rate limit.

What I verified

I tested a disposable agent, AI Thomas Testbed pickaxe-project-104, with 0 lifetime uses. I ran one completion under grok-4-1-fast-reasoning, then changed the configured model to gemini-3.5-flash without running anything, and read the same conversation back through both surfaces.

Surface	Reports	Correct?
Message Insights, “Model used”	grok-4-1-fast-reasoning	Yes
pickaxe_history, model field	gemini-3.5-flash	No

The good news. The Insights value did not move, and survived a full page reload. A per-message model is genuinely stored. So is per-message cost and a four-part latency breakdown. That is real infrastructure I can build on, and it means request 1 is smaller than I first assumed.

The gap. The API returns a live join against current configuration instead of the stored value. I have filed that separately as a bug, since it damages retrospective analysis whether or not anyone uses the fallback feature.

What I still cannot determine. Whether the stored value is the model that actually served the request, or the model that was configured at run time. My test cannot separate them, because those were the same model when the run happened. If it is the configured model, then nothing records a fallback and request 1 is a genuine new data path. If it is the serving model, request 1 is mostly a labeling change. This is question 1 and it drives the size of everything else.

The gap in practice

Question a builder needs to answer	Available today
Which model produced this response?	Stored, but only visible one message at a time in the UI
Did a fallback fire on this run?	No indicator anywhere
What is this agent’s fallback rate this week?	No aggregate view exists
Which of my 88 production tools is failing most often?	No aggregate view exists
Can I monitor any of this programmatically?	No. See below

What the API surfaces

• pickaxe_get returns one model field. No fallback configuration, no fallback state, no run history.
• run_pickaxe_completion returns {'success': True, 'result': '<text>'}. No model attribution, no retry count, no error detail. Automated tests cannot detect that they just graded output from the wrong model.
• model_list reports per-model availability but no backup mapping, so I cannot determine what my agents would fall back to.
• studio_whoami does not report the workspace Fallback Model toggle state. I had to read it in the UI.
• pickaxe_history returns only model, which is unreliable as described, and usedTokens, which reads 71 where Insights reports 10,405 tokens for the same conversation.

So the cost and latency data that Insights displays per message is UI-only. Even on a workspace with full history retention, none of it can be reached programmatically.

Maximum privacy makes the per-message view unreachable

This is the part I most want considered in the design.

My main workspace retains no transcripts, which is correct and intended for it. Patrons upload unpublished manuscripts, so I do not want their text stored.

But Message Insights hangs off stored messages. On a maximum-privacy workspace there is no message to click, which means every diagnostic currently offered is unavailable to exactly the workspaces handling the most sensitive content. That workspace also has the fallback toggle on and 88 production tools behind public landing pages.

A fallback counter needs no message content. A count of runs, failures, and fallbacks per agent per day carries no user text at all. Building the visibility as an aggregate counter rather than a transcript annotation makes it work on every privacy tier.

Why this matters

Model choice on my tools is a tested decision, not a default. I A/B test models against a fixed manuscript before committing. The Chapterizer runs gpt-5.6-luna at high reasoning because high was the only level with zero misses at full chapter length, and medium dropped an author’s note on a 2,400-word test. The Amazon Keyword Planner runs grok-4.3 at high because that was measured too.

A silent substitution invalidates every one of those tests for the runs it touches.

The real cost is misdiagnosis. Prompt tuning is model-specific. When a patron reports a bad output, my first move is to look at the prompt. If that run silently fell back to another model, I will find a prompt defect that is not there and rewrite a prompt that was working correctly. I cannot rule out a fallback before starting, so it becomes an invisible confound in every quality investigation I run.

Fallback also changes cost. A run served by a different model consumes a different number of credits than the one I budgeted for. Without a count, credit anomalies are unexplainable.

Scale makes manual checking impossible. Across 88 production tools, opening Insights modals one message at a time is not a monitoring strategy.

What I am asking for

1. Record the serving model and the fallback reason

If the stored per-message model is already the serving model, label it so builders can trust it, and add the requested model and reason when they differ:

Model requested:  claude-sonnet-5
Model served:     grok-4.6   (fallback: primary returned 529)

If it is the configured model, store the serving model alongside it. Either way the reason code matters as much as the model name, because a rate limit and a provider outage call for different responses from me.

2. Aggregate it into a per-agent fallback and failure rate

For each agent, over a selectable window: total runs, failed runs by reason, runs served by a fallback with the rate as a percentage, and which backup model served them.

Counts only, no message content, so it works on maximum-privacy workspaces. A workspace-level roll-up sorted by fallback rate would let me open one page and see which of 88 production tools is unhealthy.

3. Expose the per-message record and the counters through the Studio API

The per-message data already exists for Insights. Returning it through the API is the smallest high-value change on this list.

• Fix pickaxe_history to return the stored model rather than a join. Filed separately as a bug.
• Add the cost and latency fields Insights already computes.
• Add the counters from request 2, as fields on pickaxe_get or a dedicated endpoint.
• Report the workspace Fallback Model toggle state in studio_whoami.
• Include the serving model in the run_pickaxe_completion response.

4. Optional: an alert threshold

A notification when an agent’s fallback rate crosses a threshold I set. Lower priority than 1 through 3.

Ruled out

• Not a plan limitation. The workspace is on Pro and model_list reports all models available.
• Not solved by chat history. My main workspace retains none by design, and the API’s model field is unreliable in any case.
• Not solved by per-tool settings. pickaxe_get exposes no fallback field on the agent record.
• Not the free-trial cause from the May 29 thread. My models are all available to this account, so any fallback here would come from provider failures rather than access gating.
• Not something I can infer from output. Model fingerprinting by writing style is guesswork and does not scale to 88 tools.

Questions

1. Is the “Model used” value in Message Insights the model that actually served the request, or the configured model captured at run time? This is the single most important answer, because it decides whether request 1 is a labeling change or a new data path.
2. Are fallback events recorded server-side today, even if nothing surfaces them?
3. What is the fallback chain? A fixed backup per provider, a single global backup, or a dynamic choice?
4. What conditions trigger fallback? Overload only, or also timeouts, rate limits, content filters, and context-length errors?
5. Does a fallback run bill at the primary model’s credit rate or the backup’s?
6. Does fallback preserve reasoning effort settings? My tools depend on specific reasoning levels, and a backup that ignores the setting is a quality change beyond the model swap itself.
7. Is the per-message Insights record kept on maximum-privacy workspaces, or discarded with the transcript? If the counters can be derived there, request 2 works everywhere. If not, they may need to be incremented at run time instead.

[danny_support]

Hi @thomasumstattd ,

Two related improvements are now live:

• pickaxe_history now uses the stored generation model when that information is available, rather than automatically returning the agent’s current configured model.
• Message Insights now shows a Fallback indicator when the selected model differs from the model that generated the response.

The fallback indicator only applies to new responses generated after this tracking was introduced.

We have also created a ticket for making per-response Message Insights information available through the Workspace API. Our engineers are looking into the best way to implement this.

To answer your questions:

1. What does “Model used” mean?

   “Model used” represents the model that generated the response. For newly tracked fallback responses, the originally selected model and the model that served the response are recorded separately. The current UI shows the served model with a Fallback indicator, but does not yet display the originally selected model.

2. Are fallback events recorded?

   For new responses, yes. We can identify a fallback because the selected and served models are recorded separately.

3. How is the fallback model selected?

   Pickaxe uses a platform-managed, ordered pool of fallback models. It selects an eligible model that has not already failed and is available for the Workspace’s plan. It is not a fixed backup assigned to each provider, and the available fallback models may change over time.

4. What triggers a fallback?

   A fallback may occur when the selected model is disabled or unavailable, or when a generation encounters an error that can be retried. This generally includes provider availability problems such as outages, overload, timeouts, and rate limits.

5. How is a fallback response billed?

   Usage for the completed response is calculated using the model and provider that actually generated it, along with the recorded token usage and cost. It is not calculated using the selected model merely because that was the agent’s original configuration.

6. Does fallback preserve reasoning effort?

   Pickaxe carries the selected reasoning-effort setting into the fallback attempt when the fallback model supports it. If the fallback model does not support the same setting, the effort may be adjusted to a supported value.

7. What happens on maximum-privacy Workspaces?

   Disabling conversation history is intended for builders whose agents handle sensitive information, such as medical-related information, private business data, or unpublished manuscripts. Prompts and responses from those sessions are not made available through Workspace conversation history.

   We may be able to expose limited, non-content Message Insights data for those sessions, but this has not been confirmed. Our product and legal teams are reviewing what information, if any, can be made available through the Workspace API without undermining the privacy setting.

Per-agent and Workspace fallback rates, fallback-reason reporting, additional fallback information in API responses, and alert thresholds are not currently available. These are broader requests that will likely be considered after the core work to make per-response Message Insights available through the Workspace API.