I have a knowledge base with millions of tokens of context in it. Pickaxe does a great job of presenting the best 20k or so tokens to the model. But I can’t for the life of me find a way to get the model to pull more than about 20k tokens from the knowledge base.
I know there used to be a hard cap of 20k tokens. Is this cap still lurking behind the scenes?
Is there a way to set a minimum number of tokens the RAG engine sends to the model?
Hi @thomasumstattd,
Could you provide us with IDs of the workspaces and agents where this is occurring? It would also be helpful to have some session IDs so we can pinpoint what exactly is happening. I have looked through your account and it does seem like some of your runs do pull over 20,000 tokens. Please send over the info so we can get this resolved as soon as possible!
Here is a classic example from the took I’m trying to get to look at more of the RAG: WA0E5CJF2XCL11P3NEVS
For that big run, was that a multi turn conversation? Because I could see the tool grabbing another 20k tokens every turn.
I tried looking up f4a3cd86-e9e1-47e6-a0c8-5476e2ef906f and couldn’t find it.
Wrong session ID sorry! Looking at the session ID that you gave me, it looks like it was just a regular chat response with around 10,000 KB tokens. The token number was low for this run because it only found a limited amount of candidates from your knowledge base, not because of a cap. I couldn’t find any evidence of mutli turns. Was this the correct session ID? Please also let me know the agent that you are experiencing a RAG cap for.
I just asked a question so broad it could have hit the entire RAG. Still hitting the 20k cap. Every test is right just above 20,000 tokens of input, Which I assume is the prompt itself plus the 20k RAG.
Session ID D9NURP0TXKRXZ1MJTXW7
The session ID is with the specific Pickaxe hitting the RAG limits.
I also tested asking a multi fasciated question to hit lots of zones in the RAG. Still getting the 20k limit.
Session ID: F3E5Z49ATZ2ULM4XW3QD
Could the issue be with the LLM? I started testing different models with the exact same prompt on the same pickaxe.
The hit on the RAG was wildly different depending on which LLM I used.
DeepSeek v4 Flash with reasoning turned off and it pulled in 90,000k tokens for the run.
Session: N5P0C1QIZ7FX8PS21OLK
Sonnet 5, pulled in 74,000 tokens.
Session: EAJI0D1FSUI081I8S8LT
ChatGPT Terra/Luna both got the typical 20,000 tokens (aprox).
Session ID: UDEGTJOEIG5H1I1HHRR5
Gemini 3.5 Flash-Lite 25,971 tokens.
Session ID: CQ8PDAIZYIQTSD04DWCB
Grok 4.5 24,713 tokens
Session ID: MSWMDVQPI2O1QG0EV70G
GLM 5.2 23,373 tokens
Session: WQ9K7HKVX546LAQCWSP8
Grok 4.1 Fast Reasoning 47,778 tokens
Session: O8XQJM1VD3GUW5ZT9SVF
I tried Grok 4.1 again and it came in at the typical 23,178 tokens. So now I wonder if I’m barking up the wrong tree.
Could the issue be some models take a second bite at the RAG apple?
Hi @thomasumstattd,
So sorry for such a late reply. We were creating new features just for you! After reviewing the two session IDs that you sent over, I can confirm that there is no 20k cap being applied to your workspace. For session ID: D9NURP0TXKRXZ1MJTXW7, there were 26 total candidates found totaling 7,432 knowledge base tokens. The rest of the input tokens were used up by your prompt and various other things totaling up to 20,115 input tokens.
For session ID: F3E5Z49ATZ2ULM4XW3QD, there were 22 candidates found totaling 6,233 knowledge base tokens. Your total input tokens for this run was 18,580.
None of these hit any 20,000 knowledge base token cap. Session ID: IF65QK5U1YIGFASLF7LU had 37,342 total input tokens with 27,356 knowledge base tokens. Our engineers have released a new feature allowing users to see how many knowledge base tokens and candidates would be extracted from a given input!
For your comment about different models possibly hitting RAG differently, this is not how the system functions. The reason the tokens were vastly different was because the search mode setting for grabbing RAG might not grab identical chunks every single time. Some good news is that we have just released a couple of new features that you would definitely be interested in!
Feature 1: Input Token Transparency
In message insights, you can now see the input token breakdown of each run! The input is broken down into three categories: Knowledge Base, End-user uploads, and User Memory. Instead of having to guess how many knowledge base tokens were used per run, you can now get a concrete answer!
Feature 2: Knowledge Base Search Mode
We have now released an experimental feature where you can choose the way you want your knowledge base to be searched. We currently have two modes: fast and thorough! Fast mode finds the first promising lead and grabs chunks from there. Thorough uses iterative scanning to search for additional relevant chunks past the fast search mode. Thorough will usually grab more chunks than fast at the cost of speed. This is an experimental feature which we will be constantly monitoring and changing so please give us feedback on it!
Please go ahead and play around with these two features and let us know what you think!
WOW THIS IS AMAZING!!!
I just ran a test for the same prompt “Help me plan a book launch.” and it pulled in 316,747 tokens. Then DeepSeek went on to draft the best answer to this question I’ve gotten from any model.
Best answer ever with Deepseek v4 Flash 0731.
This new system you built allows for an entire new architecture of Pickaxe. Historically, the way to get better responses was to use a more expensive model. But the best models have really expensive output tokens and reasoning tokens count as output tokens.
But there is another way.
You can load up the context window with relevant information. Input tokens are often a fraction of the cost of output tokens and with enough context, a pickaxe can give a better answer with far less reasoning using a far less expensive model.
This is a real game changer.
Hi @thomasumstattd,
We’re so happy to know that you’re enjoying the new feature! If you have any other feedback you want to give us, we would be very happy to hear it!
