I try every provider, reduce the token outpout, knowledge base, use light/fast model but every questions to my agents have 12/15 seconds lag.
I use regularly duck.ai for small talk with random model and the response is in seconds
What’s the technical reason for this latency ?
Just the model providers (on average) contribute a median of 4 seconds. But it varies a lot between provider, some of them take MUCH longer. You can see the whole breakdown here.
On the Pickaxe side, the biggest thing slowing down responses is lots of documents. We see slower responses in folks with lots of documents. Usually faster responses if you have no documents connected.
We’re actively working on reducing latency, and upcoming releases to the product will further reduce it. Our goal is for Pickaxe to contribute less than 2 seconds of latency.
I hope that helps!
do you mean documents uploaded by the user or documents in the Knowledgebase?
thx
@dougfresh I believe Nathaniel was referencing the KB primarily, but uploaded documents that are referenced in conversation can also add to response times 
in both cases though, we are actively trying to cut things down!
Yes, I’m experiencing this latency issue as well, and it really affects the user experience. I guess we are spoiled because ChatGPT responds almost instantly, so there is that inherent expectation when people use chatbots. When I use Pickaxe, however, the wait time is so much longer. I don’t want my users to have to deal with that.
We posted an update about latency here: Pickaxe Latency Deep Dive 🕰️ - #3 by nathaniel
TLDR: The median latency added by Pickaxe is 3 seconds. In the worst 5% of cases that can get up to 10 seconds. On the other hand, for the model providers, median latency is 2 seconds but in the worst 5% of cases contributes 30 seconds. So, often switching model providers can really help worst case latency.