Any user that has spoken to me knows I love our Message Insights feature. For those of you unfamiliar, if you click the ⓘ icon below any message sent by one of your bots - either in the agent builder or in the Activity section of the Users tab in your Workspace Dashboard - a screen will pop up that gives you a detailed look at all of the elements that went into the generation of that message. This includes token allocation, cost, model, and exact details of which Knowledge Base files, User Memories, and Actions were used.
Users may also know that our team has been putting a lot of time and effort into cutting down on latency, and are pushing regular backend improvements to make sure that Pickaxe adds as little time as possible to your message generation.
To that end, we just made some improvements to Message Insights that will help you track your response generation time much more granularly!
Until today, Message Insights would show just one time value, which we called Generation Time. This value reflected the length of time from when a user input hit the Pickaxe servers → when the agent’s response finished generating.
After the update, this time value has been broken into four components:
- Pickaxe Latency: this value reflects the length of time from when a user hits enter on their input all the way to when the model receives the input package (when model time to first token starts). This is the amount of latency that we have control over, and what we’re working on the backend to reduce. The flow is as follows:
- User hits enter
- Request ingress (the message arrives at our servers)
- Pickaxe processing (including elements like User Memory and the Knowledge Base)
- Model TTFT: this reflects the amount of latency that comes from the model provider, and includes the time from when they receive the package to when they send back the first token. The flow is as follows:
- Model request is received
- Response is generated
- First token is returned
- Total TTFT: this value is the combination of Pickaxe Latency and Model TTFT. Think of it as the total amount of latency before your agent begins writing a response to your user.
- Generation Time: this value is the amount of time it takes for the agent’s response to be completed, from the first token to the last. The main elements that will impact this value are the length of the response itself as well as any action calls that are triggered during the response.
With these updates, you should be able to see exactly which components of your setup are responsible for any latency you might experience, which we hope will help you all manage latency more effectively!
And from your Support Team: here are a list of items that you can look at and update to reduce latency, depending on which of these components is causing problems:
- Pickaxe Latency: If this value is high, some practical steps to lowering it include:
- Lowering your token limits. If you have an unnecessary amount of tokens allocated to elements that you don’t need, our system will spend time and resources on elements that won’t impact the response. By far the highest impact element here is the Knowledge Base - if tons of Knowledge Base files are being pulled in for a response, it’s going to noticeably impact the processing time! You can edit your token limits in the Configure tab of your agent builder.
- Email us at info@pickaxeproject.com! Like I said above, this is the latency component that we have the ability to influence. We can’t promise to solve the problem, but we do want to know about your experience and will do everything we can to assist!
- Model TTFT: this one is a lot simpler. You can lower this value by selecting a faster model! We have a whole section of the site dedicated to model comparison, and even a tool that lets you directly compare TTFT using averages from across the entire platform
- Generation Time: as I mentioned above, this relates to both the length of the response and the time it takes for any actions to run, so you can:
- Remove any unnecessary action calls
- Lower the “output token” limits (in the Configure tab of your agent builder) to ensure that your agent gives more concise responses
Thank you all for working with us as we strive to make Pickaxe the best it can be - we are committed to building in the open, and will keep you posted as we continue to push changes that reduce latency, increase transparency, and help make your agents the best they can be 
