I want to do a quick update for accountability on our latency! We’ve learned a lot more about our latency over the last few months, and we’re actively reducing it daily.
We’re proud to disclose that we’ve reduced Pickaxe’s share of initial response latency from 5 seconds to around 3 seconds at the median with an improvement we launched on May 6th.
We’re still striving to reach our goal of sub 2 second Pickaxe caused latency.
Overall latency your users experience is currently around 6 seconds at the median. That additional 3 seconds comes from the model providers TTFT.
Sometimes users experience much higher latency than 6 seconds, and it’s important to remember that these are just median wait times. When we take a look at the wait times for the slowest 5% of cases, another story emerges.
In that worst case scenario, Pickaxe adds around 10 seconds of latency. But the model providers add a whopping 30 seconds.
We’ll continue to work to reduce wait times, and at the same time we’re excited to provide more resources for choosing models that aren’t just faster on average, but have consistently low latency across the board.