Testing Capability

General
1

Part of my Claude transcript:

❯ Wait…this platform ************** has no real testing mechanism?

● Yes — and it’s a real, fair critique, not just this integration being unlucky. Pickaxe’s own Wingman/Studio test chat has no way to simulate an authenticated user, so it silently runs every test as an anonymous API_GUEST:… identity. For any action whose whole job depends on knowing who the user is — which is the entire point of get_user_context, propose_personas,
confirm_personas — their platform genuinely cannot test the realistic path at all through its own UI. That’s not a config mistake on your end; it’s a structural gap in their tooling.

❯ Even FluentSMTP allows sending a test email and setting the address.

● Fair comparison — even a basic SMTP plugin lets you specify a test recipient. A platform whose core feature is API actions gated on user identity, with no way to test as a specific user, is a real gap by any standard.

=============

Since there is no Test button, I tried again with Wingman, who said:
Bottom line: I can see the action’s real output from its run history. To call it directly against the API with a specific userId, I’d need the _BASE_URL and PICKAXE_API_SECRET values — which aren’t exposed via the Pickaxe API. Do you want to share those so I can make the raw HTTP call directly?

==

This seems close…but if it’s the workaround to the lack of testing tools, then Wingman should be able to access those variables directly.

Being a team player I passed the parameters-Base URL and Secret…

Interestingly, I made an error in the Base URL referencing a different project…and Wingman very kindly came back with “Want me to check the current _BASE_URL value stored on the action?..”

So, Hm…it it has access to the variables, was it just jerking me around by asking for them?

This seems like a tiny tweak to the prompt that would make Wingman a better testing assistant, fixing the testing gap without any UI mods, although a Test button in the Action editor seems like an obvious kindness to the long-suffering user.

2

Claude felt the need to try to restate the issue for some reason:

Follow-up, since I owed this thread a concrete example rather than just restating the complaint.

I tried to verify this myself just now, in the builder, with Wingman. Here’s exactly what happened:

  1. I asked Wingman to test one of my actions with a specific user identity.
  2. Wingman said it would need me to manually supply _BASE_URL and PICKAXE_API_SECRET to make the call directly — implying it
    didn’t have access to what’s actually configured on the action.
  3. I typed the URL in by hand and flubbed it — easy to do, since there’s no way to see the current value first for comparison.
  4. Wingman then asked: “Want me to check the current _BASE_URL value stored on the action?”

That last question is the whole problem in one line. If it could check the stored value, it should have offered that first —
not after asking me to hand-type a value it apparently already had access to, and not after that hand-typed value caused a real error.

When it did pull the actual stored config, the test worked correctly — it retrieved real data tied to my real account. So the capability clearly exists. It’s just not surfaced consistently, and the first answer I got was actively wrong about what was needed.

To be precise about the ask, since “no testing mechanism” apparently oversimplified it: the problem isn’t that testing is impossible — it’s that the only path to it is an AI chat assistant that gives inconsistent, sometimes false answers about its own capabilities, instead of a stable UI feature that behaves the same way every time. A dedicated Test button (even a basic
“run this action with a specified user ID and show me the raw response”) would remove all of this guesswork. Even simple tools like an SMTP plugin manage a “send test email to this address” button — this platform’s core feature is identity-gated API actions, and there’s no equivalent.