PokeGPT: proving a GPT could call a real API

Context

PokeGPT is a custom GPT wired to the PokeAPI. Ask it about a Pokemon and it fetches the actual record: types, base stats, abilities, evolution chain, move pool, the numbers as they exist in the games rather than as the model half-remembers them.

I built it in November 2023, days after OpenAI shipped Actions, because I wanted to know whether the feature worked or merely demoed.

The problem

Custom GPTs launched with a lot of noise, almost all of it prompt wrappers. A personality, a tone, a system message, and underneath it the same model answering from the same training data. Not new.

Actions were the new part: a GPT could be given an OpenAPI schema and call a real HTTP service. That is the difference between a chatbot that has read about your data and one that has your data. Nobody was writing about what that was like to build, as opposed to what the announcement claimed.

The PokeAPI was the ideal test subject. It is public, well documented, free, politely rate limited, and it has a property I could not get elsewhere: a domain where the model already knows a great deal, incorrectly. Every model has absorbed years of forum arguments about base stats. So I could ask a question, watch what it answered from memory, and compare that to what the API returned. A wrong answer is visible immediately, unlike in most API integrations.

My role

Everything, on my own time, with no client. The schema, the instructions, the testing, the listing.

Approach

I treated it as an experiment with a hypothesis rather than a product with a roadmap. The question was narrow: can a GPT reliably decide when to call an endpoint, pick the right one, and use what comes back instead of what it already believes.

That framing set the scope. A Pokedex fits because the questions are varied enough to exercise routing and the correct answer is checkable in seconds. I built the smallest thing that could answer the question honestly, then wrote down where it failed.

Building it

The whole product is an OpenAPI schema plus an instruction block. No code I host, no backend. Actions means OpenAI’s runtime makes the request.

Most of the work went into the schema, and that was the finding worth having. The GPT does not read the PokeAPI documentation, it reads the operation descriptions in the schema I supply, and those are the only thing telling it when to call which endpoint. My first version described the endpoints accurately and the model still called the wrong ones, because “get a Pokemon” and “get a Pokemon species” are two resources with an accurate description each and no indication of which answers a question about evolution. Rewriting descriptions to say when to use an operation rather than what it returns fixed almost every routing error. That is a prompt engineering problem wearing an API schema costume.

The second problem was payload size. A single PokeAPI response can carry hundreds of moves, each with a nested list of the game versions it appeared in. Dropping that into a context window burns tokens on data nobody asked about and pushes the answer out. The instructions constrain which fields to attend to per question type, and tell the model to make a second narrower call rather than one enormous one. Two small requests beat one large one, which is the opposite of the instinct you bring from writing normal API clients.

The third was the interesting one: the model would answer from memory instead of calling anything. Ask about a well-known Pokemon and it is confident, so it skips the tool and gives you a plausible number from an old wiki edit. The fix was an instruction block stating that any factual claim about stats, types, abilities or evolutions requires a call, and that answering without one is not permitted. Trained knowledge competing with a live tool is a general failure mode of tool-using models, and this is where I first met it.

Chained calls were where it fell over. An evolution chain means resolving a species, then a chain URL, then walking the chain. Multi-hop sequences in that first release were unreliable, and I documented that rather than pretending otherwise.

Outcome

21 sales. That is the number, and I am not dressing it up: it is a novelty Pokedex on a storefront, and 21 people wanted it.

The outcome I cared about was the answer to the question. Actions worked. A GPT could genuinely call a live REST service and use the response, and the failure modes were about schema description quality, payload size and trained knowledge overriding tool calls rather than plumbing. Every lesson from those evenings went into how I now build tool-using systems, including the research pipeline in SignalScout, which is the same problem at a much higher stake.

What I’d do differently

I shipped a schema covering more endpoints than the product needed, on the theory that more capability is better. It is not. A smaller surface routes more accurately, because every extra operation is another thing the model can pick wrongly.

I would also have put the evolution-chain limitation into the store listing rather than leaving people to find it. Saying what a thing cannot do costs one sentence and buys credibility for the rest of the page.