LLM Logprobs Analyzer (Token Confidence, Perplexity, Uncertain Spans)
Find where a language model was unsure by analysing the logprobs from its API response.
Use the tool
Why this tool exists
With logprobs enabled, APIs return the probability of every generated token and its top alternatives. This analyzer turns that JSON into perplexity, mean log probability, an approximate entropy from the returned candidates, a histogram of token confidence and the text with low-confidence tokens marked, plus a table of those tokens and their rivals.
Token probability is not truth: a model can be confident and wrong, or unsure between harmless synonyms. Use it to decide which parts of an answer to verify, to compare prompts at the same temperature and to understand sampling. Works with chat completions (choices[0].logprobs.content), the legacy completions format and plain arrays of {token, logprob}.
How to use it
- Call the API with logprobs=true (and top_logprobs=5) and copy the response JSON.
- Paste it here and set the low-confidence threshold.
- Check the marked tokens and the alternatives.
Frequently asked questions
Does low probability mean a hallucination?
Not necessarily; it flags uncertainty, which correlates with errors but is not proof.
How is entropy computed?
From the top alternatives returned, renormalised; it is an approximation of the full distribution.
Does it call an API?
No, it only parses the JSON you paste.