Valid JSON is not a valid answer

Three checks before using model output

An API response can parse as JSON and still be unusable. Consider a classifier that must return one of three issue categories: billing, access, or bug.

{"category": "probably billing", "confidence": 4}

The syntax is valid. The category and confidence are not valid for the application.

First, parse the response. Second, validate its structure: required fields, permitted categories and numeric bounds. Third, check the application rules. An existing issue identifier must actually exist, and an action must be permitted for the signed-in user.

Make failure explicit

Keep validation failures separate from transport failures. A timeout may warrant a bounded retry. Invalid fields need a specific correction or a clear failure result. Retrying indefinitely can waste time and hide the actual problem.

For actions that change data, use a stable request identifier and check whether the action already completed before retrying it. Keep generated commands or SQL as reviewable suggestions until an authorized execution step approves them.

A small evaluation set

Test missing fields, unexpected categories, out-of-range numbers and a well-formed response with an incorrect claim. These cases reveal different failure modes; a successful parse should never stand in for all of them.

This is an engineering checklist, not a measured result from a deployed model.

Further reading

JSON Schema’s reference documents type constraints, required properties and allowed values. Schema validation can enforce a data contract; it cannot establish that a model’s claim is true.