What does research say about: Prompting LLMs to express confidence improves calibration?

Verbalized probability prompts (asking 'How confident are you? Express as a percentage') reduced Expected Calibration Error by 35% vs unelicited confidence. (Source: Tian et al., 'Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from LLMs', EMNLP 2023). Well-calibrated confidence scores enable downstream systems to route low-confidence outputs to human review, reducing error propagation.

Prompting LLMs to express confidence improves calibration.

Verbalized probability prompts (asking…Verbalized probability prompts (asking 'How confident are you? Express as a percentage') reduced Expected Calibration Error by 35% vs unelicited confidence.

Context & Methodology

Well-calibrated confidence scores enable downstream systems to route low-confidence outputs to human review, reducing error propagation.

Applies To

openaianthropicgoogle

Confidence Level

Medium

Implementation Effort

low

Recommendation

test

Execution Priority

Put This Evidence to Work

Use the STCO framework to implement findings like this in structured, testable prompts.

Start Building Free Browse All 141 Citations

ROI Calculator Token Calculator Prompt Templates