> ## Documentation Index
> Fetch the complete documentation index at: https://docs.instructorphp.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Uncertain examples

## Overview

When we have a large pool of unlabeled examples that could be used in a prompt, how should we decide which examples to manually label?

Active prompting identifies effective examples for human annotation using:

* Uncertainty Estimation: Measure uncertainty on each example.
* Selection: Choose the most uncertain examples for human labeling.
* Annotation: Humans label selected examples.
* Inference: Use newly labeled data to improve prompts.

## Uncertainty Estimation (Disagreement)

Query the same example k times and measure disagreement: unique responses / total responses.

## Example

```php theme={null}
<?php
require 'examples/boot.php';

use Cognesy\Instructor\Extras\Scalar\Scalar;
use Cognesy\Instructor\StructuredOutput;

class EstimateUncertainty {
    public function __invoke(int $k = 5) : float {
        $values = [];
        for ($i = 0; $i < $k; $i++) {
            $values[] = $this->queryHeight();
        }
        return $this->disagreement($values);
    }

    private function queryHeight() : int {
        return StructuredOutput::using('openai')->with(
            messages: [['role' => 'user', 'content' => 'How tall is the Empire State Building in meters?']],
            responseModel: Scalar::integer('height'),
        )->get();
    }

    private function disagreement(array $responses) : float {
        $n = count($responses);
        if ($n === 0) return 0.0;
        return count(array_unique($responses)) / $n;
    }
}

$score = (new EstimateUncertainty)(k: 5);
dump($score);

assert(is_float($score));
assert($score >= 0.0 && $score <= 1.0);
?>
```

### Selection & Annotation

Select the top-n most uncertain unlabeled examples for human annotation.

### Inference

Use newly annotated examples as few-shot context during inference.

### References

1. Active Prompting with Chain-of-Thought for Large Language Models ([https://arxiv.org/abs/2302.12246](https://arxiv.org/abs/2302.12246))
2. The Prompt Report: A Systematic Survey of Prompting Techniques ([https://arxiv.org/abs/2406.06608](https://arxiv.org/abs/2406.06608))