Overview

How do we remove irrelevant information from the prompt?

The S2A (System 2 Attention) technique auto-refines a prompt by asking the model to rewrite the prompt to include only relevant information.

We implement this in two steps:

  1. Ask the model to rewrite the prompt
  2. Pass the rewritten prompt back to the model

Example

<?php
$loader = require 'vendor/autoload.php';
$loader->add('Cognesy\\Instructor\\', __DIR__.'../../src/');

use Cognesy\Instructor\Extras\Scalar\Scalar;
use Cognesy\Instructor\Instructor;
use Cognesy\Instructor\Schema\Attributes\Description;

class RewrittenTask {
    #[Description("Relevant context")]
    public string $relevantContext;
    #[Description("The question from the user")]
    public string $userQuery;
}

class RefineAndSolve {
    private string $prompt = <<<PROMPT
        Given the following text by a user, extract the part
        that is actually relevant to their question. Please
        include the actual question or query that the user
        is asking.
        
        Text by user:
        {query}
        PROMPT;

    public function __invoke(string $problem) : int {
        $rewrittenPrompt = $this->rewritePrompt($problem);
        return (new Instructor)->respond(
            messages: "{$rewrittenPrompt->relevantContext}\nQuestion: {$rewrittenPrompt->userQuery}",
            responseModel: Scalar::integer('answer'),
        );
    }

    private function rewritePrompt(string $query) : RewrittenTask {
        return (new Instructor)->respond(
            messages: str_replace('{query}', $query, $this->prompt),
            responseModel: RewrittenTask::class,
            model: 'gpt-4o',
        );
    }
}

$answer = (new RefineAndSolve)(problem: <<<PROBLEM
    Mary has 3 times as much candy as Megan.
    Mary then adds 10 more pieces of candy to her collection.
    Max is 5 years older than Mary.
    If Megan has 5 pieces of candy, how many does Mary have in total?
    PROBLEM,
);

echo $answer . "\n";
?>

References

  1. System 2 Attention (is something you might need too)