Skip to main content

​
Overview

How can we improve the faithfulness of reasoning chains generated by Language Models? Faithful Chain of Thought improves the faithfulness of reasoning chains by breaking it up into two stages:
  1. Translation: Translate a user query into a series of reasoning steps - task-specific steps that can be executed deterministically
  2. Problem Solving: Execute steps and arrive at a final answer that is consistent with the reasoning steps
Examples of task-specific systems:
  • Math Word Problems: PHP code that can be evaluated to derive a final answer
  • Multi-Hop QA: Multi-step reasoning process using programming logic
  • Planning: Generate symbolic goals and use planning systems to solve queries

​
Example

<?php
require 'examples/boot.php';

use Cognesy\Instructor\Extras\Sequence\Sequence;
use Cognesy\Instructor\StructuredOutput;

class ReasoningStep {
    public function __construct(
        public int $id,
        /** @var string[] */
        public array $rationale,
        /** @var int[] */
        public array $dependencies,
        public string $eval_string
    ) {}
}

class TaskSpecificReasoner {
    public function __invoke(string $query): mixed {
        $steps = $this->generateReasoningSteps($query);
        return $this->executeSteps($steps);
    }
    
    private function generateReasoningSteps(string $query): array {
        return (new StructuredOutput)->with(
            messages: [
                [
                    'role' => 'system', 
                    'content' => 'You are a world class AI who excels at generating reasoning steps to answer a question. Generate a list of reasoning steps needed to answer the question.

At each point you should either:
- declare a variable to be referenced later on
- combine multiple variables together to generate a new result that you should store in another variable

The final answer should be stored in a variable called $answer.

Use valid PHP syntax for variable assignments.'
                ],
                ['role' => 'user', 'content' => $query],
            ],
            responseModel: Sequence::of(ReasoningStep::class),
        )->get()->toArray();
    }
    
    private function executeSteps(array $steps): mixed {
        $code = [];
        foreach ($steps as $step) {
            $code[] = $step->eval_string;
        }
        
        $fullCode = "<?php\n" . implode("\n", $code) . "\nreturn \$answer;";
        
        $result = eval(substr($fullCode, 5));
        return $result;
    }
}

$result = (new TaskSpecificReasoner)(
    'If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot after another 2 more arrive?'
);

dump($result);
?>

​
References

  1. Faithful Chain-of-Thought Reasoning
⌘I