Skip to main content
Provider rate limits restrict the number of requests or tokens you can consume within a time window. When you exceed these limits, the provider returns an HTTP 429 response and your request fails. Polyglot provides built-in retry policies to handle these transient failures, but sustained rate limiting requires application-level strategies.

​
Symptoms

  • HTTP status code 429 (Too Many Requests)
  • Error messages containing “rate limit exceeded,” “too many requests,” or “quota exceeded”
  • Requests that work in isolation but fail under load

​
Use the Built-In Retry Policy

Polyglot can automatically retry failed requests with exponential backoff and jitter. Retries are opt-in and explicit — you must attach an InferenceRetryPolicy to the inference builder: 
<?php

use Cognesy\Messages\Messages;
use Cognesy\Polyglot\Inference\Config\InferenceRetryPolicy;
use Cognesy\Polyglot\Inference\Inference;

$text = Inference::using('openai')
    ->withRetryPolicy(new InferenceRetryPolicy(
        maxAttempts: 4,
        baseDelayMs: 250,
        maxDelayMs: 8000,
        jitter: 'full',
    ))
    ->withMessages(Messages::fromString('What is the capital of France?'))
    ->get();
// @doctest id="ac64"

​
Retry Policy Parameters

ParameterDefaultDescription
maxAttempts1Total number of attempts (1 means no retries)
baseDelayMs250Base delay in milliseconds before the first retry
maxDelayMs8000Maximum delay cap in milliseconds
jitter'full'Jitter strategy: none, full, or equal
retryOnStatus[408, 429, 500, 502, 503, 504]HTTP status codes that trigger a retry
retryOnExceptions[TimeoutException, NetworkException]Exception classes that trigger a retry
The delay between retries uses exponential backoff: baseDelayMs * 2^(attempt - 1), capped at maxDelayMs. The jitter strategy adds randomness to avoid thundering herd problems:
  • none — no randomness, uses the exact computed delay
  • full — random delay between 0 and the computed delay
  • equal — half the computed delay plus a random value up to half the computed delay

​
Length Recovery

The retry policy also supports automatic recovery when a response is truncated due to token limits: 
<?php

use Cognesy\Messages\Messages;
use Cognesy\Polyglot\Inference\Config\InferenceRetryPolicy;
use Cognesy\Polyglot\Inference\Inference;

$text = Inference::using('openai')
    ->withRetryPolicy(new InferenceRetryPolicy(
        maxAttempts: 3,
        lengthRecovery: 'continue',       // or 'increase_max_tokens'
        lengthMaxAttempts: 2,
        lengthContinuePrompt: 'Continue.',
        maxTokensIncrement: 512,
    ))
    ->withMessages(Messages::fromString('Write a detailed essay about climate change.'))
    ->get();
// @doctest id="864b"

​
Retry Policy for Embeddings

Embeddings requests use a separate policy class with the same interface: 
<?php

use Cognesy\Polyglot\Embeddings\Config\EmbeddingsRetryPolicy;

$retryPolicy = new EmbeddingsRetryPolicy(
    maxAttempts: 3,
    baseDelayMs: 500,
    maxDelayMs: 10000,
    jitter: 'full',
);
// @doctest id="20c8"

​
Application-Level Throttling

When retries alone are not enough, implement request throttling in your application to stay within the provider’s rate limits: 
<?php

use Cognesy\Messages\Messages;
use Cognesy\Polyglot\Inference\Inference;

class RateLimiter
{
    private float $lastRequestTime = 0;
    private float $minTimeBetweenRequests;

    public function __construct(int $requestsPerMinute = 60) {
        $this->minTimeBetweenRequests = 60.0 / $requestsPerMinute;
    }

    public function waitIfNeeded(): void {
        $elapsed = microtime(true) - $this->lastRequestTime;

        if ($elapsed < $this->minTimeBetweenRequests) {
            usleep((int) (($this->minTimeBetweenRequests - $elapsed) * 1_000_000));
        }

        $this->lastRequestTime = microtime(true);
    }
}

$limiter = new RateLimiter(requestsPerMinute: 30);

for ($i = 0; $i < 10; $i++) {
    $limiter->waitIfNeeded();

    $text = Inference::using('openai')
        ->withMessages(Messages::fromString("This is request $i"))
        ->get();

    echo "Response $i: $text\n";
}
// @doctest id="4ea5"

​
Batch Requests to Reduce Volume

Instead of making many small requests, combine related questions into a single prompt when the use case allows: 
<?php

use Cognesy\Messages\Messages;
use Cognesy\Polyglot\Inference\Inference;

// Instead of N separate requests...
$questions = [
    'What is the capital of France?',
    'What is the capital of Germany?',
    'What is the capital of Japan?',
];

// ...combine them into one request
$batchPrompt = "Answer each question on its own line:\n";
foreach ($questions as $i => $q) {
    $batchPrompt .= ($i + 1) . ". $q\n";
}

$text = Inference::using('openai')
    ->withMessages(Messages::fromString($batchPrompt))
    ->get();
// @doctest id="ee26"
This reduces the number of API calls from N to 1, dramatically lowering rate limit pressure.

​
Additional Strategies

  • Switch providers or models. Different providers and models have different rate limits. If one provider is heavily throttled, route some requests to another.
  • Upgrade your API plan. Most providers offer higher rate limits on paid tiers.
  • Cache responses. If the same prompts recur frequently, cache the results to avoid redundant API calls.
  • Use off-peak hours. Some providers have lower contention during off-peak hours, reducing the likelihood of rate limiting.
  • Monitor usage. Track your request volume and token consumption to anticipate rate limit issues before they affect users.