Multi-Provider Fallback

Automatic failover between LLM providers for high availability.

Features

Provider Chain - OpenAI → Anthropic → Ollama fallback
Automatic Retry - Per-provider retry with backoff
Universal Fallback - Falls back on ANY error for maximum availability
Circuit Breaker - Prevents repeated failures to unhealthy providers
Health Check - Verify provider status before critical operations
Streaming Support - Works with both chat and streaming
Provider Logging - Shows which provider succeeded

Quick Start

bash

export OPENAI_API_KEY=your_key
export ANTHROPIC_API_KEY=your_key
npm run recipe:multi-provider-fallback

How It Works

This recipe creates a resilient chat system that automatically fails over between multiple LLM providers when errors occur.

Provider priority:

OpenAI (primary)
Anthropic (secondary)
Ollama (tertiary/local)

Flow:

Send request to primary provider (OpenAI)
If error occurs, retry with exponential backoff
After max retries, fall back to secondary (Anthropic)
If secondary fails, try tertiary (Ollama)
Return response from whichever provider succeeds
Throw error only if all providers fail

Provider Chain

Example Output

╭─────────────────────────────────────────────╮
│  Multi-Provider Fallback                    │
│  High Availability LLM Client               │
╰─────────────────────────────────────────────╯

Provider chain:
  1. ● OpenAI (gpt-4o-mini)
  2. ● Anthropic (claude-3-haiku-20240307)
  3. ● Ollama (llama3.2)

ℹ Configured OpenAI (openai/gpt-4o-mini)
ℹ Configured Anthropic (anthropic/claude-3-haiku-20240307)
ℹ Configured Ollama (ollama/llama3.2)

━━━ Regular Chat ━━━

ℹ Trying OpenAI...
✓ OpenAI succeeded

Response: Four
Provider: OpenAI

━━━ Streaming Chat ━━━

ℹ Trying OpenAI (streaming)...
✓ OpenAI started streaming
Response: 1
2
3
4
5
Provider: OpenAI

Fallback Triggers

The client falls back on ANY error to maximize availability. This is a deliberate design choice - when uptime matters more than knowing exactly what failed, universal fallback ensures requests succeed whenever possible.

Error	Description	Fallback?
`RateLimitError`	Provider rate limit exceeded	✓
`TimeoutError`	Request timed out	✓
`ServerError`	Provider server error (5xx)	✓
`AuthenticationError`	Invalid API key	✓
Other errors	Network issues, parsing errors, etc.	✓

Streaming Behavior

When streaming fails mid-response, the partial response is lost and fallback starts fresh with the next provider. This is intentional:

Simpler implementation
Consistent user experience (no partial + retry concatenation)
Most streaming failures happen at connection, not mid-stream

If you need to preserve partial responses, consider buffering chunks yourself and implementing custom recovery.

Code Walkthrough

Types and Configuration

typescript

/**
 * Multi-Provider Fallback Library
 *
 * Exported functions for the Multi-Provider Fallback recipe.
 * Snippet markers allow VitePress to extract code for documentation.
 */

import { ChatClient, ChatError } from '../../../src';

// [start:colors]
export const colors = {
  reset: '\x1b[0m',
  dim: '\x1b[2m',
  bold: '\x1b[1m',
  red: '\x1b[31m',
  yellow: '\x1b[33m',
  green: '\x1b[32m',
  cyan: '\x1b[36m',
  magenta: '\x1b[35m'
};
// [end:colors]

// [start:types]
export interface CircuitBreakerConfig {
  failureThreshold: number; // Open circuit after N failures
  resetTimeoutMs: number; // Try again after N ms
}

export interface ProviderConfig {
  name: string;
  provider: 'openai' | 'anthropic' | 'ollama';
  model: string;
  apiKey?: string;
  baseUrl?: string;
  retryConfig: {
    maxAttempts: number;
    initialBackoffMs: number;
    maxBackoffMs: number;
    backoffMultiplier: number;
  };
  circuitBreaker?: CircuitBreakerConfig;
}

export interface CircuitState {
  failures: number;
  lastFailure: number;
  isOpen: boolean;
  config: CircuitBreakerConfig;
}

export interface HealthCheckResult {
  name: string;
  healthy: boolean;
  latencyMs?: number;
  error?: string;
}

export interface ChatResult {
  content: string;
  provider: string;
}

export interface StreamChunk {
  content: string;
  provider: string;
  done: boolean;
}

export interface CircuitStatus {
  name: string;
  isOpen: boolean;
  failures: number;
}
// [end:types]

// [start:default-config]
export const DEFAULT_CIRCUIT_BREAKER: CircuitBreakerConfig = {
  failureThreshold: 3,
  resetTimeoutMs: 30000
};
// [end:default-config]

// [start:logging]
export function formatError(error: unknown): string {
  if (error instanceof ChatError) {
    return `${error.name}: ${error.message}`;
  }
  if (error instanceof Error) {
    return error.message;
  }
  return String(error);
}

export function printBanner(): void {
  console.log(`
${colors.cyan}╭─────────────────────────────────────────────╮
│  ${colors.reset}${colors.bold}Multi-Provider Fallback${colors.reset}${colors.cyan}                    │
│  ${colors.dim}High Availability LLM Client${colors.reset}${colors.cyan}                │
╰─────────────────────────────────────────────╯${colors.reset}
`);
}

export function log(level: 'info' | 'warn' | 'error' | 'success', message: string): void {
  const prefix = {
    info: `${colors.cyan}ℹ${colors.reset}`,
    warn: `${colors.yellow}⚠${colors.reset}`,
    error: `${colors.red}✗${colors.reset}`,
    success: `${colors.green}✓${colors.reset}`
  };
  console.log(`${prefix[level]} ${message}`);
}
// [end:logging]

// [start:fallback-client]
/**
 * Multi-provider client with automatic fallback and circuit breaker
 */
export class FallbackClient {
  private providers: ProviderConfig[];
  private clients: Map<string, ChatClient> = new Map();
  private circuits: Map<string, CircuitState> = new Map();

  constructor(providers: ProviderConfig[]) {
    this.providers = providers;

    // Pre-create clients for each provider
    for (const config of providers) {
      try {
        const client = new ChatClient({
          provider: config.provider,
          model: config.model,
          apiKey: config.apiKey,
          baseUrl: config.baseUrl,
          retryConfig: config.retryConfig
        });
        this.clients.set(config.name, client);
        this.circuits.set(config.name, {
          failures: 0,
          lastFailure: 0,
          isOpen: false,
          config: config.circuitBreaker ?? DEFAULT_CIRCUIT_BREAKER
        });
        log('info', `Configured ${config.name} (${config.provider}/${config.model})`);
      } catch (error) {
        log('warn', `Failed to configure ${config.name}: ${formatError(error)}`);
      }
    }

    if (this.clients.size === 0) {
      throw new Error('No providers could be configured');
    }
  }

  /**
   * Check if circuit breaker allows requests to this provider
   */
  private isCircuitClosed(name: string): boolean {
    const circuit = this.circuits.get(name);
    if (!circuit) return false;

    if (!circuit.isOpen) return true;

    // Check if enough time has passed to try again
    const elapsed = Date.now() - circuit.lastFailure;
    if (elapsed >= circuit.config.resetTimeoutMs) {
      circuit.isOpen = false;
      circuit.failures = 0;
      log('info', `Circuit closed for ${name} (reset after ${circuit.config.resetTimeoutMs}ms)`);
      return true;
    }

    return false;
  }

  /**
   * Record a failure for circuit breaker
   */
  private recordFailure(name: string): void {
    const circuit = this.circuits.get(name);
    if (!circuit) return;

    circuit.failures++;
    circuit.lastFailure = Date.now();

    if (circuit.failures >= circuit.config.failureThreshold) {
      circuit.isOpen = true;
      log('warn', `Circuit opened for ${name} (${circuit.failures} failures)`);
    }
  }

  /**
   * Record a success - reset circuit breaker
   */
  private recordSuccess(name: string): void {
    const circuit = this.circuits.get(name);
    if (!circuit) return;

    circuit.failures = 0;
    circuit.isOpen = false;
  }

  /**
   * Health check - ping all providers
   * @param message Optional message to send (default: "ping")
   */
  async healthCheck(message = 'ping'): Promise<HealthCheckResult[]> {
    const results: HealthCheckResult[] = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) {
        results.push({ name: config.name, healthy: false, error: 'Not configured' });
        continue;
      }

      const start = Date.now();
      try {
        await client.chat(message);
        results.push({
          name: config.name,
          healthy: true,
          latencyMs: Date.now() - start
        });
      } catch (error) {
        results.push({
          name: config.name,
          healthy: false,
          latencyMs: Date.now() - start,
          error: formatError(error)
        });
      }
    }

    return results;
  }

  /**
   * Send a chat message with automatic fallback
   * Falls back on ANY error for maximum availability
   */
  async chat(message: string): Promise<ChatResult> {
    const errors: Array<{ provider: string; error: unknown }> = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) continue;

      // Check circuit breaker
      if (!this.isCircuitClosed(config.name)) {
        log('warn', `${config.name} circuit open, skipping`);
        continue;
      }

      log('info', `Trying ${config.name}...`);

      try {
        const response = await client.chat(message);
        this.recordSuccess(config.name);
        log('success', `${config.name} succeeded`);
        return {
          content: response.content,
          provider: config.name
        };
      } catch (error) {
        this.recordFailure(config.name);
        errors.push({ provider: config.name, error });
        log('error', `${config.name} failed: ${formatError(error)}`);
        log('warn', `Falling back to next provider...`);
      }
    }

    // All providers failed
    const errorSummary = errors.map((e) => `  - ${e.provider}: ${formatError(e.error)}`).join('\n');
    throw new Error(`All providers failed:\n${errorSummary}`);
  }

  // [start:streaming]
  /**
   * Stream a chat message with automatic fallback
   *
   * Note: If streaming fails mid-stream, partial response is lost.
   * Fallback starts fresh from the beginning with next provider.
   */
  async *stream(message: string): AsyncGenerator<StreamChunk> {
    const errors: Array<{ provider: string; error: unknown }> = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) continue;

      // Check circuit breaker
      if (!this.isCircuitClosed(config.name)) {
        log('warn', `${config.name} circuit open, skipping`);
        continue;
      }

      log('info', `Trying ${config.name} (streaming)...`);

      try {
        let hasYielded = false;
        for await (const chunk of client.stream(message)) {
          if (!hasYielded) {
            log('success', `${config.name} started streaming`);
            hasYielded = true;
          }
          yield {
            content: chunk.content,
            provider: config.name,
            done: false
          };
        }
        this.recordSuccess(config.name);
        yield {
          content: '',
          provider: config.name,
          done: true
        };
        return; // Success, exit generator
      } catch (error) {
        this.recordFailure(config.name);
        errors.push({ provider: config.name, error });
        log('error', `${config.name} failed: ${formatError(error)}`);
        log('warn', `Falling back to next provider...`);
      }
    }

    // All providers failed
    const errorSummary = errors.map((e) => `  - ${e.provider}: ${formatError(e.error)}`).join('\n');
    throw new Error(`All providers failed:\n${errorSummary}`);
  }
  // [end:streaming]

  /**
   * Get list of configured providers
   */
  getProviders(): string[] {
    return Array.from(this.clients.keys());
  }

  /**
   * Get circuit breaker status for all providers
   */
  getCircuitStatus(): CircuitStatus[] {
    return this.providers
      .filter((p) => this.circuits.has(p.name))
      .map((p) => {
        const circuit = this.circuits.get(p.name)!;
        return {
          name: p.name,
          isOpen: circuit.isOpen,
          failures: circuit.failures
        };
      });
  }
}
// [end:fallback-client]

// [start:create-providers]
export function createDefaultProviders(): ProviderConfig[] {
  const providers: ProviderConfig[] = [];

  if (process.env.OPENAI_API_KEY) {
    providers.push({
      name: 'OpenAI',
      provider: 'openai',
      model: 'gpt-4o-mini',
      apiKey: process.env.OPENAI_API_KEY,
      retryConfig: {
        maxAttempts: 2,
        initialBackoffMs: 1000,
        maxBackoffMs: 5000,
        backoffMultiplier: 2
      }
    });
  }

  if (process.env.ANTHROPIC_API_KEY) {
    providers.push({
      name: 'Anthropic',
      provider: 'anthropic',
      model: 'claude-3-haiku-20240307',
      apiKey: process.env.ANTHROPIC_API_KEY,
      retryConfig: {
        maxAttempts: 2,
        initialBackoffMs: 1000,
        maxBackoffMs: 5000,
        backoffMultiplier: 2
      }
    });
  }

  // Always add Ollama as local fallback (may not be running)
  providers.push({
    name: 'Ollama',
    provider: 'ollama',
    model: 'llama3.2',
    baseUrl: 'http://localhost:11434',
    retryConfig: {
      maxAttempts: 1,
      initialBackoffMs: 500,
      maxBackoffMs: 2000,
      backoffMultiplier: 2
    }
  });

  return providers;
}
// [end:create-providers]

Default Circuit Breaker

typescript

/**
 * Multi-Provider Fallback Library
 *
 * Exported functions for the Multi-Provider Fallback recipe.
 * Snippet markers allow VitePress to extract code for documentation.
 */

import { ChatClient, ChatError } from '../../../src';

// [start:colors]
export const colors = {
  reset: '\x1b[0m',
  dim: '\x1b[2m',
  bold: '\x1b[1m',
  red: '\x1b[31m',
  yellow: '\x1b[33m',
  green: '\x1b[32m',
  cyan: '\x1b[36m',
  magenta: '\x1b[35m'
};
// [end:colors]

// [start:types]
export interface CircuitBreakerConfig {
  failureThreshold: number; // Open circuit after N failures
  resetTimeoutMs: number; // Try again after N ms
}

export interface ProviderConfig {
  name: string;
  provider: 'openai' | 'anthropic' | 'ollama';
  model: string;
  apiKey?: string;
  baseUrl?: string;
  retryConfig: {
    maxAttempts: number;
    initialBackoffMs: number;
    maxBackoffMs: number;
    backoffMultiplier: number;
  };
  circuitBreaker?: CircuitBreakerConfig;
}

export interface CircuitState {
  failures: number;
  lastFailure: number;
  isOpen: boolean;
  config: CircuitBreakerConfig;
}

export interface HealthCheckResult {
  name: string;
  healthy: boolean;
  latencyMs?: number;
  error?: string;
}

export interface ChatResult {
  content: string;
  provider: string;
}

export interface StreamChunk {
  content: string;
  provider: string;
  done: boolean;
}

export interface CircuitStatus {
  name: string;
  isOpen: boolean;
  failures: number;
}
// [end:types]

// [start:default-config]
export const DEFAULT_CIRCUIT_BREAKER: CircuitBreakerConfig = {
  failureThreshold: 3,
  resetTimeoutMs: 30000
};
// [end:default-config]

// [start:logging]
export function formatError(error: unknown): string {
  if (error instanceof ChatError) {
    return `${error.name}: ${error.message}`;
  }
  if (error instanceof Error) {
    return error.message;
  }
  return String(error);
}

export function printBanner(): void {
  console.log(`
${colors.cyan}╭─────────────────────────────────────────────╮
│  ${colors.reset}${colors.bold}Multi-Provider Fallback${colors.reset}${colors.cyan}                    │
│  ${colors.dim}High Availability LLM Client${colors.reset}${colors.cyan}                │
╰─────────────────────────────────────────────╯${colors.reset}
`);
}

export function log(level: 'info' | 'warn' | 'error' | 'success', message: string): void {
  const prefix = {
    info: `${colors.cyan}ℹ${colors.reset}`,
    warn: `${colors.yellow}⚠${colors.reset}`,
    error: `${colors.red}✗${colors.reset}`,
    success: `${colors.green}✓${colors.reset}`
  };
  console.log(`${prefix[level]} ${message}`);
}
// [end:logging]

// [start:fallback-client]
/**
 * Multi-provider client with automatic fallback and circuit breaker
 */
export class FallbackClient {
  private providers: ProviderConfig[];
  private clients: Map<string, ChatClient> = new Map();
  private circuits: Map<string, CircuitState> = new Map();

  constructor(providers: ProviderConfig[]) {
    this.providers = providers;

    // Pre-create clients for each provider
    for (const config of providers) {
      try {
        const client = new ChatClient({
          provider: config.provider,
          model: config.model,
          apiKey: config.apiKey,
          baseUrl: config.baseUrl,
          retryConfig: config.retryConfig
        });
        this.clients.set(config.name, client);
        this.circuits.set(config.name, {
          failures: 0,
          lastFailure: 0,
          isOpen: false,
          config: config.circuitBreaker ?? DEFAULT_CIRCUIT_BREAKER
        });
        log('info', `Configured ${config.name} (${config.provider}/${config.model})`);
      } catch (error) {
        log('warn', `Failed to configure ${config.name}: ${formatError(error)}`);
      }
    }

    if (this.clients.size === 0) {
      throw new Error('No providers could be configured');
    }
  }

  /**
   * Check if circuit breaker allows requests to this provider
   */
  private isCircuitClosed(name: string): boolean {
    const circuit = this.circuits.get(name);
    if (!circuit) return false;

    if (!circuit.isOpen) return true;

    // Check if enough time has passed to try again
    const elapsed = Date.now() - circuit.lastFailure;
    if (elapsed >= circuit.config.resetTimeoutMs) {
      circuit.isOpen = false;
      circuit.failures = 0;
      log('info', `Circuit closed for ${name} (reset after ${circuit.config.resetTimeoutMs}ms)`);
      return true;
    }

    return false;
  }

  /**
   * Record a failure for circuit breaker
   */
  private recordFailure(name: string): void {
    const circuit = this.circuits.get(name);
    if (!circuit) return;

    circuit.failures++;
    circuit.lastFailure = Date.now();

    if (circuit.failures >= circuit.config.failureThreshold) {
      circuit.isOpen = true;
      log('warn', `Circuit opened for ${name} (${circuit.failures} failures)`);
    }
  }

  /**
   * Record a success - reset circuit breaker
   */
  private recordSuccess(name: string): void {
    const circuit = this.circuits.get(name);
    if (!circuit) return;

    circuit.failures = 0;
    circuit.isOpen = false;
  }

  /**
   * Health check - ping all providers
   * @param message Optional message to send (default: "ping")
   */
  async healthCheck(message = 'ping'): Promise<HealthCheckResult[]> {
    const results: HealthCheckResult[] = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) {
        results.push({ name: config.name, healthy: false, error: 'Not configured' });
        continue;
      }

      const start = Date.now();
      try {
        await client.chat(message);
        results.push({
          name: config.name,
          healthy: true,
          latencyMs: Date.now() - start
        });
      } catch (error) {
        results.push({
          name: config.name,
          healthy: false,
          latencyMs: Date.now() - start,
          error: formatError(error)
        });
      }
    }

    return results;
  }

  /**
   * Send a chat message with automatic fallback
   * Falls back on ANY error for maximum availability
   */
  async chat(message: string): Promise<ChatResult> {
    const errors: Array<{ provider: string; error: unknown }> = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) continue;

      // Check circuit breaker
      if (!this.isCircuitClosed(config.name)) {
        log('warn', `${config.name} circuit open, skipping`);
        continue;
      }

      log('info', `Trying ${config.name}...`);

      try {
        const response = await client.chat(message);
        this.recordSuccess(config.name);
        log('success', `${config.name} succeeded`);
        return {
          content: response.content,
          provider: config.name
        };
      } catch (error) {
        this.recordFailure(config.name);
        errors.push({ provider: config.name, error });
        log('error', `${config.name} failed: ${formatError(error)}`);
        log('warn', `Falling back to next provider...`);
      }
    }

    // All providers failed
    const errorSummary = errors.map((e) => `  - ${e.provider}: ${formatError(e.error)}`).join('\n');
    throw new Error(`All providers failed:\n${errorSummary}`);
  }

  // [start:streaming]
  /**
   * Stream a chat message with automatic fallback
   *
   * Note: If streaming fails mid-stream, partial response is lost.
   * Fallback starts fresh from the beginning with next provider.
   */
  async *stream(message: string): AsyncGenerator<StreamChunk> {
    const errors: Array<{ provider: string; error: unknown }> = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) continue;

      // Check circuit breaker
      if (!this.isCircuitClosed(config.name)) {
        log('warn', `${config.name} circuit open, skipping`);
        continue;
      }

      log('info', `Trying ${config.name} (streaming)...`);

      try {
        let hasYielded = false;
        for await (const chunk of client.stream(message)) {
          if (!hasYielded) {
            log('success', `${config.name} started streaming`);
            hasYielded = true;
          }
          yield {
            content: chunk.content,
            provider: config.name,
            done: false
          };
        }
        this.recordSuccess(config.name);
        yield {
          content: '',
          provider: config.name,
          done: true
        };
        return; // Success, exit generator
      } catch (error) {
        this.recordFailure(config.name);
        errors.push({ provider: config.name, error });
        log('error', `${config.name} failed: ${formatError(error)}`);
        log('warn', `Falling back to next provider...`);
      }
    }

    // All providers failed
    const errorSummary = errors.map((e) => `  - ${e.provider}: ${formatError(e.error)}`).join('\n');
    throw new Error(`All providers failed:\n${errorSummary}`);
  }
  // [end:streaming]

  /**
   * Get list of configured providers
   */
  getProviders(): string[] {
    return Array.from(this.clients.keys());
  }

  /**
   * Get circuit breaker status for all providers
   */
  getCircuitStatus(): CircuitStatus[] {
    return this.providers
      .filter((p) => this.circuits.has(p.name))
      .map((p) => {
        const circuit = this.circuits.get(p.name)!;
        return {
          name: p.name,
          isOpen: circuit.isOpen,
          failures: circuit.failures
        };
      });
  }
}
// [end:fallback-client]

// [start:create-providers]
export function createDefaultProviders(): ProviderConfig[] {
  const providers: ProviderConfig[] = [];

  if (process.env.OPENAI_API_KEY) {
    providers.push({
      name: 'OpenAI',
      provider: 'openai',
      model: 'gpt-4o-mini',
      apiKey: process.env.OPENAI_API_KEY,
      retryConfig: {
        maxAttempts: 2,
        initialBackoffMs: 1000,
        maxBackoffMs: 5000,
        backoffMultiplier: 2
      }
    });
  }

  if (process.env.ANTHROPIC_API_KEY) {
    providers.push({
      name: 'Anthropic',
      provider: 'anthropic',
      model: 'claude-3-haiku-20240307',
      apiKey: process.env.ANTHROPIC_API_KEY,
      retryConfig: {
        maxAttempts: 2,
        initialBackoffMs: 1000,
        maxBackoffMs: 5000,
        backoffMultiplier: 2
      }
    });
  }

  // Always add Ollama as local fallback (may not be running)
  providers.push({
    name: 'Ollama',
    provider: 'ollama',
    model: 'llama3.2',
    baseUrl: 'http://localhost:11434',
    retryConfig: {
      maxAttempts: 1,
      initialBackoffMs: 500,
      maxBackoffMs: 2000,
      backoffMultiplier: 2
    }
  });

  return providers;
}
// [end:create-providers]

Fallback Client

typescript

/**
 * Multi-Provider Fallback Library
 *
 * Exported functions for the Multi-Provider Fallback recipe.
 * Snippet markers allow VitePress to extract code for documentation.
 */

import { ChatClient, ChatError } from '../../../src';

// [start:colors]
export const colors = {
  reset: '\x1b[0m',
  dim: '\x1b[2m',
  bold: '\x1b[1m',
  red: '\x1b[31m',
  yellow: '\x1b[33m',
  green: '\x1b[32m',
  cyan: '\x1b[36m',
  magenta: '\x1b[35m'
};
// [end:colors]

// [start:types]
export interface CircuitBreakerConfig {
  failureThreshold: number; // Open circuit after N failures
  resetTimeoutMs: number; // Try again after N ms
}

export interface ProviderConfig {
  name: string;
  provider: 'openai' | 'anthropic' | 'ollama';
  model: string;
  apiKey?: string;
  baseUrl?: string;
  retryConfig: {
    maxAttempts: number;
    initialBackoffMs: number;
    maxBackoffMs: number;
    backoffMultiplier: number;
  };
  circuitBreaker?: CircuitBreakerConfig;
}

export interface CircuitState {
  failures: number;
  lastFailure: number;
  isOpen: boolean;
  config: CircuitBreakerConfig;
}

export interface HealthCheckResult {
  name: string;
  healthy: boolean;
  latencyMs?: number;
  error?: string;
}

export interface ChatResult {
  content: string;
  provider: string;
}

export interface StreamChunk {
  content: string;
  provider: string;
  done: boolean;
}

export interface CircuitStatus {
  name: string;
  isOpen: boolean;
  failures: number;
}
// [end:types]

// [start:default-config]
export const DEFAULT_CIRCUIT_BREAKER: CircuitBreakerConfig = {
  failureThreshold: 3,
  resetTimeoutMs: 30000
};
// [end:default-config]

// [start:logging]
export function formatError(error: unknown): string {
  if (error instanceof ChatError) {
    return `${error.name}: ${error.message}`;
  }
  if (error instanceof Error) {
    return error.message;
  }
  return String(error);
}

export function printBanner(): void {
  console.log(`
${colors.cyan}╭─────────────────────────────────────────────╮
│  ${colors.reset}${colors.bold}Multi-Provider Fallback${colors.reset}${colors.cyan}                    │
│  ${colors.dim}High Availability LLM Client${colors.reset}${colors.cyan}                │
╰─────────────────────────────────────────────╯${colors.reset}
`);
}

export function log(level: 'info' | 'warn' | 'error' | 'success', message: string): void {
  const prefix = {
    info: `${colors.cyan}ℹ${colors.reset}`,
    warn: `${colors.yellow}⚠${colors.reset}`,
    error: `${colors.red}✗${colors.reset}`,
    success: `${colors.green}✓${colors.reset}`
  };
  console.log(`${prefix[level]} ${message}`);
}
// [end:logging]

// [start:fallback-client]
/**
 * Multi-provider client with automatic fallback and circuit breaker
 */
export class FallbackClient {
  private providers: ProviderConfig[];
  private clients: Map<string, ChatClient> = new Map();
  private circuits: Map<string, CircuitState> = new Map();

  constructor(providers: ProviderConfig[]) {
    this.providers = providers;

    // Pre-create clients for each provider
    for (const config of providers) {
      try {
        const client = new ChatClient({
          provider: config.provider,
          model: config.model,
          apiKey: config.apiKey,
          baseUrl: config.baseUrl,
          retryConfig: config.retryConfig
        });
        this.clients.set(config.name, client);
        this.circuits.set(config.name, {
          failures: 0,
          lastFailure: 0,
          isOpen: false,
          config: config.circuitBreaker ?? DEFAULT_CIRCUIT_BREAKER
        });
        log('info', `Configured ${config.name} (${config.provider}/${config.model})`);
      } catch (error) {
        log('warn', `Failed to configure ${config.name}: ${formatError(error)}`);
      }
    }

    if (this.clients.size === 0) {
      throw new Error('No providers could be configured');
    }
  }

  /**
   * Check if circuit breaker allows requests to this provider
   */
  private isCircuitClosed(name: string): boolean {
    const circuit = this.circuits.get(name);
    if (!circuit) return false;

    if (!circuit.isOpen) return true;

    // Check if enough time has passed to try again
    const elapsed = Date.now() - circuit.lastFailure;
    if (elapsed >= circuit.config.resetTimeoutMs) {
      circuit.isOpen = false;
      circuit.failures = 0;
      log('info', `Circuit closed for ${name} (reset after ${circuit.config.resetTimeoutMs}ms)`);
      return true;
    }

    return false;
  }

  /**
   * Record a failure for circuit breaker
   */
  private recordFailure(name: string): void {
    const circuit = this.circuits.get(name);
    if (!circuit) return;

    circuit.failures++;
    circuit.lastFailure = Date.now();

    if (circuit.failures >= circuit.config.failureThreshold) {
      circuit.isOpen = true;
      log('warn', `Circuit opened for ${name} (${circuit.failures} failures)`);
    }
  }

  /**
   * Record a success - reset circuit breaker
   */
  private recordSuccess(name: string): void {
    const circuit = this.circuits.get(name);
    if (!circuit) return;

    circuit.failures = 0;
    circuit.isOpen = false;
  }

  /**
   * Health check - ping all providers
   * @param message Optional message to send (default: "ping")
   */
  async healthCheck(message = 'ping'): Promise<HealthCheckResult[]> {
    const results: HealthCheckResult[] = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) {
        results.push({ name: config.name, healthy: false, error: 'Not configured' });
        continue;
      }

      const start = Date.now();
      try {
        await client.chat(message);
        results.push({
          name: config.name,
          healthy: true,
          latencyMs: Date.now() - start
        });
      } catch (error) {
        results.push({
          name: config.name,
          healthy: false,
          latencyMs: Date.now() - start,
          error: formatError(error)
        });
      }
    }

    return results;
  }

  /**
   * Send a chat message with automatic fallback
   * Falls back on ANY error for maximum availability
   */
  async chat(message: string): Promise<ChatResult> {
    const errors: Array<{ provider: string; error: unknown }> = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) continue;

      // Check circuit breaker
      if (!this.isCircuitClosed(config.name)) {
        log('warn', `${config.name} circuit open, skipping`);
        continue;
      }

      log('info', `Trying ${config.name}...`);

      try {
        const response = await client.chat(message);
        this.recordSuccess(config.name);
        log('success', `${config.name} succeeded`);
        return {
          content: response.content,
          provider: config.name
        };
      } catch (error) {
        this.recordFailure(config.name);
        errors.push({ provider: config.name, error });
        log('error', `${config.name} failed: ${formatError(error)}`);
        log('warn', `Falling back to next provider...`);
      }
    }

    // All providers failed
    const errorSummary = errors.map((e) => `  - ${e.provider}: ${formatError(e.error)}`).join('\n');
    throw new Error(`All providers failed:\n${errorSummary}`);
  }

  // [start:streaming]
  /**
   * Stream a chat message with automatic fallback
   *
   * Note: If streaming fails mid-stream, partial response is lost.
   * Fallback starts fresh from the beginning with next provider.
   */
  async *stream(message: string): AsyncGenerator<StreamChunk> {
    const errors: Array<{ provider: string; error: unknown }> = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) continue;

      // Check circuit breaker
      if (!this.isCircuitClosed(config.name)) {
        log('warn', `${config.name} circuit open, skipping`);
        continue;
      }

      log('info', `Trying ${config.name} (streaming)...`);

      try {
        let hasYielded = false;
        for await (const chunk of client.stream(message)) {
          if (!hasYielded) {
            log('success', `${config.name} started streaming`);
            hasYielded = true;
          }
          yield {
            content: chunk.content,
            provider: config.name,
            done: false
          };
        }
        this.recordSuccess(config.name);
        yield {
          content: '',
          provider: config.name,
          done: true
        };
        return; // Success, exit generator
      } catch (error) {
        this.recordFailure(config.name);
        errors.push({ provider: config.name, error });
        log('error', `${config.name} failed: ${formatError(error)}`);
        log('warn', `Falling back to next provider...`);
      }
    }

    // All providers failed
    const errorSummary = errors.map((e) => `  - ${e.provider}: ${formatError(e.error)}`).join('\n');
    throw new Error(`All providers failed:\n${errorSummary}`);
  }
  // [end:streaming]

  /**
   * Get list of configured providers
   */
  getProviders(): string[] {
    return Array.from(this.clients.keys());
  }

  /**
   * Get circuit breaker status for all providers
   */
  getCircuitStatus(): CircuitStatus[] {
    return this.providers
      .filter((p) => this.circuits.has(p.name))
      .map((p) => {
        const circuit = this.circuits.get(p.name)!;
        return {
          name: p.name,
          isOpen: circuit.isOpen,
          failures: circuit.failures
        };
      });
  }
}
// [end:fallback-client]

// [start:create-providers]
export function createDefaultProviders(): ProviderConfig[] {
  const providers: ProviderConfig[] = [];

  if (process.env.OPENAI_API_KEY) {
    providers.push({
      name: 'OpenAI',
      provider: 'openai',
      model: 'gpt-4o-mini',
      apiKey: process.env.OPENAI_API_KEY,
      retryConfig: {
        maxAttempts: 2,
        initialBackoffMs: 1000,
        maxBackoffMs: 5000,
        backoffMultiplier: 2
      }
    });
  }

  if (process.env.ANTHROPIC_API_KEY) {
    providers.push({
      name: 'Anthropic',
      provider: 'anthropic',
      model: 'claude-3-haiku-20240307',
      apiKey: process.env.ANTHROPIC_API_KEY,
      retryConfig: {
        maxAttempts: 2,
        initialBackoffMs: 1000,
        maxBackoffMs: 5000,
        backoffMultiplier: 2
      }
    });
  }

  // Always add Ollama as local fallback (may not be running)
  providers.push({
    name: 'Ollama',
    provider: 'ollama',
    model: 'llama3.2',
    baseUrl: 'http://localhost:11434',
    retryConfig: {
      maxAttempts: 1,
      initialBackoffMs: 500,
      maxBackoffMs: 2000,
      backoffMultiplier: 2
    }
  });

  return providers;
}
// [end:create-providers]

Streaming with Fallback

typescript

/**
 * Multi-Provider Fallback Library
 *
 * Exported functions for the Multi-Provider Fallback recipe.
 * Snippet markers allow VitePress to extract code for documentation.
 */

import { ChatClient, ChatError } from '../../../src';

// [start:colors]
export const colors = {
  reset: '\x1b[0m',
  dim: '\x1b[2m',
  bold: '\x1b[1m',
  red: '\x1b[31m',
  yellow: '\x1b[33m',
  green: '\x1b[32m',
  cyan: '\x1b[36m',
  magenta: '\x1b[35m'
};
// [end:colors]

// [start:types]
export interface CircuitBreakerConfig {
  failureThreshold: number; // Open circuit after N failures
  resetTimeoutMs: number; // Try again after N ms
}

export interface ProviderConfig {
  name: string;
  provider: 'openai' | 'anthropic' | 'ollama';
  model: string;
  apiKey?: string;
  baseUrl?: string;
  retryConfig: {
    maxAttempts: number;
    initialBackoffMs: number;
    maxBackoffMs: number;
    backoffMultiplier: number;
  };
  circuitBreaker?: CircuitBreakerConfig;
}

export interface CircuitState {
  failures: number;
  lastFailure: number;
  isOpen: boolean;
  config: CircuitBreakerConfig;
}

export interface HealthCheckResult {
  name: string;
  healthy: boolean;
  latencyMs?: number;
  error?: string;
}

export interface ChatResult {
  content: string;
  provider: string;
}

export interface StreamChunk {
  content: string;
  provider: string;
  done: boolean;
}

export interface CircuitStatus {
  name: string;
  isOpen: boolean;
  failures: number;
}
// [end:types]

// [start:default-config]
export const DEFAULT_CIRCUIT_BREAKER: CircuitBreakerConfig = {
  failureThreshold: 3,
  resetTimeoutMs: 30000
};
// [end:default-config]

// [start:logging]
export function formatError(error: unknown): string {
  if (error instanceof ChatError) {
    return `${error.name}: ${error.message}`;
  }
  if (error instanceof Error) {
    return error.message;
  }
  return String(error);
}

export function printBanner(): void {
  console.log(`
${colors.cyan}╭─────────────────────────────────────────────╮
│  ${colors.reset}${colors.bold}Multi-Provider Fallback${colors.reset}${colors.cyan}                    │
│  ${colors.dim}High Availability LLM Client${colors.reset}${colors.cyan}                │
╰─────────────────────────────────────────────╯${colors.reset}
`);
}

export function log(level: 'info' | 'warn' | 'error' | 'success', message: string): void {
  const prefix = {
    info: `${colors.cyan}ℹ${colors.reset}`,
    warn: `${colors.yellow}⚠${colors.reset}`,
    error: `${colors.red}✗${colors.reset}`,
    success: `${colors.green}✓${colors.reset}`
  };
  console.log(`${prefix[level]} ${message}`);
}
// [end:logging]

// [start:fallback-client]
/**
 * Multi-provider client with automatic fallback and circuit breaker
 */
export class FallbackClient {
  private providers: ProviderConfig[];
  private clients: Map<string, ChatClient> = new Map();
  private circuits: Map<string, CircuitState> = new Map();

  constructor(providers: ProviderConfig[]) {
    this.providers = providers;

    // Pre-create clients for each provider
    for (const config of providers) {
      try {
        const client = new ChatClient({
          provider: config.provider,
          model: config.model,
          apiKey: config.apiKey,
          baseUrl: config.baseUrl,
          retryConfig: config.retryConfig
        });
        this.clients.set(config.name, client);
        this.circuits.set(config.name, {
          failures: 0,
          lastFailure: 0,
          isOpen: false,
          config: config.circuitBreaker ?? DEFAULT_CIRCUIT_BREAKER
        });
        log('info', `Configured ${config.name} (${config.provider}/${config.model})`);
      } catch (error) {
        log('warn', `Failed to configure ${config.name}: ${formatError(error)}`);
      }
    }

    if (this.clients.size === 0) {
      throw new Error('No providers could be configured');
    }
  }

  /**
   * Check if circuit breaker allows requests to this provider
   */
  private isCircuitClosed(name: string): boolean {
    const circuit = this.circuits.get(name);
    if (!circuit) return false;

    if (!circuit.isOpen) return true;

    // Check if enough time has passed to try again
    const elapsed = Date.now() - circuit.lastFailure;
    if (elapsed >= circuit.config.resetTimeoutMs) {
      circuit.isOpen = false;
      circuit.failures = 0;
      log('info', `Circuit closed for ${name} (reset after ${circuit.config.resetTimeoutMs}ms)`);
      return true;
    }

    return false;
  }

  /**
   * Record a failure for circuit breaker
   */
  private recordFailure(name: string): void {
    const circuit = this.circuits.get(name);
    if (!circuit) return;

    circuit.failures++;
    circuit.lastFailure = Date.now();

    if (circuit.failures >= circuit.config.failureThreshold) {
      circuit.isOpen = true;
      log('warn', `Circuit opened for ${name} (${circuit.failures} failures)`);
    }
  }

  /**
   * Record a success - reset circuit breaker
   */
  private recordSuccess(name: string): void {
    const circuit = this.circuits.get(name);
    if (!circuit) return;

    circuit.failures = 0;
    circuit.isOpen = false;
  }

  /**
   * Health check - ping all providers
   * @param message Optional message to send (default: "ping")
   */
  async healthCheck(message = 'ping'): Promise<HealthCheckResult[]> {
    const results: HealthCheckResult[] = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) {
        results.push({ name: config.name, healthy: false, error: 'Not configured' });
        continue;
      }

      const start = Date.now();
      try {
        await client.chat(message);
        results.push({
          name: config.name,
          healthy: true,
          latencyMs: Date.now() - start
        });
      } catch (error) {
        results.push({
          name: config.name,
          healthy: false,
          latencyMs: Date.now() - start,
          error: formatError(error)
        });
      }
    }

    return results;
  }

  /**
   * Send a chat message with automatic fallback
   * Falls back on ANY error for maximum availability
   */
  async chat(message: string): Promise<ChatResult> {
    const errors: Array<{ provider: string; error: unknown }> = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) continue;

      // Check circuit breaker
      if (!this.isCircuitClosed(config.name)) {
        log('warn', `${config.name} circuit open, skipping`);
        continue;
      }

      log('info', `Trying ${config.name}...`);

      try {
        const response = await client.chat(message);
        this.recordSuccess(config.name);
        log('success', `${config.name} succeeded`);
        return {
          content: response.content,
          provider: config.name
        };
      } catch (error) {
        this.recordFailure(config.name);
        errors.push({ provider: config.name, error });
        log('error', `${config.name} failed: ${formatError(error)}`);
        log('warn', `Falling back to next provider...`);
      }
    }

    // All providers failed
    const errorSummary = errors.map((e) => `  - ${e.provider}: ${formatError(e.error)}`).join('\n');
    throw new Error(`All providers failed:\n${errorSummary}`);
  }

  // [start:streaming]
  /**
   * Stream a chat message with automatic fallback
   *
   * Note: If streaming fails mid-stream, partial response is lost.
   * Fallback starts fresh from the beginning with next provider.
   */
  async *stream(message: string): AsyncGenerator<StreamChunk> {
    const errors: Array<{ provider: string; error: unknown }> = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) continue;

      // Check circuit breaker
      if (!this.isCircuitClosed(config.name)) {
        log('warn', `${config.name} circuit open, skipping`);
        continue;
      }

      log('info', `Trying ${config.name} (streaming)...`);

      try {
        let hasYielded = false;
        for await (const chunk of client.stream(message)) {
          if (!hasYielded) {
            log('success', `${config.name} started streaming`);
            hasYielded = true;
          }
          yield {
            content: chunk.content,
            provider: config.name,
            done: false
          };
        }
        this.recordSuccess(config.name);
        yield {
          content: '',
          provider: config.name,
          done: true
        };
        return; // Success, exit generator
      } catch (error) {
        this.recordFailure(config.name);
        errors.push({ provider: config.name, error });
        log('error', `${config.name} failed: ${formatError(error)}`);
        log('warn', `Falling back to next provider...`);
      }
    }

    // All providers failed
    const errorSummary = errors.map((e) => `  - ${e.provider}: ${formatError(e.error)}`).join('\n');
    throw new Error(`All providers failed:\n${errorSummary}`);
  }
  // [end:streaming]

  /**
   * Get list of configured providers
   */
  getProviders(): string[] {
    return Array.from(this.clients.keys());
  }

  /**
   * Get circuit breaker status for all providers
   */
  getCircuitStatus(): CircuitStatus[] {
    return this.providers
      .filter((p) => this.circuits.has(p.name))
      .map((p) => {
        const circuit = this.circuits.get(p.name)!;
        return {
          name: p.name,
          isOpen: circuit.isOpen,
          failures: circuit.failures
        };
      });
  }
}
// [end:fallback-client]

// [start:create-providers]
export function createDefaultProviders(): ProviderConfig[] {
  const providers: ProviderConfig[] = [];

  if (process.env.OPENAI_API_KEY) {
    providers.push({
      name: 'OpenAI',
      provider: 'openai',
      model: 'gpt-4o-mini',
      apiKey: process.env.OPENAI_API_KEY,
      retryConfig: {
        maxAttempts: 2,
        initialBackoffMs: 1000,
        maxBackoffMs: 5000,
        backoffMultiplier: 2
      }
    });
  }

  if (process.env.ANTHROPIC_API_KEY) {
    providers.push({
      name: 'Anthropic',
      provider: 'anthropic',
      model: 'claude-3-haiku-20240307',
      apiKey: process.env.ANTHROPIC_API_KEY,
      retryConfig: {
        maxAttempts: 2,
        initialBackoffMs: 1000,
        maxBackoffMs: 5000,
        backoffMultiplier: 2
      }
    });
  }

  // Always add Ollama as local fallback (may not be running)
  providers.push({
    name: 'Ollama',
    provider: 'ollama',
    model: 'llama3.2',
    baseUrl: 'http://localhost:11434',
    retryConfig: {
      maxAttempts: 1,
      initialBackoffMs: 500,
      maxBackoffMs: 2000,
      backoffMultiplier: 2
    }
  });

  return providers;
}
// [end:create-providers]

Create Default Providers

typescript

/**
 * Multi-Provider Fallback Library
 *
 * Exported functions for the Multi-Provider Fallback recipe.
 * Snippet markers allow VitePress to extract code for documentation.
 */

import { ChatClient, ChatError } from '../../../src';

// [start:colors]
export const colors = {
  reset: '\x1b[0m',
  dim: '\x1b[2m',
  bold: '\x1b[1m',
  red: '\x1b[31m',
  yellow: '\x1b[33m',
  green: '\x1b[32m',
  cyan: '\x1b[36m',
  magenta: '\x1b[35m'
};
// [end:colors]

// [start:types]
export interface CircuitBreakerConfig {
  failureThreshold: number; // Open circuit after N failures
  resetTimeoutMs: number; // Try again after N ms
}

export interface ProviderConfig {
  name: string;
  provider: 'openai' | 'anthropic' | 'ollama';
  model: string;
  apiKey?: string;
  baseUrl?: string;
  retryConfig: {
    maxAttempts: number;
    initialBackoffMs: number;
    maxBackoffMs: number;
    backoffMultiplier: number;
  };
  circuitBreaker?: CircuitBreakerConfig;
}

export interface CircuitState {
  failures: number;
  lastFailure: number;
  isOpen: boolean;
  config: CircuitBreakerConfig;
}

export interface HealthCheckResult {
  name: string;
  healthy: boolean;
  latencyMs?: number;
  error?: string;
}

export interface ChatResult {
  content: string;
  provider: string;
}

export interface StreamChunk {
  content: string;
  provider: string;
  done: boolean;
}

export interface CircuitStatus {
  name: string;
  isOpen: boolean;
  failures: number;
}
// [end:types]

// [start:default-config]
export const DEFAULT_CIRCUIT_BREAKER: CircuitBreakerConfig = {
  failureThreshold: 3,
  resetTimeoutMs: 30000
};
// [end:default-config]

// [start:logging]
export function formatError(error: unknown): string {
  if (error instanceof ChatError) {
    return `${error.name}: ${error.message}`;
  }
  if (error instanceof Error) {
    return error.message;
  }
  return String(error);
}

export function printBanner(): void {
  console.log(`
${colors.cyan}╭─────────────────────────────────────────────╮
│  ${colors.reset}${colors.bold}Multi-Provider Fallback${colors.reset}${colors.cyan}                    │
│  ${colors.dim}High Availability LLM Client${colors.reset}${colors.cyan}                │
╰─────────────────────────────────────────────╯${colors.reset}
`);
}

export function log(level: 'info' | 'warn' | 'error' | 'success', message: string): void {
  const prefix = {
    info: `${colors.cyan}ℹ${colors.reset}`,
    warn: `${colors.yellow}⚠${colors.reset}`,
    error: `${colors.red}✗${colors.reset}`,
    success: `${colors.green}✓${colors.reset}`
  };
  console.log(`${prefix[level]} ${message}`);
}
// [end:logging]

// [start:fallback-client]
/**
 * Multi-provider client with automatic fallback and circuit breaker
 */
export class FallbackClient {
  private providers: ProviderConfig[];
  private clients: Map<string, ChatClient> = new Map();
  private circuits: Map<string, CircuitState> = new Map();

  constructor(providers: ProviderConfig[]) {
    this.providers = providers;

    // Pre-create clients for each provider
    for (const config of providers) {
      try {
        const client = new ChatClient({
          provider: config.provider,
          model: config.model,
          apiKey: config.apiKey,
          baseUrl: config.baseUrl,
          retryConfig: config.retryConfig
        });
        this.clients.set(config.name, client);
        this.circuits.set(config.name, {
          failures: 0,
          lastFailure: 0,
          isOpen: false,
          config: config.circuitBreaker ?? DEFAULT_CIRCUIT_BREAKER
        });
        log('info', `Configured ${config.name} (${config.provider}/${config.model})`);
      } catch (error) {
        log('warn', `Failed to configure ${config.name}: ${formatError(error)}`);
      }
    }

    if (this.clients.size === 0) {
      throw new Error('No providers could be configured');
    }
  }

  /**
   * Check if circuit breaker allows requests to this provider
   */
  private isCircuitClosed(name: string): boolean {
    const circuit = this.circuits.get(name);
    if (!circuit) return false;

    if (!circuit.isOpen) return true;

    // Check if enough time has passed to try again
    const elapsed = Date.now() - circuit.lastFailure;
    if (elapsed >= circuit.config.resetTimeoutMs) {
      circuit.isOpen = false;
      circuit.failures = 0;
      log('info', `Circuit closed for ${name} (reset after ${circuit.config.resetTimeoutMs}ms)`);
      return true;
    }

    return false;
  }

  /**
   * Record a failure for circuit breaker
   */
  private recordFailure(name: string): void {
    const circuit = this.circuits.get(name);
    if (!circuit) return;

    circuit.failures++;
    circuit.lastFailure = Date.now();

    if (circuit.failures >= circuit.config.failureThreshold) {
      circuit.isOpen = true;
      log('warn', `Circuit opened for ${name} (${circuit.failures} failures)`);
    }
  }

  /**
   * Record a success - reset circuit breaker
   */
  private recordSuccess(name: string): void {
    const circuit = this.circuits.get(name);
    if (!circuit) return;

    circuit.failures = 0;
    circuit.isOpen = false;
  }

  /**
   * Health check - ping all providers
   * @param message Optional message to send (default: "ping")
   */
  async healthCheck(message = 'ping'): Promise<HealthCheckResult[]> {
    const results: HealthCheckResult[] = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) {
        results.push({ name: config.name, healthy: false, error: 'Not configured' });
        continue;
      }

      const start = Date.now();
      try {
        await client.chat(message);
        results.push({
          name: config.name,
          healthy: true,
          latencyMs: Date.now() - start
        });
      } catch (error) {
        results.push({
          name: config.name,
          healthy: false,
          latencyMs: Date.now() - start,
          error: formatError(error)
        });
      }
    }

    return results;
  }

  /**
   * Send a chat message with automatic fallback
   * Falls back on ANY error for maximum availability
   */
  async chat(message: string): Promise<ChatResult> {
    const errors: Array<{ provider: string; error: unknown }> = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) continue;

      // Check circuit breaker
      if (!this.isCircuitClosed(config.name)) {
        log('warn', `${config.name} circuit open, skipping`);
        continue;
      }

      log('info', `Trying ${config.name}...`);

      try {
        const response = await client.chat(message);
        this.recordSuccess(config.name);
        log('success', `${config.name} succeeded`);
        return {
          content: response.content,
          provider: config.name
        };
      } catch (error) {
        this.recordFailure(config.name);
        errors.push({ provider: config.name, error });
        log('error', `${config.name} failed: ${formatError(error)}`);
        log('warn', `Falling back to next provider...`);
      }
    }

    // All providers failed
    const errorSummary = errors.map((e) => `  - ${e.provider}: ${formatError(e.error)}`).join('\n');
    throw new Error(`All providers failed:\n${errorSummary}`);
  }

  // [start:streaming]
  /**
   * Stream a chat message with automatic fallback
   *
   * Note: If streaming fails mid-stream, partial response is lost.
   * Fallback starts fresh from the beginning with next provider.
   */
  async *stream(message: string): AsyncGenerator<StreamChunk> {
    const errors: Array<{ provider: string; error: unknown }> = [];

    for (const config of this.providers) {
      const client = this.clients.get(config.name);
      if (!client) continue;

      // Check circuit breaker
      if (!this.isCircuitClosed(config.name)) {
        log('warn', `${config.name} circuit open, skipping`);
        continue;
      }

      log('info', `Trying ${config.name} (streaming)...`);

      try {
        let hasYielded = false;
        for await (const chunk of client.stream(message)) {
          if (!hasYielded) {
            log('success', `${config.name} started streaming`);
            hasYielded = true;
          }
          yield {
            content: chunk.content,
            provider: config.name,
            done: false
          };
        }
        this.recordSuccess(config.name);
        yield {
          content: '',
          provider: config.name,
          done: true
        };
        return; // Success, exit generator
      } catch (error) {
        this.recordFailure(config.name);
        errors.push({ provider: config.name, error });
        log('error', `${config.name} failed: ${formatError(error)}`);
        log('warn', `Falling back to next provider...`);
      }
    }

    // All providers failed
    const errorSummary = errors.map((e) => `  - ${e.provider}: ${formatError(e.error)}`).join('\n');
    throw new Error(`All providers failed:\n${errorSummary}`);
  }
  // [end:streaming]

  /**
   * Get list of configured providers
   */
  getProviders(): string[] {
    return Array.from(this.clients.keys());
  }

  /**
   * Get circuit breaker status for all providers
   */
  getCircuitStatus(): CircuitStatus[] {
    return this.providers
      .filter((p) => this.circuits.has(p.name))
      .map((p) => {
        const circuit = this.circuits.get(p.name)!;
        return {
          name: p.name,
          isOpen: circuit.isOpen,
          failures: circuit.failures
        };
      });
  }
}
// [end:fallback-client]

// [start:create-providers]
export function createDefaultProviders(): ProviderConfig[] {
  const providers: ProviderConfig[] = [];

  if (process.env.OPENAI_API_KEY) {
    providers.push({
      name: 'OpenAI',
      provider: 'openai',
      model: 'gpt-4o-mini',
      apiKey: process.env.OPENAI_API_KEY,
      retryConfig: {
        maxAttempts: 2,
        initialBackoffMs: 1000,
        maxBackoffMs: 5000,
        backoffMultiplier: 2
      }
    });
  }

  if (process.env.ANTHROPIC_API_KEY) {
    providers.push({
      name: 'Anthropic',
      provider: 'anthropic',
      model: 'claude-3-haiku-20240307',
      apiKey: process.env.ANTHROPIC_API_KEY,
      retryConfig: {
        maxAttempts: 2,
        initialBackoffMs: 1000,
        maxBackoffMs: 5000,
        backoffMultiplier: 2
      }
    });
  }

  // Always add Ollama as local fallback (may not be running)
  providers.push({
    name: 'Ollama',
    provider: 'ollama',
    model: 'llama3.2',
    baseUrl: 'http://localhost:11434',
    retryConfig: {
      maxAttempts: 1,
      initialBackoffMs: 500,
      maxBackoffMs: 2000,
      backoffMultiplier: 2
    }
  });

  return providers;
}
// [end:create-providers]

Configuration Options

Per-Provider Retry

Each provider can have its own retry settings:

typescript

retryConfig: {
  maxAttempts: 2,        // Retry twice before fallback
  initialBackoffMs: 1000, // Start with 1s delay
  maxBackoffMs: 5000      // Max 5s between retries
}

Provider Priority

Providers are tried in array order. Put your preferred provider first:

typescript

const providers = [
  { name: 'Primary', provider: 'openai', ... },
  { name: 'Secondary', provider: 'anthropic', ... },
  { name: 'Fallback', provider: 'ollama', ... }
];

Local Fallback with Ollama

Ollama provides a local fallback that works without internet:

bash

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull llama3.2

# Ollama runs on http://localhost:11434

Circuit Breaker

The circuit breaker prevents hammering unhealthy providers. Each provider can have its own settings:

typescript

// Per-provider configuration
const providers = [
  {
    name: 'OpenAI',
    provider: 'openai',
    model: 'gpt-4o-mini',
    circuitBreaker: {
      failureThreshold: 3,    // Open after 3 failures
      resetTimeoutMs: 30000   // Try again after 30 seconds
    }
  },
  {
    name: 'Ollama',
    provider: 'ollama',
    model: 'llama3.2',
    circuitBreaker: {
      failureThreshold: 1,    // Open immediately on failure
      resetTimeoutMs: 5000    // Try again after 5 seconds
    }
  }
];

// Check circuit status
const status = client.getCircuitStatus();
// [{ name: 'OpenAI', isOpen: false, failures: 0 }, ...]

How it works:

Each provider has an independent circuit with its own config
After N failures, circuit opens (skips provider)
After timeout, circuit closes (tries again)
Success resets failure count

Health Check

Verify provider availability before critical operations:

typescript

// Default ping
const health = await client.healthCheck();

// Custom health check message
const health = await client.healthCheck('Hello, are you there?');

for (const result of health) {
  if (result.healthy) {
    console.log(`${result.name}: ✓ ${result.latencyMs}ms`);
  } else {
    console.log(`${result.name}: ✗ ${result.error}`);
  }
}

Use health checks for:

Pre-flight checks before batch operations
Monitoring dashboards
Choosing optimal provider based on latency

Environment Variables

Variable	Required	Description
`OPENAI_API_KEY`	One of these	OpenAI API key
`ANTHROPIC_API_KEY`	required	Anthropic API key

At least one cloud API key is required. Ollama works locally without an API key.

Use Cases

High Availability API

typescript

// Production service with guaranteed uptime
const client = new FallbackClient([
  { provider: 'openai', ... },    // Fast, primary
  { provider: 'anthropic', ... }, // Reliable backup
  { provider: 'ollama', ... }     // Local emergency fallback
]);

Cost Optimization

typescript

// Try cheaper models first
const client = new FallbackClient([
  { provider: 'ollama', model: 'llama3.2', ... },  // Free, local
  { provider: 'openai', model: 'gpt-4o-mini', ... }, // Cheap cloud
  { provider: 'anthropic', model: 'claude-3-opus', ... } // Premium fallback
]);

Development/Production Split

typescript

const providers = process.env.NODE_ENV === 'production'
  ? [openaiConfig, anthropicConfig, ollamaConfig]
  : [ollamaConfig]; // Local only in dev

Full Source

View on GitHub

Multi-Provider Fallback ​

Features ​

Quick Start ​

How It Works ​

Provider Chain ​

Example Output ​

Fallback Triggers ​

Streaming Behavior ​

Code Walkthrough ​

Types and Configuration ​

Default Circuit Breaker ​

Fallback Client ​

Streaming with Fallback ​

Create Default Providers ​

Configuration Options ​

Per-Provider Retry ​

Provider Priority ​

Local Fallback with Ollama ​

Circuit Breaker ​

Health Check ​

Environment Variables ​

Use Cases ​

High Availability API ​

Cost Optimization ​

Development/Production Split ​

Full Source ​

Related ​

Multi-Provider Fallback

Features

Quick Start

How It Works

Provider Chain

Example Output

Fallback Triggers

Streaming Behavior

Code Walkthrough

Types and Configuration

Default Circuit Breaker

Fallback Client

Streaming with Fallback

Create Default Providers

Configuration Options

Per-Provider Retry

Provider Priority

Local Fallback with Ollama

Circuit Breaker

Health Check

Environment Variables

Use Cases

High Availability API

Cost Optimization

Development/Production Split

Full Source

Related