Cloud LLM Routing in React Native: OpenAI + Anthropic + Gemini Fallback Patterns

If you ship a React Native app that talks to a cloud LLM and you only have one provider wired in, you have a single point of failure for the most expensive feature in your product. Outages happen. Rate limits hit during launch spikes. Pricing changes overnight. The fix is a routing layer that picks the right provider per request and falls back when the primary fails. Primary Gemini for cost, fallback Claude for reasoning, emergency OpenAI for breadth. All three wired for Hermes-safe streaming. Real config below.

I run this exact router in the AI Pro tier of AI Mobile Launcher. The provider-selection slice is in src/store/llm-preferences-slice.ts. The streaming layer borrows from Wire RN's Hermes XHR polyfill. Everything below has shipped.

Why routing, not single-provider?

The case for one provider is simplicity. You wire OpenAI, you ship, done. It works until it does not. Three things break the assumption:

• Outages. OpenAI has had multi-hour API outages. Apps with only OpenAI wired up went down for the same window. Apps with fallback to Claude or Gemini kept working.
• Rate limit spikes during your launch. If you Product Hunt yourself to the front page and 200 users hit the chat feature in the same hour, your API key gets throttled. Fallback to a secondary provider lets the launch survive.
• Cost shaping. Gemini Flash is ~10x cheaper than Claude Sonnet per token for many tasks. If you do not route by task type, you are overpaying on most requests.

The fix is small. The wins compound across every chat session your users run.

What you need before starting

• An RN app with a working network layer (RTK Query, Tanstack Query, or fetch-based).
• API keys for at least two of: OpenAI, Anthropic, Google Gemini.
• A backend or proxy in front of the keys (do NOT ship API keys to the client; we will get to this).
• The Hermes streaming polyfill (XHR-based SSE shim).

Setup time: ~2 hours for the basic router. Plan a full day if you also want the proxy server.

Step 1: Never ship API keys to the client

Before any routing code. If your RN app calls OpenAI/Anthropic/Gemini directly with a key in the bundle, that key is compromised. Tools like react-native-decompiler extract it in minutes. The fix is a thin proxy.

// server/proxy.ts (deploy to Cloudflare Workers, Vercel Edge, Fly.io)
import { OpenAI } from "openai";
import Anthropic from "@anthropic-ai/sdk";
import { GoogleGenerativeAI } from "@google/generative-ai";

const PROVIDERS = {
  openai: new OpenAI({ apiKey: process.env.OPENAI_KEY! }),
  anthropic: new Anthropic({ apiKey: process.env.ANTHROPIC_KEY! }),
  gemini: new GoogleGenerativeAI(process.env.GEMINI_KEY!),
};

export async function handleChat(req: Request): Promise<Response> {
  const { provider, messages, model, stream } = await req.json();
  // ...auth check against your user session...
  // ...rate limit per user...
  // forward to the provider, stream back
}

The proxy is non-negotiable. Every code sample below assumes your RN client calls your proxy, not the LLM provider directly. The proxy holds the keys and adds your per-user rate limits.

Step 2: Define the routing slice

// src/store/llm-preferences-slice.ts
import { createSlice, PayloadAction } from "@reduxjs/toolkit";

export type Provider = "gemini" | "anthropic" | "openai";
export type TaskType = "chat" | "reasoning" | "extraction" | "summarization";

type ProviderConfig = {
  primary: Provider;
  fallback: Provider;
  emergency: Provider;
};

type State = {
  routing: Record<TaskType, ProviderConfig>;
  failureCounts: Record<Provider, number>;
};

const initialState: State = {
  routing: {
    chat: { primary: "gemini", fallback: "anthropic", emergency: "openai" },
    reasoning: { primary: "anthropic", fallback: "openai", emergency: "gemini" },
    extraction: { primary: "gemini", fallback: "openai", emergency: "anthropic" },
    summarization: { primary: "gemini", fallback: "anthropic", emergency: "openai" },
  },
  failureCounts: { gemini: 0, anthropic: 0, openai: 0 },
};

The routing config is per task type. Chat tasks default to Gemini for cost. Reasoning tasks (multi-step plans, code review) default to Claude. Extraction (pulling JSON from text) defaults to Gemini because Flash is fast and cheap and surprisingly good at this.

Step 3: The fallback executor

// src/features/cloud-llm/api/execute-with-fallback.ts
export async function executeWithFallback(
  task: TaskType,
  messages: ChatMessage[],
  onToken: (token: string) => void
): Promise<string> {
  const { primary, fallback, emergency } = store.getState()
    .llmPreferences.routing[task];
  const chain: Provider[] = [primary, fallback, emergency];

  let lastError: unknown;
  for (const provider of chain) {
    try {
      const result = await callProvider(provider, messages, onToken);
      store.dispatch(llmPreferencesSlice.actions.resetFailures(provider));
      return result;
    } catch (err) {
      console.warn(`[llm] ${provider} failed:`, err);
      store.dispatch(llmPreferencesSlice.actions.recordFailure(provider));
      lastError = err;
    }
  }

  throw lastError ?? new Error("All providers failed");
}

Three providers in the chain. Reset failure count on success. Log failures. If all three fail, surface the error to the UI with a “we are working on it, try again in a minute” message.

Step 4: Hermes-safe streaming for each provider

This is where most RN devs hit the wall. Hermes does not implement ReadableStream. The OpenAI SDK and Anthropic SDK both assume it works. The Gemini SDK has its own streaming format. Three providers, three streaming patterns, one Hermes engine that supports none of them out of the box.

The fix: an XHR-based SSE shim that fires a callback per chunk.

// src/features/cloud-llm/api/sse-fetch.ts
export async function sseFetch(
  url: string,
  init: RequestInit,
  onEvent: (data: string) => void
): Promise<void> {
  return new Promise((resolve, reject) => {
    const xhr = new XMLHttpRequest();
    xhr.open(init.method ?? "POST", url);

    Object.entries(init.headers ?? {}).forEach(([k, v]) => {
      xhr.setRequestHeader(k, v as string);
    });

    let cursor = 0;
    xhr.onprogress = () => {
      const chunk = xhr.responseText.slice(cursor);
      cursor = xhr.responseText.length;
      for (const line of chunk.split("\n")) {
        if (line.startsWith("data: ")) {
          const data = line.slice(6).trim();
          if (data && data !== "[DONE]") onEvent(data);
        }
      }
    };

    xhr.onload = () => resolve();
    xhr.onerror = () => reject(new Error("SSE network error"));
    xhr.send(typeof init.body === "string" ? init.body : JSON.stringify(init.body));
  });
}

XHR's onprogress fires as bytes arrive. We slice the new portion, split on newlines, parse the SSE data: prefix, forward each event. Crude. It works.

Step 5: Per-provider call functions

function extractToken(provider: Provider, payload: any): string | null {
  if (provider === "openai")
    return payload.choices?.[0]?.delta?.content ?? null;
  if (provider === "anthropic")
    return payload.delta?.text ?? null;
  if (provider === "gemini")
    return payload.candidates?.[0]?.content?.parts?.[0]?.text ?? null;
  return null;
}

Each provider has a different streaming JSON shape. OpenAI uses choices[].delta.content. Anthropic uses delta.text. Gemini uses candidates[].content.parts[].text. The extractToken helper normalizes them. Add a new provider, add a case to extractToken. The rest of the routing layer does not change.

Step 6: Cost math at scale

The reason routing matters in dollars, not just reliability.

Hypothetical: 100,000 user chat sessions per month, 3,000 tokens per session (prompt + completion).

• At Gemini Flash pricing (~$0.10 / 1M input + $0.40 / 1M output) with a 60/40 input/output split: ~$60/month total.
• At Claude Sonnet pricing (~$3 / 1M input + $15 / 1M output): ~$1,170/month for the same volume.
• At GPT-4o pricing (~$2.50 / 1M input + $10 / 1M output): ~$870/month.

The ratio is roughly 15-20x between Gemini Flash and Claude Sonnet for chat workloads. If you do not route, your cheapest provider becomes the most expensive when traffic grows. Routing by task type, with Gemini as primary for chat and Claude only for reasoning-heavy tasks, can cut total LLM spend by 60-80% without quality degradation on the chat path.

What still does not work

The honest section.

• The fallback adds latency on failures. Each failed provider costs 2-5 seconds before the next one is tried. Add a per-provider timeout (5-8 seconds) and treat slow as failure.
• Quality is not equivalent across providers. Claude is better at reasoning. Gemini is faster and cheaper. GPT is more “default-voiced.” Test prompts against all three before launch.
• Streaming format drift. Anthropic shipped a new event format in late 2025. OpenAI changed delta semantics in 2024. Gemini has had two SSE formats in 18 months. Your extractToken function is going to need maintenance.
• Cost math assumes you control the proxy. If your proxy is on a free tier, you will hit limits before your LLM costs do.
• Per-user rate limiting is not optional. If you do not enforce it in the proxy, one user can run a script and burn your monthly LLM budget in an hour. Speaking from a previous mistake.

Where to go next

The full routing layer, the Hermes streaming polyfill, the per-provider call functions, and the proxy template ship in AI Mobile Launcher AI Pro. The Hermes streaming polyfill is also documented standalone at Wire RN's docs if you want to build the routing yourself.

One issue per week on AI-native mobile development at codemeetai.substack.com.

FAQ

Should I route by task type or just always fall back on failure?

Both. Task-type routing controls cost and quality fit (cheap provider for chat, smart provider for reasoning). Failure fallback controls reliability. They are independent layers. Run both.

Can I use the OpenAI SDK directly from React Native?

You can install it, but two problems. One, it assumes ReadableStream for streaming and Hermes does not implement that, so streaming will not work without the XHR shim above. Two, if you ship your API key in the bundle, it will get extracted. Always proxy through a backend.

How do I handle gemini openai react native streaming differences?

Each provider has a different SSE event shape. OpenAI uses choices[].delta.content. Anthropic uses delta.text. Gemini uses candidates[].content.parts[].text. Normalize them in a single extractToken helper. The rest of your streaming code stays provider-agnostic.

What is react native streaming LLM and why is Hermes a blocker?

React Native runs JavaScript inside the Hermes engine by default. Hermes does not implement ReadableStream, which is what cloud LLM SDKs expect for streaming responses. You either use the XHR-based SSE shim, switch to JSC (slower, larger binary), or run completions non-streaming (worse UX).

How much can I actually save by routing to Gemini for chat?

At current pricing, Gemini Flash is ~15-20x cheaper than Claude Sonnet for chat-style workloads (3K tokens, mostly output). For 100K sessions/month, that is a difference of about $1,100/month in raw token cost. Real savings depend on your task mix, prompt size, and which provider you start from.

Do I need three providers or is two enough?

Two is usually enough. Primary + fallback covers outages and rate limits. Three providers adds redundancy at the cost of more code paths to test. I run three in AI Pro because I serve a paid product where reliability matters. For a side project, two is reasonable.

Related reading

• Gemini vs OpenAI for mobile apps in 2026
• Phi-3 Mini on-device LLM in React Native
• On-device LLM in React Native with llama.rn