Competitive Racing

Run agents in parallel, return the fastest response and cancel the others.

Execute multiple AI approaches or strategies simultaneously and return the result from whichever completes first successfully. This pattern is ideal when you have multiple ways to solve the same problem and want to minimize latency by racing them against each other.

Useful for:

• Querying multiple AI models (e.g., GPT-4, Claude, Gemini) and returning the fastest response
• Running different agents, prompts or strategies in parallel and using the first successful outcome

How does Restate help?

The benefits of using Restate for competitive racing patterns are:

• Durable coordination: Restate turns Promises/Futures into durable, distributed constructs that persist across failures and process restarts. Race multiple approaches and return the first successful result.
• Cancel slow tasks: Failed or slower approaches can be cancelled, preventing resource waste
• Serverless scaling: Deploy racing strategies on serverless infrastructure for automatic scaling while the main process remains suspended
• Works with any LLM SDK (Vercel AI, LangChain, LiteLLM, etc.) and any programming language supported by Restate (TypeScript, Python, Go, etc.).

Example

When you need a quick response and have access to multiple AI models, race them against each other to get the fastest result:

TypeScript

async function run(
  ctx: Context,
  { message }: { message: string },
): Promise<string> {
  // Start both service calls concurrently
  const slowCall = ctx.serviceClient(racingAgent).thinkLonger({ message });
  const slowResponse = slowCall.map((res) => ({ tag: "slow", res }));

  const fastCall = ctx.serviceClient(racingAgent).respondQuickly({ message });
  const fastResponse = fastCall.map((res) => ({ tag: "fast", res }));

  const pending = [slowResponse, fastResponse];

  // Wait for the first one to complete
  const { tag, res } = await RestatePromise.any(pending);

  if (tag === "fast") {
    console.log("Quick response won the race!");
    const slowInvocationId = await slowCall.invocationId;
    ctx.cancel(slowInvocationId);
  } else {
    console.log("Deep analysis won the race!");
    const quickInvocationId = await fastCall.invocationId;
    ctx.cancel(quickInvocationId);
  }

  return res ?? "LLM gave no response";
}

View on GitHub: TS / Python

In the Restate UI, you can see how multiple approaches are started simultaneously, with the first successful result being returned while other tasks are automatically cancelled:

Run the example

1. Requirements

   • AI SDK of your choice (e.g., OpenAI, LangChain, Pydantic AI, LiteLLM, etc.) to make LLM calls.
   • API key for your model provider.
2. Download the example

   TypeScript

   git clone https://github.com/restatedev/ai-examples.git &&
   cd typescript-patterns &&
   npm install
   
3. Start the Restate Server

   restate-server
   
4. Start the Service

   Export the API key of your model provider as an environment variable and then start the agent. For example, for OpenAI:

   TypeScript

   export OPENAI_API_KEY=your_openai_api_key
   npm run dev
   
5. Register the services

   UI

   CLI

   restate deployments register localhost:9080
   
6. Send a request

   In the UI (http://localhost:9070), click on the run handler of the RacingAgent service to open the playground and send a prompt to race multiple models:

   
7. Check the Restate UI

   In the UI, you can see how multiple models are queried simultaneously, with the fastest response winning the race: