Streaming

Instead of waiting for the full response, you can stream tokens incrementally as the model generates them. This gives your users a faster, more interactive experience.

The `stream()` method

client.stream() is an async generator that yields StreamChunk objects as the model streams its response via Server-Sent Events (SSE):

interface StreamChunk {
  /** Incremental text delta — empty string on the final (usage) chunk */
  delta: string
  /** Resolved model identifier */
  model: string | undefined
  /** Non-null on the final chunk */
  finishReason: string | null
  /** Token usage — only present on the final chunk */
  usage: {
    inputTokens: number
    outputTokens: number
    totalTokens: number
  } | null
}

Basic streaming example

Use a for await...of loop to iterate over chunks and accumulate the response text:

import { GatelitClient } from "@gatelit/sdk"

const client = new GatelitClient({
  gatewayUrl: "https://gateway.yourapp.com",
  getToken: async () => {
    const { token } = await fetch("/api/ai-token").then(r => r.json())
    return { token, scheme: "GatelitSigned" }
  },
})

let fullText = ""

for await (const chunk of client.stream({
  model: "anthropic/claude-3-5-sonnet-20241022",
  messages: [{ role: "user", content: "Write a haiku about distributed systems." }],
})) {
  // Append each delta to build the full response
  fullText += chunk.delta

  // Write to stdout as tokens arrive
  process.stdout.write(chunk.delta)

  // The final chunk has usage info
  if (chunk.usage) {
    console.log("\n\nTokens used:", chunk.usage.totalTokens)
  }
}

The last chunk in the stream has finishReason set to a non-null value (e.g. "stop" or "length") and usage populated with input, output, and total token counts. All preceding chunks have finishReason: null and usage: null.

Cancelling a stream with AbortSignal

Pass an AbortSignal via options.signal to cancel an in-progress stream. This is useful when the user navigates away or explicitly stops generation:

const controller = new AbortController()

// Cancel after 5 seconds
setTimeout(() => controller.abort(), 5000)

try {
  for await (const chunk of client.stream({
    model: "openai/gpt-4o-mini",
    messages: [{ role: "user", content: "Tell me a very long story." }],
    signal: controller.signal,
  })) {
    process.stdout.write(chunk.delta)
  }
} catch (err) {
  if (err instanceof Error && err.name === "AbortError") {
    console.log("Stream cancelled.")
  } else {
    throw err
  }
}

Streaming in a React component

Here is a pattern for streaming into component state:

import { useState } from "react"
import { GatelitClient } from "@gatelit/sdk"

const client = new GatelitClient({ /* config */ })

export function StreamingChat() {
  const [text, setText] = useState("")
  const [streaming, setStreaming] = useState(false)

  async function handleSubmit(userMessage: string) {
    setText("")
    setStreaming(true)

    const controller = new AbortController()

    for await (const chunk of client.stream({
      model: "openai/gpt-4o-mini",
      messages: [{ role: "user", content: userMessage }],
      signal: controller.signal,
    })) {
      setText(prev => prev + chunk.delta)
    }

    setStreaming(false)
  }

  return (
    <div>
      <pre>{text}{streaming && "▌"}</pre>
    </div>
  )
}

Using the Vue SDK composable

If you are building a Vue application, the useGatelit composable from @gatelit/sdk-vue handles streaming, accumulation, and cancellation automatically — no manual for await loop needed. See the useGatelit composable reference for full details.

Get Started

Authentication

Guides

Dashboard

The `stream()` method

Basic streaming example

Cancelling a stream with AbortSignal

Streaming in a React component

Using the Vue SDK composable

​The stream() method

​Basic streaming example

​Cancelling a stream with AbortSignal

​Streaming in a React component

​Using the Vue SDK composable

The `stream()` method

Basic streaming example

Cancelling a stream with AbortSignal

Streaming in a React component

Using the Vue SDK composable