How to Instrument LLM Calls with OpenTelemetry gen_ai.* Spans

gen_ai.* spans are a set of OpenTelemetry semantic conventions for generative AI calls, defined under the OpenTelemetry Semantic Conventions spec (v1.27.0, stable as of October 2024). They standardize the attribute names used to describe an LLM call's inputs, outputs, and metadata in distributed traces.

A conformant span for a chat completion call includes:

The purpose of standardizing these attributes is the same as standardizing http.method or db.system: any OTel-compatible backend can read and aggregate LLM telemetry without knowing the specific SDK or library that generated it.

The Vercel AI SDK (v3.3+) has built-in OpenTelemetry support. When an OpenTelemetry provider is configured and experimental_telemetry: { isEnabled: true } is set (or when the global telemetry is active), the SDK emits gen_ai.* spans automatically for every generateText(), streamText(), generateObject(), and streamObject() call.

You get:

• gen_ai.system — derived from the model provider (openai, anthropic, google, etc.)
• gen_ai.request.model — the model string you passed in
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens — from the response usage field
• gen_ai.response.model — the actual model version returned by the API (may differ from requested model)
• Span duration — the wall-clock latency of the call including streaming

What you don't get by default: prompt content, completion content, tool call payloads. These require experimental_telemetry: { recordInputs: true, recordOutputs: true }. Model Meter does not enable these flags — cost attribution only needs the metadata, not the content.

Enabling gen_ai telemetry requires an OTel SDK setup before any AI calls are made.

1. Install dependencies

npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-http

2. Create an instrumentation file (run before your app)

// instrumentation.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';

const sdk = new NodeSDK({
  resource: new Resource({ 'service.name': 'my-app' }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
    headers: { Authorization: `Bearer ${process.env.MODEL_METER_API_KEY}` },
  }),
});

sdk.start();

3. Initialize Model Meter (after OTel SDK starts)

import { init } from '@model-meter/sdk';

init({
  apiKey: process.env.MODEL_METER_API_KEY,
  endpoint: process.env.MODEL_METER_ENDPOINT,
  projectId: 'my-app',
  environment: process.env.NODE_ENV ?? 'development',
});

4. No changes needed to existing AI calls

import { generateText } from 'ai'; // unchanged
import { openai } from '@ai-sdk/openai'; // unchanged

const { text } = await generateText({
  model: openai('gpt-4o'),
  prompt: 'Summarize this ticket',
  // gen_ai.* spans emitted automatically
});

Once spans are flowing to your backend, here's what to look at:

gen_ai.usage.input_tokens — The actual token count sent to the model, including your system prompt, conversation history, and any retrieved context. This is what you're billed for on the input side. It's often larger than the literal prompt text suggests once encoding overhead is counted.

gen_ai.usage.output_tokens — Tokens generated by the model. Output tokens cost 3–4× more than input per token on most models. Longer responses or responses with structured JSON (which tends to produce verbose output) drive this up.

Span duration — Wall-clock latency including network and any streaming time. For streaming calls, this is the time from request sent to stream closed — it grows with output length.

gen_ai.response.model — The specific model version that served the request. This can differ from gen_ai.request.model if the provider routes to a different snapshot. Useful for debugging pricing discrepancies between estimated and provider-reported cost.

The OTel exporter configured in step 2 sends traces to any OTLP-compatible endpoint — Jaeger, Grafana Tempo, Honeycomb, Datadog, or Model Meter's ingestion API.

Model Meter's ingestion endpoint accepts standard OTLP traces over HTTP. Set OTEL_EXPORTER_OTLP_ENDPOINT to your Model Meter endpoint URL and pass your API key as a Bearer token in the Authorization header.

The cost engine reads gen_ai.* attributes from each span, looks up the effective token price for the reported model at the time of the request, and writes a cost record. Because pricing is effective-dated, historical cost records are never rewritten when OpenAI changes their rates — the cost at the time of the call is what's stored.

Lambda / serverless environments. The OTel SDK batches spans in memory and flushes on an interval. Lambda functions that exit before the flush interval fires will drop spans. Call sdk.shutdown() (which forces a flush) in your Lambda handler's cleanup:

context.callbackWaitsForEmptyEventLoop = false;
await sdk.shutdown();

Vercel Edge Functions. The Node.js OTel SDK doesn't run in the Edge runtime. Use @vercel/otel or a fetch-based OTLP exporter that works in the Web API environment instead.

Missing spans on the first cold start. The SDK registers instrumentations before your app code runs — if instrumentation.ts is imported after your AI provider is initialized, some spans may be missed. Use Node's --require flag or --import (ESM) to guarantee load order:

node --import ./instrumentation.js server.js

Non-Node runtimes (Deno, Bun). The @opentelemetry/sdk-node package targets Node.js. Bun has partial Node compatibility; Deno requires the Web SDK. In both cases, the gen_ai.* semantic conventions are identical — only the OTel bootstrap code differs.