Video input (understanding)

Send a video to a model and ask questions about it — summarize a recording, extract action items from a meeting, describe what happens in a clip. Bedrock-style video content blocks are supported in user messages, mirroring the bytes | s3Location pattern of Bedrock Converse’s video source. This is the input-side counterpart of Video generation, which creates video from a text prompt — don’t confuse the two.

Model support

Video input is available on the Gemini family only. Sending a video block to any other model returns an HTTP 400 before the request reaches the provider (see Limits & errors).

Model	Video input
`gemini-2.5-flash`	yes
`gemini-2.5-pro`	yes
`gemini-3-flash-preview`	yes
Every other model	no — HTTP 400

Inline video (≤ 20 MB)

For small files, hand the SDK raw bytes — it base64-encodes them into the request for you. Base64 inflates bytes by ~33% against the gateway’s 30 MB request cap, so inline video is capped at 20 MB raw per request (larger inline payloads are rejected with HTTP 413 — use the upload flow below instead). All five SDKs support video input as of SDK v0.3.0.

with open("demo.mp4", "rb") as f:
    video_bytes = f.read()

response = client.converse(
    modelId="gemini-2.5-flash",
    messages=[{
        "role": "user",
        "content": [
            {"text": "Summarize what happens in this video."},
            {"video": {"format": "mp4", "source": {"bytes": video_bytes}}},
        ],
    }],
)
print(response["output"]["message"]["content"][0]["text"])

import { readFile } from "node:fs/promises";

const videoBytes = new Uint8Array(await readFile("demo.mp4"));

const response = await client.send(
  new ConverseCommand({
    modelId: "gemini-2.5-flash",
    messages: [
      {
        role: "user",
        content: [
          { text: "Summarize what happens in this video." },
          { video: { format: "mp4", source: { bytes: videoBytes } } },
        ],
      },
    ],
  })
);

videoBytes, err := os.ReadFile("demo.mp4")
if err != nil {
    log.Fatal(err)
}

resp, err := client.Converse(ctx, &audacityruntime.ConverseInput{
    ModelId: audacity.String("gemini-2.5-flash"),
    Messages: []types.Message{{
        Role: types.ConversationRoleUser,
        Content: []types.ContentBlock{
            &types.ContentBlockMemberText{Value: "What happens in this video?"},
            &types.ContentBlockMemberVideo{Value: types.VideoBlock{
                Format: types.VideoFormatMp4,
                Source: &types.VideoSourceMemberBytes{Value: videoBytes},
            }},
        },
    }},
})

byte[] videoBytes = java.nio.file.Files.readAllBytes(java.nio.file.Path.of("demo.mp4"));

ConverseResponse resp = client.converse(request -> request
    .modelId("gemini-2.5-flash")
    .messages(Message.builder()
        .role(ConversationRole.USER)
        .content(
            ContentBlock.fromText("What happens in this video?"),
            ContentBlock.fromVideo(VideoBlock.builder()
                .format(VideoFormat.MP4)
                .source(VideoSource.fromBytes(videoBytes))
                .build()))
        .build()));

use audacity_sdk::{ContentBlock, VideoBlock, VideoFormat, VideoSource};

let video_bytes = std::fs::read("demo.mp4")?;

let response = client.converse()
    .model_id("gemini-2.5-flash")
    .messages(
        Message::builder()
            .role(ConversationRole::User)
            .content(ContentBlock::Text("What happens in this clip?".into()))
            .content(ContentBlock::Video(VideoBlock {
                format: VideoFormat::Mp4,
                source: VideoSource::Bytes(video_bytes),
            }))
            .build()?
    )
    .send()
    .await?;

Large videos: upload, then reference by URI

For files over 20 MB (up to 1 GB), upload once and reference the returned audacity://files/… URI in as many requests as you like. The flow has three steps:

POST /v1/files with content_type and size_bytes — returns a file_id, a signed upload_url (valid ~15 minutes), the uri to reference in chat requests, and expires_at.
Upload the bytes to upload_url over a resumable session (Google Cloud Storage protocol: one POST opens the session, then PUT the bytes).
Reference the video with {"video": {"format": …, "source": {"uri": …}}} in a Converse call.

The file-create step from curl:

# 1) Create an upload slot
curl -s https://api.aireserve.com/v1/files \
  -H "Authorization: Bearer $AUDACITY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"content_type": "video/mp4", "size_bytes": 52428800}'

{
  "file_id": "3f8a1e2c-9d4b-4c6a-b7e1-2a5c8d9f0e13",
  "upload_url": "https://storage.googleapis.com/…&X-Goog-Signature=…",
  "uri": "audacity://files/3f8a1e2c-9d4b-4c6a-b7e1-2a5c8d9f0e13",
  "expires_at": "2026-07-09T05:20:00.000Z"
}

# 2) Open the resumable session (the session URI comes back in the Location header)
SESSION_URI=$(curl -si -X POST "$UPLOAD_URL" \
  -H "x-goog-resumable: start" \
  -H "Content-Type: video/mp4" \
  | awk 'tolower($1)=="location:" {print $2}' | tr -d "\r")

# 3) Upload the bytes (a single PUT works; the SDK helpers chunk + auto-resume)
curl -X PUT "$SESSION_URI" --data-binary @keynote.mp4

GET /v1/files/{file_id} reports upload status — "pending" before the bytes land, then "uploaded" with size_bytes and content_type. The SDK helpers (client.files.upload in Python/TypeScript, UploadFile/uploadFile in Go/Java, upload_file() in Rust) run this whole flow for you and stream the file in 8 MB chunks, automatically resuming from the last confirmed byte after a network drop. The full loop in Python:

from audacity import Audacity

client = Audacity()   # AUDACITY_API_KEY from the environment

# Upload once — accepts raw bytes or a file path; resumable under the hood
upload = client.files.upload("keynote.mp4", content_type="video/mp4")
# {"file_id": …, "upload_url": …, "uri": "audacity://files/…", "expires_at": …}

# Reference the uploaded video by URI — reusable across requests for ~24 h
response = client.converse(
    modelId="gemini-2.5-pro",
    mediaResolution="low",   # optional cost knob — see below
    messages=[{
        "role": "user",
        "content": [
            {"text": "Summarize this keynote and list the action items."},
            {"video": {"format": "mp4", "source": {"uri": upload["uri"]}}},
        ],
    }],
)
print(response["output"]["message"]["content"][0]["text"])

Uploaded files are transient inference inputs: they expire after ~24 hours and are scoped to your workspace’s API keys — a URI leaked to another workspace resolves against that workspace’s namespace and simply does not exist. Re-referencing the same URI across conversation turns is free; the gateway caches the provider-side staging, so a large video is not re-transferred on every request.

Calling without an SDK? On the raw OpenAI-protocol endpoint, video rides the file content part — inline as a data URL in file_data, or by reference in file_id — with the MIME type in format:

{"type": "file", "file": {"file_data": "data:video/mp4;base64,…", "format": "video/mp4"}}
{"type": "file", "file": {"file_id": "audacity://files/3f8a1e2c-…", "format": "video/mp4"}}

A per-part "detail" field and a request-level "media_resolution" field control media resolution (next section).

Media resolution (video token cost)

Video is tokenized frame by frame on Gemini, so long clips get expensive fast. The request-level mediaResolution option (wire name: media_resolution) controls how densely video is sampled — "low" processes video at roughly 4× fewer tokens. Non-Gemini models ignore the field; when unset, the model’s default applies.

Value	Guidance
`low`	~4× fewer video tokens. The right default for summarization, transcription-style tasks, and long footage
`medium`	Balanced sampling for general question-answering about a clip
`high`	Finer sampling — reading small on-screen text, UI walkthroughs, fine visual detail
`ultra_high`	Maximum fidelity at maximum token cost — dense charts or frame-critical analysis

Raw-protocol callers can also set detail per content part; an explicit per-part value wins over the request-level field. On Gemini 2.5 models the highest resolution across all parts applies to the whole request; true per-part granularity starts with Gemini 3.

Supported formats

SDK `format`	MIME type (`content_type` for uploads)
`mp4`	`video/mp4`
`mov`	`video/mov`
`mkv`	`video/x-matroska`
`webm`	`video/webm`
`flv`	`video/x-flv`
`mpeg`	`video/mpeg`
`mpg`	`video/mpg`
`wmv`	`video/wmv`
`three_gp`	`video/3gpp`

Limits & errors

Inline cap — 20 MB of raw video per request (all video parts combined). Exceeding it returns 413 with a message pointing to the upload flow.
Upload cap — 1 GB per file. POST /v1/files rejects a larger size_bytes, or an unsupported content_type, with 400.
Upload URL expiry — the signed upload_url is valid ~15 minutes; create a fresh slot if it lapses.
File expiry — uploaded files auto-delete after ~24 hours. Referencing an expired or unknown URI fails with 400: video file "audacity://files/…" not found or expired; upload via POST /v1/files and retry.
Non-video model — sending video to anything outside the Gemini family returns 400 before the provider is contacted. The SDKs raise ValidationException; the raw response looks like:

{
  "success": false,
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "model \"gpt-5.4-mini\" does not support video input; video is supported on: gemini-2.5-flash, gemini-2.5-pro, gemini-3-flash-preview",
    "request_id": "…"
  }
}

Video blocks are valid in user messages only, matching image input.

boto3 / AWS SDK path. The Bedrock-compatible endpoint does not accept video content blocks — it returns a ValidationException listing its supported block types. Use one of the AI Reserve SDKs or the raw OpenAI-protocol wire format above for video.

Getting started

Connect your tool

Models

Capabilities

Migrate

Data & privacy

Model support

Inline video (≤ 20 MB)

Large videos: upload, then reference by URI

Media resolution (video token cost)

Supported formats

Limits & errors

​Model support

​Inline video (≤ 20 MB)

​Large videos: upload, then reference by URI

​Media resolution (video token cost)

​Supported formats

​Limits & errors

Model support

Inline video (≤ 20 MB)

Large videos: upload, then reference by URI

Media resolution (video token cost)

Supported formats

Limits & errors