Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

Video understanding on the Interactions API: files registered from GCS (files.register_files) produce inflated, fabricated event lists; the same bytes uploaded (files.upload) do not

Đang mở
  • Có gì không ổn?
#3,072 1 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@Venkaiahbabuneelam đang làm issue này rồi.

Từ ngày 8/10/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

priority: p2 type: bug

I use video understanding with gemini-3.1-pro-preview to count how many times a specific chain of events occurs in a video. The video is sampled at 4 fps (static video processing, processing: {"type": "static", "fps": 4}) rather than the default 1 fps, because the events are short. The model returns one structured JSON entry per occurrence, with start and end timestamps and a short description. Our test video has 13 counts of the chain (hand-counted).

This application had been running without issue on the previous API (models.generate_content), with videos both registered from GCS and uploaded through the File API. The problem appeared when I migrated the same requests to the Interactions API.

On the Interactions API, the answer depends on how the video is set as input:

  • Registered from Cloud Storage (client.files.register_files(uris=["gs://…"])): 25 of 60 calls (about 40%) returned an inflated, fabricated list. These answers report 24–52 occurrences instead of 13.
  • Uploaded with File API directly (client.files.upload(file=…)): 0 of 20 calls did this. The mean count was 13.6–13.7.
  • Old API (models.generate_content) with the same model, prompt, schema and fps: 0 of 30 testing calls did this, whether the file was registered or uploaded. (Neither did I encounter this issue with non-testing data)

Everything else about the requests in those tests was identical.

What an "inflated, fabricated list" looks like

These answers aren't small counting errors. The list stops reflecting the video:

  • Far too many entries: 24 to 52 occurrences reported where there are 13, roughly 2–4× the true count.
    Timestamps on a fixed grid: entries are spaced evenly every 2–3 seconds from the start of the video to the end, whether or not anything happens at those times.
  • Copy-pasted descriptions: most entries share one identical, generic description (for example "Consistent performance." repeated 41 times out of 51), instead of describing what actually happens in each occurrence.
  • The count is decided up front: the thought summary typically states a large total early on (for example "parsed into 52 distinct repetitions") and then fills the list to match it.

Correct answers, by contrast, have unevenly spaced timestamps that follow the real events, and a distinct description for each entry.

For counting, we classify an answer as inflated when it reports 24 or more occurrences, close to twice the true 13. Correct answers never came near that: across all clean arms the counts ranged from 8 to 19.

Environment details
  • Both macOS 27 & Cloud Run container
  • SDK: google-genai==2.28.0 (Interactions), and google-genai==2.11.0 for the generate_content baseline
  • Python: 3.11
  • Model: gemini-3.1-pro-preview
  • Test Video: 126 s, 1280×720, 30 fps, MP4
  • Request settings:
    • static video processing at fps: 4
    • no resolution set (the API default)
    • service_tier="standard"
    • store=False
    • default thinking, with thinking_summaries="auto"
    • structured output via response_format (type: text, mime_type: application/json, JSON Schema from a Pydantic model)
  • Dates: 2026-10-07 and 2026-10-08
Steps to reproduce
import json, os
from google import genai

client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])

# A) registered from GCS
reg = client.files.register_files(auth=gcs_credentials, uris=["gs://<bucket>/videos/clip.mp4"]).files[0]
# B) uploaded (same bytes)
up = client.files.upload(file="clip.mp4", config={"mime_type": "video/mp4"})
# (wait until up.state == ACTIVE)

def count_events(file_uri):
    ix = client.interactions.create(
        model="gemini-3.1-pro-preview",
        input=[
            {"type": "video", "uri": file_uri, "mime_type": "video/mp4",
             "processing": {"type": "static", "fps": 4.0}},
            {"type": "text", "text": PROMPT},  # "identify every occurrence of <chain of events>…"
        ],
        response_format={"type": "text", "mime_type": "application/json",
                         "schema": EventsAnalysis.model_json_schema()},
        generation_config={"thinking_summaries": "auto"},
        service_tier="standard",
        store=False,
    )
    return len(json.loads(ix.output_text)["events"])

Ruled out by changing one variable at a time and running set of 10 calls:

  • video codec (HEVC vs H.264)
  • audio processing (loudness-normalised vs untouched)
  • keyframe spacing (8–42 s vs 1 s)
  • JSON Schema vs SDK-converted schema
  • explicit temperature

Other observations about registered files

  • No video metadata. After register_files the file is ACTIVE immediately and files.get(...).video_metadata is None. After files.upload the file goes PROCESSING → ACTIVE (about 45 s for 25 MB) and reports video_metadata={'videoDuration': '126s'}.
  • Same token count. Both deliveries bill the same input tokens (39,801 vs 39,802), so the model receives the same amount of video. What differs seems to be the frames or timestamps it receives, not how many. That would fit the evenly spaced, ungrounded timestamps in the inflated answers.

Workaround

Download the object and use files.upload instead of register_files.

Ngôn ngữ chính
Python
Star
4k
Fork
1k
Merge trung bình
1 ngày 8 giờ
Pull request đã merge (30 ngày)
93

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của googleapis/python-genai

Tất cả issue của googleapis/python-genai

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.