AIAny
AI Video2026
Icon for item

claude-video

Enables Claude to “watch” videos by extracting timestamped frames plus captions/transcripts and feeding them to Claude for grounded Q&A. Key features: native captions first, Whisper fallback, frame deduplication, and multiple detail modes (transcript/efficient/balanced/token-burner). Useful for summarizing, debugging, and extracting moments.

Introduction

Watching video is a consistent blind spot for text-first LLM workflows: titles and isolated transcripts miss most of what’s shown on screen. This project bridges that gap by assembling timestamped visual frames and audio transcripts, then handing both to Claude’s multimodal Read tool so responses are grounded in what’s actually seen and heard.

What Sets It Apart
  • Frame+transcript-first pipeline: it prefers native captions (via yt-dlp) and falls back to Whisper (Groq or OpenAI) only when needed, so many public videos require no audio upload.
  • Tunable visual fidelity vs. cost: four detail modes — transcript (captions-only), efficient (keyframes, ~50 cap), balanced (scene-aware default, 100 cap), and token-burner (uncapped). This keeps image-token costs predictable and lets you re-run focused segments for denser coverage.
  • Frame deduplication and budgeting: a fast thumbnail-based pass drops near-duplicate frames (default threshold ~2.0) so the frame budget is spent on distinct visual content rather than held slides or static screens.
  • Multi-surface integration: packaged as a self-contained skill for Claude Code / claude.ai, and installable via the Agent Skills CLI or npm for broad agent hosts, making it easy to add video-aware capability to existing Claude workflows.
Who It's For and Tradeoffs

Great fit if you need concise, grounded answers about video content — summarization, finding what happens at a given timestamp, diagnosing UI bugs from screen recordings, or converting lecture/playlists into searchable notes. It’s tuned for practicality: fast transcript runs can cover long videos at low cost, while balanced/token-burner give richer visual coverage when required.

Look elsewhere if you need frame-level computer-vision research pipelines (no heavy CV model training here) or extreme per-frame fidelity for high-resolution OCR by default — the tool prioritizes a practical tradeoff between extraction speed, token cost, and multimodal grounding.

Information

  • Websitegithub.com
  • OrganizationsSolaris Automation
  • AuthorsBrad Bonanno
  • Published date2026/04/24

Categories

More Items

Hugging Face
AI Model2026

Simulates how a scene changes in response to text-described actions, optionally conditioned on a starting image, and generates videos up to 20 seconds long. It pairs language-model reasoning about events with a 29B diffusion model for visual prediction.

GitHub
AI Video2026

Runs local video, capture-card feeds, and game streams through a real-time GPU enhancement pipeline with super resolution, denoising, HDR processing, and frame generation. Supports NVIDIA and AMD GPUs on Windows 11; some models and features depend on specific hardware or runtimes.

Hugging Face
AI Model2026

Generates video with synchronized stereo audio from text, keyframes, or reference media in 4–8 sampling steps. Pruning, low-rank factorization, and INT8/GGUF formats reduce runtime costs, though speed and hardware needs vary by setup.