Watching video is a consistent blind spot for text-first LLM workflows: titles and isolated transcripts miss most of what’s shown on screen. This project bridges that gap by assembling timestamped visual frames and audio transcripts, then handing both to Claude’s multimodal Read tool so responses are grounded in what’s actually seen and heard.
What Sets It Apart
- Frame+transcript-first pipeline: it prefers native captions (via yt-dlp) and falls back to Whisper (Groq or OpenAI) only when needed, so many public videos require no audio upload.
- Tunable visual fidelity vs. cost: four detail modes —
transcript(captions-only),efficient(keyframes, ~50 cap),balanced(scene-aware default, 100 cap), andtoken-burner(uncapped). This keeps image-token costs predictable and lets you re-run focused segments for denser coverage. - Frame deduplication and budgeting: a fast thumbnail-based pass drops near-duplicate frames (default threshold ~2.0) so the frame budget is spent on distinct visual content rather than held slides or static screens.
- Multi-surface integration: packaged as a self-contained skill for Claude Code / claude.ai, and installable via the Agent Skills CLI or npm for broad agent hosts, making it easy to add video-aware capability to existing Claude workflows.
Who It's For and Tradeoffs
Great fit if you need concise, grounded answers about video content — summarization, finding what happens at a given timestamp, diagnosing UI bugs from screen recordings, or converting lecture/playlists into searchable notes. It’s tuned for practicality: fast transcript runs can cover long videos at low cost, while balanced/token-burner give richer visual coverage when required.
Look elsewhere if you need frame-level computer-vision research pipelines (no heavy CV model training here) or extreme per-frame fidelity for high-resolution OCR by default — the tool prioritizes a practical tradeoff between extraction speed, token cost, and multimodal grounding.
