Why this matters Many subject-driven video generation methods either sacrifice subject fidelity for flexible interactions or fail to exploit intra-subject references (OCR maps, multi-view images) to improve identity and interaction realism. HOMIE’s central insight is that extracting reference-level relationships from an MLLM and aligning those semantics directly with VAE tokens — via attention-level guidance and explicit modality-reference embeddings — lets a single system handle both inter- and intra-subject personalization without costly re-alignment steps.
Key Findings
- Global multimodal guidance in self-attention: injects MLLM-derived semantic signals into VAE token attention so that high-level reference knowledge (e.g., object roles, textual cues) influences generative decoding — this helps maintain consistent interactions across frames.
- Modality-reference embedding: tags and differentiates tokens originating from MLLM features versus VAE tokens and binds intra-subject reference image tokens, enabling the model to leverage OCR or multiple views without retraining text encoders.
- Unified inter-/intra-subject pipeline: the architecture supports both inter-subject personalization (new subjects) and intra-subject enhancement (multi-view/OCR references), improving subject fidelity while preserving controllability of text prompts.
- Empirical improvements: experiments reported state-of-the-art results on multiple HOCVP tasks, particularly where accurate human-object interaction patterns or abstract object references (e.g., logos, text) are required.
Who it's for and trade-offs
Great fit if you need high-fidelity, interaction-aware personalized video generation that must respect both subject identity and complex human–object relations, and if you have multimodal references (images, OCR) to guide generation. It’s also useful when you want to combine MLLM semantic reasoning with VAE-based video decoders without expensive re-alignment.
Look elsewhere if you need a lightweight, real-time inference stack on very constrained hardware (HOMIE focuses on architectural alignment and multimodal fusion, which adds complexity), or if your use case is unconcerned with detailed human–object interactions and only requires generic stylized avatar rendering.
Where it fits
HOMIE sits between pure text-driven subject personalization and heavy multi-view reconstruction approaches: it leverages MLLM semantics to bridge abstract references (text, logos) with pixel-level VAE tokens, making it well-suited for applications like branded content personalization, AR overlays respecting text/logos, and subject-centric video editing.