AIAny
Icon for item

JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents

Lets canvas-native agents plan, generate, edit, and organize long-horizon multimodal creative projects by representing artifacts, versions, and actions as typed canvas nodes and links. Uses a three-layer design (canvas state, protocol bridge, agent runtime) so agents act within an inspectable, editable project state.

Introduction

Most generative systems treat each asset as an isolated input→output step; real creative work is a stateful, branching project with references, drafts, edits, failures, and version relations. JarvisHub reframes the user canvas as the shared project state and external memory, enabling agents to operate inside an inspectable, editable workspace rather than as opaque tool callers.

Key Findings
  • Canvas-as-state: Multimodal artifacts, dependencies, versions, tool calls, and feedback are modeled as typed canvas nodes and links so agents can read and update an explicit project graph rather than discarding intermediate context — this preserves provenance and enables programmatic revisions.
  • Three-layer architecture: A canvas state layer, a protocol bridge, and an agent runtime separate representation, integration, and control logic, making agent actions inspectable and interruptible while easing integration of new tools and models.
  • Long-horizon workflows: Agents can progressively plan, generate, revise, and organize multimodal projects (images, video, audio, UI elements, storyboards) with versioning and failure recovery, supporting human steering and intermittent intervention.
Who it's for and tradeoffs

Great fit if you build or study interactive creative agents, multimodal pipelines, or product features that require provenance, versioning, and human-in-the-loop revision. JarvisHub is oriented toward research and platform integration rather than lightweight prompt-based generation: it adds representation and orchestration overhead, requires instrumenting a canvas and tool adapters, and depends on available multimodal models for quality. If your use case is simple single-step asset generation or a minimal-hosted UI, a simpler prompt/chat integration may be preferable.

Where it fits

JarvisHub sits between prompt/chat-first generation tools and closed commercial creative suites: it focuses on an open, inspectable harness that encodes project state and agent decisions, making it useful for experimenters and teams building reproducible, auditable agent workflows.

How it works (brief)

Agents interact via a protocol bridge that translates canvas graph edits into tool calls and back; the canvas stores typed nodes for artifacts and links for dependencies/versions, while the runtime manages planning, action sequencing, retries, and human handoffs. This design emphasizes auditability and iterative revision over opaque one-shot generation.

Information

  • Websitearxiv.org
  • AuthorsYunlong Lin, Zixu Lin, Zhaohu Xing, Biqiang Li, Chenxin Li, Haonan Wang, Haitao Wu, Hengyu Liu, Jianghai Chen, Kaituo Feng
  • Published date2026/07/26

Categories

More Items

Drives long‑horizon desktop agents by reading and manipulating program state (files, DOM, backends) instead of relying on screenshots. The main agent uses code for actions and structural verification while a lightweight GUI subagent handles rare screenshot-click steps, improving success rates and lowering per-task cost versus screenshot-only approaches.

Bridges the proprietary-to-open-source gap in agentic search by converting multi-step retrieval and reasoning traces into a structured, style-normalized JSON protocol and using it for joint distillation + RL. Produces denser supervision that improves student success rates while reducing style drift.

A PyTorch-native training framework for agentic reinforcement learning research that keeps researcher-facing code compact and editable. Uses an asynchronous loop to train multimodal and mixture-of-experts policies while never training on tokens the agent didn't generate; matches Megatron-style stacks under a comparable protocol and ships recipes and containers on GitHub.