Topology-Informed Visual Prompting for Vision-Language-Action Policies

Anonymous Authors

Topology-informed visual prompts for direct motion and topology correction — Different topological signatures require different motion directions for the same task goal. Green prompts continue the task; cyan prompts first correct a topological mismatch.

Abstract

Vision-language-action policies can struggle with manipulation tasks involving complex obstacle geometries because they operate under partial observability. Similar visual observations or robot configurations can require qualitatively different actions, a distinction that can be quantified using topological signatures.

Motion planners can reason about these signatures when environment geometry and object states are known, but this information is often unavailable at deployment. We address this gap with a topology-guided visual-prompting framework that uses simulation-based planning to augment a nominal demonstration dataset and provides vision-based guidance at deployment.

Our method uses a Gauss-linking-integral topological signature to augment VLA fine-tuning data with trajectories that correct signature mismatches and resume task execution. A VLM predicts signatures from live camera observations and end-effector waypoints that are rendered as visual prompts for the VLA. Across three simulated bimanual tasks and a real-world box-pickup task, our method outperforms a VLA fine-tuned only on nominal demonstrations and a VLM-prompting baseline. On hardware, it improves task success by 40 percentage points over the strongest baseline.

Video

Method

During training, privileged simulator geometry is used to generate signature-correction and task-continuation trajectories. Their projected end-effector paths supervise a VLA and a VLM. At deployment, the VLM predicts the current topological signature and image-space waypoints, which are rendered on the observation as a visual prompt for the VLA. The guidance is updated after every action chunk.

Training and deployment pipeline for topology-informed visual prompting — Overview of the training and deployment pipelines. Topology guides data generation in simulation and closed-loop visual prompting at deployment.

Simulation Results

Task success across two VLA backbones. Each setting uses 50 Rope Pulling, 100 Cable Threading, or 100 Box Pickup trials.

Task	Method	VLA backbone
Task	Method	π₀	π_0.5
Rope Pulling	Base Policy	70	52
	PEEK	54	50
	Ours	66	64
Cable Threading	Base Policy	49	47
	PEEK	13	20
	Ours	71	77
Box Pickup	Base Policy	31	45
	PEEK	57	66
	Ours	84	82

Ablation Results

Task-success rates using the π_0.5 backbone.

Method	Rope Pulling	Cable Threading	Box Pickup
Full Method (Ours)	64	77	82
No Visual Prompting	52	69	75
Task-Continuation Data Only	N/A	24	54

N/A indicates that the Task-Continuation Data Only ablation is not applicable to Rope Pulling because the task requires no signature correction.

Removing VLM-generated prompts while retaining the same topology-aware training data reduces success across all three tasks, indicating that the VLM helps guide the VLA during execution. Using only task-continuation data causes larger drops on tasks requiring signature correction, showing that topology also directs data generation toward relevant recovery states.

Simulation Tasks

Rope Pulling
No signature correction

Cable Threading
Correction to h^v = 1

Box Pickup
Correction to h^v = 0

Real-World Box Pickup

Method	Task success (%)	h^v-correction success (%)
Base Policy π₀	20	30
PEEK	3	20
Ours	60	83

Thirty real-world trials. All methods use the same real-world adaptation protocol.