Your Vision-Language-Action Model Already Has Attention Heads For Path Deviation Detection

TL;DR

We propose an end-to-end robotic navigation framework that pairs a high-level VLA model with a continuous low-level RL policy. During normal operation, the RL policy constantly ensures reactive, collision-free obstacle avoidance, while highly accurate LiDAR-RGB-IMU sensor fusion (LiVO) continuously logs safe waypoints. Simultaneously, by monitoring specialized internal attention heads, the system detects VLA hallucinations and path deviations in real-time with near-zero computational overhead. When a deviation occurs, the system seamlessly leverages the active RL policy to execute a direct safe recovery, autonomously navigating back to the last correct path for highly reliable real-world deployment. The entire pipeline is built on ROS 2, enabling modular integration and real-time operation on physical hardware.

Key Contributions

• Discovery of Navigation Heads — We identify a sparse subset of attention heads (H_nav) in frozen VLA models that inherently capture the spatiotemporal alignment between visual observations and linguistic intent, exhibiting a strong correlation with the agent's navigation state.
• Training-Free Anomaly Detection — We propose a real-time detection framework that monitors H_nav dynamics with near-zero computational overhead and no trainable parameters, achieving a 65.1% path deviation detection rate (recall) and a 76.4% F1 score in R2R unseen environments, thereby effectively monitoring the model hallucination issue.
• VLA–RL Integrated Pipeline — We couple the high-level anomaly signal with a continuous low-level RL policy, ensuring robust collision-free obstacle avoidance during normal operation and enabling autonomous, direct safe recovery upon detecting a path deviation.
• Real-World Deployment — The entire end-to-end framework is deployed zero-shot onto a physical robot, demonstrating practical robustness and reliable embodied navigation in real-world conditions.

Project Design

HiVLA runs two parallel loops: the high-level VLA model (~0.3 Hz) interprets egocentric observations and language instructions to plan navigation waypoints, while the low-level RL policy (~10 Hz) handles reactive obstacle avoidance in real time. Localization is provided by multi-sensor fusion, which tightly fuses LiDAR, RGB, and IMU data, and the full pipeline is orchestrated over ROS 2.

The anomaly detection pipeline is divided into Offline preparation and Online real-time inference, operating across four stages. (i) Phase Labeling: Prior data is segmented into distinct navigation phases. (ii) Head Selection: Each attention head is scored by how well its activations correlate with those phase transitions, determining the optimal head combination and hyperparameters. (iii) Anomaly Detection: At inference, residual spikes in the selected heads are monitored in real time to flag path deviations. (iv) Action Policy: Any flagged deviation immediately hands control to the RL recovery policy.

Navigation Heads (H_nav) are identified as the top-performing heads that distinctly exhibit the four predefined spatiotemporal alignment properties (S_uniform, S_diag, S_peak, S_shift) while demonstrating high sensitivity to shifts in the navigation state. Furthermore, to maximize the robustness of anomaly detection using these selected heads, we perform a grid search, as illustrated in the figures below, to derive the optimal combination of detection hyperparameters. Consequently, during online inference, the moment a path deviation begins these heads produce a sharp, detectable divergence pattern, thereby enabling immediate anomaly detection.

(a) Discriminability of attention heads: measuring which heads are most sensitive to anomalous behaviors (path deviations). (b) Episode-level detection performance across different ensemble sizes (K): showing how precision and recall vary as we increase the number of monitored heads. (c) Step-level classification performance on actual navigation failure episodes: demonstrating real-time detection capability at fine temporal granularity. Notably, an ensemble of K = 3 heads achieves the best balance between detection rate and false alarm rate.

(a) Effect of window size W on detection performance: showing how the temporal window for anomaly scoring impacts sensitivity. (b) Effect of point count P on false episode rate: demonstrating the trade-off between suppressing false alarms and detection latency. (c) Sensitivity to decision threshold τ: revealing sharp transitions in detection behavior near the optimal threshold. Ultimately, the detector is configured with (W = 10, P = 9, τ = 0.95) to maintain robust anomaly detection while minimizing unnecessary interruptions during normal navigation.

The actor-critic network combines a CNN branch that encodes the local costmap for spatial obstacle awareness with an MLP branch that encodes the goal vector. Together they produce continuous velocity commands (v_x, ω_z) that cover both routine obstacle avoidance and emergency deviation recovery.

Training is conducted in Isaac Lab using PPO. Obstacle layouts and goal positions are randomized each episode, pushing the policy to generalize beyond the specific configurations it was trained on.

The simulated environment is built at 1:1 scale to match the physical AgileX Scout 2.0, minimizing the sim-to-real gap and enabling zero-shot transfer to the real robot.

The system is deployed on an AgileX Scout 2.0 powered by an NVIDIA Jetson AGX Orin (64 GB). A custom Control Center WebUI provides live visualization of navigation states, anomaly signals, and logged safe waypoints throughout field experiments.

All components communicate over ROS 2 as modular nodes: VLA inference, Navigation Head monitoring, RL policy execution, Fast-LIVO2 localization, and sensor data streams each run independently at configurable rates, making the pipeline easy to extend or swap out.

Extrinsic calibration between the ZED 2i stereo camera and RS-LiDAR-32 ensures precise spatial alignment — a prerequisite for the tight LiDAR-RGB-IMU fusion that Fast-LIVO2 relies on for low-drift localization.

Compared to ZED VIO (vision-only odometry), Fast-LIVO2 achieves substantially lower trajectory and yaw drift, providing the reliable pose estimates needed to accurately log safe waypoints for the recovery module.

Qualitative results of our path deviation detection and recovery pipeline, evaluated in both simulation and the real world. (a) In VLN-CE simulation (VLA), once a deviation is detected, the agent is teleported back to the last logged safe waypoint for clean recovery. (b) In the real world (VLA+RL), teleportation is not possible — instead, the RL policy drives the robot directly back to the saved waypoint. Since obstacles may lie along this recovery path, the RL policy's collision-free navigation capability becomes essential to complete the recovery safely.

Experiments

Anomaly Detection — NaVILA Backbone

Training-free anomaly detection on R2R Val-Seen and Val-Unseen. All baselines require near-zero inference overhead; hyperparameters derived solely from the train split. Bold = best per column.

Anomaly Detection — NaVID Backbone

Architecture generalizability: the same pipeline applied to NaVID, with heads and hyperparameters optimized on the R2R train split only.

RL Obstacle Avoidance — Navigation Performance by Distance

SR: Success Rate ↑, CR: Collision Rate ↓, TR: Timeout Rate ↓. Bold = best, underline = second best per column.

Computational Efficiency — VLA Module

Overhead of integrating the path deviation detection module. Evaluated on a single NVIDIA RTX 6000 Ada GPU. No additional VRAM is required.

Computational Efficiency — RL Policy

Lightweight CNN + MLP (4.89 M parameters). Evaluated on a single NVIDIA RTX 6000 Ada GPU.

End-to-End Inference — Jetson AGX Orin

VLA inference time over 100 runs. Compared to standalone NaVILA, the full HiVLA pipeline adds only ~0.55 s, which includes localization and path deviation monitoring.

State Estimation — Accuracy Comparison

Fast-LIVO2 vs. ZED VIO over 10 indoor and 10 outdoor runs. Drift measured relative to a fixed AprilTag. Bold = lower (better) error per metric.

State Estimation — Resource Overhead (Jetson AGX Orin)

Deployment Hardware

Demo Videos

Conclusion

We present a robust framework for VLA-based robot navigation that successfully monitors the navigation state and detects path deviations by strictly utilizing model-intrinsic signals. Extensive evaluations across diverse VLA architectures and datasets demonstrate that our approach effectively addresses the VLA hallucination issue at near-zero additional computational cost. Going beyond anomaly detection, we seamlessly integrated this high-level signal with a continuous low-level RL policy to actively prevent obstacle collisions during movement. Ultimately, this end-to-end framework was deployed onto a physical robot without any environment-specific re-identification, confirming its safe and reliable operation in real-world environments.