Vision and touch are naturally complementary for dexterous manipulation, yet visual observations are dense and spatially structured while tactile measurements are sparse, heterogeneous, and robot-centric. Each tactile measurement is attached to a known location on the robot’s kinematic chain. Can this localization prior provide a shared representation that explicitly aligns touch with vision?
We introduce Tac2Pix, an image-space visuo-tactile fusion framework that projects sparse tactile measurements onto the RGB image plane using forward kinematics and camera calibration, then renders them as force-aware saliency maps. By expressing touch in the same pixel space as vision, Tac2Pix directly leverages pretrained 2D visual representations. Zero-initialized input adaptation preserves the original visual pathway at the start of policy learning.
We evaluate Tac2Pix on three simulated and three real-world dexterous manipulation tasks. Simulation experiments span four policy architectures and multiple visual backbones. Across the three real-world tasks, Tac2Pix improves average success under fixed visual occlusion by 26.7 percentage points over the strongest latent visuo-tactile fusion baseline. An additional three-seed Pick-and-Place evaluation tests robustness under fixed, random, and physical occlusions.
