Video-Based Optimal Transport for Feedback-Efficient Offline Preference-Based Reinforcement Learning

VOTP is a semi-supervised framework for offline preference-based RL that learns an effective reward function from only a handful of human preference labels. The key idea is to use optimal transport (OT) to align visual trajectories within the rich representation space of Video Foundation Models (ViFMs), turning a small labeled set into high-fidelity supervision over a large pool of unlabeled trajectories.

Given a small set of human-labeled trajectory pairs and a large set of unlabeled trajectories, VOTP first embeds each trajectory into the representation space of a pretrained ViFM (e.g., S3D). It then computes an optimal transport alignment between unlabeled trajectories and labeled references, using the resulting transport plan to propagate preferences from labeled to unlabeled pairs through relative alignment strengths. Training the reward model on both the human labels and these pseudo-labels substantially reduces the amount of human supervision required. The learned reward is then used to optimize the policy with an off-the-shelf offline RL algorithm (e.g., IQL).

In simulated environments, we evaluate VOTP on Gym D4RL locomotion and MetaWorld manipulation benchmarks under limited feedback budgets and compare it against state-of-the-art offline preference-based RL methods. We also evaluate its robustness to visual distractors and validate it on real-world robotic tasks.

Comparison to Baselines

We use IQL as the offline RL algorithm in VOTP. The results show that VOTP achieves strong policy performance under limited human feedback (e.g., 10 labels). Notably, by leveraging unlabeled data, VOTP significantly improves performance over P-IQL, which uses the same underlying RL algorithm but is trained only on labeled data.

OT vs. Other Heuristics

This analysis highlights the role of optimal transport in pseudo-label inference. We compare OT with heuristic methods that infer preferences solely from similarity. Unlike these heuristics, VOTP aggregates preferences from multiple labeled trajectory pairs, weighting each contribution according to the relative alignment strengths computed by the OT plan to produce more reliable pseudo-labels.

Ablation on Different Trajectory Encoders

We find that Video Foundation Models generally outperform Image Foundation Models. This improvement highlights the advantage of video representations, which capture temporal dynamics and subtle motion cues to provide richer trajectory representations. These cues are crucial for distinguishing behavioral differences when inferring preferences.

Effect of the Number of Initial Labels

Performance generally improves as the number of initial labels increases. Notably, on the door-open task, VOTP trained with only 10 labels outperforms a policy trained with ground-truth rewards.

Scatter Plots of Estimated Rewards

The strong Pearson correlation between the estimated and ground-truth rewards demonstrates that VOTP effectively leverages unlabeled data to learn a more accurate reward model. This improved reward quality explains VOTP's substantial performance gains over P-IQL, which relies solely on labeled data.

Estimated Rewards on Real-World Tasks

Lift Banana

Drawer Open

We visualize the learned rewards along successful and failed trajectories. Although both methods correctly assign appropriate rewards to successful behavior, P-IQL mistakenly assigns similarly high rewards to failed trajectories. In contrast, VOTP learns a much more discriminative rewards, clearly separating successful and failed behaviors.

We compare policies trained with VOTP against the P-IQL baseline on MetaWorld manipulation tasks under the same limited-feedback budget (For each task, use the < / > arrows to switch between VOTP and P-IQL).

We deploy VOTP on a real Sawyer robot and compare it with the P-IQL baseline on two manipulation tasks. For each task, the top row shows VOTP rollouts and the bottom row shows P-IQL rollouts.