RGB-D Video Generation for Improving Human-to-Robot Object Handover Prediction

1. Introduction: The Perception Bottleneck in Human-Robot Handovers

Human-to-robot (H2R) object handover is a foundational capability for collaborative robotics across assistive, domestic, and industrial domains. For a robotic system to execute a seamless physical transfer, its control framework must accurately anticipate human intent—determining both the precise onset timing and spatial trajectory of the handover—to initiate reaching maneuvers without endangering the human partner.

Current perception pipelines regularly suffer from premature false triggers, execution delays, and physical collision risks. Tianyu Sun et al. identify two structural perception bottlenecks in existing frameworks that contribute directly to these failure modes:

1. Lack of Full-Body Context in Hand-Centric Benchmarks: Mainstream handover datasets rely almost exclusively on egocentric or cropped hand-centric fields of view. By constraining tracking logic to localized hand kinematics, policies miss crucial global communicative cues—such as head orientation, facial micro-expressions, visual gaze focus, and upper-body posture—that naturally precede physical reaching motions.
2. The Sim-to-Real Gap in Synthetic Depth Sensing: Robotic learning frameworks frequently rely on synthetic training environments that supply pristine, noise-free depth maps. In contrast, physical depth sensors (such as structured-light or Time-of-Flight systems) introduce stochastic noise patterns, including edge scattering, surface light absorption, and zero-depth void artifacts. Because real-time depth inpainting is computationally prohibitive during closed-loop control, policies trained on idealized geometry degrade rapidly when exposed to real-world sensor degradation.

To address these twin challenges, Tianyu Sun et al. introduce PassGen, a two-stage generative RGB-D video synthesis framework, alongside Hand2Bot, a multimodal dataset engineered specifically for full-body human-to-robot handover perception. Their methodology combines dual-stage video diffusion with empirical depth noise modeling, enabling intent anticipation and zero-shot transfer on physical robotic hardware.

---

2. Benchmark Shift: Hand-Centric Limits vs. Full-Body Context in Hand2Bot

Prior handover benchmarks have advanced manipulation modeling, but they retain major structural boundaries. Synthetic datasets such as HandoverSim and GenH2R offer scalable interaction sequences but omit physical sensor noise, inducing an ideal-geometry bias. Conversely, real-world benchmarks like DexH2R, H2O, and DexYCB restrict their perceptual field of view to localized hand regions, removing global context.

The Hand2Bot dataset constructed by Sun et al. addresses these limitations by capturing a full-body field of view, incorporating physical sensor noise patterns, and providing aligned multimodal sequences across physical and synthesized captures.

Comparative Benchmark Mapping

The table below contrasts Hand2Bot against standard handover benchmarks across core perception attributes:

Dataset	Data Source (Synthetic/Real/Both)	Depth Noise Modeling	Perception Scope (Hand vs. Full Human)	Receiver Paradigm
HandoverSim	Synthetic	No ($\times$)	Hand	Parallel Gripper (H2R)
GenH2R	Synthetic	No ($\times$)	Hand	Parallel Gripper (H2R)
H2O	Real	No ($\times$)	Hand	Hand (H2H)
HOH	Real	Yes ($\checkmark$)	Hand	Hand (H2H)
DexH2R	Real	Yes ($\checkmark$)	Hand	ShadowHand (H2R)
DexYCB	Real	Yes ($\checkmark$)	Hand	Unspecified (H2A)*
Hand2Bot	Both	Yes ($\checkmark$)	Full Human	Parallel Gripper (H2R)

*Note: In benchmark classification, “H2A” denotes a Hand-to-Anything paradigm, where the receiver recipient is environment-nonspecific or unconstrained.

Dataset Parameters and Intention Labeling

The complete Hand2Bot dataset contains 5,000 multimodal RGB-D video pairs:

• 2,125 Real-World Clips (Hand2Bot-Real): Collected using an Intel RealSense L515 sensor across 7 diverse indoor environments with 33 common household objects.
• 2,875 Synthesized Clips (Hand2Bot-Syn): Generated via the PassGen pipeline to systematically expand character identity, background context, and pose diversity.

To condition models to reject non-handover interactions, the authors explicitly incorporated a 20% subset of negative intentional trials. Rather than relying on simple static backgrounds, these negative trials incorporate ambiguous physical movements, non-gaze tool passes, weak-gaze reaching motions, and ambient upper-body shifts.

For all positive handover trials, the dataset provides precise frame-level timestamp annotations for the presentation period, defining the temporal window during which handover intent is actively initiated.

Grasp Pose Annotation and Vertical-Alignment Filtering

To connect visual intent perception with robotic execution, Sun et al. integrated GraspNet-1Billion to annotate candidate 6-DoF parallel-jaw gripper grasp poses, constraining workspace masks strictly to the target object region.

To maintain kinematic feasibility and prevent high-velocity arm configurations, the authors introduced a vertical-alignment filtering strategy. Let $\mathbf{R} \in SO(3)$ represent the candidate grasp orientation matrix, $g_z$ denote the gripper approach vector, and $\mathbf{v}_z = [0, 0, -1]^T$ represent the canonical downward gravity vector. The filtering system evaluates the angular deviation:

$\theta = \arccos(g_z \cdot \mathbf{v}_z)$

The pipeline dynamically selects collision-free poses that minimize $\theta$, prioritizing top-down vertical grasp trajectories while explicitly filtering out pose candidates that establish physical contact with the human hand.

---

3. The PassGen Architecture: Dual-Stage RGB-D Video Generation

PassGen is a two-stage generative framework designed to synthesize high-fidelity interactive RGB handover videos and map them to sensor-realistic depth streams.

+-----------------------------------------------------------------------------------+
|                                 PASSGEN PIPELINE                                  |
|                                                                                   |
|  [Reference Image I_ref]  ---> [Appearance Encoder] ---\                          |
|                                                         +--> [Denoising U-Net]    |
|  [Reference Video V_ref]  ---> [PoseNet (DWPose)]  ---/      (LoRA Fine-Tuned)    |
|       |                                                 ^                         |
|       +---> [ArcFace] ---> [Temporal Face Encoder] -----+                         |
|                                                                 |                 |
|                                                         [Stage I: RGB Video]      |
|                                                                 |                 |
|                                                                 v                 |
|                                                         [DepthCrafter D_init]     |
|                                                                 |                 |
|                                                         [Height-Linear Scale]     |
|                                                                 |                 |
|                                                         [Morphological Erosion]   |
|                                                                 |                 |
|                                                         [Stage II: Noisy Depth]   |
+-----------------------------------------------------------------------------------+

3.1 Stage I — Pose-Guided & Intention-Aware RGB Synthesis

PassGen utilizes a Stable Video Diffusion (SVD) backbone. To enforce physical grounding and retain identity, the denoising U-Net is conditioned via two parallel input structures:

1. Control Signal Integration: An Appearance Encoder (ReferenceNet) extracts identity context from a single reference image $I_{\text{ref}}$, while a lightweight PoseNet processes per-frame skeletal poses $V_{\text{ref}}$ extracted by DWPose.
2. Intention-Aware Temporal Face Encoder (TFE): Conventional human animation frameworks treat the head as a rigid pose block, obscuring fine facial dynamics. PassGen incorporates a dedicated TFE branch to track facial micro-expressions and temporal gaze focus. High-dimensional face embeddings are extracted using ArcFace and processed through cascaded Temporal Attention (TA) and Feed Forward Network (FFN) blocks. This outputs temporal face tokens $\tilde{F}_t$ that track visual gaze continuity.

The integrated pose and facial guidance tokens $\tilde{P}_t$ are injected into the SVD U-Net’s cross-attention layers during LoRA fine-tuning on Hand2Bot-Real:

$U_t^l \leftarrow U_t^l + \text{CrossAttention}(Q = U_t^l, K = \tilde{P}_t, V = \tilde{P}_t)$

3.2 Stage II — Morphology-Based Depth Noise Simulation

Generic generative depth estimators (such as DepthCrafter) produce smooth depth maps that lack physical sensor noise. When deployed on physical hardware, models trained on such pristine depth maps exhibit performance drops due to unmodeled optical artifacts.

To bridge this gap, PassGen implements an empirical, morphology-based noise editing strategy designed to replicate the boundary failure patterns of the Intel RealSense L515 Time-of-Flight sensor:

1. Depth Initialization & Scale Alignment: An initial depth sequence $D_{\text{init}}$ is generated via DepthCrafter to maintain temporal smoothness. PassGen then applies a height-based linear scale alignment to project metric depth values back into the segmented human workspace mask.
2. Morphological Edge Erosion: RealSense L515 sensors generate boundary void distributions ($N_0$) resulting from edge scattering and surface light absorption. PassGen maps boundary width vectors $w(h)$ relative to pixel height $h$ to erode the generated depth boundaries $D_{\text{gen}}$.

Note on Physical Modeling Scope: The authors explicitly note that this morphological edge erosion via height-dependent boundary width vectors $w(h)$ is an empirical, heuristic approximation of RealSense L515 boundary void distributions ($N_0$), rather than a fully derived optical physics simulation of laser scattering.

Visual Evaluation Benchmarks and Component Ablations

Sun et al. evaluated PassGen against human video animation baselines across 250 test clips from Hand2Bot-Real. The quantitative comparison is detailed below:

Method	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	FID $\downarrow$	FID-VID $\downarrow$	FVD $\downarrow$
MagicPose	15.71	0.733	0.412	243.70	295.61	714.36
MagicAnimate	16.21	0.740	0.400	212.48	284.08	947.65
AnimateAnyone	16.51	0.749	0.343	190.02	203.31	623.79
DreamPose	16.58	0.753	0.352	173.90	192.13	682.55
MimicMotion	19.48	0.780	0.340	111.39	97.70	439.65
Animate-x	22.23	0.853	0.235	130.25	142.47	568.99
StableAnimator	23.17	0.867	0.218	101.67	106.73	372.81
PassGen (Ours)	25.12	0.909	0.212	91.32	99.77	337.59

To isolate the contributions of each architectural module in Stage I, the authors conducted an ablation study across the conditioning blocks. As shown in the table below, disabling the Temporal Face Encoder (TFE) and ArcFace embeddings causes significant degradation in video-level temporal consistency (FVD increases from 337.59 to 373.61) and image fidelity (PSNR drops from 25.12 to 22.77):

Backbone	Pose Guidance	TFE	ArcFace	FID-VID $\downarrow$	FVD $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	99.77	337.59	25.12	0.909
$\checkmark$	$\checkmark$	$\checkmark$	$\times$	107.44	373.61	22.77	0.829
$\checkmark$	$\checkmark$	$\times$	$\times$	111.60	399.52	21.47	0.809
$\checkmark$	$\times$	$\times$	$\times$	117.43	422.90	19.12	0.791

---

4. Intention-Gated Control & Physical Robot Evaluation

To enable socially aware interaction, Sun et al. deployed an independent Intention Gating (IG) mechanism on a physical Universal Robots UR5e platform.

                    PHYSICAL INTENTION GATING SYSTEM
                                    
  [Intel RealSense L515 Sensor] (Base-Mounted, Tilted Facing Interaction Zone)
               |
               +---> Temporal Face Tokens (F_t)  ---> Gaze Confidence f_gaze
               |
               +---> Depth Boundary Masks (D_obj) ---> Approaching Velocity v_obj
                                                              |
                                                              v
                                              [Scalar Score S_intent(t)]
                                                              |
                                              [Monotone Constraint Filter]
                                                              |
                                              [Safety Threshold Check (tau=0.80)]
                                                              |
                                                              v
                                            [Trigger Parallel Gripper Trajectory]

Physical Setup and Camera Configuration

An Intel RealSense L515 camera is mounted on the base of the UR5e robot, tilted upward toward the interaction zone. Operating at a physical distance of 1.5 m to 2.2 m from the human participant, this angle enables the wide field-of-view sensor to capture upper-body movement while retaining sufficient spatial resolution for facial gaze tracking.

Multimodal Fusion Formulation

The gating mechanism computes a continuous scalar intention score $S_{\text{intent}}(t) \in [0, 1]$ by fusing visual gaze focus with spatial approach velocity:

$S_{\text{intent}}(t) = \sigma\left(w_g \cdot f_{\text{gaze}}(\tilde{F}_t) + w_v \cdot v_{\text{obj}}(t)\right)$

Where:

• $f_{\text{gaze}}(\tilde{F}_t)$ represents the temporal gaze focus confidence score extracted from the Temporal Face Encoder.
• $v_{\text{obj}}(t) = -\frac{\partial D_{\text{obj}}}{\partial t}$ represents the object’s physical approach velocity computed from the masked depth boundaries.
• $\sigma(\cdot)$ is the standard sigmoid activation function.
• $w_g = 1.0$ and $w_v = 2.5\text{ s/m}$ serve as scaling coefficients to balance dimensionless gaze confidence with metric velocity values.

Monotone Constraint and Sensitivity Analysis

To suppress control chatter caused by momentary gaze drops or localized hand occlusions during an interaction, the control pipeline enforces a monotone non-decreasing constraint:

$\hat{S}_{\text{intent}}(t) = \max\left(\hat{S}_{\text{intent}}(t-1), S_{\text{intent}}(t)\right)$

The UR5e arm triggers its reaching trajectory if and only if $\hat{S}_{\text{intent}}(t) \ge \tau$.

Through parametric sensitivity sweeps across continuous threshold values $\tau \in \{0.60, 0.70, 0.80, 0.90\}$, the authors identified $\tau = 0.80$ as the optimal boundary. Lower settings ($\tau = 0.60$) increased responsiveness but yielded a frame-level false positive rate (FPR) of 72.5%. Conservative thresholds ($\tau = 0.90$) reduced frame-level noise to an FPR of 9.8%, but excessively restricted the motion window, causing interactive deadlocks. Setting $\tau = 0.80$ dampened the offline frame-level FPR to 13.6% while preserving adequate lead time for proactive arm activation.

Intention Score Module Ablation

To quantify the independent contributions of the gaze and velocity components, Sun et al. evaluated the Intention Score Module on a test subset of Hand2Bot-Real:

Configuration	Gaze Term	Object Term	Mean Accuracy $\uparrow$	False Positive Rate (FPR) $\downarrow$
Baseline	$\times$	$\times$	72.7%	95.5%
Gaze-Only	$\checkmark$	$\times$	86.4%	31.8%
Object-Only	$\times$	$\checkmark$	78.2%	77.3%
Full Module	$\checkmark$	$\checkmark$	90.0%	13.6%

Physical UR5e Experimental Results

The hardware deployment was evaluated across 10 distinct object types (5 Seen, 5 Unseen). Positive trials evaluated the Intention Success Rate (ISR), while negative trials measured the False Trigger Rate (FTR) during ambient human movements.

Precise Metric Definition: The authors explicitly define Intention Success Rate (ISR) as an intention-triggering metric that evaluates whether the robot correctly initiates a reaching action during the presentation period. It does not measure downstream mechanical force-closure or physical grasp completion.

OVERALL HARDWARE ACCURACY & SAFETY METRICS
===========================================================
Intention Success Rate (ISR):  [███████████████████- ] 54/60 (90.0%)
False Trigger Rate w/ IG:      [█-------------------] 2/30  (6.7%)
False Trigger Rate w/o IG:     [█████████████████---] 25/30 (83.3%)
===========================================================

Detailed Breakdown by Object Category

• Seen - Box: ISR = 10/10 | FTR (w/ IG) = 1/5 | FTR (w/o IG) = 5/5
• Seen - Cylinder: ISR = 10/10 | FTR (w/ IG) = 0/5 | FTR (w/o IG) = 5/5
• Seen - Cup: ISR = 9/10 | FTR (w/ IG) = 0/5 | FTR (w/o IG) = 4/5
• Seen - Plate: ISR = 10/10 | FTR (w/ IG) = 0/5 | FTR (w/o IG) = 5/5
• Seen - Irregular: ISR = 8/10 | FTR (w/ IG) = 1/5 | FTR (w/o IG) = 3/5
• General Unseen Objects: ISR = 7/10 | FTR (w/ IG) = 0/5 | FTR (w/o IG) = 3/5
• Overall Metrics: ISR = 54/60 (90.0%) | FTR (w/ IG) = 2/30 (6.7%) | FTR (w/o IG) = 25/30 (83.3%)

Failure Mode Analysis

The authors observed reduced performance in the Irregular (8/10 ISR) and Unseen (7/10 ISR) object categories. Their error analysis attributes these failure modes to two distinct factors:

1. Kinematic Velocity Calculation Degradation: Asymmetrical or complex object geometries introduce errors into the distance estimation pipeline. This directly degrades the physical approach velocity term $v_{\text{obj}}(t) = -\frac{\partial D_{\text{obj}}}{\partial t}$, destabilizing the scalar intention score $S_{\text{intent}}(t)$ regardless of facial gaze accuracy.
2. Mechanical Gripper Force-Closure Limits: Beyond intention gating, complex curved or irregular surfaces created mechanical force-closure challenges for the UR5e’s parallel-jaw gripper, resulting in execution failures during grasp completion.

Downstream Impact of Synthetic Augmentation

To establish the utility of synthetic data augmentation, Sun et al. compared models trained on Real data alone versus Real + Syn data under identical architectures:

Training Data	Mean Accuracy $\uparrow$	Mean FPR $\downarrow$	Unseen Object ISR $\uparrow$
Real	87.5%	22.8%	6/10
Real + Syn	90.0%	13.6%	7/10

---

5. Key Takeaways & Implications for Safety-Critical AI

The empirical findings reported by Sun et al. highlight several insights for AI safety researchers, red-teamers, and roboticists studying perception under real-world sensor degradation:

1. Explicit Noise Modeling Bridges the Sim-to-Real Gap: Training policies on clean, synthetic depth maps leaves systems vulnerable to physical deployment failures. Incorporating empirical sensor noise patterns—such as boundary erosion artifacts matching Time-of-Flight sensor distributions ($N_0$)—prevents policies from overfitting to ideal geometries.
2. Full-Body and Social Gaze Context Suppress Premature Triggers: Restricting perceptual tracking to hand kinematics causes frequent misinterpretations of human movement. Integrating full-body context, specifically facial gaze confidence and upper-body posture, enables perception systems to reliably distinguish handover intent from ambient human motion, reducing hardware false trigger rates from 83.3% to 6.7%.
3. Synthetic Augmentation Functions as a Variance-Reduction Mechanism: Augmenting real-world datasets with pose-guided generative video streams (PassGen-Syn) acts primarily as a variance-reduction and false-positive suppression mechanism. Training on combined multimodal pairs reduces the overall false positive rate from 22.8% to 13.6% while improving tracking stability on unseen objects.
4. Isolating Perception Gating Protects Mechanical Control Loops: Decoupling intent detection from downstream motion planning prevents perceptual uncertainty from destabilizing physical hardware execution. Implementing an independent, monotone-constrained gating threshold ($\tau = 0.80$) isolates control loops from transient sensor dropouts and line-of-sight occlusions.

Read the full paper on arXiv · PDF