Loading video…
Cup
Anonymous submission
DexSeed
Expanding Human Hand–Object Interactions into Robot Demonstrations for Dexterous Policy Learning
01
Grasp bank
G1
Oblique side whole-hand grasp
Human demonstration
Loading video…
Water bottle
Loading video…
Apple
Loading video…
Cylinder
Loading video…
Duck
Wuji retargeting
Loading video…
Cup
Loading video…
Water bottle
Loading video…
Apple
Loading video…
Cylinder
Loading video…
Duck
G2
Top-down deep grip
Human demonstration
Loading video…
Flashlight
Loading video…
Stapler
Wuji retargeting
Loading video…
Flashlight
Loading video…
Stapler
G3
Top-down fingertip grasp
Human demonstration
Loading video…
Cube
Loading video…
Airplane
Wuji retargeting
Loading video…
Cube
Loading video…
Airplane
G4
Top-down whole-hand grasp
Human demonstration
Loading video…
Apple
Loading video…
Elephant
Wuji retargeting
Loading video…
Apple
Loading video…
Elephant
G5
Handle-threading grasp
Human demonstration
Loading video…
Teapot · 1
Loading video…
Teapot · 2
Loading video…
Mug
Wuji retargeting
Loading video…
Teapot · 1
Loading video…
Teapot · 2
Loading video…
Mug
G6
High-wrist oblique grasp
Human demonstration
Loading video…
Banana
Loading video…
Stapler
Wuji retargeting
Loading video…
Banana
Loading video…
Stapler
G7
Lateral body wrap
Human demonstration
Loading video…
Wineglass
Loading video…
Alarm clock
Wuji retargeting
Loading video…
Wineglass
Loading video…
Alarm clock
G8
Two-finger hook
Human demonstration
Loading video…
Torus
Loading video…
Eyeglasses
Wuji retargeting
Loading video…
Torus
Loading video…
Eyeglasses
02
Data generation pipeline
Matching example numbers across the trajectory, physics, and rendering rows refer to the same demonstration.
Cup G1
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Water bottle G1
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Apple G1
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Cylinder G1
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Resize · 1.25×
Loading video…
Resize · 0.8×
Loading video…
Resize · 1×
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Duck G1
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Flashlight G2
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Stapler G2
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Cube G3
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Resize · 1.25×
Loading video…
Resize · 0.8×
Loading video…
Resize · 1×
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Airplane G3
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Apple G4
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Elephant G4
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Teapot · 1 G5
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Teapot · 2 G5
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Mug G5
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Banana G6
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Stapler G6
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Wineglass G7
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Alarm clock G7
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Torus G8
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Resize · 1.25×
Loading video…
Resize · 0.8×
Loading video…
Resize · 1×
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Eyeglasses G8
Human input
Loading video…
Human input
InteractionMesh retargeting
Original, resized, and cross-geometry objects.
L = LIM + Ltip + Lpen + Lpose + Lsmooth
LIM matches local hand–object relationships; Ltip tracks fingertip targets according to the demonstrated contacts; Lpen penalizes the hand penetrating the object or itself; Lpose regularizes changes to the wrist pose and finger joints during optimization; Lsmooth discourages rapid changes in wrist and finger motion over time. Weights and normalization are incorporated into each loss term.
Loading video…
Original
Loading video…
Resize · 0.75×
Loading video…
Resize · 0.9×
Loading video…
Resize · 1.1×
Loading video…
Resize · 1.3×
Loading video…
Resize · 1.5×
Loading video…
Cross-geometry · 1
Loading video…
Cross-geometry · 2
Loading video…
Cross-geometry · 3
Whole-arm trajectory augmentation
Object positions are sampled 0.50–0.88 m from the calibrated robot base projected onto the tabletop plane, within a 65–130° sector intersected with the usable tabletop. Angles are measured from tabletop +X (along its long edge) toward +Y (away from the robot base).
The hand positions before approach and after retreat are sampled in a 17 × 17 × 14 cm box around the reference start/return position: ±8.5 cm along each horizontal axis and ±7 cm vertically.
Planned hand-motion speeds: translation 0.10–0.12 m/s, slowing to approximately 0.01 m/s near grasp and placement; rotation up to 0.60 rad/s. Individual arm joints have speed limits of approximately 0.68–1.11 rad/s. Trajectories are filtered for arm reachability and collision constraints.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based realization
The arm tracks the reference trajectory while the learned policy controls the fingers.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Robot-centric rendering
The head camera is moved by up to ±5 cm and rotated by ±5° along/about each axis relative to its calibrated pose. The wrist camera varies by ±2.5 mm and ±1° relative to its calibrated mounting pose.
Camera focal lengths, which control image magnification, are scaled by 0.90–1.10× (head) and 0.99–1.01× (wrist). The principal point—the image location of the camera’s optical axis—is shifted horizontally and vertically by ±20 px (head) or ±2 px (wrist).
Object colors: black, gray, white, ivory, green, red, blue, purple, and yellow, with one to three color regions. Textures include stripes, cross-stripes, speckles, marble, grain, and waves.
Lighting varies between warm, neutral, and cool tones, with different directions and shadow softness. Image brightness is scaled by approximately 0.81–1.23× for the head camera and 0.76–1.32× for the wrist camera. Red and blue image channels are scaled by 0.92–1.08× to vary color balance.
The rendered robot and tabletop brightness are separately scaled by 0.55–1.15× and 0.65–1.40× relative to their original appearance. Added shadows use a blending strength of 0–0.20.
Loading video…
Example 1
Loading video…
Example 2
Loading video…
Example 3
Loading video…
Example 4
Loading video…
Example 5
Loading video…
Example 6
Loading video…
Example 7
Loading video…
Example 8
Physics-based imitation: reference-arm evaluation
Four matched cases, each showing the recorded execution of four policies on the same reference trajectory.
Water bottle · G1
Loading video…
ManipTransFailure
Loading video…
DexploreFailure
Loading video…
w/o ArmFailure
Loading video…
OursSuccess
Stapler · G2
Loading video…
ManipTransFailure
Loading video…
DexploreFailure
Loading video…
w/o ArmFailure
Loading video…
OursSuccess
Apple · G4
Loading video…
ManipTransFailure
Loading video…
DexploreFailure
Loading video…
w/o ArmFailure
Loading video…
OursSuccess
Mug · G5
Loading video…
ManipTransFailure
Loading video…
DexploreFailure
Loading video…
w/o ArmFailure
Loading video…
OursSuccess
ManipTrans, Dexplore, and w/o Arm are trained with a floating hand; Ours includes arm tracking during training. All four are evaluated with the reference arm motion here.
03
Simulation VLA results
VLA policy executions across all eight grasp modes.
Loading video…
G1 · Water bottle
Language instructionPick and place the object in the red region using an oblique side whole-hand wrap with a moderately open hand.
Loading video…
G2 · Stapler
Language instructionPick and place the object in the red region using a top-down narrow-handle deep grasp.
Loading video…
G3 · Cube
Language instructionPick and place the object in the red region using a top-down spread-fingertip pinch.
Loading video…
G4 · Apple
Language instructionPick and place the object in the red region using a top-down whole-hand grasp with a moderately open hand.
Loading video…
G5 · Mug
Language instructionPick and place the object in the red region using a handle-opening threaded grasp.
Loading video…
G6 · Banana
Language instructionPick and place the object in the red region using a high-wrist oblique top-down grasp with a moderately open hand.
Loading video…
G7 · Alarm clock
Language instructionPick and place the object in the red region using an upright-body side wrap.
Loading video…
G8 · Torus
Language instructionPick and place the object in the red region using a downward two-finger hook grasp through its opening.
04
Real-world VLA results
All eight grasp modes
Loading video…
G1 · Water bottle
Language instructionPick and place the object in the red region using an oblique side whole-hand wrap with a moderately open hand.
Loading video…
G2 · Stapler
Language instructionPick and place the object in the red region using a top-down narrow-handle deep grasp.
Loading video…
G3 · Cube
Language instructionPick and place the object in the red region using a top-down spread-fingertip pinch.
Loading video…
G4 · Apple
Language instructionPick and place the object in the red region using a top-down whole-hand grasp with a moderately open hand.
Loading video…
G5 · Mug
Language instructionPick and place the object in the red region using a handle-opening threaded grasp.
Loading video…
G6 · Bread model
Language instructionPick and place the object in the red region using a high-wrist oblique top-down grasp with a moderately open hand.
Loading video…
G7 · Alarm clock
Language instructionPick and place the object in the red region using an upright-body side wrap.
Loading video…
G8 · Torus
Language instructionPick and place the object in the red region using a downward two-finger hook grasp through its opening.
Different starting and placement locations
Loading video…
Apple · 01G4
Loading video…
Apple · 02G4
Loading video…
Apple · 03G4
Loading video…
Apple · 04G4
Loading video…
Apple · 05G4
Loading video…
Apple · 06G4
Loading video…
Apple · 07G4
Loading video…
Apple · 08G4
Loading video…
Apple · 09G4
One object, two grasp modes
Loading video…
G1 · oblique side whole-hand
Language instructionPick and place the object in the red region using an oblique side whole-hand wrap with a moderately open hand.
Loading video…
G4 · top-down whole-hand
Language instructionPick and place the object in the red region using a top-down whole-hand grasp with a moderately open hand.
Loading video…
G1 · oblique side whole-hand
Loading video…
G4 · top-down whole-hand
One grasp mode, different objects
Loading video…
AppleG1
Loading video…
Water bottleG1
Loading video…
DuckG1
More results
Loading video…
Cylinder · G1
Loading video…
Bread model · G6
Loading video…
Duck · G1
Loading video…
Eyeglasses · G8
Loading video…
Torus · G8
Loading video…
Yellow pepper · G4
Failure case
The mug handle slips along the fingers during execution, and the placement is not completed.
Loading video…
Mug · G5Handle slip during execution