Omni-modal humanoid motion

OmniHM-ZeroScaling Humanoid Motion Foundation Models with Large-Scale Human Videos

Watch full demoarXivPaper Coming soonDataset Hugging Face ↗

Code GitHub ↗

Explore the project

15MHuman motions

5MExecutable motions

10K+Hours of behavior

1.08B+Motion frames

4Input modalities

01 / Full film

See OmniHM-Zero in motion.

A five-minute real-robot film spanning live and Internet video, spoken instructions, music, and text prompts.

Full demo · 05:16Watch on YouTube

02 / Why OmniHM-Zero

Human behavior, translated for humanoids.

Motion capture is precise but difficult to scale. Internet video is diverse, yet reconstructed motion can be noisy, physically invalid, or impossible for a humanoid to execute. OmniHM-Zero bridges that gap.

Web-scale data engine

Recover, repair, retarget, and verify motion from raw video.

Omni-modal supervision

Text, RGB video, speech, and music share one motion space.

Closed-loop generation

Replan from the latest condition and robot state in real time.

OmniHM-Zero overview showing multimodal inputs, continuous humanoid behavior, dataset scale, and model scaling — OmniHM-ZeroInterleaved multimodal intent becomes continuous whole-body behavior.

03 / Dataset

Human behavior at humanoid scale.

The OmniHM dataset is built from raw Internet video. Its data engine filters and annotates clips, reconstructs human motion, repairs physical artifacts, retargets behavior, and verifies execution before training.

297Fine-grained motion types

16Scene groups

6Physical checks

4Tracking policies

Data engine

From raw video to executable motion

A five-stage data engine starts from raw Internet video, curates coherent clips, filters motion and visual quality, aligns multimodal semantics, recovers and retargets human motion, and finally refines it through closed-loop execution. Each stage narrows the gap between visually plausible human behavior and motion a humanoid can reliably track.

01Clip generationCoherent clips from segmented and filtered web video
02Quality filteringHuman-motion and visual-quality screening
03Semantic annotationVideo, text, speech, and music alignment
04Motion recoveryHuman reconstruction, repair, and retargeting
05Physical refinementClosed-loop optimization and execution verification

Five-stage OmniHM data pipeline from web video curation to physical humanoid execution verification

Inside one OmniHM record

From a human video to executable humanoid data.

Each showcase example pairs the source clip with recovered human motion, a retargeted humanoid reference, and a tracking rollout. Together, the eight records illustrate diverse motions, scenes, and objects.

01Original video02SMPL rendering03G1 retargeted04G1 tracking

Duration5.06 s

Source FPS16.0

Source audioNot detected

ObjectsNone tagged

Subjects1

Motion annotation

What the subject does

A dynamic hip-hop phrase moves from arms overhead into a deep, wide squat with the hands near the floor, then rises through a lateral weight shift and side step.

Scene context

Where the motion happens

An indoor dance studio with a light-green floor, white walls, fluorescent grid lighting, and a ballet barre.

Record provenance

Sample ID: 00056286
Sample name: 00056286_000001_009909_0_0
Motion type: dance_hip_hop

Estimated subject annotations

Count: 1
Age estimate: 22
Height estimate: 160 cm
Weight estimate: 55 kg
Gender label: female
Mood label: focused

Dense motion annotation

Time-coded, body-part captions

1 time segment

0.0-5.0sSegment 01

Whole body

The person performs a dynamic hip-hop dance sequence, starting with arms raised overhead, then dropping into a low squat with hands near the floor, and finally rising while shifting weight and stepping slightly to the side. The motion is fluid and rhythmic, characteristic of hip-hop choreography.

Upper limb

Arms start raised above the head, then lower to rest near the knees and floor during the squat, and finally move to the sides as the person rises. The upper limbs show controlled, expressive movements throughout.

Lower limb

Legs begin in a wide stance, then bend deeply into a squat with knees spread wide, and finally extend slightly as the person rises and shifts weight to the side. The lower limbs demonstrate strength and flexibility.

Hands

Hands start raised, then move downward to touch or hover near the floor during the squat, and finally move to the sides as the person rises. The hands are used for balance and expressive gesture.

Face

Face is visible and directed toward the camera for most of the clip, showing focused expression with slight head tilts and movements in sync with the dance. The person’s gaze remains engaged with the viewer.

Duration9.37 s

Source FPS30.0

Source audioPresent

Objectsbag

Subjects1

Motion annotation

What the subject does

The subject walks across the yard carrying a bundle of mulch, then bends through the hips and knees to place and spread it beside a bush.

Scene context

Where the motion happens

A sunny residential yard with tilled soil, a pond, nearby houses and trees, and an orange wheelbarrow.

Record provenance

Sample ID: 00022816
Sample name: rP1Y8adVG3k_00005_0_278
Motion type: daily_gardening

Estimated subject annotations

Count: 1
Age estimate: 35
Height estimate: 175 cm
Weight estimate: 75 kg
Gender label: male
Mood label: focused

Dense motion annotation

Time-coded, body-part captions

1 time segment

0.0-9.0sSegment 01

Whole body

The person walks from the left side of the frame toward the right, carrying a large bundle of dried leaves or mulch. They then bend forward at the waist to deposit the material onto the ground near a bush, maintaining a steady posture while distributing the material with their hands.

Upper limb

The person’s arms are initially extended forward while carrying the bundle, then lower as they bend to place the material on the ground. Their upper body leans forward during the depositing motion, with arms moving in a controlled downward arc.

Lower limb

The person walks with a steady gait, stepping forward with alternating legs. Upon reaching the target area, they bend at the knees and hips to lower their torso, keeping their feet planted and slightly apart for stability.

Hands

The hands grip the bundle firmly while walking, then release and spread the material onto the ground in a sweeping, downward motion as the person bends over.

Face

The person’s face is not clearly visible due to distance and angle, but their head is oriented downward during the bending and depositing phase, suggesting focused attention on the task.

Duration3.34 s

Source FPS30.0

Source audioPresent

Objectspot

Subjects1

Motion annotation

What the subject does

The subject bends over an outdoor fire, lowers a large metal pot onto a brick-supported stove with both hands, then straightens and steps away.

Scene context

Where the motion happens

A rural outdoor cooking area with a small wood fire, brick supports, tall grass, palm trees, and an overcast sky.

Record provenance

Sample ID: 00009486
Sample name: FE7ycQZfve0_00013_0_99
Motion type: daily_cooking

Estimated subject annotations

Count: 1
Age estimate: 28
Height estimate: 172 cm
Weight estimate: 68 kg
Gender label: male
Mood label: focused

Dense motion annotation

Time-coded, body-part captions

2 time segments

0.0-2.0sSegment 01

Whole body

The person is bent forward at the waist, leaning over a small outdoor fire setup, carefully placing a large metal pot onto a brick-supported stand above the flames.

Upper limb

Both arms are extended forward and downward; the right hand grips the pot's rim while the left hand supports the pot's side, coordinating to lower it steadily onto the stand.

Lower limb

Legs are slightly bent at the knees to maintain balance while leaning; feet are planted firmly on the ground, with the right foot slightly ahead for stability.

Hands

Right hand grips the pot's rim with fingers curled over the edge; left hand supports the pot’s side with palm flat against the surface, both hands working in unison to position the pot.

Face

Face is angled downward, eyes focused intently on the pot and fire, showing concentration and care in placement.

2.0-3.4sSegment 02

Whole body

The person straightens up and steps back from the fire, turning slightly to the left, moving away from the cooking setup while keeping eyes on the pot.

Upper limb

Arms relax and swing naturally at the sides as the person steps back; the right arm swings slightly forward as the left leg moves back.

Lower limb

Right leg steps back first, followed by the left, shifting weight smoothly; body rotates slightly to the left as the person moves away from the fire.

Hands

Hands are open and relaxed, hanging loosely at the sides, no longer engaged with the pot.

Face

Face remains oriented toward the pot, with a neutral to slightly satisfied expression, indicating the task is complete for now.

Duration7.38 s

Source FPS30.0

Source audioPresent

Objectsdoor, sponge

Subjects1

Motion annotation

What the subject does

The subject bends to scrub the lower door panels with a sponge, then rises and reaches upward to clean the higher sections while using the other hand for support.

Scene context

Where the motion happens

A bright home kitchen with a white interior door, dark wooden cabinets, granite counters, and countertop appliances.

Record provenance

Sample ID: 00008857
Sample name: JhBisU-AZ4k_00013_241_460
Motion type: daily_cleaning

Estimated subject annotations

Count: 1
Age estimate: 28
Height estimate: 165 cm
Weight estimate: 58 kg
Gender label: female
Mood label: focused

Dense motion annotation

Time-coded, body-part captions

2 time segments

0.0-5.0sSegment 01

Whole body

The person is bent forward at the waist, leaning into a white door while scrubbing its surface with a green sponge. Their body remains mostly stationary except for the arm motion, with slight shifts in weight to maintain balance.

Upper limb

The right arm is extended downward, moving the sponge in circular and up-and-down motions across the door. The left arm is bent and pressed against the door for support, occasionally adjusting position to stabilize the body.

Lower limb

The legs are slightly bent at the knees, maintaining a stable stance. The feet remain planted on the floor, with minor weight shifts from one leg to the other as the person adjusts their posture during cleaning.

Hands

The right hand grips the green sponge firmly, applying pressure and moving it across the door surface. The left hand is flat against the door, providing leverage and balance during the scrubbing motion.

Face

The face is angled downward toward the door, focused on the cleaning task. The expression appears concentrated, with the head tilted slightly to the side to maintain visual contact with the area being cleaned.

5.0-7.1sSegment 02

Whole body

The person straightens up from the bent position, turning slightly to the side while continuing to wipe the upper portion of the door. Their body rotates to reach higher areas, maintaining a steady posture.

Upper limb

The right arm extends upward, reaching the top of the door with the sponge in hand, making sweeping motions. The left arm is raised slightly for balance and then lowered as the person adjusts their stance.

Lower limb

The legs straighten as the person rises, shifting weight slightly to accommodate the upward reach. The feet remain grounded, with a subtle pivot to allow the upper body to rotate and access the higher part of the door.

Hands

The right hand continues to hold the sponge, now moving in broad, sweeping motions across the upper door panel. The left hand briefly lifts for balance and then relaxes at the side as the cleaning motion continues.

Face

The face tilts upward, following the motion of the hand as the person cleans the higher section of the door. The expression remains focused, with eyes tracking the cleaning area.

Duration3.72 s

Source FPS59.9

Source audioPresent

ObjectsNone tagged

Subjects1

Motion annotation

What the subject does

The subject swings both arms through a wide arc and pivots counter-clockwise on slightly bent legs, building momentum for a jump or turn toward a crash mat.

Scene context

Where the motion happens

A bright backyard with grass, a trampoline, blue crash mat, clothesline, ladder, brick house, and hedge; text and an emoji are overlaid on the source clip.

Record provenance

Sample ID: 00023179
Sample name: wsO6rOoi4DY_00014_0_222
Motion type: extreme_parkour

Estimated subject annotations

Count: 1
Age estimate: 15
Height estimate: 175 cm
Weight estimate: 60 kg
Gender label: male
Mood label: playful

Dense motion annotation

Time-coded, body-part captions

1 time segment

0.0-3.5sSegment 01

Whole body

The person stands upright on a grassy backyard, then initiates a rotational movement by swinging arms and pivoting on the feet, preparing for a jump or turn toward a blue mat in front of them.

Upper limb

Arms begin at the sides, then swing upward and outward in a wide arc to generate momentum, with the right arm leading the motion as the body rotates counter-clockwise.

Lower limb

Feet remain planted initially, then pivot slightly on the balls of the feet to assist the upper body’s rotation; knees are slightly bent to absorb motion and prepare for takeoff.

Hands

Hands are open and relaxed, moving in sync with the arms’ arc, fingers extended, aiding balance and momentum without gripping anything.

Face

Face is turned forward initially, then rotates with the body; expression appears focused and slightly anticipatory, with mouth slightly open as if preparing for exertion.

Duration5.06 s

Source FPS16.0

Source audioNot detected

Objectsyoga mat

Subjects1

Motion annotation

What the subject does

A smooth, controlled flow alternates between a deep squat and a forward fold while both palms remain planted on the yoga mat.

Scene context

Where the motion happens

A bright minimalist room with tiled flooring, sheer white curtains, soft natural light, and a yoga mat.

Record provenance

Sample ID: 00023395
Sample name: 00023395_000000_014644_1_0
Motion type: fitness_hatha_yoga

Estimated subject annotations

Count: 1
Age estimate: 28
Height estimate: 165 cm
Weight estimate: 55 kg
Gender label: female
Mood label: focused

Dense motion annotation

Time-coded, body-part captions

1 time segment

0.0-5.0sSegment 01

Whole body

person performs a flowing movement between a squat and a forward fold, maintaining a straight back and engaged core; motion is smooth and controlled, typical of Hatha yoga

Upper limb

arms remain extended downward with palms flat on the mat throughout; slight forward movement of the upper body as the person transitions between positions

Lower limb

legs transition from bent in a squat to straight in a forward fold; feet remain flat on the mat, knees slightly bent during the squat phase

Hands

hands are placed flat on the mat in front of the feet, fingers spread for stability; no significant hand movement observed

Face

face is turned slightly to the right, looking downward toward the mat; expression appears focused and calm

Duration17.33 s

Source FPS30.0

Source audioPresent

ObjectsNone tagged

Subjects1

Motion annotation

What the subject does

A fluid traditional dance combines sweeping arm arcs, delicate steps, turns, leg lifts, balanced poses, and expressive hand gestures.

Scene context

Where the motion happens

A softly front-lit stage with a light floor and dark curtain; vertical Chinese text appears at the right of the source frame.

Record provenance

Sample ID: 00007340
Sample name: 00007340_YouTube-Dance_movies_Fb2we1uDTtc_0000
Motion type: dance_chinese_classical

Estimated subject annotations

Count: 1
Age estimate: 25
Height estimate: 168 cm
Weight estimate: 52 kg
Gender label: female
Mood label: calm

Dense motion annotation

Time-coded, body-part captions

1 time segment

0.0-17.0sSegment 01

Whole body

whole body visible; performer executes a fluid, graceful traditional dance sequence with controlled movements and expressive posture

Upper limb

upper limbs move with elegance and precision, extending and curving in flowing arcs, often in coordination with the lower body and facial expression

Lower limb

lower limbs perform delicate steps, turns, and poses, including side steps, leg lifts, and balanced stances, contributing to the overall fluidity of the dance

Hands

hands are used expressively, with soft, controlled gestures, often forming delicate shapes and moving in synchronized patterns with the arms

Face

face is visible and expressive, with a gentle smile and focused gaze, often directed toward the audience or following the direction of movement

Duration3.44 s

Source FPS25.0

Source audioPresent

Objectsstilts

Subjects1

Motion annotation

What the subject does

The performer balances horizontally across two tall supports, lowers into an upright stance, then opens and raises both arms overhead while maintaining control.

Scene context

Where the motion happens

A dim stage with blue-lit vertical panels, atmospheric haze, shadowed audience seating, and location text overlaid on the source footage.

Record provenance

Sample ID: 00021891
Sample name: CTL03OQi7kM_00006_0_85
Motion type: performance_acrobatics

Estimated subject annotations

Count: 1
Age estimate: 28
Height estimate: 168 cm
Weight estimate: 58 kg
Gender label: female
Mood label: focused

Dense motion annotation

Time-coded, body-part captions

3 time segments

0.0-1.0sSegment 01

Whole body

The person is balancing horizontally on two vertical poles, body extended in a plank-like position with one leg stretched back and arms gripping the poles for support.

Upper limb

Both arms are bent at the elbows, gripping the poles tightly to maintain balance and support the body’s weight.

Lower limb

The right leg is extended straight back, parallel to the ground, while the left leg is bent slightly under the body for stability.

Hands

Hands are firmly gripping the tops of the poles, fingers wrapped around for maximum control and stability.

Face

Face is focused downward, showing concentration and physical exertion.

1.0-2.0sSegment 02

Whole body

The person lowers their body from the horizontal position, bending at the waist and knees, transitioning to a standing position on the platform between the poles.

Upper limb

Arms gradually straighten as the body lowers, then release the poles and move outward to the sides for balance.

Lower limb

Legs bend at the knees to absorb the descent, then straighten as the person stands upright on the platform.

Hands

Hands release the poles and open outward, palms facing forward, as the person regains balance.

Face

Face lifts slightly, expression shifting from exertion to calm focus as the transition completes.

2.0-3.5sSegment 03

Whole body

The person stands upright on the platform, arms extending outward and then raising overhead in a graceful, controlled motion, maintaining balance and poise.

Upper limb

Arms extend laterally to shoulder height, then smoothly raise overhead, palms facing inward, demonstrating control and extension.

Lower limb

Legs remain straight and stable, feet planted firmly on the platform, supporting the upright posture.

Hands

Hands open and extend outward, then raise overhead with fingers gently spread, maintaining a poised and expressive gesture.

Face

Face is calm and composed, eyes looking forward with a serene expression, conveying confidence and grace.

Dataset profile

Scale is visible in the distribution.

Four views summarize temporal scale, behavior taxonomy, scene coverage, and the relationship between motions and their contexts.

Public release

Explore OmniHM-5M.

The public OmniHM-5M release is available on Hugging Face. The examples above illustrate the source-to-execution data structure independently of any dataset viewer row.

Open dataset on Hugging Face

04 / Method

One model, many ways to express intent.

A frozen omni-modal context encoder conditions a shared flow-based action expert. The latest robot state keeps generation grounded in what the hardware is actually doing.

OmniHM-Zero architecture with multimodal context encoder, flow-based motion transformer, robot motion representation, and tracking controller

Shared prior

Task-mixed flow matching

Every modality trains one reusable motion prior instead of an isolated policy.

Responsive control

Multimodal real-time chunking

Overlapping predictions enable continuous replanning without abrupt transitions.

Physical grounding

State-conditioned generation

Each chunk uses the newest robot state before a high-frequency controller tracks it.

05 / Capability showcase

Five ways to express intent. One shared motion model.

Explore a curated set of real-robot demonstrations across live video, Internet video, speech, music, and text.

Video / Live

Real-robot demonstrations

Follow a performer as the motion unfolds.

OmniHM-Zero uses the current video stream and the robot’s latest state to continuously replan executable whole-body motion.

Real-time video input · 01

Real-time video input · 02

Real-time video input · 03

Real-time video input · 04

Real-time video input · 05

Real-time video input · 06

Video / In the wild

Real-robot demonstrations

Bring human motion from the Internet onto the robot.

In-the-wild references provide locomotion, gesture, rhythm, and direction changes that the model translates into continuous humanoid behavior.

Internet video input · 01

Internet video input · 02

Internet video input · 03

Internet video input · 04

Internet video input · 05

Internet video input · 06

Audio / Spoken instruction

Real-robot demonstrations

Say what to do—and how to do it.

Spoken instructions specify actions, body side, repetition, order, and duration, from left-leg lunges to alternating waves.

Speech input · 01“Open your arms wide then cross them in front of your chest”

Speech input · 02“Do ten left-leg lunges”

Speech input · 03“Salute with your left hand”

Speech input · 04“Strike a few bodybuilding poses”

Speech input · 05“Alternate waving your left and right hands up to your head”

Speech input · 06“Wave your left hand for seven seconds”

Audio / Music

Real-robot demonstrations

Let sound shape the movement.

Music conditions the timing and character of coordinated footwork, balance shifts, turns, and upper-body motion.

Music input · 01

Music input · 02

Music input · 03

Music input · 04

Music input · 05

Music input · 06

Language / Text

Real-robot demonstrations

Turn a prompt into a whole-body behavior.

Natural-language prompts can describe an action, a path, or a style—from counted squats and counterclockwise walking to salsa and zombie motion.

Text input · 01“Dance some hip-hop”

Text input · 02“Do three squats”

Text input · 03“Perform a salsa dance”

Text input · 04“Walk counterclockwise”

Text input · 05“Walk forward and wave”

Text input · 06“Act like a zombie”

06 / Scaling study

Scaling data, models, and execution together.

The study varies data from 1% to 100% of OmniHM and the action expert from 95.6M to 1.30B parameters under a shared execution protocol.

Dataset fraction

1%3%10%30%100%

Action expert

95.6M277M717M1.30B

Quantitative tables in the current technical report are still being finalized; no unreleased accuracy claims are shown here.

07 / Citation

Build on OmniHM-Zero.

Paper and model links will be added when the technical report is released. Code and the public dataset are available now.

@article{anonymous2026omnihmzero,
  title   = {OmniHM-Zero: Scaling Humanoid Motion Foundation Models with
             Large-Scale Human Videos},
  author  = {Anonymous Authors},
  journal = {Technical Report},
  year    = {2026}
}

OmniHM-ZeroScaling Humanoid Motion Foundation Models with Large-Scale Human Videos

See OmniHM-Zero in motion.

Human behavior, translated for humanoids.

Web-scale data engine

Omni-modal supervision

Closed-loop generation

Human behavior at humanoid scale.

From raw video to executable motion

From a human video to executable humanoid data.

Hip-hop dance

Gardening

Cooking with a pot

Household cleaning

Parkour preparation

Hatha yoga

Chinese classical dance

Acrobatics

Scale is visible in the distribution.

Explore OmniHM-5M.

One model, many ways to express intent.

Task-mixed flow matching

Multimodal real-time chunking

State-conditioned generation

Five ways to express intent. One shared motion model.

Follow a performer as the motion unfolds.

Bring human motion from the Internet onto the robot.

Say what to do—and how to do it.

Let sound shape the movement.

Turn a prompt into a whole-body behavior.

Scaling data, models, and execution together.

Build on OmniHM-Zero.