How to choose a human motion dataset
Start with the learning task
A dataset built for text-to-motion is not automatically the best source for humanoid control. A dense action-label dataset is not automatically a clean generative-motion corpus. A large body-model dataset may not include hands, objects, or language.
Write the evaluation question first: are you training a motion generator, pose estimator, tracker, controller, or world model? Is the target a human body model, animation skeleton, or humanoid robot? Do you need whole-body kinematics, hands, face, objects, scenes, contacts, or robot actions? Is the data used for training, fine-tuning, evaluation, retrieval, or simulation? Must the output support commercial deployment?
Inspect provenance
Marker-based mocap records trajectories in a calibrated capture setup. Inertial capture measures orientation through wearable sensors. Video-derived datasets estimate 3D structure from pixels. Generated datasets synthesise motion from a model. Derived datasets may crop, resample, normalise, annotate, or retarget existing motion. These categories are all useful. They have different error models.
Match the representation
SMPL, SMPL+H, SMPL-X, MMM, raw joint arrays, BVH skeletons, and robot NPZ files are not interchangeable. Conversion can change joint counts, rotations, root motion, contacts, and body shape. Document hierarchy and joint names; global versus local positions; rotation representation; root and heading convention; coordinate axes and handedness; units and scale; frame rate and timestamps; body-shape parameters; contact and confidence fields; and missing-data and interpolation rules.
Evaluate annotations and temporal granularity
Sequence captions describe an entire clip. Frame-level labels identify when an action begins and ends. Pose descriptions describe individual frames. Contact states label physical relationships. These serve different objectives. For long, multi-action motion, a single caption can hide transitions and overlapping behaviour.
Check leakage and overlap
If train and test sets derive from the same underlying AMASS sequence, performer, or source dataset, evaluation can be misleading even when filenames differ. Track source identity through every derived dataset. For model comparisons, preserve the official split unless there is a documented reason to change it.
Treat licensing as a model constraint
Public download does not mean unrestricted commercial use. Some datasets permit non-commercial research only. Some require separate registration. Some derivative repositories distribute annotations but not the underlying motion. Mixed datasets can inherit multiple licence obligations. The safest operational practice is a licence manifest at the sequence or subset level.
Best datasets by task
Text-to-motion
Start with HumanML3D when comparability with established text-to-motion work matters. Use KIT-ML when its MMM representation, annotations, or robotics context fit the project. Add Motion-X when hands, face, expressive whole-body motion, or broader mixed-source coverage is required.
Action recognition and temporal localisation
BABEL is the clearest choice in this group because it provides sequence- and frame-level action labels over AMASS motion. It is designed to represent multiple and overlapping actions rather than only one caption per clip.
Human-motion priors
AMASS remains foundational because it aggregates many optical mocap datasets in a common representation. It offers broad human motion without imposing one text-generation preprocessing pipeline. Note that the AMASS licence grants non-commercial use only.
Humanoid tracking and teleoperation
MOSAIC is the most directly packaged option in this comparison. It includes human AMASS-style files, G1-retargeted motion, and a published downstream system. AMASS also remains an important upstream source, but requires a retargeting and control pipeline.
Commercial model training and evaluation
Public research datasets often do not provide the commercial rights or support needed for deployment. Uthana is the relevant option in this set when the requirement is a negotiated commercial licence, technical delivery specification, custom enrichment, or target-embodiment mapping. The buyer should still compare a Uthana sample with internal requirements — commercial availability is not a substitute for task fit.
Ground truth, estimated motion, and pseudo-ground truth
Marker-based mocap observes tracked markers through calibrated cameras. A processing pipeline converts those trajectories into a skeleton or body model. The result is not “raw truth” — fitting and cleanup still make assumptions — but its spatial provenance differs from a monocular estimate.
Video-derived motion estimates 3D body structure from pixels. It can reach behaviours, scenes, and scale that a studio program cannot easily reproduce. Its errors depend on visibility, camera motion, depth ambiguity, motion blur, clothing, and the pose model.
Generated motion samples from a learned model rather than reconstructing a specific performer. It can target rare or missing categories, but it also reproduces the model's training distribution and failure modes.
Use the labels precisely: captured (measured through a capture system); derived (transformed from an existing motion source); estimated or pseudo-ground-truth (inferred from video or another observation); generated (synthesised by a model); and retargeted (mapped from one body representation to another). A mixed dataset should preserve these fields at the sequence level.
What public motion datasets still miss
No dataset in this comparison resolves every gap. Common limitations include limited commercial rights; body-only motion without detailed fingers or face; weak coverage of object interaction and force; few paired video and precise 3D sequences; inconsistent contact annotations; narrow performer or environment diversity; short isolated clips without long transitions; derived datasets that obscure source overlap; no target-robot representation; and no synchronised robot actions, force, torque, or tactile signals.
These are acquisition questions, not just preprocessing questions. If the missing behaviour or signal is central to the model, a custom capture or labelling program may be more efficient than forcing an existing public benchmark to serve a different purpose.
Discuss a custom dataset · Explore labeling and enrichment
A dataset procurement checklist
Before training, request or document:
- Dataset version and release date.
- Sequence-level provenance and source dataset.
- Capture, estimation, generation, and retargeting method.
- Hours, clips, frames, performers, and duplicate policy.
- Skeleton, body model, joint count, and body coverage.
- Frame rate, coordinate system, units, axes, and root convention.
- Joint position and rotation representations.
- Labels, annotator process, segmentation, contacts, and confidence.
- Object, scene, video, audio, and other paired modalities.
- Train/validation/test split and source-level leakage controls.
- QA checks, rejection criteria, and known failure modes.
- Licence for training, evaluation, commercial deployment, derivatives, and redistribution.
- Delivery format, versioning, updates, and sample access.
- Target-embodiment mapping and downstream validation scope.
Frequently asked questions
What is the largest 3D human motion dataset?
“Largest” depends on the unit and provenance. Motion-X reports 81.1K clips, 15.6 million whole-body poses, and 144.2 hours, but combines existing datasets with motion estimated from online video. AMASS's original release reports more than 40 hours of optical mocap unified from multiple sources. Compare captured hours, derived clips, frames, and annotations separately.
Is HumanML3D part of AMASS?
HumanML3D derives motion from AMASS and HumanAct12, applies standardised preprocessing, and adds natural-language descriptions. It is a distinct benchmark package, but much of its underlying motion is not independent of AMASS.
Is BABEL a separate mocap dataset?
BABEL is primarily an annotation dataset over AMASS motion. It adds sequence- and frame-level action labels to about 43 hours of AMASS sequences rather than contributing a separate 43-hour capture collection.
Which dataset is best for text-to-motion?
HumanML3D is the most established benchmark in this comparison. KIT-ML is useful for a smaller motion-language dataset with robotics lineage. Motion-X is relevant when whole-body hands, face, and expressive SMPL-X motion matter.
Which dataset is best for humanoid robots?
MOSAIC is the most directly robot-oriented package compared here because it includes Unitree G1-retargeted motion and a published tracking/teleoperation pipeline. Human motion still requires target-specific feasibility and control validation.
Can AMASS be used commercially?
The standard AMASS licence grants use for non-commercial scientific research, non-commercial education, and non-commercial artistic projects. Commercial use requires separate rights and may also depend on the underlying source datasets.
Does a public dataset allow commercial model training?
Not automatically. Public access, open-source code, a research licence, and commercial model-training rights are different. Review the dataset licence, every upstream source licence, and any restrictions on derivatives or redistribution.
What is the difference between marker-based and video-derived motion data?
Marker-based motion is measured through a calibrated capture system and then fit to a skeleton or body model. Video-derived motion estimates 3D structure from pixels. Marker-based data offers stronger spatial provenance; video-derived data offers broader behavioural and environmental scale.
