What is ego-exo data capture, and why does it matter for humanoid robot training?

What ego-exo data capture means, how it differs from single-view video, and why it matters for training humanoid robots and embodied AI models.
November 18, 2026
The Firstsource team
What is ego-exo data capture, and why does it matter for humanoid robot training?

TL;DR

  • Ego-exo data capture means recording the same action from two matched perspectives at once: egocentric (first-person, from the point of view of the person, robot, or wearable performing the action) and exocentric (third-person, from an external observer's camera).
  • The term comes largely from the academic Ego-Exo4D dataset project, which built a large research dataset around this exact pairing to advance embodied AI and video understanding research.
  • It matters for humanoid robot training because a robot needs to learn both what an action looks like from its own sensors and how that same action appears to the people or cameras around it, supporting better imitation learning, safety awareness, and human-robot interaction.
  • Firstsource applies this kind of paired capture commercially, including a 50,000-hour egocentric video program built from the robot's own point of view for a robotics studio training vision-language-action models.

Most of the video used to train robots is shot from a single camera angle. A camera sits in the corner of a room, or on a tripod a few feet away, and it records a person doing something useful: chopping vegetables, tightening a bolt, folding a towel. That footage is helpful, and it is how a lot of early robot learning got done. But it only ever answers one question, which is what the action looked like from where that one camera happened to be.

Now picture the same action recorded twice at the same moment. One recording comes from the point of view of the person doing the task, a camera on their head or wrist that sees what their own eyes and hands see. The other comes from an outside camera watching the whole scene. You now have two synchronized accounts of one event, and together they describe it far more completely than either view could alone. That pairing has a name. People call it ego-exo data capture, and it is quietly becoming a real requirement for teams building humanoid robots rather than a research curiosity.

This article defines the term fully, traces where it came from, and explains why it is turning into something a serious robotics program actually has to buy or build. It keeps a careful line throughout between the academic project that popularized the idea and the commercial capture work that companies, including Firstsource, now run in production.

What "ego" and "exo" mean in this context

The two words are short for egocentric and exocentric, and they describe where a camera sits relative to the action.

Egocentric capture is first-person. The camera is mounted on, or very close to, whoever or whatever is performing the task. For a human demonstrator, that usually means a headset-mounted camera that roughly matches their line of sight, or a wrist-worn camera that stays close to the hands. For a robot, it means the camera (or cameras) built into the robot's own head or body, the feed the robot itself would actually see when it tries the task later. Egocentric video shows intent and detail up close: which object a hand reaches for first, how a grip adjusts mid-motion, what is and is not visible to the actor at any given second.

Exocentric capture is third-person. The camera sits somewhere in the room, positioned as an outside observer. It sees the whole body, the full reach of a movement, the space around the actor, and how the actor relates to objects and to other people. Exocentric video is the view that shows posture, trajectory, and context, the things that are hard to read from inside the actor's own head.

Neither view is new on its own. Plenty of datasets contain only first-person footage, and plenty contain only third-person footage. The distinct value of ego-exo is that you record both at once and tie them together tightly in time, so that frame for frame you can line up what the actor saw with what an observer saw. Done well, the two views are synchronized closely enough that a model can learn the correspondence between them.

A simple example makes the pairing concrete. Say the task is tying a knot. The egocentric view shows the fingers up close, the exact order the loops are formed, the small corrections the hands make when the rope slips. The exocentric view shows the whole posture, both hands working together, the angle of the rope against the body, and the final result held up to check. On their own, each view leaves gaps. The close-up loses the big picture; the wide shot loses the fine detail. Recorded together and time-aligned, the two views describe one event completely, which is exactly the signal a learning system benefits from.

Where the term comes from: the Ego-Exo4D research project

The ego-exo framing owes a lot to a specific academic effort called Ego-Exo4D. It is a large-scale research dataset project that pairs egocentric and exocentric recordings of skilled human activities, the kind of tasks that take real expertise to do well: cooking, playing music, bike and equipment repair, physical assembly, and similar hands-on skills. It was built across multiple research sites so the footage spans many people, settings, and conditions rather than one lab's narrow slice of the world.

The point of Ego-Exo4D was to give researchers a shared, high-quality resource for studying how skilled activity looks from both the inside and the outside at once, and to push forward work on video understanding and embodied AI. That is worth stating plainly because it is easy to get wrong: Ego-Exo4D is a research effort. It is not a commercial vendor, it is not a product you license, and it is not a Firstsource offering. No company owns it or sells it as a service.

What has happened since is that the idea the project helped name, recording matched first-person and third-person views of the same activity, started showing up in commercial data programs that are separate from the academic dataset itself. The research established the value of the pairing and gave people a clean term for it. Companies that collect training data for robotics then began running their own synchronized multi-view capture to feed real model-training programs. So when someone today talks about buying ego-exo data, they are usually talking about that commercial capture work, which borrows the concept and the vocabulary from the research but is a distinct thing with its own operational demands.

Why humanoid robots need both perspectives, not just one

A robot already has egocentric sensors. Its own cameras give it a first-person feed by default. So a fair question is why a robotics team would go to the trouble of capturing an outside view as well.

The first reason is imitation learning. A common way to teach a robot a task is to show it demonstrations of that task done correctly, and a lot of those demonstrations were originally recorded by external cameras watching a human. If a robot only ever has its own first-person feed and nothing that connects that feed to how the demonstration looked from the outside, there is a gap it cannot bridge on its own. Paired ego-exo data gives a model the correspondence it needs: this is what the demonstrator's action looked like from the outside, and this is what the same moment looked like from inside their point of view. That link is what lets first-person sensing and third-person demonstration inform each other.

The second reason is safety and social awareness. A robot moving through a shared space needs some sense of how its own movements look to the people and other agents around it. A reach that feels fine from the robot's internal view might look abrupt or unsafe from across the room. Behavior that reads as cooperative to a person depends on how it appears from the outside, not only on what the robot's own sensors register. Egocentric data alone cannot teach that, because by definition it never shows the actor how it looks to anyone else.

Humanoid robots raise the stakes on both counts. They are being built specifically to work in human spaces, homes, warehouses, hospitals, and retail floors, rather than inside the fenced industrial cages where earlier robots lived. A robot arm bolted to a factory line operates in a controlled, repetitive environment with few people nearby. A humanoid is meant to move among people, handle unfamiliar objects, and respond to messy real-world conditions. The closer a robot gets to working alongside people, the more it matters that it understands its actions from the outside as well as from within, which is exactly what the paired view provides.

How ego-exo capture connects to the sim-to-real problem

It helps to be honest about how hard robot learning still is, because that context is what makes richer training data worth the effort. The gap between how robots perform in simulation or in a lab and how they perform in a real, unstructured environment is usually called the sim-to-real gap, and it remains wide.

One concrete measure of that gap comes from the Stanford AI Index 2026, which reports that state-of-the-art robots averaged roughly a 12% real-world success rate on household tasks in 2025. That is not a typo or an outlier framing. On the ordinary, varied chores a person does without thinking, the best current systems succeed only about one time in eight. Household tasks are a hard benchmark precisely because homes are unpredictable: lighting changes, objects differ, surfaces vary, and nothing is laid out the same way twice.

Richer, multi-perspective training signal is one concrete way teams try to narrow that gap. If a model can learn from both how an action was performed and how it appeared from the outside, it has more to generalize from than a model trained on a single view. That is a reasonable argument for investing in paired capture.

It is also important not to oversell it. Ego-exo data does not, by itself, solve the sim-to-real problem. A 12% success rate does not jump to something respectable just because the footage now has two camera angles. Lighting variation, surface friction, the sheer variety of objects a robot encounters, and countless other physical factors still have to be addressed through other means. Paired capture improves the training signal; it is one input among many, not a cure. A team that treats it as a silver bullet will be disappointed, and a vendor that sells it as one should be treated with caution.

What ego-exo data collection actually requires operationally

Describing the concept is easy. Collecting ego-exo data at quality and at scale is where the real work sits, and it is worth walking through what it actually takes.

You start with two or more synchronized camera or sensor rigs, at minimum one egocentric and one exocentric, running at the same time on the same activity. Those rigs have to be time-aligned precisely. If the first-person and third-person feeds drift apart by even a fraction of a second, the correspondence between them, the whole reason to capture both, starts to break down. Tight synchronization is the foundation everything else rests on.

Spatial calibration has to hold up as people move. It is one thing to calibrate two cameras while everything sits still. It is another to keep that calibration consistent while a participant walks around a kitchen, bends down, turns away, and comes back. The system needs to track the geometric relationship between the views as the scene changes, not just once at the start.

Then there is diversity. A dataset collected from a handful of people in one room will produce a model that generalizes poorly, because it has only ever seen a narrow version of the task. Useful ego-exo data needs enough different participants, settings, objects, and scenarios that a model trained on it can handle variation it has not seen before. Getting that breadth is a recruiting and logistics problem as much as a technical one.

Quality control has to happen continuously, during capture, not weeks later when someone finally reviews the footage. Dropped frames, clock drift between rigs, a camera knocked out of position, a sensor that quietly stopped recording: these failures are common, and if they are only caught after the fact, you lose the session and have to bring participants back. Catching them in the moment is far cheaper than discovering them in review.

And capture is not the finish line. Once footage clears quality control, both views usually still need downstream labeling, annotating objects, actions, and relationships across the egocentric and exocentric streams, so the data is usable for training. The capture gets you the raw paired signal; labeling turns it into something a model can learn from. That is a second, substantial body of work layered on top of collection.

Here is how the operational picture lines up against the proof points a buyer can actually check:

Figure Value Source
Real-world task success rate for state-of-the-art robots on household tasks (2025) ~12% average Stanford AI Index 2026
Egocentric video accepted for a robotics studio's VLA training program 50,000+ hours Firstsource
Program activation time via the Gigsourcing Platform Under 48 hours Firstsource
Countries covered by Firstsource's participant recruitment network 100+ countries Firstsource
Ego-Exo4D Academic research dataset pairing egocentric and exocentric video of skilled human activities Academic research project

Why this matters for choosing a physical AI data partner

For a team deciding who should collect this data, the operational picture above has a direct implication. Running synchronized multi-view capture is a different kind of work from labeling footage that was collected by someone else. Labeling pre-collected video is a known, well-served category with many capable vendors. Standing up calibrated, time-aligned, first-person-and-third-person rigs, recruiting diverse participants, and holding synchronization quality across thousands of hours of capture is a harder and less common capability. A partner that is good at the second thing is not automatically good at the first, and vice versa.

This is where Firstsource's own work is directly relevant as evidence rather than as a claim. Firstsource, part of the RP-Sanjiv Goenka Group, runs this kind of paired, point-of-view capture commercially through its GenAI Data Services business unit. For a robotics studio training vision-language-action (VLA) models, Firstsource built a 50,000-hour egocentric video program recorded from the robot's own point of view, exactly the kind of embodied, first-person data these models need. The participant recruitment behind programs like that runs through the Gigsourcing Platform, which spans more than 100 countries and can activate a program in under 48 hours, which is what makes the diversity and scale requirements described above achievable in practice. You can read more on the physical AI solutions page and in the robotics studio VLA case study.

If you are vetting any physical AI data partner, including this one, a few concrete questions separate real capability from a pitch. Ask for a real past program at relevant scale, not a slideshow, and listen for specifics about what was collected and for whom. Ask a pointed synchronization-QA question: how do they detect and handle clock drift between rigs, and do they catch it during capture or afterward? And ask for a realistic timeline to deliver a small sample dataset, then judge whether the answer sounds like someone who has actually done it versus someone describing it in the abstract. General claims are easy. Specific answers to those three questions are harder to fake.

Firstsource's broader data annotation solutions cover the downstream labeling that paired capture still requires once footage clears quality control, and the full set of capabilities sits under the GenAI Data Services hub.

See how Firstsource captures egocentric and embodied robotics data

Want to understand how paired, point-of-view capture works in a real program? Explore Physical AI solutions.

What to expect as this space matures

Ego-exo is still an emerging term in commercial settings, and it is worth saying where it is likely headed so buyers can plan rather than guess.

Expect the terminology to become more common as humanoid robot programs move past the pilot stage and start collecting data at scale. Right now a lot of ego-exo work is early and bespoke. As more programs reach production, the vocabulary will settle and a clearer vendor field will form around it. That is the normal path for a data category. Two-dimensional image annotation and LiDAR point-cloud labeling both went through the same arc, starting as specialized, ad hoc work and ending up as well-defined services with a recognizable set of providers. Ego-exo capture is earlier on that curve, and most of the vendor field has not caught up to offering it as a named, packaged service yet.

Because that vendor field does not really exist yet, buyers are better off focusing on evaluation criteria than on shopping a list of providers. The criteria that matter are the ones the operational section laid out: synchronization accuracy, calibration rigor, and participant diversity. A partner that can speak precisely to those three things can probably deliver, whatever label they put on the service. A vendor list, by contrast, would be short and would change quickly, so anchoring a decision to it is shaky ground.

The small number of vendors with a real, delivered ego-exo program is itself useful information for a buyer today. In a mature category, the fact that a provider has done the work is table stakes and tells you little. In an unsettled one, it is a genuine signal. If only a handful of companies can point to an actual paired-capture program at scale, then being able to point to one, as Firstsource can with its 50,000-hour egocentric program, narrows the field in a way that matters. Buy on demonstrated capability and clear evaluation criteria, not on a vendor roster that has not formed.

FAQ

What is ego-exo data capture, and why does it matter for humanoid robot training?

Ego-exo data capture is the practice of recording the same action from two matched perspectives at once, egocentric (first-person) and exocentric (third-person), so a model can learn how an action looks both from the actor's own point of view and from an outside observer. It matters for humanoid robot training because robots need to understand their own sensory experience as well as how their actions appear to the people and cameras around them.

What is the Ego-Exo4D dataset?

Ego-Exo4D is an academic research dataset project that pairs egocentric and exocentric video recordings of skilled human activities to advance video and embodied AI understanding research. It is a research effort, not a commercial vendor or product, and no company owns or sells it as a service.

What is the difference between egocentric and exocentric video?

Egocentric video is recorded from the first-person point of view of the person, robot, or wearable device performing an action. Exocentric video is recorded from a third-person, external camera observing that same action from outside, giving a different vantage point on the identical event.

Why can't a robot just learn from egocentric video alone?

Egocentric video alone does not show a robot how its actions look to the people and other agents around it, which matters for tasks that involve safety awareness, social interaction, and imitation learning from demonstrators who were originally filmed from an external viewpoint. Without the outside view, a model is missing half the picture of how its behavior lands in the world.

How is ego-exo data collected commercially outside of academic research?

Vendors that run ego-exo-style capture recruit participants and calibrate synchronized first-person and third-person camera or sensor rigs to record the same activity from both views at once. Firstsource, for example, built a 50,000-hour egocentric video program for a robotics studio's VLA model training using this kind of point-of-view capture, sourced through its Gigsourcing Platform.

Recent Blogs

What is named entity recognition (NER) annotation used for in LLM training?

What is named entity recognition (NER) annotation used for in LLM training?

Technology
November 18, 2026
What are the main types of data annotation used in computer vision vs. NLP?

What are the main types of data annotation used in computer vision vs. NLP?

Technology
November 18, 2026
What are the best providers of pre-built, licensed AI training datasets?

What are the best providers of pre-built, licensed AI training datasets?

Technology
November 18, 2026
Technology