When pixels learn space.
Three research ideas, one shared pattern: preserve what can be seen and checked, and let richer structure emerge from there.
SCROLL TO ENTER THE NOTE ↓
The premise
For years, many intelligent systems learned through translations: pixels became labels, movement became coordinates, and scenes became handcrafted representations.
These three papers make a different bet. Train and evaluate closer to the space humans can actually inspect—the rendered image, the driving video, the keypoints on screen. Each removes a lossy middleman. Together, they suggest a broader design principle: fewer unnecessary translations means fewer places for meaning to leak.
How to read thisEach chapter moves from a source paper to an authored explanation to an interaction you can inspect. The diagrams explain the ideas; they are not model output.
Pixels become structure.
One model learns both 3D generation and reconstruction directly through images, without forcing every task through a separate representation.
PixWorld treats views of a scene as a shared visual language. Give it one or more images and a target camera; it learns to predict what that camera should see.
The important shift is not just technical. Generation and reconstruction stop being separate worlds. They become different requests made of the same underlying visual system.
The same points can read as a flat image or a navigable surface. The evidence stays visual; the structure becomes spatial.
Motion keeps its context.
A character-animation system learns appearance, motion, interaction, and occlusion together from a driving video.
SCAIL-2 avoids splitting a performance into too many isolated ingredients. Instead of separately extracting motion, identity, pose, and camera information, it conditions on the video in context.
That matters when the details are the point: how hands meet an object, how one body passes in front of another, or how a prop changes the shape of a movement.
Switch the layer being emphasized. The result works because the performance remains connected to the full scene.
Flat motion gains depth.
A 3D human-motion model learns from accurate 2D poses in ordinary video—then checks its predictions by projecting them back onto the screen.
3D motion datasets usually require specialized capture. VideoMDM uses 2D keypoints people can obtain from everyday video, together with rough depth estimates from an existing model.
The system learns in 3D but is graded in 2D: project the predicted body back through the camera and compare it with the screen evidence you can verify.
The dashed figure is the camera evidence. Rotate the view to reveal the 3D hypothesis behind the same movement.
The common thread
Skip the lossy middleman.
All three systems evaluate as close as possible to the thing we can check: rendered pixels, driving video, or keypoints on screen. Less translation between representations means fewer places for meaning to disappear.
That lesson travels beyond research. Creative systems, operating systems, and personal knowledge systems all lose something at each hand-off. Design checkpoints close to the real work, preserve context, and let people see why the system reached its result.
Receipts
The three papers.
- 01PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Space (opens in a new tab)ARXIV ↗
- 02SCAIL-2: Unifying Controlled Character Animation with End-to-end In-Context Conditioning (opens in a new tab)ARXIV ↗
- 03VideoMDM: Towards 3D Human Motion Generation From 2D Supervision (opens in a new tab)ARXIV ↗
The interactive diagrams are authored explanations of each paper’s central idea, not model output. For technical detail and results, use the original papers.