Reasoning and Planning

Scan, Materialize, Simulate

Scan, Materialize, Simulate turns RGB-D observations into a reconstructed scene, infers geometry and material properties, and runs Genesis-based simulation to optimize physically grounded robot actions.

Tool Introduction

Core parameters, trigger timing, and visual before/after demo references.

Short Explanation

Use SMS when the robot must plan through latent physical properties such as shape, contact, or material instead of relying on purely geometric state.

InputRGB-D frames or single scene image + experiment config

OutputReconstructed scene state, inferred materials, optimized action parameters, experiment JSON

Trigger TimingTriggered on demand after the required input files and configuration are prepared.

RuntimePython / Genesis / OpenAI-assisted material inference

BeforeRGB-D frames or single scene image + experiment config

Prepare the scene, image, video, sensor stream, prompt, or configuration expected by the original project.

AfterReconstructed scene state, inferred materials, optimized action parameters, experiment JSON

Read the produced visualization, prediction, map, trajectory, mask, grasp pose, or other documented artifact.

Preset Example

A quick-run style example for the documentation page.

Inputtools/scan_materialize_simulate scene observation or generated experiment scene

Promptexperiment: billiards; preset: lite

ExpectedA serialized experiment result with reconstructed geometry, inferred materials, optimized action parameters, and simulation-derived objective values.

Parameters And Output

Readable controls and the meaning of each returned artifact.

Parameter Explanation

experimentselectbilliards

Selects the experiment pipeline, such as billiards or quadrotor.

presetselectlite

Controls simulation scale, optimization loops, and rendering fidelity.

outputpath

Destination JSON path for experiment artifacts and metrics.

rgb_d_framespath

Optional RGB-D observations used by the scan and reconstruction stage.

Output Explanation

scene_representation

Reconstructed geometry or scene handle used for downstream simulation.

material_estimates

Inferred material attributes used in the physics model.

action_parameters

Optimized control or interaction parameters produced by the solver.

objective_value

Simulation-derived score used to select or compare candidate actions.

How To Use

Official resources, deployment steps, academic context, citation, and source-reported benchmark numbers.

Resources

Genesis GitHubhttps://github.com/Genesis-Embodied-AI/Genesis Code Downloadhttps://github.com/Genesis-Embodied-AI/Genesis/archive/refs/heads/main.zip Genesis Project Pagehttps://genesis-embodied-ai.github.io/Paperhttps://arxiv.org/abs/2505.14938 Genesis Benchmarkshttps://github.com/zhouxian/genesis-speed-benchmark

Deployment Notes

Run `./setup.sh sms_env` to install the SMS framework dependencies and the bundled Genesis simulator.
Use `python -m sms_framework.main` with an experiment name, preset, and output path to execute the end-to-end pipeline.
The local framework keeps lightweight fallbacks for segmentation, material inference, and scene handling so tests can run even when heavyweight assets are absent.
Use `pytest tests -q` to validate the reconstructed engineering pipeline after installation.

Relative Path Example

# Relative-path local entry for the SMS framework deployment
cd tools/scan_materialize_simulate
./setup.sh sms_env
python -m sms_framework.main --experiment billiards --preset lite --output tools/scan_materialize_simulate/outputs/billiards_result.json
python -m sms_framework.main --experiment quadrotor --preset lite --scene_id 0 --start_positions 4 --output tools/scan_materialize_simulate/outputs/quadrotor_result.json

Expected Result Shape

{
  "tool": "scan_materialize_simulate",
  "status": "ok",
  "results": [
    {
      "label": "Physically grounded scene materialization",
      "score": 0.87,
      "output": "Reconstructed scene state, inferred materials, optimized action parameters, experiment JSON"
    }
  ],
  "timing": {
    "runtime": "Official runtime notes include 1.1 s/iteration for billiards Nelder-Mead optimization, 8.2 s/iteration for quadrotor optimization on an RTX 4090, and about 3 minutes for 60-frame mapping.",
    "device": "documented in source benchmark when available"
  },
  "artifacts": {
    "visualization": "tools/scan_materialize_simulate/runs/visualization.png",
    "raw_predictions": "tools/scan_materialize_simulate/runs/predictions.json"
  }
}

Paper figure

Academic Info

Paper identity and contribution summary.

TitleScan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning

AuthorsAdd authors

VenueAdd venue

ContributionComposes scanning, material inference, and physics simulation into one planning loop so robot actions can be optimized against a physically grounded scene model.

Citation

@misc{scan_materialize_simulateYEAR,
  title={Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning},
  author={Author},
  year={YEAR},
  note={Add venue or arXiv identifier},
  url={https://arxiv.org/abs/2505.14938}
}

Benchmark

Only compact, source-reported numbers are shown here.

Dataset	Metric	Value	Runtime	Source
Billiards, 18 scenes x 30 seeds	Task objective, SMS predicted / realized	5.5 (12.7) cm / 20.5 (17.9) cm	Nelder-Mead averages 1.1 s/iteration, converges in 15-20 of 30 iterations	Official SMS paper, Table 1
Billiards, per-scene best	SMS realized vs baseline realized	5.5 (8.3) cm vs 28.2 (21.7) cm	PyBullet simulation timestep 0.0025 s	Official SMS paper, Table 1
Quadrotor landing, 4 scenes x 10 starts	Landing success rate, SMS vs visual prompting	100% / 80% / 90% / 90% vs 50% / 50% / 60% / 50%	Genesis optimization averages 8.2 s/iteration on RTX 4090	Official SMS paper
Scene reconstruction	Scan/mapping setup	60 RGB-D images; about 3 s/frame optimization; about 3 min total mapping	FR3 controller 400 Hz for billiards setup	Official SMS paper

Artifacts

Official SMS paper, Table 1, billiards/quadrotor experiment details, wrapper source, SMS framework README, bundled Genesis assets, tests, and experiment entrypoints.

Demo Images

Visual references from the original tool. Click any image to inspect the original size.