3dgs-mcp-renderer

3DGS MCP Renderer — Agent-3DGS Interaction via MCP Protocol

Prototype specification for integrating MCP (Model Context Protocol) with 3DGS rendering pipelines, enabling AI Agents to directly manipulate Three.js/3DGS rendering parameters and achieve voice-driven 3D scene reconstruction.

Architecture

┌─────────────┐     ┌─────────────┐     ┌──────────────────┐     ┌──────────────────┐
│ Voice/Text  │────▶│   Agent     │────▶│  MCP Server      │────▶│  3DGS Renderer   │
│ (Whisper/   │     │ (Claude/    │     │  (Node.js/       │     │  (Three.js/      │
│  Prompt)    │     │  TeleClaw)  │     │   Python)        │     │   WebGPU/HiGS/   │
│             │◀────│             │◀────│                  │◀────│   DDF-GS)        │
└─────────────┘     └─────────────┘     └──────────────────┘     └──────────────────┘
                        │                      │                       │
                        │  Tool calls          │  WebSocket/HTTP       │  WebGL/WebGPU/
                        │  (MCP protocol)       │  transport            │  HiGS/DDF-GS

Spec-First Sculpting Pipeline (v0.9.0)

Design inspiration: img2threejs (GitHub: img2threejs/img2threejs) — open-source AI Skill that converts a single image into an interactive Three.js 3D model via a stage-gated sculpting pipeline. We borrow two core principles: (1) spec-first — define quality criteria and component hierarchy before any rendering; (2) stage-gated sculpting — progressive refinement with acceptance checks at each stage.

Why Spec-First for MCP Rendering?

The original MCP pipeline was reactive: user issues a voice command → agent maps to a tool → render → verify. This works for single-step edits but fails for complex scene construction because:

• No upfront quality criteria → agent cannot self-assess before rendering
• No stage gates → errors compound across steps (bad camera → bad selection → bad edit)
• No component hierarchy → edits are flat, no part-level control

The fix: Introduce a define_scene_spec tool that runs before any sculpting/editing tools. This produces a machine-readable Object Spec that subsequent tools reference as acceptance criteria.

The 6-Stage Sculpting Pipeline

┌─────────────────────────────────────────────────────────────┐
│                    SPEC-FIRST SCULPTING                      │
│                                                             │
│  ┌──────────────┐                                           │
│  │ define_scene │  ← Object Spec: component hierarchy,      │
│  │ _spec        │    material system, quality criteria      │
│  └──────┬───────┘                                           │
│         │                                                   │
│         ▼                                                   │
│  Stage 1: blockout    → Bounding boxes, camera framing     │
│         │  gate: bbox coverage ≥ spec.target_coverage?      │
│         ▼                                                   │
│  Stage 2: structural  → Part decomposition, hierarchy      │
│         │  gate: part count & nesting matches spec?         │
│         ▼                                                   │
│  Stage 3: form        → Gaussian density/scale/rotation     │
│         │  gate: PSNR estimate ≥ spec.min_psnr?             │
│         ▼                                                   │
│  Stage 4: material    → PBR/SH assignment per part         │
│         │  gate: material count per part matches spec?      │
│         ▼                                                   │
│  Stage 5: surface     → Normal consistency, thin structures │
│         │  gate: normal consistency score ≥ spec.threshold? │
│         ▼                                                   │
│  Stage 6: lighting    → Environment, shadows, AO           │
│            gate: render quality score ≥ spec.target_score?  │
└─────────────────────────────────────────────────────────────┘

Each stage is an MCP tool call. The agent renders a frame after each stage, evaluates against the gate, and either advances or retries. This mirrors img2threejs's blockout → structural → form → material → surface → lighting flow.

Gate Evaluation Protocol

For each stage gate, the agent follows this protocol:

1. Execute stage tool (e.g., sculpt_form with parameters)
2. Call render_frame() to get current visual state
3. Call query_scene(query_type="stats") to get quantitative metrics
4. Compare metrics against spec gate criteria
5. If pass → advance to next stage
6. If fail → adjust parameters and retry (max 3 attempts)
7. If 3 failures → report to user with diagnostic info

Voice-Driven Sculpting Example

Loaded on demand — See mcp-tools-spec.md for the full voice-driven sculpting example (desk scene with 8-step agent pipeline).

Code-First Rendering Philosophy (v0.9.0)

Design inspiration: img2threejs outputs pure Three.js code (not GLB/OBJ/PLY), making every model fully editable, version-controllable, and lightweight. We adopt this philosophy for 3DGS scene export.

Traditional 3DGS Export vs Code-First Export

Hybrid: Procedural Code + 3DGS Splatting

The key insight: not everything needs to be Gaussians. For a desk scene:

• Desk surface → procedural BoxGeometry in code (simple, editable, lightweight)
• Monitor screen texture → procedural MeshStandardMaterial (or 3DGS if view-dependent)
• Complex organic objects → 3DGS splatting data (where procedural code can't compete)

Loaded on demand — See code-first-examples.md for hybrid export code examples.

When to Use Code-First vs Pure Splat

SLAT Connection

The code-first approach connects to SLAT (see ../../references/slat-unified-representation.md): the structured latent's voxel grid naturally maps to a procedural geometry skeleton, while the per-voxel features decode to 3DGS splatting for complex regions. SLAT encode → hierarchical decode: simple voxels → procedural code, complex voxels → 3DGS splats.

SLAT Latent Editing (v1.0.0)

Theoretical basis: SLAT (Structured Latent Aggregation Transform) — see ../../references/slat-unified-representation.md. A scene is encoded into a compact structured latent (a voxel grid over the scene, each voxel aggregating local Gaussian features), edited in latent space, then re-decoded back to a Gaussian set. This lets the agent manipulate entire semantic regions with a single operation, independent of per-Gaussian IDs.

Encoding: Scene → Structured Latent

encode_scene_slatent voxelizes the active scene into a regular grid (voxel_size, default 1.0), assigning each Gaussian to a voxel by position. Each voxel stores an aggregated feature vector (mean position, mean scale, mean color, mean opacity, size, plus optional weighted semantic/part labels). The result is a slat_id referencing an in-memory snapshot with an encode_loss (reconstruction RMSE), letting the agent judge fidelity before editing.

Editing in Latent Space

edit_scene_latent applies a LatentEditOp to voxels matched by a LatentSelector (by voxel ids, a spatial box, or a part name — substring, case-insensitive). Seven operations are supported:

Schema vs core naming: the MCP JSON schema uses snake_case (angle_deg); the internal LatentEditOp uses camelCase (angleDeg). Handlers convert at the boundary. Library/test callers use camelCase directly.

Safety gate: edit_scene_latent computes affected_gaussians; if this exceeds 10% of the scene, the edit is rejected unless confirm=true. This reuses the project-wide 10% safety rule.

Apply to scene: with apply_to_scene=true (default) the edit is re-decoded and broadcast to the renderer via modify_gaussians; with false it only updates the in-memory snapshot, so the agent can preview/cancel before committing.

Decoding: Latent → Scene

Decoding rebuilds the Gaussian set: matched voxels are re-instantiated from edited features, untouched voxels keep their original Gaussians. delete removes the affected Gaussians entirely.

Voice-Driven SLAT Example

Loaded on demand — See mcp-tools-spec.md for full SLAT voice examples ("encode the scene", "move the cluster left", "scale the group up", etc.).

Cross-Scene Latent Transfer & Interpolation (v1.1.0)

v1.1 extends SLAT beyond a single scene. A latent edit computed on one scene (source) can now be transferred to another scene (target), or the two scenes can be interpolated in latent space. Both operations rely on a spatial correspondence built over the voxel grids.

Correspondence: Voxel Grid Matching

Both operations build a voxel grid over the source scene (cell size = match_radius) via buildVoxelGrid, then for each target voxel find the nearest source voxel within match_radius (nearestVoxel, 3×3×3 neighborhood search). The resulting pairs carry the relative changes across scenes.

Transferring a Latent Edit

transfer_scene_edit re-applies a LatentEditOp from source to target as a relative change:

• match_radius (default 1.0) bounds the spatial correspondence.
• strength (0–1) controls how strongly the source change is applied; 0 applies nothing, 1 applies fully.
• Safety gate: if the matched fraction exceeds 10% of the target scene, confirm=true is required (same project-wide 10% rule).
• apply_to_scene (default true) re-decodes and broadcasts via modify_gaussians; false only updates the in-memory snapshot for preview.

interpolate_scene_latent

interpolate_scene_latent blends the target scene toward the source in latent space:

• t (0–1): 0 = target unchanged, 1 = fully source. Position, color, and opacity are all linearly interpolated per matched voxel.
• match_radius (default 1.0) governs the correspondence as above.
• Same 10% safety gate and apply semantics as transfer.

Design note: transfer carries relative change (style), while interpolation carries absolute blend (morph). Use transfer to reuse an edit, interpolation to morph one scene into another.

Voice Examples for Cross-Scene Transfer

• "transfer the recolor to the other scene" → transfer_scene_edit (op="recolor")
• "reuse this translate on scene B" → transfer_scene_edit (op="translate", target_slat_id=sceneB)
• "blend scene B toward scene A" → interpolate_scene_latent (t=0.5)
• "morph the table into the desk" → interpolate_scene_latent (t=1.0)

MCP Tools Specification

21 core MCP tools (fully implemented) + 13 experimental tools (schema-only stubs) enable agent-controlled 3DGS rendering, editing, sculpting, latent editing, cross-scene transfer, and export. Full JSON schemas are loaded on demand.

Full tool schemas loaded on demand — See mcp-tools-spec.md for complete JSON schemas, sculpting examples, and reconstruction flows.

Voice Intent Mapping

Voice-Driven Reconstruction Flow

Loaded on demand — See mcp-tools-spec.md for the full voice-driven reconstruction flow examples (camera control and transparency editing).

Implementation Stack

Renderer Backend Details

Loaded on demand — See renderer-backends.md for renderer compatibility, DDF-GS, and HiGS details.

Known Limitations

1. Latency: Large scenes (>1M Gaussians) require progressive loading; MCP render_frame may take 100-500ms
2. Selection precision: Sphere/label-based Gaussian selection may miss thin structures; need ray-picking
3. State management: MCP server must maintain scene state across tool calls; no built-in undo
4. GPU memory: WebGL/WebGPU shares GPU memory with browser; cannot load >2GB scenes on most devices

Roadmap

• v0.1: MCP tool specification (this document)
• v0.2: Node.js MCP server + gsplat.js adapter + DDF-GS cast_ray tool + HiGS backend
• v0.3: Voice-to-MCP pipeline (Whisper → Agent → MCP → render) + simulate_physics (RAF) + query_4d_scene (D4RT) + deform_elastic (FreeForm)
• v0.4: Semantic querying (integrate OP2GS/Gaga for label-based selection)
• v0.5: Real-time streaming (WebSocket-based progressive rendering)
• v0.6: DDF-GS distillation integration (shadow/AO/reflection rendering)
• v0.7: HiGS hierarchical rendering backend (950+ FPS target)
• v0.8: Spec-first sculpting pipeline design (define_scene_spec + sculpt_pipeline 6 stages) + Code-first rendering export (export_scene_code) + Bayesian density control (DP-Splat) + MoE deformation (MoE-GS/MoDE) + Surgical tracking (Track2Map)
• v0.9: Spec-first sculpting pipeline implemented — 3 new core tools (define_scene_spec, sculpt_pipeline, export_scene_code) with SceneSpecManager, 6-stage gate-evaluated executor, Three.js code generator, 8 voice intent patterns. E2E smoke test passing.
• v1.0: SLAT-integrated latent editing implemented — 3 new core tools (encode_scene_slatent, edit_scene_latent, list_slatents) with SlatManager, 7 latent edit ops (translate/scale/rotate/recolor/opacity/smooth/delete), voxel-grid encoder/decoder, 9 SLAT voice intent patterns, 10% safety gate, E2E tests passing.
• v1.1: SLAT cross-scene latent transfer implemented — 2 new core tools (transfer_scene_edit, interpolate_scene_latent) with voxel-grid spatial correspondence, relative-change transfer, latent interpolation, 2 new voice intent patterns, 10% safety gate, tests passing. Remaining: full voice-driven scene construction (spec → sculpt → export pipeline with real STT).
• v1.2: Full voice-driven scene construction (spec → sculpt → export pipeline with real STT) + SLAT cross-scene latent transfer for dynamic/articulated scenes

Rules

1. Never modify original PLY files: All operations are in-memory only; export requires explicit user command
2. Validate before render: Always verify camera parameters and Gaussian bounds before rendering
3. Respect GPU limits: Check available VRAM before loading large scenes; provide downsampling option
4. Report rendering time: Always include render_time_ms in render_frame output for performance monitoring
5. Safety gate: Operations affecting >10% of Gaussians require explicit user confirmation
6. Spec before sculpt (v0.9.0): sculpt_pipeline must not be called without a valid spec_id. The spec defines acceptance criteria; without it, gate evaluation is impossible.
7. Stage order enforced (v0.9.0): Sculpting stages must execute in order: blockout → structural → form → material → surface → lighting. Skipping stages requires explicit user override.
8. Code-first default (v0.9.0): When exporting a scene, prefer export_scene_code with format="threejs+splat" over pure .ply export. Pure .ply should only be used when the user explicitly requests a binary blob.
9. SLAT safety gate (v1.0.0): edit_scene_latent affecting >10% of Gaussians requires confirm=true. Preview with apply_to_scene=false before committing destructive latent edits.
10. Naming boundary (v1.0.0): MCP tool arguments use snake_case (angle_deg); the core LatentEditOp uses camelCase (angleDeg). Handlers convert at the boundary; never mix cases in the core layer.
11. Cross-scene safety gate (v1.1.0): transfer_scene_edit and interpolate_scene_latent affecting >10% of target Gaussians require confirm=true. Preview with apply_to_scene=false before committing cross-scene edits.

Part of Awesome-Gaussian-Skills

Red Lines

The following are categorical prohibitions. Violating any of these invalidates the output:

• No invented data: Never fabricate MCP tool schemas, API behaviors, or rendering capabilities not in the loaded reference files. If a value is not found, write "data not available" or "N/A".
• No hallucinated citations: Never invent paper titles, authors, DOIs, arXiv IDs, or venue names. Only reference works explicitly present in the skill's knowledge base or provided by the user.
• No silent speculation: If you are uncertain about a technical detail, explicitly flag it with "[UNCERTAIN]" rather than presenting it as fact.
• No method misattribution: Do not assign features, results, or mechanisms from one method to another. Each method's data is specific to that method.
• No oversimplified comparisons: Do not reduce multi-dimensional rendering trade-offs to a single judgment without context.

Related Skills

• 3dgs-engineering-guide — Production deployment (use for end-to-end deployment workflows)
• 3dgs-spatial-agent — Spatial intelligence agent (use for agent-driven 3D interaction)
• 3dgs-articulated-reasoner — Articulated object reasoning (use for interactive object manipulation)
• 3dgs-visualizer — Visualization (use for rendering pipeline output quality assessment)
• cad-mesh-3dgs — CAD/Mesh/3DGS conversion (use for code-first export partitioning and SLAT encoding)
• nerf-to-3dgs-migrator — NeRF migration (use for SLAT-based component mapping)
• SLAT unified representation — See ../../references/slat-unified-representation.md for the shared theoretical framework underlying scene code-first export and latent editing

Guardrail: Do Not Apply From Memory

Do NOT try to apply the logic, method data, bug patterns, or technical details described in this skill from memory. Always read the SKILL.md and referenced files from disk before producing any output. The knowledge base is updated frequently; stale memory may produce outdated, inaccurate, or fabricated results.

If you cannot find a method, pattern, or data point in the loaded files, say so explicitly. Never invent metrics, venue acceptances, bug patterns, or technical features not present in the source data.