📦 Shipment Dispatched

Simulated delivery to:
Items have left the replication bay. A simulated delivery confirmation has been recorded in the agent log.
Drag to orbit · Scroll to zoom · Right-drag to pan
Agent Matrix · 20-Agent Core
[Agent-00]Orchestrator online. 20 agents connected.
Active Agents: 20 / 20
Recursive Loop: 0 passes

ArmOS — System Documentation

Simulation guide & real-time hardware reference

What Is ArmOS

Browser-based digital twin of a 4-DOF robotic arm. Type a natural-language command; a 20-agent pipeline parses intent, solves inverse kinematics, and executes a full pick-and-place sequence in 3D — entirely client-side, no install required.

Command Syntax

  • pick up [object] and place by [object] — e.g. "pick up the red cube and place by the blue sphere"
  • put the syringe next to the vial — industry sets are addressed by object name
  • [task] and then [task] and then [task] — chains run sequentially

Object sets — Classic Shapes · Desktop/Office · Electronics Bench · Workshop · Medical · Parts Assembly · Welding & Cooling. Switch with the dropdown in Scene Objects; the legend always lists the exact names the parser accepts.

AI parsing — when the backend is online, commands are interpreted by Google Gemini, so looser phrasing works too ("swap the cubes", "move the battery by the board"). Offline or without an API key, the onboard grammar takes over automatically.

Assembly Missions

Say build a small table (or stool, gantry crane, lamp, tower) and ArmOS runs a full mission pipeline: engineering → virtual FDM fabrication → simulated perception → component matching → dependency-ordered installs → measured inspection with a correction loop (max 3 passes per part).

  • Creation Capsule — every mission generates a portable replication contract before fabrication: engineered component artifacts (dimensions, material, tolerance, print orientation, grasp regions, geometry hash, SIMULATED_MACHINE_CODE reference, material/energy/time estimates), the assembly graph, success criteria, and node compatibility. Downloadable as JSON from the mission panel.
  • Full materialization lifecycle — components never just appear: QUEUED → SLICING → PRINTING (progressive reveal of the actual engineered geometry, estimated physical time vs accelerated sim time shown) → COOLING → READY → staged → scrambled with a recorded random seed → rediscovered through perception. The timeline strip tracks INTENT through COMPLETE, and each phase emits provenance events.
  • Replicator Nodes — the sidebar node panel shows the active node's manifest (RP-0001 FDM, RP-0002 CNC-only, RP-0003 large dual-arm FDM). Missions validate against the manifest; a node that can't fabricate returns a structured ROUTE_REQUIRED. Physical execution stays PHYSICAL_EXECUTION_DISABLED throughout.
  • Built-in catalog — table, stool/chair, gantry crane, desk lamp, tower, bench, step ladder. Type "build a [name]" or use color/size/leg-count modifiers: "build a large 6-legged red table", "build a huge cyan bench". Modifiers change actual printed geometry — dimensions, leg count, and material color all update in the engineered artifact.
  • Engineering output is the single source of truth — parts list, ghost preview, reference render, target poses, and success conditions all derive from one plan. Known designs come from the onboard library (no AI call); novel goals are engineered by Gemini server-side.
  • Honest simulation — the planner consumes noisy observations (±2 mm), not hidden object IDs; color is never the sole matching cue; every placement carries an injected error the inspector must measure and correct. Success means measured state within tolerance (4 mm), never "the animation finished".
  • Mission panel — goal, live status, phase timeline, fab-bay progress, Creation Capsule, per-component states, and running trial stats. Difficulty (Normal / Varied / Challenge) injects distractor parts, larger placement errors, and grasp slips. ↺ Same re-runs the mission in a freshly randomized workspace; ✕ New clears it for a fresh goal.
  • Known limits, logged honestly — the 4-DOF arm has no wrist yaw, so piece yaw is normalized at mate rather than commanded; fabrication toolpaths are representative, not production G-code.

Scene Controls

  • Object set dropdown — swaps the manipulation library: classic primitives or industry benches (office, electronics, workshop, medical, assembly, welding).
  • ↺ Reset — re-scatters the current set to fresh positions within reach.
  • Agents toggle — sidebar button shows/hides the agent activity overlay and 3D neural ring (hidden by default). Firing agents pulse their ring node and its links.
  • Phantom trajectory — before the arm moves, the planned end-effector path is drawn as a glowing spline (Agent-08), then fades once the object is placed.
  • Safety envelope — the translucent dome is the arm's safe workspace. Agent-13 bound-checks every waypoint; a violation flashes the dome red and Agent-09 inhibits motion.
  • Scene Objects panel — lists every object; held items are dimmed. On phones the same list rides above the command bar on the 3D view.
  • IK — 2-link planar solver; hover, grasp, lift, transit, and place poses solved independently.
  • Object tracking — held mesh follows the gripper world position each frame; snaps to rest on release.

3D Viewport Controls

  • Orbit — Left-drag
  • Zoom — Scroll wheel
  • Pan — Right-drag

4-Phase Execution Pipeline

Phase 1
200–400 ms
Perception & IntentAgent-01 parses the natural-language goal. Agent-02 resolves object world coordinates from the scene registry.
Phase 2
700–900 ms
Kinematics & SafetyAgent-07 runs the 2-link IK solver for hover, grasp, lift, transit, and place poses. Agent-09 validates all trajectories against workspace limits.
Phase 3
1 200 ms
Hardware Motion DispatchAgent-10 sequences the five-phase pick-and-place motion: hover → descend → grasp → lift → transit → place → retreat. The gripper mesh closes/opens and the held object follows the gripper world position each frame.
Phase 4
1 600–2 000 ms
Recursive Self-ImprovementAgent-15 evaluates the mission outcome. Agent-19 records the trial for future policy optimization, incrementing the Recursive Loop counter.

Reserved Agent Roles

  • Agents 03–06 — Vision: segmentation, classification, pose estimation, point-cloud filtering.
  • Agent-08 — Trajectory optimiser: cubic-spline smoothing to minimise jerk.
  • Agent-11 — Environmental surveillance: continuous workspace sweep every 4 s, drift detection, position-change alerts mid-mission, pre-install re-observation trigger. Agents 12–14: actuator control, bounds enforcement, inspection.
  • Agent-16 — Workspace spatial map: persistent world model tracking last-known position and confidence for every scene object; confidence decays over time so stale observations are distinguished from fresh ones. Agents 17–19: sensor stubs, reward evaluation, trial recording.

Required Parts — Minimum Build (desktop, ≈$170–260)

QtyPart (exact)Spec / role
1Raspberry Pi 5, 8 GBBridge host: runs local_bridge.py + WebSocket. 27 W USB-C PSU, 32 GB microSD, internet access (Gemini parsing)
1ESP32-S3 DevKitC-1Servo MCU: runs arm_controller.ino, 50 Hz PWM dispatch (Arduino Mega 2560 also works)
1PCA9685 driver board16-ch 12-bit PWM over I²C (addr 0x40), drives all servos
3MG996R servoBase yaw / shoulder / elbow — 11 kg·cm @ 6 V, 180°, metal gear
1MG90S micro servoGripper — 2.2 kg·cm, 180°
16 V / 10 A regulated PSUServo rail → PCA9685 V+ terminal (never power servos from the Pi)
15 V / 3 A PSUSeparate logic rail for MCU (Pi uses its own 27 W supply)
14-DOF arm frame, ≤500 mm reach3D-printed PLA/PETG (M3 hardware) or aluminium bracket kit

Wiring Map

ConnectionFromTo
USB serial, 115 200 baudPi 5 USB-AESP32-S3 USB-C
I²C (SDA / SCL)ESP32 GPIO 8 / GPIO 9PCA9685 SDA / SCL
PWM ch 0 / 1 / 2 / 3PCA9685base / shoulder / elbow / gripper servo
6 V 10 AServo PSUPCA9685 V+ / GND screw terminals
GroundALL railsCommon ground — required, or servos jitter

Bring-Up — Plug and Play

  • 1. Flash MCU — upload firmware/arm_controller.ino (channel map above, 50 Hz).
  • 2. Start bridge — on the Pi: pip install websockets pyserial, then python local_bridge.py --serial /dev/ttyUSB0 (serves ws://0.0.0.0:8080).
  • 3. Open the twin — browse to this app from any machine on the same network; it auto-connects to the bridge (WebSocket tab has the payload spec).
  • 4. AI parsing — the ArmOS server needs Node 18+, internet, and a GEMINI_API_KEY; without it the onboard grammar parser takes over automatically.

Optional Upgrades

  • Vision — Intel RealSense D435i (1280×720 RGB-D @ 30 fps) + NVIDIA Jetson Orin Nano (40 TOPS) for on-device object detection.
  • Closed-loop feedback — AS5600 magnetic encoders (12-bit, I²C) per joint, 100 Hz PID on the MCU.
  • Mid-scale (500–900 mm) — aluminium T-slot frame + DS3218 (20 kg·cm) servos on the same electronics.
  • Industrial — Dynamixel XM540-W270 smart actuators, RS-485 @ 4 Mbps (replaces PCA9685 chain).
Safety: Add a hardware e-stop on the servo rail — the MCU watchdog must cut servo power if no valid command arrives within 500 ms.

Connecting a Real Backend

The simulation connects to ws://localhost:8080 and auto-retries every 3 s. Replace with your server address to drive the twin from live encoder telemetry.

Inbound Command Payload

{ "target_angles": { "base_yaw": 45.0, // –180 to +180° "shoulder_pitch": 30.0, // –90 to +90° "elbow_pitch": -20.0, // –90 to +90° "gripper": 75.0 // 0 (closed) – 100 (open) } }

Minimal Python Server

import asyncio, json, websockets async def handler(ws): while True: await ws.send(json.dumps({"target_angles": { "base_yaw": read_encoder(0), "shoulder_pitch": read_encoder(1), "elbow_pitch": read_encoder(2), "gripper": read_gripper_pct() }})) await asyncio.sleep(0.033) asyncio.run(websockets.serve(handler, "0.0.0.0", 8080))

Update Rates

  • Twin rendering — 60 Hz (rAF loop)
  • WebSocket telemetry — 30 Hz sufficient
  • Servo PWM — 50 Hz standard
  • Encoder PID — 100–500 Hz on MCU
CORS / TLS: Use wss:// when serving over HTTPS, or run locally over HTTP for development.

Bundled Local Dependencies

lib/ three.min.js OrbitControls.js

No internet required. Air-gapped judges see the full 3D viewer with object manipulation.

Cross-Platform Compatibility

  • Runtime — HTML5 + ES6. Chrome, Firefox, Safari, Edge on Windows/macOS/Linux/ChromeOS.
  • Renderer — WebGL 1.0+ (all hardware post-2012).
  • No build step — open index.html directly or python -m http.server 8000.

Hardware Bridge

Cloud
ArmOS Digital TwinHosts 3D viewer, agent matrix, and object manipulation. WebSocket endpoint bridges to local hardware.
Local
Agent Controller ScriptPython bridge on the host PC: connects WebSocket, translates angles to PWM, writes to MCU over USB/Serial, streams encoder feedback back.
import asyncio, json, serial, websockets ser = serial.Serial("/dev/ttyUSB0", 115200, timeout=0.1) async def bridge(): async with websockets.connect("wss://YOUR-URL/ws") as ws: async for msg in ws: a = json.loads(msg).get("target_angles", {}) ser.write(f"{a.get('base_yaw',0)},{a.get('shoulder_pitch',0)},{a.get('elbow_pitch',0)},{a.get('gripper',0)}\n".encode()) line = ser.readline().decode().strip() if line: await ws.send(json.dumps({"telemetry": line})) asyncio.run(bridge())

Repository Checklist

  • ✅ index.html — self-contained, no build step
  • ✅ lib/three.min.js — bundled offline
  • ✅ lib/OrbitControls.js — bundled offline
  • ⬜ local_bridge.py — add for hardware demo
  • ⬜ firmware/arm_controller.ino — MCU sketch