Late August got the Spatial Cognition Platform running on physical Quest headsets without crashing tracking. Late September was about transforming that scaffold into an environment an engineer would actually want to spend hours inside: the Control Room.
We retired the early, monolithic AgentOps room (#504) in favor of modular
spatial substrates: a planning table (#575), gate benches (#562), gateway
racks, and dedicated orb sockets. Moving from wireframe prototypes to an integrated
room revealed dozens of ergonomic and serialization edge cases that only appear
under real headset usage.
The first major bug affected the planning table substrate (#575). To achieve
zero-allocation rendering for thousands of node links, we use a custom
InstancedPrimitiveCatalog. In the initial implementation, Configure() stored
the catalog in plain private C# fields. Because plain C# instances are not
serialized by Unity, saving a composed scene and reopening it left the catalog
null. The moment an operator or integration test loaded the saved scene and
applied state, the table threw an uninitialized exception. We resolved this by
persisting the underlying mesh and material references in [SerializeField]
arrays on the renderer component and lazily rebuilding the runtime catalog on
Awake and OnEnable.
VR typography brought its own physical constraints (#606). Rendering small
text labels directly in 3D space causes severe TextMeshPro sampling artifacts,
leaving node labels muddy and unreadable from distance. We standardized on a
strict two-size type scale (Primary and Secondary), authoring font assets at 10x
resolution and scaling them down via local component transforms. This produced
tack-sharp text across both near-field inspection desks and wall-mounted boards.
Physical interactions were smoothed out with HeldOrbFollowModel (#588). When
an operator grabs an agent reasoning orb, rigidly pinning its orientation to the
hand controller causes the orb to tumble wildly with every micro-twitch of the
wrist. The new model eases position to a forward offset using a tuned spring rate
while decoupling orientation entirely, letting the orb maintain a calm, slow yaw
rotation. A flick classifier was added to let operators toss an orb back toward
its seat with a natural release gesture.
Finally, we gave the Helper companion an audible presence (#612). Built as a
stateless outbound port (IHelperVoice) against local TTS endpoints, it streams
audio through an engine-free RIFF PCM16 decoder. If network latency spikes or the
TTS endpoint is offline, the synthesis fails completely silent, ensuring voice
playback never blocks core room simulation.
Lessons learned
- Rebuild pure C# catalogs lazily from serialized arrays across scene loads.
Unity’s scene serialization will silently drop any state held in standard C#
data structures that do not inherit from
UnityEngine.Object. To survive scene reloads, serialize raw primitives (mesh IDs, material arrays) in component fields and expose a lazy restore method that reconstructs runtime caches on first access. - Scale down large font assets in transforms to keep VR text crisp. Small font point sizes in TextMeshPro render poorly when magnified in stereoscopic VR. Rendering fonts at ten times normal size onto high-resolution SDF atlases and scaling the transform down by 0.1 creates crisp, legible text across varying view distances.
- Decouple held object rotation from controller wrist tumbling. Rigidly coupling 3D interactive objects to raw controller quaternions exaggerates wrist movement and makes inspected data difficult to read. Easing position to controller-forward while keeping object rotation on a calm, independent spring produces an interaction feel that is tactile rather than erratic.
- Keep audio synthesis ports stateless and fail-silent. Ancillary features like companion voice acting or sound cues should never be load-bearing. Modeling voice synthesis as a stateless HTTP client with automatic fallback ensures that a missing audio pack or an inference timeout never interrupts room-scale tracking.