Skip to content

Viz controls & interactions architecture

How interactive UI — control views, overlays, banners, editors, file choosers, and interactive 3D objects — is wired end-to-end in py/pytanga/viz/. Follow this contract when adding a new control, view, or interactive element. For the overall scene/layout/overlay model, see viz-architecture.md.

Concepts

Three orthogonal concerns, one model:

  • Scene — a 3D/2D world of entities (incl. interactive objects) + a camera
  • scene chrome.
  • Layout — a tree arranging scene panes (SceneView) and widgets (*View) in split/stack/overlay views (one per page).
  • Control — one model id + kind + value + handlers, wrapped by a *View(ControlView) and placed in an overlay GroupView or a SceneView(overlay=[...]), a layout pane, or a banner/dialog.
  • Interaction — an interactive object (ActPoint, …) under the same event model; it differs only in payload richness and coalescing.

Fixed contract

  1. Single global id namespace. Every control, interactive object, and View has a globally unique, stable id (views auto-generate v0… and can override it via id=); scene is a routing hint, not a storage key. Stable ids are what make viz.remove_view(id) possible, and what the frontend reconciles a view_layout re-push on — there is one live-view registry (_viewRegistry, see viz-architecture.md) and no parallel id-indexed dicts.
  2. One (id, event) registry. ControlHandlerRegistry keys handlers by (id, event) — change, click, press, release, cell_change, row_add, column_add, row_delete, toggle, close, accept. Layout views, banners, dialogs, editors, and interactions all register here.
  3. One control model. Each *View wraps a pytanga.viz._controls.Control (exposed as view.control) and serializes its fields from it via Control.serialize (_fields() per kind). ControlView.__getattr__ forwards attribute reads and set_value/get_value/undo/redo/ can_undo/can_redo to the control, so value and history live on the control, not on a host. Two handler shapes share that one registry, distinguished by the stored HandlerOrigin: ControlHandler (async def h(value, event: ControlEvent)) for controls, and InteractionHandler (async def h(event)) for interactive objects. Every event derives from the single ControlEvent; the interaction events (InteractionEvent → ClickEvent/DragEvent/ScrollEvent) add the target object_id, the event_type and the camera. See Handler types below for the full set of aliases.
  4. One client→server envelope. The frontend sends every user action through sendEvent(target, event, data) in templates/events.js: { type: "event", target: "<id>", event: "<name>", data: {…} }. server.py maps event → the control or interaction callback.
  5. Tree-walk dispatch, one registry. Inbound control:* events route to LayoutHost.dispatch_control_event, which resolves the control id by walking the layout trees and dialog contents (resolve_control — there is no _control_views index) and delegates to control.handle_event. Interactions route through InteractionHost._dispatch_interaction_event (handler (event), with drag-move coalescing + camera caching) — reading from the same registry.
  6. One async request/reply seam. Control.handle_event_async(event, payload) -> Dispatch (default delegates to the sync handle_event) lets a kind await a provider before reporting its dispatch. A Dispatch may also carry a reply message dict, which dispatch_control_event sends only to the requesting browser via Transport.send_to_browser. Request/response events (e.g. a custom enum column's enum_options) use reply instead of a broadcast control_update.

Handler types

Handlers are async callables. Their arity and event type say which family they belong to; all aliases are public (exported from pytanga.viz).

Alias Signature Attached through
ControlHandler async def h(value, event: ControlEvent) a control's on_* fields (on_change, on_click, on_press, on_release, on_cell_change, on_row_add, …)
EnumOptionsHandler async def h(request: TableEnumOptionsRequest, event: ControlEvent) -> list[str] \| tuple[str, ...] \| None Table.on_enum_options (custom enum columns; reply-only)
InteractionHandler async def h(event: InteractionEvent) Visualizer.on_interaction(object_id, event_type, h), VizSceneHandle.on_interaction, VizObjectRef.on_interaction
ActHandler · ActEventHandler · ActClickHandler async def h(event: DragEvent \| ClickEvent, obj: ActSceneObject) an ActSceneObject subclass' own handler= / on_drag_start= / on_drag_end= / on_click= constructor kwargs

Both ControlHandler and InteractionHandler are Awaitable-returning, but the interaction one is declared as Coroutine because the interaction dispatcher schedules it with asyncio.create_task — a custom awaitable that is not a coroutine is not accepted there.

The control value argument depends on the kind (float for sliders, str for dropdowns / text / textarea / colour pickers, bool for checkboxes, None for buttons and group toggles, a change/selection dataclass for tables) and is annotated Any in ControlHandler; see ControlHandler's docstring for the per-kind table. ActHandler returns bool: True means "I handled it completely (including the flush)", False lets the default behaviour run (move the object and flush).

Event hierarchy

ControlEvent                      # shared base: browser_id (+ future fields)
└── InteractionEvent              # + object_id, event_type, camera
    ├── ClickEvent                # + mouse_button, modifiers, screen_position,
    │                               world_position, world_normal
    ├── DragEvent                 # + screen_position, delta_pixels, world_position,
    │                               world_delta, drag_mode, ray_origin,
    │                               ray_direction, delta_transform
    └── ScrollEvent               # + screen_position, scroll_delta

Two consequences worth knowing when writing a handler:

  • Annotate the concrete event. world_position lives on ClickEvent / DragEvent, not on InteractionEvent, so async def h(event: InteractionEvent) cannot read it — use DragEvent for drag handlers.
  • Events may grow. ControlEvent is extensible: the event is always the last argument, after the value, so extra fields can be added without breaking existing handler signatures.

Registration

  • Controls. Handlers are dataclass fields named on_<event> on the Control; Control.register_handlers (called by LayoutHost.register when a view tree is mounted) maps on_change → "change", on_cell_change → "cell_change", … and registers each under (control_id, event) with HandlerOrigin.CONTROL.
  • Interactive objects. on_interaction(object_id, InteractionEventType.DRAG_MOVE, handler) registers under (object_id, "drag_move") with HandlerOrigin.INTERACTION. ActSceneObject does this for you in _init, mapping its constructor kwargs onto DRAG_MOVE (the per-frame handler), DRAG_START, DRAG_END and CLICK.
  • One registry, one key space. ControlHandlerRegistry stores both families side by side, tagged with HandlerOrigin; read control entries with get() and interaction entries with get_interaction(), and use clear_controls() to drop only the control family. Registration is last-writer-wins per (id, event), and the family is not checked — a control and an interactive object sharing an id and an event name shadow each other.
  • Request/reply. EnumOptionsHandler is registered like any other on_* field, but its Dispatch.reply is sent only to the requesting browser instead of broadcasting a control_update.

Handler (the ambiguous old single alias) no longer exists — use the explicit aliases above. Typing guidance for the handlers you write is in typing-and-annotations.md: annotate value: Any, event: ControlEvent for controls and event: DragEvent, obj: ActSceneObject for act objects, and narrow (isinstance) before touching a subclass-only attribute such as ActPoint.point.

Host layer

Control/overlay concerns are split out of Visualizer into hosts in py/pytanga/viz/_hosts.py:

  • OverlayHost — base that holds the two ports (Transport + LayoutHost); _handler_registry/_push_message are thin aliases over Transport.
  • LayoutHost (in _layout.py) — the owner of scenes + layouts. It registers a mounted view tree (register walks iter_control_views and calls each Control.register_handlers, injecting the _push callback), resolves control ids (resolve_control — layouts + dialogs), and runs the control:* / file_browser_* dispatch core (dispatch_control_event). There is no ControlHost.
  • ThemeHost / InteractionHost — theme selection and interactive-object handling.
  • OverlayContainer (in _layout.py) — the per-layout overlay: add(view), show_banner/alert/confirm, show_dialog, open_editor, and their _on_close/_on_accept handlers (the old BannerHost/DialogHost/ EditorHost are folded into it).

Visualizer exposes the public API through explicit forwarders (no __getattr__); _register_routes() installs the inbound routing table on the Transport.

View & layout model

The layout tree is the single render path for every page:

  • View is the base for every pane/container (SceneView, StackView, SplitView, GroupView, MenuView, and the *View control wrappers); every view has a stable id (auto v0…, overridable) so it can be addressed/removed at runtime.
  • SceneView is a pane that renders a named scene; its overlay lists views (e.g. a GroupView) that float over that pane's canvas, anchored by each child's position (EAnchor).
  • A GroupView(parent_id="<entity>") in a SceneView(overlay=[...]) is instead attached to that entity via CSS2D. Because view_layout arrives before the scene entities, ThreeJsView defers the attach until the parent entity is registered (_pendingAttachedGroups), so anchored groups appear on first load.
  • GroupView is a titled StackView with an optional leading icon, icon_only mode, and a borderless fold button — the control-group container.
  • MenuView is a hamburger dropdown or a permanent bar of options (EControlVariant.MENU flattens its control children).

Controls are added declaratively — build the *View and either pass it to set_layout (or viz.add(view), which mounts it in the default layout's overlay) or list it in a SceneView(overlay=[...]). The whole tree serializes to one view_layout message via serialize_layout(root, name=..., overlay=[...]).

View-mode unification

There is one view mode. A "single scene" is served as a layout whose root is a one-SceneView stack (StackView("vertical", [SceneView(name)])) merged with the global overlay (base scene "" only) and per-scene overlays:

  • LayoutHost._scene_layout_for(scene_name) resolves any scene to its serialized view_layout (the base scene reuses the default layout).
  • The server always resolves a view_layout on ready (layout → _layout_callback, else _scene_layout_callback(scene_name)) and the frontend always renders through _buildLayout — there is no separate single-scene bootstrap.
  • Global-overlay changes are granular: viz.add(view) pushes an overlay_define and viz.remove_view(id) pushes an overlay_remove (only the overlay view is sent, not the whole layout); per-scene overlays and set_layout re-push the full view_layout instead.
  • On every view_layout the frontend reuses the existing ThreeJsView scene panes (keyed by scene name) rather than tearing them down, so only the DOM chrome rebuilds and the WebGL scene/camera survive; a scene pane newly introduced by a re-push fetches its state via scene_sync_request.

Event names

Control events (→ LayoutHost.dispatch_control_event): change, click, press, release, cell_change, row_add, column_add, row_delete, undo, redo, toggle (group), close (banner/editor — data.value is the editor's text or null), file_browser_navigate, file_browser_select.

Interaction events (→ InteractionHost._dispatch_interaction_event, coalesced): interaction:click, interaction:dblclick, interaction:drag_start, interaction:drag_move, interaction:drag_end, interaction:scroll.

Client → server logging

The browser reports warnings/errors to the backend over the same event envelope, targeting a backend-only ClientLog control (_controls.py, id "client_log"):

  • Frontend sendLog(level, message, { source, data }) → sendEvent("client_log", "log", { level, message, source, data }) (templates/events.js); level ∈ debug|info|warn|error.
  • server.py::_EVENT_MSG_MAP maps "log" → "control:log", routing it to LayoutHost.dispatch_control_event, which resolves the reserved id to the ClientLog control (a LayoutHost hook — the control is not in a layout).
  • ClientLog.handle_event("log", payload) normalizes the payload into a ClientLogRecord and fires ("client_log", "log"); the default sink logs via logging.getLogger("tanga.viz.client"), replaceable with viz.on_client_log(handler).
  • ClientLog is backend-only: never serialized, never placed in a layout.
  • Opt-in trace forwarding: setLogForwarding(true) / ?log=1 additionally forwards the frontend _log(...) init/WS lines at info level (default off — only console.warn/console.error are forwarded).

Adding a new control kind

  1. Backend model — add a Control dataclass in py/pytanga/viz/_controls.py (id/label/tooltip + kind fields) with _value_type + _fields() (or set_value/get_value overrides for table-like kinds). There is no central serialization/coercion switch — serialize() merges _fields() and set_value coerces via _value_type.
  2. Layout view — add a *View(ControlView) in py/pytanga/viz/views.py whose __init__ builds self.control = <Control>(...) (keep the constructor signature); the base ControlView._serialize emits the fields from self.control, and ControlView.__getattr__ forwards set_value/undo/… Handler registration is automatic: the on_* constructor kwargs are dataclass fields on the control, and Control.register_handlers maps each on_<event> to its (id, event) registry entry at mount time.
  3. Frontend — add a create<Kind> factory in templates/controls/<kind>.js (importing the shared registry/event/icon helpers from controls-panel.js, and registering a _controlRegistry entry with an owner and an apply(value)), and, for a layout view, a views/<kind>-view.js whose render() calls it with owner: 'layout'. Send events via sendEvent(id, "<event>", { value }).
  4. Server routing — if the kind introduces a new event (not just a value), add the event→message mapping to server.py::_EVENT_MSG_MAP. Event handling lives on the control: override Control.handle_event(event, payload) -> Dispatch in _controls.py to mutate the model and report which (id, event) handler to fire and what to push back (see Table.handle_event). For a kind whose handler must still fire when the control id is not resolvable, mirror the parse_table_event helper. For an async request/response event, override Control.handle_event_async (declare the provider as an on_* field so register_handlers registers it) and return Dispatch(reply=<message>); the dispatcher sends that reply to the requesting browser only (see Table.handle_event_async for enum_options).
  5. Tests — serialization round-trip, registration, and dispatch.

Interactive objects

Register handlers with Visualizer.on_interaction(object_id, event_type, handler) (stored under (object_id, event_type.value)). The per-pane frontend InteractionController captures/throttles pointer events and sends them under interaction:* names; the backend coalesces drag_move.

Active rectangles

ActRectangle2D (_active.py) is an interactive axis-aligned rectangle: a visual-only Rectangle2D body plus child ActPoint handles (4 corners for resize, one centre handle for translate). It is a composite — the frontend raycasts one mesh per entity, so each grabbable part is its own ActPoint entity and the body's interaction_config is disabled. Default resize/translate behaviour is overridable (on_corner_drag / on_translate / on_change); the rectangle_labeling.py example composes a disabled left-drag DragBinding plus a mode flag to drag out a preview Rectangle2D and finalize an ActRectangle2D.

SquarePointStyle (a PointStyle variant, dispatched in factory.js by style_type exactly like CrossHairPointStyle) renders a Point as a flat square marker — used for the rectangle handles.

Per-handler enable/disable + cursors

Every active element (ActSceneObject) can toggle its handlers individually: DragBinding/ClickBinding carry a mutable enabled flag, and the general handler/on_click are toggled via set_handler_enabled/set_click_enabled. refresh_interaction() re-registers the trigger set after a change. A cursor can be attached to an element (ActSceneObject(cursor=…) → hover_cursor, shown on hover) or to the gesture (InteractionConfig.cursor, shown during the drag); Visualizer.set_cursor()/VizSceneHandle.set_cursor() set a per-scene override (via the scene_config message) for mode switches.

Image canvas

ImageCanvas (_image_view.py) displays images on an interactive ActImagePlane (_active.py). The plane reuses the standard ActSceneObject contract — set_interaction + on_interaction + a drag_anchor that returns the ray↔plane hit, so world_position is the pixel coordinate (the canvas scene uses a y-down frame with 1 unit = 1 pixel). Mouse handlers modify shader uniforms via ImageCanvas.set_uniform, which sends an image_update JSON message; the pixel data itself travels as binary WebSocket frames (_image_wire.py, Transport.send_bytes) and is never re-sent on uniform/overlay changes.

The canvas also supports multiple specific handlers via DragBinding / ClickBinding (a mouse button + optional modifier set; the most specific match wins, falling back to the general on_drag/on_click), and lets you rebind the camera navigation (pan/dolly/rotate) per scene through a controls mapping (SceneConfig.controls, applied by configureControls in view_mode.js).

Coordinate frame overlay/underlay

CoordinateSystem(display_mode="overlay") (see viz-architecture.md) emits two payload-style scene objects — axes_overlay (overlay) and grid_underlay (underlay) — each carrying a static spec. They register no (id, event) handlers and send no interaction/control events: the frontend renderers read the live camera every frame and recompute ticks/grid locally, so pan/zoom needs no backend round-trip.

Per-pane camera view & visibility

SceneView(camera_view=…) (a CameraView bundling camera + lock + navigation + per-pane controls + viewport + background_image) and SceneView(hide=…) / SceneView(show=…) are per-pane attributes, not new (id, event) controls. They serialize as fields on the scene_view node and are applied by ThreeJsView (setCamera, applyCameraLock, configureControls for the "2d" navigation mode, the viewport crop via applyPinhole, the NDC background quad, and a build-time visibility filter) — they register no handlers and send no control/interaction events.

Viewport state is set at runtime through the existing message dispatch, not a new channel: Visualizer.set_viewport(view, …) pushes a view_viewport message (handled in viewer.js next to view_camera, dispatched to ThreeJsView.setViewport), and set_viewport(scene_name=…) / VizSceneHandle.set_viewport(…) re-push the existing scene_config with a viewport field. See viz-architecture.md.

Background images are swapped the same way: Visualizer.set_background_image(view, image) sends the new pixel frame (binary) followed by a view_background_image message (handled in viewer.js next to view_camera/view_viewport, dispatched to ThreeJsView.setBackgroundImage) — no view_layout re-push, so the pane's WebGL scene and every other pane are untouched.

Control enable/disable/hide (control_state)

Controls carry two runtime state flags on the model — Control.enabled (rendered but greyed out + non-interactive when False) and Control.visible (False hides it without removing it from the layout). Both are serialized only when non-default (False), mirroring tooltip.

Set them at runtime without re-pushing view_layout:

  • ControlView.set_enabled / set_visible (+ enable/disable/show/hide sugar) mutate self.control and push through the injected _push_state callback (injected by LayoutHost.register, next to _push).
  • Visualizer.set_control_enabled(cid, bool) / set_control_visible(cid, bool) resolve the control via resolve_control (layouts and dialogs) and push — so they work for dialog/banner controls too.

The wire message is { type: "control_state", id, enabled?, visible? } (only changed fields), pushed by LayoutHost._push_control_state and handled in viewer.js next to control_update. applyControlState(id, msg) looks up the rendered control in _controlRegistry (each factory stores el: wrapper) and applyControlStateToElement toggles the .tanga-control-disabled class, the native disabled attribute on input/select/textarea/button, and display:none. For a layout ControlView, visible=false also collapses the hosting view: viewer.js looks it up in _viewRegistry (the control id doubles as the ControlView's stable view id) and calls View.setHidden(true), which removes it from flow and reports zero minimum / no preferred size so the enclosing Stack/Split views re-layout; the initial visible:false is applied the same way in build.js. Disabled grey-out uses the theme tokens --tanga-disabled-opacity / --tanga-disabled-fg (overridable per theme).

Follow-ups

  • Fold interaction.js onto sendEvent — the interactive-object frontend (templates/interaction.js) still sends interaction:* messages directly rather than through sendEvent (the server already routes event: "interaction:*"). Deferred: it touches the timing-sensitive drag path and isn't covered by the browser-less test suite.
  • Active menu element — if menus gain a backend-visible "active element", add value / on_change / on_activate to MenuView directly (menus are overlay containers, not control leaves, and keyboard navigation is frontend-only).