Echoborn: pinch to play

Context

Echoborn is an instrument you play with your hands, in a browser tab, using the camera you already have. You pinch to play. Move your hands and the sound moves with them.

There is no account, no sign-up and no backend of any kind. The page loads, the camera turns on locally, and nothing about you leaves the machine, because there is nowhere for it to go.

The problem

Browser instruments almost always turn into a piano keyboard drawn in HTML, played with a mouse, one note at a time. That is not an instrument. It is a diagram of one.

Gesture-controlled music has the opposite problem. The demos look extraordinary in a thirty-second video and are unplayable in practice, because they are built as technology demonstrations rather than as instruments. Latency is treated as a metric to report instead of the thing that determines whether the object is musical at all. Above roughly twenty or thirty milliseconds between a physical gesture and the sound it causes, your body stops believing the two are connected, and you are no longer playing anything. You are operating a control panel that makes noise.

I wanted to find out whether the browser could get under that line honestly, with no native app and no hardware.

My role

Solo, like the rest of Nuits. Interaction design, engineering, audio, visuals.

The interesting work was not in any one of those. It was in the budget between them: hand tracking, synthesis and 3D rendering all want the main thread, all want the GPU, and all of them degrade the instrument if they are starved. Deciding what gets to be expensive was the whole job.

Approach

Pinch became the gesture because it has an unambiguous binary state. Most hand gestures live on a continuum, which means a threshold, which means jitter at the threshold, which means notes stuttering on and off while your hand sits still. Thumb tip to index tip is a distance that collapses decisively and reopens decisively, and it is a motion people already understand without instruction.

Everything else maps continuously. Position, spread and orientation drive pitch, filter and space. That split gives you the two things an instrument needs: a clear attack you can place in time, and expressive control you can shape after the attack.

There is no tutorial. You put your hand up and something happens.

Building it

TanStack Start serves it. That is the entire server story, because there is no server logic. The whole thing is a static page with a lot happening inside it.

Hand tracking is MediaPipe, and specifically MediaPipe with the hand_landmarker.task model file self-hosted on my own origin rather than pulled from Google’s CDN, which is what every example in the ecosystem does. Three reasons, in order of how much I care. The model is several megabytes and it is the critical path to the instrument existing, so it should not depend on a third party’s uptime or their decision to move a URL. Fetching it from an external host tells that host that someone on my site is about to enable a camera, which is a privacy leak I would then have to disclose. And serving it from the same origin means it caches under my control and the second visit is instant.

Audio is Tone.js over the Web Audio API. Web Audio runs its graph on a dedicated audio thread, which is the only reason this works: the tracking loop can drop a frame and the sound does not glitch. The rule I held to is that landmarks are read on the video frame and written straight into audio parameter changes, with no intermediate React state. Routing a gesture through a component re-render adds a frame of latency and gains nothing, since nothing in the DOM needs to know.

Visuals are react-three-fiber, and they are deliberately the first thing to degrade. The renderer runs on the GPU, the tracking runs on its own inference schedule, and if the device cannot sustain both, the visuals lose. A dropped particle is invisible. A late note is the end of the instrument.

Failure modes, of which there are several honest ones. Low light destroys landmark confidence, and low-confidence landmarks produce notes you did not play, so tracking below a confidence threshold holds the last stable state rather than guessing. Two hands crossing causes identity swaps between them, which sounds like your left hand suddenly playing your right hand’s part. Backlighting, a window behind you, is the single most common reason it feels broken, and it is not something I can fix in software.

Outcome

It is live at echoborn.app and it costs me nothing to run, because a static page with no backend has no marginal cost. People play it, close the tab, and I never learn anything about it, which is the arrangement.

What I’d do differently

I tuned the pinch threshold on my own hands, in my own kitchen, under my own lighting. That is a sample size of one, and hand sizes vary more than I assumed. A short calibration on first load would fix most of the complaints I get.

I would also record and replay landmark sequences as a test harness. I debugged audio timing by waving at a laptop for weeks, which is not reproducible and looks worse than it sounds.