embodied-sensor-kit
A library that lets your body act as an input to a web app.
embodied-sensor-kit is a library for building web apps you control with your body. A smartwatch reports how your wrist is moving, a webcam tracks your hand, and the kit turns both into things an app can use: a point on the screen, and named gestures such as a double pinch or a wrist flick. A developer can wire a gesture to a command the way they would wire a click to a button, without writing the sensing code themselves.
The clip above is a drawing app built with it. You draw in the air with your fingertip, double pinch to turn drawing on and off, and flick your wrist to undo the last stroke.
How it fits together
| Input | Detected by | In the drawing app |
|---|---|---|
| Hand position | MediaPipe hand tracking | Draws a trace in the air |
| Double pinch | Watch gesture | Turns drawing on and off |
| Wrist flick | Watch motion sensors | Undoes the last stroke |
The watch side is deliberately simple. A small Wear OS app streams raw motion data, and everything else happens in the library: smoothing, calibration, mapping arm movement to a point on the screen, and recognizing gestures. That keeps the watch code the same for every app and lets each app tune its own gestures. A simulator streams fake motion, so you can build and test without a watch.
Why I built it
My research is about how the body helps people think and communicate, and I wanted a way to build software that takes the body as input without starting from scratch each time. I first tried hand tracking on research video and found it harder than I expected. This kit is what I built to make that kind of input easier to work with.
Where it stands
It works as a prototype. The code is not public yet, and I haven’t chosen a license. I’d like to bring in more devices than the Pixel Watch.