Phonk Hand

A study in hand-sound gestural interaction. MediaPipe Hands tracks both hands from the webcam and translates them into a Tone.js audio engine with always-on oscillators, modulated without envelopes for pitch stability.

Two performance modes grounded in real music theory. CHORDS, for vocal accompaniment. The left hand selects the degree of a progression, the right hand the chord type, wrist rotation the inversion. MELODY, for improvisation. The left hand picks one of the seven diatonic chords, the right plays over one of thirteen scales (pentatonics, diatonic modes, blues, hirajoshi, hijaz, bhairav, pelog), with motion-controlled vibrato and tremolo.

Each hand's volume responds to camera distance; a closed fist mutes one voice without stopping the other. Built-in video recording with native sharing, to document performances. Bilingual IT/EN. A case study for harmony teaching and interaction design.