Works and composition tools
Phonk Hand
A two-hand gestural interaction study. The webcam becomes an instrument for harmonic accompaniment and melodic improvisation (MediaPipe, Tone.js).
- Author's contribution
- Concept, mapping between gesture and sound, development.
- Status
- Study
The project
A study in hand-sound gestural interaction. MediaPipe Hands tracks both hands from the webcam and translates them into a Tone.js audio engine with always-on oscillators, modulated without envelopes for pitch stability.
Two performance modes grounded in real music theory. CHORDS, for vocal accompaniment. The left hand selects the degree of a progression, the right hand the chord type, wrist rotation the inversion. MELODY, for improvisation. The left hand picks one of the seven diatonic chords, the right plays over one of thirteen scales (pentatonics, diatonic modes, blues, hirajoshi, hijaz, bhairav, pelog), with motion-controlled vibrato and tremolo.
Each hand's volume responds to camera distance; a closed fist mutes one voice without stopping the other. Built-in video recording with native sharing, to document performances. Bilingual IT/EN. A case study for harmony teaching and interaction design.
Features
- Both-hands tracking
- CHORDS and MELODY modes
- 6 progressions, 12 keys, 13 scales
- Camera-distance volume control
- Gestural vibrato and tremolo
- Video recording and sharing
Characteristics
- Audio
- AI
Tech stack
- JavaScript
- MediaPipe Hands
- MediaPipe FaceMesh
- Tone.js
- Web Audio API
- MediaRecorder
- Web Share API
- PWA
Collaborations and research
If this work is relevant to your field, profile and contacts are on the main site.
Contact on francescomariano.art