Where it started
When someone raises a hand and asks “How many?”, text alone misses the context. The application needs to connect perception, language, speech and character-state feedback.
My contribution
The KKChatBot series explores frontend and backend integration, multimodal input and character feedback. The public repository includes Vue 3, FastAPI, gesture recognition and chat interfaces.
Natural interaction means understanding what someone says and noticing what they do.
Three engineering decisions
Use vision as context
Connect MediaPipe gesture recognition and recent visual observations to conversation logic.
Give deterministic events a short path
Handle selected gestures directly instead of always waiting for a model. Documentation speed claims are not presented as measured results.
Make state perceptible
Use thinking and talking character assets alongside streamed feedback, with interruption and call-ending interactions.
What came out of it
- The public repository contains frontend and backend code, container deployment configuration and interaction documentation.
- Talking and thinking character clips are retained. The playable clip below demonstrates an asset only; it does not connect to a model or camera.
- No unverified accuracy, latency, user-count or production-scale metrics are used.
Public source and documentation; no production-scale claims
Character asset clip
A character animation from the public project. Plays offline; not a live AI conversation or performance demonstration.
What's next
Add reproducible multimodal recordings and end-to-end evaluations for each input path and its failure handling.
Project notes are grounded in reviewed source, experiment records and personal confirmations.
Content reviewed: 2026.10.28