Design and implementation of a collaborative and interactive visualization with voice input

A collaborative game that translates voices into lively, shared visualisations. Multiple users can sing at the same time, creating a colourful underwater world together, a mixture of artistic experiment, game and edutainment.

Text
Main image: Excerpt from the actual game Startscreen

Initial situation

Singing is not only enjoyable on its own, it is even more exciting when done together. Inspired by the joy of making music together, the idea arose for an exhibition in which groups create not only sound but also fascinating visuals through singing. The aim was to use the voices of several people as input to create an interactive, visual experience that invites both experimentation and learning.

Problem statement / Project goal

How can singing voices be processed and visualised as input for a game in such a way that the result is a collaborative, motivating experience that is accessible to everyone? The technical challenge was to evaluate audio parameters such as volume, pitch, brightness and harmony in real time and visualise them in a way that is not only fun, but also promotes learning and strengthens interaction within the group. The goal was to design and implement a system that turns voices into visible, playful patterns and makes the connection between sound and visualisation intuitively tangible.

The calibration
                            
                            At the beginning, all participants calibrate their microphones and individual voice pitches. Personal minimum and maximum values for pitch, volume and brightness are determined in order to optimally adjust the game to each voice and ensure fairness.                        
The game
                            
                            Colourful corals grow in the underwater world, fish appear, bubbles form, all of this is created and changed live by the voices of the players. Each parameter (e.g. volume, pitch, brightness) has its own visual effect, and harmony between multiple users creates additional animations.                        
The end screen:
                            
                            At the end, everyone receives a personalised evaluation. The most important voice parameters are explained graphically and in simple terms, with humorous analogies (‘your average frequency corresponds to a guitar, for example’), so that everyone immediately understands what they have achieved.

Solution developed and its benefits

A web-based, collaborative game was developed in which up to four users can contribute their voices simultaneously via their own microphones. Using robust backend logic, various voice parameters are analysed and sent in real time via WebSocket to a high-performance visualisation (PIXI.js) in the frontend. The frontend translates the audio data into a colourful underwater landscape in which each voice becomes visible. Various mini-games encourage interaction and make the musical parameters intuitively tangible.

Real-time visualisation of multiple voices as a shared game world
Individual calibration for each voice
Interactive feedback: corals, bubbles, fish depending on voice parameters
Harmonies are rewarded with special animations
Several mini-games for motivation and playful learning
Clear, easy-to-understand end screen with personal evaluation
Barrier-free, low-threshold access, no prior musical knowledge required
Modular architecture: Python (backend) and JavaScript/PIXI.js (frontend)

The system was developed iteratively, based on usability tests and real user feedback. Test subjects particularly highlighted the motivating game environment and the immediately comprehensible feedback. The solution combines technical performance with artistic visualisation, creating a unique, motivating learning experience for all age groups.

In practical use, the game has proven to be accessible, robust and intuitive for a wide range of users – from spontaneous group performances to educational applications in exhibition or teaching contexts.

Testimonial
The Project presents a creative and unusual concept: visualising the growth of corals in an underwater world through pitch, timbre, and other vocal qualities. The idea is charming and offers a fresh perspective on how voice input can be transformed into a playful and interactive tool. The students demonstrated curiosity and a willingness to experiment with unconventional mechanics. While there is certainly room for refinement and the handling of vocal parameters may not be entirely intuitive, they succeeded in creating an engaging and visually appealing experience. Collaboration throughout the process was exemplary, and the result invites a joyful, playful engagement with the voice. Gabriel Imthurn

Key terms

Voice Visualization Votes are made directly visible
Real-time Audio Processing Processing and visualisation without noticeable delay
Collaborative Game Interaction and fun arise from working together
PIXI.js, Python, WebSocket Modern web technologies for performance and flexibility
Edutainment Learn and experiment through play

Customer

Head of the Chair of Music Education in Adolescence, PH FHNW
Prof. Dr. Gabriel Imthurn
Bahnhofstrasse 6
5210 Windisch

Team

Luca Bertonazzi
Marco Worni

Advisor
Prof. Dr. Kevin Kim