Gesture-based interactions with a shop window display

The interactive shop window display now allow users to control content with simple hand gestures, offering a touch-free experience.

Initial situation

In the contemporary retail landscape, numerous shops leverage large screens as a way to communicate information and entice potential customers. Despite their widespread use, these displays are for one-way communication, lacking interactivity. To add interactivity in this situation, one challenge arises from the common placement of these displays behind shop windows, making traditional touchscreen approaches impractical. Thus, the quest for another method to capture user input is necessary, paving the way for pioneering solutions that reimagine the possibilities of customer engagement within the retail environment.

Project goal

By introducing gesture-based interaction, the system creates a dynamic and engaging way for customers to explore displayed content. This approach aligns with the evolving expectations of modern consumers, who increasingly value innovative and interactive shopping experiences. Additionally, the contactless nature of the system directly addresses hygiene concerns, providing a safer alternative to traditional touch-based interfaces.

Solution developed

The result is an interactive shop window where users can explore content with simple hand gestures movements. With MediaPipe, we were able to use an RGB camera to recognize gestures through light and glass reflection to reduce resource costs. The system is designed to be intuitive, hygienic due to it non-touch based interaction system and provide users with a modern experience. Through careful system performance testing, usability testing and agile improvements, we have created a reliable and user-friendly solution that is tailored to the needs of the retail industry.

• The shop window can be navigated by swiping left and right with the hand.
• Gesture recognition works through machine learning.
• The gesture recognition rate through usability testing is 80% accurate.
• Contactless interactions have become a priority, with 71% of consumers preferring touch-free solutions post-COVID-19.
• RGB cameras, combined with advanced ML software, provide a cost-effective alternative to depth sensors for gesture recognition.
• Our interactive shop window can increase the attention of passers-by more than a regular shop window.

Key terms

• InterfaceAn interface is the point of interaction between a user and a system, enabling communication and control through elements like visual displays, buttons, or gestures.
• Gesture-based interactionGesture-based interaction allows users to control digital systems or interfaces using natural hand movements, without the need for physical contact.
• One-way communicationRefers to a communication model where information flows in a single direction, from the sender to the receiver, without the opportunity for feedback or interaction.
• Modern consumerIs an individual who values convenience, personalization, and innovation in their shopping experiences, often driven by digital advancements.
• MediaPipeIs an open-source framework developed by Google for building multimodal applied machine learning pipelines. It provides pre-built solutions for tasks like gesture recognition, face tracking, and object detection, using real-time data from cameras or other sensors.
• Machine learning (ML)Machine learning is a subset of artificial intelligence that enables systems to learn and improve from experience without being explicitly programmed.
• PipelineA pipeline in computing refers to a structured sequence of processes or steps through which data flows to achieve a specific outcome.
• System performance testEvaluates a system's responsiveness, stability, and resource usage under various conditions to ensure it meets performance requirements.
• Usability testUsability testing is a method of evaluating how easily and effectively users can interact with a product or system. It involves observing real users as they complete tasks to identify design issues, gather feedback, and ensure the product meets user needs and expectations.
• AgileAgile is a flexible project management methodology focused on iterative development, collaboration, and adaptability to change.
• RGB cameraAn RGB camera captures images using three primary color channels: red, green, and blue, which combine to produce full-color images.
• Depth sensorA device that measures the distance between the sensor and objects in its field of view, creating a 3D representation of the environment.

Customer

ShopWindow
Selim Jorns
kontakt.de@shopwindow.app
+41 76 706 05 57
ShopWindow

Team

Adrian Miller
Shukanand Ananthavel

Advisor
Kevin Kim
Luca Fluri