Classification of product changes in an automated retail environment.
Python, Machine Learning, Image Processing, Computer Vision
The project aimed to address several critical questions and challenges in the context of classifying product changes into “grabs”, “drops”. Currently, existing approaches require two cameras to accurately detect and classify these changes on store shelves. This requires time to install to calibrate both cameras perfectly and is therefore more costly. The central question was whether the current approach could be effectively translated into a single camera view with minimal loss of performance, thereby lowering the barrier to entry for potential clients. Detecting changes in product positioning was a vital aspect of this project, and investigating additional processes that can be applied to enhance change detection capabilities was a primary goal of the project. The image below shows a pair of images from the dataset, depicting the before and after images of a product change in the shelf.
The image below shows the different classes that need to be classified. For this project, we focused exclusively on the "grab" and "drop" classes.
AIRS’s long term goal is to be able to create a fully automated retail store. This would be done by cameras and an AI System that take note of which products a customer has taken from the shelves and would have to pay for when they leave. With an automated store, fewer employees are needed, and theft can be mitigated. Currently AIRS uses two cameras per shelf to determine changes in a shelf. This way, the system can make use of depth information that can be calculated from the two captured images. This stereo camera setup has some major downsides: it needs to be precisely calibrated and maintained with the correct angles and distances between the cameras and the shelf, and it uses double the cameras that a single camera setup would use.
The study has found that with state of the art machine learning techniques the transition from a stereo to a mono camera setup is feasible. On the provided dataset an accuracy and f1-score of 97.34% was achieved. Initially, a baseline model was developed to serve as a fundamental reference point. Other models were then developed and compared against this baseline model. The image below shows results of the baseline model. The learning curve and confusion matrix, along with the accuracy and f1-scores were the metrics that were used to evaluate the performance of the models.
Project Duration: 18/09/23 - 19/01/24
Person hours: 360 hours
Team size: 2
AI Retailer Systems
Homepage
Christopher Trachsel, BSc in Computer Science
Janic Berger, BSc in Computer Science
Prof. Dr. Hilko Cords
hilko.cords@fhnw.ch
