This project is supported under the
European Union's Horizon 2020
research and innovation programme under
Grant Agreement No. 965486
MIMEX’s Visual People-Tracking System
Unmanned micro-markets, like those in development in the MIMEX project, are designed to be completely self-service shops where customers’ shopping activities and payment systems are handled by distributed sensors equipped with artificial intelligence.
by Paul Chippendale
Such systems must guarantee robust performance 24-7, 365 days a year, whether day or night, during weekdays or over the holidays, without the need for permanent human staff. Unmanned micro-markets must grant customers the same, or higher, levels of service quality compared to conventional shops, both in terms of products and the services they offer, with the added advantage that shopping becomes a just-walk-out experience. To achieve this high-level functional goal, multi-sensor technologies need to be exploited that take advantage of state-of-the-art camera networks, audio systems, RFID (radio-frequency identification) and UWB (ultra-wideband) sensors. The process of shopping and also operating/maintaining this new form of shop must be user friendly, intuitive, unobtrusive, extremely accurate, and respectful of privacy policies. Flexibility is key, as products change their appearances, shop layouts change and product form factors are modified to match consumer trends, the season or current promotions.
Computer vision systems play a crucial role in this flexibility. Cameras connected to AI units can interpret a wide variety of visual signals,providing new and interesting solutions for shopping automation. Camera systems are easy to install and they are able to track products and people in shops without the need of additional and invasive tags. Such vision systems can monitor the flow of customers over time and provide valuable shopping analytics about habits, preferred routes, effective marketing/display strategies, etc. This data can help shop owners/operators to better organize the placement and displacement of products on shelves, helping to improve experiences and indicate where potential shopper bottlenecks may occur, that could lead to gatherings that disappoint customers or create COVID pandemic issues.
Shopper/researcher analysis tool for SmarTrack. Customers are assigned different colour particles and bounding boxes. Faces (on the left) are blurred for privacy reasons. If people are closer than a certain distance apart (e.g. 1.5m), a red vertical bar appears next to their face. Occupancy likelihood is shown in the top left box.
In MIMEX, people tracking is performed using the patented multiple camera system SmarTrack, a technology developed by Fondazione Bruno Kessler (Italy). SmarTrack has already been used for surveillance applications, ambient assisted living and sports analysis. The technology detects people by processing shape and colour features from a calibrated multi-camera network. Tracking is performed in a 3D space through a Bayesian probabilistic approach exploiting particle filtering. This method determines people's locations in 3D space with high accuracy, even in the presence of occlusions and changes of scene illumination. SmarTrack also integrates a human-joint-separable model which enables the detection of finer person details such as head, chest, arms and legs, which can lead to the generation of richer customer analytics, like the direction of a shopper’s gaze towards product displays. In this way, the output of SmarTrack can be integrated with outputs from other sensors in the micro-market, e.g. RFID and UWB sensors placed on products, to better understand shopper preferences.
The performance of the visual tracking system is being evaluated in the MIMEX Testbed created in FBK: a micromarket-like environment equipped with four calibrated cameras mounted in the four corners. Shelves have been stocked with products, and mock-shoppers are encouraged to pick-up products, place products in their baskets, and also pick-up and replace products on shelves, whilst being tracked. SmarTrack’s task is to detect and track each customer and then create a shopping map of their habits. The live output from the tracker can be monitored through a custom interface that shows the four images captured by the four cameras over time (see the figure below).
Shopper tracks can be downloaded for further analysis, revealing statistics like the most attractive shelves/products, preferred routes through a shop or the customer throughput. Preliminary experiments are being evaluated qualitatively, showing that SmarTrack is an effective tool for detecting and tracking people as well as for controlling interpersonal distances between customers.