How Amazon's Just Walk Out Works
The amazing technology behind
If you ever wanted to automate lead sourcing on LinkedIn through AI, here’s a workshop from our GrowthSchool, book a spot asap! (100 free seats only):
In this AI Powered LinkedIn Workshop, get hands-on with solutions to key challenges for growth-minded product professionals:
👉 Automate lead sourcing with AI, creating a consistent flow of qualified prospects for your product’s growth.
👉 Quickly analyze customer data with AI to prioritize features and respond to market needs faster.
👉 Drive engagement and boost product traction using Vaibhav’s proven $100K LinkedIn Strategy for measurable growth.
Amazon has been trying 17 years to reinvent how people do physical grocery shopping.
So far, we have been doing grocery shopping pretty much the old way:
Customers typically select items, proceed to a checkout line, and wait for a cashier to scan their purchases and process payment.
This process can be time-consuming, especially during peak hours when lines are longer.
Shoppers often interact with cashiers for payment and assistance, which can vary in efficiency and satisfaction.
But what if customers can selecting their items and then walk out?
That is what Amazon’s Just Walk Out solutions tries to achieve.
Arthur C. Clarke's Third Law states that "any sufficiently advanced technology is indistinguishable from magic"
But what is even more fascinating is the tech behind this Walk Out technology.
In today’s newsletter, we deep dive into various components that make this experience happen.
The total experience has been eight years in the making, and continues to evolve.
What makes this tech happen?
Amazon's Just Walk Out technology is currently implemented in over 170 third-party locations across the U.S., UK, Australia, and Canada, including various venues such as airports, stadiums, universities, and hospitals
In addition to these third-party stores, the technology is also utilized in Amazon's own retail locations, particularly in 22 Amazon Go stores that operate primarily in major cities.
The Walk Out technology has been possible through:
Computer Vision
Object Recognition
Advanced Sensors
Machine Learning Models
Generative AI
All of these technologies, systems and algorithms have to work perfectly well so that when a customer picks up a Caesar Salad, tech can detect what’s picked up, charge them correctly and then allow them to walk out.
Other complications may arise, when a picked up item is returned.
This seamless experience is powered by a sophisticated integration of various technologies. Below is a detailed exploration of the core components that make this system work effectively.
Computer Vision
At the heart of Amazon's Just Walk Out technology is computer vision, which allows the system to interpret and understand visual data from the store environment. This technology employs numerous cameras strategically placed throughout the store to monitor customer movements and interactions with products.
Tracking Shoppers: The system continuously tracks each customer from the moment they enter until they exit. This involves detecting and identifying individuals, even in crowded or occluded situations, where visibility may be compromised.
Pixel Analysis: Each camera captures images that are processed to identify distinct pixels associated with shoppers. The system can differentiate between individuals based on their movements and interactions without collecting biometric data, ensuring privacy
Real-Time Processing: The computer vision algorithms operate in real-time, allowing immediate updates to the virtual shopping cart as items are picked up or returned to shelves. This capability is crucial for maintaining an accurate account of purchases
Object Recognition
Object recognition is a critical aspect of Just Walk Out technology, enabling the system to identify various products accurately.
Training with Data Sets: The technology has been trained using extensive datasets, including millions of synthetic images and shopping scenarios. This training helps it recognize a wide variety of items under different conditions, such as varying lighting and store layouts
Handling Misplaced Items: If a customer returns an item to a different location, the system adjusts accordingly by utilizing both visual cues and weight sensors on shelves to ensure accurate inventory management
Small Item Detection: For smaller items that may not be easily visible to cameras, weight sensors are integrated into shelves. These sensors detect when an item is picked up or returned, assisting in accurate tracking
Advanced Sensors
The integration of advanced sensors enhances the capabilities of Just Walk Out technology beyond what computer vision can achieve alone.
Load-Cell Sensors: These sensors measure weight changes on shelves when items are picked up or returned. They work in conjunction with cameras to provide additional data points for item recognition
Sensor Fusion: By combining inputs from various sensors (cameras and weight sensors), the system can more accurately determine which items have been selected or returned. This fusion of data improves overall accuracy and reliability in tracking shopper behavior
LiDAR Technology: Amazon also utilizes LiDAR technology for creating three-dimensional maps of store layouts, enhancing spatial awareness for tracking shopper movements
Machine Learning Models
Machine learning plays a pivotal role in processing data collected from computer vision and sensor inputs.
Multimodal AI Models: Amazon has developed new AI models that analyze data from multiple sensor types simultaneously. This approach improves efficiency in identifying items taken from shelves and generating receipts quickly
Continuous Learning: The machine learning algorithms are designed to adapt over time, learning from new shopping behaviors and improving their accuracy in item recognition and shopper tracking. This continuous improvement is essential for maintaining high service levels as store environments evolve
Generative AI Techniques: The latest models leverage generative AI methods similar to those used in applications like ChatGPT, allowing for sophisticated pattern recognition and decision-making processes within the shopping environment
Generative AI
The application of generative AI within Just Walk Out technology enhances its ability to simulate various shopping scenarios for training purposes.
If you’re interested in learning how the AI behind Amazon Walk Out works, go here: How the Amazon Go Store’s AI Works | by Ryan Gross | Towards Data Science (also the source behind above image)
Synthetic Data Generation: Amazon uses generative AI to create synthetic training data that mimics real-world shopping experiences. This includes variations in customer behavior, product placement, and environmental conditions, which helps train the system more comprehensively
Improving Accuracy and Efficiency: By employing generative AI techniques, Amazon aims to improve the accuracy of its models in real-time operations while also streamlining the receipt generation process for customers upon exiting the store
Future Applications: As generative AI continues to evolve, it may provide further enhancements to Just Walk Out technology, potentially allowing for even more complex interactions and better customer service capabilities.
The Pivot: Walk Out Technology to Dash Carts
Amazon has recently decided to phase out its Just Walk Out technology from some of its grocery stores, particularly Amazon Fresh and Whole Foods locations in the U.S.
This shift is primarily due to the high operational costs and challenges associated with maintaining the technology, which relies heavily on computer vision and AI to enable cashier-less shopping experiences.
Reports indicated that a significant portion of transactions using this technology required manual verification by employees, undermining the initial promise of a fully automated system
The Just Walk Out technology, while innovative, has faced difficulties in accurately processing transactions without human intervention. It was reported that around 70% of transactions needed manual review by workers in India, which contradicted the notion of a fully automated system
Customers expressed a preference for more interactive shopping experiences that include features such as locating products easily and tracking savings while shopping, which led Amazon to reconsider its approach
As part of broader cost-reduction strategies amid changing economic conditions, Amazon is reevaluating its retail ambitions, which has included closing several retail chains and halting new store launches.
Instead of Just Walk Out, Amazon is now focusing on Dash Carts, which are smart shopping carts designed to enhance the shopping experience by allowing customers to track their selections and avoid long checkout lines.
These carts provide functionalities such as:
Product Location Assistance: Helping shoppers find items within the store.
Deal Access: Allowing customers to view deals while they shop.
Receipt Review: Enabling customers to see their receipts as they shop, enhancing transparency and satisfaction
Despite this pivot in its grocery segment, Amazon continues to expand Just Walk Out technology in other settings, including airports, stadiums, and universities, where it remains popular for its convenience and efficiency