Your Guide Through the Store
Our Commitment
Innovative Technology
Innovative Technology
At Perceive, we are dedicated to making grocery shopping accessible for all individuals. We believe that everyone has the right to shop without barriers.
Innovative Technology
Innovative Technology
Innovative Technology
Our assistive shopping app leverages cutting-edge technology to provide real-time assistance. Users can easily identify items on their shopping lists with our intuitive interface.
Designed for Users
Innovative Technology
Designed for Users
Our app is built with a focus on the needs of the visually impaired. We offer a hands-free experience that allows users to navigate stores with confidence.
Our Shared Vision
We are working toward a world where grocery shopping is intuitive, independent, and accessible to everyone, regardless of vision. Beginning with a mobile app and expanding into wearables, we are building technology that doesn’t just assist, but empowers people to navigate everyday life with confidence and ease.
How Perceive Works
Using our app is simple—designed with visually impaired shoppers in mind:
- Create a voice-powered list before you head to the store.
- Walk through the produce section.
- Receive audio alerts when an item is nearby.
- Check it off. Keep going.
- No camera handling. No guessing. Total independence.
System Architecture
Our architecture enables real-time, on-device object detection filtered by user input, triggering cl
- Connection & Setup: Upon app launch, the device connects via API Gateway, registering its unique device and websocket IDs for session management.
- User Interaction & List: The user interacts with the mobile app (UI) to build a shopping list.
- On-Device Object Detection: The app uses the device camera and an integrated YOLO model for real-time object detection from the video feed. It can filter detections based on the user's shopping list.
- Cloud Upload (on Match): When a relevant object is detected (matching the list), the app (via Amplify) uploads the corresponding frame to an S3 bucket. The image file name includes the device ID and detected object.
- Backend Processing Trigger: The S3 upload triggers a Lambda function.
- Device Lookup: Lambda retrieves the device ID from the S3 object and queries a DynamoDB table to find the corresponding websocket connection ID.
- AI Guidance Generation: Lambda sends the image and a predefined prompt to a multimodal LLM hosted on SageMaker. The LLM returns text-based guidance related to the detected object.
- Response Delivery: Lambda uses the retrieved connection ID and API Gateway to send the LLM's text response back specifically to the originating device's websocket connection.
- User Feedback: The mobile app receives the text data and provides audio feedback to the user.
