IoT Project -- Smart Refridgerator Assistant

Posted on Edited on In

Introduction

This project is the final project for my CSS532 of UW CSS. Health is a significant topic as we always talk about, the foods we eat every day have very important effects on our body health. The refridgerator is the place where most of families store the foods, however, life is so busy that we also forget what we have put into our food store and what time we put them in. Normally, we have wasted the freshness of the good food even have some food gone bad and pollute the whole store. So, to address this, I want to create a smart refridgerator assistant who can take care the problems.

The idea is about recording the foods in your refridgerator and remind you of what we have and when we put them in. To implement a small prototype to make the requirements come true, I need to combine the techniques of IoT, Cloud computing and Machine Learning. With the help of this smart refridgerator assistant, we can avoid forgetting foods in our refridgerators and keep track on the freshness of them. The whole picture looks like that the assistant is going to monitor the refridgerator and send message to IoT hub, the cloud will store the food status and send notification to our mobile phone app and let us know the status of our refridgerator, so I can know what we have in our food store and decide the what to eat next.

Project Presentation

Technique Background

Devices

This project uses two devices, one is Raspberry Pi served as monitor device, the other is an Android mobile phone served as an end device. Two devices are not connected directly, AWS services act as the communication media. The lambda function will control the message flow, and all the messages are over MQTT protocal.

Raspberry Pi

The Raspberry Pi is a series of small single-board computers developed in the United Kingdom by the Raspberry Pi Foundation to promote teaching of basic computer science in schools and in developing countries. The one I use in this project is Raspberry Pi 3 Model B+. The Raspberry Pi is also equiped with a video camera for image capturing. 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Raspberry Pi 3 Model B+

CPU:     Quad core 64-bit processor clocked at 1.4GHz
Memory:  1GB LPDDR2 SRAM
Network: Dual-band 2.4GHz and 5GHz wireless LAN
         Bluetooth 4.2 / BLE
         Higher speed ethernet up to 300Mbps
Power:   Power-over-Ethernet capability (via a separate PoE HAT)
++++++++++++++

The Arducam 5 Megapixels 1080p Sensor

- High-Definition video camera for Raspberry Pi Model A or B, B+, model 2, Raspberry Pi 3,3 B+
- 5MPixel sensor with Omnivision OV5647 sensor in a fixed-focus lens
- Integral IR filter
- Still picture resolution: 2592 x 1944
- Max video resolution: 1080p

Raspberry Pi
Raspberry Pi

The Raspberry Pi device provides all runtime environment for the monitor part of this project, the device is installed with Raspbian and Python environment. The camera driver is provided by the default system.

Android Device (emulator)

As for Android Device, I use the android emulator provided by Google Android Studio for its pure Android OS. The client application will run in the emulator.

Android Device

AWS

AWS provides dozens of services that support our needs. In this project, I mainly use the service of IoT Hub, Dynamodb, Lambda, Cognito and CloudWatch.

IOT Hub

AWS IoT Hub is a managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices. In this project, IoT Hub is used to manage the communication between the Raspberry Pi and the cloud. It handles the messages sent from the Raspberry Pi and routes them to the appropriate AWS services.

Lambda

AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers. In this project, Lambda functions are used to process the data received from the IoT Hub. These functions perform operations such as storing data in DynamoDB, sending notifications, and triggering other AWS services.

Dynamodb

Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. In this project, DynamoDB is used to store the data related to the food items in the refrigerator. Each item is stored with details such as name, quantity, and the date it was added to the refrigerator.

Cognito

Amazon Cognito lets you add user sign-up, sign-in, and access control to your web and mobile apps quickly and easily. In this project, Cognito is used to manage user authentication and authorization. It ensures that only authorized users can access the data and perform operations on the smart refrigerator assistant.

CloudWatch

Amazon CloudWatch is a monitoring and observability service built for DevOps engineers, developers, site reliability engineers (SREs), and IT managers. In this project, CloudWatch is used to monitor the performance and health of the AWS services used. It collects and tracks metrics, collects and monitors log files, and sets alarms.

Machine Learning

In this project, the Raspberry Pi runs the recognition code to identify the food items in the refrigerator. This involves several machine learning techniques.

Image Recognition

Image recognition is the process of identifying and detecting an object or a feature in a digital image or video. In this project, image recognition is used to identify the food items in the refrigerator. The Raspberry Pi captures images of the food items and uses image recognition algorithms to identify them.

Deep Learning

Deep learning is a subset of machine learning that uses neural networks with many layers. In this project, deep learning techniques are used to improve the accuracy of the image recognition algorithms. The neural network is trained on a dataset of food images to recognize different types of food items.

Caffe

Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by the Berkeley Vision and Learning Center (BVLC). In this project, Caffe is used to implement the deep learning algorithms for image recognition. The trained model is deployed on the Raspberry Pi to perform real-time recognition of food items.

System Design

Implementation

Reference