Using Machine Learning for IoT

THE FUTURE OF SMART APPLICATIONS IN HEALTH AND ENERGY

Maestro SmartBox

Home: Welcome

Home: About

About

Using Machine Learning for IoT is a Human-Computer Interaction project which uses ambient sensing to infer human activity. The goal is to enable smart applications in healthcare and energy. We are using Maestro, a hardware system which consists of two modules: the sensing node which contains external sensors measuring IMU, PIR, color and illumination, audio, pressure, humidity, temperature and is attached to an Arduino Uno, and the Raspberry Pi which is the central unit and acquires data from these sensors. The use of sensors in conjunction results in 18 data points per unit time. The device collects data in real-time at a rate of 30 samples/sec and updates the centralized database running on the local server every 10 seconds.

These components of Maestro allow for various functionalities such as occupancy counting, user identification, and activity recognition. Past research in our lab has shown that active learning is significant and the accuracy of the system increases with the number of queries. These were queries to receive the label associated with the activity that was being performed at that moment in time. The goal is to develop a user interface with some shared functionalities with the previous webpage being used for data visualization and labeling but meant for a different purpose.

Human-Computer Interaction

Studying and understanding the influence that computers (and technology in general) has on its users, humans, and vice versa. This also involves improving systems to enhance accessibility and the user experience.

Ambient Sensing

Detecting and reacting to human presence using ambient sensors which measure characteristics such as light and motion in the immediate surroundings

IoT

The Internet of Things which refers to the network of physical objects which interact with sensors, software and other technologies to connect and exchange data with other devices and systems over the internet

Raspberry Pi

A small integrated circuit which supports the features and functionalities of a computer. It can be plugged into a computer monitor or TV and can be used with a keyboard and mouse

Arduino

Microcontroller board with digital and analog input/output pins that can be interfaced to sensors and other circuits in the form of shields

Sensors

A small a device which detects or measures a physical property and records, indicates, or otherwise responds to it

Goals for the Summer

Objective

The main responsibility for this summer includes constructing the user interface to align and display data from various streams. I will be graphing the sensor data in a dynamic line graph to reflect the changes to the values in real time. Each data value will have a separate graph, so there will be a total of 18 plots on the webpage, updating every 10 seconds. This will be synchronized with video footage from a camera so that along with the visual representation of the data, hopefully patterns and/or trends can be observed based on what someone is doing.

Nandana used HTML and Javascript to create the webpage. The code includes the functionalities to store data from each sensor in an array and plot the points on the corresponding graphs, as well as to communicate with the database to acquire this data. An open source tool known as chart.js supports the plots and associated aesthetic features on the webpage. Nandana has also been working on ensuring the functioning of the overall system on the backend. She has worked extensively with the data collection process on the PostgreSQL database. Additionally she has worked with Linux on the server to write python scripts to query from the database every 10 seconds and "grab" data and send it to the webpage using MQTT, an open source messaging tool.

Dhyanashri joined the team recently as a high school intern and she has been helping with the webpage design using HTML and CSS. Some changes she has implemented are to improve the design to look more professional and increase readability.

This is a block diagram displaying all of the components that are a part of the system, both hardware and software. The solid lines represent hardwired, or physical, connections where the dashed lines represent online connections

Beyond the Summer

Home: Objective

Future Work

The plan is to eventually deploy this system into the industry as a product to be used for ambient sensing with a live demo experience through the webpage which will be accessible to the user. The system will consist of multiple maestro boxes collecting data in conjunction for more accurate capabilities to determine human occupancy and infer human activity. However, before Using Machine Learning for IoT is ready for this real world use, further developments need to be made to the system.

We need to establish an experimental collection for practical and more realistic data collection. The goal is to establish the method for data collection and outline what participants would have to do. While the testing procedure may seem invasive or violating privacy because there will be a camera constantly monitoring an individual, this is only so that the ML model can be most accurately trained to infer human activity and help recognize any visual trends. Going forward, having a well-trained model would no longer require the use of a camera constantly watching someone. The camera module is being implemented strictly for testing purposes to serve as a supplemental source of data to the values from the sensor.

The hardware maestro box is currently a bit large due to the Raspberry Pi with a protective case and heat syncs as a separate unit from the Arduino with a PCB and the sensors, with both being connected via USB cable. Other members of the lab are working on a PCB that attaches directly to the Raspberry Pi and has all the sensors embedded, eliminating the need for the additional Arduino. Once complete, this will make the maestro box more sleek and compact.

Some immediate next steps would be to expand the functionalities of the webpage. Another teammate in the lab is working on a mobile application for active learning which will have a voice agent that will determine when it is an appropriate time to interrupt the user and ask what they are doing. This will allow for the data to be labeled with an action or activity, which is beneficial when we may not be able to determine exactly what an individual is doing from the sensor values and the video feed. Once this is set up, it would be ideal to form a connection between the mobile app and webpage so that an occurrence of a prompted question would be visible in the live timeline on the graphs, with the user's response being displayed at the corresponding time.

Home:Future Work

Weekly Progress

Week 1

Used MQTT to establish trial connection between personal Raspberry Pi and temporary webpage
Displaying basic graphs on webpage with randomly generated values