Professional Documents
Culture Documents
CST 499
Joe Sarabia
Executive Summary
This project enhances the usability of home automation by correlating user location and
user activity within a home to build behavioral models of individual users which can then used to
automatically apply state based on detection of a user and other environmental factors. This
project affects all home automation users. This project delivers a functional prototype
demonstrating how machine learning and home automation could be combined to achieve
enhanced usability.
3
Executive Summary 2
Overview 5
Project Description 5
The Issue in Technology 6
Solution to the Issue 6
Feasibility Discussion 13
Paper 1: Bluetooth Low Energy for Smart Home Energy Management 13
Paper 1: Options and Justifications 14
Paper 2: Machine Learning On Real-Time Data to Enhance Home Automation 15
Paper 2: Options and Justifications 15
Design Requirements 16
Functional Decomposition 16
Selection of Design Criterion 18
Final Deliverables 18
Methodology 18
Milestone 1: Market Analysis and Feasibility Assessment 19
Milestone 2: Procure Equipment 19
Milestone 3: Prototype Implementation 20
Milestone 4: Prepare Demonstration 20
Ethical Considerations 20
Ethical Concerns 20
Negative Impacts 23
4
Environmental Impacts 24
Final Implementation 31
Beacon Devices 31
Home Assistant and Raspberry Pi 32
iOS Application 34
Estimote Cloud Manager 35
Estimote Proximity Manager 37
Conclusion 40
Vision of Projects Future 41
References 42
Appendix 43
Home Assistant Configuration 43
MQTT for Room Presence Detection Configuration 44
Detecting Lights On after Entering Room 44
Event Samples for State Changes 46
5
Overview
first part of the the project was research-based and incorporated a survey of the industry in order
to hone in on a specific area of focus for the second part of the capstone. The second part was
Project Description
While substantial progress has been made towards delivering on the promise of the smart
home of tomorrow, even the modest smart home capabilities of today generally require a
thereby excluding vast portions of this technologys potential user base. This research
investigated the extent to which todays technology can be used to create a learning home by
implementing precise indoor location tracking combined with a state machine tracking device
usage from which an artificial intelligence (AI) could be derived which would be able to learn
the behavior of its occupants and derive the desired state of the home at any given time based
upon a combination of parameters such as time of day, biometrics, weather conditions and the
specific location of a particular occupant or combination of occupants within the home. The
ultimate goal is for this state engine to be able to control common Internet of Things (IoT)
devices such as lighting, music speakers, electrical outlets, appliances, and televisions. The
research delivers a functional prototype of this state engine demonstrating control of a subset of
home? This research-based portion of the capstone project sought to answer whether particular
hardware such as IoT devices and presence detection devices can be used to accurately and
precisely track the location of specific individuals within a home. Furthermore, this research
sought to answer whether a software implementation can be created to detect and track the
change of state of IoT devices within the home and correlate that to the location of a specific
individual within the home. Finally, this research sought to answer whether an AI can be built to
learn the behaviors of a homes occupants and automatically apply desired state based upon
applicable parameters such as time of day, weather conditions and other factors which influence
a particular behavior.
Given the findings from the survey of the current state of the industry, this capstone then
shifted toward answering a second question, specifically, how well can todays existing
technology be used to create an early, functional prototype of a learning home? Based upon the
findings from the research-based portion, the project-based portion of this capstone implemented
a prototype which is capable of precisely tracking indoor location of occupants within a home,
observing state changes of IoT devices within that home and then correlating these data together
to produce a recommendation for desired state of a home based upon the location of a particular
occupant and at least one other desirable parameter such as time of day.
The final culmination of this capstone project was an early prototype that effectively
demonstrates the ideas embodied within this research as outlined above. While the machine
7
learning and AI aspects of this project will be left for further study, the prototype does enable a
home to exhibit the capability to accurately respond to particular combinations of inputs based
upon previous learned behaviors and detection of occupants. The idea is to facilitate the
The three most important criteria needed to demonstrate the ideas investigated in this
research are that a change of state occurs based upon a combination of the following factors: a
particular users location within a home, some other pre-condition such as time of day and that it
exhibits control over more than one class of devices and preferably in combination with each
other.
Phase I: Research-based
During the research based portion of the this capstone, a survey of the industry was
conducted in order to identify suitable technologies and their capabilities to fulfill the above use
cases. For purposes of precise location tracking, the objective was to identify 1-2 technologies
which are able to determine specific location within a room. For example, detecting an occupant
in the living room may not be sufficient: the technology should be able to detect the difference
between sitting on the couch in the living room and just walking through it. A requirement for
these technologies is that they have a public API which can be used for integration in the broader
solution. For purposes of detecting behaviors and ultimately automatically applying state or at
least generating a recommended state, an objective of this portion of the research was to identify
2-3 IoT devices with APIs that can be used to read and detect state changes.
8
This research investigated products from both Pixie Technologies which use UWB
technology for RTLS and products from Estimote using a combination of iBeacon technology,
UWB technology and their own technologies for accurately tracking indoor location. While the
initial review of Pixies tracking devices looked promising due to an apparently high level of
precision, their SDK was not extensively documented, and was thus unclear how that capability
could be rapidly converted to a prototype to correlate indoor movement to rooms within a home.
Additionally, the Pixie devices non-replaceable batteries were exhausted after less than 6 weeks,
despite their promise to last one year. Pixie customer support did an excellent job of replacing
the devices, however, such delays posed a major encumbrance to a this projects relatively short
timeline. Ultimately after evaluating both Pixies and Estimotes devices, Estimote was chosen
because of the breadth of its product line, including 4 different types of beacons, along with a
mature SDK, and active developers community, and various sample applications to demonstrate
a variety of product capabilities. Additionally, Estimote has been producing beacons and SDKs
for beacon based applications since the release of iBeacon by Apple and appeared to have the
most mature, most active and most well defined approach to this space.
There are two sample use cases that I explored as early prototypes based upon findings in
the research-based portion of the project. The ultimate goal with these was to investigate
Objective 1: Follow Me
The objective with this use case was to demonstrate the feasibility and a prototype of some
desired state that follows an occupants traversal through various parts of a home. Similar to how
9
light sensors work, this can be thought of a sensor with persona, that is, the sensors know the
difference between multiple occupants such that they only activate for particular occupants under
particular circumstances. One interesting use case to demonstrate here is follow-me audio, where
a multi-speaker, multi-room audio system is able to continuously adapt its state based upon an
occupants location. An example of this could be a playlist that starts playing on a speaker when
an occupant walks into the bathroom to get ready for the day, then switches from the bathroom
speaker to the bedroom speaker while the occupants finishes getting ready and then switches to
the living room speaker when the occupant moves to couch to prepare a to-do list for the day. Its
not necessary and indeed inefficient for speakers to be playing in a room the occupant doesnt
occupy, and this use case is intended to demonstrate the ability for a home to apply a state, in this
case a playlist, and have it follow that user around the home.
across multiple IoT devices or systems applied by a particular occupant in a discernible and
predictable manner. The objective of this use case was to demonstrate the ability to learn and
appropriately apply such a behavior. An example use case of this is a breakfast making routine
where every weekday morning between 8:15a and 8:45a an occupant enters the kitchen, turns on
two specific lights, requests to hear the news from a home assistance device, and then plays a
playlist consisting of 3-4 songs while making and eating breakfast. Before leaving the kitchen,
the occupant turns off the playlist and turns off the lights while heading out the door for work.
The objective of this use case was to learn this behavior and then appropriately apply it if the
occupant is present during the learned timelines. Importantly, this use case should also
10
demonstrate that state changes dont occur for that same occupant in that location at times
previously unobserved, that is, it shouldnt apply the breakfast behavior during dinner, and it
should only apply the behavior for intended occupants and state should be unaffected by the
presence of some other occupant. For example a second occupant passing through the kitchen
during the same time should have no impact on the state of the home.
As previously mentioned, the stakeholders and community for this project are any current
or future home automation user. What this group of individuals stands to gain is an improved
realization of home automation benefits, many of which could be construed as comfort level
benefits. Nonetheless, home automation promises advances in many aspects of ones home
including security and energy efficiency. One common IoT device with interesting capabilities is
the smart camera which can be used to detect activity within zones, and more recently, human
presence in a specified area. This activity detection can then be used to trigger desired home
automation actions. While useful, today these are not personalized. For example, the cameras can
detect a person but not a specific person. This paper asserts that the ability to detect particular
people in particular spaces coupled with behavioral modeling will add greater persona and
reduced rigidity to the usability of home automation. On the other end of the spectrum, another
consideration for the home automation user community is one of privacy. There are some
prospective users of home automation who may have no interest in having cameras storing
images of their activity in some external storage service, or there might be objections to the use
of cameras in particular parts of the home, such as bedrooms and bathrooms. The lack of
cameras in these areas otherwise reduces home automation capabilities which might otherwise be
11
achieved. This paper seeks to overcome such privacy objections by researching feasible options
to trigger actions based on the presence of a particular person in a particular position within a
One particular area of interest that this research sought investigate is to what extent the
techniques discussed herein might benefit disabled people using home automation. As an
example, great strides have been made in speech based home assistant type devices, but an
individual with a speech impediment may not benefit from some of the capabilities such devices
provide to control their home. Additionally, in the case of some disability that encumbers
interaction with physical devices such as lights and doors, this technology could be used to learn
the persons desired behavioral patterns in a home, perhaps by enlisting help from an assistant,
and then once learned, the behaviors could be applied automatically as the person moves around
the house, assuming that mobility is not a restriction, thus proving that person greater autonomy.
Lastly, anybody new to home automation or perhaps intimidated by the technology such
Given the slow rate of adoption of smart home technology and its current barriers to
adoption, this research seeks to reduce or neutralize some of these impediments which inhibit
adopters from fully realizing its potential benefits. It also provides a prototype that establishes a
pattern for other consumer and commercial IoT applications where the application of learned
Who Benefits
Potentially any current or future user of home automation based technologies will benefit
from this research. In particular, classes of users who are otherwise excluded from or
substantially disadvantaged in leveraging home automation and its capabilities today would
benefit.
Why Is It Important
ubiquitous and its capabilities are increasingly impressive. There is frequent conversation in the
national dialogue today amongst policymakers and others about the role of automation and
artificial intelligence in our future economy. On one level, this research is important to
demonstrate the power of artificial intelligence in a tangible way that is readily accessible and to
close the gap between the sci-fi based depictions of a smart home, such as those depicted in Back
to the Future and The Jetsons, with the real capabilities of todays technology. On another level,
I think the realization of these capabilities opens up more possibilities for societal and individual
benefit when our surrounding environments can intuitively adapt to our presence.
Background Information
smart home technologies in my home over the last 3.5 years. Initially, my attempts at
implementing home automation yielded a very low wife acceptance factor, and I realized that not
everyone would find it appealing to control their home from their phone, or even have a phone
from which they could control their home. Ive observed numerous deficiencies in existing
products such as geofences that are orders of magnitude larger than my home and which dont
13
intelligently account for the presence of multiple people within a home. More than once, I might
leave to go to the store and all the lights in the house turned off as I drove away while my wife
was sitting on the couch watching television. This is where the initial ideas behind occupant
between multiple apps on my phone to yield a desired state in my home could be improved. I
realized that I was typically taking the same set of actions from the same location within the
home, and that the this sequence of steps along with precise occupant detection should be
sufficient to create a home that can learn and apply my desired behaviors and truly be called a
smart home.
Feasibility Discussion
A brief review of existing research yielded two results which have some similar concepts
yet not entirely overlapping concepts and proposals. The two papers reviewed are covered
below.
appliances communicate to an energy management unit (EMU), and some intermediary system
between the appliances and the user of the appliance is able to make a recommendations about
the use of the appliance when the user attempts to use it, generally in an effort to reduce peak
load demand and consumptions charges. For example, the EMU might suggest that the electric
clothes dryer be used at a later time when the usage fee for electricity is lower. While this does
involve a use case where home automation interacts with the user which is something The
14
Learning Home project proposes, in this case it occurs through an intermediary system and is not
completed automated based on behavior. The Learning Home proposes that similar activity with
appliances be applied by presence detection. Nonetheless, there could certainly be use cases
where both applications might exist, or where the data collection by the EMU provides an input
into The Learning Homes state engine as a parameter to consider in determining whether some
mechanism for home energy management (HEM). The choice of wireless protocol is critical
according to this paper. BLE connection setup and data transfer occur in a range from 3ms to
1.28s. Tens or even hundreds of milliseconds to transfer data would likely be sufficient to satisfy
the perception of near real-time reaction in the cases proposed by The Learning Home. The
authors point out the substantial BLE penetration in the industry as a major benefit, for example,
its already technology included in commonly owned devices such as smart phones, smart
watches, and tablets. They also point out that general reliability is key to achieving accurate
automation functionality. The paper asserts that BLE is the best choice in applications of home
automation.
Given the maturity of BLE relative to UWB and the broader availability of BLE based
devices in the industry, this research elected to adopt devices that at least had BLE as one of its
available protocols. To the papers point about ubiquity, BLE is a required component for
iBeacon technology to function and is usually implemented alongside other protocols such as
Googles Eddystone and vendors custom protocols such as Estimotes Monitoring. One early
15
area of examination conducted during this research was whether BLEs data transfer rates are
sufficient to achieve the perception of near real-time reaction. At least in iOS, there are non
user-modifiable constraints on how often and iOS device polls for bluetooth devices. This
research investigated various options for combining iOS polling constraints with configurable
packet broadcasting time in order to achieve a balance between quasi-real-time applications and
battery life of the beacon devices themselves. The outcome of this research will be discussed in a
later section.
This particular project sounds similar in that their goal is to use ML to generate
personalized and time-variant home automation plans. It posits a solution to the same problem
that The Learning Home seeks to improve upon, namely the rigidity of usability. This paper also
proposes a system that will mimic a users actions based upon previously detected behavioral
patterns.
While these concepts have some overlap with concepts that The Learning Home also
seeks to investigate, this paper is two pages long and is primarily conceptual in its nature with no
reference to the user differentiation detection capabilities or precise real-time location services
This research validate the concepts introduced in this paper and seeks to pickup where
this research left off by implementing a prototype embodying and demonstrating some of its
ideas.
16
Design Requirements
The following section of the document provides detailed specifications which served as a
basis for the project and guided the entire development process.
Functional Decomposition
The diagram below is a depiction of the major components of the project and how the
flow of activities and data works to achieve the desired outcomes. From a hardware perspective,
the major components of the design are a home outfitted with home automation devices, a
raspberry pi device to host open source software components, beacons used to track indoor
location and mobile devices which are used by occupants to detect signals from the beacons. On
the software side there are three primary components. The first component is an open source
software package called Home Assistant. This package has three major architectural components
of interest for this project, namely, a state machine which tracks the state of all devices it knows
about over time, and event bus which listens to events to be fired such as state changes and a rule
engine which can be used to listen to the event bus, evaluate the event againsts a set of rules and
then change desired state. Using Home Assistant APIs, a software package built as part of this
project extracts data for examination and to perform behavioral pattern detection. Additionally, a
software package built as part of this project tracks an occupants location within a home and
sends messages to the Home Assistant platform when the occupant enters or leaves a particular
17
room.
While the original goal of this research was to apply machine learning algorithms to the
datasets generated in order to create recommendations for state changes to perform based on an
individual occupants location, the mix of research and implementation in this project posed a
time constraint such that this final architecture was achieved about 75% of the way through the
available timeline. This did not leave adequate time to generate a sufficient amount of data from
which to train models, so in lieu of that, SQL queries of the Home Assistant databases state
table were used to derive a first pass of automation rules that combined presence detection and
18
state changes. The rules were then added to Home Assistant such that after placement of the
beacons and running the iOS app, the home is able to automatically apply state changes at
The primary factors that influenced choosing the appropriate hardware to support this
project were the cost, reliability and ease of use. On the cost side, the goal was to develop a
solution where equipment needed for indoor location tracking was no more than $20 per room
and would be easy to install. Additionally, it was important to select a vendor with strong
expertise in the area of beacon technology which also had a mature and well documented API.
The clear winner after testing devices from two vendors was Estimote who offers a broad
portfolio of hardware products and has been working with beacon technology since its inception.
Final Deliverables
The final deliverables for this project will include an iOS mobile application used to track
and record indoor location along with the software packages discussed above which are used to
extract data from Home Assistant, correlate it with location data, produce state recommendations
and finally create rulesets and apply them to Home Assistant. Due to the nature of this project,
the final deliverables will also include a video demonstrating the capabilities of this project.
Methodology
The methodology I followed in completing this project was Agile, using iterative and
incremental development techniques and fast feedback loops to adjust as the project went on. I
used real-world consumer devices combined with software to implement prototypes of the topics
19
discussed herein. I had identified four major milestones as part of this project at its outset, which
Specifically, I researched the smart cameras which are suitable candidates for real-time facial
detection and devices which can be used to precisely track real-time location of a user within a
home. As part of the solution intended to correlate the activity of devices with the location of
users in order to build behavioral models which could later apply state, I researched home
automation smart hubs which aggregate the control of multiple devices where some such model
can be applied. One important part of research that I completed in this milestone was the
assessment of APIs for all devices that I research, as that was important for later stages of
project. Ultimately during this phase, I selected the use of Estimote beacons for tracking location
and the Wink Hub for aggregated control of home automation devices.
Based upon the previous milestone, I procured Estimote Nearables, Estimote Proximity
Beacons, Estimote Location Beacons and Estimote Location Beacons with UWB. Unfortunately,
the Location Beacons with UWB, which were the most promising piece of hardware for my
desired application, were received days before the end of research completion and as such were
not obtained in sufficient time to incorporate into this project. I already owned a Wink Hub
device, which moderately accelerates the development process and reduces the budget.
20
prototype of the concepts discussed in this paper. Specifically, I built a prototype to demonstrate
the ability to correlate user activity with user location to build a behavior model. The camera
based approach quickly proved to be unreliable as the facial detection capabilities built into
commercial products were unreliable and yielded frequent false positives. Finally, I conducted
some initial research on machine learning, but descoped that element from this project due to
time constraints.
Most of the testing was conducted in my home and I have devised ways to present a
scaled down version of this project on-campus. To that extent, during this portion of the project I
experimented with a scaled down version or other mechanisms to effectively demonstrate the
broader ideas behind the project. Finally, I opted to use Zoom as the platform to create my
demonstration as the most effective way to display this projects capabilities incorporated live
video while simultaneously capturing events in a browser and iOS application state.
Ethical Considerations
Given that this research deals with tracking user activity and location very precisely
within ones home, there are important ethical considerations to consider both in its design and
implementation.
Ethical Concerns
The primary ethical concerns associated with this capstone project relate to privacy. This
project sought to investigate two fundamental methods as mechanisms to track user behavior and
21
build corresponding behavioral models using machine learning, each of which has a discrete set
of privacy considerations.
Location Services. While location tracking capabilities, which are commonly called
location services, provide substantial capabilities in a variety of mobile devices, they can be
viewed as useful or invasive. Mobile devices commonly leverage location services to provide
useful functions such as GPS based directions and geo-tagging of objects such as pictures.
Nevertheless, many device users are not comfortable with the notion of their every movement
and location being tracked and recorded. As a result of these concerns, some users opt to turn off
A common use of location services in home automation is for the control of lighting, for
example, to turn lights on or off when entering or exiting a region such as a geofence around
somebodys home. In the case of iOS devices, Apple states that Location Services uses GPS
and Bluetooth (where they're available), along with crowd-sourced Wi-Fi hotspots and cellular
towers to determine the approximate location of your device. One of the primary mechanisms
that this project uses for precisely tracking user location within a home is via Bluetooth
communications between a personal device such as a mobile phone (iPhone) or wearable device
(Apple Watch) and beacons. Any user who wishes to completely turn off all location services
would not be able to benefit from the techniques discussed in this paper.
Nevertheless, the opposition to tracking everywhere a person goes outside of their home
is expected to be more more objectionable than tracking where they are inside their home. For
example, the former may expose illicit or morally objectionable behavior that a device user may
wish to conceal, while in the case of the latter, a visit to the closet or the living room is less likely
22
merits validation with potential users now that this research has yielded some viable prototypes.
One method of mitigating this concern, should it become one, would be to use some
ancillary device other than mobile device as a tracking mechanism to represent the user, such as
Cameras. The other primary method this research evaluated as part of its efforts to build
user behavioral models was the use of IoT cameras. As demonstrated by the Persirai IoT botnet
in May 2017, IoT cameras have proven to be a favorite target of malware authors and used as a
launching point for distributed denial of service (DDOS) attacks. Additionally, while users have
generally been accepting of storing streaming video of activity outside their homes with some
external service provider as a method of home security, it is expected that users would have
strong opposition to continuous video recording inside their home, particularly in certain rooms
within the home such as bedrooms and bathrooms where a generally high level of privacy should
rightfully be expected.
This research investigated the use of IoT cameras to perform facial recognition as a
means to discern and identify various actors within the home and potentially use those
recognition capabilities to apply learned behaviors. For example, this could provide for the
detection and application of different lighting based upon who enters the front door. This
research originally aimed to mitigate concerns about cameras within a home by positioning them
to cover main points of ingress and egress such as a front door or garage entry door and main
living spaces such as a family room and kitchen; however, due to aforementioned technical
constraints, this approach was not extensively pursued nor was it incorporated into the final
23
prototype. This research did not consider the application of IoT devices in other parts of the
home, and therefore concluded that a hybrid approach between the bluetooth and camera based
methods could be used to achieve the desired effects, should the technical constraints be
Negative Impacts
non-automated counterparts, and thus the barrier for entry could be very high for lower income
individuals and families. According to Pew Research, while 77% of all Americans own a
smartphone, as many as 64% of low income owners also own smartphones. Bluetooth is a
leverage at least some of techniques which are part of this research and nearly all smartphones
users should theoretically be able to, at least from a technology perspective, using capabilities
built-in to their phone. The bigger issue which may exclude low-income groups would be the
additional cost of the items required to track location inside a home such as beacons, IoT
A long term goal of this research is to determine whether it would be useful or feasible to
incorporate BLE chips in more commonly used devices such as light switches and power plugs
to facilitate some of the techniques that this research has examined. The ability to do so in a
viable manner may reduce the potential financial burdens associated with implementation of the
technology.
24
Environmental Impacts
One of the primary motivations behind this project and indeed home automation in
general is to have a positive environmental impact by enabling people to use their homes more
efficiently. Automation of lighting is a common entry point into home automation and this
research intends to build techniques that will make the use of homes and devices beyond just
During different milestones of this project, it was at times ahead of schedule, at times
behind schedule, at times way over budget and at times right on budget. Overall, this project
primarily ran slightly behind its timeline and was slightly over budget.
Milestone 1
This milestone ran ahead of time and ultimately on budget. I had procured some
equipment in advance of the start of class which enabled this milestone to be achieved more
quickly. Additionally, I had been following Estimotes technology for some time and had an idea
25
that it would be the more suitable option for rapid prototyping, but to avoid confirmation bias, I
Milestone 2
This milestone ran behind schedule and on budget. As previously discussed, Estimotes
Location Beacons with UWB sounded promising for this research and provided all the
capabilities of its other beacons combined into a single form factor. Unfortunately, these devices
were backordered and then encountered additional delays during the backorder and were not
incorporated into this project. I ultimately ended up procuring every type of beacon that Estimote
produces and after testing returned the ones that were not useful for this research.
Milestone 3
This milestone ran behind schedule and over budget. The implementation of the
prototype took much longer than initially anticipated. There were several factors which
Firstly, once it was known that this researchs first choice of hardware would likely not
be available in sufficient time for incorporation into the final project, alternative approaches had
to be evaluated and implemented. Secondly, one of the final deliverables was on a platform and
in a language which I had never worked with before, specifically Swift on iOS. There was a
significant learning curve to overcome. During the initial implementation of the prototype, I
encountered a few iBeacon protocol limitations that were substantial encumbrances, specifically,
I found that it was difficult to control distance in software and that the reachability of a beacon
from a mobile device was mostly achieved through physical placement of the device and by
adjusting signal strength. Additionally, I discovered that iBeacon imposes a 30 second wait for
26
exit events. This was problematic because as it created false positives in terms of an occupants
actual location, that is, it made for the appearance of an occupant being in two rooms
underestimated the time that the Machine Learning component would take and I wound up with
insufficient data and time to train models. I discovered that the general recommendation for
training is to apply a 80%/20% rule, that is train with 80% of the data and then validate it with
the remaining 20%. In this case, because I know that I have variations of activity throughout a
week, it would have been useful for this project to treat a week as an atomic unit and have 4
weeks of data to train on and then 1 week of data to validate on. Unfortunately, with the solution
architecture finalized and a prototype completed around the 6.5 week mark, there wasnt time to
complete all of the machine learning milestone and it had to be descoped from the project and
This milestone ended up being over budget because during the course of implementing
the prototype, I came across the excellent Home Assistant platform which required some sort of
device to run on to collect state data. I purchased a Canakit Raspberry Pi to run Home Assistant
Milestone 4
shortly after the prototype was completed. There was no budget I anticipated being required for
Target Audience
My target audience is all ages, but Im initially targeting digital natives or generally any
person who was born after 1980, i.e. 18-37 year olds. I am initially targeting this audience as
familiarity with technology, mobile devices and and mobile apps is a prerequisite in the early
prototype stages. Other age groups are also reasonable targets if they satisfy these criteria.
during the prototyping phase. Longer term, my project will target people with disabilities or
mobility restrictions that could benefit from this project in additional to non-technology focused
people who might appreciate its benefits but dont have the interest or knowledge to configure
Project Testers
hardware component to it, there were substantial limitations on the ability to test the project.
Testing of the project was constrained to the occupants of the home where the prototype has been
implemented.
The names of the testers were Joe and Kasie. I fit the target audience as I am a digital
native who is familiar with technology and I am a home automation enthusiast. My wife Kasie
fits my target audience as she is also a digital native but part of the later target group of people
who appreciates home automation but dont want to hassle with it and doesnt have the time or
Testing was rather straightforward. It involved simply behaving in the manner in which
one normally would in the home, but while carrying a mobile device running the prototype
1. Install The Learning Home iOS application on the device via XCode.
2. Run application.
3. Visit at least three distinct rooms in the home, such as the Living Room, Kitchen
5. In 2 of the 3 rooms, manually interact with at least one class of device such as a
light switch.
6. In 1 of these 2 rooms interact with more than one class of device such as a light
Testing Observations
There are two parts of this testing that were important to analyze. The first was end-user
behaviour, which was quite straightforward: start the mobile app, put the phone in your pocket,
and then proceed with your typical routine. In the case of both testers, the tasks were completed
in an unremarkable fashion.
29
The second part of the testing analysis had more to do with the data generated by the
application and how well the data generated correlates to each testers activities. On one hand,
testing demonstrated the ability to accurately detect and capture the state of the home, as
The testing also demonstrated the ability to detect the testers location in particular rooms
A primary issuing uncovered during testing, however, related to the reliability of room
detection. There are frequent false positive enter and exit events reported by the application
where the tester and its nearest beacon were stationary and had been stationary for over 10
seconds, but where an exit and then enter beacon region events would be reported. Additionally,
30
testing uncovered that the detection of a room enter event usually lags physical entrance by
approximately 2-5 seconds. These creates a perceptible delay that for certain tasks, such as
turning on lights when entering a dark room, was too lengthy and was viewed as an encumbrance
during testing.
Testers were able to complete all of the tasks, as the prototype required them to launch an
application on their phone and then use the home in the manner they normally would.
As mentioned above, the most immediate short term need is to improve the timing of
activities. Currently, there are is a delay between physically entering a room and a room entrance
event being detected in software. Additionally, exit events for leaving beacon range are software
controlled via iOS to occur 30 seconds after leaving the region which creates circumstances
where the data represents the tester as being in two locations at same time even after having
recently left one of the rooms. This creates issues for data correlation down the road.
Estimote recently released a new version of its SDK using their own protocol which
purports to overcome some of these limitations of iBeacon and allows for greater precision
around measuring entry and exit events that may address some of these shortcomings. In the
short-term, I plan to use this new Proximity SDK for room detection to see if its possible to
The long term plan will be to use UWB based technologies, which I received too late in
this course to incorporate into the current prototype, to further investigate higher precision and
Final Implementation
The final implementation of this project comprises a substantial portion of the originally
envisioned functionality. The prototype that was built is able accurately track the presence of a
homes occupant in various rooms throughout the house. Additionally, the prototype
implementation tracks state changes to devices within the home. All of this data is tracked in the
same database and was used to derive automation rules which are then applied based on user
presence.
Beacon Devices
The final implementation uses Estimotes Proximity Beacons, with one placed in the
kitchen, living room and master bedroom in my home. After encountering some of the issues
with the iBeacon protocol discussed earlier in this paper, Estimotes proprietary Monitoring
32
protocol was used in the final prototype to overcome some of these constraints. Note that in the
configuration screens above that the iBeacon protocol has been disabled and the Estimote
Monitoring protocol is enabled. This is a limitation of the Estimote Proximity Beacon: it can
only broadcast one protocol at at time. Estimote Location Beacons are able to simultaneously
Home Assistant is an open source application that provides a very extensible and robust
platform for home automation with a copious number of plugins and configuration options.
Included in the Home Assistant platform and of interest for this project were a state machine to
33
track changes to entities over time, as well as an event bus that other components in the platform
are able to plug into. One of these key components is an automation rule engine which can
monitor events on the bus, validate rules, and then apply state changes.
In order to have dedicated hardware to run software packages that would enable type of
data collection that I desired for this project, I obtained a CanaKit Raspberry Pi 3 Complete
Starter Kit - 32 GB Edition and installed Raspbian Jessie. Initially, I used the Raspberry Pi to
host software packages that provided Apple HomeKit support, anticipating that the integration of
HomeKit would be desirable for an iOS application, and a software package for text-to-speech
(TTS) support via the Sonos API, which enabled me to create announcement messages on
speakers during early prototyping. Ultimately, upon discovering Home Assistant, I abandoned
the use of these other software packages, installed Docker, and then installed hass.io, the
34
dockerized version of Home Assistant. After installation, I configured Home Assistant with most
of my devices, which was facilitated greatly by Home Assistant's Wink component. Finally, I
configured some rooms in Home Assitant to group devices together that I wanted to test for this
project.
iOS Application
An iOS application using Swift was implemented to read Estimote Monitoring packets
and send notification messages to the Home Assistant platform using the MQTT protocol.
The implementation of this prototype included two critical pieces, an MQTT client for
iOS that could send notifications to Home Assistant, along with an extension of an application
template that would allow reading beacon information from the Estimote Cloud.
35
beacon information from the Estimote Cloud where various configuration data lives.
Unfortunately, the provided template only fetches the beacons identifier which looks something
like this: 5a1b4d9ad7fe92f5adc70c9f74894f3e. It was clear that the sending such an identifier
to Home Assistant would only obfuscate later efforts to correlate user position to other activities,
so I wrote an extension to Cloud Manager that fetches the name of the beacons, which Ive
This implementation involved three components. The first was to add a new function to
The second part was to call the new function as part of a pre-fetch routine so that it could
self.cloudManager.fetchNames(beaconIdentifiers:
self.beaconIdentifiers) { (result) in
switch result {
case .error:
self.presentFetchingNamesFailedAlert()
return
in range:
The piece which tied everything together was the implementation of an MQTT client for
iOS. Home Assistant provides an MQTT broker along with an MQTT Room presence detection
capability in a class of entities that it refers to as sensors. By defining an MQTT topic associated
with a particular device id, the Home Assistant platform allows publishing messages to a
subtopic where the subtopic represents a room. If the payload of the message published to a
subtopic contains a device id which matches the configuration of the sensor, then Home
Assistant fires a state change indicating that the device now occupies whatever subtopic, or
track when beacons come in and out of range. Two of the primary advantages of using Estimote
Monitoring packets over iBeacon packets was the elimination of 30 second exit delays and,
perhaps more importantly, the ability to define desired distances in software that would
The first part of this implementation and perhaps the trickiest for somebody new to iOS
development like I was, was using CocoaPods to properly import the Moscapsule library, a
commonly used MQTT client for iOS. Once this was done, the next step involved setting up the
self.cloudManager.fetchNames(beaconIdentifiers: identifiers) {
(result) in
s witch result {
case .error:
print("error attempting to retrieve beacon names")
return
There are two items worth noting in the above code. The first is the protocol version. If
unspecified, this defaults to MQTT version 3.1; however, the broker that ships with Home
Assistant 0.55.2 expects MQTT version 3.1.1. The second item worth noting is the reuse of the
fetchNames() function previously created for the UI updates. This turns out to be very useful for
The next step in updating the ProximityManager was to expand the capabilities of the
fire events whenever the initial state is observed, when a beacon enters the desired range and
when a beacon exits the desired range. In each of these cases, the ProximityManager class calls a
if beaconIdentifiersInRange.count == 0 {
self.sendAwayMessage()
}
else {
for identifier in beaconIdentifiersInRange {
self.sendEntryMessageForIdentifier(identifier: identifier)
}
}
self.delegate?.proximityManager(self, didUpdateBeaconsInRange:
beaconIdentifiersInRange)
}
happens: either an away message is sent when the array of in range beacons is empty or an entry
message is sent for the identifier in range. These two functions are what ultimately publish the
MQTT messages and tie the presence detection back to Home Assistant. The entry message
l et i d = self.mqttConfig.clientId
l et n ame = self.mqttConfig.clientId
l et d istance = Parameters.desiredDistance
Its worth noting again that the clientId specified in the original configuration of the
MQTT client above should match the sensor configuration in Home Assistant, otherwise the
message will not have the desired effect. The function publishes a message containing the client
id, the client name, and a distance. Also note that the message is published to a subtopic that
The away message function operates in a similar fashion; however it publishes to the
room_presence/away subtopic to indicate that the occupant is not in a monitored room within
the home.
Conclusion
This project serves as a proof of concept and validates some of the core capabilities
The most immediate work to carry forward with this project will be the implementation
of the machine learning and AI features which were descoped from the project. Firstly, this will
involve collecting sufficient data over the next four weeks to adequately train and validate
machine learning models. Part of this process will involve data extraction and data cleansing.
Once the models are trained and generating reliable output, a software package will be
created that uses the output from machine learning to construct the rules, represented as YAML
configuration files, and then uploads the rules back to Home Assistant such that over time after
placement of the beacons and running the iOS app, the home is able to automatically apply state
changes at appropriate times based on upon user occupancy with little to no additional
References
Apple Inc. (2017, February 24). Turn Location Services and GPS on or off on your iPhone, iPad,
https://support.apple.com/en-us/HT207092
Collotta, M., & Pau, G. (2015). A Solution Based on Bluetooth Low Energy for Smart Home
Shaikh, S., Attar, A., Pathan, S., & Shaikh, R. (2016). Machine Learning On Real-Time Data to
Smith, A. (2017, January 12). Record shares of Americans now own smartphones, have home
http://www.pewresearch.org/fact-tank/2017/01/12/evolution-of-technology/
43
Appendix
Rooms:
rooms:
name: Rooms
view: yes
icon: mdi:home
entities:
- group.kitchen
- group.living_room
- group.master_bedroom
k itchen:
name: Kitchen
entities:
- light.kitchen
- light.kitchen_island
- switch.kitchen_cabinet_underlights
- media_player.kitchen
l iving_room:
name: Living Room
entities:
- light.living_room
- light.floor_lamp
m aster_bedroom:
name: Master Bedroom
entities:
- light.joes_nightstand
- light.kasies_nightstand
- light.master_bedroom
- light.master_hanging_lamp
- media_player.master_bedroom
44
{
"id": "joephone",
"name": "joephone",
"distance": 1.5
}
sensor:
- platform: mqtt_room
device_id: joephone
name: 'Joe Room Detection'
state_topic: 'room_presence'
timeout: 3
away_timeout: 36000
select *
from states
where entity_id like '%kitchen%'
and
last_changed >= datetime (
(
select last_changed
from states
where state_id = 13279
), '-15 seconds')
and
last_changed <= datetime (
(
select last_changed
from states
where state_id = 13279
), '+15 seconds')
automation:
- alias: 'Kitchen Lights On for Joe'
46
initial_state: 'on'
trigger:
- platform: state
entity_id: sensor.joe_room_detection
to: 'kitchen (ice)'
action:
- service: light.turn_on
entity_id: light.kitchen_island
data:
brightness: 100
{
"entity_id":"sensor.joe_room_detection",
"old_state":{
"entity_id":"sensor.joe_room_detection",
"state":"away",
"attributes":{
"distance":0.0,
"friendly_name":"Joe Room Detection"
},
"last_changed":"2017-10-17T05:01:41.425214+00:00",
"last_updated":"2017-10-17T05:01:41.425214+00:00"
},
"new_state":{
"entity_id":"sensor.joe_room_detection",
"state":"kitchen (ice)",
"attributes":{
"distance":1.5,
"friendly_name":"Joe Room Detection"
},
"last_changed":"2017-10-17T05:01:53.556113+00:00",
"last_updated":"2017-10-17T05:01:53.556113+00:00"
}
}
47
{
"entity_id":"sensor.joe_room_detection",
"old_state":{
"entity_id":"sensor.joe_room_detection",
"state":"kitchen (ice)",
"attributes":{
"distance":1.5,
"friendly_name":"Joe Room Detection"
},
"last_changed":"2017-10-17T05:01:53.556113+00:00",
"last_updated":"2017-10-17T05:01:53.556113+00:00"
},
"new_state":{
"entity_id":"sensor.joe_room_detection",
"state":"away",
"attributes":{
"distance":0.0,
"friendly_name":"Joe Room Detection"
},
"last_changed":"2017-10-17T05:02:24.752570+00:00",
"last_updated":"2017-10-17T05:02:24.752570+00:00"
}
}
Event Sample of device state change (turning kitchen island light on):
{
"entity_id":"light.kitchen_island",
"old_state":{
"entity_id":"light.kitchen_island",
"state":"off",
"attributes":{
"manufacturer_device_model":"lutron_p_pkg1_w_wh_d",
"device_manufacturer":"lutron",
"model_name":"Caseta Wireless Dimmer & Pico",
48
"friendly_name":"Kitchen Island",
"supported_features":19
},
"last_changed":"2017-10-17T05:02:30.919073+00:00",
"last_updated":"2017-10-17T05:02:30.919073+00:00"
},
"new_state":{
"entity_id":"light.kitchen_island",
"state":"on",
"attributes":{
"brightness":255,
"min_mireds":154,
"max_mireds":500,
"manufacturer_device_model":"lutron_p_pkg1_w_wh_d",
"device_manufacturer":"lutron",
"model_name":"Caseta Wireless Dimmer & Pico",
"friendly_name":"Kitchen Island",
"supported_features":19
},
"last_changed":"2017-10-17T13:51:38.853085+00:00",
"last_updated":"2017-10-17T13:51:38.853085+00:00"
}
}