Microsoft Hololens / Augmented Reality

Independent Study (Co-advised by Maribeth Gandy Coleman and Brian D. Jones)


Whack-A-Holomole is a Microsoft Hololens gaming application, which aims to change turning on and off lights, a boring task, into a fun and intriguing experience. Players will whack moles, which will constantly turn on the lights and change the color of the bulbs, until they finally manage to turn off the unnecessary lights.

-Presented demo to TechSAge State of the Science Conference 2017


Augmented reality provides a unique experience because not only does it show immersive 3D graphics, but also the 3D graphic images are placed in the real world. I think the experiences AR technology creates can be enhanced if the 3D graphic images interact with real world objects, so I wanted to connect Microsoft Hololens with smart home devices, such as smart plugs, smart bulbs, and WiFi thermostats.


I had two scenarios for this project. The first idea is an educational game that will help people understand how IoT devices work. The second is to help people engage with smart devices and energy-saving activities.


1) Augmented Reality and Education

This project started with the idea that AR could be used as a learning tool in education and everyday life. I hoped that the immersive feeling of the AR environment would raise educational efficacy and interest and intrigue children who otherwise become bored easily while studying.


I researched educational AR applications that are currently being developed. Even though the subject and techniques applied were different, the apps enable users to interact with virtual screens or objects in the real world, which make the experience very special for students.

2) IoT (Internet of Things)

IoT technology is more advanced than Internet or mobile Internet technology based on existing wired communication, and devices connected to the Internet communicate with each other without any human intervention. In addition to home appliances and electronic devices, information can be shared by networking objects in various fields such as the healthcare, automotive, and smart home fields.

The most widely used environment for IoT technology is the residential space, which is called a ‘Smart home’. This means home appliances, lighting, energy, and security functions as well as smart cars and smart education.


1) Environment setting - connection test

wanted to incorporate IoT technology with AR to improve the home education environment. So, the first thing I needed to do was make a connection between the Hololens and smart devices. Firstly, I tried to make a DIY-style smart plug using Arduino and Relay. I was also able to connect Philips Hue smart bulbs to the Hololens. Lastly, I connected the Nest thermostat to the Hololens. I used Unity3D (C#) and RESTful API to control the Philips Hue smart bulbs and the Nest thermostat.

[Hololens + Smart plug using Arduino: Smart plug using Arduino used for this test]
[Hololens + Philips Hue: Screenshot from the Philips Hue testing video clip]
[Hololens + Nest Thermo: Nest thermostat that is used for this project and connected to 24V external power adaptor]

Finally, In this project I chose Philips Hue because it supports dimming control as a default feature as well as a color change feature, which may provide various effects for the future prototype. To create a gaming environment, I replaced the lightbulb in the room with a Philips Hue lightbulb which can communicate with the Microsoft Hololens. When a participant wearing a Hololens gazes and air-taps, a lighted bulb turns off in response to the participant’s behavior.

1) Possible Gaming Scenarios

The motivation for this project, as part of the whole home gaming project, was to make an entertaining experience, which will enable people to move from room to room or space to space in their home. The reason I chose the Hololens headset as a base technology for this project is that I believe the Hololens headset can create an immersive and explorable experience which makes the ordinary home environment into an amusing space.

While testing the IoT devices, I had another idea to motivate people to engage with smart devices and energy saving activities. Since Philips Hue can provide a colorful, party-like environment, I thought it might be a delightful experience if they can play with holograms that are synchronized with smart bulbs.

Scenario 1: Let people know how smart devices work

For this scenario, holograms can be used to visualize the invisible relationships between smart home devices. It can be a powerful educational application because the Hololens headset is very good at showing something otherwise unseen, such as pipes behind walls.
When the user starts the application, it shows the location of smart hubs, such as Amazon Echo or Google Home, and they can select one of the devices using gaze and air-tap gestures. This application also teaches users about basic troubleshooting processes. The most common problem with smart devices is “connection lost”. Holograms can be used to visualize where the lost connection occurred and show users the steps they have to take to fix it.

Scenario 1: Scenario 2: Whack-A-Mole + Smart bulbs

The purpose of this scenario is provide an entertaining experience by playing an augmented reality game which is connected to smart bulbs. It also aims to provide a message about energy saving. The reason I choose ‘Whack-A-Mole’ is to adapt a media convention that encourages people to naturally engage in the AR experience it provides. When a participant plays the game, a mole that moves up and down in the hole in front of their eyes appears, and the participant knows how to engage in the game without any explanation. The game begins with the mole appearing under the lit light, and the participant catches the mole with gaze and air-tap. If the participant catches the mole, the light goes out and they win the game.


Microsoft Hololens’s spectator view

I used Microsoft hololens’s spectator view to capture this gaming project demo video. In the past, video could only be shot from a first person perspective, but it has now become possible to take videos from the view of a third party observer through the recently released spectator view technique.

Demo at TechSAge State of Science Conference Open House Reception

As a student project, this prototype was demonstrated at TechSAge State of Science Conference, which held March 27th, 2017 at Georgia Tech.