We turned my phone into a universal remote that uses augmented reality. We can track the location of objects and identify them in real time, so we can overlay an icon representing the object on the video shown on the touchscreen. Simply click on the object to interact with it. It’s all wireless, no base station required.

Actual video is 1080p, I recommend you view it full screen.

Presentation slides: without notes and with notes

Photos:
http://imgur.com/a/oRjll

Our group did circuit and PCB design, we did the soldering, we designed the phone casing, we designed all the IR and RF protocols, we wrote our own firmware and Android app.

We have many visions on how to improve our system. We know how to improve our noise and interference immunity, plus increase the tracking speed performance. We have several ideas on how to make the phone-attached-circuit less bulky. We know how to use multiple emitters to estimate the object’s size and orientation.

We are trying to gauge interest in our project, to see whether or not to continue the project, and in which direction to take it in.

Abstract (exactly as we submitted it)

Augmented reality provides individuals with a virtually enhanced
method of viewing and interacting with the world around them. The
Augmented Reality Universal Controller and Identifier (ARUCI) is a
real-time system that integrates remote object control with augmented
reality to view and control various objects through an interactive
visual interface that can be installed on an android smartphone. The
ARUCI system incorporates both physical hardware, and a platform that
allows users to upload and configure control mechanisms for various
electronic objects such as lighting and appliances. By pointing ARUCI
in the direction of any compatible object, the user can identify
details about it. For configured electronic devices, ARUCI can also
control them remotely via an augmented reality overlay on the
smartphone. The goal of this system is to improve the way people view
and interact with all of the electronics around them.

Q & A (updated as people ask questions)

• Augmented reality can already be done with tags, why are you using IR?
  • At long distances, the camera won’t be able to process the tag image, but IR has much longer range.
• What image processing library are you using?
  • There is no image processing being used at all, object identification and location are both done using only infrared.
• Why are you using RF if you have IR communication already?
  • More flexibility in our design. Off-screen objects and objects without IR are still controllable. IR is slow and traffic would become overwhelming.
• You talk about both USB and Bluetooth, and your pictures show a USB connector and a RN42 module, how is that circuit talking with the phone?
  • Originally we were using the MAX3421 to communicate via USB, but it turns out the phone does not enumerate unless you give it 5V. We then gave it 5V by rigging up a 5V boost converter, but since boost converters are not 100% efficient, it meant we were wasting more power than what a RN42 module would draw.
  • That did not explain why both footprints were on the PCB. The reason is simple, we simply foreseen the possibility of the MAX3421 method failing, and added Bluetooth as a backup just in case, the PCB isn’t that dense so it wasn’t a problem just including a footprint..
• I want this, how does the average person set this up?
  • Like we said, we are looking at options for our next steps. We can ask device makers to integrate our system into their own products. We can design nodes for specific models of products (with the help from the device manufacture).
  • Or we can just provide DIY kits. The node circuit is a 2 part design with one board sharing the same layout, while the other board is a “perfboard”, this makes it friendly for DIY. Simple nodes like a light switch can be a simple relay circuit. We also feature a USB port, which can be used for firmware updating, reconfiguration, and other things.
  • For security and privacy, the ATmega128RFA1 we use features a AES engine that can be used for encryption. We can use the USB port to update the key. The IR data is not encrypted, mostly because there’s not much point, if you can “man in the middle” the IR data which only contains device identifiers, you can probably just use your eyes to look at the object to figure out what it is.
• What inspired this project?
  • Originally we wanted a magic wand, wave a stick at an object to control it. This is actually totally possible with our current system. We then decided that aiming a wand would be hard, we didn’t want to add a laser pointer for safety reasons, so we decided on using augmented reality, which is pretty cool itself.