
We started this week’s creative meeting with a quick exercise. We each jotted down examples of design problems that are challenging because of the separation between the digital and the physical worlds. Then, we looked at the evolution of Augmented Reality technology. Like most burgeoning technologies, the evolution of AR is peppered with examples of applications best described as gimmicky and useless. We wanted to look instead at the best of the best in order to find ones that either help solve a real need, or at least create a truly unique experience. Our ultimate goal was to identify the types of design problems for which a solution involving AR could be appropriate, either now or in the future (you know, when the technology is better).
Here’s what we found.
AUGMENTED REALITY THAT ENHANCES ENTERTAINMENT:
Harry Potter Deathly Hallows DVDs

The animated characters that animate from each of the two Deathly Hallows DVDs actually sense each other and start to battle. By taking “augmented packing” to a new level, they gave customers a fun incentive to purchase DVDs for both movies.
AR Basketball

A sort of AR version of the classic time-killer paper football, this iPhone game puts a mini-mini-hoop right on your desk and let’s you flick virtual basketballs.
Topps 3d live trading cards

Trading cards are all about showcasing a player, so seeing an animated 3d version of that player pop up on your desktop - and actually play a virtual game with him - is a natural evolution.
AUGMENTED REALITY THAT ENCOURAGES CREATIVITY:
Lost Valentinos music video

It’s already fun posing the bandmates in a music video as if they were action figures, but using them to record your own interactive music video for all the world to see is a win.
Wrigley’s 5 Gum

We’re not sure of the connection between gum and DJ-ing, but Wrigley’s came up with a clever way to create a “virtual” physical instrument that you can actually play and record.
AUGMENTED REALITY AS ADVERTISING:
Vampire Diaries Magic Mirror

Still one of the coolest uses of AR, pedestrians pausing to look at this billboard for a series about vampires think they’re looking into a mirror. Then poof their reflection disappears.
VW Beetle Juiced Up billboards

You have to pre-install the app and aim your iPad at the ad, but Volkswagen’s static billboards will come alive in an impressive display of 3d animation.
Esquire’s Find Brooklyn Decker

Goldrun’s iPhone app gives you rewards for taking fake photographs with celebrities. It’s a bit hard to imagine posing in front of my phone with a celebrity photo, but at least this campaign makes it more of a paparazzi game, driving customers right to the magazine aisle where they are (in theory) more likely to buy Esquire’s latest issue. Generally speaking, this probably works best with kids and animated characters.
AUGMENTED REALITY AS UTILITY:
Holition’s Augmented Retail

Holition has a whole line of “augmented retail” experiences. They use printed cutouts that you strap on your body, or facial recognition, to help you try on luxury items such as rings, watches, and sunglasses.
USPS Priority Mail Virtual Box simulator

The postal service tried a clever solution for all the time and materials people waste selecting the wrong shipping boxes. Just put your item on your desk and slip a virtual box on top. The idea is great, though it’s hard to say if all the prep is simpler than just grabbing a tape measure.
Browse AR

There are many of these “real life browsers” coming out now, and until something like Google Glasses comes out and removes the need for a viewfinder, their utility is suspect. Yelp’s Monocle might have been the first (and Google Glasses might be the endgame), but they all use GPS and your smartphone’s camera to show you where points of interest are in real time.
Word Lens

If it worked as well as advertised, this comes as close to magic and a perfect, transparent UX, as anything I’ve seen. Aim your phone at a sign, a document, or any word or words in a foreign language, and this app not only translates it for you in real-time, but replaces the foreign text with the translation. This was our favorite in concept.
Conclusions
With an understanding of the range of modern AR applications, we now reviewed the results of our initial exercise to determine whether any of them would be appropriate for an Augmented Reality solution. We discussed the challenges in knowing the true color of physical objects online, mimicking the emotional depth of face-to-face conversation through video, checking the tactile properties of fabrics and similar items, representing network maps (i.e. real devices in digital space), and of course the ability to taste food and smell perfumes across the web. Challenges with physical space seemed more promising, such as when you’re trying to buy furniture, replacement parts for mechanical objects like a bicycle, or looking at apartment rentals and understanding the size and layout of the rooms. We also discussed the many applications within remote healthcare, and possible ways of using AR to teach someone to play an instrument remotely.

But what about the problem of learning how to tie a tie? Drawings on how to tie a tie are confusing and difficult to translate into 3d space. And it’s still a challenge translating the motions of someone in a video to your own reflection in a mirror. AR’s “magic mirror” effect might be able to solve this. It could place a 3d tie on your real-time video and animate itself being tied. Or it could create a semi-transparent tie for each step, allowing you to simply match the shape of your own tie below it, guided by helpful arrows. A fancy version might allow you to manipulate the tie and give you immediate feedback on whether you’re doing it the right or wrong way.
In the end, we believe that AR has two big hurdles to surmount if it’s going to hit the mainstream and have real utility. Whether through better algorithms or hardware, it has to recognize and track objects better so we can get beyond printing out all these silly paper symbols. Ideally, this would better help it mimic the dynamic occlusion effect (which would make trying on sunglasses feel a lot better). The second hurdle is the display screen. Mobile phones are an integral part of daily life, but pointing it this way and that in public to view AR content feels just dorky enough to be a barrier. Only when the screens are big enough (digital billboards) or small enough (contact lenses) that the screen disappears will AR really take hold.