After coordinating scientific research for the United States during World War II, including initiating the Manhattan Project, the engineer Vannevar Bush set his sights on a pacifist instrument for world knowledge.
In the July 1945 issue of The Atlantic, Bush outlined his vision for a head-mounted camera attached to “a pair of ordinary glasses” that would record comments, photographs, and data from scientific experiments: “One can now picture a future investigator in his laboratory. His hands are free, and he is not anchored.” His “camera … of the future,” no “larger than a walnut,” worn on “a pair of ordinary glasses … where it is out of the way of ordinary visions” was in many ways a forerunner of today’s augmented-reality devices.
For decades we’ve been inching closer to popular augmented-reality technologies to enhance the physical world—each new iteration promising to turn the entire world into a computing interface—but only in the past couple of years have headsets no longer needed to be enormous, bulky, and expensive, and superimposed images advanced beyond thin lines.
* * *
Coined in the 1990s, “augmented-reality” describes any technology that overlays digital interfaces onto the physical world. Unlike virtual reality, which immerses you in a simulation using stereoscopic 3D on a screen in front of your eyes, augmented-reality technologies embed opaque holograms directly into the environment. As early as 1990, assembly workers at Boeing were wearing see-through head displays that superimposed computerized images of where to place the wires on the 777 aircraft, which saved them from looking back and forth at their manuals. Thad Starner, now a professor of computing at Georgia Institute of Technology, was in 1993 one of the first people to don homemade computing devices with searchable text and recording functions—much like Bush imagined and long before the tech was fashionable or feasibly compact.
Spurred by developments in mobile computing and graphics processing, both augmented and virtual-reality technologies have finally become wearable, portable, and affordable enough for popular use. Six decades and several engineering challenges later, the first wave of market-ready devices are starting to battle for consumer attention: Last month, Facebook-owned Oculus VR announced that its Oculus Rift headset would ship to consumers by next year. Both HTC’s Vive and Gear VR, Oculus’ partnership with Samsung’s mobile phones, have said they will hit the market by the end of this year. Augmented reality might be a little farther off: Two of the highest-profile augmented-reality devices—Microsoft’s HoloLens and the Google-backed Magic Leap—have vague timelines for release.
No longer the stuff of prescient scientists, science fiction, and engineers, augmented and virtual visions are now competing for how we see the world. Whereas immersion in virtual reality means you can’t move around without bumping into walls, or see the plane you’re repairing, Magic Leap and HoloLens’ applications have the potential to transform how we go about everyday life. At stake is the chance to replace the smartphone, the desktop—even the desk. Starner says that once he saw how augmented reality enhanced his productivity and interactions in the physical world—he uses his custom device like a hyper-personalized assistant—he lost interest in virtual reality. Many others have similarly moved on from virtual reality to augmented reality, or AR as it’s commonly known.
“AR could be really seamlessly integrated into everyday tasks: communicating, working, information visualization, gaming outside—pretty much anything you can come up with,” said Gordon Wetzstein, an assistant professor of electrical engineering at Stanford. “It would fundamentally change the way we communicate with people.”
Magic Leap is still in what the company characterizes as “stealth mode,” but some of the lucky few who’ve tried its technology have described what they’ve seen with language typically reserved for religious experiences. The veteran game designer Graeme Devine told Polygon that although he was skeptical at first, he eventually was convinced to visit Magic Leap’s offices and “saw something that I did not think was possible. I like to think I know technology and I am not easily impressed. I worked at Apple, but when I saw what they were doing, I just said, immediately, ‘How can I help?’”
Whatever investors at Google saw last year caught their attention, too. They led Magic Leap’s $542 million round of funding in October, giving the 4-year-old company a valuation of $2 billion. “I believe that people may want to use this new form of computing as much, if not more than their mobile device,” Magic Leap’s founder and CEO, Rony Abovitz, said on Reddit.
Aspirational images from Magic Leap’s website give a good idea as to the scale of their ambitions: A tiny elephant frolics in someone’s hands; children clap their hands together in joy at seahorses floating in their classroom; dragons flap across a purple sky. And a particularly vivid patent illustration shows Magic Leap turning the world into a touchscreen—without the screen. (Or as one of the slogans they’ve applied to trademark puts it, “The World Is Your New Silver Screen.”)
Microsoft has hinted at potential functions for the device, unveiling applications for education, workplaces, operating rooms, and, as in Bush’s vision, labs at a conference in April. Researchers at Case Western University have teamed up with Microsoft to try to transform medical education and healthcare. With the HoloLens, for example, it could be possible to float and rotate a fully dimensional heart model that pumps with blood, providing crucial visual cues to practiced surgeons and medical students.
In another enticing demonstration, Microsoft showed how HoloLens could use data from the Mars rover to make it look like you were standing on the red planet. NASA plans to work with the technology this summer to control its Mars rovers. Using the device’s gesture commands for annotations, the agency’s scientists will be able collaborate with each other.
What the startup depicts looks like magic. Is it really possible? Both Microsoft and Magic Leap have kept hardware details largely under wraps, citing concerns about competition and industry spies, but patent and trademark filings reveal that Magic Leap’s technology projects light directly onto your retina, producing images that work with how your eye normally sees. This tricks the brain, which won’t distinguish between light from the outside world and light from Magic Leap’s tiny fiber optic projector, explains Wetzstein. These dense light-field displays allow the eye to perceive objects at different distances—creating a sense of depth that allows the eyes to focus on digital objects as though they existed in the real world. Unlike the flat screens suspended in space by the Google Glass, Magic Leap’s objects projected into the environment will, ideally, look realistic.
Microsoft hasn’t exactly said how the HoloLens shines light into the eye, but Wetzstein’s own papers on near-eye light displays, show that it’s possible to create the same light effects as Magic Leap through multiple means. The technical specifications released by Microsoft reveal that the HoloLens uses computer sensors to capture instant data from the real world—to make sure its holographic objects accurately fit into the environment—and a custom-built Holographic Processing Unit (HPU) takes all that data in at a speed that’s fast enough to avoid visual delays. Adding to the illusion, the headset uses binaural audio, what’s known as “spatial sound,” to allow you to hear the holograms’ audio accurate to its relative distance from your ear. You will actually be able to hear a dragon thumping down your hall towards you and roaring in your right ear. Before that can happen, augmented-reality devices still face many obstacles. Chief among them: packing their hardware into a wireless, light-weight headset, about the size and cost of an expensive pair of sunglasses. In other words, the question is no longer if holographs can blend with reality, but how, how well, and at what price.
“I’ve tried everything that’s on the market and I still think we have a long way to go,” Wetzstein said. “If Magic Leap delivers what they promise, it will be a significant improvement. However, AR faces a lot of challenges, because the devices need to be so small and low power, and there’s a lot of challenges with optics, electronics, vision.” Augmented-reality displays must be graphically robust and deliver “an amazing visual experience” while at the same time being small, comfortable, and portable enough that people would want to wear them, Wetzstein explained.
Technology Review’s Rachel Metz, who had the opportunity to try both devices this winter, wrote that Magic Leap’s target prototype size could not show her the same quality of images as the larger optometrist-sized version of the headset. She also noted that while HoloLens showed off a fully mobile headset at Build, it’s unclear whether Magic Leap has fit its tech into a wearable size.
“A lot of these digital miracles all come down to a common culprit: power,” said David Whittinghill, an assistant professor of computer graphics at Purdue who’s interested in developing for these devices. “How do you get the batteries to power these sort of video-records? How are you going to miniaturize it? How do you make it mobile and get it down to a weight that people can carry around?” As Wetzstein and others pointed out, for an augmented-reality device to be wearable in everyday life it also needs a bright enough display to work outdoors. But then again, better brightness takes up more battery.
There’s also the problem of creating realistic illusions in the first place, and fitting them naturally into the clutter of objects already in our spaces. In a head-mounted device, any slight misalignment or lag-time with its projections might make people sick—a little glitch and the device is too annoying; a major mismatch and it’s dysfunctional. This is where computer vision comes in: The headwear has to know where and what the eye is looking at, and the device must sense what’s in the real world, what’s moving and what’s staying still.
But even when virtual images successfully align with the physical world, and the devices achieve a comfortable size, augmented reality’s viewing experience is still likely to be trumped by virtual reality’s wider field of view. While the demos show fully immersive augmented experiences, the actual headset’s capabilities remain in a somewhat smaller window. Reporters at the Microsoft Build Developer Conference noted that images kept dropping out of the HoloLens field of view, an area that spans about 40 degrees, according to an estimate in Wired. Some described the HoloLens’ narrow viewing area—about the size of a TV—as shattering the experience’s illusion, whereas others found these “glimpses of the unenhanced world on the periphery” to be a persistent distraction. The Verge’s Adi Robertson explained that anything that slipped outside this window disappeared, rendering peripheral vision useless. Alice Truong at Quartz called the HoloLens limited perspective the “AR version of tunnel vision.”
David Luebke, who leads research on near-eye displays at NVIDIA, says that limited field of view is a common and serious problem with augmented-reality displays. Luebke is collaborating on a project at the University of North Carolina called Pinlight Displays, whose 100-degree field of view shows that this tunnel vision might not be a limitation in future AR devices. But until these displays no longer induce tunnel vision, Luebke believes that VR’s full field of view might make it the more likely candidate to replace traditional cubicle monitors at work and for entertainment too. Why watch a movie floating in a small window before your eyes—one that disappears if you approach it from the wrong distance—when you could instead see it in your entire field of vision on the Oculus Rift?
Given these trade-offs, it’s unlikely that one altered vision of reality will rule them all. Starner predicts that instead of a single company dominating the market, users will gladly choose from a full suite of devices—both AR and VR—ranging from low resolution to high resolution to wide field of view depending on their needs. “For a video game, a large field of view can be fun, but for surgery, messaging, personal-information management, and many other things, you only need a relatively small field of view,” he said. “One car does not fit all. Just try to move your whole apartment in a Chevy Corvette.”
These preferences point to unspoken law underwriting all computing wearables: People have to want to wear them. The ghost of Google Glass haunts conversations about the HoloLens and Magic Leap. Early Google Glass adopters were stigmatized (remember “Glassholes”?) for wearing the product in public. For Magic Leap’s mass adoption, it will need to have appealing everyday applications, like the iPhone does—but also appear non-threatening to those not wearing the device. “If you can’t wear Google Glass in public because people are suspicious of you, then who cares if it makes you sick or not?” asked Thomas Stoffrengen, a professor of kinesiology at the University of Minnesota who researches motion sickness in near-eye displays. “Augmented reality may reduce these problems, but we’ll see if there’s still a social problem.”
Microsoft’s instinct to create a device that you don’t wear all the time—but you have to have handy for professional tasks—might be spot-on and make it more socially acceptable than Magic Leap. (Microsoft writes in a press release that “the device quickly fades to the background.”) Tobias Höllerer, a professor at the University of California, Santa Barbara, who worked on some of the earliest mobile AR systems, predicts that “the first wave of these new devices will be occasional use”—meaning that they will be used to do tasks our smartphones can’t already do. These niche scenarios—an architect showing his blueprints, tourists visiting ancient ruins, a plumber advising repairs—will not in themselves prompt mass adoption, but they will at least enable socially-valid professional and personal uses among pioneering consumers.
Perhaps the best litmus test of the public’s readiness for head-mounted wearables is Starner, who has worn personal computing systems to enhance his life ostensibly longer than anyone. When in 1993 Starner first carried around his clunky personal computer device in a shoulder bag, a fellow computer scientist asked him why he would want a mobile computer. Starner says that when he wears his sleeker head display nowadays, people pull him aside to ask, “What is it? Can I try it? Can I buy it?” Computers and their scientists have, of course, come a long way.
Vannevar Bush’s prediction, half a century later, rings true: “The world has arrived at an age of cheap complex devices of great reliability; and something is bound to come of it.”
This article was originally published at http://www.theatlantic.com/technology/archive/2015/06/dueling-realities/395126/
Image may be NSFW.Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Clik here to view.
Clik here to view.
Clik here to view.