Transdisciplinary RFP: Center for Human-Centric Augmented and Virtual Reality
As cool as augmented and virtual reality might be, we have barely scratched the surface of their potential. This transdisciplinary team seeks to change that.
TL;DR
With a fundamental focus on understanding human perception and cognitive and social functions, faculty members from optics, engineering, natural sciences, medicine, and the libraries seek to awaken the awesome potential of augmented and virtual reality. They aim to do it through a center that catalyzes fundamental research, advances in hardware, technology translation, enhancement of computing capabilities, and the development of applications. Jump to: their big idea; the why and why us; implications for Rochester’s reputation; the last word.
Before we dive in, let’s establish some fundamental definitions.
Augmented reality (AR) puts digital content on top of our physical world, changing how we see and interact with it in real-time. A classic example is Pokémon Go, but day-to-day, many of us experience it through Snapchat and Instagram filters.
Virtual reality (VR) is a fully immersive experience. A VR headset cuts you off from the physical world and replaces it with a wholly new, simulated environment—such as a stream of music notes you need to slash with lightsabers in the rhythm-based game Beat Saber.
Extended reality (XR) is an umbrella term used to refer to AR, VR, and everything in between.
(If this has you itching to experience XR, visit the Mary Ann Mavrinac Studio X, Rochester’s XR hub, on the first floor of the Carlson Science and Engineering Library. There, you can try the Microsoft HoloLens 2 (AR), Meta Quest 3 (VR), and other XR equipment.)
Now that we’ve covered the basics, we can get to the good stuff.
In the mid-1960s computer scientist Ivan Sutherland envisioned a world with AR and VR, which he presented in his essay “The Ultimate Display.” (You can read the essay in full via Wired, which calls it a “seed-bomb for emergent technology.”) Despite this technology being on our innovation radar for decades, we’re still waiting for it to live up to its potential and transform how we access information and interact with the world.
Immature optics, a lack of powerful computational hardware, and insufficient digital content have slowed progress in the visual realm. Similarly, audio AR/VR systems—from basic stereo to more sophisticated object-based, multichannel setups—have been the subject of intensive research but are far from the ceiling of possibility. Finally, seamless integration with human perception, cognition, and behavior is limited. The proposed Center for Human-Centric Augmented and Virtual Reality (CHAVR) has designs to advance the state of AR/VR.
The team
Co-leads:
-
Nick Vamivakas
Marie C. Wilson and Joseph C. Wilson Professor of Optical Physics
Dean, Graduate Education and Post-Doctoral Affairs -
Duje Tadin
Professor, brain and cognitive sciences, ophthalmology, neuroscience, Center for Visual Science
Director, training for Center for Visual Science -
Meg Moody
Assistant director, Studio X -
Mujdat Cetin
Professor, electrical and computer engineering, computer science
Robin and Tim Wentworth Director, Goergen Institute for Data Science and Artificial Intelligence
Director, New York State Center of Excellence in Data Science and Artificial Intelligence
-
Jannick Rolland
Brian J. Thompson Professor of Optical Engineering
Professor, optics, biomedical engineering, Center for Visual Science
Director, Center for Freeform Optics
Director, R.E. Hopkins Center for Optical Design and Engineering -
Susana Marcos
Nicholas George Endowed Professor in optics, ophthalmology, optics
David R. Williams Director, Center for Visual Science -
Benjamin Suarez-Jimenez
Associate professor, neuroscience
Associate professor, Center for Visual Science
Okay—Picture this…
What’s the main idea behind the center?
With a focus on human-centric engineered systems, this team seeks to awaken the potential of AR/VR through a center that catalyzes fundamental research, advances in hardware, technology translation, enhancement of computing capabilities, and the development of applications.
In the context of the team’s work, “human-centric” considers the following:
- User experience. Is it easy to use and comfortable? How well does it respond to gestures and eye movement?
- Perceptual capabilities. How well does it align with how we perceive and experience the world?
- Can it be used by people with disabilities? Does it support a diverse user base?
- How can it be used to support critical aspects of human life, such as education, health care, etc.?
The CHAVR team will take a dual approach to achieving their goals.
Top-down strategy
There are currently several fundamental challenges that lead to a poor AR/VR experience, which this approach aims to tackle head-on.
The CHAVR team has already done groundwork around a system that allows users to experience visual and audio stimuli in ways that feel real. They’re doing this by addressing issues that confuse our brains and otherwise ruin the illusion—and the fun.
- Field of view. Right now, many AR/VR systems offer visual windows that are varying degrees of narrow. This team wants to blow off the frames to create a more natural, immersive experience.
- Image resolution. It’s hard to feel like you’re in space if the alien in front of you is pixelated. The fix here is to deliver images in human eye-level resolution.
- Visual comfort. Common side effects of current AR/VR technology are headaches and visual fatigue, and it’s because of how we focus on objects and perceive their distance. For example, hold your finger in front of your eyes, and they’ll adjust to see it clearly as you move it closer and further away. But this process gets wonky in the AR/VR world because our eyes and brain have to reckon with the closeness of the screen and the varying distances of AR/VR content. (If you want to explore this issue, it’s called the vergence-accommodation conflict.)
- Eye movements. Here, the team is looking to take what can be a design barrier and make it an asset.
The ultimate goal is to take all of that and incorporate it into high-performing, wearable tech that is lightweight and has an ergonomic design. To support this effort, the team will pursue large-scale grants, which includes resubmitting their application to become a National Science Foundation Engineering Research Center.
Bottom-up strategy
Partnering with Studio X and the Health Lab, the center aims to be a University-wide support system, providing researchers with technical and human resources, such as high-end equipment and XR specialists. This approach will focus on three kinds of service:
- Facilitating AR/VR-related research and the development of novel applications
- Developing potential users’ understanding of and capacity to leverage AR/VR.
- Cultivating a research landscape that actively seeks the use of AR/VR technology to achieve innovative and groundbreaking results.
To further develop the comprehensive vision of the center, the team used planning grant funds to hold an AR/VR symposium on October 25, 2024 to highlight the breadth and depth of AR/VR work at Rochester through presentations, hands-on demonstrations, and showcases.
Success is in the air
Why and how is Rochester poised to take this on, and what strengths is this center bringing to the table?
As of the fall of 2022, Rochester’s AV/VR research takes place in about 50 labs and is supported by more than $156 million in funding. What’s more, the University already has an established relationship with Meta (formerly Facebook), one of the AR/VR industry leaders.
Currently, XR work at the University of Rochester is organized around three focus areas:
- Fundamental science. A deep understanding of human perception (e.g., vision, audition, touch, and balance) and how it interfaces with our cognitive systems is fundamental to creating seamless and convincing AR/VR experiences.
- Core technology. Future advances in AR/VR will depend on fundamental advances in optics, photonics, imaging, haptics, displays, AI, data science, acoustics and audio, material science, computation and electronics, and sensing.
- Novel applications. Nearly every aspect of the University will benefit from the means to create realistic, immersive environments to advance research, educational mission, workplace development, and medicine.
It is becoming clear that end-to-end design and simulation tools are needed to guide the development of AR/VR interfaces that focus on haptic, visual and auditory elements and how they interact with our senses. This center will stand apart from other academic and industrial research labs through efforts that combine human perception and engineering. Rochester is exceptionally well-equipped to integrate perceptual science into hardware and system design. Additionally, Rochester’s combination of expertise in optical design and visual, haptic and auditory science is also unique, creating the opportunity for the University to create an entirely new approach to designing systems that interact with human senses.
Advances in visual, haptic and auditory displays will propel progress in AR/VR, but they pale in importance compared to the need for breakthroughs in low-power mobile computing. Think about the difference between Zack Morris’s cell phone and today’s smartphone—we need that kind of upgrade for AR/VR headsets, which are still too bulky and consume too much power. Also, existing CMOS-based technologies are begging for improvements as they are being pushed to their extreme limits. With Rochester’s resources in its electrical and computer engineering, computer science, chemical and mechanical engineering, and materials science, this center could lead the high-risk, high-reward research that develops specialized processors and computation-in-memory architectures that could help unlock the next level of AR/VR.
Reputation. Reputation. Reputation.
How will the success of the center enhance Rochester’s prominence and prestige?
In 2024, Rochester was ranked 46th in—to our knowledge—the only rankings that exist for AR/VR programs at a college or university. The CHAVR team does not believe this is representative of Rochester’s work or capabilities in the space. A center would help increase the University’s visibility and output by facilitating and enhancing AR/VR-related work on two levels.
Macro. Although Rochester has demonstrated excellence in many key areas of AR/VR (e.g., optics, sound and human cognition), the cross-fertilization of ideas and research across these disciplines is limited. A center would also drive future breakthroughs by facilitating research in human perception, engineering, and computing. In addition, the center would fuel the exchange of ideas among researchers, creating new opportunities for collaboration.
Micro. Having a structure that facilitates transdisciplinary work in AR/VR would naturally enhance the quality of the work and its impact on the field, raising the profile of Rochester and its faculty. The center would also help the University attract top students and encourage their involvement as it would serve as the home for the XR PhD training program after the NSF grant is completed.
The last word
If you have a smartphone, chances are it has become an essential tool for your day-to-day life because of its broad functionality. A common criticism of our reliance on—or maybe infatuation with—our phones is they pull our focus to a screen, away from the real world around us. (Think about the last time you experienced something through the lens of your phone’s camera rather than your eyes.)
But let’s put the phone aside and consider people in the Deaf community. A deaf person learning how to make something might face the challenge of splitting their focus between what’s at their hands and a person signing instructions.
The CHAVR team is envisioning a world with less disconnect between how we experience the world and the technology that enhances it. Team co-lead Nick Vamivakas refers to the work as “human-in-the-loop engineering design,” which seeks to create a seamless connection between the hardware and the user. “The idea is that the person isn’t using the device as much as the device becomes part of the user,” Nick says. “At the moment, I don’t know of anywhere this is being done. Because Rochester has expertise that would allow us to leverage native cues, such as eye movements and brain signals, we have an opportunity to lead the development of systems that fully integrate the user.”
