Over a video conference presentation, Eugene Jahn showed viewers an augmented reality program he created to help aspiring Michael Jordans shoot the perfect basket, showing the best path and angle to become a better shooter. The Allen School sophomore is a student in the Virtual Reality Systems CSE 490V taught by affiliate instructor Douglas Lanman, Director of Display Systems Research at Facebook Reality Labs.
The students — five graduate students, one sophomore, two juniors and 23 seniors, all studying computer science or computer engineering — spent the winter quarter learning about virtual reality systems, building software and hardware for a complete VR headset in 10 weeks. The hands-on course introduced students to the cross-disciplinary skills needed to create their own VR systems. Jahn and his classmates presented their final projects over Zoom as part of a virtual demo day, instead of in person as originally planned, after the University of Washington moved classes online due to COVID-19.
“I gave the students the option of not doing the final project because they didn’t have as much time in the lab to build their system, but all of them chose to do it — they all have a ‘maker’ mentality,” Lanman said during his introduction to the online demo day. “Although the students planned to have more time in the maker space in the Reality Lab, they made the most of finishing up their projects and presenting them over a conference call.”
Lanman said he had long dreamed of teaching a course at UW. He saw his chance when he moved to Seattle to join Facebook Reality Labs, reaching out to Allen School faculty shortly after his arrival to discuss possible courses. He said the moment he had been waiting for came when Facebook co-funded the launch of the UW Reality Lab.
“The creation of this center positioned UW, and the greater Seattle region, at the center of augmented and virtual reality revolution,” Lanman said. “When I read about the lab, I knew it was the right time to train a new generation of students to join the growing AR/VR industry being built in our backyard.”
Lanman emailed his idea to Brian Curless and Steve Seitz, who, along with Ira Kemelmacher-Shlizerman, comprise the Reality Lab’s leadership team. They agreed to the course and Lanman began preparing the materials.
Putting the course together, Lanman said, was a major undertaking.
“The class goes through almost everything you need to know about the process of building VR headsets, the research that goes into it and actually building one over the course of the quarter,” said Andrew Wei, a graduate student who completed the course. “Then you get to go a little further and you get to explore a bit and do something in VR that you’re interested in, so it made it a great experience.”
As you might expect, the students needed to become very comfortable with computer graphics to set up the rendering engine, Lanman explained. In order to track the headsets, they also had to master a good bit of linear algebra, signal processing, and computer vision methods. Over six assignments the students programmed projects in JavaScript, WebGL, OpenCV, and Arduino frameworks. On top of all this programming, they also had to physically assemble their headsets.
“For many, this was their first time building a piece of hardware, since most are CS students,” he said. “Learning to accept that you’ll break some, or many, parts is an important, albeit stressful, lesson. Thanks to funding from Facebook, it wasn’t an expensive lesson for the students! I’m happy to report that more than 30 students went through this course and are now the fearless hardware-software hackers I dreamed of inspiring.”
From sourcing hardware components for the headset kits to preparing comprehensive lecture notes, the course was custom-built from the ground up. Gordon Wetzstein, an electrical engineering professor at Stanford, had already built course materials over the last five years. He and Lanman have published numerous academic papers together and share an enthusiasm for augmented reality and VR display technology.
“My first call after getting the green light to bring this course to UW was to Gordon,” Lanman said. He had referred to this as an ‘experimental course’ when he first started offering it at Stanford and he said he was excited to see it ‘graduate’ to other schools.”
Building off of Wetzstein’s course, Lanman added his own spin, focusing both on computer graphics and computer vision. One unique addition was to implement a positional tracking system for the headset that used the students’ webcams. This eliminated the need for any custom hardware and meant that every student would be able to demo a fully working modern VR system to their friends and family, starting from a box of fairly simple parts. Lanman said they also spent quite a lot more time talking about the optics and display technologies behind modern AR/VR systems, “which happens to be what I work on in my day job.”
Kirit Narain was an undergraduate teaching assistant for the course. Last summer, Narain decided to try Wetzstein’s course on his own, out of an interest in VR and wanting to build his own headset. He said the two courses have some similarities and many differences. One being Lanman.
“490V leans slightly more into Doug’s expertise in displays, graphics and optics – and is more focused on understanding the cutting edge in each of these fields, including solutions still in the research stage,” Narain said. “There is also a bigger focus on augmented reality displays as well, which in my opinion makes it a more rounded class and gives students taking it an amazing insight into exactly how each of these systems works and prepared to take on their challenges.”
Lanman also added field trips and guest lectures into the course. After all, Lanman explained, Seattle is the growing center of the AR/VR universe, so how hard could it be to give students an opportunity to see the latest, greatest technologies — and meet the people behind them — at our local companies?
“As with the UW faculty, researchers and engineers at Microsoft, Valve, and Facebook Reality Labs were eager to be part of this course. This is the part of the course I’m most proud of: providing an opportunity to unite individuals passionate about AR/VR across the region and try to make something truly unique for the students,” Lanman said. “It was very exciting to see members of these companies, and others, offer guest lectures and sit in on some of the classes.”
Lanman said that building state of the art AR/VR display systems is not a theoretical or pure software exercise: his students had to get their hands dirty and yes, occasionally break things.
“I hope that this course put hardware projects on students’ radars. This course was designed for students receptive to this path: if you are scared of building a VR headset from scratch, you probably didn’t register. By the end of this course, I can see the students have really sharpened their maker skills,” Lanman said. “I told them at the beginning of the class that research is not as mysterious or difficult as it’s depicted in movies. I’m not sure they believed me. By the end, I am happy to see many students adopt the mentality of a graduate student: if you’re diligent in understanding what’s been done before, it’s not too difficult to figure out the path forward, even if it seems like walking into the unknown.”
For Narain, who was a first time TA, the experience was “unbeatable.” He was particularly keen to play a role in helping students to build a strong conceptual foundation and understanding of the numbers and algorithms, in addition to answering questions and helping them to debug their code. According to Narain, the course material was not easy, and having that maker attitude was crucial to student success. There also was a lot more nitty-gritty mathematics involved than most CSE classes, and working directly with hardware feels unfamiliar to some students.
“Having conquered these challenges over the last 10 weeks, the boost in student ambition and confidence could clearly be seen with their innovative final projects,” Narain said. “The novelty of AR/VR means there are no standard set of tools or methodologies for interaction yet. Watching the students experiment with different approaches and discovering what works and doesn’t is very interesting for me, too.”
Neil Sorens, a senior majoring in computer science, said he appreciated the way the class pulled together knowledge from many different disciplines: math, physics, computer science and various areas within computer science such as graphics and computer vision.
“Having an incredibly passionate and fun instructor was really motivating because the class was highly challenging,” Sorens said. “It was a good mix of practical and theoretical work, and we had a lot of freedom in choosing our final projects.”
During the virtual demo day, Sorens and his fellow students in the VR Systems class presented their final projects to Lanman, their TAs and each other. Allen School faculty and industry experts were also invited to drop into the Zoom meeting and see the final projects. Students worked on projects that varied, including hardware, body and hand tracking, eye tracking, rendering, audio, training and education, and even applications. In pairs or on their own, students presented a total of 19 projects as a culmination of what they learned over the quarter.
In addition to Jahn’s basketball training project, other presentations include:
Wide-FOV VR headsets using Fresnel Lenses (Andrew Wei and Daoyi Zhu): A headset with peripheral vision, the team created a prototype with a wide field of vision.
360 degree vision using FOV compression (Neil Sorens): A headset with a 110 degree field of vision and equipped it with a 360 degree camera for specialized real-world applications to give users “eyes in the back of their head.”
Finger tracking using magnetometers (Alex Mastrangelo and Paul Yoo): A glove with magnetometers built in the fingertips that has high accuracy and low power for gaming.
Inverse kinematics and full-body tracking (Terrell Strong): A headset and controllers with full tracking points and animated arms and a torso to better understand a user’s sense of presence in VR experiences.
Exploring two-handed interactions (Andrew Rudasics): A task for users to manipulate a two-handed object to determine the most comfortable and accurate way to model for users.
VR Wings (Rory Soiffer and Everett Cheng): A game that allows players to fly like a bird through a virtual world by flapping their arms, using custom wing-like controllers.
Eye tracking for VR gaming (Alex Zhang): A shooting game using virtual reality by only using eye tracking techniques.
Real-time foveated ray tracing (Frank Qin and Anny Kong): Implemented foveated ray-tracing to make images more detailed and therefore look more realistic.
VR Volume Rendering (Xiao Liang, Jeffrey Tian and Nguyen Duc Duong): A 3D experience to use for medical scans in a more realistic manner.
Spatial audio for VR gaming (Thomas Hsu and Christie Zhao): A game that makes the player focus on spatial audio to navigate through the VR world.
VR batting cage (Dylan Hayre): A game that allows users to practice hitting baseballs and adjust their swing to build his or her skills.
3D drawing in VR (Daniel Lyu and Lily Zhao): A VR app that supports letter animation to enhance the learning of vocabulary.
VR galaxy tour (Natalia Abrosimova and Wenqing Lan): An experience that allows users to experience the galaxy through a VR headset.
Crime scene investigation (Zhu Lu and Weihan Lan): A 3D degree crime scene that allows users 360 degree view of the room to explore for clues.
Sketching in AR with the ability to add 3D model support (Anthony Lu and Jacky Mooc): An application that allows users to import avatars and modify them by drawing or sketching, drawing on their AR device.
VR dueling (Robin Schmit): A multiplayer game was created to sync real-time information from two or more headsets in order to play games with multiple participants.