“In recent years, many machine learning models have achieved really high accuracy scores on a number of natural language processing tasks, often coming very close to or outperforming human accuracy,” Guestrin said. “However, when these models are evaluated carefully, they show critical failures and a basic lack of understanding of the tasks, indicating that these NLP tasks are far from solved.”
Much of lead author Ribeiro’s prior work is on analysis and understanding of machine learning and NLP models. As the driving force of the project, he began testing and found critical failures in several state-of-the-art models used for research and commercial purposes — despite the fact that these models out-performed humans on standard benchmarks. To remedy this problem, Ribeiro and his colleagues created a process and a tool called CheckList, which provides the framework and tooling necessary to enable developers to break their tasks into different “capabilities” and then write tests for these using a variety of test types.
People in several industries found new and actionable bugs to fix by using CheckList, even in systems that had been extensively tested in the past. In fact, NLP practitioners found three times more bugs by using CheckList than compared to existing techniques.
“I did a case study with a team at Microsoft and they were very happy to discover a lot of new bugs with CheckList”,” Ribeiro said. “We also tested models from Google and Amazon and we’ve received positive feedback from researchers in both companies.”
Including CheckList, University of Washington researchers earned three out of the five recognized paper awards at ACL this year. The second of these, a Best Paper Honorable Mention, went to a group of researchers that included professor Noah Smith, Ph.D. student Suchin Gururangan, and postdocs Ana Marasović and Swabha Swayamdipta for their work on “Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks.” The team — which also included applied research scientists Kyle Lo and Iz Beltagy of AI2 and Northwestern University professor Doug Downey — explored whether they could apply a multi-phased, adaptive approach to pretraining to improve the performance of language models that are pretrained on text from a wide variety of sources, from encyclopedias and news articles, to literary works and web content. Using a method they referred to as domain-adaptive pretraining (DAPT), the researchers showed how tailoring a model to the domain of a specific task can yield significant gains in performance. Further, the team found that by adapting the model to a task’s unlabeled data — an approach known as task-adaptive pretraining (TAPT) — they could boost the model’s performance with or without the use of DAPT.
In addition, Allen School adjunct professor Emily Bender of the UW Department of Linguistics and Saarland University professor Alexander Koller were awarded Best Theme Paper for “Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data,” in which they draw a distinction between form and meaning in assessing the field’s progress toward natural language understanding. While recognizing the success of large neural models like BERT on many NLP tasks, the authors assert that such systems cannot capture meaning from linguistic form alone. Instead, Bender and Koller argue, these models have been shown to learn some reflection of meaning into the linguistic form that has proven useful in various applications. With that in mind, the authors offer some thoughts on how members of their field can maintain a healthy — but not overhyped — optimism with respect to communicating about their work with language models and the overall progress of the field.
Congratulations to all on an outstanding showing at this year’s conference!
Allen School professor emeritus Richard Ladner, a leading researcher in accessible technology and a leading voice for expanding access to computer science for students with disabilities, has been named the 2020 recipient of the Public Service Award for an individual from the National Science Board (NSB). Each year, the NSB recognizes groups and individuals who have made significant contributions to the public’s understanding of science and engineering. In recognizing Ladner, the board cited his exemplary science communication, diversity advocacy, and well-earned reputation as the “conscience of computing.”
A mathematician by training, Ladner joined the University of Washington faculty in 1971. For much of his career, he focused on fundamental problems underpinning the field of computer science as one of the founders of what is now the Allen School’s Theory of Computation research group. After making a series of significant contributions in computational complexity and optimization — and later, branching out into algorithms and distributed computing — his career would take an unexpected but not altogether surprising turn toward accessibility advocacy and research.
Ladner enrolled in an American Sign Language course at a local community college, a move that represented a “return to his roots” after growing up in a household where both parents were deaf. That experience spurred him to begin volunteering in the community with people who were deaf and blind and to occasionally write about accessibility issues.
Then, in 2002, Ladner began working with Ph.D. student Sangyun Hahn at the UW. Hahn, who is blind, related to Ladner how he was having trouble accessing the full content of his textbooks; mathematical formulas had to be read aloud to him or converted into Braille, while graphs and diagrams had to be manually traced, labeled in Braille, and printed on an embosser. His student’s frustration was the impetus for Ladner and Hahn to launch the Tactile Graphics project, which automated the conversion of textbook figures into an accessible format. Ladner followed that up with MobileASL, a collaboration with Electrical & Computer Engineering professor Eve Riskin to enable people who are deaf to communicate in American Sign Language using mobile phones. Ladner also mentored many Ph.D. students in accessibility research — among them Anna Cavender (Ph.D., ‘10), who developed technology to consolidate a teacher, display screen, sign language interpreter, and captioning on a single screen; Jeffrey Bigham (Ph.D., ‘09), who developed a web-based screen reader that can be used on any computer without the need to download any software; Information School alumnus Shaun Kane, who developed technology to make touchscreen devices accessible to people who are blind; and Shiri Azenkot (Ph.D., ‘14), who developed a Braille-based text entry system for touchscreen devices.
Ladner’s approach to accessibility research is driven by the recognition that to build technology that is truly useful, you have to work with the people who will use it. It’s a lesson he took from his earlier experience as a volunteer, and one that he has emphasized with every student who has worked with him since. During his career, Ladner has mentored 30 Ph.D. students and more than 100 undergraduate and Master’s students — many of whom followed his example by focusing their careers on accessible technology research.
“I visited Richard’s lab at the University of Washington just over 10 years ago. While I did get to see Richard, he was most interested in my meeting his Ph.D. students — and I could see why,” recalled Vicki Hanson, CEO of the Association for Computing Machinery. “Richard had provided an atmosphere in which his talented students could thrive. They were extremely bright, enthusiastic, and all involved in accessibility research. I spent the day talking with his students and learning about their innovative work.
“All were committed to developing technology that would overcome barriers for people with disabilities. Sometimes there are barriers in being able to use technology – in other cases, however, the use of technology actually provides opportunities to remove barriers in various aspects of daily living,” Hanson continued. “Richard’s students were working on both of these aspects of accessibility. The collegial and inspiring interactions among his students would serve as a model of research collaboration for computing labs everywhere.”
Ladner’s impact on students extends far beyond the members of his own lab. In addition to his research contributions and mentorship, Ladner has been a prominent advocate for providing pathways into computer science for students with disabilities. To that end, he has been a driving force behind multiple initiatives designed to engage a population that, until recently, was often overlooked in technology circles.
“When we think about diversity, we must include disability as part of that,” Ladner noted. “The conversation about diversity should always include disability.”
To that end, Ladner has been a leading voice for the inclusion of people with disabilities in conversations around improving diversity in technology. He served as a founding member of the board of the national Center for Minorities and People with Disabilities in Information Technology (CMD-IT). The organization hosts the ACM Richard Tapia Celebration of Diversity in Computing, which attracts an estimated 1,500 attendees of diverse backgrounds and abilities each year. Ladner was also a member of the steering committee that established the Computing Research Association’s Grad Cohort Workshop for Underrepresented Minorities and Persons with Disabilities (URMD) for beginning graduate students. In discussions leading up to the program’s launch, Ladner was instrumental in making sure that the “D” made it into the name and scope of the workshop.
Ladner has also worked directly with colleagues and students around the country to advance diversity in the field. The longest-running of these initiatives is the Alliance for Access to Computing Careers (AccessComputing), which he co-founded with Sheryl Burghstahler, Director of the UW’s DO-IT Center, with funding from National Science Foundation’s Broadening Participation in Computing program. AccessComputing and its 60 partner institutions and organizations support students with disabilities to successfully pursue higher education and connect with career opportunities in computing fields. Since its inception in 2006, that initiative has served nearly 1,000 high school and college students across the country. For seven consecutive years, Ladner also organized the annual Summer Academy for Advancing Deaf and Hard of Hearing in Computing to prepare students to succeed in computing majors and careers.
More recently, Ladner partnered with Andreas Stefik, a professor at the University of Nevada, Las Vegas, on AccessCSForAll. That initiative is focused on developing accessible K-12 curricula for computer science education along with professional development for teachers. The duo also partnered with Code.org to review and modify the Computer Science Principles Advanced Placement course to ensure that online and offline course activities met accessibility standards for students with disabilities. This included developing accessible alternatives to visually-based unplugged activities as well as making interactive tools that would work with screen readers. Ladner and his collaborators on the project earned a Best Paper Award at last year’s conference of the ACM’s Special Interest Group on Computer Science Education (SIGCSE 2019) for their efforts.
This past spring, Ladner was one of nine researchers to co-found the new Center for Research and Education on Accessible Technology and Experiences (CREATE) at the UW. The mission of CREATE is to make technology accessible and to make the world accessible through technology. The center, which was established with an inaugural $2.5 million investment from Microsoft, consolidates the efforts of faculty from the Allen School, Information School, and departments of Human Centered Design & Engineering, Mechanical Engineering, and Rehabilitation Medicine who work on various aspects of accessibility.
“Richard is a gifted scientist and mentor who really helped to put UW on the map when it comes to accessible technology,” said professor Magdalena Balazinska, Director of the Allen School. “As a staunch advocate for innovation that serves all users, his impact on computing education and research cannot be overstated.”
Since his retirement in 2017, Ladner has remained engaged with the Allen School community and continues to invest his time and energy in accessible technology research and increasing opportunities for students with disabilities in computing fields. In accepting this latest accolade — one in a long line of many prestigious awards he has collected during his career — Ladner expressed optimism that accessibility’s importance is recognized by an increasing number of his peers.
“I am honored to receive this recognition from the National Science Board and heartened that the scientific community is rising to the important challenge of supporting students with disabilities,” Ladner said.
Allen School professor and alumnus Michael Ernst (Ph.D., ‘00) has had a distinguished research career that spans more than two decades and includes multiple, influential contributions to programmer productivity through software analysis, testing and verification. In recognition of his impressive body of work, Ernst was recently recognized with the 2020 Outstanding Research Award from the Association for Computing Machinery’s Special Interest Group on Software Engineering (ACM SIGSOFT).
Since he returned to his alma mater in 2009 to take up a faculty position with the Allen School’s Programming Languages and Software Engineering (PLSE) group, Ernst has advanced the state of the art in the field on multiple fronts and earned many accolades along the way. The latest of these, SIGSOFT’s most prestigious award, recognizes his enduring impact on the theory and practice of software engineering.
“Michael Ernst is one of the most accomplished software engineering researchers. His work has led to many tools that are applied daily to real life software products,” said SIGSOFT Chair Tom Zimmermann, Senior Principal Researcher at Microsoft. “His Daikon tool was revolutionary and an elegant solution that impacted the research community in testing, verification, and programming languages.”
Daikon is a software system that dynamically detects likely program invariants. Ernst and his co-authors first presented Daikon in the 1999 paper, “Dynamically discovering likely program invariants to support program evolution,” while he was a Ph.D. student with fellow graduate student Jake Cockrell (M.S., ’99), Allen School alumnus and UCSD professor William G. Griswold (Ph.D., ’91), and the late Allen School professor David Notkin. In the paper, Ernst and his collaborators described novel techniques, such as tracing variables of interest, for enabling programmers to easily identify program properties that must be preserved when modifying code. The team’s approach initiated an important new line of research, earned the 2013 Impact Paper Award, and continues to be relevant more than two decades later.
Another research highlight of Ernst’s impressive career so far was a paper he co-authored with Carlos Pacheco of Google, and Shuvendu Lahiri and Thomas Ball of Microsoft Research called “Feedback-directed random test generation” in 2007. In the paper, Ernst and his colleagues presented a test generation technique and corresponding tool, Randoop, which generates tests for programs written in object-oriented languages such as Java and .NET. The technique put forward by the team generates one test at a time, executes the test, and classifies it as a normal execution, a failure, or an illegal input. Based on this information, the technique biases the subsequent generation process to extend good tests and avoid bad tests. Ernst and the team earned the Most Influential Paper Award from the International Conference on Software Engineering (ICSE) in 2017, and Randoop remains the standard benchmark against which other test generation tools are measured.
A year after his ICSE recognition, Ernst received a 2018 Impact Paper Award from the International Symposium on Software Testing and Analysis (ISSTA) for the 2008 paper, “Practical pluggable types for Java.” With this project, Ernst and his co-authors set out to address a well-known verification problem for programmers working in Java: while type checking could prevent many bugs, it couldn’t prevent enough bugs. To address this, Ernst and his colleagues developed the Checker Framework, a scalable tool for creating expressive and concise pluggable type systems for Java. The Checker Framework incorporated a type system component for defining type qualifiers and their semantics, and a compiler plug-in that enforces the semantics. It is also backward-compatible and integrates easily with standard development environments and tools, making it useful to programmers seeking to write bug-free code as well as type-system designers looking to evaluate and deploy their type systems. At the time of publication, Ernst was a faculty member at MIT, working with researcher Jeff H. Perkins, graduate student Matthew M. Papi, and undergraduates Mahmood Ali and Telmo Luis Correa Jr.
Among his many other accolades, Ernst earned a 2019 ISSTA Impact Paper Award for “HAMPI: a solver for string constraints,” a 2011 European Conference on Object-Oriented Programming (ECOOP) Best Paper Award and a total of nine ACM Distinguished Paper Awards. He was the recipient of the inaugural John Backus Award (2009) — created by IBM to honor mid-career university faculty members — and was elected a Fellow of the ACM in 2014.
“Michael is an exceptional software-engineering researcher and his continual momentum to produce research has advanced the field and has made some incredible real-world impacts,” said professor Magdalena Balazinska, director of the Allen School. “In addition to his research, he’s been an admirable influence on our next generation of computer scientists and engineers. We’re grateful he is a part of the Allen School faculty.”
In 2018, Ernst was formally recognized for his work as an educator and mentor with the CRA-E Undergraduate Research Faculty Mentoring Award from the Computing Research Association. The award recognizes faculty who provide exceptional mentorship and support to student researchers.
“I’m passionate about mentoring because doing research as an undergraduate changed the course of my career and my life,” Ernst said at the time. “I love making new discoveries — and I get a vicarious thrill from helping others to experience that same wonder.”
No one could accuse Ph.D. student Vikram Iyer of just winging it. Since his arrival at the University of Washington, Iyer has advanced ground-breaking innovations in low-power wireless communication and computation to expand the Internet of Things, from 3D-printable wireless objects capable of storing and transmitting data, to insect-scale platforms that provide a bug’s eye view of the world. As a sign of just how his ideas have taken flight, today Iyer was named one of three recipients of the Paul Baran Young Scholar Award by the Marconi Society.
“By creating low cost, mobile IoT devices that can help answer questions and solve problems in any environment, Vikram’s work supports the Marconi Society’s mission of bringing the opportunity of the network to everyone,” said internet pioneer Vint Cerf, Chair of the Marconi Society, in a press release. “We are proud to welcome him to the Marconi family.”
The award recognizes innovative young engineers who show extraordinary technical acumen, creativity and promise for creating tomorrow’s information and communications technologies to support a digitally inclusive society. The honor came as no surprise to Iyer’s advisor, Allen School professor Shyam Gollakota, who recognized early on that his student would be a highflier.
”Vikram is a one-of-a-kind creative interdisciplinary researcher who is also humble,” said Gollakota. “He develops creative solutions that are at the intersection of hardware, software and biology. In so doing, he transforms what was once science fiction into reality.”
Iyer, a student in the UW Department of Electrical & Computer Engineering, began working with Gollakota in the Networks & Mobile Systems Lab in 2015. Among their first projects together was a collaboration with Allen School and ECE professor Joshua Smith of the Sensor Systems Laboratory on an ultra-low power system to provide wireless connectivity for implantable devices. Interscatter — short for intertechnology backscatter — employs a technique called backscatter communication to convert Bluetooth transmissions to WiFi and ZigBee signals over the air using commodity devices. As part of that project, Iyer and his collaborators created the first prototype contact lens antenna and an implantable neural recording interface capable of communicating directly with smartphones and smart watches. The team earned a Best Paper Award from the Association for Computing Machinery’s Special Interest Group on Data Communications (ACM SIGCOMM).
More recently, Iyer, Gollakota and their colleagues teamed up with professor Sawyer Fuller of the UW Mechanical Engineering Department’s Autonomous Insect Robotics (AIR) Lab to enable new wireless robotic technologies to take flight. The result was Robofly, the world’s first wireless fly-sized drone to achieve liftoff. Unlike previous insect-scale drones, Robofly does not require a wire to the ground to supply power and control signals — a significant achievement on the path toward autonomous robot flight. The team’s bio-inspired design featured dual flapping wings driven by a pair of piezoelectric actuators and directed by a lightweight microcontroller, which issues a series of pulses mimicking the action of a biological fly’s wings. An onboard photovoltaic cell converts light from a laser beam into electricity to power the onboard components without the need for heavy batteries, while the first sub-100 milligram boost converter and piezo driver sufficiently boosts the voltage to enable RoboFly’s ascent.
While news of Robofly’s exploits took off, Iyer recognized that there are limitations to what a drone insect could do. For one thing, robotic liftoff was difficult to achieve. And commercial drones are limited in how long they can fly uninterrupted.
“This made me wonder, rather than building a system that mimics an insect, could we augment live insects with sensing, computing and communication functionalities to create a mobile IoT platform?” Iyer explained. “We could use this platform to study micro-climates on large farms, answer questions about insects’ behavior or collect air quality data at a more granular level than by using a handful of stationary sensors.”
Iyer with his advisor, Shyam Gollakota, unveiling 3D-printed wireless smart objects in 2017
To explore the idea, Iyer set up an amateur beekeeping operation in a room in the Paul G. Allen Center on campus. The result was Living IoT, a mobile platform that combines sensing, computation, and communication packaged into a tiny wireless backpack light enough to be carried by a bumblebee. The entire system — antenna, envelope detector, sensor, microcontroller, backscatter transmitter, and rechargeable battery — weighed in at just 102 milligrams, or around half a bumblebee’s potential payload. Because the system did not need to power flight, only data collection, the team could keep the weight down by designing the system to transmit data and recharge the battery when the bee returned to the hive each day.
For his latest project, which was recently published in Science Robotics, Iyer’s insect subjects kept their feet on the ground. Building off of the previous work with bees, Iyer and his collaborators created a new wireless backpack containing a tiny, steerable video camera operated via Bluetooth. This time, they fitted their system on two species of beetle to demonstrate the potential for insect-scale robotic vision. Dubbed “BeetleCam,” the system emulates a real bug’s energy-efficient approach to gathering visual information, which relies on head motion independent of its body, while a built-in accelerometer prolongs the battery life by allowing the system to capture images only when the beetle is in motion. Weighing in at a mere 250 milligrams, or roughly half the payload the insects can carry, the system enables the beetles to freely navigate terrain and climb trees.
The team used what it learned to design the world’s smallest power-autonomous terrestrial robot with vision — proving, once again, that good things really do come in small packages.
“This is the first time that we’ve had a first-person view from the back of a beetle while it’s walking around. There are so many questions you could explore, such as how does the beetle respond to different stimuli that it sees in the environment?” Iyer said in a UW press release. “But also, insects can traverse rocky environments, which is really challenging for robots to do at this scale. So this system can also help us out by letting us see or collect samples from hard-to-navigate spaces.”
Iyer and a bumblebee demonstrate Living IoT
Iyer is the second UW student — and second from Gollakota’s lab — to earn this prestigious award. His labmate and frequent collaborator, Rajalakshmi Nandakumar (Ph.D. ‘19), now a faculty member at Cornell Tech, was honored in 2018 for her work on mobile apps for detecting life-threatening health issues. In addition to Iyer, the Marconi Society recognized two other researchers with 2020 Young Scholar Awards: Yasaman Ghasempour at Rice University (soon joining the Princeton University faculty) for her work on efficient, ultra-high speed network connections for next-generation IoT, and Piotr Roztocki at Canada’s Institut National de la Recherche Scientifique (INRS) for his work on scalable quantum resources for “future-proofing” telecommunications network security. The honorees were selected by an international panel of engineers drawn from leading universities and companies.
“Our Young Scholars are the braintrust that will put the speed, security and applications of next generation networks into the hands of billions,” said Cerf.
Our latest student spotlight features Vancouver, Washington native Robert Minneker, who graduated in June with degrees in biomedical engineering and computer science and is continuing his studies in the Allen School’s fifth-year Master’s program. Using the skills they gained as undergraduates, Minneker and two friends created Tremor Vision, a web-based tool to help diagnose and treat Parkinson’s disease. The creators — Minneker, Allen School alumna Janae Chan (B.S. ‘19) and informatics graduate Drew Gallardo — were runners up in the 2020 Microsoft Imagine Cup.
Allen School: Congratulations on your success in the Imagine Cup! What was the experience like, particularly as it was virtual this year?
Robert Minneker: Thank you so much! I still can’t believe how far we made it. The Imagine Cup was an unforgettable experience. I learned so much about what it takes to pitch your idea in a concise and convincing way and what it takes to get a product to market and in the hands of those it’s intended for. I definitely think it would’ve been more fun to be in the same room as Drew and Janae pitching our idea to the judges, but being virtual was the right call given the circumstances. I really appreciate that Microsoft was flexible with the competition and continued it despite COVID-19. None of us had been to the Imagine Cup before and had no idea what to expect. It is one of the most memorable and influential experiences of my undergraduate career and something I will never forget.
Allen School: What is Tremor Vision, and how did the three of you come to be a team and bring it to fruition for the Imagine Cup?
RM: Tremor Vision began during DubHacks 2019, the University of Washington’s annual hackathon. I skateboarded with Drew pretty often — we actually met at a skatepark a few years ago — and quickly realized he shared a love for programming. We have been good friends ever since. Janae and I both studied BioE and CS, so we shared classes and were interns at Sage Bionetworks at the same time. We got pretty close. I knew both of them, but they didn’t know each other, and I knew they were both going, so we kind of just formed a team from there.
When we got to the hackathon, we had no idea what we were going to work on. Janae and I, both being BioE majors, wanted to do something related to health care. Drew is quite passionate about telehealth, so we started looking for open problems in that area that we thought we could tackle, particularly using machine learning. After some digging, we came across the challenges associated with detection and tracking of Parkinson’s and thought we would be able to propose a solution. We played around with many ideas and many approaches that night, and by the morning we had a working computer vision model that could detect Parkinson’s from a patient’s hand-drawn spiral, with a nice user interface to display the results to their physician. This was our first time at a hackathon, so we definitely didn’t know what to expect and especially didn’t expect to make something that worked and that was so well received by the judges and our fellow participants.
One of the prizes we won at DubHacks was the Azure Champ Prize for our use of Azure services, and part of that included fast-track entry to the Imagine Cup competition. None of us had heard of the competition before DubHacks but decided to apply since they showed interest in our project and believed in us. We definitely thought we would be quickly screened out of the competition and did not think we would go as far as we did!
Allen School: How has a top performance in the Imagine Cup impacted the future of Tremor Vision?
RM: We are still working on Tremor Vision. We have a minimum viable product developed and are currently participating in the NSF I-CORPS program via UW CoMotion focused on customer discovery. We plan to complete user testing and have a version ready for market by the end of the year.
Allen School: In addition to your success with Tremor Vision, you are continuing your studies for a fifth year in the B.S./M.S. program. What piqued your interest in computer science initially?
RM: I decided to study in the Allen School after completing most of the bioengineering major and taking CSE 142/143 as my engineering electives and participating in computational research — specifically, biomedical informatics. I quickly realized the potential that computing would have on health and medicine in the coming years and decided that I would really enjoy being a part of it. After seeing firsthand in my research how artificial intelligence and machine learning could transform health care, I knew I needed to become more knowledgeable in the area. Computer science seemed like the logical choice given the school’s strengths and faculty research interests.
Allen School: What do you enjoy most about being in the Allen School?
RM: The quality of the courses and the resources available to students were invaluable. Everything from the TA program to computer labs, career fairs, and the opportunities posted via the undergraduate blog. I felt as if I was part of a school that was doing its best to help me succeed.
The University of Washington has established a new endowed professorship, the Senosis–Paul G. Allen Endowed Professorship in Computer Science & Engineering, through the generosity of professors Shwetak Patel and Julie Kientz. The $1 million endowment, which was made possible in part by the acquisition of Patel’s mobile health startup Senosis Health by Google in 2017, will support recruitment and retention of Allen School faculty who pursue high-impact research aimed at solving meaningful, real-world problems for the benefit of society.
Patel directs the UW’s Ubiquitous Computing (UbiComp) Lab and holds the Washington Research Foundation Entrepreneurship Endowed Professorship in the Allen School and the Department of Electrical & Computer Engineering. Kientz serves as chair of the Department of Human-Centered Design & Engineering, where she directs the Computing for Healthy Living and Learning (CHiLL) Lab, and is an adjunct professor in the Allen School and the Information School. The pair joined the UW faculty in 2008 after earning their Ph.D.s from the Georgia Institute of Technology. Since arriving in Seattle, Patel and Kientz have applied their expansive view of computing to explore solutions for health care, education, environmental sustainability, and more.
The results have paid off not just for the university and the Seattle region, but also for society at large. Their success — and Patel’s direct experience with the benefits of holding an endowed professorship — inspired the couple to pay it forward by supporting future faculty members who, they hope, will not so much follow in their footsteps as blaze their own trails.
“As the beneficiary of the WRF Entrepreneurship Professorship, I have appreciated firsthand the role of endowments in enabling faculty to be creative and flexible in research. This kind of support empowers you to think outside the box and challenge existing assumptions,” explained Patel. “Julie and I hope that this new professorship will offer future faculty members that same freedom to pursue innovation that truly pushes the envelope — and pushes technology in new and unexpected directions that will have a positive impact on people’s lives.”
Thinking outside the box and pushing the limits of technology have been hallmarks of both Patel’s and Kientz’s research. Already in their careers, each of them has made groundbreaking contributions and set new directions for research that have established the UW as a leader in human-computer interaction, digital health, accessible technology, low-power sensing, and mobile computing.
Patel began his career focused on new capabilities for whole-home sensing. Recognizing that each appliance emits a distinct signal on a home’s electrical system, he and his students developed a technique for measuring power consumption at the individual device level with a single sensor. They later expanded their approach to monitoring water usage based on the pressure waves generated as each plumbing fixture is turned on and off. To commercialize this research, Patel and his team launched a company, Zensi, that was later acquired by Belkin. He followed that up with SNUPI Technologies, a company he and his collaborators started to commercialize a consumer-facing whole-home sensing system called WallyHome, which was subsequently acquired by Sears.
Inspired by the global growth in smartphone use, Patel began exploring how he and his team could leverage the increasingly sophisticated sensors built into today’s mobile devices to aid in the early detection and monitoring of disease — pioneering an entirely new field focused on mobile health sensing that has become even more compelling since the outbreak of the COVID-19 pandemic. By combining sensing data from inputs like the phone’s camera, microphone and accelerometer with new machine learning algorithms, Patel’s lab has worked with health care providers on a variety of health monitoring tools. Patel and a group of collaborators started Senosis Health with a view to commercializing their initial work in this space. The team also approached the Food and Drug Administration to obtain approval for the use of mobile apps in both clinical and at-home settings — a novel idea at the time, as the agency had no prior experience with mobile app development.
Kientz and Patel earned their Ph.D.s from Georgia Tech before joining the UW faculty
Since Google’s acquisition of Senosis three years ago, Patel has built on this work as the leader of the company’s Seattle-based engineering team focused on mobile health technologies. He hopes that by leveraging the acquisition to establish a new professorship, more faculty will be inspired to unite academic research with entrepreneurial impact.
“Being an entrepreneur has helped me to identify research problems I wouldn’t have previously considered solely as an academic,” Patel explained last year when he received the ACM Prize in Computing — the premier mid-career award in the field of computer science — from the Association for Computing Machinery. “That experience opened up opportunities for me to venture down research paths I wouldn’t have otherwise thought about.”
In addition to supporting an entrepreneurial mindset and a more expansive view of the role of academic research, Patel and Kientz also hope that holders of the professorship will model a commitment to diversity and inclusion in computing through their teaching, outreach, and service.
“We need to design and develop computing technology for all of society, not just a privileged subset. One of the ways we make sure we do that is to make the computing discipline representative of the people we are trying to serve,” Kientz said. “Shwetak and I feel strongly that our duty as educators and as researchers is to advance technologies and create an academic community that reflects a rich diversity of backgrounds and experiences. It’s important to us that the holders of this professorship also model these values.”
Kientz is keenly aware of the power of technology and education to be the great equalizer, and in more ways than one. Her research takes a human-centered approach to technology, combining ubiquitous computing, human-computer interaction, and informatics to design and study novel interactive technologies for health and education while also working to understand and reduce the burden technology places on the people who use it. It’s an approach that she honed early in her career, when she explored technology to support caregivers of young children in data-based decision-making. Rather than approach the topic solely as a computer scientist, though, Kientz spent significant time among the communities for whom she was designing — giving her firsthand insight into how technology could best support their work by, for example, making tracking childhood development data more fun and meaningful by linking it with sentimental mementos in a digital baby book and allowing it to be shared with family and friends.
Since then, Kientz and her students have developed tools to help people improve their sleep quality, support physical fitness goals of people with visual impairments, support inclusive education, help parents teach their children to self-regulate screen time, and to monitor children’s developmental progress. Her work on the latter led her to work on the launch of startup, BrightSteps, to assist parents and caregivers in monitoring children’s development and connecting them to resources.
The Senosis–Paul G. Allen Professorship leverages funds made available through the Paul G. Allen Professorship Matching Program. That program, which is supported by earnings of the endowment established by Mr. Allen and Microsoft upon creation of the school, provides a 1:2 match on individual gifts aimed at attracting and retaining exceptional faculty who will advance UW’s leadership in computer science education and research.
Patel and Kientz on a backpacking trip to the Enchantments last summer
The professorship is one of two UW initiatives announced today and funded with gifts from the couple. The other is a gift to Kientz’s home department to create the Kientz & Patel HCDE Student Emergency Support Fund. That gift will offer support to students facing near-term financial hardship, such as unexpected health care costs, car repairs, legal fees, emergency travel, and housing insecurity. The goal, Kientz explains, is to alleviate the burden for students who suddenly find themselves facing unexpected financial emergencies.
“Students who cannot make a rent payment may struggle with housing security, or one unpaid bill can begin to collect fee upon fee, quickly making payment completely unattainable,” she said. “This can make the difference between being able to stay in school or have to drop out.”
“Julie and Shwetak are both superstars who have enriched our university, our community, and our field in countless ways,” said professor Magdalena Balazinska, Director of the Allen School. “Time and again, they have demonstrated in tangible ways how technology can help solve some of society’s most vexing problems — from addressing disparities in education and health care, to conserving natural resources for a more sustainable planet. They are also both dedicated members of our community, supporting our mission through their service roles as department chair and associate director for research and innovation. With these gifts, they are once again leading the way in showing how UW faculty are truly a force for good.”
The Senosis–Paul G. Allen Professorship will become the Shwetak N. Patel & Julie A. Kientz–Paul G. Allen Endowed Professorship when the couple eventually become emeritus faculty or retire from the university. The professorship is the third endowment in the Allen School created as the result of faculty entrepreneurial activity. In 2017, the Guestrin Endowed Professorship in Artificial Intelligence and Machine Learning was established following Apple’s acquisition of Carlos Guestrin’s machine learning startup Turi. And the establishment of the Washington Research Foundation Entrepreneurship Endowed Professorship — the professorship held by Patel — was related to the acquisition by Microsoft of Oren Etzioni’s startup Farecast.
Way to go, Shwetak and Julie — thank you for your leadership and your generous support of faculty and students!
Tompa in Sieg Hall, 1999. Photo credit: John Zahorjan
Allen School professor Martin Tompa, an early and leading expert in computational molecular biology and a beloved mentor to undergraduate and graduate students, retired from the Allen School at the end of June. Tompa’s long and distinguished career spans an array of computer science research, including computational complexity, algorithms, and computational molecular biology. And then, of course, there’s his mastery of Schnapsen — the 300+ year old card game with an avid following in Europe and, thanks to Tompa, in the halls of the Allen School.
As an undergraduate at Harvard University in the early 70s, Tompa wanted to study computer science. It wasn’t offered as a major at Harvard at the time so he graduated in 1974 with a degree in applied mathematics. He still took every opportunity he could to take classes in computing and conducted undergraduate research with Thomas Cheatham, an early pioneer of extensible programming languages. As a computer science Ph.D. student at the University of Toronto, Tompa’s research focused on theoretical computer science, work that he continued at the University of Washington after he was hired onto the faculty in 1978.
“I first met Martin when, in the fall of 1977, I was on sabbatical leave at the University of Toronto where he was a graduate student. I was thrilled when he decided to come to UW when he finished up his Ph.D. in 1978,” Richard Ladner, Allen School professor emeritus, said. “Through all these years he has been a valued colleague and friend. I especially appreciated Martin at faculty meetings where he would be a stickler to make sure our decision processes were fair and unbiased. Martin is also a wonderful game player in social games like contract bridge and his favorite, Schnapsen.”
McTiernan and Tompa in Buffalo, 1976
In 1984, Tompa won an inaugural Presidential Young Investigator Award for his work in computing foundations. The next year, Tompa and his wife Anne McTiernan and two daughters left Seattle for New York, where McTiernan earned her medical degree. From 1985 to 1989, Tompa was on the staff of the IBM Research Division at the Thomas J. Watson Research Center and became manager of its Theory of Computation Group.
Four years after their departure for the east coast, McTiernan was offered a residency position at UW Medicine and they were thrilled to return to Seattle. Tompa was welcomed back to the CSE department’s Theory of Computation research group. He switched gears in 1998 to study computational molecular biology.
“I enjoyed working in both theoretical computer science and computational molecular biology,” Tompa said. “There’s something wonderful about theoretical computer science — mathematics is so clean and proving things is wonderful. Molecular biology has been really exciting over the last few decades and it was exciting to learn about it. It was great fun to work with biologists and write programs to help with their analyses.”
Tompa was interested in the structure of DNA and learning the functions of its various parts. He worked on comparative sequence analysis, comparing the DNA sequences of multiple organisms to see what they had in common and where they differ.
“It was a lot of work to switch fields but also a lot of fun ― for the first several years it felt like I was a student again,” Tompa recalled. “Larry Ruzzo was also learning about it, so we would attend classes and seminars together, ask a lot of stupid questions and learn a lot from each other. It’s one of the beauties of being in an academic job, you have the freedom to, say, give up your research area, become a student again and change your field of research. You couldn’t do that in most jobs and, in the end, the change paid off. Larry and I learned a lot in the area and made some good contributions.”
Allen School professor Larry Ruzzo joined the UW faculty in 1977, just one year before Tompa’s arrival. The two were collaborators in theoretical computer science and the first Allen School faculty members to work in computational molecular biology. Tompa said working with Ruzzo was one of the best aspects of his academic career, as they basically built their careers together.
“Martin and I basically ‘grew up’ together academically, and worked closely together on a variety of projects, including curriculum development, co-taught courses, service on each other’s student’s supervisory committees and many joint papers, Ruzzo said. “Martin is an academic gem — a delightful colleague, a brilliant researcher, caring and inspiring advisor, and a gifted classroom teacher.”
Members of the Allen School’s Computational and Synthetic Biology group collaborate with biologists on a wide range of computational problems that will ultimately enable them to better understand complex biological systems. Tompa’s research in this area has been extensive, particularly in motifs in genomic sequences, where his work helped biologists understand the mechanisms that regulate how, when, where, and at what rate genes express their products. An important aspect of this challenge is the identification of binding sites in the genome for the proteins involved in such regulation. Finding those binding sites is where sequence motifs come in.
In 2001, Tompa, who was also an adjunct professor in the Department of Genome Sciences by then, and graduate student Jeremy Buhler, now a professor at Washington University in St. Louis, wrote “Finding motifs using random projections.” The paper introduced a novel randomized algorithm called PROJECTION for the discovery of short sequence motifs such as protein binding sites. The duo’s approach remedied weaknesses observed in existing motif discovery algorithms and solved difficult motif challenge problems. Its impact was so great that it earned a Test of Time Award from Research in Computational Molecular Biology in 2013.
While Tompa published many papers in both theoretical computation and computational molecular biology, he took the greatest pleasure from his collaborations with interdisciplinary researchers.
Tompa showing Husky pride, 1995
“I was very proud of any paper that was really about biology,” he said. “Co-authoring with experimental molecular biologists allowed me to be a part of their discovery process. But I was also really proud of the papers I wrote with CSE graduate students, discovering new algorithms and solving problems in biology.”
Saurabh Sinha (Ph.D. ‘02), now a professor of computer science at the University of Illinois at Urbana-Champaign, co-authored several papers on motifs in genomic sequences as a graduate student with Tompa and has fond memories of working in his research group. Everything, from socials at Tompa’s house to the patience and time Tompa took to help Sinha edit his papers to maximize their chance of acceptance, meant a lot to him.
“The more diffuse memory I have is of how much respect he showed his students, and how much trust he had in us, and how much independence he gave us in doing our work while also keeping an eye on the progress and providing key technical advice at the right junctures,” Sinha said. “Also, it was from him that I learned the lessons of academic and research integrity, and the importance of prioritizing means over ends.”
Amol Prakash (Ph.D. ‘06) published several papers on comparative genomics with Tompa before going on to become the founder and CEO of Optys.
“Martin was the best mentor I could have imagined. It was an honor to be his student. There are so many things that I learned from him that reflect on my thinking and writing style,” Prakash said. “I chose a non-academic career, but I am sure if I had students to advise, I would have tried to emulate a lot of what I learned from his mentoring me. I wish him the best of times in the next phase.”
Tompa admits that, while researching and studying new disciplines was a favorite part of his career, it was the teaching that gave him the most joy. Six years ago he gave up half of his appointment as an experiment in retirement. In a move that is almost unheard of, he opted to give up research rather than teaching. Since then, Tompa has spent all of his university time teaching undergrads and loved every minute of it. It’s evident that his joy has made a positive impact on his students, and they made sure it was celebrated. In 1998 and 1999, Tompa earned back to back UW ACM Teaching Awards, the recipient of which is chosen each year by undergraduate students in the Allen School.
“In the days when lots of faculty teach using prepared PowerPoint slides, Martin has a unique way of engaging his classes and making sure that the material feels fresh,” Allen School professor Paul Beame said. “Instead of slides, Martin writes everything out on the board or using a data projector, developing each point in the lecture in interaction with the class as he goes along. Powerpoint slides can feel stale and rehearsed, Martin’s unique method makes the class feel dynamic. I marvel at how he can do it.”
Several of Tompa’s students were happy to share their best memories of their time learning from Tompa, and most of them talked about his patience, mentorship and his favorite game, Schnapsen.
Some of the founding members of the UW Schnapsen Club, 2015
Tompa learned Schnapsen from his father, who grew up in Vienna and taught him the popular Austrian card game when he was a child. Schnapsen is a two-person game that has some similarities with another well-known game, Bridge. While he enjoyed the game as a child, Tompa admits he hadn’t played it for quite some time until 2011, when two former graduate students, Dick Garner (Ph.D. ‘82) and Jeff Scofield (Ph.D. ‘85), came to him looking for a good card game they might implement on an iPhone in OCaml. Martin told them he had just the game. As it turned out, Schnapsen was the ideal game for the smaller screens of early iPhones. When Garner and Scofield sent him an early version of their app to test out, it beat Tompa far more often than not.
“I couldn’t fathom what it was doing, and I wanted to understand how it was beating me. So I started taking notes on each game,” Tompa recalled. “They had done a marvelous job, developing a program that allowed the computer in the phone to explore a lot of possible moves and look ahead quite far in the game.”
Schnapsen is the perfect pastime for computer scientists, and Tompa began a blog about the winning strategies he learned from their app, employing concepts such as expected value and other aspects of probability theory. He turned the blog into a book, “Winning Schnapsen,” the definitive guide to mastering the centuries-old game that enjoys a popular following in Europe.
Tompa also incorporated Schnapsen into his CSE 312 course on the foundations of computing, using it as a running example to illustrate various concepts from probability theory. This inspired a group of students who took the course to establish a UW Schnapsen Club. Varun Mahadevan (B.S. ‘17), a teaching assistant for four quarters of the course, fondly remembers the experience and the opportunity to learn and play Schnapsen.
Tompa with Mahadevan and his family on graduation day, 2017
“It was an absolute pleasure working with him,” Mahadevan said. “He had the most unique flavor for the class where he uses Schnapsen as a teaching tool for combinatorics. It was from this that we formed a Schnapsen club and we’d meet every Friday in a conference room on the 5th floor of the Allen Center and play cards and chat.”
One of those who fondly remembers these weekly meetups is Alex Tsun (B.S., ‘18), currently a graduate student at Stanford University.
“I remember when I first ‘met’ Martin during CSE 312 lecture more than five years ago in Winter 2015. He was so excited to be teaching probability and to share his love of Schnapsen with as many people as he could,” Tsun said. “He invited us, as he always does for his classes, to Schnapsen club on Fridays, at which I eventually became a regular. I’m not sure if he remembers this, but I first met him on the first Friday and challenged him to a game of Schnapsen. I believe I actually won my first game against him, probably by luck, and he promptly wiped me out in a second game, and in future weeks. I really enjoyed his class.”
Tompa and the TAs for CSE 312, 2017
According to Beame, when students talk about Tompa, Schnapsen is prominently mentioned, in addition to his sheer joy in teaching.
“In teaching CSE 312, which introduces CSE students to the many aspects of counting and probabilistic reasoning that are so essential in machine learning, modern algorithms, and other areas of computer science, Martin has found ways to integrate Schnapsen in the course material and homework assignments,” Beame said. “While the use of familiar games like poker is common in such courses, because this is such a new game to the students, they cannot rely on the familiar and have to think outside the box, developing real new understanding along the way.”
While students and colleagues alike will miss seeing Tompa regularly in the halls of the Allen School, Tompa says he’s excited for this new chapter of his life. He’s looking forward to spending more time with his three grandchildren in Seattle and continuing his research on his family’s genealogy. He’s working on a book about his parents and their families, who lived in central Europe in the 1930s. Their families had to flee Europe when the Nazis came to power and Tompa has been writing about what happened to them.
Although he will no longer be in the classroom, he will still have a direct impact in the lives of many students.
“On the whole, I would say that CSE has always valued its educational mission, but amidst the competing demands on everyone’s time for teaching, research, and service, I can’t think of any of my colleagues who has more consistently prioritized the needs of students, at all levels,” Ruzzo said. “Martin really cares, it shows, and he pushed all of us to do the same.”
Congratulations, Martin, enjoy your well-deserved retirement!
Allen School professor Byron Boots was recognized with an Early Career Award at the Conference on Robotics: Science & Systems (RSS 2020) for his work spanning machine learning, artificial intelligence and robotics. Each year, the RSS Foundation selects one or more early career researchers for the honor based on their outstanding research accomplishments and demonstrated potential to advance the field of robotics. Boots, who directs the Allen School’s Robot Learning Laboratory, focuses on the development of theory and systems that tightly integrate perception, learning and control to enable new capabilities in motion planning, high-speed navigation, robot manipulation, and more.
Among Boots’ major contributions to date was the Gaussian Process Motion Planner (GPMP), an efficient, gradient-based algorithm that represents motion planning as continuous-time trajectories. He and his colleagues designed GPMP to address limitations associated with two strategies that, at the time, represented the state of the art in motion planning. The first, sampling-based algorithms, tend to require post-processing or the addition of optimal planners that are computationally inefficient in responding to high-dimensional problems with challenging constraints; the second, trajectory optimization algorithms, require fine discretization to integrate cost information under certain constraints and are themselves costly to rerun when faced with changing conditions. GPMP overcame such computational inefficiencies while maintaining smoothness in the result. It also served as the foundation for GPMP2, an extensible algorithm that treats the problem of motion planning as one of probabilistic inference. Under this approach, GPMP2 uses factor graphs to compute a more efficient solution. Boots and his collaborators extended GPMP2 to an incremental algorithm, iGPMP2, capable of efficiently replanning trajectories as conditions change. This combined work earned the team Paper of Year from the International Journal of Robotics Research in 2018.
The following year, Boots and his colleagues presented a novel online learning-based framework for model predictive control, a powerful technique for optimizing control tasks in dynamic environments. As part of that work, they devised a new algorithm, Dynamic Mirror Descent Model Predictive Control (DMD-MPC), that can be applied to a variety of settings and cost functions. The team’s approach, which earned Best Student Paper and was a finalist for Best Systems Paper at RSS 2019, was yet another example of Boots’ keen interest in advancing robot learning in ways that combine new levels of flexibility with increased efficiency.
“Machine learning offers huge potential for robots to learn dynamically by interacting with their environments instead of requiring any new functionality to be hand-designed by engineers. But that level of flexibility and adaptability can come at a high cost,” Boots explained. “Machine learning algorithms are notoriously data-hungry as well as computationally expensive. My goal is to leverage a mix of machine learning and prior knowledge to accelerate robot learning for real-world applications while making the process more efficient and scalable.”
One of those real-world applications Boots is particularly keen to accelerate is high-speed navigation. He recently secured a grant from the United States Army Research Laboratory as part of its Scalable, Adaptive and Resilient Autonomy (SARA) program aimed at expediting research in autonomous mobility and maneuverability in complex, unknown and adversarial environments. The grant will support Boots’ work, alongside Allen School colleagues Dieter Fox and Siddhartha Srinivasa and collaborators at the Georgia Institute Technology, to develop new capabilities in perception, planning and model predictive control that will enable autonomous ground vehicles (AGVs) to operate safely and fluidly under dynamic conditions involving a variety of obstacles and terrain.
Boots joined the Allen School faculty in 2019 after five years as a professor at Georgia Tech’s School of Interactive Computing. He was no stranger to the University of Washington, having previously completed a postdoc working with Fox in the Allen School’s Robotics and State Estimation Lab after earning his Ph.D. from Carnegie Mellon University. Boots has published nearly 100 peer-reviewed papers and his work has earned recognition at many of the top conferences in the field, including RSS, the International Conference on Machine Learning (ICML), International Conference on Robotics & Automation (ICRA), International Conference on Artificial Intelligence and Statistics (AISTATS), Conference on Neural Information Processing Systems (NeurIPS).
“It was already clear during his postdoc at UW that Byron would become a trailblazer in robotics and machine learning,” said Fox. “The RSS Early Career Award is only given to a very small group of the most innovative and influential researchers in robotics, and I can’t think of anybody more deserving of this honor than Byron.”
Boots and his fellow Early Career Award recipients — Luca Carlone of the Massachusetts Institute of Technology and Jeannette Bohg of Stanford University — were formally honored during the RSS 2020 conference held online this week.
Professor Sham Kakade, a member of the Allen School’s Machine Learning and Theory of Computation groups, received a Test of Time Award at the International Conference on Machine Learning (ICML 2020) for his work on “Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design.” In the winning paper, which was originally presented at ICML 2010, Kakade and his colleagues established the first sublinear regret bounds for Gaussian process (GP) optimization in the nonparametric setting. The team’s work was lauded by the machine learning community for its technical depth and for its enduring impact on both theory and practice.
Kakade, who holds a joint faculty appointment in the Allen School and the University of Washington’s Department of Statistics and is also a senior data science fellow at the eScience Institute, co-authored the paper while a professor at the University of Pennsylvania. His collaborators on the project included Niranjan Srinivas, a Ph.D. student at the California Institute of Technology; Andreas Krause, a professor at CalTech at the time; and Matthias Seeger, then a faculty member at the Universität des Saarlandes in Germany. Together, the team set out to address an open question in machine learning: how to optimize an unknown, noisy function that is expensive to evaluate while minimizing sampling.
“We were interested in finding a principled framework for addressing the problem of Bayesian optimization, and we realized that one way to formalize this was through the theory of the sequential design of experiments,” explained Kakade. “One that I was particular excited about was how we could provide a sharp characterization of the learning complexity through a novel concept we introduced, the ‘information gain.’”
The question has numerous applications in both laboratory and real-world settings, from determining the optimal control strategies for robots, to managing transportation and environmental systems, to choosing which advertisements to display in a sponsored web search. To answer the challenge, Kakade and his colleagues united the fields of Bayesian optimization, bandits and experimental design. The team analyzed GP optimization as a multi-armed bandit problem to offer up a novel approach for deriving cumulative regret bounds in terms of maximal information gain. In the process, they succeeded in establishing a novel connection between GP optimization and experimental design.
By applying a simple Bayesian optimization method known as the Gaussian Process Upper Confidence Bound (GP-UCB) algorithm, the team demonstrated that they could obtain explicit sublinear regret bounds for a number of commonly used covariance functions. In experiments using real-world network sensor data, Kakade and his collaborators showed that their approach performed as well or better than existing algorithms for GP optimization which are not equipped with regret bounds. In the decade after the researchers unveiled their results, Bayesian optimization has become a powerful tool in machine learning applications spanning experimental design, hyperparameter tuning, and more. The method, proof techniques, and practical results put forward by Kakade and his colleagues have been credited with sparking new research directions and subsequently enriching the field of machine learning in a variety of ways.
Since the paper’s initial publication, Srinivas joined 10x Genomics as a computational biologist after completing a postdoc at the University of California, Berkeley, while Krause moved from CalTech to the faculty of ETH Zürich in Switzerland. Seeger is now a principal machine learning scientist at Amazon. The team is being formally recognized during the ICML 2020 conference taking place virtually this week.
Jeong Joon (JJ) Park, a Ph.D. student working with Allen School professor Steve Seitz in the Graphics and Imaging Laboratory (GRAIL) has been named a 2020 Apple Ph.D. Scholar. Park was recognized in the “Artificial Intelligence/Machine Learning” category for his focus on 3D reconstruction and understanding. Apple selects its scholars based on their innovative research, record as thought leaders and collaborators in their fields, and unique commitment to take risks and push the envelope in machine learning and AI.
Park’s research focuses on recovering the underlying 3D properties of environments from photos and depth sensing. His work spans computer vision, deep learning, and augmented and virtual reality (AR/VR).
“3D computer vision can be very useful in many exciting areas of 3D applications, including VR/AR — visiting new places virtually, or robotics — using them to navigate and pick up stuff,” Song said. “I work on visually realistic reconstructions and study what is the most suitable representation for 3D geometry and appearance.”
Park enjoys the work because it is an intersection of various fundamental areas that interest him, including machine learning, physics, computer vision and graphics. In his most recent paper, “Seeing the World in a Bag of Chips,” he and co-authors Alexsander Holynski and Seitz address problems of novel view synthesis and environment reconstruction from hand-held RGBD sensors. The team developed a method for modeling highly specular objects, inter-reflections and fresnel effects to enable surface light field reconstruction of the environment with the same input used to reconstruct the shape alone. The results can be seen in this video, where Park films a bag of chips and then reconstructs a high-resolution image of the surrounding room, including lights, furniture, windows — even trees and people seen through the windows. The team presented its work at the Computer Vision and Pattern Recognition 2020 Computer Vision and Pattern Recognition (CPVR 2020) conference in June, and the project was also featured in Scientific American and WIRED. This helps to create more realistic 3D views in VR and AR.
In another paper, “Deepsdf: Learning continuous signed distance functions for shape representation,” Park introduces a new way of representing shape using neural networks. The new representation enables computers to more effectively learn 3D shapes prior from large datasets, enabling applications in vision, graphics and robotics. That work was among the candidates for Best Paper at CVPR 2019. This technique can be used for applications in AR/VR for indoor scene geometry reconstruction, inferring 3D shape of humans from an image, robotics for object grasping, or autonomous-driving for predicting shape of cars around the driver. Previously, in his paper “Surface light field fusion” published at the 2018 International Conference on 3D Vision, Park shows how to scan highly reflective objects with an RBGD sensor, using infrared dot patterns themselves to recover specular coefficients.
“While most of my students look to me to choose research directions in the beginning, JJ was driving his own research program from the start,” said Seitz. ”He’s passionate about scene modeling and rendering and wants to solvethis problem for real. Large scale, general objects, and real systems that work for real users. And it’s been thrilling to watch him make major inroads on solving this problem with a series of strong technical papers.”
Park is looking forward to the opportunities the program will afford him.
“I’m very happy to be supported by Apple,” he said. “The company is already taking initiatives in augmented reality and other research areas and I expect to see synergy between our research.”
Congratulations, JJ — and thanks to Apple for generously supporting student research!
