Heer, who holds the Jerre D. Noe Endowed Professorship at the Allen School, co-authored the paper while a professor at Stanford University. The project was led by first author Mike Bostock, Heer’s former Ph.D. student, who spent the last 10 years continuing to develop D3, author hundreds of examples, and support the D3 user community. Co-author and graduate student Vadim Ogievetsky helped to develop and evaluate the system. The result was a novel approach to create web-based visualizations by flexibly binding data directly to document elements.
“I’m so excited to see D3 honored at IEEE VIS,” said Heer. “I don’t see this as just an award for 10-year-old work. I see it as a recognition of 10 years of incredible effort by Mike and others, alongside an outpouring of teaching and sharing by visualization creators and researchers. It has been a thrill to help web-based visualization come of age.”
D3, a de facto standard for interactive visualizations on the web, is an approach that helps turn data into powerful visualizations using HTML, SVG and CSS. In the paper, the team shows how representational transparency improves expressiveness and better integrates with developer tools. D3 also simplifies debugging and allows iterative development. Designers using the framework input data into their document to generate and modify visual content, providing a compelling way for web developers to enable interactive visualizations. As a free and open-source tool, D3 changed the field of data visualization research and practical uptake of interactive visualization on the web.
D3 sits atop years of visualization systems research, building on ideas from Bostock and Heer’s earlier Protovis toolkit, as well as Heer’s prior Prefuse and Flare frameworks. This line of work has continued at the Allen School, resulting in visualization languages such as Vega and Vega-Lite.
Since the paper’s initial publication, Heer joined the Allen School faculty and has been recognized with the Association for Computing Machinery’s ACM Grace Murray Hopper Award, the IEEE Visualization Technical Achievement Award, Best Paper Awards at the ACM Conference on Human Factors in Computing (CHI), EuroVis and IEEE InfoVis conferences, and another IEEE InfoVis 10-Year Test-of-Time Award. After Ogievetsky earned his Master’s from Stanford, he went on to co-found Imply, a multi-cloud data platform, while Bostock worked at the New York Times to create data-rich stories using D3 before going on to form Observable, a provider of web-based computational notebooks for data analysis, visualization and more.
Allen School professor Maya Cakmak, who directs the Human-Centered Robotics Lab, was recently named a 2021 Trailblazer by the University of Washington’s Disabilities, Internetworking, and Technology Center (DO-IT). The Trailblazer awards honor members of the DO-IT community who forge new pathways to increase the potential for people with disabilities to succeed in college, careers and life. Cakmak was honored for her continued work with DO-IT’s AccessEngineering, AccessComputing and Summer Study Program. The initiatives introduce students with disabilities to engineering and computer science and make these disciplines more accessible.
Cakmak’s research focuses on creating robots that can be programmed and controlled by a diverse group of users with unique needs and preferences to do useful tasks. By taking a human-centered approach to build robots that assist people in everyday tasks with state-of-the-art robotic functionalities accessible to users with no technical background, Cakmak’s goal is to make personal robotic assistants in the home a reality to improve the quality of people’s lives, including giving independence to people with motor limitations and enabling older adults to age in place.
“I was introduced to accessibility research by emeritus professor Richard Ladner when I first joined the faculty in 2013,” said Cakmak. “Richard made me realize the potential role that robots could play in addressing inequities that people with mobility and dexterity limitations face in accessing the physical world.”
In 2014, Cakmak was introduced to DO-IT director Sheryl Burghstahler and started working with her as a co-PIs on the AccessEngineering project. Ladner encouraged Cakmak to propose a DO-IT robot programming workshop. Both of those experiences taught her a lot about working with people with disabilities and shaped her research interests for years to come. Her continued commitment to DO-IT programs has given Cakmak the opportunity to serve a diverse student body while integrating relevant disability-related and universal design content into engineering courses. In the robot programming workshop, she taught high school students with disabilities to program different types of robots. For instance, in one workshop students with little robotic experience programmed interactive social robots to help people deal with stress and regulate emotions. In another, students collaborated to program a mobile manipulator robot to be a grocery store attendant.
“We generally think in terms of the 99 percent of humans, and human-centered design targets 100 percent of them,” said Dillen, a student in the workshop.
Cakmak also participated in the AccessComputing OurCS workshop series, which involved university women with disabilities in a two-day program where they created a robot that supports mental health. By brainstorming and bodystorming, the group explored user and robotic interaction to reach their goal. The students identified a need to journal to improve mental health and built journaling robots with friendly faces that were tuned into listening to a person and asking them questions based on the techniques and tools the students found to be the most important to relieve stress.
“I continue to work with DO-IT, as well as other great initiatives on campus like the Taskar Center and CREATE, because they are the subject domain experts and bring so much to the table. I continue to learn from them in every project,” said Cakmak. “Like our rockstar Allen School grad student and 2020 Trailblazer Award recipient Ather Sharifrecently wrote, we ought to include people with disabilities in every part of accessibility research.”
Mones, who joined the Allen School in 1999, has built a career working in and teaching computer graphics and animation production. Her research is in animation, visual storytelling, content development, fast prototyping, facial expression for stylized characters and the animation production pipeline design for games, film and immersive environments. As director of the Reality Studio, Mones teaches students about effective production pipelines and clear storytelling for and in VR, AR and MR. Students develop and create their own animation and immersive projects under her guidance.
“Since joining the Allen School, Barbara grew a single course in digital animation into a suite of courses extending from traditional to immersive — virtual reality — digital animation,” said Ed Lazowska, Professor and Bill & Melinda Gates Chair Emeritus in the Allen School. “Year after year, the animated shorts that her students create are invited to prestigious animation festivals in the U.S. and beyond. Year after year, her graduates take positions at leading animation houses. Year after year, we receive messages from former students describing how their careers were shaped by the experiences they had with Barbara at University of Washington.”
The curricula Mones has developed in computer animation has been widely recognized and influential — she has lectured at institutions globally on animation and curriculum development. She also coordinated an international student animation competition for the ACM SIGGRAPH for 17 years and served as Art Chair for the organization’s Education Committee.
“Barbara gave me my first opportunity to learn the skills for animation and visual effects. She introduced me to my grad program and set me on the path to working in the film industry,” said Elizabeth Muhm (B.S., Computer Science and Mathematics, ‘09), a former teaching assistant for Mones who is currently a software engineer at Google. “She both shared her passion for the craft and taught practical skills I use daily like how to manage up and how to think of all your work as a draft to iterate on.”
Students who study digital animation at the UW have the opportunity to put what they learned into practice in the Allen School’s Animation Capstone, in which they collaborate on the production of an animated short film following an industry-standard production pipeline that spans modeling, shading, lighting, animating, rendering and post-production. With her capstone students, Mones has produced and directed 20 animated shorts, many of which were screened at domestic and international film festivals. Along the way, she has developed a curriculum that now incorporates AR, VR and MR into storytelling, content development and filmmaking.
Furthermore, her graphics and animation have been shown in museums and institutions worldwide, including the Smithsonian Institution and the Villa Ciani Museum in Switzerland and the ACM SIGGRAPH Electronic Theater. She also has designed and implemented training programs in the areas of digital modeling, animation and 3D paint at Dreamworks/Pacific Data Images and Industrial Light & Magic.
Watch Mones talk about the award and speak more about her work here. All of her screenings and awards can be seen on the Animation Research Labs website.
Allen School Ph.D. students Saadia Gabriel and Dhruv (DJ) Jain each won a dissertation Fellowship from Google Research and the CMD-IT Diversifying LEAdership in the Professoriate (LEAP) Alliance. In an effort to make computer science research careers more accessible, Google Research partnered with the LEAP Alliance, which is operated by the national Center for Minorities and People with Disabilities in Information Technology to increase the diversity of Ph.D. graduates in computing. Together, the organizations provided a total of six dissertation awards this year to support doctoral students from historically underrepresented groups as they complete their Ph.D. requirements.
Gabriel, advised by Allen School professor Yejin Choi, researches natural language generation and social commonsense reasoning. Gabriel has previously worked on evaluating factuality in generation, as well as improving fairness and explainability in toxic language detection. In her most recent work, she investigates how people might react to Covid-19 and climate misinformation online. She aims to find how well machine learning models interpret and understand reactions and emotions of people in everyday situations and whether or not these models are capable of recognizing text that is factually consistent with prior context. She also seeks to determine if machine learning algorithms are designed with accessibility and interpretability in mind.
Gabriel will design algorithms for machine learning approaches to find implications captured by written language then develop frameworks that can understand headlines that are harmless versus headlines that have malicious intentions. Ultimately she plans to develop a system prototype and mobile application for artificial intelligence-augmented news reading, using resources she develops for generative neural models to be trained with misinformation detection formalisms.
Jain, who is co-advised by Allen School professor Jon Froehlich and Human Centered Design & Engineering professor and Allen School adjunct professor Leah Findlater, works in the Makeability Lab to advance sound accessibility by designing, building and deploying systems that leverage human computer interaction (HCI) and artificial intelligence (AI). His primary aim is to help people who are d/Deaf and hard of hearing (DHH) to receive important and customized sound feedback.
Jain created HomeSound, a smart home system that senses and alerts users to sound activity like a beeping microwave, blaring smoke alarm or barking dog. To increase the portability of HomeSound, Jain created SoundWatch, an app that provides always-available sound feedback on smartwatches. When the app picks up a nearby sound like a car honking, a bird chirping or someone hammering, it sends the user a notification along with information about the sound. The next phase of his research will be devoted to building on this work, which was well-received by users, to enable feedback to be customized to individual needs, such as the calls of each of their children or the beep of a new home appliance. For example, Jain is currently working on ProtoSound, a sound recognition system that can be personalized by end-users by inputting a few labelled examples of each sound.
Jain has earned two Best Paper Awards, four Best Paper Award Honorable Mentions and one Best Artifact Award at top conferences in the field of HCI. In addition to the Google Research/CMD-IT LEAP Alliance grant, he previously received a Microsoft Research Dissertation Grant to support his work.
The National Science Foundation (NSF) has selected Allen School professors Amy Zhang, who directs the Social Futures Lab, and Franziska Roesner, who co-directs the Security and Privacy Research Lab, to receive Convergence Accelerator funding for their work with collaborators at the University of Washington and the grassroots journalism organization Hacks/Hackers on tools to detect and help stop misinformation online. The NSF’s Convergence Accelerator program is unique in that its structure offers researchers the opportunity to accelerate their work over the course of a year to find tangible solutions. The curriculum is designed to strengthen each team’s convergence approach and further develop their solution to move on to a second phase with the potential for additional funding.
In their proposal, “Analysis and Response for Trust Tool (ARTT): Expert-Informed resources for Individuals and Online Communities to Address Vaccine Hesitancy and Misinformation,” Zhang, Roesner, Human Centered Design & Engineering professor and Allen School adjunct professor Kate Starbird, Information School professor and director of the Center for an Informed Public Jevin West, and internet and Hacks/Hackers researcher at large Connie Moon Sehat, who serves as principal investigator on the project, aim to develop a software tool — ARTT — that helps people identify and prevent misinformation. This currently happens on a smaller scale by individuals and community moderators with few resources or expert guidance on combating false information. The team, made up of experts in fields such as computer science, social science, media literacy, conflict resolution and psychology, will develop a software program that helps moderators analyze information online and present practical information that builds trust.
“In our previousresearch, we learned that rather than platform interventions like ‘fake news’ labels, people often learn that something they see or post on social media is false or untrustworthy from comment threads or other community members,” said Roesner, who serves as co-principal investigator on the ARTT project alongside Zhang. “With the ARTT research, we are hoping to support these kinds of interactions in productive and respectful ways.”
While ARTT will help prevent the spread of any misinformation, the team’s focus right now is on combating false information on vaccines — vaccine hesitancy has been identified by the World Health Organization as one of the top 10 threats to global health.
In addition to her participation in the ARTT enterprise, Zhang has another Convergence Accelerator project focused on creating a “golden set” of guidelines to help prevent the spread of false information. That proposal, “Misinformation Judgments with Public Legitimacy,” aims to use public juries to render judgments on socially contested issues. The jurors will continue to build these choices to create a “golden set” that social media platforms can use to evaluate information posted on social media. Besides Zhang, the project team includes the University of Michigan’s Paul Resnick, associate dean for research and faculty affairs and professor at the School of Information, and David Jurgens, professor at the Information School and in the Department of Electrical Engineering & Computer Science, and the Massachusetts Institute of Technology’s David Rand, professor of management science and brain and cognitive sciences and Adam Berinsky, professor of political science.
Online platforms have been increasingly called on to reduce the spread of false information. There is little agreement on what process should be used to do so, and many social media sites are not fully transparent about their policies and procedures when it comes to combating misinformation. Zhang’s group will develop a forecasting service to be used as external auditing for platforms to reduce false claims online. The “golden sets” created from the jury’s work will serve as training data to improve the forecasting service over time. Platforms that use this service will also be more transparent about their judgments regarding false information posted on their platform.
“The goal of this project is to determine a process for collecting judgments on content moderation cases related to misinformation that has broad public legitimacy,” Zhang said. “Once we’ve established such a process, we aim to implement it and gather judgments for a large set of cases. These judgments can be used to train automated approaches that can be used to audit the performance of platforms.”
Participation in the Convergence Accelerator program includes a $749,000 award for each team to develop their work. Learn more about the latest round awards here and read about all of the UW teams that earned a Convergence Accelerator award here.
As anyone who has visited a website knows, online ads are taking up an increasing amount of page real estate. Depending on the ad, the content might veer from mildly annoying to downright dangerous; sometimes, it can be difficult to distinguish between ads that are deceptive or manipulative by design and legitimate content on a site. Now, Allen School professor Franziska Roesner (Ph.D., ‘14), co-director of the University of Washington’s Security and Privacy Research Lab, wants to shed light on problematic content in the online advertising ecosystem to support public-interest transparency and research.
Consumer Reports selected Roesner as a 2021-2022 Digital Lab Fellow to advance her efforts to create a public-interest online ads archive to document and investigate problematic ads and their impacts on users. With this infrastructure in place, Roesner hopes to support her team and others in developing new user-facing tools to combat the spread of misleading and potentially harmful ad content online. She is one of three public interest technology researchers to be named in the latest cohort of Digital Lab Fellows focused on developing practical solutions for addressing emerging consumer harms in the digital realm.
This is not a new area of inquiry for Roesner, who has previously investigated online advertising from the perspective of user privacy such as the use of third-party trackers to collect information from users across multiple websites. Lately, she has expanded her focus to examining the actual content of those ads. Last year, amidst the lead-up to the U.S. presidential election and the pandemic’s growing human and economic toll — and against the backdrop of simmering arguments over the origins of SARS-CoV-2, lockdowns and mask mandates, and potential medical interventions — Roesner and a team of researchers unveiled the findings of a study examining the quality, or lack thereof, of ads that appear on news and media sites. They found that problematic online ads take many forms, and that they appeared equally on both trusted mainstream news sites and low quality sites devoted to peddling misinformation. In follow-up work, Roesner and her collaborators further studied people’s — not just researchers’ — perceptions of problematic ad content, and in forthcoming work, problematic political ads surrounding the 2020 U.S. elections.
“Right now, the web is the wild west of advertising. There is a lot of content that is misleading and potentially harmful, and it can be really difficult for users to tell the difference,” explained Roesner. “For example, ads may take the form of product ‘advertorials,’ in which their similarity to actual news articles lends them an appearance of legitimacy and objectivity. Or they might rely on manipulative or click-baity headlines that contain or imply disinformation. Sometimes, they are disguised as political opinion polls with provocative statements that, when you click on them, ask for your email address and sign you up for a mailing list that delivers you even more manipulative content.”
Roesner is keen to build on her previous work to improve our understanding of how these tactics enable problematic ads to proliferate — and the human toll that they generate in terms of the time and attention wasted and the emotional impact of consuming misinformation. Building out the team’s existing ad collection infrastructure, the ad archive will provide a structured, longitudinal, and (crucially) public look into the ads that people see on the web. These insights will support additional research from Roesner’s team as well as other researchers investigating how misinformation spreads online. Roesner and her collaborators ultimately aim to help “draw the line” between legitimate online advertising content and practices, and problematic content that is harmful to users, content creators, websites, and ad platforms.
But Roesner doesn’t think we should wait for the regulatory framework to catch up. One of her priorities is to protect users from problematic ads, such as by developing tools that automatically block certain ads or empower users to recognize and flag them. While acknowledging that online advertising is here to stay — it funds the economic model of the web, after all — Roesner believes that there is a better balance to be struck between revenue and the quality of content that people consume on a daily basis as they point and click.
“Even the most respected websites may be inadvertently hosting and assisting the spread of bogus content — which, as things stand, puts the onus on users to assess the veracity of what they are seeing,” said Roesner. “My hope is that this collaboration with Consumer Reports will support efforts to analyze ad content and its impact on users — and generate regulatory and technical solutions that will lead to more positive digital experiences for everyone.”
Consumer Reports created the Digital Lab Fellowship program with support from the Alfred P. Sloan Foundation and welcomed its first cohort last year.
“People should feel safe with the products and services that fill our lives and homes. That depends on dedicated public interest technologists keeping up with the pace of innovation to effectively monitor the digital marketplace,” Ben Moskowitz, director of the Digital Lab at Consumer Reports, said in a press release. “We are proud to support and work alongside these three Fellows, whose work will increase fairness and trust in the products and services we use everyday.”
Allen School professor Shyam Gollakota has received a 2021 Moore Inventor Fellowship in recognition of his work at the nexus of low-power wireless communication, biology and living organisms. Gollakota, who directs the Allen School’s Networks & Mobile Systems Lab, is the first University of Washington faculty member to receive this prestigious award that nurtures the next generation of scientist-inventors. The Gordon and Betty Moore Foundation established the fellowship program aimed at supporting “50 inventors to shape the next 50 years” in 2016 to mark the 50th anniversary of Moore’s Law positing the exponential growth in computer processing power.
Power figures prominently in Gollakota’s work — but in his case, the focus has been on finding ways to wirelessly fuel computation in order to cut the cord and lighten the load. Much like the results of the eponymous law articulated by Gordon Moore, Gollakota’s research has led to expanded computational capabilities accompanied by a shrinking form factor.
His first foray into wireless computing resulted in a breakthrough known as ambient backscatter. Together with his UW colleague Joshua Smith, who holds a joint appointment in the Allen School and the Department of Electrical & Computer Engineering, Gollakota developed a battery-free system that used television, WiFi and other wireless signals as both a power source and a mode of communication. In a series of subsequent projects, Gollakota and his collaborators expanded these capabilities to cover greater distances and bestow the capability to perform wireless computation on a greater variety of objects.
After looking skyward to enable devices to pull power out of thin air, Gollakota cast his eyes in the opposite direction as he contemplated how to make the most of these new capabilities.
“Outside there’s a whole world on every square foot, with living beings that you don’t even think about. We just walk over it,” Gollakota said in a UW News story. “But there’s so much happening — feats of engineering. There’s so much beauty in these tiny things.”
Gollakota and his colleagues drew inspiration from those “tiny things” to engineer a new line of research he has dubbed the Internet of Biological and Bio-Inspired Things. The concept began to take off with the development of a lightweight wireless sensor backpack small enough to be carried by bumblebees. The onboard sensors gather data about the surrounding environment as the bees go about their daily business; upon their return to the hive each evening, the data they logged is uploaded using backscatter while the tiny battery is wirelessly recharged for the next day’s flight. Gollakota and his collaborators followed up that buzz-worthy project with a wireless sensing package that could be safely air-dropped from great heights by live moths or drones into remote or impassable areas, followed by a miniature remote-control camera that can ride on the back of a beetle. Looking to the future, Gollakota is keen to explore ways to more deeply integrate biology and technology to achieve his vision.
Credit: Mark Stone/University of Washington
“Simply put, Shyam is amazing — he is easily the most creative person I have ever met,” his Allen School colleague Thomas Anderson observed earlier this year. “He repeatedly invents and builds prototypes that, before you see them demonstrated, you would have thought impossible.”
While Gollakota’s notion of an Internet of Biological and Bio-Inspired Things may at first seem to belong in the realm of science fiction, it has many practical applications, from wildlife conservation, to smart agriculture, to large-scale environmental monitoring. In parallel with this work, Gollakota has also collaborated with colleagues and clinicians on a series of mobile sensing projects to support contactless disease detection and health monitoring using smartphones and smart speakers.
Gollakota is one of five innovators to be named in the 2021 cohort of Moore Inventor Fellows. He and his fellow honorees were selected from nearly 200 nominations received by the Foundation and will each receive $825,000 to further their inventions. Gollakota, who holds the Torode Family Career Development Professorship in the Allen School, previously earned the Association for Computing Machinery’s ACM Grace Murray Hopper Award in recognition of his early-career technical contributions, MIT Technology Review’s TR35 Award recognizing the world’s top innovators under the age of 35, and a Sloan Research Fellowship — among many other honors since his arrival at the UW in 2012.
University of Washington professor Shwetak Patel has earned a place in the Georgia Tech College of Computing’s Hall of Fame and a spot on Business Insider’s recent list of “30 leaders under 40” who are changing health care for his innovative work combining low-power sensing, signal processing and machine learning for applications ranging from non-invasive disease screening to monitoring appliance-level energy consumption. Patel, who holds the Washington Research Foundation Entrepreneurship Endowed Professorship in the Allen School and the UW Department of Electrical & Computer Engineering, is also a serial entrepreneur and director of health technologies at Google Health and Fitbit Research.
“Ten years ago, computing’s role in health care was basically billing, data collection and databases,” Patel observed to Business Insider. “Now computing is playing a critical role in the actual discovery of new interventions in health outcomes.”
Patel himself has been largely responsible for transforming computing’s contribution from staid billing software to pocket-sized personal health monitor. Evidence of that work is strewn around the Allen School’s Ubiquitous Computing Lab on the UW’s Seattle campus, where stacks of mobile phones and coils of charger cable jostle for space with a variety of 3D-printed accessories, camera color correction cards, the odd blood pressure monitor, and even a life-sized plastic baby doll. (That last one is used for demonstrating an app for detecting infant jaundice.)
Patel’s drive to “democratize diagnostics,” as he once described it, stemmed from his realization that the proliferation of smartphones and their increasingly sophisticated sensing capabilities had the potential to improve health care outcomes for millions of people around the globe. He and his students began thinking about how they could employ these on-board sensors — such as the phone’s camera, microphone, accelerometer, and gyroscope — to augment traditional in-person care by enabling early detection and intervention. They also saw an opportunity to empower people to monitor their health on an ongoing basis, without the need for repeated trips to a clinic or access to specialized equipment, with the help of a device they already carry around with them.
Working with clinicians at UW Medicine, Seattle Children’s and others, Patel and his team developed apps for assessing lung function in people with respiratory illnesses, detecting jaundice in babies and adults, measuring blood hemoglobin in people with anemia — to name only a few. Patel and his collaborators started a company, Senosis Health, to commercialize their research. After Senosis was acquired by Google, Patel began splitting his time between the UW and the company in order to lead the latter’s mobile health efforts. Patel and the Google Fit team have since released tools for measuring heart rate and respiratory rate to permit users to monitor their general health and wellness with the aid of their smartphone camera that was based in part on research originating in his UW lab.
Patel’s efforts to advance mobile health sensing were a natural progression from his visionary work on low-power sensing that stretches back to his student days at Georgia Tech. His first foray into the technology was as an undergraduate working on the Aware Home, a demonstration project that sought to imagine the connected home of the future. After earning his bachelor’s, Patel remained in Atlanta to pursue his doctorate, during which time he developed a system for measuring residential energy and water consumption by individual appliances and fixtures from a single point in the home — research that Patel continued to refine and expand upon following his arrival at the UW. He and his Georgia Tech collaborators started a company, Zensi, to commercialize that work which was subsequently acquired by Belkin.
Next, Patel and his students zoomed out from looking at individual appliances to monitoring the entire home via an ultra-low-power sensing system known as SNUPI, short for Sensor Nodes Utilizing Powerline Infrastructure. SNUPI consisted of a network of low-power sensors that transmitted data about a building’s condition — for example, increased moisture level in the walls — via the structure’s electrical circuit. The system was designed to function for decades without having to replace the batteries. Patel and his team created another spinout company, SNUPI Technologies, to commercialize a residential whole-home hazards monitoring platform under the name of WallyHome that was later acquired by Sears.
Through the years, Patel and his students have also advanced innovations in motion tracking, object detection, wearable technologies, hyperspectral imaging, and more. Throughout his career, he has earned more than two dozen Best Paper Awards and multiple “test of time” awards at the field’s preeminent conferences focused on ubiquitous computing, mobile computing, pervasive computing and human-computer interaction.
Patel’s induction into his alma mater’s Hall of Fame and the Business Insider recognition are the latest in a string of accolades recognizing the wide-ranging impact of his work. Previously, he was named a Fellow of the Association for Computing Machinery and received the organization’s ACM Prize in Computing for mid-career contributions to the field. Patel is also a past recipient of a MacArthur Foundation “Genius” Award and a Presidential Early Career Award for Scientists and Engineers (PECASE).
UW and UCSD Golden Goose Award recipients (clockwise from top left): Stephen Checkoway, Karl Koscher, Stefan Savage and Tadayoshi Kohno
In 2010 and 2011, a team of researchers led by Allen School professor Tadayoshi Kohno and Allen School alumnus and University of California San Diego professor Stefan Savage (Ph.D., ‘02) published a pair of papers detailing how they were able to hack into a couple of Chevrolet Impalas and take command of a range of functions, from operating the windshield wipers to applying — or even denying — the brakes. A decade later, Kohno, Savage, and University of Washington alumni Karl Koscher (Ph.D., ‘14), now a research scientist in the Allen School’s Security & Privacy Research Lab, and Stephen Checkoway (B.S., ‘05), a UCSD Ph.D. student at the time who is now a faculty member at Oberlin College, have received the Golden Goose Award from the American Association for the Advancement of Science for demonstrating “how scientific advances resulting from foundational research can help respond to national and global challenges, often in unforeseen ways.”
“More than 10 years ago, we saw that devices in our world were becoming incredibly computerized, and we wanted to understand what the risks might be if they continued to evolve without thought toward security and privacy,” explained Kohno in a UW News release.
Achieving that understanding would go on to have significant real-world impact, influencing “how products are built and how policies are written,” noted Savage. It would also transform not just the automobile manufacturing landscape, but the computer security research landscape as well.
“The entire automotive security industry grew from this effort,” recalled Kohno. “And I imagine that neighboring industries saw what happened here and didn’t want something similar happening to them.”
“What happened here” was that Kohno and his colleagues demonstrated how a motor vehicle’s computerized systems could be vulnerable to attackers, theoretically endangering the car’s occupants and those who share the road with them. The quartet was aided and abetted by collaborators that included, on the Allen School side, then-student and current professor Franziska Roesner (Ph.D., ‘14), fellow student Alexei Czeskis (Ph.D., ‘13), and professor Shwetak Patel; on the UCSD side, they were joined by postdoc Damon McCoy, master’s student Danny Anderson, professor Hovav Shacham, and the late researcher Brian Kantor.
This “dream team,” as Kohno describes it, set out to reverse-engineer the various vehicle components. The goal was to figure out how they communicated with each other so that they could use that to gain access to the systems that control the vehicle’s functions. The researchers published two papers in rapid succession detailing their findings; the first established how a car’s internal systems were vulnerable to compromise, while the follow-up explored the external attack surface of the vehicle by demonstrating how an attacker could infiltrate and control those systems remotely. The team presented the former at the 2010 IEEE Symposium on Security & Privacy and the latter at the 2011 USENIX Security Symposium.
In a way, Savage recalled, the researchers’ ignorance about how the vehicle’s systems were actually designed to work ended up working to the team’s advantage; it enabled them to approach their task without any preconceptions of what should happen. An example is the brake controller, which they broke into via a technique known as black-box testing or “fuzzing.” As the label suggests, these efforts involved less precision and more “throwing stuff at it,” according to Savage, to see what would stick. The results were enough to stop anyone in their tracks — including the technical experts at GM.
“We figured out ways to put the brake controller into this test mode,” Koscher explained to UW News. “And in the test mode, we found we could either leak the brake system pressure to prevent the brakes from working or keep the system fully pressurized so that it slams on the brakes.”
As the senior Ph.D.s on the project, Koscher and Checkoway spearheaded that discovery, which involved calling into the car’s OnStar unit and instructing it to download and install remote command and control software that they had written. With that in place, they were able to compel the system to download the software that would enable them to remotely control the brakes from a laptop — as demonstrated later in a famous “60 Minutes” segment in which the team surprised correspondent Leslie Stahl by bringing the car to a complete stop while she was behind the wheel.
While that made for good television, what is most gratifying for the researchers are the industry and regulatory frameworks that grew out of their discovery.
For example, GM — along with other manufacturers — hired an entire security team as a direct result of the UW and UCSD research; likewise, the National Highway Traffic Safety Administration (NHTSA) — which previously had no one on staff with computer security expertise and “were very unsure what to do with us,” according to Savage — wound up creating an entire unit devoted to cybersecurity, complete with its own testing lab. In other positive changes, the Society of Automotive Engineers — later renamed SAE International — established a set of security standards that all automobile manufacturers adhere to, and the industry created the Auto-ISAC, a national Information Sharing and Analysis Center, to enhance vehicle cybersecurity and address emerging threats.
The team’s work also paved the way for new research outside of the automotive industry. For example, its results inspired the U.S. Defense Advanced Research Projects Agency (DARPA) to create its HACMS project, short for High-Assurance Cyber Military Systems, to examine the security of cyber-physical systems. And that was just the start.
“My gut tells me that the attention directed at this project helped to build up expertise in this embedded systems realm,” observed Koscher. “What was initially focused on automotive security was then applied to other industries, such as medical devices.”
The project also served to highlight the advantages of working as part of a larger team, as Checkoway discovered to his delight. While various members of the group may have approached a problem from different angles, they would often meet in the middle to come up with a solution.
“This was an extremely collaborative effort,” Checkoway explained last year. “No task was performed by an individual researcher alone. I believe our close collaboration was the key to our success.”
At the time the researchers quietly revealed their results to GM, they couldn’t be sure such a happy outcome was a foregone conclusion. At first, the company representatives didn’t believe they could do some of the things they had done — or how they could have possibly done them. But the team’s non-adversarial approach, in which they opted to walk company representatives through their process and findings while refraining from naming the manufacturer publicly, went a long way toward steering the conversations in a positive direction.
“As academics, we have the opportunity to approach the dialogue around vulnerabilities without really having a stake in the game,” explained Kohno. “We’re not selling vulnerabilities, we’re not selling a product to patch vulnerabilities, and we aren’t a competing manufacturer. So we discovered something, and once we had the results, we wanted to figure out, how can we use this knowledge to make the world a better place?”
The team is quick to credit the federal government for driving investment in a project for which they didn’t have a precise destination in mind when they started. According to Savage, the National Science Foundation’s willingness to back a project that was not guaranteed to pan out was key to enabling them to identify these latent security risks. “We’re extremely grateful to NSF for having flexibility to fund this work that was so speculative and off the beaten path,” Savage said.
Checkoway (left) and Koscher reunite with Emma the Impala in the UW’s Central Garage Mark Stone/University of Washington
It is just the kind of work that the Golden Goose Award was created to recognize. In answer to the late U.S. Senator William Proxmire’s “Golden Fleece Award” ridiculing federal investment that he deemed wasteful, U.S. Representative Jim Cooper conceived of the Golden Goose Award to honor “the tremendous human and economic benefits of federally funded research by highlighting examples of seemingly obscure studies that have led to major breakthroughs and resulted in significant societal impact.”
For Kohno, that impact and this most recent recognition — the team previously earned a Test of Time Award from the IEEE Computer Society Technical Committee on Security and Privacy — are motivation enough to explore where the next security risk may come from.
“The question that I have now is, as security researchers, what should we be investigating today, such that we have the same impact in the next 10 years?”
Forget social media memes — searching for cat photos is serious business for members of the Molecular Information Systems Lab.
Picture this: You are a researcher in the Molecular Information Systems Lab housed at the Paul G. Allen School. You and your labmates have developed a system for storing and retrieving digital images in synthetic DNA to demonstrate how this extremely dense and durable medium might be used to preserve the world’s growing trove of data. And you have a particular fondness for cats.
If you wanted to sift through photos of frisky felines — and really, what better way to spend an afternoon — how would you pick out the relevant files floating around in your test-tube database without having to sequence the entire pool?
In a paper published recently in the journal Nature Communications, a team of MISL scientists presented the first technique for performing content-based similarity search among digital image files stored in DNA molecules. The approach is akin to that of a modern search engine, albeit in a much smaller form factor than your average server farm and with the potential to be much more energy efficient.
”Content-based search enables us to type a word or phrase into a hundred-page document and be taken to the exact page it appears, or upload a photo of a daisy and get flower images in return,” said co-author Yuan-Jyue Chen, senior researcher at Microsoft and an affiliate professor at the Allen School. “We don’t know the specific page numbers or files we’re looking for, so that computation saves us the trouble of reading the entire document or scrolling through every photo on the internet. Our team took that same idea and applied it to data stored in molecular form.”
Files, whether stored in digital or molecular form, make use of a process known in database parlance as key-based retrieval. In an electronic database, it is typically a file name; in a molecular database, it is a unique sequence, reminiscent of a barcode, that is encoded in the snippets of DNA associated with a particular file. Items with this barcode can be amplified via polymerase chain reaction (PCR) to reassemble a file in its entirety, since a single digital file might be split among hundreds — possibly even thousands — of DNA oligonucleotides, depending on its total size. Generally speaking, key-based retrieval works great when you know the contents of the files and can pick out which ones you want to retrieve; if you don’t and your data is stored as the As, Ts, Cs and Gs of DNA instead of 0s and 1s, the entire database would have to be sequenced in order to perform a content-based search.
Left to right: Luis Ceze, Yuan-Jyue Chen, Callista Bee, Karin Strauss, David Ward and Aaron Liu. Tara Brown Photography
To move beyond the limitations of key-based retrieval, the researchers leveraged DNA’s natural hybridization behavior along with machine learning techniques to enable similarity search to be performed on the stored data. In a conventional digital database, similarity search relies on a set of feature vectors that are stored separately from the original data. When a search is executed, its results point to the location of each data file associated with a particular feature vector. For the molecular version, the MISL team developed an image-to-sequence encoding scheme that employs a convolutional neural network to designate image feature vectors as “similar” or “not similar” and then maps them to DNA sequences that will predictably hybridize — or bind — with the reverse complement of a query feature vector processed by the same neural network during execution of a search. The technique can be applied to new images not seen during training, and the entire process is easily extended to other types of data such as videos and text.
The researchers created an experimental database by running a collection of 1.6 million images through their encoder, which converted them to DNA sequences incorporating their feature vectors, and tacked on a unique barcode identifier for each file. They then performed a similarity search for three photos — including one of a tuxedo cat named Janelle — using the reverse complement of each query image’s encoded feature sequence against a sample of the database. After filtering out the hybridized target/query pairs for high-throughput sequencing, they found the most frequently sequenced oligos did, indeed, corresponded to images in the database that were visually similar to the query images.
The team found that its molecular-based approach was comparable to that of in silico algorithms representing the state of the art in similarity search. Unlike those algorithms, however, the team points out that DNA-based search has the potential to scale to significantly larger databases without a correspondingly significant increase in processing time and energy consumption due to its inherently parallel nature. In this way, researchers have barely scratched the surface of what DNA computing can do.
Left to right: Lee Organick, Melissa Queen and Georg Seelig. Tara Brown Photography
“As DNA data storage is made more practical by advances in fast and low-cost synthesis, sequencing and automation, the ability to perform computation within these databases will pave the way for hybrid molecular-electronic computer systems. Starting with similarity search is exciting because that is a popular primitive in machine learning systems, which are quickly becoming pervasive,” Allen School professor and co-corresponding author Luis Ceze said.
In addition to Ceze and Chen, contributors to the paper include lead author and Allen School alumna Callista Bee (Ph.D., ‘20), Ph.D. students Melissa Queen and Lee Organick, former research scientist Xiaomeng (Aaron) Liu, lab manager David Ward, Allen School and UW Electrical & Computer Engineering professor Georg Seelig, and co-corresponding author Karin Strauss, an affiliate professor in the Allen School and senior principal research manager at Microsoft. Bee initially presented the team’s proof of concept and precursor to this work at the 24th International Conference on DNA Computing and Molecular Programming (DNA 24), for which she earned a Best Student Paper Award.
