Jain, who is co-advised by Allen School professor Jon Froehlich and Human Centered Design & Engineering professor and Allen School adjunct professor Leah Findlater, works in the Makeability Lab to advance sound accessibility by designing, building and deploying systems that leverage human computer interaction (HCI) and artificial intelligence (AI). His primary aim is to help people who are d/Deaf and hard of hearing (DHH) to receive important and customized sound feedback. The dissertation grant will support Jain’s continuing work on the design and evaluation of three of these systems.
One of his projects, HomeSound, is a smart home system that senses and visualizes sound activity like the beeping of a microwave, blaring of a smoke alarm or barking of a dog in different rooms of a home. It consists of a microphone and visual display, which could be either a screen or a smartwatch, with several devices installed throughout the premises. Another system, SoundWatch, is an app that provides always-available sound feedback on smartwatches. When the app picks up a nearby sound like a car honking, a bird chirping or someone hammering, it sends the user a notification along with information about the sound. Jain also has contributed to the development of HoloSound, an augmented reality head-mounted display system that uses deep learning to classify and visualize the identity and location of sounds in addition to providing speech transcription of nearby conversations. All three projects are currently being deployed and tested with DHH users.
“Dhruv is a dedicated researcher who draws on his own unique life experiences to design and build interactive systems for people who are deaf or hard of hearing,” Froehlich said. “DJ cares not just about academic results and solving hard technical problems but in pushing towards deployable solutions with real-world impact. SoundWatch is a great example: DJ helped lead a team in building a real-time sound recognizer in the lab that they then translated to an on-watch system deployed on the Google Play Store. Thus far, it’s been downloaded over 500 times.”
Over the course of his research, Jain has published 20 papers at top HCI venues such as the Association for Computing Machinery’s Conference on Human Factors in Computing Systems (CHI), Symposium on User Interface Software and Technology (UIST), Designing Interactive Systems Conference (DIS) and Conference on Computers and Accessibility (ASSETS). His work has received two Best Paper Awards, three Best Paper Honorable Mentions and one Best Artifact Award.
How can we endow artificial intelligence with the ability to comprehend and draw knowledge from the immense and varied trove of online documents? Is there a way to make testing and debugging of software that pervades our increasingly technology-dependent world more efficient and robust? Speaking of pervasiveness, as technologies like machine learning are more embedded into our society, how can we be sure that these systems reflect and serve different populations equitably?
Those are the questions that Allen School professors Hannaneh Hajishirzi, René Just and Jamie Morgenstern are grappling with in their latest research. And while each focuses on a different area, they all have at least two things in common: Their commitment to integrating research and education earned them National Science Foundation CAREER Awards recognizing junior faculty who exemplify the role of teacher-scholars, and their contributions are helping to cement the Allen School’s leadership in both core and emerging areas of computing.
Hannaneh Hajishirzi: Knowledge-rich neural text comprehension and reasoning
Hajishirzi joined the Allen School faculty full-time in 2018 and is the director of the H2Lab, where she focuses on finding and addressing foundational problems in artificial intelligence (AI) and natural language processing (NLP). She is also a Fellow at the Allen Institute for AI. In her CAREER award-winning work, Hajishirzi aims to improve textual comprehension and reasoning using AI capabilities and deep learning algorithms to help applications better understand text and draw logical conclusions. For example, her research seeks to enable AI systems to answer questions such as “What percentage of Washington state’s budget has been spent on education over the last 20 years?” or verify claims like “Baricitnib prevents 2019-COV from infecting AT2 lung cells.” To do so, the AI system needs to comprehend the claim, find data from reliable sources like scientific articles, and perform reasoning skills to integrate explicit and implicit evidence. Hajishirzi will integrate AI capabilities into deep learning algorithms to understand and reason about textual comprehension to devise hybrid, interpretable algorithms that understand and reason about textual knowledge across varied formats and styles. It will also generalize to emerging domains with scarce training data and operate efficiently under resource limitations.
“Effectively unlimited quantities of ever-changing knowledge are available online in diverse styles, such as news vs. science text and formats — knowledge bases, financial reports and textual documents,” Hajishirzi said. “The content and language style in these domains might evolve over time; for example, scientific articles about the Covid-19 pandemic may use different vocabulary and style as scientists learn more about the virus and communicate findings to different audiences. It is really important to build AI systems that make sense of this enormous amount of knowledge. As a result, textual comprehension is a fundamental problem in AI and NLP.”
René Just: Toward effective, predictable, and consistent software testing
Since joining the Allen School faculty in 2018, Just has focused on software engineering and data science, in particular static and dynamic program analysis, empirical software engineering and applied statistics and machine learning. Through his CAREER-winning research, Just is working to increase the quality of the software that powers modern technology and to improve software development more generally with a framework and methodology for systematic software testing. By developing more effective and consistent software testing approaches, Just will provide developers with concrete test goals quantifying the degree to which these test goals are representative of the defects that were experienced during development in the past. By developing more effective and consistent software testing, his method will be able to assess test goals to check for bugs that were experienced during the development of the software.
“Given that software affects nearly every aspect of our daily lives, its defects have serious implications, and significant advances in software quality benefit every corner of society,” Just said. “This is the main motivation for my work on software testing, which aims to prevent fatal software incidents, million-dollar defects, and a lot of frustration caused by software that simply isn’t working as expected.”
Jamie Morgenstern: Strategic and equity considerations in machine learning
Morgenstern, who joined the Allen School faculty in 2019, studies the social impact of machine learning (ML) and the impact of social behavior because of ML. In her CAREER award-winning work, Morgenstern is researching the performance of machine learning systems that are trained and tested on heterogeneous and strategically generated data and make both predictions and decisions. This work will help guarantee high-quality predictions in domains with varied data sources. It will also help ensure that insights from ML will apply to diverse populations rather than just the majority. Morgenstern’s work will transform the way both researchers and practitioners reason about human-centric ML models. Additionally, it will shed light on interesting technical questions about how optimizing for different measures of performance affects minority communities.
“This will provide guarantees on the future performance of systems built on human-generated historical data, even when those people will continue to interact with the system,” Morgenstern said. “My research will inform the design of learning algorithms trained and deployed on human-generated data, in domains such as commerce, medicine, risk modeling, and social benefit allocation.”
Morgenstern, Just and Hajishirzi are the most recent Allen School faculty members to advance leading-edge research with support from the NSF CAREER program. A total of 64 faculty members have received one of these prestigious awards or their predecessor, the Presidential/NSF Young Investigator Award, during their time at the Allen School.
MISL researcher Nicolas Cardozo pipes cell cultures containing NanoporeTERs onto a portable MinION nanopore sensing device for processing as professor Jeff Nivala looks on. Dennis Wise/University of Washington
Genetically encoded reporter proteins have been a mainstay of biotechnology research, allowing scientists to track gene expression, understand intracellular processes and debug engineered genetic circuits. But conventional reporting schemes that rely on fluorescence and other optical approaches come with practical limitations that could cast a shadow over the field’s future progress. Now, thanks to a team of researchers at the University of Washington and Microsoft, scientists are about to see reporter proteins in a whole new light.
In a paper published today in the journal Nature Biotechnology, members of the Molecular Information Systems Laboratory housed at the UW’s Paul G. Allen School of Computer Science & Engineering introduce a new class of reporter proteins that can be directly read by a commercially available nanopore sensing device. The new system ― dubbed “Nanopore-addressable protein Tags Engineered as Reporters,” also known as NanoporeTERs or NTERs for short ― can perform multiplexed detection of protein expression levels from bacterial and human cell cultures far beyond the capacity of existing techniques.
You could say the new system offers a “nanopore-tal” into what is happening inside these complex biological systems where, up until this point, scientists have largely been operating in the dark.
“NanoporeTERs offer a new and richer lexicon for engineered cells to express themselves and shed new light on the factors they are designed to track. They can tell us a lot more about what is happening in their environment all at once,” said co-lead author Nicolas Cardozo, a graduate student in the UW’s molecular engineering Ph.D. program. “We’re essentially making it possible for these cells to ‘talk’ to computers about what’s happening in their surroundings at a new level of detail, scale and efficiency that will enable deeper analysis than what we could do before.”
Raw nanopore signals streaming from the MinION device, which contains an array of hundreds of nanopore sensors; each color represents data from an individual nanopore. The team uses machine learning to interpret these signals as NanoporeTERs barcodes. Dennis Wise/University of Washington
Conventional methods that employ optical reporter proteins, such as green fluorescent protein (GFP), are limited in the number of distinct genetic outputs that they can track simultaneously due to their overlapping spectral properties. For example, it’s difficult to distinguish between more than three different fluorescent protein colors, limiting multiplexed reporting to a maximum of three outputs. In contrast, NTERs were designed to carry distinct protein “barcodes” composed of strings of amino acids that, when used in combination, enable a degree of multiplexing approaching an order of magnitude more. These synthetic proteins are secreted outside of the cell into the surrounding environment, where they are collected and directly analyzed using a commercially available nanopore array — in this case, the Oxford Nanopore Technologies MinION device. To make nanopore analysis possible, the NTER proteins were engineered with charged “tails” that get pulled into the tiny nanopore sensors by an electric field. Machine learning is then used to classify their electrical signals in order to determine the output levels of each NTER barcode.
“This is a fundamentally new interface between cells and computers,” explained Allen School research professor and corresponding author Jeff Nivala. “One analogy I like to make is that fluorescent protein reporters are like lighthouses, and NanoporeTERs are like messages in a bottle. Lighthouses are really useful for communicating a physical location, as you can literally see where the signal is coming from, but it’s hard to pack more information into that kind of signal. A message in a bottle, on the other hand, can pack a lot of information into a very small vessel, and you can send many of them off to another location to be read. You might lose sight of the precise physical location where the messages were sent, but for many applications that’s not going to be an issue.”
In developing this new, more expressive vessel, Nivala and his colleagues eschewed time-consuming sample preparation or the need for other specialized laboratory equipment to minimize both latency and cost. The NTERs scheme is also highly extensible. As a proof of concept, the team developed a library of more than 20 distinct tags; according to co-lead author Karen Zhang, the potential is significantly greater.
Co-authors of the Nature Biotechnology paper (left to right): Karen Zhang, Nicolas Cardozo, Kathryn Doroschak and Jeff Nivala. Not pictured: Aerilynn Nguyen, Zoheb Siddiqui, Nicholas Bogard, Karin Strauss and Luis Ceze. Tara Brown Photography
“We are currently working to scale up the number of NTERs to hundreds, thousands, maybe even millions more,” Zhang, who graduated this year from the UW with bachelor’s degrees in biochemistry and microbiology, explained. “The more we have, the more things we can track. We’re particularly excited about the potential in single-cell proteomics, but this could also be a game-changer in terms of our ability to do multiplexed biosensing to diagnose disease and even target therapeutics to specific areas inside the body. And debugging complicated genetic circuit designs would become a whole lot easier and much less time consuming if we could measure the performance of all the components in parallel instead of by trial and error.”
MISL researchers have made novel use of the ONT MinION device before. Allen School alumna Kathryn Doroschak (Ph.D., ‘21), one of the lead co-authors of this paper, was also involved in an earlier project in which she and her teammates developed a molecular tagging system to replace conventional inventory control methods. That system relied on barcodes comprising synthetic strands of DNA that could be decoded on demand using the portable ONT reader. This time, she and her colleagues went a step further in demonstrating how versatile such devices can be.
“This is the first paper to show how a commercial nanopore sensor device can be repurposed for applications other than the DNA and RNA sequencing for which they were originally designed,” explained Doroschak. “This is exciting as a precursor for nanopore technology becoming more accessible and ubiquitous in the future. You can already plug a nanopore device into your cell phone; I could envision someday having a choice of ‘molecular apps’ that will be relatively inexpensive and widely available outside of traditional genomics.”
Additional co-authors of the paper include research assistants Aerilynn Nguyen and Zoheb Siddiqui, former postdoc Nicholas Bogard, Allen School affiliate professor Karin Strauss, senior principal research manager at Microsoft; and Allen School professor Luis Ceze.
Kyle Johnson, a Ph.D. student working with professor Shyam Gollakota in the Allen School’s Network and Mobile Systems Lab, received a Generation Google Scholarship for his academic performance, leadership and commitment to diversity, equity and inclusion. The company created the scholarship to help students pursuing computer science degrees excel in technology and become leaders in the field.
As a researcher, Johnson aims to create battery-less micro robots designed to operate autonomously for prolonged periods of time, implementable as the sensory notes in a swarm algorithm. To achieve this, he plans to leverage the properties of structures like leaf-out origami to create origami robots. These have potential applications in many environments requiring both low-power and small-scale devices, like in the deployment of space rovers for interplanetary exploration.
“As a first year graduate student, Kyle is leading embedded systems and robotic research that span aerospace, electrical and mechanical engineering and computer science,” said Gollakota. “The miniaturized robotic systems he is building, if successful, are creative and more importantly useful for achieving wireless sensor deployment at a scale that has not been possible before.”
In addition to his research, Johnson has been a leader in working towards improving diversity, equity and inclusion in academia and technology fields. For the past three years, Johnson has taught middle and high school students how to code and hardwire a multitude of different sensors and devices, demonstrating to students that they can apply the technical concepts that they learn in class towards solving real-world problems.
In the summer of 2019, Johnson traveled to Cape Town, South Africa to study racism, education and development. There, he learned from high school math and physics teachers about creating lesson plans and teaching to diverse students before eventually teaching some of those classes himself. At the University of South Africa, Johnson shared his experiences during the “Trauma, Educational Exclusions and Survival: Examining Global Student Experience & Resilience” workshop at an academic development symposium and encouraged those at the university to be more engaged in public schools to better prepare students for higher education.
Upon his return to the United States, Johnson co-founded the student group A Vision for Electronic Literary & Access (AVELA) to provide more opportunities for underrepresented students to pursue their interests at a university level. Johnson and the other AVELA members accomplish this by representing the populations that they aim to support while leading workshops, camps and other forms of community outreach. AVELA has partnered with Seattle Public Schools, the Kent School District, the National Society of Black Engineers, Seattle MESA, The Urban League of Metropolitan Seattle, InspireOne, and the city of Seattle to teach engineering workshops and create lesson plans catered towards aspiring, underrepresented students. The organization is in the process of earning a 501(c) status; as a nonprofit, AVELA can apply for grants and additional funding.
Over the past year, Johnson has worked to help change policies and job descriptions at the University of Washington to make them more equitable and created tools for students to more quickly report racism, sexism and other discriminatory actions. He also co-founded the Black Student Graduate Association (BSGA).
““The BGSA focuses on giving UW’s Black graduate students the space to relax, network and share experiences,” Johnson said. “As the 100-member organization expands, I hope more Black graduate students will find solace in a community of their peers.”
In its latest round of funding intended to strengthen the United States of America’s leadership in artificial intelligence research, the National Science Foundation today designated a new NSF AI Institute for Future Edge Networks and Distributed Intelligence (AI-EDGE) that brings together 30 researchers from 18 universities, industry partners and government labs. Allen School professor Sewoong Oh is among the institute researchers who will spearhead the development of new AI tools and techniques to advance the design of next-generation wireless edge networks. The focus of AI-EDGE, which is led by The Ohio State University, will be on ensuring such networks are efficient, robust and secure.
Among the exciting new avenues Oh and his colleagues are keen to explore is the creation of tools that will enable wireless edge networks to be both self-healing and self-optimizing in response to changing network conditions. The team’s work will support future innovations in a variety of domains, from telehealth and transportation to robotics and aerospace.
“The future is wireless, which means much of the growth in devices and applications will be focused at the network edge rather than in the traditional network core,” Oh said. “There is tremendous benefit to be gained by building new AI tools tailored to such a distributed ecosystem, especially in making these networks more adaptive, reliable and resilient.”
AI-EDGE, which will receive $20 million in federal support over five years, is partially funded by the Department of Homeland Security. It is one of 11 new AI research institutes announced by the NSF today — including the NSF AI Institute for Dynamic Systems led by the University of Washington.
“These institutes are hubs for academia, industry and government to accelerate discovery and innovation in AI,” said NSF Director Sethuraman Panchanathan in the agency’s press release. “Inspiring talent and ideas everywhere in this important area will lead to new capabilities that improve our lives from medicine to entertainment to transportation and cybersecurity and position us in the vanguard of competitiveness and prosperity.”
Oh expects there will be synergy between the work of the new AI-EDGE Institute and the NSF AI Institute for Foundations in Machine Learning unveiled last summer to address fundamental challenges in machine learning and maximize the impact of AI on science and society. As co-PI of IFML, he works alongside Allen School colleagues Byron Boots, Sham Kakade and Jamie Morgenstern and adjunct faculty member Zaid Harchaoui, a professor in the UW Department of Statistics, in collaboration with lead institution University of Texas at Austin and other academic and industry partners to advance the state of the art in deep learning algorithms, robot navigation, and more. In addition to tackling important research questions with real-world impact, AI-EDGE and IFML also focus on advancing education and workforce development to broaden participation in the field.
Allen School alumna Rajalakshmi Nandakumar (Ph.D., ‘20), now a faculty member at Cornell University, received the SIGMOBILE Doctoral Dissertation Award from the Association for Computing Machinery’s Special Interest Group on Mobility of Systems Users, Data, and Computing “for creating an easily-deployed technique for low-cost millimeter-accuracy sensing on commodity hardware, and its bold and high-impact applications to important societal problems.” Nandakumar completed her dissertation, “Computational Wireless Sensing at Scale,” working with Allen School professor Shyam Gollakota in the University of Washington’s Networks & Mobile Systems Lab.
In celebrating Nandakumar’s achievements, the SIGMOBILE award committee highlighted “the elegance and simplicity” of her approach, which turns wireless devices such as smartphones into active sonar systems capable of accurately sensing minute changes in a person’s movements. The committee also heralded her “courage and strong follow-through” in demonstrating how her technique can be applied to real-world challenges — including significant public health issues affecting millions of people around the world.
Among the contributions Nandakumar presented as part of her dissertation was ApneaApp, a smartphone-based system for detecting a potentially life-threatening condition, obstructive sleep apnea, that affects an estimated 20 million people just in the United States alone. Unlike the conventional approach to diagnosing apnea, which involves an overnight stay in a specialized lab, the contactless solution devised by Nandakumar and her Allen School and UW Medicine collaborators could be deployed in the comfort of people’s homes. ApneaApp employs the phone’s speaker and microphone to detect changes in a person’s breathing during sleep, without requiring any specialized hardware. It works by emitting inaudible acoustic signals that are then reflected back to the device and analyzed for deviations in the person’s chest and abdominal movements. ResMed subsequently licensed the technology and made it available to the public for analyzing sleep quality via its SleepScore app.
Her early work on contactless sleep monitoring opened Nandakumar’s eyes to the potential for expanding the use of smartphone-based sonar to support early detection and intervention in the case of another burgeoning public health concern: preventable deaths via accidental opioid overdose. This led to the development of Second Chance, an app that a person activates on their smartphone to unobtrusively monitor changes in their breathing and posture that may indicate the onset of an overdose. Catching these early warning signs as soon as they occur would enable the timely administration of life-saving naloxone. Nandakumar’s colleagues created a startup, Sound Life Sciences, to commercialize this and related work that employs sonar to detect and monitor a variety of medical conditions via smart devices.
The SIGMOBILE Dissertation Award is the latest in a string of honors recognizing Nandakumar for her groundbreaking contributions in wireless systems research. She previously earned a Paul Baran Young Scholar Award from the Marconi Society, a Graduate Innovator Award from UW CoMotion, and a Best Paper Award at SenSys 2018.
“Rajalakshmi is brilliant, creative and fearless in her research. She repeatedly questions conventional wisdom and takes a different path from the rest of the community,” said Allen School professor Ed Lazowska, who supported Nandakumar’s nomination. “Her work points to the future — a future in which advances in computer science and computer engineering will have a direct bearing on our capacity to tackle societal challenges such as health and the environment. Rajalakshmi is a game-changer.”
Way to go, Rajalakshmi!
Photo credit: Sarah McQuate/University of Washington
Allen School professor Yejin Choi is among a team of researchers recognized by the Computer Vision Foundation with its 2021 Longuet-Higgins Prize for their paper “Baby talk: Understanding and generating simple image descriptions.” The paper was among the first to explore the new task of generating image captions in natural language by bridging two fields of artificial intelligence: computer vision and natural language processing. Choi, who is also a senior research manager at the Allen Institute for AI (AI2), completed this work while a faculty member at Stony Brook University. She and her co-authors originally presented the paper at the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Baby talk is the process by which adults assist infants in acquiring language and building their understanding of the world that is characterized in part by the use of grammatically simplified speech. Drawing upon this concept, Choi and her collaborators set out to teach machines to generate simple yet original sentences describing what they “see” in a given image. This was a significant departure from conventional approaches grounded in the retrieval and summarization of pre-existing content. To move past the existing paradigm, the researchers constructed statistical models for visually descriptive language by mining and parsing the large quantities of text available online and paired them with the latest recognition algorithms. Their strategy enabled the new system to describe the content of an image by generating sentences specific to that particular image, as opposed to requiring it to shoehorn content drawn from a limited document corpus into a suitable description. The resulting captions, the team noted, had greater relevance and precision in the way they describe the visual content.
Yejin Choi
“At the time we did this work, the question of how to align the semantic correspondences or alignments across different modalities, such as language and vision, was relatively unstudied. Image captioning is an emblematic task to bridge the longstanding gap between NLP research with computer vision,” explained Choi. “By bridging this divide, we were able to generate richer visual descriptions that were more in line with how a person might describe visual content — such as their tendency to include not just information on what objects are pictured, but also where they are in relation to each other.”
This incorporation of spatial relationships into their language generator was key in producing more natural-sounding descriptions. Up to that point, computer vision researchers who focused on text generation from visual content relied on spatial relationships between labeled regions of an image solely to improve labeling accuracy; they did not consider them outputs in their own right on a par with objects and modifiers. By contrast, Choi and her colleagues considered the relative positioning of individual objects as integral to developing the computer vision aspect of their system, to the point of using these relationships to drive sentence generation in conjunction with the depicted objects and their modifiers.
Some of the results were deemed to be “astonishingly good” by the human evaluators. In one example presented in the paper, the system accurately described a “gray sheep” as being positioned “by the gray road”; the “gray sky,” it noted, was above said road. For another image, the system correctly pegged that the “wooden dining table” was located “against the first window.” The system also accurately described the attributes and relative proximity of rectangular buses, shiny airplanes, furry dogs, and a golden cow — among other examples.
The golden cow
The Longuet-Higgins Prize is an annual “test of time” award presented during CVPR by the IEEE Pattern Analysis and Machine Intelligence (PAMI) Technical Committee to recognize fundamental contributions that have had a significant impact in the field of computer vision. Choi’s co-authors on this year’s award-winning paper include then-master’s students Girish Kulkarni, Visruth Premraj and Sagnik Dhar; Ph.D. student SiMing Li; and professors Alexander C. Berg and Tamara L. Berg, both now on the faculty at University of North Carolina Chapel Hill.
Allen School alumnus Adrian Sampson (Ph.D., ‘15) has been recognized by the IEEE Computer Society’s Technical Committee on Computer Architecture with the 2021 Young Computer Architect Award for “contributions to approximate computing and hardware synthesis from high-level representations.” This award honors an outstanding researcher who has completed their doctoral degree within the past six years and who has made innovative contributions to the field of computer architecture. Sampson, who completed his Ph.D. working with Allen School professor Luis Ceze and Allen School professor and vice director Dan Grossman, is now a faculty member in the Department of Computer Science at Cornell University in the Cornell Ann S. Bowers College of Computing and Information Science.
“Adrian’s work on programming language-hardware co-design for approximate computing led to a new research area with lots of follow-on research by the community,” said Ceze. “His research impact is complemented by him being an amazingly creative, caring and fun human being. I could not be more proud of him.”
Sampson devoted his early career to new abstractions in approximate computing, with a focus on rethinking modern computer architecture — specifically reducing the energy consumption of computer systems. For instance, Sampson, along with Ceze and Grossman, created EnerJ, a language for principled approximate computing that allows programmers to indicate where it is safe to permit occasional errors in order to save energy. While power consumption of computers is often strained by correctness, guarantees like EnerJ, an extension to Java, exposes hardware faults in a safe, principled manner allowing power-saving techniques like lower voltage. Sampson’s research shows that approximate computing is a promising way of saving energy in large classes of applications running on a wide range of systems, including embedded systems, mobile phones and servers.
“Modern architecture gives everyone a license to try ideas that are weird and ambitious and potentially transformative,” Sampson said about his work. “It would be silly not to take up that opportunity.”
At Cornell, Sampson and his research group, Capra, are focused on programming languages and compilers for generating hardware accelerators. One area in particular is field-programmable gate arrays (FPGAs), which allow the co-design of applications with hardware accelerators. These are still hard to program, so Sampson and his team created Dahlia, a programming language that leverages an affine type system to constrain programs to only represent valid hardware designs. Dahlia aims to compile high-level programming languages into performant hardware designs and offer open source tools to help programming language and architecture.
“What Adrian has leveraged in clever and impactful ways throughout his career is that many potential amazing performance advantages at the hardware level are possible only if given extra information or assurance from the software as to where the techniques are safe to use,” Grossman said.
In addition to his Computer Architecture Award, Sampson previously was recognized with an NSF CAREER Award in 2019 and a Google Faculty Research Award in 2016, just to name a few, and has published more than 40 papers.
Allen School professors Maya Cakmak and Dieter Fox, along with their collaborators at NVIDIA’s AI Robotics Research Lab, earned the award for Best Paper in Human-Robot Interaction at the IEEE International Conference on Robotics and Automation (ICRA 2021) for introducing a new vision-based system for the smooth transfer of objects between human and robot. In “Reactive Human-to-Robot Handovers of Arbitrary Objects,” the team employs visual object and hand detection, automatic grasp selection, closed-loop motion planning, and real-time manipulator control to enable the successful handoff of previously unknown objects of various sizes, shapes and rigidity. It’s a development that could put more robust human-robot collaboration within reach.
“Dynamic human-robot handovers present a unique set of research challenges compared to grasping static objects from a recognizable, stationary surface,” explained Fox, director of the Allen School’s Robotics and State Estimation Lab and senior director of robotics research at NVIDIA. “In this case, we needed to account for variations not just in the objects themselves, but in how the human moves the object, how much of it is covered by their fingers, and how their pose might constrain the direction of the robot’s approach. Our work combines recent progress in robot perception and grasping of static objects with new techniques that enable the robot to respond to those variables.”
The system devised by Fox, Cakmak, and NVIDIA researchers Wei Yang, Chris Paxton, Arsalan Mousavian and Yu-Wei Chao does not require objects to be part of a pre-trained dataset. Instead, it relies on a novel segmentation module that enables accurate, real-time hand and object segmentation, including objects the robot is encountering for the very first time. Rather than attempting to directly segment objects in the hand, which would not provide the flexibility and adaptability they sought, the researchers trained a fully convolutional network for hand segmentation given an RGB image, and then inferred object segmentation based on depth information. To ensure temporal consistency and stability of the robot’s grasps in response to changes in the user’s motion, the team extended the GraspNet grasp planner to refine the robot’s grasps over consecutive frames over time. This enables the system to react to a user’s movements, even after the robot has begun moving, while consistently generating grasps and motions that would be regarded as smooth and safe from a human perspective.
Maya Cakmak (left) and Dieter Fox
Crucially, the researchers’ approach places zero constraints on the user regarding how they may present an object to the robot; as long as the object is graspable by the robot, the system can accommodate its presentation in different positions and orientations. The team tested the system on more than two dozen common household objects, including a coffee mug, a remote control, a pair of scissors, a toothbrush and a tube of toothpaste, to demonstrate how it generalizes across a variety of items. That variety goes beyond differences between categories of object, as the objects within a single category can also differ significantly in their appearance, dimensions and deformability. According to Cakmak, this is particularly true in the context of people’s homes, which are likely to reflect an array of human needs and preferences to which a robot would need to adapt. To ensure their approach would have the highest utility in the home for users who need assistance with fetching and returning objects, the researchers evaluated their system using a set of everyday objects prioritized by people with amyotrophic lateral sclerosis (ALS).
“We may be able to pass someone a longer pair of scissors or a fuller plate of food without thinking about it — and without causing an injury or making a mess — but robots don’t possess that intuition,” said Cakmak, director of the Allen School’s Human-Centered Robotics Lab. “To effectively assist humans with everyday tasks, robots need the ability to adapt their handling of a variety of objects, including variability among objects of the same type, in line with the human user. This work brings us closer to giving robots that ability so they can safely and seamlessly interact with us, whether that’s in our homes or on the factory floor.”
ICRA is the IEEE’s flagship conference in robotics. The 2021 conference, which followed a hybrid model combining virtual and in-person sessions in Xi’an, China, received more than 4,000 paper submissions spanning all areas of the field.
Visit the project page for the full text of the research paper, demonstration videos and more.
Let’s give a big hand to Maya, Dieter and the entire team!
Allen School professor Emina Torlak, co-founder of the UNSAT group and a member of the Programming Languages & Software Engineering (PLSE) group, earned the 2021 Robin Milner Young Researcher Award from the Association for Computing Machinery’s Special Interest Group on Programming Languages (ACM SIGPLAN) for her research contributions in automated reasoning to make computer programming easier. The award was named in honor of Milner, the legendary British computer scientist who was a leader in programming language research and had a passion for mentoring his younger colleagues. Torlak is the 10th recipient of the award, which is given to a researcher within 20 years of their start of graduate school.
“This prestigious award is explicit recognition of what is well-known in the programming-languages research community: Emina builds beautifully engineered systems that are ready for serious use not just by her group but by other researchers, and she does so by extending the elegant logical foundations of our field in sophisticated ways,” said Dan Grossman, professor and vice director of the Allen School. “Her tools don’t just work; they do things nobody else knows how to do.”
In her work, Torlak combines new language abstractions, algorithmic insights, robust design, and thorough implementation. From the start of her career as a doctoral student at MIT, she has been building tools and applications that find and solve problems in programming languages.
“Automatic verification of programs ensures that programs are free of bugs that hackers could exploit to break into the system. Unfortunately, verifying program correctness is a specialized task that programmers are not trained to perform,” said Allen School professor Rastislav Bodik. “Emina’s research goal has been to make automatic verification of programs accessible to programmers who lack expertise in verification. She has accomplished this vision by automating a lot of the verification process, in a clever way. Her key insight was that it was impossible to construct a super-verifier that would automatically verify all kinds of programs, and that to automate the process, verifiers would need to be tailored to narrower classes of programs. She designed a tool that allowed programmers to automatically construct such automatic verifiers for their programs.”
An example is Torlak’s Rosette framework, which allows users to quickly develop their own domain-specific program synthesis algorithms.
“Rosette is an ingenious design that combines advances from programming languages such as meta-programming, compilation as in partial evaluation and formal methods such as symbolic computation,” Bodik said. “Using Rosette, programmers were able to construct verifiers in a few days where it would take months to do so with traditional methods. Rosette was also used to construct synthesizers of programs, which are tools that automatically write programs. Using Rosette, researchers have built synthesizers that reached parity with human programming experts on a range of advanced programming tasks. This success owes both to Emina’s design skills and the leadership in supporting the open-source community around Rosette.”
Leveraging Rosette, Torlak helped to create Serval, a framework for creating automated verifiers for systems software. It overcomes several obstacles in scaling automated verification, including the developer effort required to write verifiers, the difficulty of finding and fixing performance bottlenecks, and limitations on their applicability to existing systems. From there, she contributed to the development of Jitterbug, a tool for writing and proving the correctness of just-in-time (JIT) compilers for computers using the extended Berkeley Packet Filter (eBPF) technology. It produces a formal correctness guarantee for eBPF JITs, a security-critical component of the Linux operating system, through a precise specification of JIT correctness and an automated verification strategy that scales to practical implementations.
“Emina Torlak is a leader in the area of automated verification. She has made both conceptual contributions and built tools that are state-of-the-art and widely used,” said the Milner Award selection committee. “On the conceptual side, the notion of a solver-aided programming language is hers and has quickly become standard terminology.”
