Each year, Google recognizes approximately 75 exceptional graduate students from around the world through its Google Ph.D. Fellowship Program. The students, who come from a variety of backgrounds, are selected based on their potential to influence the future of technology through their research in computer science and related fields. As part of its 2023 class of Fellows, the company selected two future leaders from the Allen School: Miranda Wei in the Security and Privacy category and Mitchell Wortsman in Machine Learning.
Miranda Wei
Wei joined the Allen School in 2019 to work with professors Tadayoshi Kohno and Franziska Roesner, co-directors of the Security and Privacy Research Lab. Now in the fifth year of her Ph.D., Wei seeks to empower people and mitigate harms from emerging technologies through her dissertation research that explores how factors like gender affect people’s experiences with technological security and privacy. Her work has already contributed to an important new subfield that centers on sociotechnical factors in security and privacy research.
“Miranda’s focus on assessing conditions of disempowerment and empowerment, and then developing mechanisms to help users improve their computer security and privacy, is truly visionary,” said Kohno, who also serves as the Allen School’s Associate Director for Diversity, Equity, Inclusion and Access. “Miranda has not only identified important work to do, but she has identified a strategy and the key components for moving the whole field forward.”
Grounding her research in critical and feminist theories, Wei explores the (dis)empowerment of users through security and privacy measures in several contexts. Her work draws from the social sciences and multiple fields of computer science, including human-computer interaction and information and communication technologies for development in addition to security and privacy. In recent work, Wei has applied both quantitative and qualitative approaches, including case studies and participant interviews, to examine topics such as gender-based stereotypes and computer security and the connection between digital safety and online abuse.
“This research sets the foundation for learning from experiences of marginalization to understand broader sociotechnical systems,” explained Wei. “This enables equitable improvements to security and privacy for all online users.”
Wei’s academic journey began as an undergraduate student at the University of Chicago, where she earned a degree in political science with a minor in computer science. In addition to having published over a dozen peer-reviewed papers, Wei volunteers her time to support new and prospective graduate students through the Allen School’s Pre-Application Mentorship Service (PAMS) and Care Committee. She is also active with DUB as a student coordinator and participates in the University of Washington’s graduate application review process as an area chair.
“Miranda has great insight for research problems at the intersection of computer security and privacy and society, and she pursues this vision passionately and independently,” said Roesner. “At the same time, she is a wonderful collaborator and community member who looks out and advocates for others.”
Mitchell Wortsman
Wortsman, who is also in his fifth year at the Allen School, earned a Google Ph.D. Fellowship for his work with professors Ali Farhadi, co-director of the Reasoning, AI and VisioN (RAIVN) Lab and CEO of the Allen Institute for AI, and Ludwig Schmidt, who is also a research scientist in the AllenNLP group at AI2. Wortsman has broad interest in large-scale machine learning spanning deep learning, from robust and accurate fine-tuning to stable and low-precision pre-training. His dissertation work seeks to improve large pre-trained neural networks as reliable foundations in machine learning.
“One of my main research goals is to develop computer vision models that are robust, meaning that their performance is less degraded by changes in the data distribution,” explained Wortsman. “This will enable the creation of models which are useful and reliable outside of their training distribution.”
With the progress in pre-training large-scale neural networks, machine learning practitioners in the not-so-distant future could potentially spend most of their time fine tuning these networks. Wortsman studies the loss landscape of large pretrained models and explores creative solutions for fine tuning with the goal of improving accuracy and robustness. Wortsman wants his models to be useful to society at large and not exclusively for academic and commercial applications. One of his ongoing projects includes a collaboration with the UW School of Medicine.
Wortsman is first author on over nine peer-reviewed publications, several of which he co-authored as a predoctoral young investigator at AI2, and collaborated on the development of an open source reproduction of OpenAI’s CLIP model. He has also served as a teaching assistant in the Allen School and as a reviewer for PAMS.
“Michell’s work has laid the foundations for many open models that let computers understand and generate images,” said Schmidt. “Mitchell is one of the core developers of OpenCLIP, which is downloaded several thousand times per day and has become part of many AI projects. Every time someone uses Stable Diffusion, one of Mitchell’s models provides the text guidance for the image generation process.”
The Paul G. Allen Center for Computer Science & Engineering shortly after it opened in fall 2003. The theme of the dedication was “Opening the Doors to Our Future.” Photo by Ed LaCasse
In the late 1990’s, members of the Allen School faculty experimented with a new — some would say unorthodox — way to mark the conclusion of Visit Days, the annual pilgrimage made by prospective graduate students to computer science programs around the country. To commemorate the visitors’ time in Seattle, professors in what was then the Department of Computer Science & Engineering would cheerfully send them on their way with a surprise parting gift: a palm-sized chunk of concrete.
The concrete in question had, without any human intervention, become dislodged from the crumbling facade of Sieg Hall — the building that, should the recipients choose the University of Washington, would become a home away from home for the duration of their Ph.D.
“The souvenir definitely made us memorable, and it helped our cause when it came to recruitment,” Allen School professor Ed Lazowska, who chaired the department at the time, recalled wryly. “One student emailed that they just couldn’t say ‘no’ to us after we literally gave them a piece of our building. But giving out chunks of the building, like the building itself, was a joke. We were woefully behind other top computing programs when it came to facilities.”
While outside the building was crumbling, inside it was cramped — so much so that, as a prank, someone set up a “graduate student office” on a ledge in the stairwell, complete with a handy rope ladder for access. More than two decades after it first housed UW’s burgeoning Computer Science & Engineering program, Sieg was no longer fit for the purpose. In 1999, the department stepped up its campaign for a new, permanent home.
Lazowska and local technology industry leaders led the charge, forging a public-private partnership that was unprecedented in UW’s history. All told, they raised $42 million in private funds — substantially more than half the project’s cost — from more than 250 donors. Lazowska’s faculty colleague Hank Levy oversaw the design and construction of the building in tandem with LMN Architects and general contractor M.A. Mortenson. He saw to it that the funds were put to good use.
“Our goal was to create a warm and welcoming environment that would facilitate teaching, research and collaboration,” said Levy. “Every aspect of the building — the materials, the artwork, the abundant natural light, the open spaces that encourage people to gather and exchange ideas — were intentional choices made with this goal in mind.”
Those choices were supported in large part by leadership gifts from the building’s namesake, the late Paul G. Allen, along with the Bill & Melinda Gates Foundation and Microsoft. Completion of the 85,000 square-foot facility, which was dedicated on October 9, 2003, tripled the program’s available space and set off a chain of events that made the Allen School into the powerhouse it is today.
“What really sets UW’s computer science program apart are the people.” Paul G. Allen at the dedication of the building that bears his name.
Allen himself understood at the time that he was investing in something more meaningful than bricks and mortar.
“I’m proud to have supported this beautiful and unique facility, but what really sets UW’s computer science program apart are the people,” Allen observed during the grand opening celebration. “The faculty here is unparalleled, and the undergrad and graduate students are dedicated and inspiring.”
Allen’s faith would inspire a period of expansion that no one — including Lazowska, who has been the program’s most vocal cheerleader over the years — could have foreseen in 2003.
“I cannot stress enough the importance of the Allen Center to the trajectory of our program,” he said. “It provided us with competitive space for the first time in our history. It was the spark that set us on a path to triple our degree production, ramp up our ability to deliver computer science education to students across campus, and attract the brightest researchers in the field to Seattle.
On move-in day in the summer of 2003, fewer than 40 faculty members unpacked boxes in their shiny new offices; two decades later, that number is approaching 100. And faculty recruiting has barely kept pace with the explosive growth in student interest, with the Allen School the most requested major among freshman applicants to the University for several years running. It now serves roughly 2,800 students across its degree programs — and thousands more who take one or more courses as non-majors each year.
As the program grew in size, it also grew in stature, thanks in no small part to its new and improved laboratory space.
“Computer Science & Engineering at the University of Washington is an engine of opportunity,” Allen had said at the time, “and I want to ensure it’s an even more cutting-edge resource for the coming generation.”
That engine has been going full throttle ever since. One high-profile example of how the move to the Allen Center greased the wheels of innovation is UW’s emergence as a center for mobile health. By tapping into the built-in sensing capabilities of smartphones coupled with advances in machine learning, Allen School researchers, in conjunction with UW Medicine clinicians, have developed a range of mobile tools for screening and monitoring of a variety of health conditions spanning fever, pre-diabetes, sleep apnea, infant jaundice, reduced lung function, ear infection, newborn hearing loss and more. All got their start in the Allen Center’s labs, and several led to the creation of Allen School spinout companies.
The collaborations don’t stop there, as the Allen Center provided a launch pad for multiple cross-campus initiatives, some supported by significant federal and/or private investment. These include efforts to advance accessible technologies and more accessible communities, data science for discovery and innovation, neurotechnologies for people with spinal cord injury, stroke or other neurological disorders, next-generation cloud computing infrastructure, computing for environmental sustainability and more. In the past five years alone, the Allen School has secured more than $200 million in grants and contracts to support its research. Along the way, the school has strengthened its leadership in core areas such as systems, architecture and theoretical computer science even as it has expanded its expertise to encompass new areas, including cryptography, molecular programming, quantum computing and natural language processing.
The Allen Center and Gates Centers on the UW campus provide a unified home for the Allen School, which has grown significantly in both size and stature over the past two decades. Photo by Tim Griffith, courtesy of LMN Architects
And that list is by no means exhaustive.
“We took Paul’s words to heart, and the impact of the community’s investment continues to be felt today far beyond the Allen Center’s walls,” said Magdalena Balazinska, director of the Allen School and Bill & Melinda Gates Chair in Computer Science & Engineering. “It is felt through the graduates we’ve mentored, the technologies we’ve developed, the companies we’ve started, the opportunities we’ve created, and the leadership we’ve provided.”
The growth sparked by the Allen Center eventually led UW to break new ground in computing literally as well as figuratively; nearly 16 years later, with its first building now bursting at the seams, the Allen School dedicated its second building, the Bill & Melinda Gates Center, which doubled its physical space.
That additional space came just in time, too. Thanks to advocacy by the University and additional investments from the state legislature, the school is currently on track to award 820 degrees annually and has cemented its place in the top echelon of computer science programs in the nation.
“I said back then that the true measure of this building will be what we do inside to take our programs to the next level of excellence,” said Levy. “I’d like to think that we lived up to that promise, and then some.”
After a two-year hiatus, the University of Washington’s Bonderman Travel Fellows are back, independently traveling the world and benefitting from the monumental growth that comes with immersing oneself in unfamiliar spaces. Since its inception in 1995, the fellowship has supported over 300 UW students on their travels based on their curiosity, openness, resilience and creativity.
Soon, it will be Allen School Ph.D. student Gus Smith’s turn to hit the road, along with seven other graduate students who were named 2023 Bonderman Fellows. Smith, who is in his fifth year at the Allen School co-advised by professors Luis Ceze and Zach Tatlock, focuses his research on using programming language tools to automatically generate compilers for custom hardware.
With support from the fellowship, he will have an opportunity to explore a different kind of language, far from the computing lab.
“I will use my Bonderman journey to bridge gaps between myself and my international friends and colleagues, not only by experiencing their home countries directly, but by challenging myself to experience the feeling of being an outsider in new and unfamiliar countries,” explained Smith. “In the process, I hope to gain more empathy for what it is like to live so far from your country of origin.”
Inspired by the international and first-generation American friends and colleagues he has encountered during his time at UW, Smith proposed to visit at least six countries over a period of five months following graduation. His itinerary — which is still taking shape — will span the continent of Asia from Taiwan in the east to Israel and Jordan in the west. Other highlights will include two months in India, a brief stay in Singapore and an exploration of Chiang Mai and Bangkok in Thailand.
Along the way Smith hopes to connect with the cultures of the people he has known to develop a better understanding of their backgrounds. He also anticipates personal growth that will facilitate deeper connections with his friends, fellow computer scientists and other people he meets throughout his life.
“During my travels, I’ll seek to understand how people across the world engage with the search for happiness and how they cope with the knowledge that what they seek may be elusive or impermanent,” said Smith. “If I meet a thousand people, I’ll find a thousand different answers to that question.”
Researchers at the University of Washington developed small robotic devices that can change how they move through the air by “snapping” into a folded position during their descent. Shown here is a timelapse photo of the “microflier” falling in its unfolded state, which makes it tumble chaotically and spread outwardin the wind. Photo by Mark Stone/University of Washington
On a cool afternoon at the heart of the University of Washington’s campus, autumn, for a few fleeting moments, appears to have arrived early. Tiny golden squares resembling leaves flutter then fall, switching from a frenzied tumble to a graceful descent with a snap.
Aptly named “microfliers” and inspired by Miura-fold origami, these small robotic devices can fold closed during their descent after being dropped from a drone. This “snapping” action changes the way they disperse and may, in the future, help change the way scientists study agriculture, meteorology, climate change and more.
“In nature, you see leaves and seeds disperse in just one manner,” said Kyle Johnson, an Allen School Ph.D. student and a first co-author of the paper on the subjectpublished in Science Robotics this month. “What we were able to achieve was a structure that can actually act in two different ways.”
When open flat, the devices tumble chaotically, mimicking the descent of an elm leaf. When folded closed, they drop in a more stable manner, mirroring how a maple leaf falls from a branch. Through a number of methods — onboard pressure sensor, timer or a Bluetooth signal — the researchers can control when the devices transition from open to closed, and in doing so, manipulate how far they disperse through the air.
How could they achieve this? By reading between the lines.
“The Miura-ori origami fold, inspired by geometric patterns found in leaves, enables the creation of structures that can ‘snap’ between a flat and more folded state,” said co-senior author Vikram Iyer, an Allen School professor and co-director of the Computing for the Environment (CS4Env) initiative. “Because it only takes energy to switch between the states, we began exploring this as an energy efficient way to change surface area in mid-air, with the intuition that opening or closing a parachute will change how fast an object falls.”
That energy efficiency is key to being able to operate without batteries and scale down the fliers’ size and weight. Fitted with a battery-free actuator and a solar power-harvesting circuit, microfliers boast energy-saving features not seen in larger and heavier battery-powered counterparts such as drones. Yet they are robust enough to carry sensors for a number of metrics, including temperature, pressure, humidity and altitude. Beyond measuring atmospheric conditions, the researchers say a network of these devices could help paint a picture of crop growth on farmland or detect gas leaks near population centers.
“This approach opens up a new design space for microfliers by using origami,” said Shyam Gollakota, the Thomas J. Cable Endowed Professor in the Allen School and director of the school’s Mobile Intelligence Lab who was also a co-senior author. “We hope this work is the first step towards a future vision for creating a new class of fliers and flight modalities.”
Weighing less than half a gram, microfliers require less material and cost less than drones. They also offer the ability to go where it’s too dangerous for a human to set foot.
For instance, Johnson said, microfliers could be deployed when tracking forest fires. Currently, firefighting teams sometimes rappel down to where a fire is spreading. Microfliers could assist in mapping where a fire may be heading and where best to drop a payload of water. Furthermore, the team is working on making more components of the device biodegradable in the case that they can’t be recovered after being released.
“There’s a good amount of work toward making these circuits more sustainable,” said Vicente Arroyos, another Allen School Ph.D. student and first co-author on the paper. “We can leverage our work on biodegradable materials to make these more sustainable.”
Besides improving sustainability, the researchers also tackled challenges relating to the structure of the device itself. Early prototypes lacked the carbon fiber roots that provide the rigidity needed to prevent accidental transitions between states.
The research team took inspiration from elm and maple leaves in designing the microfliers. When open flat, the devices tumble chaotically, similar to how an elm leaf falls from a branch. When they are “snapped” into a folded position, as shown here, they descend in a more stable, straight downward manner like a maple leaf. Photo by Mark Stone/University of Washington
Collecting maple and elm leaves from outside their lab, the researchers noticed that while their origami structures exhibited the bistability required to change between states, they flexed too easily and didn’t have the venation seen in the found foliage. To gain more fine-grained control, they took another cue from the environment.
“We looked again to nature to make the faces of the origami flat and rigid, adding a vein-like pattern to the structure using carbon fiber,” Johnson said. “After that modification, we no longer saw a lot of the energy that we input dissipate over the origami’s faces.”
In total, the researchers estimate that the development of their design took about two years. There’s still room to grow, they added, noting that the current microfliers can only transition from open to closed. They said newer designs, by offering the ability to switch back and forth between states, may offer more precision and flexibility in where and how they’re used.
During testing, when dropped from an altitude of 40 meters, for instance, the microfliers could disperse up to distances of 98 meters in a light breeze. Further refinements could increase the area of coverage, allowing them to follow more precise trajectories by accounting for variables such as wind and inclement conditions.
Related to their previous work with dandelion-inspired sensors, the origami microfliers build upon the researchers’ larger goal of creating the internet of bio-inspired things. Whereas the dandelion-inspired devices featured passive flight, reflecting the manner in which dandelion seeds disperse through the wind, the origami microfliers function as complete robotic systems that include actuation to change their shape, active and bi-directional wireless transmission via an onboard radio, and onboard computing and sensing to autonomously trigger shape changes upon reaching a target altitude.
“This design can also accommodate additional sensors and payload due to its size and power harvesting capabilities,” Arroyos said. “It’s exciting to think about the untapped potential for these devices.”
The future, in other words, is quickly taking shape.
“Origami is inspired by nature,” Johnson added, smiling. “These patterns are all around us. We just have to look in the right place.”
The project was an interdisciplinary work by an all-UW team. The paper’s co-authors also included Amélie Ferran, a Ph.D. student in the mechanical engineering department, as well as Raul Villanueva, Dennis Yin and Tilboon Elberier, who contributed as undergraduate students studying electrical and computer engineering, and mechanical engineering professors Alberto Aliseda and Sawyer Fuller.
In 2020, a group of researchers unveiled a tool called Face Depixelizer that would take a low-resolution image as an input and, with the help of a generative machine learning model called StyleGAN, produce a high-resolution image in its place. But the model, which was not designed to “fix” the original low-quality image but instead generate an imaginary replacement, had a tendency to predominantly imagine white people — even when the original image depicted someone of another race.
The following year, a group of web developers and accessibility experts signed an open letter urging website owners to avoid using accessibility overlays on their sites. The signatories had become alarmed by the growing reliance on these automated tools, which are marketed under the guise of helping website owners improve the user experience while avoiding potentially costly litigation, when it became apparent that they can actually make the experience worse for people with screen readers — to the point of making a site unusable. To date, nearly 800 individuals have added their names to the letter.
These are just two examples of how technology can have unforeseen, and ostensibly unintended, negative consequences in the real world. Spurred on by these and other cautionary tales, a team of researchers at the Allen School want to assist their colleagues in anticipating and mitigating the consequences of their own work. With support from a five-year institutional transformation grant through the National Science Foundation’s Ethical and Responsible Research (ER2) program, the team hopes their project will usher in a new paradigm in computing-related research not just at the University of Washington, but across the field.
One member of the team, Allen School Ph.D. student Rock Yuren Pang, already had begun thinking about how society increasingly bears the brunt of unintended consequences from new technologies. After enrolling in a graduate-level computer ethics seminar taught by professor Katharina Reinecke, he began to fully appreciate the difficulties researchers face in attempting to understand, let alone mitigate, what those might be.
“Emerging technologies are being used for a growing range of applications that directly impact people’s lives — from how communities are policed, to which job applicants are called for an interview, to what content someone sees online,” Pang said. “As a young Ph.D. student, I thought the question of how we as researchers might think about the downstream impacts of our work to be a really important problem. But I also felt overwhelmed and didn’t know how to even begin tackling it.”
Rock Yuren Pang (left) and Katharina Reinecke
In a new white paper, Pang, Reinecke and Allen School professors Dan Grossman and Tadayoshi Kohno offer a potential starting point. Dubbed PEACE — short for “Proactively Exploring and Addressing Consequences and Ethics” — their proposal offers a vision for empowering researchers to anticipate those consequences “early, often, and across computer science.”
The latter is important, Reinecke notes; while artificial intelligence may dominate the headlines at the moment, these issues extend throughout the field.
“We can’t just point fingers at AI; every technology, no matter how seemingly benign, has the potential to have undesirable impacts,” said Reinecke, the PI on the NSF grant whose research in the Allen School’s Wildlab includes investigating how people relate to technology differently across languages, cultures and abilities. “When we interviewed researchers across multiple subfields, they generally acknowledged the importance of trying to anticipate the consequences of innovation. But to translate that into practice, they need some scaffolding in place.”
To that end, Reinecke and her co-authors propose a holistic approach that would weave such considerations into the school’s teaching and research while making it easier for researchers to tap into existing resources for assistance in anticipating and mitigating undesirable impacts. Two of the resources the team intends to explore as part of the NSF grant, the Tarot Cards of Tech and guided stakeholder analysis, have Seattle roots. The latter is a pillar of Value Sensitive Design, co-conceived by UW iSchool professor and Allen School adjunct faculty member Batya Friedman, that engages individuals or groups who could be directly or indirectly affected by technology. As part of the process, researchers could save the results of their analysis in the form of a PEACE report that could be shared with collaborators on a particular project and updated anytime.
Researchers will also have the option to share their PEACE reports with an ethics board comprising faculty colleagues from across campus with expertise in areas such as law, bioethics, science and technology studies, and gender, women and sexuality studies. Members of this group will act as a sounding board for researchers who wish to follow up on the results of their exploration — and help them think through how they could address any potential unintended consequences they’ve identified in the process.
As with other elements of the proposed PEACE process, consultation with the ethics board would be entirely voluntary.
“We want to give researchers a low-friction, low-stakes mechanism for seeking diverse perspectives on how a technology might be used or misused. This could help surface potential implications we may not think of on our own, as computer scientists, that can inform how we approach our work,” Reinecke said. “We aren’t saying ‘don’t do this risky piece of research.’ What we’re saying is, ‘here’s a way to anticipate how those risks might manifest’ in order to mitigate potential harm.”
Tadayoshi Kohno (left) and Dan Grossman
In his role as co-director of the Allen School’s Security and Privacy Research Lab and the Tech Policy Lab at UW, Kohno has had ample opportunity to analyze the harm that can result when researchers haven’t thought ahead.
“Many times during my career have I wondered if the original researchers or developers could have prevented a problem before deployment,“ said Kohno. “For years, I and my colleagues have encouraged the people who build new technologies to apply a security and privacy mindset from the start rather than having to fix vulnerabilities later, after damage has been done. That’s essentially what we’re suggesting here — we’re asking our colleagues to apply a societal mindset, and to front-load it in the research process instead of relying on hindsight, when it may be too late.”
Grossman is vice director of the Allen School and often teaches the undergraduate computer ethics seminar, which the school began offering to students on a quarterly basis in 2020. He sees an opportunity for the PEACE project to eventually transform computing education and research on a massive scale.
“We are in a position to guide the future leaders of our field toward thinking not only about the technical aspects of computing, important as they are, but also the ethical ones — to train future researchers and technologists how to rigorously consider the potential ramifications socially, politically, environmentally, economically or any combination thereof,” said Grossman. “We need the people who understand their proposed technology to grapple with these issues as well as to learn how to interact with non-technologists, such as public-policy experts, who have complementary expertise.”
The team will deploy and evaluate the PEACE project within the Allen School to start, with plans to extend access to other academic units on campus in later years. Eventually, Pang and his colleagues plan to distill the findings from their evaluation of the UW deployment into detailed design guidelines that can be adapted by other institutions and companies.
“I want to create the go-to place for UW researchers to learn, anticipate and bounce ideas off other researchers about the potential consequences of our work,” Pang said. “But I hope this initiative encourages a broader culture in which computer scientists are unafraid to think critically and openly about these issues. And I believe we can do it in a way that supports, not stifles, innovation.”
Allen School professor Dieter Fox will be honored at the 32nd International Joint Conference on Artificial Intelligence (IJCAI) with the 2023 John McCarthy Award. The award is named for the eponymous scientist, widely regarded as one of the founders of the field of artificial intelligence (AI), and recognizes established researchers who have built up a distinguished track record of research excellence in AI. Fox will receive his award this week and give a presentation on his work at the conference held in Macao, S.A.R.
“Receiving the John McCarthy Award is an incredible honor, and I’m very grateful for the truly outstanding students and collaborators I had the pleasure to work with throughout my career,” Fox said. “I also see this award as a recognition of the importance the AI community places on building intelligent systems that operate in the real world.”
Fox has made a number of key contributions to the fields of AI and robotics, developing powerful machine learning techniques for perception and reasoning, as well as pioneering Bayesian state estimation and the use of depth cameras for robotics and activity recognition.
His research focuses on systems that can interact with their environment in an intelligent manner. Currently, most robots lack the intelligence to perceive and understand changing environments over time. They move objects in a set, programmable way. In a factory, where conditions are tightly controlled, this is a strength. Everywhere else, it’s a problem.
During his time as a Ph.D. student at the University of Bonn, Fox’s work on Markov localization tackled a fundamental problem in robotics and is now considered a watershed moment for the field. Near the start of the 21st century, researchers concentrated on the problem of tracking, giving a robot a map and its initial location. But these robots lacked true autonomy. They were unable to estimate their location and recover from mistakes out in the field — traits, importantly, displayed by a human pathfinder.
Fox and his collaborators developed grid-based and sampling-based Bayes filters to estimate a robot’s position and orientation in a metric model of the environment. Their work produced the first approach that allowed a robot to reorient itself and recover from failure in complex and changing conditions. Fox’s pioneering work in robotics touches virtually every successful robot navigation system, be it indoors, outdoors, in the air or on streets.
In 2009, Fox began a two-year tenure as director of Intel Labs Seattle. There, he and his collaborators developed some of the very first algorithms for depth camera-based 3D mapping, object recognition and detection. Back at the University of Washington, Fox and his colleagues set an additional precedent with a separate study on fine-grained 3D modeling. Called DynamicFusion, the approach was the first to demonstrate how depth cameras could reconstruct moving scenes and objects, such as a person’s head or hands, with impressive resolution in real time. The work won a Best Paper Award from the Conference on Computer Vision and Pattern Recognition (CVPR) in 2015.
For Fox, the McCarthy Award represents another milestone in a journey that began in his youth. As a high school student in Germany, he stumbled upon the book “Gödel, Escher, Bach: An Eternal Golden Braid” by Douglas Hofstadter. The pages, he found, flew by. When he finally closed its cover, he was spellbound.
“From the book, I was fascinated by the ideas behind logic, formal reasoning and AI,” Fox said. “I learned that by studying computer science, I’d be able to continue to have fun investigating these ideas.”
Fox currently shares his time between the UW and NVIDIA, joining the company in 2017. He directs the UW Robotics and State Estimation Laboratory and is senior director of robotics research at NVIDIA. His work at NVIDIA stands at the cutting edge of deep learning for robot manipulation and sim-to-real transfer, bringing us ever closer to the dream of smart robots that are useful in real world settings such as factories, health care and our homes
“Working with Wikipedia data is really exciting because there is such a robust community of people dedicated to improving the platform, including contributors and researchers,” Tsvetkov said. “In contrast, when you work with, for example, social media data, no one is going to go back and rewrite old Facebook posts. But Wikipedia editors revise articles all the time, and prior work has encouraged edit-a-thons and other initiatives for correcting biases on the platform.”
For the continuously evolving site, the research fills crucial content gaps in its data and how it is ultimately used. In the past, related studies focused mainly on one variable, binary gender, and lacked tools to isolate variables of interest, limiting the conclusions that could be drawn. For example, previous research involved comparing the complete sets of biographies for women and men in order to determine how gender influences their portrayals in these bios.
Tsvetkov’s team developed a matching algorithm to build more comprehensive and comparable sets, targeting not just gender but also other variables including race and non-binary gender. For instance, given a set of articles about women, the algorithm builds a comparison set about men that matches the initial set on as many attributes as possible (occupation, age, nationality, etc.), except the target one (gender).
The researchers could then compare statistics and language in those two sets of articles to conduct more controlled analyses of bias along a target dimension, such as gender or race. They also used statistical visualization methods to assess the quality of the matchings, supporting quantitative results with qualitative checks.
To examine gender bias, instead of comparing all articles about women with all articles about men, the team’s algorithm constructs matched sets: For each article about a woman, it identifies the most similar article about a man. Analyzing these matched sets serves to isolate gender from other correlating variables.
As a result, the researchers saw a significant difference when analyzing articles with and without their matching approach. When the approach was implemented, they found data confounds decreased — a boon for better evaluating bias in the future.
From left: Anjalie Field, Chan Young Park and Kevin Z. Lin
“We did a lot of data curation to be able to include analyses of racial bias, non-binary genders, and intersected race and gender dimensions,” said lead author Anjalie Field, a professor at Johns Hopkins University who earned her Ph.D. from Carnegie Mellon University working with Tsvetkov. “While our data and analysis focus on gender and race, our method is generalizable to other dimensions.”
Future studies could further build upon the team’s methodology, targeting biases other than gender or race. The researchers also pointed to shifting the focus from the data sets to the natural language processing (NLP) models that are deployed on them.
“As most of our team are NLP researchers, we’re also very interested in how Wikipedia is a common data source for training NLP models,” Tsvetkov said. “We can assume that any biases on Wikipedia are liable to be absorbed or even amplified by models trained on the platform.”
The study’s co-authors also included Chan Young Park, a visiting Ph.D. student from Carnegie Mellon University, and Kevin Z. Lin, an incoming professor in the University of Washington’s Department of Biostatistics. Lin earned his doctorate from Carnegie Mellon University and was a postdoc at the University of Pennsylvania when the study was published.
Ilharco, a fourth-year Ph.D. student in the Allen School’s H2Lab, is advised by professors Ali Farhadi and Hannaneh Hajishirzi. His research focuses on advancing large multimodal models as reliable foundations in AI.
“I believe the next-generation of models will push existing boundaries through more flexible interfaces,” Ilharco said. “There is much progress to be made towards that vision, both in training algorithms and model architectures, and in understanding how to design better datasets to train the models. I hope my research will continue to help in all of these directions.”
During his fellowship, Ilharco said he hopes to continue his work in building more reliable machine learning systems. While models such as GPT-4, Flamingo and CLIP have demonstrated impressive versatility across applications, there is still room for growth. In the past decade, machine learning systems have become highly capable, particularly when performing specific tasks such as recognizing objects in images or distilling a piece of text. Yet their abilities can advance further, Ilharco said, with the end goal being a single model that can be deployed across a wider range of applications.
To meet this challenge, Ilharco is targeting dataset design. A recent project, DataComp, acts as a benchmark for designing multimodal datasets. Ilharco was part of the research team that found smaller, more stringently filtered datasets can lead to models that generalize better than larger, noisier datasets. In their paper, the researchers discovered that the DataComp workflow led to better training sets overall.
“DataComp is designed to put research on datasets on rigorous empirical foundations, drawing attention to this understudied research area,” Ilharco said. “The goal is that it leads to the next generation of multimodal datasets.”
Another project introduced a framework for editing neural networks and appeared at the 11th International Conference on Learning Representations (ICLR 2023) this spring. Ilharco co-authored the paper that investigated how the behavior of a trained model could be influenced for the better using a technique called task arithmetic. In one example, the team showed how the model could produce less toxic generations when negating task vectors. Conversely, adding task vectors improved a model’s performance on multiple tasks simultaneously as well as on a single task. Ilharco and his collaborators also found that combining task vectors into task analogies boosted performance for domains or subpopulations in data-scarce environments.
Their findings allow users to more easily manipulate a model, expediting the editing process. Because the arithmetic operations over task vectors involve only adding or subtracting model weights, they’re more efficient to compute compared to alternatives. Additionally, they result in a single model of the same size, incurring no extra inference cost.
“We show how to control the behavior of a trained model — for example, making the model produce less toxic generations or learning a new task — by operating directly in the weight space of the model,” Ilharco said. “With this technique, editing models is simple, fast and effective.”
For Ilharco, the next wave of multimodal models is fast-approaching. He wants to be at the center of it.
“I hope to be a part of this journey,” he said.
Sharma, also a fourth-year Ph.D. student, is advised by professor Tim Althoff in the Allen School’s Behavioral Data Science Lab. He studies how AI can support mental health and well-being.
“I’m excited to be selected for this fellowship which will help me further my research on human-AI collaboration,” he said. “AI systems interacting with humans must accommodate human behaviors and preferences, and ensure mutual effectiveness and productivity. To this end, I am excited to pursue my efforts in making these systems more personalized.”
Sharma’s long-term goal focuses on developing AI systems that empower people in real-world tasks. His research includes work on AI to assist peer supporters to increase empathy in their communications with people seeking mental health support and exploring how AI can help users regulate negative emotions and intrusive thoughts.
Both put the user — the human being — at the center.
“Effectively supporting humans necessitates personalization,” Sharma said. “Current AI systems tend to provide generalized support, lacking the ability to deliver experiences tailored to the specific needs of end-users. There is a need to put increased emphasis on developing AI-based interventions that provide personalized experiences to support human well-being.”
Sharma’s work with mental health experts and computer scientists was among the earliest efforts to demonstrate how AI and natural language processing-based methods could provide real-time feedback to users in making their conversations more empathetic.
At The Web Conference 2021, he and his co-authors won a Best Paper Award for their work on PARTNER, a deep reinforcement learning agent that learns to edit text to increase “the empathy quotient” in a conversation. In testing PARTNER, they found that using the agent increased empathy by 20% overall and by 39% for those struggling to engage empathetically with their conversational partners.
“PARTNER learns to reverse-engineer empathy rewritings by initially automating the removal of empathic elements from text and subsequently reintroducing them,” Sharma said. “Also, it leverages rewards powered by a new automatic empathy measurement based on psychological theory.”
Earlier this year, Sharma was also lead author on a paper introducing HAILEY, an AI agent that facilitates increased empathy in online mental health support conversations. The agent assists peer supporters who are not trained therapists by providing timely feedback on how to express empathy more effectively in their responses to support seekers in a text-based chat. HAILEY built upon Sharma’s work with PARTNER.
With a focus on people and process, Sharma sees how his research area can continue to grow. He said he hopes to advance AI’s ability to personalize to the user, while also remaining safe and secure.
“Utilizing my experience in designing and evaluating human-centered AI systems for well-being, I will investigate how such systems can learn from and adapt to people’s contexts over time,” Sharma said. “I’ve always been fascinated by technological efforts that support our lives and well-being.” Read more →
An evening view of Toronto, where the 61st Annual Meeting of the Association for Computational Linguistics (ACL) took place last month. Photo by Lianhao Qu on Unsplash.
Allen School researchers took home multiple Best Paper and Outstanding Paper Awards from the 61st Annual Meeting of the Association for Computational Linguistics (ACL) held in Toronto last month. Their research spanned a number of projects aimed at enhancing the performance and impact of natural language models, including how artificial intelligence (AI) processes humor, the impact of built-in political biases on model performance, AI-assisted cognitive reframing to support mental health, identifying “WEIRD” design biases in datasets and how to imbue language models with theory of mind capabilities. Read more about their contributions below.
Best Paper Awards
Do Androids Laugh at Electric Sheep? Humor ‘Understanding’ Benchmarks from The New Yorker Caption Contest
The tasks included matching jokes to cartoons, identifying a winning caption and explaining why an image-caption combination was funny. For an AI model, it’s no joke. Humor, the authors point out, contains “playful allusions” to human experience and culture. Its inherent subjectivity makes it difficult to generalize, let alone explain altogether.
“Our study revealed a gap still exists between AI and humans in ‘understanding’ humor,” said Choi, who holds the Wissner-Slivka Chair at the Allen School and is also senior research manager for the Allen Institute for AI’s MOSAIC project. “In each task, the models’ explanations lagged behind those written by people.”
Clockwise, from top left: Jack Hessel, Jena D. Hwang, Robert Mankoff and Ana Marasovic; not pictured: Lillian Lee
The team applied both multimodal and language-only models to the caption data. Compared to human performance, the best multimodal models scored 30 accuracy points worse on the matching task.
Even the strongest explanation model, GPT-4, fell behind. In more than two-thirds of cases, human-authored explanations were preferred head-to-head over the best machine-authored counterparts.
Future studies could focus on other publications or sources. The New Yorker Cartoon Caption Contest represents only a “narrow slice” of humor, the authors note, one that caters to a specific audience. New research could also explore generating humorous captions by operationalizing feedback produced by the team’s matching and ranking models.
To do this, they developed a framework based on political science literature for measuring bias found in pretrained models. Then they analyzed how these biases affected the models’ performance in downstream social-oriented tasks, such as measuring their ability to recognize hate speech and misinformation.
They found that bias and language are difficult to separate. Both non-toxic and non-malicious data, they note, can cause biases and unfairness in NLP tasks. If political opinions are filtered from training data, however, then questions arise concerning censorship and exclusion from political participation. Neither is an ideal scenario.
“Ultimately, this means that no language model can be entirely free from social biases,” Tsvetkov said. “Our study underscores the need to find new technical and policy approaches to deal with model unfairness.”
The consistency of the results surprised the team. Using data from Reddit and several news sources, the researchers found that left-leaning and right-leaning models acted according to form. The left-leaning models were better at detecting hate speech towards minority groups, while worse at detecting hate speech towards majority groups. The pattern was reversed for right-leaning models.
In evaluating data from two time periods — before and after the 2016 U.S. presidential election — they also discovered a stark difference between the levels of political polarization and its attendant effect on the models’ behavior. With more polarization comes more bias in language models.
Future studies could focus on getting an even more fine-grained picture of the effect of political bias on NLP models. For example, the authors note that being liberal on one issue does not preclude being conservative on another.
“There’s no fairness without awareness,” Tsvetkov said. “In order to develop ethical and equitable technologies, we need to take into account the full complexity of language, including understanding people’s intents and presuppositions.”
The study’s co-authors also included Chan Young Park, a visiting Ph.D. student from Carnegie Mellon University, and Yuhan Liu, an undergraduate at Xi’an Jiaotong University.
Outstanding Paper Awards
Cognitive Reframing of Negative Thoughts through Human-Language Model Interaction
Clockwise, from top left: Tim Althoff, Ashish Sharma, Inna Lin and David Wadden
Working with experts at Mental Health America, the team developed a model that generates reframed thoughts to support the user. For example, the model could produce reframes that were specific, empathic or actionable — all ingredients for a “high-quality reframe.” The study was the first to demonstrate that these all make for better reframes, Althoff said, and the team illustrated this with gold standard randomized experiments and at scale.
The research has already seen real-world impact. Since its introduction late last year, the team’s reframing tool has had more than 60,000 users.
“The findings from this study were able to inform psychological theory — what makes a reframe particularly effective?” Sharma said. “Engaging with real users helped us assess what types of reframes people prefer and what types of reframes are considered relatable, helpful and memorable.”
Those “high-quality reframes” could be particularly helpful for those who lack access to traditional therapy. The team pointed out several obstacles to care, including clinician shortages, lack of insurance coverage and stigmas surrounding mental health, that served as motivations for the study.
Clockwise, from top left: Kevin Rushton, Khendra G. Lucas, Theresa Nguyen and Adam S. Miner
The model can also be integrated into existing therapy workflows, Sharma added, helping both clients and therapists in the process. For example, therapists often assign “homework” to clients, asking them to practice cognitive reframing, a technique by which a person can picture a negative thought through a different, more balanced perspective.
But many clients report having difficulty in applying those techniques following a session. Sharma and Althoff said the team’s reframing tool can provide support in those moments.
“It turns out that often our thoughts are so deep-rooted, automatic and emotionally triggering that it can be difficult to reframe thoughts on our own,” Althoff said. “This kind of research not only helps us improve our intervention itself, but could also inform how clinicians teach these skills to their clients.”
Minding Language Models’ (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker
Top row, from left: Melanie Sclar, Peter West and Yejin Choi; bottom row, from left: Yulia Tsvetkov, Sachin Kumar and Alane Suhr
Choi and Tsvetkov worked with Allen School Ph.D. students Melanie Sclar and Peter West and collaborators on SymbolicToM, an algorithm that improves large language models’ abilities to reason about the mental states of other people that earned the team an Outstanding Paper Award. The team also won the Outstanding Paper Award at the ToM Workshop at the 2023 International Conference on Machine Learning (ICML) for this work.
Theory of mind (ToM), or the ability to reason about others’ thoughts and intentions, is a key part of human intelligence. But today’s AI models lack ToM capabilities out of the box. Prior efforts at integrating ToM into language models required training, with existing reading comprehension datasets used for ToM reasoning remaining too simplistic and lacking diversity.
“This implied that models trained solely on this data would not perform ToM reasoning,” Sclar said, “and rather only mimic these skills for the simplistic data they were trained on.”
Enter SymbolicToM. Without requiring ToM-specific training, the decoding-time algorithm takes a divide-and-conquer approach. SymbolicToM splits a given problem into subtasks, Sclar said, solving each with off-the-shelf large language models. The result is a better, more robust model.
“We knew from the get-go that our approach needed to focus on having good generalization capabilities, and thus would benefit from not requiring training,” Sclar said. “SymbolicToM is to the best of our knowledge the first method for theory of mind reasoning in natural language processing that does not require any specific training whatsoever.”
The team tasked SymbolicToM with answering reading comprehension questions based on a story featuring multiple characters. They tracked each character’s beliefs, their estimation of others’ beliefs and higher-order levels of reasoning through graphical representations. In doing so, the models could reason with more precision and interpretability.
“Our method in particular is not focused on training neural language models, but quite the opposite: given that we have imperfect language models trained with other objectives in mind, how can we leverage them to dramatically improve theory of mind performance?” Tsvetkov said. “This is key because data with explicit theory of mind interactions are scarce, and thus training directly is not a viable option.”
Sclar pointed to potential avenues for future applications, including education and business. For example, AI agents with ToM reasoning skills could assist in tutoring applications, providing a deeper understanding of students’ knowledge gaps and designing tests based on their mental model of each student.
Another instance involves negotiation strategy. If AI agents can intuit what each party hopes to achieve and how much they value certain aspects of a deal, Sclar said, they can provide support in reaching a fair consensus.
“Imbuing neural language models with ToM capabilities would improve these models’ potential on a wide range of applications,” Sclar said, “as well their understanding of human interactions.”
The study’s authors also included visiting Ph.D. student Sachin Kumar of Carnegie Mellon University and professor Alane Suhr of the University of California, Berkeley.
NLPositionality: Characterizing Design Biases of Datasets and Models
Top row, from left: Katharina Reinecke, Sebastin Santy and Ronan Le Bras; bottom row, from left: Jenny Liang and Maarten Sap
“Language is a social phenomenon,” Santy said. “Many in the NLP field have noticed how certain datasets and models don’t work for different populations, so we felt it was the right time to conduct this large-scale study given these gaps and with the right kind of platform.”
That platform, LabintheWild, provides more reliable data from a more diverse set of users. Reinecke, one of the platform’s co-founders and director of the Wildlab at the Allen School, noted that as opposed to Mechanical Turk, a popular paid crowdsourcing site, LabintheWild collects results from a greater pool of countries.
With LabintheWild, the personal is emphasized over the pecuniary. After completing a study, users can see personalized feedback and compare their results with others’ performance on the platform.
This feedback is eminently shareable, Reinecke added, increasing its reach. The researchers’ recent study collected 16,299 annotations from 87 countries — one of the first NLP studies to reach that scale. They applied their framework, called NLPositionality, to LabintheWild’s vast participant pool, implementing users’ annotations from existing datasets and models for two tasks: social acceptability and hate speech detection.
Their findings aligned with Reinecke’s previous work, which shows that technology is often designed for people who are Western, Educated, Industrialized, Rich and Democratic, or “WEIRD.”
“WEIRD bias is well-known in psychology and our hypothesis was that we might find similar results in AI as well, given most of the recent advances make use of mostly English data from the internet and filter for ‘high-quality,’ ” said Reinecke, who holds the Paul G. Allen Career Development Professorship. “While we had a feeling that there would be Western bias because of how most of the datasets are curated in the Western Hemisphere, we did not expect it to be this pronounced.”
The study’s co-authors also included Allen School alumni Maarten Sap (Ph.D., ‘21) and Jenny Liang (B.S., ‘21), now professor and Ph.D. student, respectively, at Carnegie Mellon University, and Ronan Le Bras of AI2. Read more →
Single-cell datasets are transforming biomedical research aimed at understanding the mechanisms and treatment of diseases such as acute myeloid leukemia (AML) pictured above. A new deep learning framework called ContrastiveVI enables researchers to explore single-cell data in finer detail by applying contrastive analysis, which is capable of revealing subtle effects that previous computational methods might miss. Credit: National Cancer Institute
In the days before single-cell RNA sequencing, researchers investigating the mechanisms and treatment of disease had to make do with running experiments on bulk cell profiles created by taking tissue samples and grinding them up, “sort of like putting them in a blender,” in the words of Allen School Ph.D. student Ethan Weinberger.
That milkshake may have brought all the biomedical scientists to the lab, but the bulk sequencing technique limited them to studying aggregations of populations of cells, with no way to distinguish among individual cell types. Nowadays, researchers can take measurements at the level of individual cells, enabling the exploration of such finer-grained distinctions and advancing our understanding of various biological functions. But without the right computational tools, even single-cell datasets can yield distinctions without a difference.
Weinberger is a member of the Allen School’s AIMS Lab, where he works with fellow Ph.D. student Chris Lin and professor Su-In Lee to leverage advances in artificial intelligence to help scientists get the most out of these increasingly robust datasets. In a paper published this week in Nature Methods, the team introduced ContrastiveVI, a deep learning framework for applying a powerful technique called contrastive analysis, or CA, to single-cell datasets to disentangle variations in the target, or treatment, cells from those shared between target and control cells when running experiments.
“Scientists want to investigate questions like ‘How does perturbing this particular gene affect its response to a pathogen?’ or ‘What happens when I hit a diseased cell with such-and-such a drug?’,“ explained Weinberger. “To do that, they need to be able to isolate the variations in the cell data caused by that perturbation or that drug from those that are shared with a control dataset. But existing models can’t separate those out, which might lead someone to draw erroneous conclusions from the data. ContrastiveVI solves that problem.”
“There are so many contexts in which scientists would want to do this”: Ethan Weinberger (left) and Chris Lin
CA has proven effective at this type of isolation in other contexts, but its utility in relation to single-cell datasets has so far been limited. That’s because existing computational models for analyzing single-cell data mostly rely on a single set of latent variables to model all variations in the data, effectively lumping them all together and precluding the ability to perform CA.
ContrastiveVI is the first deep learning model designed for performing CA on single-cell data. Unlike other approaches, the ContrastiveVI model explicitly separates latent variables into two categories, each with their own encoding function: shared variables, or those that are found in both the target and control cells, and salient variables, which are found exclusively among the target cells.
It is that second category that will excite scientists testing potential cancer drugs or analyzing the role of gene expression in the body’s response to disease.
“ContrastiveVI effectively distinguishes the factors that are salient — that is, relevant — to an experiment from confounding factors. This enables us to capture variations that are unique to the treated cells,” said Lee, senior author of the paper and holder of the Paul G. Allen Career Development Professorship in the Allen School. “ContrastiveVI will reveal tiny but important variations in the data that may be obscured by other models.”
Lee and her co-authors validated ContrastiveVI using real-world datasets with previously verified results as their ground truth. In one experiment, the researchers applied ContrastiveVI to a target dataset of measurements taken from two dozen cancer cell lines treated with idasanutlin. This small-molecule compound has shown therapeutic potential owing to its activation of a tumor-suppressing protein in wild type — that is, unmutated — TP53 genes. The team used ContrastiveVI to analyze data on both wild type and mutated TP53 cell lines, which are non-responsive to idasanutlin, using a background dataset from the same cell lines treated with a different compound, dimethyl sulfoxide, as the control.
“A good result — one that agreed with prior knowledge — would show separation by cell line accompanied by increased mixing of treatment and control cells in the shared latent space, but mixing across mutant cell lines with clear separation based on mutation status in the salient latent space,” said Lin, co-lead author of the paper with Weinberger. “And that is exactly what we observed. In addition, our model indicated a separation between wild-type cell lines in the salient space that suggested a differential response to treatment, which spurred us to run additional analyses to identify the specific genes that contribute to those variations.”
A comparison of ContrastiveVI’s shared and salient latent spaces in the idasanutlin experiment. Top row: Cancer cells in the shared latent space separate according to cell line and whether they are wild type or have the TP53 mutation, with treatment and control cells mixed within each cluster. Bottom row: Cells separate in the salient latent space based on whether they are wild-type or mutant, while displaying increased mixing across the mutant cell lines. Further analysis revealed four genes highlighted by ContrastiveVI that contributed to a differential treatment response observed in the wild-type cells. Credit: Nature Methods
Such findings, which could build upon prior knowledge and lead scientists to new hypotheses, is precisely the sort of progress Lin and his colleagues hope their model will support. In another demonstration of ContrastiveVI’s potential, the researchers applied the model to a dataset drawn from intestinal epithelial cells of mice displaying variations in gene expression due to infection with the bacteria Salmonella or the parasite H. polygyrus (H. poly), a type of roundworm, using healthy cells as the control. Once again, the model aligned with expectations by separating along cell type and mixing across infections in the shared latent space, while largely mixing across cell types and separating by pathogen in the salient latent space.
Like the cancer cell example, the pathogen infection experiment also yielded unexpected patterns that prompted the team to analyze further. These patterns included differences in the upregulation of multiple genes between H. poly–infected tuft cells and other infected cell types that may have been masked in prior experiments — and could point to a distinctive role in the body’s immune response.
Su-In Lee
The researchers also explored how the model could be adapted to isolate variations in multimodal single-cell datasets, such as a combination of RNA and surface protein expression data in CRISPR-perturbed cells. They layered their CA modeling techniques onto TotalVI, a deep generative model developed to analyze joint RNA-protein datasets, to create TotalContrastiveVI. In a series of experiments, they showed how their extended model could be used to identify clusters of cells in the salient latent space and apply downstream analysis to identify patterns that warranted further investigation.
TotalContrastiveVI may be a proof of concept, but the underlying model is no mere demonstration project. The team designed ContrastiveVI to make it easy for researchers to integrate the tool into existing workflows.
“Our software is essentially plug and play,” noted Lin. “Computational biologists can deploy ContrastiveVI right now in conjunction with standard tools in the field such as Scanpy to begin exploring single-cell datasets in greater detail than they could before.”
Those details could yield new hypotheses that could, in turn, lead to new biomedical breakthroughs.
“There are so many contexts in which scientists would want to do this,” said Weinberger. “People were already excited by the potential of single-cell datasets. With ContrastiveVI, they can unlock even more insights and expand our knowledge of the mechanisms and treatment of disease.
“To borrow a popular metaphor in biomedical circles: before, we had a smoothie; now we can zoom in on each part of the corresponding fruit salad.”
