Allen School News

The University of Washington and Meta are launching a new partnership today that will support graduate student research while providing opportunities to collaborate with industry-leading scientists and engineers. The Meta AI Mentorship Program is designed to enable Allen School Ph.D. students who are interested in artificial intelligence, machine learning or natural language processing to advance their dissertation research under the guidance of both their faculty advisor and researchers in Meta’s Seattle office by collaborating on projects of mutual interest.

“The Meta AI Mentorship Program encourages students to tackle ambitious research projects that will enable them to make progress toward their degree while enjoying the opportunity to work side by side with industry researchers,” said Luke Zettlemoyer, a professor in the Allen School’s Natural Language Processing group and a research director at Meta AI. “I think this program will yield exciting results and new lines of inquiry in our field. I’m thrilled to be able to bring this program to our students and colleagues.”

There are a variety of potential synergies between UW and Meta research that students and their mentors might choose to explore, including but not limited to representation learning, natural language generation, machine translation, question answering, semantic parsing and large-scale optimization. In keeping with the principles of academic freedom and open research, participating students will not only be permitted, but encouraged to publish the results of their work with their advisors and Meta mentors — and to incorporate that work into their dissertation research.

“Our goal is to encourage students to think big and to make progress on important research questions, so we intentionally designed the mentorship program with open research in mind,” Zettlemoyer explained. “We want to ensure that students not only have a great experience, but also come away with results they can build off of to advance their careers.”

The program is currently open to Allen School Ph.D. students who are in their second year or later, with the potential to expand to students in other units on campus in subsequent years. Participants in the inaugural cohort, which will number up to five students, will receive sponsored research support that will cover their tuition along with a stipend. In addition, the students will spend eight paid hours per week working with researchers at Meta’s Seattle office during the academic year, and 40 hours per week paid during the summer. The expected duration of each student’s participation will be one year, with the option to extend it for up to an additional year.

“This program offers students the best of both worlds: the excitement of discovery under the guidance of their faculty advisor, and the opportunity to work alongside industry leaders — all of whom are at the forefront of their respective fields,” said Magdalena Balazinska, professor and director of the Allen School. “Our school has already enjoyed the benefits of multiple collaborations with Meta. I look forward to seeing how this latest partnership drives innovation and enriches our students’ early research careers.”

The Allen School and Meta have enjoyed a long and fruitful partnership leading up to today’s announcement, including Meta researchers who are affiliate professors in the Allen School, Allen School professors who have deep engagement with Meta, student fellowships and internships, and Meta’s support for the UW Reality Lab focused on another area of mutual interest: augmented and virtual reality.

The UW–Meta AI Mentorship program is part of a portfolio of employment-based, collaborative, open research programs the company has launched around the country since 2020. The others are with Carnegie Mellon University and New York University.

Interested students have until June 17 to apply to be part of the first cohort of the Meta AIM Program at the UW, which will get underway in October. Read more →

Eight years ago, Allen School professor Jennifer Mankoff and a group of like-minded researchers who cared about, or needed, accessibility put their heads together after coming to a realization about SIGCHI, the Association for Computing Machinery’s Special Interest Group on Computer-Human Interaction. While a growing swath of researchers in the community had begun to focus on the design and evaluation of technologies for diverse users, including those with disabilities, Mankoff and her colleagues noted that the venues for showcasing that work — including the group’s flagship annual conference, CHI — were not themselves accessible to many of these same audiences. And thus, AccessSIGCHI was born.

“Around the time we launched, nearly one-tenth of the papers presented at CHI were related to accessibility or disability in some way, yet only about 20% of conferences included accessibility support,” recalled Mankoff, who holds the Richard E. Ladner Professorship in the Allen School where she directs the Make4All Group. “I and Jennifer Rode, who founded AccessSIGCHI, asked ourselves, ‘how can we ensure that our community’s publications and events and procedures make accessibility a priority?’ It’s a necessary conversation, if not always a comfortable one. And it’s ongoing.”

In 2015, Mankoff and her colleagues in AccessSIGCHI (originally known as the SIGCHI Accessibility Community) released a seminal report documenting the state of accessibility within the SIGCHI community and laying out a vision for the future. In it, the authors noted, “​SIGCHI​ ​can​ ​attract​ ​new​ ​members,​ ​and​ ​make​ ​current​ ​members feel​ ​welcome​ ​by​ ​making​ ​its​ ​events​ ​and​ ​resources​ ​more​ ​inclusive.​ ​This​ ​in​ ​turn​ ​will​ ​enrich SIGCHI,​ ​and​ ​help​ ​it​ ​to​ ​live​ ​up​ ​to​ ​the​ ​ideal​ ​of​ ​inclusiveness​ ​central​ ​to​ ​the​ ​concept​ ​of user-centered​ ​design.​” The group has since released three subsequent reports on a biennial basis assessing current conditions and issuing recommendations for improvement. This month, in recognition of Mankoff’s efforts to make not only technologies but also the community that creates them more accessible to members with diverse needs and experiences, that same community honored her with its 2022 SIGCHI Social Impact Award.

“Jen exemplifies the meaning of the SIGCHI Social Impact Award,” said colleague and previous award recipient Richard Ladner, professor emeritus in the Allen School. “Her career has been driven by her desire for social impact in multiple ways, including but not limited to improving the lives of people with disabilities through technology, improving the lives of everyone through technology to support environmental sustainability, and improving the accessibility of CHI sponsored conferences for the benefit of the entire CHI community.”

Mankoff is known for her holistic approach to research, including an understanding of how structural factors can impede people’s access to technologies in addition to the technologies themselves. She and her collaborators at Carnegie Mellon University, where she was a faculty member before she joined the Allen School in 2017, helped to document and advance the potential for consumer-grade fabrication tools and techniques, combined with a wider variety of materials, to support “medical making.” 

Says graduate student Megan Hofmann, who led much of this work and starts as faculty at Northeastern University in the fall, “At the onset of the COVID pandemic, medical making suddenly took on a new urgency. Jen encouraged me to engage with this through my contacts in Colorado, my home state, while she worked to support efforts at the University of Washington.” This work resulted in the delivery of hundreds of PPE (personal protective equipment) devices in Washington and over 100,000 PPE devices in Colorado. It also led to a series of conference publications, including an examination of the role of medical makers in resolving acute and chronic shortages of PPE while upholding standards of clinical safety, and recommendations for building a more robust infrastructure to support medical making communities in meeting local needs — both of which earned honorable mentions at CHI 2021.

Another area in which Mankoff has been a leader is in the development of tools and materials for fabrication that are accessible to — and relevant to — people with disabilities. Her work in this area has ranged from machine knitting to 3D printing, and helped to establish the importance of tools that allow re-use and design by domain experts who are not also fabrication experts.

Mankoff has also been instrumental in bridging the gap between disability studies and assistive technology research. Her work to apply the theoretical underpinnings of the former to promote a more inclusive and self-advocating model for the latter earned the 2021 SIGACCESS ASSETS Impact Award from the ACM Special Interest Group on Accessible Computing for a paper 10 years or older with lasting impact. More recently, Mankoff has advanced mixed-method approaches, including interviews and biometric data, for understanding university students’ mental health and well-being on a large scale via the cross-disciplinary UW EXP study, such as the impact of discrimination, pandemic-driven remote learning, and accessibility innovations and challenges of the same.

In an effort to promote cross-campus collaboration and community partnerships to advance accessibility research, education and translation, Mankoff was one of the driving forces behind the establishment of the UW Center for Research and Education on Accessible Technology and Experiences (CREATE). She currently serves as founding co-director of CREATE with her colleague Jacob O. Wobbrock, a professor in the UW’s Information School and adjunct professor in the Allen School. Seeded with a $3 million from Microsoft, CREATE’s mission is “to make technology accessible and to make the world accessible through technology.”

Beyond her contributions in accessibility research spanning almost three decades, Mankoff helped to steer the SIGCHI community in new directions through her research linking sustainability issues with HCI. It was a connection that few had made before she began co-organizing conference workshops on the topic in 2007; in the process, she and Ph.D. student Tawanna Dillahunt, now a tenured professor at the University of Michigan, also set a new standard for HCI and sustainability research by deliberately engaging underserved populations as part of this work.

“Jen was one of the first in HCI, and the first among sustainability researchers, to specifically seek out people from low-income communities to help uncover new concerns such as how sustainability plays out in the landlord-tenant relationship,” Ladner noted. “Jen’s work emphasized the importance of working with diverse populations at the intersection HCI and sustainability, and her research with underserved communities continues to this day.” 

While she has never shied away from pushing her colleagues to take into account perspectives and experiences different from their own, Mankoff’s activism recently took a more personal turn. Last fall, she suddenly lost her voice. With her primary means of sharing information cut off, she resorted to her rudimentary knowledge of American Sign Language — something she and her son had begun learning by chance just two months prior — and tools such as a portable whiteboard. Mankoff, whose first experience with inaccessibility came during graduate school due to a repetitive strain injury before she was later diagnosed with Lyme disease while building her research career, was already keenly aware that the world is not organized for people with disabilities. But suddenly, she understood the ableism and artificial obstacles that people face when going about their daily lives in a whole new way:

“In essence, this is the first disability experience I’ve had that is defined entirely by the numerous barriers put up by others,” Mankoff wrote in a candid blog post about the episode. “I will admit to being surprised by the sheer amount of discrimination I’ve encountered in a single month.” She goes on to describe gatekeeping practices that threatened her ability to obtain services, ableist jokes and disbelieving and dismissive health care providers. Mankoff ends by saying “I suspect this is mostly about being in a new situation. But I’d argue that is exactly when compassion and support are most needed!”

Mankoff has since recovered her voice; it would surprise no one who knows and works with her to learn that this recent experience has galvanized her intent to use it to speak up even more forcefully for those whose voices, too often, still go unheard.

“Jennifer has contributed significantly to accessibility through both her research and her activism, from creating technologies and tools that empower others to ensuring conferences are inclusive and accessible to all,” said Wobbrock. “She is a highly deserving recipient of the SIGCHI Social Impact Award, and I am proud to call her a colleague, collaborator and friend.”

Mankoff was formally recognized during the CHI 2022 conference that took place as a hybrid-onsite event earlier this month. She previously was elected to the CHI Academy in 2019.

Congratulations, Jen! Read more →

One is defying traditional gender-based norms through her choice of career path while making it easier for her peers to do the same. Another was motivated to turn tragedy into triumph and dedicate his research to making the world more accessible to people with disabilities. And still others are working to unlock the mysteries of disease, empowering communities through data, and questioning how we can ensure that emerging technologies are designed to serve diverse people and communities.

What all five have in common is that they are Allen School students who are seeking ways to apply their education to benefit others — and the distinction of being named to the 2022 class of the Husky 100. This annual honor recognizes students across the three University of Washington campuses who are making the most of their Husky experience. Read on to learn more about how these students exemplify the UW’s ethos of “we > me” while demonstrating how computing can contribute to a more equitable and inclusive society.

Nuria Alina Chandra

Nuria Alina Chandra is making the most of her Husky experience in part by helping others on their journey through her roles as an Undergraduate Research Leader and Honors Peer Mentor. Currently pursuing a major in computer science with a minor in global health, she has taken what could be described as the scenic route. The Olympia, Washington native entered the UW with a clearly defined plan: study biochemistry, go to graduate school, devote her career to investigating the causes of and cures for chronic disease. Before the end of her first year, during which she earned the distinction of being named a Presidential Freshman Medalist, she began to question her direction.

It was around the same time that Chandra enrolled in a global health class, where she learned how demographic and socioeconomic factors, as well as physical factors, influence health. That revelation, combined with the growing urgency to reckon with structural racism in the United States following the murder of George Floyd by police in Minneapolis, inspired Chandra to dig deeper into data about morbidity and mortality. Her findings caused her to reevaluate the path she had set for herself.

“As my time as a Husky progressed, I realized that my true path is far from a clean line, but is, in fact, a branching and twisting tree,” she said.

Those twists included enrolling in a computer science course on a whim, which led Chandra to realize that what drew her to biochemistry was the quantitative and algorithmic thinking involved. As Chandra discovered her passion for computer science, she also became excited by its potential impact on the world’s most complex problems. She found an opportunity to combine computation with her burgeoning interest in the social determinants of health as an undergraduate researcher at Seattle Children’s Hospital. As part of a project led by Dr. Jennifer Rabbitts investigating the development of chronic and acute pediatric pain, Chandra applied her newfound computational skills to analyzing clinical trial data in combination with data on family income and race. She also discovered the power of storytelling to advance both science and social justice while writing for the student-run publications Voyage and The Daily.

Chandra recently branched out again, this time into research focused on the development of new machine learning and statistical methods for studying human health and disease under the guidance of Allen School professor Sara Mostafavi. This and her previous work with Dr. Rabbitts has helped Chandra to realize that she needn’t abandon her original goal; she just discovered a new and exciting way to reach her destination. 

“I will research the root causes of chronic diseases, as I planned when I started college,” Chandra said. “However, the nature of my planned research has been reshaped into something far different than what I ever imagined thanks to new ways of thinking developed in my global health and computer science classes.”

Hayoung Jung

Hayoung Jung’s Husky experience started off tinged with doubt. Having served as a volunteer with the American Red Cross since middle school, he decided before even applying to the UW that what he wanted most out of a career was to have a positive impact on the world. Hailing from Vancouver, Washington, Jung was thrilled to earn direct admission to the Allen School as a freshman; shortly after he arrived on campus, however, his confidence in his choice of major began to waver.

“I was haunted as a first-year student by my inability to reconcile technology’s omnipresence in society with its seeming disregard for ethical responsibilities,” Jung explained. “I had never programmed before college — my burgeoning interest in computer science was mainly fueled by academic curiosity — and I began to doubt my future in a field so often stereotyped as being soulless and materialistic.”

That doubt quickly dissipated once Jung enrolled in the Allen School’s first-year seminar and attended lectures on “Computing for Social Good.” It was there that Jung discovered how computing could be used to benefit diverse people and communities, such as robotics to assist people with motor disabilities; the seminar also offered an unvarnished look at how developer biases can infuse the technologies they create, potentially leading to harm. The course convinced Jung of the importance of diversity in the technology industry — and of sticking with his choice of major. 

“The seminar showed me that computer science is more than coding: computer science is inherently social,” Jung said. “I left ‘Computing for Social Good’ with a new perspective on the influence of technology on the world and a refreshed confidence in my ability to forge my own path at UW.”

That path includes a second major in political science. He combined the two to great effect during the pandemic, when he founded Polling and Open Data Initiative at the University of Washington (PODUW), a registered student organization dedicated to informing communities and improving policy through polling and data science. After noticing some of his fellow students were unequally impacted by the move to remote learning, Jung spearheaded an initiative to explore the impact more broadly on diverse communities of students at UW to inform future practices. To that end, PODUW collaborated with Student Regent Kristina Pogosian to poll more than 3,700 students across the university’s Seattle, Bothell and Tacoma campuses and prepare a report to the UW Board of Regents.

The pandemic also made clear for Jung the dangers of misinformation online. That lesson inspired him to join the Social Computing and Algorithmic Experiences (SCALE) Lab led by Information School professor and Allen School adjunct professor Tanu Mitra, where he is leading an audit study on YouTube to investigate the differences in COVID-19 misinformation exposure between the United States and countries in Africa. Closer to home, Jung has translated his renewed confidence in computer science as a force for good into initiatives that will uplift members of the Allen School community. These include supporting first-year students through the Big/Little mentorship program sponsored by the UW chapter of the Association for Computing Machinery (UW ACM) — of which he is vice chair — and arranging a variety of social and research-focused events for students during the pandemic.

“My vision of positively impacting the world through technology began at UW,” Jung said. “UW has empowered me to use technology to serve, advocate for diverse communities, and combat social inequities.”

Chase King

In his sophomore year, Chase King had an opportunity to witness firsthand the power of artificial intelligence to make a positive difference in people’s lives on a global scale. As an intern at Beewriter, a fledgling company based out of UW’s Startup Hall that develops AI-assisted multilingual writing tools, King received an email from a grateful user in Vietnam who had successfully used the tools to improve his resumé and job applications to secure employment as a technology consultant. It was an “aha” moment for King that helped shape his Husky experience going forward.

“Knowing that my independent research had a positive impact on thousands of people around the world speaking a diversity of languages solidified my desire to focus on the social impact of my work going forward,” explained King, who originally hails from the Bay Area. “I want to transcend the boundaries of my own language and culture to positively influence the lives of many — including those who speak a different language and live half a world away.”

Another epiphany came after King enrolled in the “Data & Society” course offered by the UW Department of Sociology, which challenged him to consider the potential downsides to AI that hasn’t been designed to equitably serve all users. The course opened his eyes to the various ways in which algorithmic unfairness manifests in practice, including how tools such as facial recognition can compound structural racism and cause real harm to already marginalized communities. He subsequently used what he learned in that class to formulate new homework problems in his capacity as a teaching assistant in the Allen School’s machine learning course, to encourage students to contemplate the potential shortcomings of the AI models and their real-world implications. He also has incorporated such lessons into his discussions with students during office hours, finding teaching to be a “powerful force multiplier” for encouraging students to share the same insights with their peers and apply those lessons to their future work.

“These conversations are especially relevant for those of us in purely technical fields,” King noted, “where it can be easy to fall into the trap of perpetuating the status quo, which does not adequately acknowledge minority voices.”

In addition to pursuing a double major in computer science and applied mathematical and computational sciences, King is minoring in neural computation and engineering. The latter has enabled him to explore how technology can be used to assist people with sensory and motor impairments. As part of his capstone project, King collaborated with a group of students to design a neurotechnology device that uses computer vision to help visually impaired people avoid obstacles while navigating indoor spaces. In addition to providing him with an opportunity to build a technical solution to a real-world problem, the capstone experience reinforced the importance of technologists engaging directly with the individuals and communities they hope to serve through their work.

“I envision a future where all information technology graduates are equipped with skills to think critically on a societal level and to design inclusive, equitable and impactful technology,” King said.

Simona Liao

Growing up in what she describes as a traditional Chinese family in Beijing, Zhehui (Simona) Liao heard early and often about the value of a college education as the pathway to securing a good job. What she did not hear was encouragement to pursue a technical field; both her father, who majored in computer science in college, and a high school teacher discouraged her from learning coding when she was younger by observing that “girls are not as good as boys in STEM.”

While that may have deterred Liao in high school, once she arrived at the UW, she re-discovered coding via the Allen School’s introductory computer science course. Here, she found great joy in completing the Java programming assignments as well as a huge sense of accomplishment. Liao’s mother, for one, embraced her daughter’s newfound career path. This “subversive” experience reminded Liao of what she learned in her Gender, Women & Sexuality Studies classes about how gender binary functions to discipline women and girls via institutions such as family and school — also in the GWSS classrooms, she further gained inspiration and encouragement by learning about Chinese feminists and revolutionists from 100 years ago who transformed Imperial China. 

“My mom, despite being independent and brilliant enough to become an engineering professor, is still impacted by conventional gender ideologies to take on the invisible reproductive labor in our household,” noted Liao. “I recognized my privilege as an international student with a supportive mother who invests in my education. I know not all girls have the same opportunities to discover their hidden interests in STEM.”

Putting the two together — quite literally, as Liao is pursuing a double major in computer science and GWSS — has inspired her to use that privilege to support other women and girls interested in technical fields and to work to increase diversity, equity and inclusion more generally, both inside and outside of the Allen School. To help others in China who may not enjoy the same support for their own career choices, Liao founded Forward with Her, a non-profit networking and mentorship program that connects women college students with women professionals in STEM fields. Although such programs are commonplace in countries like the United States, they are relatively rare in China; when Liao presented FWH as part of the 2020 Social Innovation Competition hosted by UN Women and Generation Equality in Beijing, it earned recognition among the top 10 programs. To date, FWH has connected more than 400 women students with professional mentors and peers in China.

“As a woman in CS, I understood the feelings of being alone in a classroom surrounded by men and the difficulty in finding a sense of belonging,” explained Liao. “FWH created a community where women students can find women role models with developed careers in STEM and peers with whom to share their experiences.”

Back on the UW campus, Liao’s passion for sharing diverse experiences led her to serve first as event coordinator and then as president of Minorities in Tech, an Allen School student group focused on building community and advancing allyship in support of students with diverse backgrounds and experiences. She has helped organize educational events focused on Black History Month and community conversations that provide a safe space in which to explore issues around identity, inequality and the culture of computing. As the vice president of outreach, Liao also led multiple STEM outreach programs for students from low-resource school districts in Washington state on behalf of the UW chapter of the Society of Women Engineers.

More recently, Liao decided to apply her interest in gender issues and technology to research. She joined the Social Futures Lab, led by Allen School professor Amy Zhang, where she is investigating approaches for addressing sexual harassment in social games played in virtual reality.

“My Husky experience started with first understanding the world in terms of systems of power through a transnational feminist perspective and then leveraging my educational privileges to support a larger community,” Liao said. “In the future, I hope to keep expanding the impact I can make by drawing upon all the knowledge, experience, and leadership skills I gained at UW.”

Ather Sharif

Nine years ago, Allen School Ph.D. student Ather Sharif fell asleep in the back seat of a car and woke up in a hospital bed. His spinal cord had been severed in an accident, leaving him paralyzed from the neck down. At the time, Sharif was pursuing his master’s degree at the University of North Dakota; the prospect of not being able to use a computer again made him question his purpose in life.

“The struggles and limitations from the spinal cord injury forced me to contemplate the decision to continue living,” Sharif said. “But I chose to keep moving forward. And seeing where I am and who I am today, I am glad I did.”

Moving forward included restarting his master’s program at Saint Joseph’s University in Philadelphia — after spending over a year in a rehabilitation hospital relearning how to live and regaining his independence. Back on campus, Sharif was one of the only people who used a wheelchair; since the building that housed the computer science department was not accessible, he contended with the added complexity of scheduling classes and meetings in different locations. The situation deprived him of any sense of community within his own discipline.

That changed when he discovered AccessComputing, a program focused on supporting students with disabilities to pursue computer science education and careers. It was through this online community that Sharif met Allen School professor emeritus Richard Ladner, who encouraged him to apply to the Ph.D. program. Sharif’s research focuses on making online data visualizations accessible to blind and low-vision users and has been published at the ACM SIGACCESS Conference on Computers & Accessibility (ASSETS) and ACM Conference on Human Factors in Computer Systems (CHI). Since his arrival at the UW in 2018, Sharif has worked with his advisors, Allen School professor Katharina Reinecke and iSchool professor and Allen School adjunct professor Jacob O. Wobbrock, on a variety of projects, including reassessing the measurement of device interaction based on fine motor function. As part of that work, he interviewed users about their experience with digital pointing devices. He fondly remembers how one participant’s eyes lit up when it dawned on them that they might one day “use a mouse like an able-bodied person.”

“Those words, and the genuine excitement and honest hope they embodied, have not only gotten engraved in my mind but have also defined and motivated my accessibility-related research work,” recalled Sharif.

Sharif has also collaborated with professor Jon Froehlich in the Allen School’s Makeability Lab on tracking the evolution of sidewalk accessibility over time, and he and Dhruv Jain and Venkatesh Potluri earned a Best Paper nomination at ASSETS 2020 for their ethnographic study of how students with disabilities navigate graduate school. Outside of his research, Sharif recognized that one of the reasons so many barriers exist for people with disabilities is their lack of representation in leadership positions. He decided to be the change he’d like to see by volunteering to serve as co-chair of the Allen School’s Graduate Student Committee and as a member of the G5PAC — short for Graduate Student, Fifth-year Master’s, and Postdoc Advisory Council — and the LEAP Alliance (Diversifying LEAdership in the Professoriate), among other roles. He also serves as a member of the user advisory group for the Office of the ADA Coordinator at UW, and mentors other students in accessibility research and founded a grassroots organization, EvoXLabs, to advance universal web design and other accessible technologies.

“While the world has made significant progress in recognizing the inequities and disenfranchisement disabled people face in their everyday lives, we are nowhere close to achieving our desired goal of an equitable society,” Sharif observed. “My future is that of an advocate, a leader, and a researcher devoted to making this world a better place for disabled people.”

Congratulations to all of our 2022 Husky 100 honorees — you make the Allen School and UW proud! Read more →

“Nothing about us without us.”

That statement has become a rallying cry for people with disabilities to ensure they have a direct voice in shaping the policies and conditions that, in turn, shape their access to employment, education, and lately, technology. With the growing proliferation of human-centered applications powered by artificial intelligence, it has become clear that the question of who will benefit from these emerging technologies will be determined in no small part by who makes them.

Venkatesh Potluri, a Ph.D. student currently working with professor Jennifer Mankoff in the Allen School’s Make4all Group, is among the makers determined to advance a more inclusive approach. He is also legally blind, which gives him firsthand knowledge of the barriers people with disabilities face in their interactions with technology on both the front and back ends. 

“Conversations about inclusion in AI typically highlight the role of fair hiring practices and equal access to education,” Potluri said. “For people with disabilities, having access to the actual tools of development is equally critical to making the field and the technology it produces more inclusive. Right now, when it comes to the development of human-centered machine learning systems, that access is extremely limited.”

This is particularly the case for blind or visually impaired (BVI) developers, a group historically underrepresented in computing to begin with. For Potluri and his peers, access to critical elements of modern programming, such as user interface and user experience design, is essentially non-existent. Human-centered ML systems depend heavily on these elements, however; without accessibility support for tools beyond basic programming, BVI developers are sidelined from the development process in this rapidly growing area. Beyond the impact on the individuals’ careers — which is bad enough — the exclusion of developers who are BVI or have other disabilities from teams developing these technologies can have real-world repercussions across the field of AI.

“Human-centered machine learning holds a lot of promise for solving important societal problems ranging from health care to transportation. But those solutions won’t benefit everyone if diverse needs and perspectives aren’t taken into account,” Potluri explained. “And in some cases, embedded biases in these technologies can result in real harm. For example, a self-driving car that relies on a collision avoidance system trained solely on a ‘typical’ pedestrian profile may not recognize a wheelchair user crossing the street.”

Earlier this year, Potluri was named a 2022 Apple Scholar in AI/ML for his efforts to advance a new paradigm in UI design, one that uses AI to improve accessibility for BVI developers. Potluri’s attempts to investigate how blind or visually impaired computer users understand visual semantics such as shape, spacing, and size of user interface elements show that users do want access to this information, but many current screen readers do not surface it. Current UI accessibility tools offer basic — one might even say paltry — descriptions of core functionality like menus and links, while disregarding visual semantics such as shape, size, spatial arrangement and overall consistency. With the support of his Ph.D. fellowship from Apple, Potluri plans to construct a dataset and machine learning models for automatically generating UI descriptions that incorporate these rich visual semantics. His goal is to address a fundamental challenge BVI users have in understanding visual layout and aesthetics. 

The enhanced descriptions produced by his new models will lay the groundwork for improvements in text-based search for design templates as well as machine understanding of UIs. And that’s just phase one; Potluri intends to build upon that foundation by developing an accessible UI editor that will improve the design experience for BVI developers. The new tool would enable developers to search for, apply and iteratively assess design templates, and to obtain suggestions for repairing deviations from convention that could detract from the user experience. 

Potluri’s efforts will empower BVI developers to make meaningful UI design decisions that determine form as well as function. Like previous advances in accessible technology, the benefits will most certainly extend beyond people with disabilities.

“Ultimately, the results of this work will improve the quality of AI-based assistance for BVI and sighted users alike when designing UIs for BVI and sighted users alike,” Potluri noted.

Potluri is one of just 15 graduate students at universities around the world to be recognized by Apple in its latest cohort of Scholars. He and his fellow honorees were chosen based on their innovative research, leadership and collaboration, and commitment to advancing their respective fields. Previously, as a first-year Ph.D. student working with Mankoff and professor Jon Froehlich in the Allen School’s Makeability Lab, Potluri earned a Google Lime Scholarship in recognition of his leadership, academic excellence and passion for computer science and technology.

“Venkatesh is all too familiar with the limitations and biases encoded into today’s systems with regard to what BVI programmers are capable of, or even interested in, doing,” noted Mankoff. “His research is raising the bar for what’s possible, and will help to include BVI people in fields such as interface design and data science.

“Not only is Venkatesh’s work of the highest quality, but also, his work works. He is committed to developing not just ideas but practical solutions that will lift up people with disabilities by giving them equitable access to these growing segments of our field,” she continued. “I cannot emphasize enough how important this is, in a world that often still thinks of people with disabilities as the subjects of research rather than the originators.”

Learn more about the Apple Scholars in AI/ML Ph.D. Fellowship here.

Congratulations, Venkatesh! Read more →

When Sewon Min first arrived at the University of Washington as an exchange student in the fall of 2016, little did she know how those three months would change the course of her academic career. After completing a brief stint as an undergraduate research assistant under the guidance of Allen School professors Hannaneh Hajishirzi and Ali Farhadi, she returned to Seoul National University in Korea to complete her bachelor’s degree. In the interim, Hajishirzi “tried really hard” to convince Min to move back to the Pacific Northwest to begin her Ph.D. — an effort that ultimately led to a happy reunion with her former advisor as well as a new mentor in professor Luke Zettlemoyer.

In the end, it doesn’t seem like Min took much convincing.

“It was very clear that returning to the Allen School for my Ph.D. would be the right choice for me. I loved every interaction I had with Hanna and Ali,” recalled Min. “All of our discussions and their suggestions opened up a lot of new directions. Oftentimes, when I was stuck, I would leave our meeting excited about new ideas I could try. Also, Hanna has been a world-leading expert in the topic I’ve been especially excited about — question answering — and I had a strong desire to continue working on that.”

Since her arrival, Min has wasted no time in establishing herself as one of the most promising up and coming researchers in the field. She has published more than 15 papers at the premier NLP conferences that have earned more than 1,700 citations combined. Now in her fourth year at the Allen School, Min recently earned a 2022 JP Morgan Ph.D. Fellowship in artificial intelligence to build on her already impressive record of leading the field of natural language processing in new — and sometimes unexpected — directions. 

“Sewon is a rising star who aims to take NLP paradigms to the next level. She not only pushes performance for long-standing hard problems, but also blazes entirely new directions for the field,” said Hajishirzi, who splits her time between the Allen School’s H2Lab and the Allen Institute for AI, where she is a senior research manager. “Sewon’s work is both technically sophisticated and highly impactful, and she continues to push the boundaries of what natural language models can do for a range of real-world applications.”

Min has proven particularly adept at pushing boundaries through her work on question answering. Whereas much of the previous research in this area has focused on restricted questions — ones for which a single answer can be extracted from a given document — Min has chosen to focus on broadening the capabilities of NLP models to respond to questions more akin to those posed by humans.

“My goal is to build a system that understands and can reason about natural language at a level that will help people solve problems they face in their daily lives, from answering their queries, to detecting false information on the internet,” explained Min. “Current systems assume a well-defined user query and a single, definite answer, but that’s not how the human quest for knowledge works! People ask ambiguous and open-ended questions, sometimes built on false presuppositions and requiring complex processing and reasoning about real-world conditions.”

One of Min’s early research breakthroughs was in multi-hop question answering — that is, questions that require reasoning about one or more facts to arrive at the correct answer. In a paper that appeared at the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019), Min and her collaborators proposed a novel approach to the problem that involves decomposing multi-hop questions into their component parts. Their system, DecompRC, generates sub-questions that can be answered by state-of-the-art single-hop QA models, and then chains the answers to arrive at the correct response to the original question. The technique proved to be both clever and efficient; even with the benefit of only 400 labeled training examples — a relatively miniscule amount of data in machine learning terms — Min and the team demonstrated that DecompRQ could generate high-quality sub-questions on a par with human-authored ones.

Another area in which Min has already made significant contributions is open-ended question answering. She became interested in exploring how to bestow models with the ability to field questions that are inherently ambiguous after realizing the extent to which existing models make faulty assumptions about the nature of the questions real-world users would ask. For example, after examining a corpus of over 14,000 questions based on Google search queries, Min found more than half to be ambiguous in their references to events, entities, time dependency or other factors. Given that ambiguity can be difficult for both machines and humans to spot, she conceived of a new open-ended QA task, AmbigQA, in which the model has to retrieve every plausible answer based on the various potential interpretations of what the questioner was searching for, and then produce a disambiguated question for each answer. 

Min and her colleagues presented their results at the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), where it caught the attention of the NLP community as an example of how to effectively model ambiguity as well as study other challenges associated with more realistic open-domain question answering. According to her co-advisor, Zettlemoyer, Min’s work on this and several other projects illustrates her dedication to not only developing solutions to important problems, but also to making sure they are the right problems to solve in the first place.

“Sewon has an uncanny ability to figure out what methods will work well in practice while opening up entirely new ways of thinking about problems,” said Zettlemoyer, who is also a research director at Meta AI. “Beyond her technical contributions, what really sets Sewon apart is how she considers problems as a whole and pushes the machine learning community into more realistic and open-ended territory instead of focusing narrowly on well-trodden challenges.”

With support from the JP Morgan Ph.D. Fellowship, Min is eager to continue down the path less traveled as she continues to grapple with an open-ended question of her own.

“How do we build models that can deal with that level of ambiguity and imperfection — and do so in a way that is computationally efficient?” she asked. “That is the problem that I’m trying to solve.”

Min is one of 11 students who were named JP Morgan Ph.D. Fellows last month as part of the company’s AI Research Awards program. Learn more about the 2022 class of fellows here.

Congratulations, Sewon! Read more →

When he was a student in high school, computer science major Alex Mallen had what he describes as a “rough” introduction to research. Fortunately, the Bellevue, Washington, native didn’t let that experience deter him at the University of Washington, where as a freshman he decided to try again as a step toward pursuing a Ph.D. after graduation. Mallen’s persistence has paid off in the form of multiple, positive research experiences that have helped him to solidify his plans to enroll in graduate school and, most recently, a prestigious Goldwater Scholarship to support his goal of helping to build artificial intelligence that people can trust to be beneficial.

Of his renewed focus on research, Mallen describes how he cast a wide net and kept an open mind — useful advice for any student hoping to incorporate time in the lab as part of their own undergraduate experience.

“I reached out to professors, postdocs and graduate students whose research I found interesting,” he explained. “I also enrolled in a graduate class in an area I was interested in with my mentor, Professor Kutz.”

Nathan Kutz is a professor in the Department of Applied Mathematics and director of the AI Institute in Dynamic Systems at the UW. In collaboration with Kutz and postdoc Henning Lange, Mallen contributed to the development of a simple and computationally efficient new technique, Deep Probabilistic Koopman (DPK), that enables probabilistic forecasting of complex phenomena thousands of timesteps into the future with a reasonable degree of accuracy. The new class of models, which leverages recent advances in linear Koopman operator theory, returns a probability distribution that assumes parameters will vary quasi-periodically with time. Mallen and his co-authors demonstrated how their approach could be effectively applied in a variety of domains, from forecasting energy demand, to predicting atmospheric pollution levels, to modeling a mouse’s cortical function for neuroscience research. 

“I began working with Alex when he was just a freshman, and I’m not sure how you could find someone as talented and creative and productive as he has been so early in his career,” said Kutz. “He spearheaded our work on DPK, for which he provided critical missing theory for how nonstationary data relate to building Koopman embeddings to transform nonlinear dynamical systems into linear dynamical systems. When we applied this work to a challenge data set for power grid monitoring, his new method placed within the top three—whereas most of the other algorithms had been improved over several years. This is but one illustration of the quality of his work and his potential for transformative impact.”

Mallen subsequently contributed directly to neuroscience research working with members of the Allen Institute for Brain Science. There, he helped to construct and analyze the dataset underpinning the MICrONS Explorer, which offers a comprehensive visualization of the mouse visual cortex. The team developed the tool as part of the Machine Intelligence from Cortical Networks Program to pave the way for a new generation of machine learning algorithms based on an enhanced understanding of “the algorithms of the brain.” More recently, Mallen has been collaborating with a group of researchers based predominantly in Europe and members of the grassroots research collective EleutherAI on a project to direct and characterize the behavior of large pretrained transformers, such as GPT-3, using the example of a large transformer pretrained on human chess games. 

Mallen aims to combine his passion for research with a commitment to effective altruism, which espouses an evidence-based approach to developing solutions to society’s most pressing problems. To that end, he and other members of the UW Effective Altruism group are working to build a community of people on campus who are looking to apply their expertise to do good.

He believes the approach could be particularly effective for addressing the outsized influence AI could have on society in the future.

“It seems reasonably likely that AI will have a very large impact on the world in the next hundred years, and that this shift will have a large and lasting effect on people’s lives for many generations,” Mallen observed. “The effects of AI systems we design are in theory predictable and controllable, but the challenge of properly steering them gets harder as they become more capable.

“I hope to tackle some of the general problems that may arise when training capable AI systems, such as misalignment with human values,” he continued. “We can already see some of these issues in current algorithms that produce toxic or biased output, or social media that harm discourse and mental health by overoptimizing for engagement.”

Mallen is one of two UW students to be named 2022 Goldwater Scholars by the Barry Goldwater Scholarship & Excellence in Education Foundation. Sharlene Shirali, a junior majoring in neuroscience, joined him among this year’s honorees, who are chosen for their potential to make significant research contributions in the natural sciences, engineering or mathematics.

While he is interested in many disciplines, Mallen chose to pursue computer science at the Allen School as an effective means for making sense of the world around him — and for achieving the altruistic impact that he seeks.

“I’m really interested in understanding things — society, philosophy, math, the world — but also I want to do something useful to other people,” Mallen said. “I think computer science is a really important tool to do both.”

Read the Goldwater Foundation announcement here, and the UW Undergraduate Academic Affairs announcement here.

Congratulations, Alex! Read more →

Artificial intelligence tools have the potential to become as essential to medical research and patient care as centrifuges and x-ray machines. Advances in high-accuracy predictive modeling can enable providers to analyze a range of patient risk factors to facilitate better health care outcomes — from preventing the onset of complications during surgery, to assessing the risk of developing various diseases.

When it comes to emergency services or critical care settings, however, the potential benefits of AI in the treatment room are often outweighed by the costs. And in this case, the talk about cost in health care isn’t just about money.

“With sufficient data and the right parameters, AI models perform quite well when asked to predict clinical outcomes. But in this case, ‘sufficient data’ often translates as an impractical number of patient features to collect in many care settings,” noted Gabriel Erion (Ph.D., ‘21), who is combining an M.D. with a Ph.D. in computer science as part of the University of Washington’s Medical Scientist Training Program. “The cost, in terms of the time and effort required to collect that volume and variety of data, would be much too high in an ambulance or intensive care unit, for example, where every second counts and responders need to prioritize patient care.”

But thanks to Erion and collaborators at the UW’s Paul G. Allen School and the UW School of Medicine, providers needn’t make a choice between caring directly for patients and leveraging advances in AI to identify the interventions with the highest likelihood of success. In a paper published in Nature Biomedical Engineering, the team presents CoAI, short for Cost-Aware Artificial Intelligence, a new framework for dramatically reducing the time, effort and resources required to predict patient outcomes and inform treatment decisions without sacrificing the accuracy of more cost-intensive tools. 

To reduce the number of clinical risk factors required to be collected in real time, the researchers trained CoAI on a massive dataset combining patient features, prediction labels, expert annotations of feature cost, and a budget representing total acceptable cost. They applied Shapley values to calculate a quantitative measure of the predictive power of every single feature in the dataset; since Shapley values are additive, this approach enables CoAI to calculate the importance of a group of features relative to their cost. CoAI then recommends which subset of features would enable the most accurate prediction of patient risk within a specified budget. And some of those budgets are very tight, indeed.

“Fifty seconds. That’s how long first responders told us they can spare to score patient risk factors when they are in the midst of performing a life-saving intervention,” said co-senior author and professor Su-In Lee, who leads the Allen School’s AIMS Lab focused on integrating AI and the biomedical sciences. “CoAI deals with this constraint by prioritizing a subset of features to gather while achieving the same or better accuracy in its predictions as other, less cost-aware models. And it is generalizable to a variety of care settings, such as cancer screening, where different feature costs come into play — including financial considerations.”

As co-author Joseph Janizek (Ph.D., ‘22) explained, CoAI has a significant advantage over even other cost-sensitive methods owing to its efficiency and flexibility.

“A notable difference between CoAI and other approaches is its robustness to ‘cost shift,’ wherein features become more or less expensive after the model has been trained. Since our framework decouples feature selection from training, CoAI continues to perform well even when this shift occurs,” noted Janizek, who is also pursuing his M.D. in combination with a Ph.D. from the Allen School via the MSTP. “And because it’s model-agnostic, CoAI can be used to adapt any predictive AI system to be cost-aware, enabling accurate predictions at lower cost within a wide variety of settings.”

Janizek and his AIMS Lab colleagues teamed up with clinicians at the UW School of Medicine and first responders with Airlift Northwest, American Medical Response and the Seattle Fire Department to validate the CoAI approach. In a series of experiments, the researchers evaluated CoAI’s performance compared to typical AI models in predicting the increased bleeding risk of trauma patients en route to the hospital and the in-hospital mortality risk of critical care patients in the ICU. They also surveyed first responders and nurses to understand how patient risk scoring works in practice — hence the aforementioned 50-second rule. In the case of trauma response, their experiments showed that CoAI dramatically reduces the cost of data acquisition — by around 90% — while still achieving levels of accuracy comparable to other, more cost-intensive approaches. They achieved similar results for the inpatient critical care setting.

According to co-senior author Dr. Nathan White, associate professor of Emergency Medicine at the UW School of Medicine, these results speak to what is possible when researchers break down barriers between disciplines and prioritize how new technologies will be put to real-world use.

“A key contributor to the success of this project included the great synergy afforded by working across traditional silos of medicine and engineering,” said White. “AI is an important component of healthcare today, but we must always be aware of the clinical situations where AI is being used and seek out input from frontline health care workers involved directly in patient care. This will ensure that AI is always working optimally for the patients it intends to benefit.”

Lee agreed, noting that the UW’s MSTP serves to enhance this synergy with each new student who enters the program.

“Gabe and Joe were the first UW MSTP students to earn their Ph.D. in the Allen School. They exemplify the best of both worlds, combining rigorous computer science knowledge with hands-on clinical expertise,” Lee said. “This nexus of knowledge, spanning two traditionally disparate disciplines, will be essential to our future progress in developing AI as an effective and efficient tool used in biomedical research and treatment decisions.”

Dr. White’s colleagues in the Department of Emergency Medicine, Drs. Richard Utarnachitt, Andrew McCoy and Michal Sayre, along with Dr. Carly Hudelson of the Division of General Internal Medicine, are co-authors of the paper. An early preview of the project earned the Madrona Prize sponsored by Madrona Venture Group at the Allen School’s annual research day in 2019. The research was funded by the National Science Foundation, American Cancer Society, and National Institutes of Health.

Read the paper in Nature Biomedical Engineering. Read more →

Less than a year after her arrival at the University of Washington, professor Yulia Tsvetkov is making her mark as the newest member of the Allen School’s Natural Language Processing group. As head of the Tsvetshop — a clever play on words that would likely stymie your typical natural language model — Tsvetkov draws upon elements of linguistics, economics, and the social and political sciences to develop technologies that not only represent the leading edge of artificial intelligence and natural language processing, but also benefit users across populations, cultures and languages. Having recently earned a 2022 Sloan Research Fellowship from the Alfred P. Sloan Foundation, Tsvetkov is looking forward to adding to her record of producing new tools and techniques for making AI and NLP more equitable, inclusive and socially aware.

“One of the goals of my work is to uncover hidden insights into the relationship between language and biases in society and to develop technologies for identifying and mitigating such bias,” said Tsvetkov. “I also aim to build more equitable and robust models that reflect the needs and preferences of diverse users, because many speakers of diverse language varieties are not well-served by existing tools.”

Her focus at the intersection of computation and social sciences has enabled Tsvetkov to make inroads when it comes to protecting the integrity of information beyond “fake news” by identifying more subtle forms of media manipulation. Even with the growing attention being paid to identifying and filtering out misleading content, tactics such as distraction, propaganda and censorship can be challenging for automated tools to detect. To overcome this challenge, Tsvetkov has spearheaded efforts to develop capabilities for discerning “the language of manipulation” automatically and at scale. 

In one project, Tsvetkov and her colleagues devised computational approaches for detecting subtle manipulation strategies in Russian newspaper coverage by applying agenda-setting and framing — two concepts from political science — to tease out how one outlet’s decisions about what to cover and how were used to distract readers from economic conditions. She also produced a framework for examining the spread of polarizing content on social media based on an analysis of Indian and Pakistani posts following the 2019 terrorist attacks in Kashmir. Given the growth in AI-generated text, Tsvetkov has lately turned her attention to semantic forensics, including the analysis of the types of misinformation and factual inconsistencies produced by large AI models with a view to developing interpretable deep learning approaches that will control for factuality and other traits of machine-generated content. 

“Understanding the deeper meaning of human- or machine-generated text, the writer’s intent, and what emotional reactions the text is likely to evoke in its readers is the next frontier in NLP,” said Tsvetkov. “Language technologies that are capable of doing such fine-grained analysis of pragmatic and social meaning will be critical for combating misinformation and opinion manipulation in cyberspace.”

Another of the ways in which Tsvetkov’s work has contributed to researchers’ understanding of the interplay between language and social attitudes is by surfacing biases in narrative text targeting vulnerable audiences. NLP researchers — including several of Tsvetkov’s Allen School colleagues — have demonstrated effective techniques for identifying toxic content online, and yet more subtle forms continue to evade moderation. Tsvetkov has been at the forefront of developing new datasets, algorithms and tools grounded in social psychology to detect discrimination, at scale and across multiple languages, based on gender, race and/or sexual orientation that manifests in online text and conversations. 

“Although there are tools for detecting hate speech, most harmful web content remains hidden,” Tsvetkov noted. “Such content is hard to detect computationally, so it propagates into downstream NLP tools that then serve to amplify systematic biases.”

One approach that Tsvetkov has employed to great effect is an expansion of contextual affective analysis (CAA), a technique for examining how people are portrayed along dimensions of power, agency and sentiment, to multilingual settings in an effort to understand how narrative text across different languages reflects cultural stereotypes. After applying a multilingual model to English, Spanish and Russian Wikipedia entries about prominent LGBTQ figures in history, Tsvetkov and her team found systematic differences in phrasing that reflected social biases. For example, entries about the late Alan Turing, who was persecuted for his homosexuality, described how he “accepted” chemical castration (English), “chose” it (Spanish), or “preferred” it (Russian) — three verbs with three very different connotations as to Turing’s agency, power and sentiment at the time. Tsvetkov applied similar analyses to uncover gender bias in media coverage of #MeToo and assist the Washington Post in tracking racial discrimination in China, and has since built upon this work to produce the first intersectional analysis of bias in Wikipedia biographies that examines gender disparities beyond cisgender women alongside racial disparities.

The fact that most existing NLP tools are grounded in a specific variant of English has been a driving force in much of Tsvetkov’s research. 

“We researchers often say that a model’s outputs are only as good as its inputs,” Tsvetkov noted. “For the purposes of natural language models, those inputs have mostly been limited to a certain English dialect — but there are multiple English dialects and over 6,000 languages besides English spoken around the world! That’s a significant disconnect between current tools and the billions of people for whom English is not the default. We can’t achieve NLP for all without closing that gap.”

To that end, Tsvetkov has recently turned her attention to developing new capabilities for NLP technologies to adapt to multilingual users’ linguistic proficiencies and preferences. For example, she envisions tools that can match the ability of bilingual and non-native speakers of English and Spanish to switch fluidly between the two languages in conversation, often within the same sentence. Her work has the potential to bridge the human-computer divide where, currently, meaning and context can get lost in translation.

“Yulia is intellectually fearless and has a track record of blending technical creativity with a rigorous understanding of the social realities of language and the communities who use it,” said Magdalena Balazinska, professor and director of the Allen School. “Her commitment to advancing language technologies that adapt to previously ignored users sets her apart from her research peers. By recognizing that AI is not only about data and math, but also about people and societies, Yulia is poised to have an enormous impact on the field of AI and beyond.”

Tsvetkov joined the Allen School last July after spending four years on the faculty of Carnegie Mellon University. She is one of two UW researchers who were honored by the Sloan Foundation in its class of 2022 Fellows, who are chosen based on their research accomplishments and creativity as rising leaders in selected scientific or technical fields. Briana Adams, a professor in the UW Department of Biology, joined Tsvetkov among a total of 118 honorees drawn from 51 institutions across the United States and Canada.

Read the Sloan Foundation press release here and a related UW News release here. 

Congratulations, Yulia!

Rebekka Coakley contributed to this story. Read more →

Recent advances in open-ended text generation could enable machines to produce text that approaches or even mimics that generated by humans. However, evaluating the quality and accuracy of these large-scale models has remained a significant computational challenge. Recently, researchers at the Allen School and Allen Institute for AI (AI2) offered a solution in the form of MAUVE, a practical tool for assessing modern text generation models’ output compared to human-generated text that is both efficient and scalable. The team’s paper describing this new approach, “MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers,” earned an Outstanding Paper Award at the Conference on Neural Information Processing Systems (NeurIPS 2021) in December.

The goal of open-ended text generation is to achieve a level of coherence, creativity, and fluency that mimics human text. Because the task is, as the name suggests, open-ended, there is no correct answer; this makes evaluation of a model’s performance more difficult than with more concrete tasks such as translation or summarization. MAUVE solves this problem by employing information divergence frontiers — heretofore a little-used concept in NLP — to reduce the comparison between model-generated text and human text to a computationally tractable yet effective measurement.

“For open-ended text generation to make that next leap forward, we need to be able to evaluate a model’s performance on two key aspects that are prone to error: how much weight it gives to sequences that truly resemble human text, as opposed to gibberish, and whether the generated text exhibits the variety of expression we would expect to see from humans, instead of boring or repetitive text that reads like a template,” explained lead author Krishna Pillutla, a Ph.D. candidate in the Allen School. “The beauty of MAUVE is that it enables us to quantify both, using a simple interface and an approach that is easily scaled to whatever sized model you’re working with.”

MAUVE computes the divergence between the model distribution and target distribution of human text for the above-mentioned pair of criteria in a quantized embedding space. It then summarizes the results as a single scalar that illustrates the gap between the machine-generated and human text. To validate MAUVE’s effectiveness, the team tested the tool using three open-ended text completion tasks involving web text, news articles and stories. The results of these experiments confirmed that MAUVE reliably identifies the known properties of machine-generated text, aligns strongly with human judgments, and scales naturally with model size — and does so with fewer restrictions than existing distributional evaluation metrics. And whereas other language modeling tools or statistical measures are typically limited to capturing a single statistic or correspond to only one point on the divergence curve, MAUVE offers expanded insights into a model’s performance.

“MAUVE enables us to identify the properties of machine-generated text that a good measure should capture,” noted co-author Swabha Swayamdipta, a postdoctoral investigator at AI2. “This includes distribution-level information that enables us to understand how the quality of output changes based on the size of the model, the length of text we are asking it to generate, and the choice of decoding algorithm.”

While Swayamdipta and her colleagues designed MAUVE with the goal of improving the quality of machine-generated text — where “quality” is defined according to how closely it resembles the human-authored kind — they point out that its capabilities also provide a foundation for future work on how to spot the difference. 

“As with every new technology, there are benefits and risks,” said senior author Zaid Harchaoui, a professor in the University of Washington’s Department of Statistics and adjunct professor in the Allen School. “As the gap narrows between machine and human performance, having tools like MAUVE at our disposal will be critical to understanding how these more sophisticated emerging models work. The NLP community can then apply what we learn to the development of future tools for distinguishing between content generated by computers versus that which is produced by people.”

Additional co-authors of the paper introducing MAUVE include Allen School Ph.D. student Rowan Zellers, postdoc Sean Welleck, alumnus John Thickstun (Ph.D., ‘21) — now a postdoc at Stanford University — and Yejin Choi, the Brett Helsel Career Development Professor in the Allen School and a senior research manager at AI2. The team received one of six Outstanding Paper Awards presented at NeurIPS 2021, which are chosen based on their “clarity, insight, creativity, and potential for lasting impact.”

Members of the team also studied the statistical aspects of MAUVE in another paper simultaneously published at NeurIPS 2021. Together with Lang Liu, Ph.D. candidate in Statistics at UW, and Allen School professor Sewoong Oh, they established bounds on how many human-written and machine-generated text samples are necessary to accurately estimate MAUVE. 

Read the research paper here and the NeurIPS award announcement here. Explore the MAUVE tool here.

Congratulations to the entire team! Read more →

Luke Zettlemoyer, a professor in the Allen School’s Natural Language Processing group and a research director at Meta AI, was recently elected a Fellow of the Association for Computational Linguistics (ACL) for “significant contributions to grounded semantics, semantic parsing, and representation learning for natural language processing.” Since he arrived at the University of Washington in 2010, Zettlemoyer has focused on advancing the state of the art in NLP while expanding its reach into other areas of artificial intelligence such as robotics and computer vision.

Zettlemoyer broke new ground as a Ph.D. student at MIT, where he advanced the field of semantic parsing through the application of statistical techniques to natural language problems. He and his advisor, Michael Collins, devised the first algorithm for automatically mapping natural language sentences to logical form by incorporating tractable statistical learning methods — specifically, the novel application of a log-linear model — in a combinatory categorial grammar (CCG) with integrated semantics. He followed up that work, for which he received the Best Paper Award at the Conference of Uncertainty in Artificial Intelligence (UAI 2005), by developing techniques for mapping natural language instructions to executable actions through reinforcement learning that rivaled the performance of supervised learning methods. Those results earned him another Best Paper Award with MIT colleagues, this time from the Association for Computational Linguistics (ACL 2009). 

After he arrived at the Allen School, Zettlemoyer continued pushing the state of the art in semantic parsing by introducing the application of weak supervision and the use of neural networks, among other innovations. For example, he worked with student Yoav Artzi (Ph.D., ‘15) on the development of the first grounded CCG semantic parser capable of jointly reasoning about meaning and context to execute natural language instructions with limited human intervention. Later, Zettlemoyer teamed up with Allen School professor Yejin Choi, postdoc Ionnas Konstas, and students Srinivasan Iyer (Ph.D., ‘19) and Mark Yatskar (Ph.D., ‘17) to introduce Neural AMR, the first successful sequence-to-sequence model for parsing and generating text via Abstract Meaning Representation, a useful technique for applications ranging from machine translation to event extraction. Previously, the use of neural network models with AMR was limited due to the expense of annotating the training data; Zettlemoyer and his co-authors solved that challenge by combining a novel pretraining approach with preprocessing of the AMR graphs to overcome sparsity in the data while reducing complexity.

Question answering is another area of NLP where Zettlemoyer has made multiple influential contributions. For example, the same year he and his co-authors presented Neural AMR at ACL 2017, Zettlemoyer and Allen School colleague Daniel Weld worked with graduate students Mandar Joshi and Eunsol Choi (Ph.D., ‘19) to introduce TriviaQA, the first large-scale reading comprehension dataset that incorporated full-sentence, organically generated questions composed independent of a specific NLP task. According to another Allen School colleague, Noah Smith, Zettlemoyer’s vision and collaborative approach are a powerful combination that has enabled him to achieve a series of firsts while steering the field in exciting new directions.

“Simply put, Luke is one of natural language processing’s great pioneers,” said Smith. “From his graduate work on semantic parsing, to a range of contributions around question answering, to his extremely impactful work on large-scale representation learning, he’s shown foresight and also the ability to execute on his big ideas and the charisma to bring others on board to help.”

One of those big ideas Smith cited — large-scale representation learning — went on to become ubiquitous in NLP research. In 2018, Zettlemoyer, students Christopher Clark (Ph.D., ‘20) and Kenton Lee (Ph.D., ‘17), and collaborators at the Allen Institute for AI (AI2) presented ELMo, which demonstrated pretraining as an effective tool for enabling a language model to acquire deep contextualized word representations that could be incorporated into existing models and fine-tuned for a range of NLP tasks. ELMo, which is short for Embeddings from Language Models, satisfied the dual challenges of modeling the complex characteristics of word use such as semantics and syntax while also capturing how such uses vary across different linguistic contexts. Zettlemoyer subsequently did some fine-tuning of his own by contributing to new and improved pretrained models such as the popular RoBERTa — with more than 6,500 citations and counting — and BART. In addition to earning a Best Paper Award at the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2018), the paper describing ELMo has been cited more than 9,200 times.

Zettlemoyer pioneered another exciting research trend when he began connecting the language and vision aspects of AI. For example, he worked with Yatskar and Allen School colleague Ali Farhadi to introduce situation recognition, which applies a linguistic framework to a classic problem in computer vision — namely, how to concisely and holistically describe the situation an image depicts. Situation recognition represented a significant leap forward from independent object or activity recognition with its ability to summarize the main activity in a scene, the actors, objects and locations involved, and the relationship among all of these elements. Zettlemoyer also contributed to some of the first work on language grounding for robotic agents, which built in part on his original contributions to semantic parsing from his graduate student days. He and a team that included Allen School professor Dieter Fox, students Cynthia Matuszek (Ph.D., ‘14) and Nicholas FitzGerald (Ph.D., ‘18), and postdoc Liefeng Bo developed an approach for joint learning of perception and language that endows robots with the ability to recognize previously unknown objects based on natural language descriptions of their physical attributes. 

“It is an unexpected but much appreciated honor to be named an ACL Fellow. I am really grateful to and want to highlight all the folks whose research is being recognized, including especially all the students and research collaborators I have been fortunate enough to work with,” Zettlemoyer said. “The Allen School has been an amazing place to work for the last 10+ years. I really couldn’t imagine a better place to launch my research career, and can’t wait to see what the next 10 years — and beyond — will bring!”

Zettlemoyer previously earned a Presidential Early Career Award for Scientists and Engineers (PECASE) and was named an Allen Distinguished Investigator in addition to amassing multiple Best Paper Awards from the preeminent research conferences in NLP and adjacent fields. In addition to his faculty role at the Allen School, he joined Facebook AI Research in 2018 after spending a year as a senior research manager at the Allen Institute for AI. He is one of eight researchers named among the ACL’s 2021 class of Fellows and the third UW faculty member to have attained the honor, following the election of Smith in 2020 and Allen School adjunct faculty member Mari Ostendorf, a professor in the Department of Electrical & Computer Engineering, in 2018.

The designation of Fellow is reserved for ACL members who have made extraordinary contributions to the field through their scientific and technical excellence, service and educational and/or outreach activities with broad impact. Learn more about the ACL Fellows program here.

Congratulations, Luke! Read more →