Allen School News

Luke Zettlemoyer, a professor in the Allen School’s Natural Language Processing group and a research director at Meta AI, was recently elected a Fellow of the Association for Computational Linguistics (ACL) for “significant contributions to grounded semantics, semantic parsing, and representation learning for natural language processing.” Since he arrived at the University of Washington in 2010, Zettlemoyer has focused on advancing the state of the art in NLP while expanding its reach into other areas of artificial intelligence such as robotics and computer vision.

Zettlemoyer broke new ground as a Ph.D. student at MIT, where he advanced the field of semantic parsing through the application of statistical techniques to natural language problems. He and his advisor, Michael Collins, devised the first algorithm for automatically mapping natural language sentences to logical form by incorporating tractable statistical learning methods — specifically, the novel application of a log-linear model — in a combinatory categorial grammar (CCG) with integrated semantics. He followed up that work, for which he received the Best Paper Award at the Conference of Uncertainty in Artificial Intelligence (UAI 2005), by developing techniques for mapping natural language instructions to executable actions through reinforcement learning that rivaled the performance of supervised learning methods. Those results earned him another Best Paper Award with MIT colleagues, this time from the Association for Computational Linguistics (ACL 2009). 

After he arrived at the Allen School, Zettlemoyer continued pushing the state of the art in semantic parsing by introducing the application of weak supervision and the use of neural networks, among other innovations. For example, he worked with student Yoav Artzi (Ph.D., ‘15) on the development of the first grounded CCG semantic parser capable of jointly reasoning about meaning and context to execute natural language instructions with limited human intervention. Later, Zettlemoyer teamed up with Allen School professor Yejin Choi, postdoc Ionnas Konstas, and students Srinivasan Iyer (Ph.D., ‘19) and Mark Yatskar (Ph.D., ‘17) to introduce Neural AMR, the first successful sequence-to-sequence model for parsing and generating text via Abstract Meaning Representation, a useful technique for applications ranging from machine translation to event extraction. Previously, the use of neural network models with AMR was limited due to the expense of annotating the training data; Zettlemoyer and his co-authors solved that challenge by combining a novel pretraining approach with preprocessing of the AMR graphs to overcome sparsity in the data while reducing complexity.

Question answering is another area of NLP where Zettlemoyer has made multiple influential contributions. For example, the same year he and his co-authors presented Neural AMR at ACL 2017, Zettlemoyer and Allen School colleague Daniel Weld worked with graduate students Mandar Joshi and Eunsol Choi (Ph.D., ‘19) to introduce TriviaQA, the first large-scale reading comprehension dataset that incorporated full-sentence, organically generated questions composed independent of a specific NLP task. According to another Allen School colleague, Noah Smith, Zettlemoyer’s vision and collaborative approach are a powerful combination that has enabled him to achieve a series of firsts while steering the field in exciting new directions.

“Simply put, Luke is one of natural language processing’s great pioneers,” said Smith. “From his graduate work on semantic parsing, to a range of contributions around question answering, to his extremely impactful work on large-scale representation learning, he’s shown foresight and also the ability to execute on his big ideas and the charisma to bring others on board to help.”

One of those big ideas Smith cited — large-scale representation learning — went on to become ubiquitous in NLP research. In 2018, Zettlemoyer, students Christopher Clark (Ph.D., ‘20) and Kenton Lee (Ph.D., ‘17), and collaborators at the Allen Institute for AI (AI2) presented ELMo, which demonstrated pretraining as an effective tool for enabling a language model to acquire deep contextualized word representations that could be incorporated into existing models and fine-tuned for a range of NLP tasks. ELMo, which is short for Embeddings from Language Models, satisfied the dual challenges of modeling the complex characteristics of word use such as semantics and syntax while also capturing how such uses vary across different linguistic contexts. Zettlemoyer subsequently did some fine-tuning of his own by contributing to new and improved pretrained models such as the popular RoBERTa — with more than 6,500 citations and counting — and BART. In addition to earning a Best Paper Award at the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2018), the paper describing ELMo has been cited more than 9,200 times.

Zettlemoyer pioneered another exciting research trend when he began connecting the language and vision aspects of AI. For example, he worked with Yatskar and Allen School colleague Ali Farhadi to introduce situation recognition, which applies a linguistic framework to a classic problem in computer vision — namely, how to concisely and holistically describe the situation an image depicts. Situation recognition represented a significant leap forward from independent object or activity recognition with its ability to summarize the main activity in a scene, the actors, objects and locations involved, and the relationship among all of these elements. Zettlemoyer also contributed to some of the first work on language grounding for robotic agents, which built in part on his original contributions to semantic parsing from his graduate student days. He and a team that included Allen School professor Dieter Fox, students Cynthia Matuszek (Ph.D., ‘14) and Nicholas FitzGerald (Ph.D., ‘18), and postdoc Liefeng Bo developed an approach for joint learning of perception and language that endows robots with the ability to recognize previously unknown objects based on natural language descriptions of their physical attributes. 

“It is an unexpected but much appreciated honor to be named an ACL Fellow. I am really grateful to and want to highlight all the folks whose research is being recognized, including especially all the students and research collaborators I have been fortunate enough to work with,” Zettlemoyer said. “The Allen School has been an amazing place to work for the last 10+ years. I really couldn’t imagine a better place to launch my research career, and can’t wait to see what the next 10 years — and beyond — will bring!”

Zettlemoyer previously earned a Presidential Early Career Award for Scientists and Engineers (PECASE) and was named an Allen Distinguished Investigator in addition to amassing multiple Best Paper Awards from the preeminent research conferences in NLP and adjacent fields. In addition to his faculty role at the Allen School, he joined Facebook AI Research in 2018 after spending a year as a senior research manager at the Allen Institute for AI. He is one of eight researchers named among the ACL’s 2021 class of Fellows and the third UW faculty member to have attained the honor, following the election of Smith in 2020 and Allen School adjunct faculty member Mari Ostendorf, a professor in the Department of Electrical & Computer Engineering, in 2018.

The designation of Fellow is reserved for ACL members who have made extraordinary contributions to the field through their scientific and technical excellence, service and educational and/or outreach activities with broad impact. Learn more about the ACL Fellows program here.

Congratulations, Luke! Read more →

The Allen School has a proud tradition of nurturing undergraduate student researchers whose work has the potential for real-world impact. This year, three of those students — Jerry Cao, Mike He and Yu Xin — earned honorable mentions from the Computing Research Association (CRA) as part of its 2022 Outstanding Undergraduate Researcher Awards competition for their contributions in health sensing and fabrication, programming languages and machine learning, and building robust computer systems.

Jerry Cao

The CRA recognized senior Jerry Cao, who is majoring in computer science and applied mathematics, for his research in health sensing and fabrication. Advised by professors Jennifer Mankoff and Shwetak Patel, his work aims to apply computing and fabrication to improve individuals’ quality of life. To reduce the burden of health monitoring and make it easier for users to prototype custom tools that fit their personalized needs, Cao is creating a wearable device in compression sleeve-form for the leg that records changes in the blood volume in the body’s superficial tissue. This can help predict the onset of adverse symptoms throughout the day for conditions such as Postural Orthostatic Tachycardia Syndrome (POTS) where blood flow is improperly regulated throughout the body.

Cao is also working on a project to rapidly prototype physical objects. He aims to reduce the number of iterations — currently requiring several to reach the final product — by reconfiguring a model to support real-time iteration. He is developing a pipeline to take a parametric model and produce a reconfigurable prototype where each parameter can be adjusted up to a specified and allowed range. Users can more easily change the size of the physical model this way and record all the necessary measurements to fabricate a final version. For example, when building a cabinet, builders must ensure it fits in its designated space. The reconfigurable prototype will limit the iterations and allow users to explore different configurations of the object, then create the final version using actual materials.

Mike He

Mike He, a senior studying computer science, was acknowledged for his work in programming languages, formal verification, compilers and machine learning systems. Advised by professor Zachary Tatlock, He worked with the Allen School’s PLSE group on Dynamic Tensor Rematerialization (DTR), an algorithm that trains deep learning models under constrained memory budgets. Since deep learning models use up a lot of GPU memory while training, He and his colleagues created DTR to train these models under restricted memory budgets. DTR removes restrictions on classic compilers and when memory fills up, DTR evicts the oldest, stalest, cheapest-to-recompute tensor to make room for the next allocation. If the training loop later tries to access a previously evicted tensor, DTR recomputes it on demand by tracking operator dependencies. 

In addition to his contributions to DTR, He led the push to develop new flexible accelerator matching compiler techniques to easily target new hardware accelerators in deep learning frameworks. To do so, the team is enabling devices to be more easily incorporated into an existing DL framework and, in principle, for formal functional verification down to the hardware implementation. The project, 3LA, has a built-in pattern-matching algorithm that can find accelerator supported workloads in deep learning models using equality saturation. The project addressed the mapping gap between deep learning models represented in high-level domain-specific languages and specialized accelerators using instruction-level abstraction as the software-hardware interface.

Yu Xin 

Yu Xin, a senior studying computer science and applied and computational mathematical science, was honored by the CRA for his work with Allen School professor Arvind Krishnamurthy in building effective and robust computer systems. In particular, Xin worked to develop a scheduler for serving deep learning inference tasks. Applications using cloud-based deep learning models, when deployed on a large scale, tend to flood data center GPU clusters, slowing down the time it takes to respond and causing delays and extra expense. To help with the cost and speed, Xin and his collaborators created Symphony, a centralized dispatcher to satisfy requests within a latency bound, ensuring load-balance across GPUs and maximizing their efficiency by using appropriate dynamically-sized batches of inference requests. By loading dozens of deep learning models on each GPU, Symphony enables burst amortization across models and has the potential to eliminate the need for overprovisioning. Enabling multiple dispatchers for better scalability, Xin designed an algorithm to partition the model space into many disjoint subsets in which each dispatcher handles one of the models. The algorithm finds the partitioning scheme that minimizes the deviation between partitions in terms of total request rates and model sizes by generating and solving a Mixed Integer Linear Programming (MILP) problem.

Xin’s previous work includes developing tools for analyzing images of proteins generated from a cryo-electron microscope. For example, filtering out high-frequency noises by generating an artificial image based on a mathematical model and comparing it against every patch of the image to see if there is a match and then output all the matched results. This approach saves researchers time while increasing their effectiveness by directing their attention to the most relevant sites.

Congratulations to Jerry, Mike and Yu! 

Read more →

During their time at the Allen School, recent alumni Maarten Sap (Ph.D., ‘21) and Ivan Evtimov (Ph.D., ‘21) tackled some of the thorniest issues raised by emerging natural language processing and machine learning technologies — from endowing NLP systems with social intelligence while combating inequity and bias, to addressing security vulnerabilities in the convolutional neural networks that fuel state-of-the-art computer vision systems. Recently, the faculty honored both for their contributions with the William Chan Memorial Dissertation Award, which was named in memory of the late graduate student William Chan to recognize dissertations of exceptional merit. Evtimov earned additional recognition for his work from the Western Association of Graduate Schools and ProQuest as the recipient of the WAGS/ProQuest Innovation in Technology Award, which recognizes distinguished scholarly achievement at the master’s or doctoral level.

Sap — who is currently a postdoctoral/young investigator at the Allen Institute for AI (AI2) — worked with Allen School professors Yejin Choi and Noah Smith. His dissertation, “Positive AI with Social Commonsense Models,” advanced new techniques for making NLP systems more human-centric, socially aware and equity-driven.

“Maarten’s dissertation presents groundbreaking work advancing social commonsense reasoning and computational models serving equity and inclusion. More specifically, his work presents technical and conceptual innovations that make deep learning methods significantly more equitable,” said Choi and Smith, both of whom are also senior research managers at AI2. “Maarten’s research steers the field of NLP and its products toward a better future.”

One example is ATOMIC, a large-scale social commonsense knowledge graph Sap and collaborators created to help machines comprehend day-to-day practical reasoning about events, causes and effects. To create equity-driven NLP systems, he also helped develop PowerTransformer, a controllable text rewriting model that helps authors mitigate biases in their writing, particularly biases related to how the public describes people of different genders. Sap also tackled the problem of detecting biases and toxicity in language by identifying issues with the current hate speech detectors that lead to racial biases. His work introduced Social Bias Frames, a linguistic framework for explaining the biased or harmful implications in text. The papers supporting this, The Risk of Racial Bias in Hate Speech Detection and Social Bias Frames: Reasoning about Social and Power Implications of Language were nominated for a Best Short Paper Award by the Association for Computer Linguistics in 2019 and won the Best Paper Award at the West Coast NLP Summit in 2020, respectively. Sap was also a member of the team that won the first Amazon Alexa Prize for a conversational chatbot called Sounding Board that engages with users about current topics.

TechCrunch, Forbes, Fortune and Vox have all covered Sap’s research. After completing his postdoc with AI2’s MOSAIC team, he will join Carnegie Mellon University’s Language Technology Institute as a professor in the fall.

Evtimov’s dissertation, “Disrupting Machine Learning: Emerging Threats and Applications for Privacy and Dataset Ownership,” makes significant contributions to the security of adversarial machine learning. His research as a member of the Allen School’s Security & Privacy Research Lab focused particularly on the vulnerabilities of convolutional neural networks (CNN) that allow maliciously crafted inputs to affect both their inference and training. Evtimov said that understanding new technologies in terms of  security and privacy is important in order to think ahead of adversarial actors. 

“Ivan’s dissertation is highly innovative, and contributed significant results to the field of real-world attacks against computer vision algorithms. His work is of fundamental importance to the field,” Allen School professor and lab co-director Tadayoshi Kohno said. “Computer vision is everywhere — in autonomous cars, in computer authentication schemes, and more. Ivan’s dissertation helps the field develop secure computer vision systems and also provides foundations for helping users protect their privacy in the face of such systems.”

Evtimov’s work shows that the vulnerabilities of CNNs exhibit a duality when it comes to security and privacy. For example, he found the computer algorithms for cameras reading traffic signs in autonomous cars could be tricked by an object as simple as a sticker on a stop sign. The sticker could fool the cameras into reading the stop sign as a speed limit sign. In the case of autonomous driving, it is critical to identify anything that could be exploited by malicious parties in such a safety-critical setting. Machine learning, Evtimov found, can also be used in an unauthorized manner. Take, for example, a search engine for facial recognition. To protect privacy, Evtimov studied the conditions in which people could flood a database full of photos gathered from the public without permission with decoys. He proposed FoggySight, a tool that involves community users uploading modified photos — for instance, labeling photos of Madonna as photos of Queen Elizabeth  — to poison the facial search database and throw off searches in it. He also found ways to protect visual data released for human consumption from misuse through machine learning, including developing a protective mechanism that can be applied to the information contained in datasets before public release to prevent unauthorized parties from training their own models using the data. 

Evtimov’s research has been covered by Ars Technica, IEEE Spectrum and more. He previously won a Distinguished Paper Award at the Workshop on Adversarial Machine Learning in Real-World Computer Vision Systems and Online Challenges for his work examining the vulnerability of combined image and text models to adversarial threats. After graduating from the Allen School, Evtimov joined Meta as a research scientist. 

Congratulations to Maarten and Ivan! 

Read more →

Allen School professor emeritus Richard Ladner has been elected a Fellow of the American Association for the Advancement of Science (AAAS) for his “distinguished contributions to the inclusion of persons with disabilities in the computing fields.” One of 26 leading scientists in the organization’s Information, Computing & Communications section to attain the rank of Fellow this year, Ladner has devoted the past two decades to research and advocacy aimed at making computing education and careers more accessible while designing technologies that empower all users.

A mathematician by training, Ladner helped establish the University of Washington’s theoretical computer science group shortly after joining the faculty in the early 1970’s. At the time, Ladner’s interest in disability issues was personal, having been raised by two parents who were deaf. Later, after he completed an American Sign Language course at a local community college, Ladner began doing volunteer work with people who were deaf and blind as well as writing about accessibility issues. Having worked on several accessibility projects in the 1980s and 1990s, his first full-time foray into accessible technology development — an experience that would alter the course of his career in terms of both research and advocacy — would not come until 2002.

That year, Ladner met Sangyun Hahn, a graduate student who was blind. Hahn related to his new advisor his frustration at being unable to easily access certain content in his textbooks, such as mathematical formulas and diagrams. Their discussions led to the launch of the Tactile Graphics project to automate the conversion of textbook figures into an accessible format. A series of accessibility projects followed, including MobileASL, a collaboration between Ladner and UW Electrical & Computer Engineering professor Eve Riskin to enable people to communicate using American Sign Language via a mobile phone; WebAnywhere, a non-visual platform enabling people who are blind to navigate the web using any browser, on any device, with Jeffrey Bigham (Ph.D., ‘09); Perkinput, a Braille-based text entry system for touchscreen devices, with Shiri Azenkot (Ph.D., ‘14) and iSchool professor Jacob Wobbrock; and Blocks4All, an accessible, touchscreen-based blocks environment for children who are blind to learn programming, with Lauren Milne (Ph.D., ‘18). 

As a student with a disability, Hahn was a relative rarity in computer science graduate programs at the time he and Ladner met. When the latter turned his attention full-time from exploring the theoretical underpinnings of computing to making computing more accessible to all users, he recognized that one of the obstacles was the lack of pathways for more people with disabilities to pursue computer science and bring their perspectives to the development of new technologies. This led him to partner with Sheryl Burgstahler, director of UW’s DO-IT Center, to establish the Alliance for Access to Computing Careers, or AccessComputing, in 2006 with support from the National Science Foundation’s Broadening Participation in Computing program. AccessComputing helps high school, undergraduate and graduate students to build skills and connections with mentors and professional opportunities in the computing fields. So far, the program has directly served more than 2,400 students with disabilities through a range of activities, from academies and workshops to research and work-based internships.

Ladner subsequently teamed up with Andreas Stefik at the University of Nevada, Las Vegas to launch AccessCSforAll, an initiative aimed at providing accessible curriculum and resources to engage students with disabilities in K-12 computer science education. That work led Code.org and the Computer Science Teachers Association to name Ladner, Stefik and the Quorum programming team 2018 Computer Science Champions. A year later, Ladner and Stefik were again recognized — this time alongside collaborators William Allee and Sean Mealin — with a Best Paper Award from the Association for Computing Machinery’s Special Interest Group in Computer Science Education at the SIGCSE 2019 conference. In the winning paper, “Computer Science Principles for Teachers of Blind and Visually Impaired Students,” the team presented the results of its partnership with Code.org to review and revamp the Advanced Placement CSP curriculum and tools for accessibility.

In 2020, the National Science Board recognized Ladner with its Public Service Award for his exemplary science communication and diversity advocacy — the latest in a long line of previous accolades for his leadership on accessible technology and education that includes the Strache Leadership Award from the Center on Disabilities at University of California, Northridge, the Award for Outstanding Contributions to Computing and Accessibility from the ACM Special Interest Group on Accessible Computing (SIGACCESS), the Richard A. Tapia Achievement Award for Scientific Scholarship, Civic Science and Diversifying Computing from the Center for Minorities and People with Disabilities in Information Technology (CMD-IT), and more. Along the way, Ladner also earned the 2019 Harrold and Notkin Research and Graduate Mentoring Award — named in part for the late David Notkin, former chair of what was then known as the UW Department of Computer Science & Engineering — from the National Center for Women and Information Technology (NCWIT) for his long-standing efforts to advance gender diversity in computing.

Even after he officially attained emeritus status at the Allen School in 2017, Ladner remained active in research and mentoring students in addition to advocacy and program leadership. Over the course of his career, he has supervised or co-supervised 30 Ph.D. students and more than 100 undergraduate researchers — many of whom sought Ladner out for his focus on accessibility before that line of research entered the mainstream. Some of those same students later established the Richard E. Ladner Endowed Professorship, currently held by his faculty colleague Jennifer Mankoff, and the Richard Ladner Endowed Fund for Graduate Student Support in his honor.

Ladner also continues to build on his legacy of advocacy for engaging people with disabilities in technology development. The same year he was recognized for making the K-12 computer science curriculum more accessible, he helped establish the UW’s Center for Research and Education on Accessible Technology and Experiences (CREATE) alongside eight colleagues from multiple UW departments with an inaugural investment from Microsoft. Under the slogan “making technology accessible and making the world accessible through technology,” CREATE supports transformational, multidisciplinary research that will translate into real-world impact while building expertise in accessible technologies and increasing representation in the field for people with disabilities.

“Richard is truly a pioneer in the field of accessible computing,” said professor Magdalena Balazinska, director of the Allen School. “He understood the importance of fully including people with disabilities long before the rest of the field recognized this challenge and he continues to innovate today. He’s an inspiration to all of us.”

Ladner was previously elected a Fellow of the ACM and of the IEEE. He is one of four UW faculty members recognized in the 2021 class of AAAS Fellows, including Emily Carrington, who was honored in the Biological Sciences, and Julia A. Kovacs and Stefan Stoll, who were both honored in Chemistry. Founded in 1848, AAAS is the world’s largest general scientific society.

Learn more about the newly elected AAAS Fellows here.

Congratulations, Richard! Read more →

Mohit Shridhar, a Ph.D. student working with Allen School professor Dieter Fox, has been named a 2022-2023 NVIDIA Graduate Fellow for his research in building generalizable systems for human-robot collaboration. Shridhar’s work is focused on connecting language to perception and action for vision-based robotics.

Shridhar aims to use deep learning to connect abstract concepts to concrete physical actions with long-term reasoning to develop robot butlers. The Fellowship will help him continue his work in building robots that learn through embodied interactions rather than from static datasets. Using his own creation CLIPort, a language-conditioned imitation-learning agent, will advance precise spatial reasoning and learning generalizable semantic representations for vision and language. Shridhar’s framework combines two-streams with semantic and spatial pathways, where the semantic stream uses an internet pre-trained vision language model to bootstrap learning. This end-to-end framework can solve a variety of language-specified tabletop tasks, from packing unseen objects to folding clothes with centimeter-level precision.

“Mohit’s CLIPort work is the first to show the power of combining general language and image understanding models with fine-grained robot manipulation capabilities,” said Fox, who leads the Allen School’s Robotics & State Estimation Lab and is senior director of robotics research at NVIDIA..

In order to communicate with the butlers, Shridhar developed the Action Learning From Realistic Environments and Directives dataset (ALFRED). This is a dataset for agents to learn mapping from natural language instructions and egocentric vision to sequences of actions for household tasks. ALFRED consists of 25,000 natural language directives, including high-level instructions like “rinse off a mug and place it in the coffee maker” and lower-level language directions like “walk to the coffee maker on the right.” Tasks given to ALFRED are more complex in terms of sequence length, action space and language than previous vision-and-language task datasets.

Taking the next step beyond communicating tasks to the robots, Shridhar wants the robots to think about long-term actions without directly dealing with the complexities of the physical world. An example he gives is telling an agent to make an appetizer with sliced apples. Without any physical interactions, ALFWorld, a simulator that enables agents to learn abstract, “textual” policies in an interactive TextWorld, will train the robot to check the fruit bowl for apples and look in the drawers for a knife to make the appetizer. Before ALFWorld, agents did not have the infrastructure necessary for both reasoning abstractly and executing concretely. 

Shridhar intends to deploy ALFRED-trained models in household environments where a mobile manipulator can be commanded to perform tasks such as putting two plates on the dining table.

“I hope to build collaborative butler robots that aid and better human living,” Shridhar said.

Before coming to the Allen School, Shridhar received his Bachelor’s in Engineering from the National University of Singapore. He has interned at Microsoft Research, NVIDIA and an augmented reality startup. 

Shridhar is only one of 10 students recognized by the Graduate Fellowship Program based on their innovative research in Graphics Processing Unit (GPU) computing. Previous Allen School recipients of the NVIDIA Fellowship include Anqi Li (2020) and Daniel Gordon (2019).

Read more about the 2022-2023 NVIDIA Graduate Fellowship awards here.

Congratulations, Mohit! Read more →

“You are what you eat,” as the saying goes. But not everyone has the same degree of choice in the matter. An estimated 19 million people in the United States live in so-called food deserts, where they have lower access to healthy and nutritious food. More than 32 million people live below the poverty line — limiting their options to the cheapest food regardless of proximity to potentially healthier options. Meanwhile, numerous studies have pointed to the role of diet in early mortality and the development of chronic diseases such as heart disease, type 2 diabetes and cancer.

Researchers are just beginning to understand how the complex interplay of individual and community characteristics influence diet and health. An interdisciplinary team of researchers from the University of Washington and Stanford University recently completed the largest nationwide study to date conducted in the U.S. on the relationship between food environment, demographics, and dietary health with the help of a popular smartphone-based food journaling app. The results of that five-year effort, published today in the journal Nature Communications, should give scientists, health care practitioners and policymakers plenty of food for thought. 

“Our findings indicate that higher access to grocery stores, lower access to fast food, higher income and college education are independently associated with higher consumption of fresh fruits and vegetables, lower consumption of fast food and soda, and less likelihood of being classified as overweight or obese,” explained lead author Tim Althoff, professor and director of the Behavioral Data Science Group at the Allen School. “While these results probably come as no surprise, until now our ability to gauge the relationship between environment, socioeconomic factors and diet has been challenged by small sample sizes, single locations, and non-uniform design across studies. Different from traditional epidemiological studies, our quasi-experimental methodology enabled us to explore the impact on a nationwide scale and identify which factors matter the most.”

Althoff ‘s involvement in the study dates from when he was a Ph.D. student at Stanford working with professor and senior author Jure Leskovec and fellow student and co-author Hamed Nilforoshan. Together with co-author Dr. Jenna Hua, a former postdoctoral fellow at Stanford University School of Medicine and founder and CEO of Million Marker Wellness, Inc., the team analyzed data from more than 1.1 million users of the MyFitnessPal app — spanning roughly 2.3 billion food entries and encompassing more than 9,800 U.S. zip codes — to gain insights into how factors such as access to grocery stores and fast food, family income level, and educational attainment contribute to people’s food consumption and overall dietary health. 

The team measured the association of the aforementioned input variables with each of four dietary outcomes: fresh fruit and vegetable consumption, fast food consumption, soda consumption, and incidence of overweight or obese classified by body mass index (BMI). To understand how each variable corresponded positively or negatively with those outcomes, the researchers employed a matching-based approach wherein they divided the available zip codes into treatment and control groups, split along the median for each input. This enabled them to compare app user logs in zip codes that were statistically above the median — for example, those with more than 20.3% of the population living within half a mile of the nearest grocery store — with those below the median.

Among the four inputs the team examined, higher educational attainment than the median, defined as 29.8% or more of the population with a college degree, was the greatest positive predictor of a healthier diet and BMI. All four inputs were found to positively contribute to dietary outcomes, with one exception: high family income, defined as income at or above $70,241, was associated with a marginally higher percentage of people with a BMI qualifying as overweight or obese. But upon further investigation, these results only scratched the surface of what is a complex issue that varies from community to community.

“When we dug into the data further, we discovered that the population-level results masked significant differences in how the food environment and socioeconomic factors corresponded with dietary health across subpopulations,” noted Nilforoshan.

As an example, Nilforoshan pointed to the notably higher association between above-median grocery store access and increased fruit and vegetable consumption in zip codes with a majority of Black residents, at a 10.2% difference, and with a majority of Hispanic residents, at a 7.4% difference, compared to those with a majority of non-Hispanic white residents, where he and his colleagues found only a 1.7% difference. These and other findings indicate that factors such as proximity to grocery stores or higher income, on their own, are not sufficient for people to bypass the drive-thru or kick the (soda) can to the curb — and that future attempts to address dietary disparities need to take variations across zip codes into account.

“People assume that if we eliminate food deserts, that will automatically lead to healthier eating, and that a higher income and a higher degree lead to a higher quality diet. These assumptions are, indeed, borne out by the data at the whole population level,” explained Hua. “But if you segment the data out, you see the impacts can vary significantly depending on the community. Diet is a complex issue! While policies aimed at improving food access, economic opportunity and education can and do support healthy eating, our findings strongly suggest that we need to tailor interventions to communities rather than pursuing a one-size-fits-all approach.”

Althoff believes that both the team’s approach and its findings can guide future research on this complex topic that has implications for both individuals and entire communities.

“We hope that this study will impact public health and epidemiological research methods as well as policy research,” said Althoff. “Regarding the former, we demonstrated that the increasing volume and variety of consumer-reported health data being made available due to mobile devices and applications can be leveraged for public health research at unprecedented scale and granularity. For the latter, we see many opportunities for future research to investigate the mechanisms driving the disparate diet relationships across subpopulations in the U.S.”

Read the paper in Nature Communications here. Access the publicly available data and code associated with the study here. Read more →

As the world watched COVID-19 grow from a mysterious virus in far-off places to a planetary pandemic, news outlets worked hard to keep the world informed on how, where and why it was spreading. At the start of the outbreak, Matthew Conlen, a Ph.D. student in the Allen School’s Interactive Data Lab, was working as a graphic/multimedia editor for the New York Times helping with their elections forecasting application, also known as “The Needle.” He switched gears to contribute to the paper’s coverage of the novel coronavirus as it enveloped the globe — work that contributed to stories that earned the New York Times a Pulitzer Prize in Public Service for, in part, filling “a data vacuum that helped local governments, healthcare providers, businesses and individuals to be better prepared and protected.” 

Conlen’s forte is creating interactive data visualizations that do precisely that: help the public to comprehend what is happening in the country and throughout the world. In this particular case, he led a data collection effort on COVID-19 in nursing homes, and he also worked with epidemiologists and modelers to give readers an understanding of what could happen in different scenarios when schools reopened, as the vaccine rolled out and when the U.S. could reach herd immunity.

“Data journalism can provide a valuable perspective on our world, complementing traditional narrative reporting with additional context, more comprehensive accounts and increased audience engagement,” Conlen’s advisor and Allen School professor Jeffrey Heer said. “Interactive visualizations rank among the most visited and revisited pieces published by major news outlets. Though data visualization on the web has largely ‘come of age,’ a major remaining challenge is empowering more journalists — as well as educators and others  —  without an extensive technical background to author and collaborate on interactive articles.”

Conlen’s work, Heer said, leads on all of these fronts: publishing data-driven news at multiple major outlets and via Parametric Press, which Conlen co-founded, while simultaneously researching at the Allen School, which has resulted in new open-source languages and tools that make these articles easier to create. 

“It’s a virtuous cycle of research and practice,” Heer said.

After earning dual bachelor’s degrees in computer science and applied mathematics, Conlen began working on a big data analytics platform at an advertising technology, or ad tech, company. Despite interesting technical challenges, he found more fulfilling work in journalism, using digital news tools at the Huffington Post, FiveThirtyEight, The New Yorker and NASA’s earth science communications team. He became interested in data visualization because it combines math and statistics with tough programming challenges and a creative design aspect. This combination of technical and creative elements is, he notes, hard to come by in other fields.

“I’m interested in computer science generally because I think computers can be tremendously empowering tools,” Conlen said. “I want to develop systems that enable people to do things that were otherwise out of reach. It’s like giving someone superpowers but all you have to do is write some code.”

He said he saw his pursuit of a Ph.D. as the next phase of a career oriented around data visualization and digital publishing.

“Within the world of academic research I can spend more time understanding how people learn from data visualizations and interactive graphics and what makes certain designs effective, and I can engage with rich fields like human-computer interaction, or HCI, to better understand how to build effective digital tools for journalists and others.”

Combining his journalism and research, Conlen could see that visual forms are effective for communicating complex data sets. As a journalist, he understands the real-world constraints that HCI needs to account for in order to be useful in practice. He and Heer created Idyll, a toolkit that reduces the amount of effort and custom code required to make it easier to author and publish interactive articles, based on the challenges Conlen observed in the newsrooms in which he worked. The interactive capabilities of Idyll are seen in Unraveling the JPEG by graphics programmer Omar Shehata. Conlen explained that by using the interactive capabilities of Idyll, Shehata constructed a narrative walkthrough of the JPEG compression algorithm that connected with a big audience online — an audience that might not be interested to learn about that topic if not for the graphics that he made. 

“It wouldn’t have been possible to create this system without the practical knowledge that I gained as a journalist or without the space and time to think deeply and build ambitious research systems that the Allen School affords,” Conlen said.

In addition to Idyll, Conlen published the Beginner’s Guide to Dimensionality Reduction, which earned a Best Paper Honorable Mention in 2018 at the IEEE Visualization and Visual Analytics Workshop on Visualization for AI Explainability. The article used interactive graphics to introduce a complex technical topic to new readers in a gentle and engaging way.

“I’m regularly impressed by the resilience of my students,” Heer, who leads the Interactive Data Lab said. “Matt’s ability to bridge the worlds of professional journalism and academic research is a standout example, animated by Matt’s commitment to a more just and better-informed society.”

After spending a year at The New York Times, Conlen returned his focus to academic research. In October he presented his paper, “Idyll Studio: A Structured Editor for Authoring Interactive & Data-Driven Articles,” at the Association for Computing Machinery’s Symposium on User Interface Software and Technology. Idyll Studio is a new graphical interface for writing interactive and data-driven stories. 

“Think Microsoft Word but you can create documents that are dynamically driven by databases and include interactive visualizations and graphics,” Conlen said.

Conlen defended his dissertation last week and is currently working on a short-term contract with the New York Times. In early 2022 he will continue his work on the Idyll ecosystem — the open source project received a donation from venture capitalist Albert Wegner that will allow Conlen to put more time into improving the core project and refining Idyll Studio. He will continue doing data journalism and building tools to support that work. 

To view more of Conlen’s work combining journalism and data visualization, check out his collection of published articles on his website. 

Read more →

Earlier this month, the Allen School held a virtual celebration showcasing efforts to increase diversity in computing and honoring members of our community who have demonstrated their commitment to diversity, excellence and leadership. An annual tradition, the event this year also offered Allen School leaders an opportunity to share highlights from its five-year strategic plan to increase diversity, equity, inclusion and access (DEIA), spanning curriculum and programs, professional development, policies and procedures, internal community engagement, external outreach and budget. Allen School professor and director Magdalena Balazinksa had the happy task of introducing two undergraduate scholarship winners: senior Nayha Auradkar, recipient of the Allen AI Outstanding Engineer Scholarship for Women and Underrepresented Minorities from the Allen Institute for Artificial Intelligence (AI2), and senior Caiwei Tian, recipient of the Lisa Simonyi Prize. 

Auradkar, who is enrolled in the Allen School’s B.S./M.S. program, exemplifies the goal of the AI2 scholarship to encourage students from underrepresented groups to excel in computer science and engineering and become leaders and role models in their fields. Finding a passion for machine learning and human computer interaction, Auradkar used it to conduct accessibility research in the Make4All lab with Allen School professor Jennifer Mankoff. As an undergraduate she published two papers, one aimed at analyzing the features of personal protective equipment in response to the pandemic and the other focused on automating the process of creating complex textured knitting objects to make it easier for people with mobility-related disabilities to knit. Auradkar said that as someone with a disability, accessibility research has deep personal value to her and enables her to use her skills to help other people with disabilities. 

Auradkar isn’t focused solely on academics, though; she’s determined to make a difference on campus through leadership, too. She is the chair of the ACM-W, founded and leads the Allen School affinity group Ability, founded and leads Huskies Who Stutter and served as the outreach director for the Society of Women Engineers. In these roles she teaches middle school girls introductory engineering, cultivates a strong community of women in tech, promotes disability community and accessibility awareness and supports other UW students who stutter.

“This scholarship will enable me to learn from and collaborate with top research scientists, which will allow me to grow my research skills as I transition in my graduate degree,” Auradkar said. “It will also provide me with extra support in my DEIA work.”

The Lisa Simonyi Prize was established by longtime Allen School supporters Lisa and Charles Simonyi. The couple created the scholarship to recognize and support students who exemplify excellence, leadership, and diversity. This year’s recipient, Tian, is a double major in computer science and applied and computational mathematical sciences. She added the former after a data structure and algorithm course inspired a newfound interest in using programming as a tool to turn complex ideas into practice and discussing algorithms and the tradeoff between runtime and memories. Tian works in the Allen School’s UbiComp Lab with professor Shwetak Patel on developing a generalized deep learning model that uses video signals from smartphones to measure blood oxygen saturation (SpO2) levels, a crucial test in modern medicine. This work focuses on building a unified platform-agnostic model that works on all major smartphone systems.

Tian also has worked as a software development engineering intern at Amazon, a research assistant in the Make4All Lab and a research assistant at Fred Hutch. Tian co-founded a Chinese student choir group, MotE, a 50-member group that performs at festivals. She also assisted underprivileged students and provided academic support and encouragement as a math and science tutor for students at Licton Spring K-8 Public School.

“I’m really excited and honored to get this scholarship,” Tian said. “Knowing nothing about computer science before coming to UW and now graduating with this award, I think it is an evidence of my hard-work and my growth at the Allen School. It also encourages me to go further and keep learning.”

Thanks to AI2 and the Simonyis for supporting diversity and excellence, and thanks to everyone who logged on to celebrate the people who are making our school and our field a more welcoming destination for all. And congratulations to Nayha and Caiwei! 

For more about the Allen School’s efforts to advance diversity in computing, please visit our website. 

Read more →

Researchers affiliated with the Allen School took home all three Best Paper Awards at the Association for Computing Machinery’s 28th Symposium on Operating Systems Principles (SOSP). Current Ph.D. student Jacob Van Geffen, recent alumnus James Bornholt (Ph.D.,’19), former postdoc and incoming professor Simon Peter and affiliate professor Daniel Berger contributed to the winning papers that presented new advances in debugging, distributed computing and caching. 

In the paper, “Using Lightweight Formal Methods to Validate a Key-Value Storage Node in Amazon S3,” Van Geffen and Bornholt, now a professor at the University of Texas, Austin, present ShardStore, a new storage backend for Amazon Simple Storage Service (S3). Built on 40,000 lines of Rust code, ShardStore optimizes disk IO efficiency and currently stores hundreds of petabytes of customer data. The paper describes how ShardStore is resilient, resilient and crash-safe, and how AWS uses formal methods to catch and fix bugs early. Additional authors of the paper include Vytautus Astrauskas, a Ph.D. student at ETH Zurich and a team of researchers from Amazon Web Services that included Rajeev Joshi, Brendan Cully, Bernhard Kragl, Seth Markle, Kyle Sauri, Drew Schleit, Grant Slatton, Serdar Tasiran and Andrew Warfield.

Focused on building automated mechanisms to help engineers ensure the correctness of every change they make, ShardStore was developed to employ techniques like property-based testing and model checking with far lower overhead than traditional provable correctness. According to the team, this lightweight formal method prevented a number of issues like crash consistency and concurrency problems, before reaching production. The team plans to continue improving their techniques for cloud-based data storage. 

Peter, who is currently a professor at the University of Texas, Austin and will join the Allen School faculty in January, co-authored “LineFS: Efficient SmartNIC Offload of a Distributed File System with Pipeline Parallelism,” about fitting the high demands of a distributed file system (DFS) onto smart network interface cards (SmartNICs). In the paper, the team presents LineFS, a SmartNIC-offloaded, high-performance DFS with support for client-local persistent memory. LineFS moves CPU-intensive tasks to a SmartNIC, improving latency in LevelDB — a fast, key-value store — up to 80%. Korea Advanced Institute of Science and Technology researchers Jongyul Kim, Insu Jang, Jaeseong Im and Youngjin Kwon, along with Waleed Reda and Dejan Kostic from the KTH Royal Institute of Technology and Emmett Witchel also at UT Austin, contributed to the paper.

In “Kangaroo: Caching Billions of Tiny Objects on Flash,” Berger, a researcher at Microsoft and UW,  and his co-authors present a new flash cache that enables more efficient caching of tiny objects — often in social media and IoT services — called Kangaroo. Kangaroo overcomes challenges in existing flash cache designs such as minimizing main memory usage, which is expensive and energy hungry, and reduces load on back-end storage systems. Additionally, Kangaroo reduces flash memory wear out, extending flash cache lifetimes by multiple years. This also helps cost and sustainability. The paper was written with Carnegie Mellon University researchers Sara McAllister, Benjamin Berg, Julian Tutuncu-Macias, Juncheng Yang, Nathan Beckman and Gregory Ganger and Facebook researchers Sathya Gunasekar and Jimmy Lu.

Congratulations to all! 

Read more →

Professor Joshua R. Smith, who holds a joint appointment in the Allen School and the Department of Electrical & Computer Engineering, was elected into the 2021 class of Fellows of the National Academy of Inventors for his impactful creations in the fields of wireless power, communication, sensing and robotics. Smith, who leads the Sensor Systems Lab, is one of only five University of Washington faculty members to have received this prestigious award that highlights the prolific spirit of innovation in academic inventors. The NAI Fellows program was created to recognize inventors and their contributions to society, which stimulate the economy, improve and save lives, and make the world a better place. It is the highest professional distinction given solely to academic inventors. 

The NAI Fellow selection process considers inventions that have been licensed or commercialized. Smith holds 48 U.S. patents and 16 international patents, 44 of which are licensed by companies. His inventions have led to hundreds of millions of dollars in product revenues, bolstering the economy and the creation of approximately 70 full-time jobs, according to Suzie Pun, a professor in the Department of Bioengineering and another UW faculty member who is an NAI Fellow. 

Smith’s first six patents, developed while he was a graduate student at MIT, pioneered mutual capacitance sensing and led to the creation of a smart airbag system that was included in every Honda car between 2000 and 2015. Before his arrival at UW, Smith spent five years at Intel Research Seattle, creating new capabilities in wireless power, wireless sensing and robotics. He led the creation of the Wireless Identification and Sensing Platform (WISP), the first fully programmable platform for wireless, battery-free sensing and computation powered by radio waves.

Soon after, he developed Wireless Resonant Energy Link (WREL), which uses magnetically coupled resonators to efficiently transfer wireless power even as range, orientation and load vary. With the help of a heart surgeon from Yale, Smith was able to power a ventricular assist device designed for implantation in the human body without requiring a cable through the patient’s chest, called the Free-range Resonant Electrical Energy Delivery System (FREED). This wireless power work at UW is commercialized by WiBotic, a company Smith co-founded with ECE alumnus Benjamin Waters (Ph.D., ‘15). The UW patents are also licensed for implanted heart pumps by Corisma.

“Among the many outstandingly inventive engineers at Intel Research Seattle, we were especially excited that Josh joined our faculty, he is extraordinary in every imaginable respect,” said Ed Lazowska, professor and Bill & Melinda Gates Chair Emeritus at the Allen School. “He is an academic inventor and entrepreneur of the highest caliber and in the finest tradition.”

In 2013, Smith, together with Allen School professor Shyam Gollakota and a team of graduate students, developed Ambient Backscatter using existing wireless signals to provide power and communication for low-power sensing and computing devices. This next led to the creation of Passive-Wi-Fi, bringing low-power Wi-Fi to transmissions. They also invented Interscatter, using wireless transmissions over the air from one technology to another for internet-connected implanted devices. Smith also co-led the UW team behind the world’s first battery‐free phone, as well as a series of ultra-low-power battery-free wireless cameras that communicate via backscatter.

The team’s research is being commercialized by Jeeva Wireless, a UW spinout co-founded by Smith, Gollakota, and ECE alumni Vamsi Talla (Ph.D., ‘16) and Aaron Parks (Ph.D., ‘17).

“Josh has a consistent record of impactful inventions,” said Pun. “I have gotten to know him through a research collaboration to develop touchscreen-based sensors for detection of pathogens such as SARS-CoV-2. Josh devised a creative method to improve detection sensitivity for the virus; he is in the process of testing this idea in his laboratory. If successful, his design could be applied for next generation biosensing devices.”

Smith also co-founded Proprio, which provides surgical visualization and navigation, together with UW neurosurgeon Sam Browd, Allen School graduate student Jim Youngquist, UW Foundation board member Ken Denman, and Michael G. Foster School of Business alumnus Gabe Jones (MBA, ‘14). Smith served on advisory councils and task forces for the United States Postal Service and the Smithsonian Institution and is an IEEE Fellow. His work has earned multiple Best Paper Awards, and he is known for his dedicated mentorship of student researchers.

“I feel so privileged to collaborate with my outstanding UW faculty and student co-inventors,” said Smith, who holds the Milton and Delia Zeutschel Professorship in Entrepreneurial Excellence in ECE. “And invention is just one part of a long process to bring new things into the world.

“I am very grateful to the many people who have worked so hard to take these inventions from the lab to the world, including UW CoMotion, many patent attorneys, and most of all the co-founders and employees at the companies making these technologies real.”

Read the NAI announcement here, and the full list of 2021 Fellows here.

Congratulations, Josh! 

Read more →