Examples of computational illusion knit design including a gray slate that transitions to the Mona Lisa, and an image of Vincent Van Gogh’s sunflowers shifting into his self portrait.
When you look at a knitted panel head on, all you might see is gray noise, but taking a step to the side and looking at an angle reveals another image — a hidden Mona Lisa.
This technique is called illusion knitting, where stitches of varying heights can create different images depending on what angle you view the piece. Traditionally, the intricate process of designing these illusions, by selecting stitches, choosing their colors and deciding whether they are raised or not was done by hand. Now, a team of Allen School researchers have introduced computational illusion knitting. The design framework helps automate the process by computationally generating knitting patterns for input designs, making illusion knitting more accessible and allowing for more complex and multi-view patterns that were previously impossible. The researchers presented the paper at the SIGGRAPH 2024 conference.
Allen School Ph.D. student Amy Zhu
“If we write down the mathematical properties of the illusion knits, then we have a foundation with which to create algorithms that help the user iterate on and optimize their designs,” lead author and Allen School Ph.D. student Amy Zhu said. “With this foundation you can start to think, ‘where can I push the boundaries of what’s possible with illusion knitting?’”
To create an illusion knit object, the researchers first characterize the design’s unique microgeometry, or the small-scale geometry on the surface of a knitted piece. The goal is to then add a layer of abstraction over the complex microgeometry to streamline the design process, Zhu explained. This layer uses logical units called bumps, or raised sections made by a knit with a purl in the next row below it, that can block previous stitches, as well as flats, where the knit surface is level and is made by a knit stitch with another in the next row.
These observations led the researchers to develop various constraints that capture both the viewing behavior, or what image is seen at each angle, and the physical behavior, also known as the result from the knitted design or fabrication choices. Single image illusion knits, such as the hidden Mona Lisa, are easier to design as they only change the color between rows.
Using computational techniques, the team became the first to design and execute illusion knitting patterns that require mixed colorwork and texture between rows. The researchers first used automated methods such as gradient descent and the maximum satisfiability problem (MaxSAT) to find a compromise that satisfies the most constraints and relaxes others, Zhu explained. However, these systems are still limited by how well they can express constraints about readability, knittability as well as how accurate the theoretical model is to reality.
A knit image of Edvard Munch’s “The Scream” quantized using a diffusion model on the left, and one quantized by hand on the right.
Instead of relying solely on the algorithm for solutions, the researchers created a user-in-the-loop framework that allows the user to easily edit the design. The user can then further simplify the design by breaking it down into tiles made of bumps and flats to fill the image.
“There’s many tricky things to consider when you’re in fabrication land that are often difficult to anticipate or capture when working theoretically — maybe the machine settings are wrong and the white stitches are smaller, making the contrast between colors lower,” Zhu said. “One reason I like our approach is that we put the fabricated reality first, so our priority is providing ways for the user to edit things that come out of the machine, so they can observe if something didn’t look right and know how to fix it.”
For example, using a knitting machine and this design framework, Zhu created a knit piece that shows artist Vincent Van Gogh’s self portrait from one angle and his sunflowers from another. The quantized input image of the self portrait had too much noise around his head which made the design unreadable when knit. For a cleaner result, all she had to do was edit the input image to remove the noise or modify the tiles.
Zhu said she hopes this framework gives users the foundation to create more innovative illusion knit designs such as clothes that display images with certain poses or knit animations. Next, Zhu is researching ways to capture and explore different fabrication plans of knit objects.
“Approaching crafts from a computer science angle allows us to expand the boundaries of what can be achieved — whether that’s developing new algorithms for machine knitting, creating novel design tools or even inventing new forms of visual art,” said senior author and Allen School professor Zach Tatlock, who co-advises Zhu alongside colleague Adriana Schulz. “It’s fascinating to see how the rigor of computer science can enhance the creativity of traditional crafts.”
Allen School Ph.D. student Shangbin Feng envisions the work of large language models (LLMs) as a collaborative endeavor, while fellow student Rock Yuren Pang is interested in advancing the conversation around unintended consequences of these and other emerging technologies. Both were recently honored among the 2024 class of IBM Ph.D. Fellows, which recognizes and supports students from around the world who pursue pioneering research in the company’s focus areas. For Feng and Pang, receiving a fellowship is a welcome validation of their efforts to challenge the status quo and change the narrative around AI.
Shangbin Feng: Championing AI development through multi-LLM collaboration
Shangbin Feng
For Shangbin Feng, the idea of singular general purpose LLM can be difficult, the same way there are no “general purpose” individuals. Instead, people have varied and specialized skills and experiences, and they collaborate with each other to achieve more than they could on their own. Feng adapts this idea into his research into multi-LLM collaboration, helping to develop a range of protocols enabling information exchange across LLMs with diverse expertise.
“I’m super grateful for IBM’s support to advance multi-LLM collaboration, challenging the status quo with a collaborative and participatory vision,” Feng said. “As academic researchers with limited resources, we are not powerless: we can put forward bold proposals to enable the collaboration of many in AI development, not just the resourceful few, such that multiple AI stakeholders can participate and have a say in the development of LLMs.”
Feng’s research focuses on developing different methods for LLM collaboration. He introduced Knowledge Card, a modular framework that helps fill in information gaps in general purpose LLMs. The researchers augmented these LLMs using a pool of domain-specialized small language models that provide relevant and up-to-date knowledge and information. Feng and his collaborators then proposed a text-based approach, where multiple LLMs evaluate and provide feedback on each others’ responses to identify knowledge gaps and improve reliability. Their research received an Outstanding Paper Award at the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024).
He also helped develop the framework Modular Pluralism that enables aggregation of community language models representing the preferences of diverse populations at the token probabilities level. Most recently, Feng proposed the collaborative search algorithm called Model Swarms. In this weight-level approach, diverse LLM experts collectively move in the parameter search space using swarm intelligence.
In future research, Feng plans to focus on ways to reprocess and recycle the over one million publicly available LLMs.
“It is often costly to retrain or substantially modify LLMs, thus reusing and composing existing LLMs would go a long way to reduce the carbon footprint and environmental impact of language technologies,” Feng said.
Rock Yuren Pang: Uncovering unintended consequences from emerging technologies
Rock Yuren Pang
As AI becomes more commonplace and integrated into different sectors of society, Rock Yuren Pang wants to help researchers and practitioners grasp the potential adverse effects of these emerging technologies.
“I work in the intersection of human-computer interaction (HCI) and responsible AI,” Pang said. “Through the IBM fellowship, I’ll continue to design systems and sociotechnical approaches for researchers to anticipate and understand the unintended consequences of our own research products, especially with fast-growing AI advancements. I’d like to change the narrative that doing so is a burden, but rather a fun and rewarding experience to communicate potential risks as well as the benefits for many diverse user populations.”
While tracking the downstream impacts of AI and other technologies can be overwhelming, Pang has worked with his Ph.D. advisor, Allen School professor Katharina Reinecke, to introduce tools to help make it more manageable. In a white paper dubbed PEACE, or “Proactively Exploring and Addressing Consequences and Ethics,” Pang and his collaborators propose a holistic approach that both makes it easier for researchers to access resources and support to predict and mitigate unintended consequences of their work, and also intertwine these concerns into the school’s teaching and research. Their work is supported through a grant from the National Science Foundation’s Ethical and Responsible Research (ER2) program.
He also helped develop Blip, a system that consolidates real-world examples of undesirable impacts of technology from across online articles. The system then summarizes and presents the information in a web-based interface, assisting researchers in identifying consequences that they “had never considered before,” Pang explained. Most recently, Pang has been investigating the growing influence of LLMs on HCI research.
Outside of his work in responsible AI, Pang is also interested in accessibility research. As part of the Center for Research and Education on Accessible Technology and Experiences (CREATE), he and his team developed an accessibility guide for data science and other STEM classes. Their work was honored last year as part of the UW IT Accessibility Task Force’s Digital Accessibility Awards. Pang also contributed to AltGeoViz, a system that allows screen-reader users to explore geovisualizations by automatically generating alt-text descriptions based on their current map view. The team received the People’s Choice Award at the Allen School’s 2024 Research Showcase.
In the future, Pang aims to design more detailed guidelines for addressing potential unintended consequences from AI, and novel interactions to collaborate with AI.
Allen School professor Amy X. Zhang (Photo by University of Washington)
As the head of the Allen School’s Social Futures Lab, professor Amy X. Zhang wants to reimagine how social platforms can empower end users and communities to take control of their own online experiences for social good. Her research draws on the design of offline public institutions and communities to then develop new social computing systems that can help online platforms become more democratic instead of top-down, and more customizable as opposed to one-size-fits-all.
These efforts were commended by the Alfred P. Sloan Foundation, which recognized Zhang among its 2025 class of Sloan Research Fellows. The fellowship highlights early-career researchers whose innovative work represents the next generation of scientific leaders.
“It’s an honor to be recognized for my work, so that we can further the impact our lab’s research has had across social computing and human-computer interaction (HCI) research communities,” Zhang said. “This fellowship will support my research into redesigning social and collaborative computing systems to give greater agency to users and further societally good aims.”
One of Zhang’s main lines of research focuses on participatory governance in online communities. Taking inspiration from the idea of “laboratories for democracy,” she has developed tools to help users have a greater say in the policies governing their online actions. For example, Zhang and her collaborators introduced PolicyKit, an open-source computational framework that enables communities to design, carry out and implement procedures such as elections, juries and direct democracy on their platform of choice such as Slack or Reddit. Their research has prompted follow-up projects including Pika, which enables non-programmers to author a wide range of executable governance policies, and MlsGov, a distributed version of PolicyKit for governing end-to-end encrypted groups.
At the same time, she is also interested in how governance can manifest at the platform level. Because platforms are much larger than communities and have millions of users, it becomes a challenge to maintain democratic systems, Zhang explained. To address this, Zhang focuses on workflows supporting procedural justice in platform-level content moderation design. She found that using digital juries for content moderation were perceived as more procedurally just compared to existing algorithms and contract workers, but overall, expert panels had the highest perception. Because of her expertise, Zhang was invited by Facebook to give input on their Community Review program and X (formerly known as Twitter) guidance on their Community Notes program for everyday users to weigh in on potential content violations.
Another line of Zhang’s research addresses the issue of content moderation, but from a different angle. Instead of platforms moderating content, Zhang builds tools to help users “decide for themselves how they would like to customize what content they do or do not want to see.” For example, when it comes to online harassment, Zhang found that platform-level solutions do not account for the different ways that people can define and want to address harassment. Zhang helped develop the system called FilterBuddy that helps content creators who face harassment create their own filters to moderate their comment sections. She has also introduced other personalized moderation controls, customized social media feed curation algorithms as well as content labels for misinformation.
Zhang’s other research interests lie in the space of public interest technology. This includes supporting the many people, often acting in a volunteer capacity, who take on civic roles online such as spreading important information or responding to misinformation. In their research, Zhang and her team found that these people may spend significant amounts of time, effort and emotional labor often without knowing if they are making a difference. To help support their work, the team developed an augmented large language model that can address misinformation by identifying and explaining inaccuracies in a piece of content. Zhang also helped interview fact-checkers on how they prioritize which claims to verify and what tools may assist them in their work. From this research, Zhang and her collaborators introduced a framework to help fact-checkers prioritize their efforts based on the harm the misinformation could cause.
In her ongoing and future research, Zhang plans to explore how offline institutional structures can also be useful for rethinking the governance of artificial intelligence technologies.
“My long-term goal is to build social computing systems that make our online spaces as rich and varied as our offline ones, while also striving for a more pro-social, resilient and inclusive society,” Zhang said.
Zhang, in addition to being named a Sloan Research Fellow, has received Best Paper Awards at the Association of Computing Machinery CHI conference on Human Factors in Computing Systems (ACM CHI) and ACM SIGCHI Conference on Computer-Supported Cooperative Work & Social Computing (ACM CSCW), a National Science Foundation CAREER Award and a Google Research Scholar Award.
Joining Zhang in the 2025 class of Sloan Research Fellows is Allen School alum Lydia Chilton (Ph.D. ‘16), now faculty at Columbia University. Her research focuses on HCI and how AI can help people with design, innovation and creative problem-solving.
Zhang is one of three University of Washington faculty to earn Sloan Research Fellowships this year. The other honorees are Amy L. Orsborn, a professor of electrical & computer engineering and bioengineering who earned a fellowship in the neuroscience category, and chemistry professor Dianne J. Xiao.
Each year, the Association for Computing Machinery (ACM) recognizes the top 1 percent of its members who have made notable contributions to the field of computing science and technology as ACM Fellows.
Allen School professors Brian Curless and Jeffrey Heer were among the 55 ACM fellows who will be recognized at a ceremony in June in San Francisco, California. The 2024 inductees’ contributions range from 3D scanning to interactive machine learning and much more. Their work has helped transform how we use computing technologies today.
Brian Curless: “For contributions to 3D shape and appearance capture and reconstruction and to computational photography.”
The ACM commended Curless for his “contributions to 3D shape and appearance capture and reconstruction and to computational photography.”
“It is an incredible honor to be named an ACM Fellow,” Curless said. “This recognition is as much mine as it is the amazing collaborators, both at the Allen School and beyond, that I’ve worked with over the years researching computational photography, 3D scanning and more.”
Curless’ work has helped lay the groundwork for major milestones in 3D scanning research. He invented what is now the standard for merging range images together to reconstruct 3D surfaces such as the ridges and scales in a detailed dragon model. In addition, previous methods for extracting range data from optical triangulation scanners, which use lasers to shine light on an object and then measure the reflection to estimate its shape, were often not accurate for surfaces that are curved, discontinuous or had varying degrees of reflectance. Curless introduced a new and more accurate method using spacetime analysis. His volumetric 3D reconstruction research has been used for feature film development, and has since then been incorporated into Google’s Tango augmented reality (AR) computing platform and Microsoft’s Hololens.
The volumetric reconstruction method was also key to the success of the Digital Michelangelo Project, the first, detailed 3D computer archive of the Renaissance artist’s sculptures including the statue “David.” The project led to more research into using 3D scanning for archaeological preservation.
His work in 3D scanning technology extends beyond digitizing artwork. Curless has pioneered methods for using large 3D datasets to create accurate human body models to fit the general population, ushering in new research in data-driven human body shape modeling. Further, he and collaborators combined 3D scanning with traditional photography to introduce a new paradigm — surface light fields — to concisely represent the view-dependent appearance of objects.
Outside of 3D shape scanning, Curless has made strides in computational photography. He helped develop the interactive digital photomontage method, a technique for combining multiple images together without visible seams. The technique was featured in Adobe Photoshop CS4 as well as in Google Maps to create satellite imagery. Its interface for stitching together the best parts of photos has led to Google Pixel phone’s “Best Take” feature and “Photomerge Group Shots” in Adobe Photoshop Elements 10.
In addition to being named an ACM Fellow, Curless has also received the National Science Foundation CAREER Award, Alfred P. Sloan Foundation Faculty Fellowship and the UW ACM Teaching Award.
“Brian is a pioneer in 3D shape scanning and computational photography,” said Seitz. “He invented techniques that became widely used both in academia and industry, and it’s amazing to see him recognized by this major honor.”
Jeffrey Heer: “For contributions to information visualization, human-centered data science and interactive machine learning.”
Jeffrey Heer
Allen School professor Jeffrey Heer, who co-directs the UW Interactive Data Lab alongside colleague Leilani Battle, has published over 120 peer-reviewed papers and has developed some of the most popular open-source visualization tools used by developers around the world. Heer’s interest in visualization began during his undergraduate studies at University of California, Berkeley, and since then, his work has already helped change the way that people interact with data, charts and graphs. But for Heer, “there’s still so much to do.”
“I’m honored to be named an ACM Fellow, and I am particularly grateful to my amazing students and collaborators over the years,” said Heer, who holds the Jerre D. Noe Endowed Professorship in the Allen School. “Whether visualizing complex information, wrangling data into shape, authoring analyses or making sense of statistical and machine learning results, together we’ve helped make it easier for people to work with, make sense of and communicate data more effectively.”
The ACM recognized Heer for “contributions to information visualization, human-centered data science and interactive machine learning.”
Heer is best known for his work on interactive information visualization. Building interactive visualizations requires knowledge in fields such as user interface development, graphic design as well as algorithmic implementation. Over the years, Heer and his collaborators introduced widely-adopted visualization languages such as Prefuse, Protovis, D3, Vega, Vega-Lite and Mosaic that have become the standard in fields like industry, data science and journalism.
“I am incredibly excited that Jeffrey Heer is being recognized by the ACM for the tremendous impact he has had in the areas of Visualization and Human-Computer Interaction,” said Maneesh Agrawala, computer science professor at Stanford University. “His work has opened new directions for research on things like narrative in visualization, data preparation for analysis and grammars for visualization. It has also had immense impact beyond research with his open source tools like D3 and Vega regularly used by developers worldwide.”
Since joining the Allen School faculty in 2013, Heer has received the ACM’s Grace Murray Hopper Award, the 2017 IEEE Visualization Technical Achievement Award and Best Paper Awards at the ACM Conference on Human Factors in Computing (CHI), EuroVis and IEEE InfoVis conferences. He was also inducted into the IEEE Visualization Academy in 2019 — one of the most prestigious honors in the field of visualization.
Shayan Oveis Gharan (Photo by Dennis Wise/University of Washington)
Allen School professor Shayan Oveis Gharan and Ph.D. alumnus Kuikui Liu, now a professor at MIT, are part of a team of researchers that received this year’s Michael and Sheila Held Prize from the National Academy of Sciences for introducing a new method for counting the bases of matroids. The annual prize honors “outstanding, innovative, creative and influential research” in the field of combinatorial and discrete optimization published in the past eight years.
In their winning paper “Log-Concave Polynomials II: High-Dimensional Walks and an FPRAS for Counting Bases of a Matroid,” the researchers bridge three different and unexpected subfields — Hodge theory for matroids and combinatorial geometries, Markov chains analysis and high dimensional expanders — to resolve a 30-year-old conjecture by Mihail and Vazirani, one of the most important open questions in the field of approximate counting. Their work has a number of real-world applications including network reliability, data transmission and machine learning and has led to other notable developments in theoretical computer science.
“We came up with a solution that not only answered an open problem many people had tried for years, but the answer was so simple, elegant, universal and easy to use that it changed the field,” Oveis Gharan said. “Because these different research communities have evolved independently, they have a lot of tools that the other one doesn’t have. Often, you can use tools from one community to answer questions in the other community — which is what we did.”
Oveis Gharan, Liu and their collaborators University of Washington Mathematics professor Cynthia Vinzant and Nima Anari from Stanford University introduced the first fully polynomial randomized approximation scheme (FPRAS) to count the number of bases of any matroid given by an independent set oracle. Their algorithm utilizes the simple two-step Monte Carlo Markov Chain (MCMC) method to address the Mihail-Vazirani conjecture which says that the bases exchange graph of any matroid has edge expansion of at least one.
This algorithm is especially useful in situations that require the assignment of resources or transmission of information such as network reliability problems. For example, if there was a network of routers connected by wires, this algorithm can help estimate the probability that the network gets disconnected based on which wires become independently damaged. These types of problems are usually solved using Markov chains to check if random subnetworks are connected.
However, in most cases it can be very difficult to figure out how long you should run the Markov chain to get the correct answer. The researchers developed a new tool which uses polynomials to analyze the Markov chains, answering the long-standing open problem.
In another example, a company has a certain number of jobs it needs completed along with a set of machines that can do a subset of those jobs. If you assume each machine can only be assigned one job and account for the probability of each machine independently failing, then this method can help generate an efficient algorithm that can approximate the probability that all the jobs will be finished with up to a 1 percent error margin.
“Since our research came out, there has been a sudden birth of so many papers using our techniques and finding new applications and answering other problems,” Oveis Gharan said. “It’s rare for a theoretician to feel an impact of their work.”
The Held Prize is not the first award Oveis Gharan, Liu and their collaborators have received for their research. The team received the Best Paper Award from the Association for Computing Machinery’s 51st annual Symposium on the Theory of Computing (STOC 2019).
“As far as I am concerned, Shayan’s combination of brilliance, creativity, relentless enthusiasm and generosity are unsurpassed. My collaborations with him have been one of the absolute highlights of my career,” Allen School professor Anna Karlin said.
Many students first develop their research skills in their Allen School courses. Professors Leilani Battle and Maya Cakmak, who co-chair the undergraduate research committee, designed a sequence of seminars that introduce students to research and give them the opportunity to work on a hands-on research project with a faculty or graduate student mentor.
“Participating in research allows students to exercise their creativity and apply their computing skills to advance science and technology, in a way that is not possible in their classes,” Cakmak said. “Undergraduate research is often the first step toward a research career and the Allen School is committed to enabling as many of our students to experience research and have those career opportunities.”
Research opportunities can help students realize new and unexpected applications for computer science.
“Research reveals a side of computer science that students might not see in a class or a typical internship, such as how computer science can lead to community-driven initiatives, exciting opportunities with nonprofit organizations and even founding your own startup company,” Battle said.
From helping farmers improve their networks to sell their products to developing innovative ways for children to interact with technology, these CRA-recognized students have shown that computer science research does not have to look like how you expect it to be.
Chun-Cheng Chang: Pushing the boundaries of wireless communication, robotics and AR
Chun-Cheng Chang
Chun-Cheng Chang’s passion for combining hands-on creativity with technical innovation began when he founded his middle school’s carpentry club. Since then, Chang has focused on developing cutting-edge “solutions that push the boundaries of wireless communication, robotics and augmented reality (AR) interfaces.”
“My research interest lies within the paradigm of low-power sensing and human-computer interaction (HCI). My low-power sensing work focuses on the practical solutions for battery-free systems that may be used in environmental monitoring and health sensing,” Chang said. “For HCI, I designed systems that benefit different applications, including improving robotics manipulation data collection, user experience and designing large language model (LLM) assistance systems for mobile health data analysis.”
Analog backscatter communication is promising for use in low-power wireless devices as it can send messages with less energy compared to popular methods such as Bluetooth. However, existing systems are limited in their transmission range and security. Instead, Chang, as part of the Allen School’s UbiComp Lab led by professor Shwetak Patel, helped develop an analog backscatter tag that reaches a transmitting range of up to 600 meters in outdoor line-of-sight scenarios, compared to state-of-the-art systems that can only transmit a maximum of 20 meters.
Chang has also turned his research skills toward building artificial intelligence (AI) assistants for mental health. The UbiComp Lab’s project looked at how LLMs could be used to interpret ubiquitous mobile health data from smartphones and wearables to generate mental health insights. Chang helped finetune and train three LLMs for mental health evaluation, and he conducted user study interviews with clinicians. The team’s study revealed an opportunity for clinicians and patients to use collaborative human-AI tools to explore self-tracking data. Their research was published in the Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT 2024).
In addition to wireless sensing systems and AI, Chang’s research spans the intersection between sensing and robotics. He helped design millimeter scale battery-free microrobots that can be used in environmental monitoring or exploration. When Chang joined the project, he noticed issues with the MilliMobile microrobot prototype’s precise motor and wheel alignment. To address this, Chang redesigned and improved the prototype’s schematics, printed circuit board and software. Chang’s changes helped the team, alongside collaborators at Columbia University, develop Phaser, a system framework that enables laser-based wireless power delivery and communication for microrobots.
As robots become more popular, Chang has also introduced software to help everyday users control the devices. He worked with professor Ranjay Krishna, who co-directs the Allen School’s RAIVN Lab, to design EVE, an iOS application that allows users to train robots using intuitive AR visualizations. With the app, users can preview the robot’s planned motion and see the robot arm’s reach-range boundary for the action. Chang and his collaborators presented their research at the 37th Annual ACM Symposium on User Interface Software and Technology (UIST 2024).
Ritesh Kanchi: Designing AI tools for accessibility and education
Ritesh Kanchi
Ritesh Kanchi was drawn to the HCI research community because it gave him “the opportunity to design technologies for those who benefit most; everyone stands to gain from HCI — it’s all about the humans.” His work as an undergraduate researcher tackles issues across HCI, computer science education and accessibility.
“Across my research experiences, I’ve defined my focus on developing Intelligent User Interfaces (IUIs) that empower niche communities in accessibility and education,” Kanchi said. “My work is rooted in creating context-aware, personalized and adaptive interfaces that address diverse user needs, making traditionally inaccessible experiences more inclusive.”
Since his first quarter at the University of Washington, Kanchi has been interested in designing new technology for children. He joined the Information School’s KidsTeam research group exploring how children use emerging generative AI tools to support, rather than take over, creative activities such as stories, songs or artwork. Their research received an honorable mention at the ACM CHI conference on Human Factors in Computing Systems (CHI 2024). In collaboration with the University of Michigan, Kanchi and his KidsTeam co-authors investigated how children understand computational data concepts such as “the cloud” and their privacy implications. They presented their paper at the 23rd annual ACM Interaction Design and Children Conference (IDC 2024) — the top conference in design of technology for children.
“How children interact with technology today influences the way that adults interact with technology tomorrow,” Kanchi said.
His HCI research continued at the Allen School’s Makeability Lab led by professor Jon Froehlich. Kanchi worked with Allen School Ph.D. student Arnavi Chheda-Kothary to investigate using AI for accessibility, especially among mixed visual-ability families. Many children express themselves through artwork that is primarily visual and non-tactile, such as using crayons, markers or paints, which can make it difficult for blind or low-vision (BLV) family members to experience and engage with the art.
To tackle this issue, Kanchi helped develop ArtInsight, a novel app that allows mixed visual-ability families to use AI to engage with children’s artwork. Previous research explored how AI can help BLV family members read with children, but none focused on art engagement. The app uses large language models to generate creative and respectful initial descriptions of the child’s artwork as well as questions to facilitate conversations between BLV family members and children about the art. For Kanchi, this research project was “a culmination of who I am as a researcher, combining my passions and prior research, including children’s design research, user interface development and accessibility.”
Kanchi’s passion for accessibility has even benefited his Allen School community. As a lead teaching assistant for CSE 121, the first course in the Allen School’s introduction to computer programming series, he converted course content to make it compatible with screen readers. He is working with Allen School professors Matt Wang and Miya Natsuhara on designing a system for creating accessible course diagrams in different content modalities including data structures such as binary trees and linked lists.
Alexander Metzger: Using mobile technologies for environmental sustainability and agricultural economic development
Alexander Metzger
From helping farmers in developing nations expand their networks to making it easier for users to understand the environmental impact of their electronics, Alexander Metzger has set his sights on making a global impact with his work.
“My research aims to bridge gaps in information access for underserved languages and communities through a combination of algorithm design, machine learning and edge device technology,” Metzger said.
During his first year at the Allen School, he joined the Information and Communication Technology for Development (ICTD) Lab, co-led by professors Richard Anderson and Kurtis Heimerl, to work on the eKichabi v2 study. Smallholder farmers across Tanzania often lack accessible networking platforms making them reliant on middlemen to sell their produce. The research team, alongside collaborators at Cornell University, collected the largest agricultural phone directory to date; however, they still faced the challenge of ensuring that the digital directory was accessible to users with limited internet and smartphone access.
To tackle this issue, Metzger helped develop and maintain an Unstructured Supplementary Service Data (USSD) application that allowed users to access the directory using simple menus on basic mobile phones. As the project advanced, he analyzed the USSD application’s technical performance to answer the question of how well an information system could address the divide between basic phones and smartphones. Metzger was the co-lead author on the paper examining the application’s HCI and economic challenges published at the ACM CHI Conference on Human Factors in Computing Systems (CHI 2024).
Continuing his research into mobile technologies, Metzger turned his attention toward addressing information access challenges that other smallholder farmers face. Smallholder farmers across countries including Kenya, India, Ethiopia and Rwanda often lack access to trustworthy sources to learn about sustainable farming practices and how to tackle issues such as crop disease. Metzger worked with Gooey.AI to help develop and deploy Farmer.CHAT, a GPT4-based, multilingual platform that helps extension workers with personalized advice and guidance. Farmer.CHAT presented their platform in front of the United Nation General Assembly.
Environmental decision making is not just a problem for farmers. As a member of the Allen School’s UbiComp Lab, Metzger is working on developing multi-modal hybrid machine learning and computer vision algorithms that can make environmental impact estimates more accessible. He is also designing a battery-free balloon and glider-based sensor systems that provide developing countries with the data to help predict and prevent environmental disasters, including wildfires.
Outside of his Allen School work, Metzger recently founded Koel Labs, a research-focused startup that trains open-source audio models to help language learners improve their pronunciation.
Kenneth Yang: Removing inefficiencies in software engineering and neuroscience
Kenneth Yang
When Kenneth Yang solves a problem — whether it is a tricky programming puzzle or a neuroscience research headache — his first thought is, “there has to be a better way to do this.”
“Because our lives are technology-infused, I turned to computer science as a tool to help me solve problems efficiently, predictably and accurately,” Yang said. “My research focuses on identifying and solving inefficiencies in existing processes and aims to improve algorithms and methods used in computer graphics, software engineering and neuroscience.”
As a research assistant to Allen School professor Michael Ernst in the Programming Languages and Software Engineering (PLSE) group, Yang is turning his problem-solving skills toward the challenge of merging code. Version control systems are merge tools that allow multiple developers to work on a code at the same time and then combine their edits into a single version. While there are many algorithms and merge tools available to automatically handle merging code, they often fail — leading to unexpected bugs and developers having to turn their attention to manually resolving and integrating changes.
However, these different merge tools, including new approaches, have not been evaluated against each other. The limited experiments that have been performed have excluded systems such as Git, the dominant version control system, or have only measured the benefits of correct merges, not the costs of incorrect ones. Yang and his collaborators addressed these problems and showed that many earlier merge tools are less effective than previously thought. They presented their results in their paper “Evaluation of Version Control Merge Tools” that appeared at the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE 2024).
Yang is also using his software skills to help neuroscientists accelerate their research. Neuroscientists gather brain data using electrophysiology, where they insert probes into the brain to detect neurons’ electrical activity. The small scale of different brain regions alongside other factors can make the process difficult and hard to use in larger studies.
Instead, Yang’s research with the Virtual Brain Lab group within the UW Department of Neurobiology & Biophysics’ Steinmetz Lab develops software and tools that can make electrophysiology experiments more efficient, repeatable and automated. He co-developed the software platform Pinpoint that allows neuroscientists to interactively explore 3D brain models, plan experiments and control and maneuver the robotic electrode manipulators into the correct location.
To work alongside Pinpoint, he designed and wrote the Electrophysiology Manipulator Link, or Ephys Link, a unifying communication platform for popular electrophysiology manipulators that translates planning data into specific robotic movements. Yang’s tools have reduced the insertion process from 15 minutes per probe to 15 minutes total and with minimal researcher intervention. In future research, he aims to enable researchers to fully automate electrophysiology and make complex multi-probe studies possible.
In addition to honoring these four undergraduates at UW, the CRA recognized another student with an Allen School connection. Awardee Gene Kim, an undergraduate student at Stanford University, has collaborated with Allen School professor Jen Mankoff on the development of assistive technology to help make data visualization, personal fabrication and design tools more accessible for blind and visually impaired people.
Despite their growing potential and increasing popularity, large language models (LLMs) often produce responses that are factually inaccurate or nonsensical, also known as hallucinations.
Allen School Ph.D. student Akari Asai has dedicated her research to tackling these problems with the use of retrieval-augmented language models, a new class of LLMs that pull relevant information from an external datastore with a query that the LLM generates. For being a pioneer in Artificial Intelligence & Robotics, Asai was recognized as one of MIT Technology Review’s 2024 Innovators Under 35 Japan. The award honors young innovators from Japan who are “working to solve global problems.”
“Being named to MIT Technology Review 35 Under 35 is an incredible honor,” said Asai, who is a member of the Allen School’s H2 Lab led by professor Hannaneh Hajishirzi. “It highlights the power of collaboration and the potential of AI to address real-world challenges. With the rapid adoption of LLMs, the need to investigate their limitations, develop more powerful models and apply them in safety-critical domains has never been more urgent. This recognition motivates me to keep working on projects that can make a meaningful impact.”
Asai’s paper on adaptive retrieval-augmented LMs was one of the first to show that retrieval-augmented generation (RAG) is effective at reducing hallucinations. Without using RAG, traditional LLMs generate responses to user input based on the information it was trained on. In contrast, RAG adds an information retrieval component to the LLM that uses the user input to first pull information from a new, external data source so the LLM can generate responses that incorporate the latest information without needing additional training data.
As part of the study, Asai and her team compared the response accuracy from various conventional and retrieval-augmented language models across a dataset of 14,000 wide-ranging questions. They found that augmentation methods such as RAG significantly and more efficiently improved the performance of LMs compared to increasing their training data and scaling up models. Asai and her collaborators presented their work at the 61st Annual Meeting of the Association for Computational Linguistics (ACL 2023), where they received the Best Video Award.
Building off of that research, Asai has helped develop the foundational components of retrieval-augmented language models along with improved architectures, training strategies and inference techniques. In 2024, she introduced a new framework called Self-reflective RAG, or Self-RAG, that enhances a model’s quality and factual accuracy using retrieval and self-reflection. While RAG only retrieves relevant information once or a fixed number of time steps, Self-RAG can retrieve multiple times, making it useful for diverse downstream queries such as instruction following. The research received the best paper honorable mention at the 37th Annual Conference on Neural Information Processing Systems (NeurIPS) Instruction Workshop and was evaluated at the top 1% of papers at the 12th International Conference on Learning Representations (ICLR 2024).
Asai is also interested in advancing the ways that retrieval-augmented language models can tackle real-world challenges. Recently, she launched Ai2 OpenScholar, a new model designed to help scientists more effectively and efficiently navigate and synthesize scientific literature. She has also explored how retrieval augmentation can help with code generation and helped develop frameworks that can improve information access especially among linguistic minorities. The latter includes Cross-lingual Open-Retrieval Answer Generation (CORA), Cross-lingual Open Retrieval Question Answering (XOR QA) and AfriQA, the first cross-lingual question answering dataset focused on African languages. In 2022, she received an IBM Ph.D. Fellowship to advance her work to ensure anyone can find what they need efficiently online including across multiple languages and domains.
In future research, Asai aims to tackle other limitations that come with LLMs. Although LLMs are becoming increasingly useful in fields that require high precision, they still face challenges such as lack of attribution, Asai explained. She aims to collaborate with domain specialists “to create systems that experts can truly rely on.”
“I’m excited about the future of my research, where I aim to develop more efficient and reliable AI systems. My focus is on creating models that are not only scalable but also transparent, making AI more accessible and impactful across diverse fields,” Asai said.
GEN1 members tabled and gave out self-care packages for National First-Gen Day.
About a quarter of all Allen School students are first-generation or first-gen, meaning that they are one of the first in their family to pursue a Bachelor’s degree in the U.S., according to Allen School demographics data.
In 2020, a group of first-gen Allen School students including Aerin Malana (B.S. ‘22) approached Chloe Dolese Mandeville, the Allen School’s Assistant Director for Diversity & Access, and Leslie Ikeda, who manages the Allen School Scholars Program, wanting to build a club to help support and recognize their fellow first-gen students. Today, the club called GEN1 has helped first-gen students find community, connect with resources at the Allen School and University of Washington and more — all with the goal of highlighting the first-gen experience.
“A really important goal of GEN1 was to celebrate that being a trailblazer and coming into this space has a lot of weight, and also there’s a lot of joy and things to celebrate in that journey,” said former GEN1 adviser Ikeda.
Fostering community
For Ikeda, who was also a first-gen student, the experience can feel like an “invisible identity” that comes with unique challenges.
“The first-generation experience can feel very isolating, especially coming into a larger institution and a competitive program such as the Allen School,” Ikeda said. “If you don’t have the social capital or the networks, you can feel really lost.”
GEN1 hosted a coding game night in collaboration with the student organization Computing Community (COM2).
GEN1’s goal is to make the first-gen community more visible and provide members with a space to share their stories with other first-gen students. Many students come to the club looking for others who understand their experience.
“I joined GEN1 because I was looking for a computer science and engineering community who have similar backgrounds as me,” said Christy Nguyen, Allen School undergraduate student and GEN1 vice chair. “I felt really behind compared to my peers when starting my computer science journey because I couldn’t ask my parents about coding-related difficulties. I’m really happy that I’m in GEN1 because we share these experiences and help each other in times when our parents can’t.”
Over the years, the club has led multiple initiatives and programs to help students thrive in their academics, future careers and their overall wellbeing. These include bi-weekly study sessions, an alumni mentorship program, an internship program to onboard new club officers and self-care packages during midterms and finals. GEN1 also hosts social events such as pumpkin painting to help students destress and unwind. At the same time, GEN1 also collaborates with other groups at the Allen School such as Women in Computing, recognizing how “intersectional” the first-gen identity can be, Ikeda said.
Czarina Dela Cruz, Allen School undergraduate student and GEN1 chair, has been involved with GEN1 since her freshman year and ran for the position to provide other students with the same sense of community that welcomed her in.
GEN1 members at the First-Gen Graduation celebration for the class of 2024.
“As someone who came into the Allen School without a strong technical background, joining GEN1 has helped me find a community who I can rely on for advice, laughs and connections,” Dela Cruz said.
Dela Cruz said her goal for this year as the GEN1 chair is to “increase GEN1’s engagement and reach all first-gen students in the Allen School, to encourage and support as they go along their journey in computing and beyond.” For example, to celebrate National First-Generation Day on Nov. 8, GEN1 hosted a career night featuring companies such as Google and Microsoft along with technical interview workshops. The holiday commemorates the signing of the Higher Education Act, ushering in programming to support first-gen college students.
But National First-Generation Day is not the only time during the year when the Allen School highlights first-gen students. Shortly after the club started, GEN1 began hosting a first-generation Allen School graduation ceremony. The event has grown beyond just a celebration for the graduates, but a chance for the Allen School community to come together and show their support.
“This celebration aspect of GEN1 has been really impactful,” Dolese Mandeville said. “Being here at the Allen School is a huge accomplishment, but it’s also important to highlight everything they do beyond. Having GEN1 as a space for first-gen students is amazing, but I hope they know that everyone else is also rooting for them to succeed.”
Each year, the InfoSys Science Foundation (ISF) recognizes the achievements and contributions of researchers and scientists of Indian origin who are making waves in their field and beyond.
Allen School professor Shyam Gollakota, who leads the Mobile Intelligence Lab, received this year’s Infosys Prize in Engineering and Computer Science for his research that uses artificial intelligence to change the way we think about speech and audio. He is among one of six award winners who will be honored at a ceremony in Bangalore, India, next month and receive a prize of $100,000.
“This prize supports our work on creating a symbiosis between humans, hardware and AI to create superhuman capabilities like superhearing, with the potential to transform billions of headphones, AirPods, and improves the lives of millions of people who have hearing loss,” said Gollakota, the Washington Research Foundation/Thomas J. Cable Professor in the Allen School.
The award is one of the largest in India recognizing science and research excellence. This year, the ISF decided that the award will honor researchers younger than 40, “emphasizing the need for early recognition of exceptional talent,” the organization said in a statement.
For the past few years, Gollakota has been building innovative ways to boost the power of headphones using AI. Most recently, he developed a prototype for AI-powered headphones that create a “sound bubble” around the wearer. The headphones use an AI algorithm that allows the wearer to hear others speaking inside the bubble, while sounds outside of it are quieted. Gollakota said that the award money from the InfoSys Prize will go toward commercializing the technology.
These AI-enabled software that work in real-time can be difficult to run on smaller devices like headphones due to size and power restraints, however, Gollakota helped create knowledge boosting. The system can increase the performance of the small model operating on headphones using the help of a remote model running on a smartphone or in the cloud.
“His work on mobile and wireless communications is game-changing,” said Jayathi Murthy, Engineering and Computer Science Infosys Prize jury chair. “Particularly impressive is his work on active sonar systems for physiological sensing, battery-free communications and the use of AI to selectively tailor acoustic landscapes. These innovations will continue to benefit humanity for years to come.”
Allen School Ph.D. student Joe Breda holds a smartphone against a patient’s head to show how the FeverPhone app works. Dennis Wise/University of Washington
When you need to test if you are running a fever, you may not have an accurate at-home thermometer handy. However, many people may have the next best thing right in their pocket — a smartphone.
A team of researchers in the Allen School’s UbiComp Lab and UW School of Medicine developed the app FeverPhone that turns smartphones into thermometers without the need for additional hardware. The research, which was originally published in the Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, received an IMWUT Distinguished Paper Award at the ACM International Joint Conference on Pervasive and Ubiquitous Computing/International Symposium on Wearable Computing (UbiComp/ISWC) in Melbourne, Australia, in October.
Joe Breda
“It feels great to have this work spotlighted by the community like this because it brings light to the still unexpected utility inside these — already ubiquitous — devices,” lead author Joe Breda, a Ph.D. student in the Allen School, said. “I like to do this type of research as it demonstrates what can be done with the sensor hardware already in people’s pockets so the next generation of devices might include these kinds of techniques by default. This is particularly important for applications like health sensing where ensuring access to diagnostics at the population scale can have serious impacts.”
The app is the first to use smartphones’ existing sensors to gauge whether or not someone has a fever. Inside most off-the-shelf smartphones are small sensors called thermistors that monitor the temperature of the battery. These sensors happen to be the same ones clinical-grade thermometers use to estimate body temperature. The researchers found that the smartphone’s touchscreen could sense skin-to-phone contact, while the thermistors could estimate the air temperature and the rise in heat when the phone was pressed against someone’s forehead.
The team tested out FeverPhone’s temperature-sensing capabilities against a traditional oral thermometer on patients at the UW School of Medicine’s Emergency Department. FeverPhone’s readings were within the clinically acceptable range. Although the app still needs more testing before it can be widely used, FeverPhone’s potential to help during sudden times of demand when thermometers may be less available is still exciting for doctors.
“People come to the ER all the time saying, ‘I think I was running a fever.’ And that’s very different than saying ‘I was running a fever,’” said study co-author Mastafa Springston, M.D., in a UW News release. “In a wave of influenza, for instance, people running to the ER can take five days, or even a week sometimes. So if people were to share fever results with public health agencies through the app, similar to how we signed up for COVID exposure warnings, this earlier sign could help us intervene much sooner.”
Since the team published the paper last year, some smartphone makers have introduced their own body temperature sensors.
“For example, it’s not unlikely that this paper directly inspired Google Pixel to introduce this new functionality, which was exactly why I pursued this work in the first place,” Breda said. “I wanted to advocate for these big tech companies to consider minimal hardware or software changes to these ubiquitous devices to make health care more accessible. I actually met with someone on the team shortly after this work was submitted to share my findings.”
Breda has turned his attention toward other devices with potential health care capabilities. For example, he is currently researching how smartwatches can be used for early detection of influenza-like illnesses. Breda has been collaborating with the National Institutes of Health (NIH) to build models that can passively detect if the wearer is getting sick, starting on the first day of virus exposure, using signals from their heart rate, temperature and skin conductance. Last October, he traveled to Washington, D.C. to test the technology on patients at the NIH.
“In the next phase of my career, I am looking at how ubiquitous computing and artificial intelligence-powered sensing on these devices can improve public health through analyzing biomarkers for disease prevention, and even more broadly through improving urbanism and digital civics,” Breda said. “These devices offer a unique opportunity to detect, or in some cases even predict, personal and public health issues before they even happen.”
