The team onstage at AAAI 2020 (from left): conference program co-chair Vincent Conitzer, Ronan Le Bras, Yejin Choi, Chandra Bhagavatula, Keisuke Sakaguchi, and conference program co-chair Fei Sha
Datasets like the Winograd Schema Challenge (WSC) are used to measure neural models’ ability to exercise common-sense reasoning. They do this by testing whether they can correctly discern the meaning of pronouns used in sentences describing social or physical relationships between entities or objects based on contextual clues. These clues tend to be easy for humans to comprehend but pose a challenge for machines. The models are fed pairs of nearly identical sentences that primarily differ by a “trigger” word, which flips the meaning of the sentence by changing the noun to which the pronoun refers. A high score on the test suggests that a model has achieved a level of natural language understanding that goes beyond mere recognition of statistical patterns to a more human-like grasp of semantics.
But the WCS, which consists of 273 problem sets hand-written by experts, is susceptible to built-in biases that paint an inaccurate picture of a model’s performance. Because individuals have a natural tendency to repeat their problem-crafting strategies, they also have a tendency to introduce annotation artifacts — unintentional patterns in the data — that reveal information about the target label that can skew the results of the test.
For example, if a pair of sentences asks the model to determine whether a pronoun is referring to a lion or a zebra based on the use of the trigger words “predator” or “meaty,” the model will note that the word “predator” is often associated with the word “lion.” In a similar fashion, a reference to a tree falling on a roof will lead the model to correctly associate the trigger word “repair” with “roof,” because while there are very few instances of trees being repaired, it is quite common to repair a roof. By choosing the correct answers to these questions, the model is not indicating an ability to reason about each pair of sentences. Rather, the model is making its selections based on a pattern of word associations it has detected across the dataset that just happens to correspond with the right answers.
“Today’s neural models are adept at exploiting patterns of language and other unintentional biases that can creep into these datasets. This enables them to give the correct answer to a problem, but for incorrect reasons,” explained Choi, who splits her time between the Allen School’s Natural Language Processing group and AI2. “This compromises the usefulness of the test, because the results are not an accurate reflection of the model’s ability. To more accurately assess the state of natural language understanding, we came up with a new solution for systematically reducing these biases.”
That solution enabled the team to produce WinoGrande, a dataset comprising 44,000 sentence pairs that follow a similar format to that of the original WSC. One of the shortcomings of the WSC was its relatively small size owing to the need to hand-write the questions. Choi and her colleagues got around that difficulty by crowdsourcing question material using Amazon Mechanical Turk, following a carefully designed procedure to ensure that problem sets avoided ambiguity or word association and covered a variety of topics. To eliminate any unintentional biases embedded in the dataset at scale, the researchers developed a new algorithm, dubbed AFLite, that employs state-of-the-art contextual representation of words to identify and eliminate annotation artifacts. AFLite is modeled on an existing adversarial filtering (AF) algorithm but is more lightweight, thus requiring fewer computational resources.
The team hoped that their new, improved benchmark would provide a clearer picture of just how far machine understanding has progressed. As it turns out, the answer is “not as far as we thought.”
“Human performance on problem sets like the WSC and WinoGrande surpass 90% correctness,” noted Choi. “State-of-the-art models were found to be approaching human-like levels of accuracy on the WSC. But when we tested those same models using WinoGrande, their performance dropped to between 59% and 79%.
“Our results suggest that common sense is not yet common when it comes to machine understanding,” she continued. “Our hope is that this sparks a conversation about how we approach this core research problem and how we design the benchmarks for assessing future progress in AI.”
In the latest Allen School undergrad spotlight, Nathan Wacker, can proudly say he’s helped build something that is truly out of this world. The third-year Allen School student from Seattle worked on HuskySat-1, a 3U CubeSat that was launched into space on November 2, 2019 and left Northrup Gruman’s Cygnus cargo spacecraft on January 31.
Allen School: What interested you in working on HuskySat-1, and what was your job in the Husky Satellite Lab?
Nathan Wacker: I was interested in joining because I wanted to work on something that was going to space. It still feels surreal to say that I have done it. Working at the lab seemed more tangible than most of the programing I had done up to that point.
Since I joined the team in the fall of 2017, I have worked on the flight software for the power distribution system, plasma thruster and various other systems on the spacecraft. I also worked quite extensively on our ground station command and control software. Since we are now in space, I have been maintaining the ground station and leading satellite operations.
Allen School: What was it like watching the cargo craft launch from NASA’s Wallop Flight Facility last November, knowing HuskySat-1 was on it?
NW: Folks from the lab that weren’t able to attend the launch in person, myself included, gathered in a basement classroom at 6:30 a.m. Saturday, November 2 to watch the NASA stream live. There was a lot of excitement in the room as the rocket disappeared into the clouds, but it was hard not to think about the failure scenarios and how we wouldn’t know until months later whether our satellite had survived.
Allen School: What kind of information have you learned since the satellite’s deployment?
NW: Deployment from Cygnus occurred on January 31 and we made first contact several hours later. Since then, we have determined that the power system is healthy, the primary communications system works, and that we are sitting at a cozy temperature — for space. We have also commissioned the camera and taken some low-resolution images of earth.
Allen School: What are some unique lessons you learned while on the satellite team that you might not have experienced anywhere else?
NW: A few lessons I have learned are: Don’t prematurely optimize, don’t over-engineer, document early and often, test everything. Also, the ability and drive to learn is more valuable than knowledge. These lessons all came from working on a long-term and large-scale project, which is difficult to teach in a classroom.
Allen School: Several academic areas would allow you to work on a project like HuskySat-1, why did you choose to major in computer science?
NW: Computer programming has always been appealing because smaller-scale projects can be put together quickly and iterated on at no cost other than the computer. Growing up, that was a much quicker path to gratification than woodworking or electronics projects, for instance, but still scratched an itch of mine to build something.
Allen School: What do you like most about being in the Allen School?
NW: The education is excellent. As a student, it is extremely rewarding to take classes from people who have significantly advanced their field and still care about student success. Upper-division CSE classes are the most interesting classes I have taken at UW.
Allen School: What are some of your favorite activities or experiences here at the UW?
NW: The Husky Satellite Lab has by far been my most fulfilling experience here at UW. Not only has it been a great engineering experience that has informed my career interests, but I have also had the pleasure of getting to know the talented individuals I work with in a professional and social context.
Read more about HuskySat-1 and follow the live orbital tracker to stay up-to-date on the satellite’s mission. We are proud to have Nathan as a member of the Allen School community!
Portrait of Rosalind Franklin by Seattle artist Kate Thompson. Dennis Wise/University of Washington
British scientist Rosalind Franklin, who spent the early 1950s researching the structure of DNA at King’s College London, should have won the Nobel Prize. She very well may have, except that her untimely death from ovarian cancer at the age of 37 meant that the Nobel Committee, which does not award posthumously, did not even consider her. For it was Franklin, not the famous scientific duo Watson and Crick, who captured the first image proving the shape of deoxyribonucleic acid — better known as DNA, the building block of all life.
At the time, Franklin was applying her expertise in x-ray crystallography to determine the structure of DNA in collaboration with King’s College Ph.D. student Raymond Gosling. The researchers captured an image of moistened DNA fibers using x-ray diffraction techniques and equipment refined by Franklin herself. The so-called Photo 51, which revealed a helical shape consisting of two strands, and other unpublished data from Franklin’s lab would find their way into the hands of fellow scientists James Watson and Francis Crick at Franklin’s alma mater, the University of Cambridge. The material confirmed the three-dimensional structure of DNA as a double helix — a structure the pair would race to describe in a paper published in April 1953 in the journal Nature.
Although Franklin would publish her own paper co-authored with Gosling on the double helix in the same issue, it was Watson and Crick, along with Franklin’s King’s College colleague Maurice Wilkins, who went on to share the 1962 Nobel Prize in Physiology or Medicine and thus ensure their places in the pantheon of scientific achievement. The contributions of Franklin, who had passed away 4 years before the Nobel announcement, would only begin to be more widely appreciated decades later. In any event, the Nobel Prize can only be shared by up to 3 individuals, so we will never know whether Franklin, had she lived long enough, would have received her due. Given the limitation on sharing the prize and the attitudes toward women in science at that time — and her own colleagues’ attitudes toward Franklin, in particular — it seems unlikely.
More than 60 years after Watson and Crick laid eyes on Photo 51, Allen School professor Luis Ceze met Seattle-based multimedia artist Kate Thompson in a bar not far from the University of Washington campus. Ceze co-directs the Molecular Information Systems Laboratory, a partnership between UW and Microsoft that is exploring synthetic DNA’s potential as a long-term storage solution and computational platform for digital data. He had already crossed paths with Thompson on campus, where she was doing an artist’s residency in the Nemhauser Lab focused on evolutionary biology. Ceze was intrigued by what she told him about her work, which is focused on making science visual.
Kate Thompson at work in her studio. Mary Bruno
For more than a year, he and his colleagues had been collecting images submitted by people around the world as part of the #MemoriesInDNA project under the tagline “What do you want to remember forever?” After meeting Thompson, Ceze became interested in exploring a way to ensure that the world would remember Franklin and her contributions forever — and he had a particularly fitting medium in mind.
“Rosalind Franklin was largely responsible for uncovering the structure of DNA, nature’s own perfected storage medium. Her work opened up a whole new avenue of scientific research and discovery for which, to this day, she does not really get the credit that she deserves,” Ceze said. “We had this massive collection of image files signifying what people want to preserve for posterity, and this new storage method. So we thought, why not use them to demonstrate the science we’ve been working on while paying tribute to the scientist who started it all?”
The idea he and Thompson hashed out over glasses of wine was elegantly simple. The lab would work with Thompson to create a piece of art to commemorate Franklin that incorporates the very medium that she revealed to Watson, Crick, and the world: DNA. To be precise, the medium would be synthetic DNA containing thousands of copies of images people had voluntarily contributed to advance a new wave of molecular systems research.
“I was fascinated by the idea of honoring this brilliant but mostly forgotten woman using the same material that should have made her famous in her own lifetime,” said Thompson, who took on the role of artist-in-residence at MISL after meeting with Ceze. “Until recently, her legacy to science and the world went largely unnoticed. I hope this project helps to ensure it will not be ignored.”
Dennis Wise/University of Washington
The result of Thompson’s collaboration with the MISL is now on display in the Bill & Melinda Gates Center for Computer Science & Engineering on the UW Seattle campus. The work, which measures 40 inches high by 30 inches wide and was created using acrylic ink on archival paper, took nearly 8 months from conception to completion. It is one of three original copies Thompson produced at the behest of the lab.
Viewed at a distance, the work is clearly a portrait of Franklin. Thompson reproduced her likeness from an old black and white photograph, combining soft, dark brush strokes with a mosaic of nearly 2,000 meticulously arranged images — the majority of which measure just ⅞ inch square — from the #MemoriesInDNA collection. The artist arranged the latter with the help of a macro she wrote for the purpose, which sifted through the images and sorted them by tonal values.
The images that make up Franklin’s face are contained within a larger collection that serves as a colorful backdrop for the subject of the painting. Up close, the images themselves come into sharper relief; if they linger long enough, viewers can make out the contents of the individual photos in detail. Each image depicts a person, place or object someone wants remembered in perpetuity.
But Thompson didn’t just paint on the images; she painted with them, infusing Franklin’s likeness with the actual data files ensconced in their microscopic storage medium. Using roughly half of the trove of more than 10,000 photos submitted as part of #MemoriesInDNA, MISL researchers first converted the digital image files — around 325 megabytes of data — into the As, Ts, Cs and Gs of DNA and encoded them into synthetic DNA ordered from Twist Bioscience. Thompson then took the vial of DNA furnished by the lab, which consisted of a mere 1.5 milliliters of liquid, and mixed the contents with black acrylic ink along with a binding substance that would help it adhere to the paper. She combined yet more of the image-laden DNA with a clear acrylic hardener, which she used to coat the finished piece.
“I painted several practice portraits using plain ink first,” the artist recalled with a laugh. “I didn’t want to mess up when it came time to use the real thing. Ink is relatively inexpensive but specially encoded DNA from a lab, not so much!”
The process the lab and artist followed to turn digital photos into a DNA-infused painting. Kate Thompson
Before Thompson could pick up her brush, lab members David Ward, Bichlien Nguyen, Xiaomeng Liu, and Jeff Nivala conducted experiments to ensure that, in the latter’s words, “the science was as rigorous as the art.” First and foremost, they needed to establish that no chemical reaction would occur between the synthetic DNA and acrylic medium when Thompson mixed the two in her studio. The team also wanted to be confident that, once mixed and applied to paper, the DNA could be subsequently retrieved from the material. The outcome is both a work of art and an artifact of science.
“Not that I’m suggesting you do this — in fact, please don’t! — but if you were to scrape a little bit of the portrait off, with the right equipment you could retrieve the data and convert it back from DNA molecules to digital 0s and 1s,” explained MISL co-director Karin Strauss, principal research manager at Microsoft Research and affiliate professor at the Allen School. “This portrait is not only preserving Franklin’s memory but preserving the data as well, in a form that will be accessible to future generations.”
To increase the likelihood that the data contained in the work will, indeed, be accessible for generations to come, the team built in a high degree of redundancy. Nguyen estimates it took 30 minutes to amplify copies of each image file using polymerase chain reaction, or PCR.
“Because DNA as a storage medium is so dense, we were able to provide Kate with around a trillion copies of each image to mix into the paint,” explained Nguyen, a senior researcher at Microsoft Research who oversaw the DNA storage process. “That way, we can be certain that we will be able to retrieve all of the data — even if a portion of one set of images is somehow lost or damaged, we still have many back-ups.”
This is not the first time that the MISL team has applied its science to the arts. The lab previously partnered with Twist Bioscience to preserve significant cultural and historical artifacts in DNA as a way of demonstrating its potential for archival data storage and retrieval, including iconic musical performances at the Montreux Jazz Festival, the top 100 books of Project Gutenberg, the Universal Declaration of Human Rights in 100 languages, and the non-profit Crop Trust’s entire seed database. Along the way, the team set a new record for the amount of digital data stored in and successfully retrieved in DNA that appeared in a peer-reviewed journal, presented new techniques for random access, developed a new platform for microfluidics automation for DNA data storage at scale, and demonstrated the world’s first end-to-end automated system for encoding digital data in DNA.
Members of the team with the finished portrait, left to right: Luis Ceze, David Ward, Bichlien Nguyen, Kate Thompson, and Karin Strauss (not pictured: Xiaomeng Liu, Jeff Nivala). Dennis Wise/University of Washington
Members of the public may view the portrait of Franklin on the ground floor of the Bill & Melinda Gates Center, at the base of the Anita Borg Grand Stairway. The lab is also exploring the potential to exhibit the artwork in additional locations in the future to reach an even wider audience.
“Our team was excited to partner with Kate on this project, which highlights an often forgotten figure who helped usher in the age of molecular storage,” Ceze said. “I hope the artwork itself, the science upon which it’s based, and the story of Rosalind Franklin will inspire people.”
Read more about Thompson’s work here, and learn more about the MISL’s work on DNA data storage here. Read the original #MemoriesInDNA announcement here. The lab members and the artist would like to express their appreciation to the thousands of people, spread out over 80 countries, who shared their personal photos as part of the #MemoriesInDNA project. The lab will continue to use the images as part of its research to advance DNA-based computation and image search.
“I am thrilled that the Sloan Foundation has honored Hanna and Yin Tat for their outstanding work on fundamental problems that have broad relevance and potential for impact,” said professor Magdalena Balazinska, director of the Allen School. “Hanna is working at the leading edge of artificial intelligence to transform the way we conceive of and build AI systems that touch people’s everyday lives, from education and media, to financial services and scientific documents. And Yin Tat is doing groundbreaking — even audacious — work that pushes past decades-old limits of computing to create faster, better solutions to a range of modern-day problems.”
Hanna Hajishirzi
Hajishirzi, who joined the Allen School faculty in 2018 and is also an AI research fellow at the Allen Institute for Artificial Intelligence (AI2), addresses foundational problems in natural language processing, artificial intelligence, and machine learning. Her goal is to develop general-purpose algorithms that can represent, comprehend, and reason about diverse forms of data efficiently and on a large scale. Hajishirzi’s research spans multiple domains, including representation learning, question answering, knowledge graphs, and applications such as conversational dialogue and knowledge extraction from unstructured text.
“Enormous amounts of information are available online in multiple forms across diverse resources; for example, in news articles, web pages, textbooks and technical documents,” explained Hajishirzi. “An important challenge in AI is how to represent and integrate diverse resources to facilitate further comprehension and reasoning. It is the right time to address this challenge at large scale and in real-world settings, using a unified representation that combines the best features of deep neural models and symbolic formalisms.”
Hajishirzi is among the pioneers in designing novel, end-to-end neural models for question answering and reading comprehension. One of her key contributions is Bi-Directional Attention Flow for Machine Comprehension, or BiDAF, which is a deep neural model for end-to-end question-answering about text and diagrams that has been widely adopted in academia and industry. Hajishirzi and her collaborators designed the system to be both scalable and modular, thus enabling its use with multiple modalities and knowledge bases. Hajishirzi is also among the first to address the problem of understanding scientific articles and data across multiple modalities, such as diagrams, math and geometry word problems. For example, she led the development of DyGIE, a system for enabling knowledge extraction from computer science and biomedical scientific papers. She also led the GeoS project, the first automated system for solving geometry word problems that can answer SAT geometry test questions on a par with the average American 11th grade student. More recently, Hajishirzi devised a new interpretable neural model for solving math problems, MathQA, that maps word problems to operation programs, and DenSPI and DecompRC, systems for real-time and multi-hop question answering that achieves state-of-the-art results by decomposing compositional questions into simpler sub-questions.
Hajishirzi has garnered numerous accolades for her research. She received an Allen Distinguished Investigator Award in AI for her work on the Spoon Feed Learning (SPEL) framework that combined principles of child education and machine learning to enable computers to interpret diagrams. Hajishirzi later earned a Google Faculty Research Award for her efforts to develop practical, scalable methods for open-domain question answering. She has also received an Amazon Research Award, a Bloomberg Data Science Award, and a Best Paper Award from the Special Interest Group on Discourse and Dialogue (SIGDIAL). Hajishirzi regularly publishes at the top conferences in the field, including the annual meeting of the Association for Computational Linguistics (ACL), the Conference on Empirical Methods in Natural Language Processing (EMNLP), and the Conference on Computer Vision and Pattern Recognition (CVPR).
Yin Tat Lee
Lee, who joined the Allen School faculty in 2017 and is also a visiting researcher at Microsoft Research AI, combines ideas from continuous and discrete mathematics to produce state-of-the-art algorithms for solving optimization problems that underpin the theory and practice of computing. His work encompasses multiple domains, including convex optimization, convex geometry, spectral graph theory, and online algorithms.
“From machine learning and experiment design, to route planning and medical imaging, convex optimization is used everywhere,” Lee said. “My group develops new techniques and algorithms to optimize faster, with the goal to design a universal optimization algorithm without compromising performance.”
Lee has already expanded convex optimization techniques to break long-standing running time barriers for a variety of problems. For example, he and his colleagues presented a new general interior point method that yielded the first significant improvement in linear programming in more than 20 years and a new algorithm for approximately solving maximum flow problems in near-linear time. Lee also has demonstrated the applicability of optimization techniques to an even broader class of problems than previously was considered feasible, devising a faster cutting plane method that improved the running time for solving classic problems in both continuous and combinatorial optimization. More recently, Lee contributed to a pair of new algorithms that achieve optimal convergence rates for optimizing non-smooth convex functions in distributed networks. That same year, Lee contributed to a total of six papers that appeared at the Symposium on Theory of Computing (STOC 2018) — a record high for an individual researcher at the conference.
Lee’s work has earned him multiple Best Paper and Best Student Paper awards at premier conferences in the field, including the IEEE Symposium on Foundations of Computer Science (FOCS), the ACM-SIAM Symposium on Discrete Algorithms (SODA), and the Conference on Neural Information Processing Systems (NeurIPS 2018). Last year, he earned a Microsoft Research Faculty Fellowship in recognition of his efforts to advance the field of theoretical computer science for real-world applications. In 2018, the Mathematical Optimization Society awarded Lee the A.W. Tucker Prize, which recognizes the best doctoral thesis in optimization in the prior three years, for his work on faster algorithms for convex and combinatorial optimization. That same year, he received a CAREER Award from the National Science Foundation to build upon that work and overcome multiple obstacles to optimization.
“To receive a Sloan Research Fellowship is to be told by your fellow scientists that you stand out among your peers,” Adam F. Falk, president of the Alfred P. Sloan Foundation, said in a press release. “A Sloan Research Fellow is someone whose drive, creativity, and insight makes them a researcher to watch.”
Hajishirzi and Lee are among four University of Washington researchers to watch in this latest group of Fellows, which includes Kyle Armour, a professor in the School of Oceanography and Department of Atmospheric Sciences, and Jacqueline Padilla-Gamiño, a professor in the School of Aquatic and Fishery Sciences.
The UW Reality Lab has launched a new incubator where students can develop innovative projects in augmented and virtual reality (AR/VR) with guidance and resources from lab faculty and staff. The Reality Lab, which launched two years ago, allows researchers to focus on the pursuit of leading-edge research and educating the next generation of innovators in this growing field. The incubator gives students a space to work on AR/VR projects while fostering a community of collaboration and organic mentorship. It also supports the greater UW community in AR/VR research and allows the novel research taking place in the incubator to be shared with the whole world.
“We select projects and teams for the incubator with the goal of having every project ultimately be released to the community in the form of research results or an application,” said John Akers, director of research and education in the lab. “Projects can either be in support of other groups in the greater UW community, such as other labs and departments, or student motivated projects based on ideas they are committed to developing fully.”
According to Ira Kemelmacher, professor of computer science and director of the UW Reality, Lab, one of the biggest challenges in AR/VR adoption is content and experiences creation.
“We are opening the incubator to allow undergraduate researchers to team up, come up with fresh ideas, and invent the future of AR/VR. We started by teaching a series of AR/VR capstones where teams of students came up with application ideas and implementations, in an amazing variety of fields from visualization of homelessness to cooking in AR,” she said. “In capstones they only have 10 weeks to bring their ideas into life. Due to its high popularity in the undergrad community and successful results, we decided to open an incubator that will allow more time for development. We believe undergrads have an immense potential in creating breakthroughs in AR/VR technology and our incubator encourages them to do exactly that, while getting advising, state of the art hardware, and full support from us and our collaborators.”
At the moment, the incubator is hosting two projects. One is led by Max Needle, a Ph.D. student in the Department of Earth and Space Sciences. Needle’s research is in rocks bending. He flew a drone around a strip mine in Pennsylvania that features a large folded rock layer, as well as several fossils and lots of faults. He was then able to generate a high-resolution 3D model of the mine from the drone photos, with the goal of developing immersive geological adventure and educational experiences.
Team discussion in the UW Reality Lab incubator
“The strip mine is a field-trip destination for many university geology classes, however, like many exquisite exposures of geologic structures, there are geographical and physical limiting factors,” Needle said. “To overcome obstacles related to access, my group at the Reality Lab Incubator is developing a virtual field trip through the strip mine. The geology-specific tools that we develop for VR with the Reality Lab, as well as the format and gameplay, can be put in a pipeline to enhance VR experiences of other geologic sites that have been mapped digitally in 3D.
His work will open new doors for teaching geology, and who has access to field geology.
“The incubator is great, so far,” said Andrew Wang, a second year Allen School student working with Needle. “All of the necessary tools are available. Having these tools so accessible will help us debug and further develop our knowledge in VR/AR technology.”
The other active incubator project is led by an Allen School senior and explores how different forms of locomotion mechanics in virtual reality can create emergent gameplay. He is working to see if some of these modes could reduce the simulator sickness some people feel in VR.
According to Akers, the incubator hopes to increase projects in the future as the process process for how projects are accepted and teams are composed is solidified. Learn more about the incubator and how to get involved on their website.
With continuous tracking, AuraRing can pick up handwriting — potentially for short responses to text messages
Sometimes a ring symbolizes a promise, sometimes it shows a person’s birth month or mood, and sometimes it’s a statement about their taste in jewelry. But thanks to researchers in the Allen School’s Ubicomp Lab, a ring can now do a lot more.
The latest in smart technology, the AuraRing is a ring and wristband combination with high-fidelity input tracking. The combination is a magnetic tracking system designed to report precise finger movement.
“We’re thinking about the next generation of computing platforms,” said Allen School alumnus and co-lead author Eric Whitmire (Ph.D. ‘19), now a research scientist at Facebook Reality Labs. “We wanted a tool that captures the fine-grain manipulation we do with our fingers — not just a gesture or where your finger is pointed, but something that can track your finger completely.”
The ability to track a finger enables freeform and subtle input for wearable platforms like smartwatches and augmented and virtual reality headsets. AuraRing enables applications like object manipulation, drawing, sliding, swipe-based text input and hand pose reconstruction because of its absolute, continuous tracking with millimeter-level accuracy. Due to a high bandwidth and data rate, AuraRing is also capable of detecting taps of various intensities which enables new kinds of always-available ambient interfaces.
The ring is a single transmitter coil tightly wrapped around a 3D-printed loop
The system, which is worn on the index finger, consists of a single transmitter coil tightly wrapped around a 3D-printed ring and a wristband with three embedded sensor coils that measure the resulting magnetic field. Using these measurements, the wristband tracks the absolute position and orientation of the ring in real-time, making free-form drawing, handwriting short text messages, controlling games and moving virtual objects with mixed reality headsets possible.
The writstband is embedded with sensor coils that measure the ring’s magnetic field
AuraRing is also a low-power, battery operated device that generates an oscillating magnetic field around the hand. By focusing on short-range tracking over distances between 10 and 15 centimeters, the system is less susceptible to using up too much power and encountering environmental interference.
“To have continuous tracking in other smart rings you’d have to stream all the data using wireless communication. That part consumes a lot of power, which is why a lot of smart rings only detect gestures and send those specific commands,” said co-lead author Farshid Salemi Parizi, a Ph.D. student in electrical and computer engineering. “But AuraRing’s ring consumes only 2.3 milliwatts of power, which produces an oscillating magnetic field that the wristband can constantly sense. In this way, there’s no need for any communication from the ring to the wristband.”
With these minimal, low-power electronics, AuraRing can operate for about a day on self-contained batteries and therefore has the potential to do a lot more.
“Because AuraRing continuously monitors hand movements and not just gestures, it provides a rich set of inputs that multiple industries could take advantage of,” said professor Shwetak Patel, who holds a joint appointment in the Allen School and the Department of Electrical & Computer Engineering. “For example, AuraRing could detect the onset of Parkinson’s disease by tracking subtle hand tremors or help with stroke rehabilitation by providing feedback on hand movement exercises.”
AuraRing was developed with support from the UW Reality Lab.The team, which has open-sourced the hardware designs and algorithms for their work, published their findings in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies. To learn more, watch the group’s video and check out the UW News release and coverage by KING 5 News and VentureBeat.
February 1 marked five years since the passing of professor Gaetano Borriello. Gaetano famously applied to only one program – ours – when he entered the academic job market in 1988 after earning his Ph.D. from the University of California, Berkeley. He spent the next 27 years on the Allen School faculty, six of those valiantly fighting the cancer that would eventually take him from us. Gaetano is one of several leaders, and dare we say, legends, whom the Allen School community lost in the last decade. As a new decade dawns, the memories of those who knew and worked with Gaetano haven’t faded.
“We continue to feel Gaetano’s loss every day. When I moved into the director’s office last month, one of my first thoughts was that now I will be near Gaetano’s bench, which is just across Sylvan Grove from my window,” said Magdalena Balazinska, professor and director of the Allen School. “It makes me feel like Gaetano is still with us — and I do believe his wisdom and compassion continue to influence our decisions and actions in a very positive way.
“It’s not the same without Gaetano,” she continued, “but we will continue to build upon his legacy by working to create a more inclusive community here within the Allen School and pushing state-of-the-art technologies to benefit people and communities around the world.”
Here, members of the extended Allen School family pay tribute to a friend, mentor and colleague whose presence is still very much felt by members of our community — and whose impact endures both here on campus and across the globe.
Gaetano’s vision:
Gaetano (center) poses with newly-minted Ph.D. graduates (left to right) Brian DeRenzi, Benjamin Birnbaum, Yaw Anokwa, and Carl Hartung
“Not all research ideas go beyond the lab, so it’s a testament to the power of Gaetano’s vision that the tools he created are in active use in every country in the world,” said Allen School alumnus Yaw Anokwa (Ph.D., ‘12), who worked with Gaetano on the Open Data Kit project and continues to support the deployment of ODK via his company, Nafundi.
“I wish he could see how much impact he has had, and I wish I could still ask for his guidance,” continued Anokwa. “I miss him. A lot.”
“Gaetano stands out to me as a visionary and as an educator,” said professor Richard Anderson of the Allen School’s Information and Communication Technology for Development Lab, which is building on Gaetano’s legacy in ICTD research. “He was able to look 10 years ahead and develop high impact applications of emerging technology, and he loved to recount all the times people told him his approaches would never be practical.
“Throughout his career, he put students above all else,” Anderson continued, “both as an advisor and mentor of graduate and undergraduate students, and as a teacher in Computer Engineering.”
Gaetano’s humanity:
“I knew Gaetano from my role as technical staff at CSE, overlapping with him for almost two decades,” said Scott Rose, who retired from the Allen School in 2015. “Gaetano was an exceptionally smart engineer, but what made him stand out from the crowd were his humor and his humanity — he was a genuine man of the people, without an arrogant bone in his body and with no appetite for status. He just wanted everybody to have the opportunity to do their best, to live with dignity, and he’d bend over backwards to make sure they did.
“Students at any level, staff, colleagues — they were all part of Gaetano’s community of peers,” Rose continued. “I hope that I make my exit with the same dignity that he did. I miss him greatly, and at his advice, yeah, I get my regular colonoscopies.”
Gaetano’s devotion to students and staff also stuck with Crystal Eney, Director of Student Services at the Allen School.
Crystal Eney (left) poses with Gaetano in front of Drumheller Fountain on the UW Seattle campus
“Gaetano was thoughtful, warm, passionate, entertaining and extremely student-focused. He always put students’ needs first. He was their advocate and their mentor. He would give them tough love when they needed it, but he was also a great listener and problem solver,” recalled Eney. “It wasn’t just the students who looked up to him — it was the staff, too. He treated us like equals, and was always there to say ‘good job’ or ‘nice work.’
“In this day and age when everything seems to move at lightning speed, Gaetano was one of those people who would stop by to ask how your day was going, and it wasn’t just a mindless pleasantry,” she noted. “He cared deeply about others, and he is so very deeply missed.”
Gaetano’s relationship with staff was such that he could bring people back into the fold after they had left — in the case of Kay Beck-Benton, the Allen School’s director of external relations, eight years after.
“I loved working with Gaetano,” said Beck-Benton, who started at the Allen School as a program manager before leaving UW for a series of startups. “So much so, that he is a major reason why I returned to — and currently work in — the Allen School. Gaetano was an advocate and mentor for staff and for students, as well as being passionate about research and teaching.
“No matter who you were or what position you held in the school, Gaetano made you feel welcome in his circle,” she concluded.
Gaetano’s mentorship:
Given how devoted he was to students, it is no surprise that mentorship and setting students up for success would be a recurring theme for those who remember Gaetano.
“If you asked Gaetano what should be the UW’s top priority, he would have one answer: the students,” said Allen School Ph.D. candidate Waylon Brunette, who began working with Gaetano as an undergraduate researcher nearly 20 years ago. “His focus and dedication to students pervaded his views on education, research, and service, and he measured his own success based on his students’ success.
“Gaetano was great at finding students who might have a few rough edges, like me, and helping us to realize that we were actually diamonds in the rough,” Brunette continued. “Mentoring was fundamental to who Gaetano was as a person, and I have tried to model my own mentorship on his shining example.”
Melissa Westbrook (left), Gaetano’s widow, recognizes Scott Hauck with the Gaetano Borriello Professorship for Education Excellence
“I know that advisors can have a big impact on people’s careers, but Gaetano was truly a role model to me,” Hauck said. “I saw in him someone who combined impactful research, careful mentoring, and a passion for education, and have tried to be as graceful and compelling a faculty member as he was. I often think about what Gaetano would have done when trying to structure my own teaching.
“I also try to put the students first, just as Gaetano did for me,” Hauck continued. “And I do still find myself wandering over to Sylvan Grove to sit on Gaetano’s bench sometimes and commune with him a bit.”
Gaetano also served as a mentor and role model to his faculty colleagues — whether intentional or not.
“Out of everything I can think of that has made me a better mentor and advisor over my 17 years at UW, being next door to Gaetano’s office for about five of those years, in a building with fairly thin walls, is near the top of the list,” observed professor Dan Grossman, vice director of the Allen School. “When I needed thoughtful advice on something, I would ask Gaetano. Nowadays, I try to guess the advice he would give me if he were still here.
“Gaetano was the full package — an excellent researcher who evolved his focus over his career, a gifted teacher, an award-winning mentor, and always one to carry more than his share of service,” Grossman continued. “He was also humble, self-aware, and grateful — a true role model for me and many others.”
Gaetano’s legacy:
Gaetano’s affection for students extended to willingly taking a pie in the face at the ACM spring picnic
Gaetano’s ability to educate and inspire those around him ensures his legacy will endure beyond the impact of the technologies he created. It’s a legacy that his former students and colleagues aim to expand through their own work.
“I would not be where I am today without Gaetano. As I learn how to advise Ph.D. students myself, I become more and more grateful for the incredible advocate and champion that Gaetano was for me, and for all his students,” remarked Allen School alumna Nicola Dell (Ph.D., ‘15), who is now a faculty member at Cornell Tech.
“I can’t believe that he has been gone five years,” she said. “I also can’t count the times I’ve wished I could call him up and ask for his advice. He is truly missed. I hope every day that he would be proud of the work we continue to do, that he continues to inspire.”
“Gaetano’s impact is felt every day, at UW and around the world,” said professor Ed Lazowska, who holds the Bill & Melinda Gates Chair in Computer Science & Engineering in the Allen School. “Scott Hauck, who was advised by Gaetano and his close colleague Carl Ebeling, now holds the Gaetano Borriello Professorship for Educational Excellence at UW. Open Data Kit, a project Gaetano launched with several of his students, sees ever-expanding adoption for mobile data collection around the world — an expansion supported by Nafundi, a company co-founded by two of Gaetano’s former Ph.D. students, Yaw Anokwa and Carl Hartung. And ODK-X is building on that legacy, guided by Gaetano’s faculty colleague Richard Anderson and alumni who previously worked with Gaetano.
“Other alumni have gone on to faculty positions at other universities, where they are building on Gaetano’s legacy in technology for development,” Lazowska continued. “Gaetano may not be physically with us, but five years on, he is everywhere.”
Following Gaetano’s passing, tributes poured in from colleagues and collaborators around the world, including primatologist and wildlife conservationist Jane Goodall, the International Red Cross, IEEE, PATH, and others. Read more here.
Knowing the distance between the center of display and the entry point of the blind spot area (s), and given that α is always around 13.5 degrees, the authors can calculate the viewing distance (d) as part of the Virtual Chinrest.
Behavioral studies in labs on university campuses are overwhelmed with participants who are WEIRD: western, educated, and from industrialized, rich and democratic countries. They are usually college students participating in the studies for class credit.
In an effort to expand these studies to non-WEIRD people too, virtual labs like LabintheWild and Amazon’s Mechanical Turk, open up the study to anyone with access to the internet. These labs give researchers a broader glimpse at the way people think and behave from young to old, around the globe, with diverse cultural beliefs and geographical locations. But, researchers still hesitate to rely too much on these virtual labs because they need a more controlled environment.
Allen School researchers have come up with a tool to allow for more control. The Virtual Chinrest “enables remote, web-based psychophysical research at large scale, by accurately measuring a person’s viewing distance through a 30-second task,” according to the lead author and Allen School Ph.D. student, Qisheng Li.
She said that online studies in psychophysical experiments allow researchers to analyze human perception and performance. Study participants in labs often need to rest their chins in a certain location to control the exercise, making sure each performer is viewing the test from the same place.
“We don’t know how far people in any online environment sit from their computers and we don’t know how big their display of the test is,” said Allen School professor Katharina Reinecke, co-founder of LabintheWild. “The virtual chinrest can monitor both the resolution on the screen and the physical distance from the monitor so researchers have more control over the online studies.”
To first calculate a participant’s display, participants are asked to place a credit card-sized card on the screen and adjust the slider on the screen to fit the credit card. That allows the researchers to calculate the pixel density on the monitor.
To measure the user’s distance from his or her monitor, there is also a blind spot task. Testers are asked to focus on a black square on the screen with their right eye closed, while a red dot repeatedly sweeps from right to left. They must hit the spacebar on their keyboards whenever it appears that the red dot has disappeared. That allows researchers to determine the distance between the center of the black square and the center of the red dot when it disappears from eyesight and understand how far the participant is from the monitor.
In an online test of the Virtual Chinrest on LabintheWild that included 1153 participants, Reinecke’s team was able to replicate and extend the results of previous in-lab studies to prove that the Virtual Chinrest can allow psychophysical studies to be done online, allowing for more diverse participant samples.
Reinecke and her collaborators presented Virtual Chinrest in a recent paper published in the research journal Nature’s Scientific Reports. Additional authors include professor Sung Jun Joo of Pusan National University and professor Jason D. Yeatman of Stanford University.
From computer science to linguistics and kickboxing to baking, this month’s undergraduate student spotlight, Noelle Merclich, is driven to creating a welcoming environment in the Allen School, serving others and always being kind and compassionate. During a month when people are making and struggling to stick to new resolutions, the Maple Valley, Wa., native and junior computer science major resolved long ago to do her best and try new things every day.
Allen School: Why did you choose to major in computer science?
Noelle Merchlich: After taking a couple years of programming in high school, I realized I really enjoyed the puzzle solving nature of it and how computer science impacts nearly every other possible career. Also my dad constantly jokes about how his return on investment for my college education is my ability to pay for him to take trips around the world, so the financial stability doesn’t hurt either.
Allen School: What do you like most about being an Allen School student?
NM: Definitely the people. Some of my favorite memories of the last few years include playing card games in the labs until 2 a.m. after barely finishing an assignment before the deadline and baking a surprise birthday cake in the residence halls. My experience at UW wouldn’t be the same without the friends I’ve met in the Allen School, and I know I definitely wouldn’t have survived most finals weeks without their help.
Allen School: What do you like about being a TA for the CSE Startup course and Direct Admission seminars?
NM: I’ve had the opportunity to be a TA for CSE Startup for three years now. It’s been exciting to see how the course has changed and grown over time. I enjoy working with the instructor, Lauren Bricker, and undergrad adviser Leslie Ikeda to improve the curriculum to best fit the needs of our students during their first experience in college and at the Allen School. I appreciate how I’m able to help the curriculum evolve along with my own experiences at UW. As for the DA seminar, I like how I’m able to help a larger scale of students with their transition to the Allen School. Realistically, my experience in computer science has always been a positive one since my parents have always supported my aspirations. And it’s never seemed abnormal to be a woman in the industry since my first two computer science teachers were women. However, I realize this is definitely not how everyone is initially exposed to the field, so I try to use my role in the DA seminar to help make each incoming student feel as though they have a place in computer science in whatever way I can.
Allen School: Why did you choose to minor in linguistics?
NM: My interest in linguistics began after giving a presentation on Noam Chomsky for a psychology class. When I took Ling 200: Intro to Linguistic Thought, I was really fascinated by the universality of how we break down language, so I took a few more classes in the linguistics department. I committed to completing my minor because over the past two years I’ve developed an interest in natural language processing, and understanding concepts like how syntax and semantics work together to form meaning is helpful with that.
Allen School: What are some of your favorite experiences or activities at the UW?
NM: I started taking kickboxing classes my freshman year because I’ve always thought it seemed really fun. Now it’s become a way for me to punch my stress away. Also, before I decided on pursuing computer science I thought I would go to culinary school to become a pastry chef. It’s safe to say that I don’t just like desserts, I love them. I’m constantly trying out different bakeries around campus and Seattle to find the best macarons, tres leches, or anything sweet. I highly recommend Cubes and Le Panier.
Allen School: Who or what inspires you?
NM: My grandmother, Rosa. In spite of all the drama and tragedy she faced throughout her life, she maintained such kindness and compassion for everyone. When I was about 14, I remember helping her make homemade spaghetti and meatballs to give to the construction workers doing renovations on her neighbor’s house at the end of the block. Although she passed away at the end of my freshman year, she has always motivated me to be more kind, patient, and helpful to those around me. Her example is part of why I’m so passionate about computer science outreach. I try to find ways to connect students to computer science who normally wouldn’t have the resources to get started themselves. As a result, when one of the Allen School advisers sent out an application for HCDE’s alternative spring break group, I jumped at the chance. For two years I’ve had the incredible opportunity of being part of a team that created curricula for teaching introductory programming concepts to middle and high school students in rural Neah Bay, WA. It was definitely one of the more challenging and fulfilling college experiences I’ve had.
We are inspired by Noelle’s contributions to the Allen School and her outreach work!
Tree-based machine learning models are among the most popular non-linear predictive learning models in use today, with applications in a variety of domains such as medicine, finance, advertising, supply chain management, and more. These models are often described as a “black box” — while their predictions are based on user inputs, how the models arrived at their predictions using those inputs is shrouded in mystery. This is problematic for some use cases, such as medicine, where the patterns and individual variability a model might uncover among various factors can be as important as the prediction itself.
While there are multiple ways of computing global measures of feature importance that gauge their impact on the model as a whole, TreeExplainer is the first tractable method capable of quantifying an input feature’s local importance to an individual prediction while simultaneously measuring the effect of interactions among multiple features using exact fair allocation rules from game theory. By precisely computing these local explanations across an entire dataset, the tool also yields a deeper understanding of the global behavior of the model. Unlike previous methods for calculating local effects that are impractical or inconsistent when applied to tree-based models and large datasets, TreeExplainer produces rapid local explanations with a high degree of interpretability and strong consistency guarantees.
“For many applications that rely on machine learning predictions to guide decision-making, it is important that models are both accurate and interpretable — meaning we can understand how a model combined and weighted the various input features in predicting a certain result,” explained lead author and recent Allen School alumnus Scott Lundberg (Ph.D., ‘19), now a senior researcher at Microsoft Research. “Precise local explanations of this process can uncover patterns that we otherwise might not see. In medicine, factors such as a person’s age, sex, blood pressure, and body mass index can predict their risk of developing certain conditions or complications. By offering a more robust picture of how these factors contribute, our approach can yield more actionable insights, and hopefully, more positive patient outcomes.”
Many predictive AI models are a “black box” that offer predictions without explaining how they arrived at their results. TreeExplainer produces local explanations by assigning a numeric measure of credit to each input feature, such as factors that contribute to mortality risk shown in the example above. The ability to compute local explanations across all samples in a dataset can yield a greater understanding of global model structure.
Lundberg and his colleagues offer a new approach to attributing local importance to input features in trees that is both principled and computationally efficient. Their method draws upon game theory to calculate feature importance as classic Shapley values, reducing the complexity of the calculation from exponential to polynomial time to produce explanations that are guaranteed to always be both locally accurate and consistent. To capture interaction effects, the team introduces Shapley Additive Explanation (SHAP) interaction values. These offer a new, richer type of local explanation that employs the Shapley interaction index — a relatively recent concept in game theory — to produce a matrix of feature attributions with uniqueness guarantees similar to Shapley values.
This dual approach enables separate consideration of the main contributions and the interaction effects of features that lead to an individual model prediction, which can uncover patterns in the data that may not be immediately apparent. By combining local explanations from across an entire dataset, TreeExplainer offers a more complete global representation of feature performance that both improves the detection of feature dependencies and succinctly shows the magnitude, prevalence, and direction of each feature’s effect — all while avoiding the inconsistency problems inherent in previous methods.
In a clinical setting, TreeExplainer can provide a global view of the dependency of certain patient risk factors while also highlighting variabilities in individual risk. In their paper, the UW researchers describe several new methods they developed that make use of the local explanations from TreeExplainer to capture global patterns and glean rich insights into a model’s behavior, using multiple medical datasets. For example, the team applied a technique called local model summarization to uncover a set of rare but high-magnitude risk factors for mortality. These are inputs such as high blood protein that are shown to have low global importance, and yet they are extremely important for some individuals’ mortality risk. Another experiment in which the researchers analyzed local interactions for chronic kidney disease revealed a noteworthy connection between high white blood cell counts and high blood urea nitrogen; the team found that the model assigned higher risk to the former when it was accompanied by the latter.
In addition to discerning these patterns, the researchers were able to identify population sub-groups that shared mortality-related risk factors and complementary diagnostic indicators for kidney disease using a technique called local explanation embeddings. In this approach, each sample is embedded into a new “explanation space” to enable supervised clustering in which samples are grouped together based on their explanations. For the mortality dataset, the experiment revealed certain sub-groups within the broader age groups that share specific risk factors, such as younger individuals with inflammation markers or older individuals who are underweight, that would not be apparent using a simple unsupervised clustering method. Unsupervised clustering also would not have revealed how two of the strongest predictors of end-stage renal disease — high blood creatinine levels, and a high ratio of urine protein to urine creatinine — can each be used to identify a set of unique at-risk individuals and should be measured in parallel.
AIMS Lab researchers, top row from left: Su-In Lee, Scott Lundberg, and Gabriel Erion; bottom row, from left: Hugh Chen and Alex DeGrave
Beyond revealing new patterns of patient risk, the team’s approach also proved useful for exercising quality control over the models themselves. To demonstrate, the researchers monitored a simulated deployment of a hospital procedure duration model. Using TreeExplainer, they were able to identify intentionally introduced errors as well as previously undiscovered problems with input features that degraded the model’s performance over time.
“With TreeExplainer, we aim to break out of the so-called black box and understand how machine learning models arrive at their predictions. This is particularly important in settings such as medicine, where these models can have a profound impact upon people’s lives,” observed Allen School professor Su-In Lee, senior author and director of the AIMS Lab. “We’ve shown how TreeExplainer can enhance our understanding of risk factors for adverse health events.
“Given the popularity of tree-based machine learning models beyond medicine, our work will advance explainable artificial intelligence for a wide range of applications,” she said.
