Allen School News

Professor Ira Kemelmacher-Shlizerman of the Allen School’s Graphics & Imaging Laboratory (GRAIL) flies her geek flag high in this week’s edition of GeekWire’s “Geek of the Week.” Kemelmacher-Shlizerman is one of the brains behind a series of high-profile research projects that combine facial recognition, modeling and 3-D reconstruction — including a new technique to create photorealistic videos of people from audio files, as demonstrated by a lip-syncing Obama.

The goal of her work, Kemelmacher-Schlizerman explains to GeekWire, is to transform how we interact with one another and the world around us through advancements in computer vision, computer graphics, and augmented and virtual reality technologies.

“Our photos and videos tell a ton of about ourselves, our histories, how people grow, age, learn to walk, and change over time,” she said. “Exploring and learning from that data will enable magical applications in telepresence, health, sports, entertainment and many other unexpected ones.”

Read the full article here, and also check out recent Allen School “Geeks of the Week” Shyam Gollakota, director of the Networks & Mobile Systems Lab, and Ph.D. alumna Irene Zhang.

DawgBytes is the Allen School’s K-12 outreach program. One component of DawgBytes is a series of girls and co-ed computer science summer day camps for middle school and high school students. Always wildly over-subscribed, they’re in full swing now. This was the second week of a two-week “High School Girls Leadership Camp,” during which the students were joined by (and mentored) students in the Middle School Girls Creative Coding Camp.

More photos here. Info on our summer camps here. Info on DawgBytes here. DawgBytes Facebook page here.

“With the nation’s fastest-growing economy and an all-star business roster of household names and up-and-comers, Washington — the Evergreen State — soars above the competition as America’s Top State for Business in 2017.

“The home of Amazon and Costco, Boeing and Expedia, as well as rising stars like Adaptive Biotechnologies, online marketplace OfferUp and space company Blue Origin, Washington has the old and new economies covered — as well as pretty much everything in between.

“But the success story does not end there. At a time when the best workforce rules, Washington boasts the nation’s largest concentration of STEM (science, technology, education and math) workers. Nearly 1 in every 10 Washington workers is in those professions, according to the U.S. Bureau of Labor Statistics. The University of Washington’s computer science school — recently named for one of the state’s most famous natives, Microsoft co-founder Paul Allen — is world class. There is no brain drain here; no state does better at hanging on to its college graduates. And the state is consistently a magnet for investment capital. Washington businesses attracted nearly $1.6 billion in venture capital last year, the sixth-highest total in the nation.”

Professor Michael Ernst of the Allen School’s Programming Languages & Software Engineering group (PLSE) has been recognized by the International Conference on Software Engineering with its Most Influential Paper Award for 2017. Ernst — along with co-authors Carlos Pacheco of Google, and Shuvendu Lahiri and Thomas Ball of Microsoft Research — earned the award for their ICSE 2007 paper, “Feedback-directed random test generation.”

Each year, ICSE selects a paper from 10 years earlier that it judges to have had “the most influence on the theory or practice of software engineering during the 10 years since its original publication.” In their winning paper, Ernst and his colleagues presented a test generation tool, Randoop, which generates tests for programs written in object-oriented languages such as Java and .NET. The technique put forward by the team generates one test at a time, executes the test, and classifies it as (probably) a normal execution, a failure, or an illegal input. Based on this information, it biases the subsequent generation process to extend good tests and avoid bad tests.

By contrast, a typical test generation technique would generate many tests and then try to determine which ones were of value. For example, an error-revealing test is one that makes legal calls that yield incorrect results. Without a formal specification, it is difficult to know whether a given call is legal and whether its outcome is desired. Furthermore, multiple generated tests might fail for the same reason.

Automated test generation is a practically important research topic. In 2002, the National Institute of Standards and Technology (NIST) estimated the annual costs of inadequate infrastructure for software testing in the U.S. to be at least $22.2 billion and as high as $59.5 billion — with more than half of those costs borne by software users on error avoidance and mitigation.

Ernst and Pacheco first introduced feedback-directed test generation in their ECOOP 2005 paper, “Eclat: Automatic generation and classification of test inputs,” when Ernst was a member of the MIT faculty and Pacheco was his student. The ICSE 2007 paper expanded and improved upon the technique with the introduction of Randoop based on more extensive experiments. Randoop continues to be actively maintained 10 years on, and thanks to its scalability and simplicity, it remains the standard benchmark against which other test generation tools are measured.

Ernst and his colleagues were formally recognized at the ICSE 2017 conference held in Buenos Aires, Argentina in May.

Congratulations, Michael!

Researchers in the Allen School’s Graphics & Imaging Laboratory (GRAIL) have developed a new technique that enables them to generate photorealistic videos from audio clips. The team, which includes recent Ph.D. graduate Supasorn Suwajanakorn and professors Steven Seitz and Ira Kemelmacher-Shlizerman, demonstrated their approach by producing a video of former president Barack Obama lip-syncing audio on a range of topics, complete with natural-looking facial expressions and mouth movements.

To achieve such a lifelike result, the researchers had to overcome the “uncanny valley” problem that typically plagues synthesized human likenesses — giving them a creepiness factor that most viewers will find hard to look past.

“People are particularly sensitive to any areas of your mouth that don’t look realistic,” noted Suwajanakorn, lead author of the paper describing the team’s results. “People can spot it right away and it’s going to look fake…you have to render the mouth region perfectly to get beyond the uncanny valley.”

For its demonstration with Obama, the team trained a neural network to view existing videos of the former president and translate sounds into mouth shapes. They then superimposed and blended those shapes onto a reference video — drawing on their previous research in 3-D facial reconstruction and digital modeling — to depict Obama accurately lip syncing speeches from unrelated audio clips. According to Kemelmacher-Shlizerman, it is the first time researchers have achieved such realistic results in an audio-to-video conversion.

The team’s approach, which will be presented at the SIGGRAPH 2017 conference in Los Angeles, California next month, could yield significant advancements in video conferencing and virtual reality applications.

Read the UW News release here and visit the project page here. Also check out coverage of the project by The Atlantic, IEEE Spectrum, Wired, New Atlas, Engadget, GeekWire, PCMag, and The Verge.

The Allen School continues to build its expertise in leading-edge areas of the field with the recruitment of Kevin Jamieson as the Guestrin Endowed Professor in Artificial Intelligence & Machine Learning. Jamieson, whose research focuses on adaptive data collection algorithms for machine learning, will join the University of Washington this fall after completing a postdoc at University of California, Berkeley.

Jamieson’s research explores how to leverage already-collected data to inform what future measurements to make next, in a closed loop. In both theory and practice, Jamieson has demonstrated that adaptive data collection — or active learning — can extract considerably richer insights than any measurement plan fixed in advance, using the same statistical budget. This is particularly valuable for machine learning applications when training data is time-consuming or expensive to collect, such as when labeled examples are provided by humans. His work has been adopted in a range of applications, from measuring human perception in psychology studies, to numerical optimization and choosing hyperparameters for deep neural networks, to recommending what beer you should try next.

Keen to explore how applications and collaborations can inform his research questions, Jamieson also led the development of NEXT, an open-source software system that facilitates the development, testing, and deployment of active learning for online, real-time applications. The New Yorker magazine employs the NEXT platform to crowd-source its cartoon caption contest, in which readers view captions submitted by other readers for a fixed cartoon one at a time online and rate them as funny, somewhat funny, or unfunny. It’s a contest for the algorithms as well as the readers: behind the scenes, a suite of different active-learning algorithms compete against each other to identify the captions that are consistently rated as funny, and to stop showing those that are unfunny. The exercise provides Jamieson and his collaborators valuable information about which algorithms work best in practice, and which can be discarded.

Jamieson is returning to his roots by joining the UW, where he obtained his bachelor’s degree in electrical engineering before going on to earn his master’s at Columbia University and a Ph.D. from University of Wisconsin, Madison. He is the inaugural recipient of the Guestrin Endowed Professorship, which the Allen School created following Apple’s acquisition of Turi, the machine learning startup created by professor Carlos Guestrin.

Jamieson is the latest in a series of impressive new additions to the Allen School faculty: last week, the Allen School announced that two outstanding researchers in human-computer interaction — Jennifer Mankoff from Carnegie Mellon University and Jon Froehlich from University of Maryland, College Park — will arrive as part of a cluster hire that will make UW one of the leading centers of HCI research and education. Previously, the school revealed that leading robotics researcher Siddhartha “Sidd” Srinivasa of CMU and pioneering computer engineer Michael Taylor of University of California, San Diego would join the faculty this fall along with Yin Tat Lee, a rising star in theoretical computer science.

Jennifer Mankoff

The University of Washington is preparing to welcome outstanding new faculty hires who will bolster its reputation as a center of human-computer interaction research and teaching. Two of the impending arrivals, Jennifer Mankoff and Jon Froehlich, will join the Allen School faculty in advancing solutions to society’s greatest challenges, focusing on accessibility, education, health, sustainability, and more.

Jennifer Mankoff will join the Allen School as the Richard E. Ladner Endowed Professor in Computer Science & Engineering.  She is currently a Professor in Carnegie Mellon University’s Human Computer Interaction Institute, where she leads the Make4All Lab. Previously, Mankoff was a member of the faculty in Electrical Engineering & Computer Science at the University of California, Berkeley.

Mankoff’s research combines empirical methods and technical innovation to solve problems related to accessibility, health, and sustainability. Most recently, her work has focused on 3D printing and its potential for creating personalized assistive technologies for people with disabilities.

For example, Mankoff and her colleagues have shown that a 3D printed physical interface can augment a screen reader and improve interaction speed by a third. This do-it-yourself (DIY) theme is carried over into her work with e-NABLE, a collaborative effort to engage volunteers and clinicians in the design and fabrication of upper-limb assistive technologies. Mankoff’s assistive technology work also extends beyond fabrication. For example, she and her colleagues have explored navigation aids for the blind, and mobile sound capture and transcription tools for the deaf.

As is the case with a lot of accessibility research, Mankoff’s people-centric approach to technology and novel use of 3D printing will wind up benefiting everybody. Mankoff has studied and created solutions for addressing uncertainty in measurement during 3D printing, attaching adaptations to everyday objects (Encore and Reprise) and prototyping custom assistive technologies. Another stream of work aims to explore more natural materials, including embedding textiles in 3D printing, creating knitted objects programmatically, and printing using layers of fabric.

Mankoff’s work also extends to chronic illness. She has explored the impact of chronic disease on quality of life, predictors of trust in peer versus professionally produced health content, visualization techniques for infant oxygen monitoring, and the design of both mobile and web tools for managing chronic illness and physical therapy.

Mankoff has been recognized with an Alfred P. Sloan Fellowship, IBM Faculty Fellowship, and Best Paper awards from ASSETS, CHI, and Mobile HCI. She earned her B.A. in computer science from Oberlin College in 1995 and her Ph.D. in computer science from the Georgia Institute of Technology in 2001.

While she is eager to take advantage of new research opportunities as a member of the Allen School faculty and the university’s Design Use Build (DUB) group, Mankoff is particularly excited about the culture she encountered during her time on campus.

“I visited UW feeling that it would take a lot for me to leave my wonderful colleagues and position at CMU,” Mankoff admitted. “However, the wonderful welcome, fascinating research, many opportunities for collaboration, and warm community in the Allen School and DUB won me over.”

With Mankoff’s arrival, the Allen School adds an exciting new dimension to its accessibility research that will complement the existing efforts of its HCI research group and the Taskar Center for Accessible Technology.

Jon Froehlich’s arrival will be a homecoming of sorts. Froehlich, who earned his Ph.D. at the Allen School working with professors James Landay and Shwetak Patel, will return to his alma mater as an associate professor after having spent the past five years on the faculty of the computer science department at University of Maryland, College Park and director of the Makeability Lab.

Jon Froehlich

Froehlich focuses on the design of interactive tools and techniques that span the virtual and the physical to address pressing global challenges in accessibility, education, and environmental sustainability. He aims to invent or reappropriate methods for sensing physical or behavioral phenomena, leveraging techniques in computer vision and machine learning to interpret and characterize the data. He then builds software and hardware tools that are uniquely enabled by this approach for people of different ages and abilities.

One of Froehlich’s most recent projects, MakerWear, earned him a CAREER Award from the National Science Foundation. MakerWear is a modular toolkit that enables children without prior programming knowledge to create their own interactive wearables using tangible plug-and-play modules. Froehlich and his colleagues won a Best Paper at CHI 2017 for demonstrating how children between the ages of five and 12 can use MakerWear to create their own diverse, personally meaningful designs while developing computational thinking. Another project developed for children, BodyVis, explores how wearable sensing technology can help develop scientific inquiry skills and knowledge of anatomy and physiology. Froehlich and his team created a custom e-textile shirt that senses the wearer’s physiology in real-time and then visualizes this information via responsive anatomical models.

Froehlich is also interested in developing assistive technologies that support independent living. For example, as part of the HandSight project, Froehlich and his colleagues repurposed tiny, endoscopic cameras to augment a blind person’s hand with computer vision. The technology enables users to sense non-tactile information about the world — such as color, texture, and printed text — to support the activities of daily living and increase their independence. Froehlich also initiated Project Sidewalk to collect street-level accessibility information and develop new location-based tools to assist people with disabilities in navigating their communities.

Similar to Mankoff, Froehlich identified the collaborative culture — exemplified by the interdisciplinary DUB group, of which Froehlich was a founding member during his student days — as a motivating factor in his decision to join the UW faculty.

“I am delighted to rejoin the Allen School — a place that helped shape and support my desire to work on applied problems with social relevance,” Froehlich said. “The school has a reputation not just of research excellence and creative innovation, but of collegiality and a unique collaborative culture.”

“UW is a world-leader in HCI research and education, with top experts in nearly every facet of the discipline,” he continued. “I am excited to once again have the opportunity to work with the fantastic faculty and students not just in computer science and engineering, but also in art, geography, social science, HCDE, and the iSchool. Cross-campus collaborations like DUB are what make UW one of the top HCI institutions in the world.”

With the arrival of Froehlich and Mankoff, the Allen School adds leading-edge expertise in wearable technology, personal fabrication, and pervasive computing — and reinforces the school’s reputation as a destination for rising stars in HCI and accessibility research.

Mankoff and Froehlich will be accompanied by Anind K. Dey and Leah Findlater, respectively, in their moves to UW.

Anind K. Dey (left) and Leah Findlater

Dey, who currently serves as the Director and Charles M. Geschke Chair of CMU’s Human Computer Interaction Institute, will join UW as Dean of the Information School. Dey’s research spans human-computer interaction, ubiquitous computing, machine learning, and sensing, with projects focused on intelligibility and end-user control in context-aware computing, active learning to improve usability and performance, modeling and predicting human behavior, and development of programming support for context-aware and sensor-rich environments.

Findlater, currently a faculty member in the University of Maryland College of Information Studies and Institute for Advanced Computer Studies, will join the faculty of UW’s Department of Human Centered Design & Engineering (HCDE). At UMD, Findlater leads the Inclusive Design Lab, where her research focuses on the design of new mobile and wearable technologies that can be adapted to accommodate individual users’ needs and preferences — including novel interfaces and interactions that are accessible to people with visual and motor impairments.

In addition to their primary appointments, both Dey and Findlater will be adjunct faculty members of the Allen School.

Alexis Hiniker

The group announced today are part of a larger HCI cluster hire by UW that includes Alexis Hiniker, who is joining the iSchool faculty after earning her Ph.D. at UW in human-centered design and engineering. Hiniker is an HCI researcher and full-stack developer who builds technologies to empower users by reducing vulnerability and fostering well-being, especially among children, including work in child-computer interaction, technology use in families, and technology for supporting children with atypical patterns of development.

Together, this group of new hires will significantly expand UW’s capacity to develop novel solutions to some of the most pressing problems facing society today and advance the Seattle region’s reputation as a center of innovation and real-world impact.

The Allen School previously announced the impending arrival of two other game-changing faculty hires — leading robotics researcher Siddhartha “Sidd” Srinivasa from CMU, and pioneering computer engineer Michael Taylor from University of California, San Diego.

Martin Kellogg (left) and Calvin Loncaric

Allen School students Martin Kellogg and Calvin Loncaric have earned national recognition as grand finalists in the Association for Computing Machinery’s 2017 Student Research Competition. The competition, which is sponsored by Microsoft, highlights exemplary student researchers and encourages their participation in top computing research conferences. Kellogg captured third place in the undergraduate category, and Loncaric placed third in the graduate category. A total of 330 students participated in this year’s competition by presenting their work at one of two dozen ACM conferences held around the world.

Kellogg is a first-year Ph.D. student who works with professor Michael Ernst in the Allen School’s Programming Languages & Software Engineering (PLSE) group. He presented his winning project, “Combining Bug Detection and Test Case Generation,” at the Foundations of Software Engineering (FSE 2016) conference in December. The paper, which was the result of work he initiated as an undergraduate at University of Virginia, presents N-prog, an efficient, new tool for the detection of software bugs.

Loncaric, who also works with Ernst, captured third place in the graduate competition for another project featured at FSE 2016, “Cozy: Synthesizing Collection Data Structures.” Cozy is a novel tool for implementing new data structures that uses counter-example guided inductive synthesis in place of the more tedious and error-prone process of handwritten implementation.

ACM recognized Kellogg, Loncaric and their fellow competition winners at its annual awards banquet this past weekend in San Francisco, California.

Read the ACM press release here and our prior blog coverage of the Allen School at FSE 2016 here.

Way to go, Martin and Calvin!

Check out the live webcams here, here, and here. Learn about the project here.

From left: professor Dan Grossman, students Justin Kotalik and Chadi Moussi, and professor Ed Lazowska at the end-of-year poster fair

Each year our amazing Allen School undergraduates spend many hours combining their creativity and problem-solving skills with their technical computing acumen to design and implement course projects, particularly in our senior-level courses and capstone design courses. At the end of each quarter, the Microsoft Atrium in the Paul G. Allen Center hosts a variety of course-specific poster sessions where students describe what they have accomplished. Last week, we tried something new: a school-wide, end-of-year poster session where student teams from across the curriculum could reprise their posters and talk about their work with a broader audience.

Participating teams also competed for a prize awarded by Allen School alumni and friends serving as judges. Our volunteer panel evaluated the students’ work based on technical achievement, creativity, presentation, and potential for impact. After viewing all 19 projects on display — which spanned programming languages, natural language processing, data visualization, mobile apps, robotics, and more — the judges selected two for special recognition: “Leo,” a decentralized chat app for iOS that provides end-to-end encryption for mobile messaging, and “Global Terrorism,” an interactive data visualization tool for objectively presenting and analyzing information on terrorist attacks.

The winning project, Leo, was developed by computer science undergraduate Austin Bhisharat, computer engineering undergraduate Justin Kotalik, and combined bachelor’s/master’s student Chadi Moussi to address security concerns with popular mobile messaging apps. These apps typically rely on a central server to relay messages between sender and recipient, which enables them to collect metadata about users’ communications as the messages pass through a central server. With Leo, messages are encrypted and sent peer-to-peer, preventing any single entity from knowing who is communicating with whom or when.

Honorable mention went to the Global Terrorism data project, which was developed by computer science undergraduates Sheen Dudwadkar, Carson Gulledge, Michelle Lee, and Grace Qiu. The team created a narrative visualization tool that enables citizens to explore data and questions about terrorism themselves, without going through the filter of emotionally and politically charged media coverage.

Learn more about all 19 featured projects here.

Congratulations to our winners and to all the students who participated — and many thanks to the extended Allen School family for supporting our undergraduates!