Jeong Joon (JJ) Park, a Ph.D. student working with Allen School professor Steve Seitz in the Graphics and Imaging Laboratory (GRAIL) has been named a 2020 Apple Ph.D. Scholar. Park was recognized in the “Artificial Intelligence/Machine Learning” category for his focus on 3D reconstruction and understanding. Apple selects its scholars based on their innovative research, record as thought leaders and collaborators in their fields, and unique commitment to take risks and push the envelope in machine learning and AI.
Park’s research focuses on recovering the underlying 3D properties of environments from photos and depth sensing. His work spans computer vision, deep learning, and augmented and virtual reality (AR/VR).
“3D computer vision can be very useful in many exciting areas of 3D applications, including VR/AR — visiting new places virtually, or robotics — using them to navigate and pick up stuff,” Song said. “I work on visually realistic reconstructions and study what is the most suitable representation for 3D geometry and appearance.”
Park enjoys the work because it is an intersection of various fundamental areas that interest him, including machine learning, physics, computer vision and graphics. In his most recent paper, “Seeing the World in a Bag of Chips,” he and co-authors Alexsander Holynski and Seitz address problems of novel view synthesis and environment reconstruction from hand-held RGBD sensors. The team developed a method for modeling highly specular objects, inter-reflections and fresnel effects to enable surface light field reconstruction of the environment with the same input used to reconstruct the shape alone. The results can be seen in this video, where Park films a bag of chips and then reconstructs a high-resolution image of the surrounding room, including lights, furniture, windows — even trees and people seen through the windows. The team presented its work at the Computer Vision and Pattern Recognition 2020 Computer Vision and Pattern Recognition (CPVR 2020) conference in June, and the project was also featured in Scientific American and WIRED. This helps to create more realistic 3D views in VR and AR.
In another paper, “Deepsdf: Learning continuous signed distance functions for shape representation,” Park introduces a new way of representing shape using neural networks. The new representation enables computers to more effectively learn 3D shapes prior from large datasets, enabling applications in vision, graphics and robotics. That work was among the candidates for Best Paper at CVPR 2019. This technique can be used for applications in AR/VR for indoor scene geometry reconstruction, inferring 3D shape of humans from an image, robotics for object grasping, or autonomous-driving for predicting shape of cars around the driver. Previously, in his paper “Surface light field fusion” published at the 2018 International Conference on 3D Vision, Park shows how to scan highly reflective objects with an RBGD sensor, using infrared dot patterns themselves to recover specular coefficients.
“While most of my students look to me to choose research directions in the beginning, JJ was driving his own research program from the start,” said Seitz. ”He’s passionate about scene modeling and rendering and wants to solve this problem for real. Large scale, general objects, and real systems that work for real users. And it’s been thrilling to watch him make major inroads on solving this problem with a series of strong technical papers.”
Park is looking forward to the opportunities the program will afford him.
“I’m very happy to be supported by Apple,” he said. “The company is already taking initiatives in augmented reality and other research areas and I expect to see synergy between our research.”
Congratulations, JJ — and thanks to Apple for generously supporting student research!