Abhijit Kundu

selected publications

1. 

   Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation

   Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, Thomas Funkhouser.

   In CVPR 2022

   Abs arXiv Bib PDF Website

   We present Panoptic Neural Fields (PNF), an object-aware neural scene representation that decomposes a scene into a set of objects (things) and background (stuff). Each object is represented by an oriented 3D bounding box and a multi-layer perceptron (MLP) that takes position, direction, and time and outputs density and radiance. The background stuff is represented by a similar MLP that additionally outputs semantic labels. Each object MLPs are instance-specific and thus can be smaller and faster than previous object-aware approaches, while still leveraging category-specific priors incorporated via meta-learned initialization. Our model builds a panoptic radiance field representation of any scene from just color images. We use off-the-shelf algorithms to predict camera poses, object tracks, and 2D image semantic segmentations. Then we jointly optimize the MLP weights and bounding box parameters using analysis-by-synthesis with self-supervision from color images and pseudo-supervision from predicted semantic segmentations. During experiments with real-world dynamic scenes, we find that our model can be used effectively for several tasks like novel view synthesis, 2D panoptic segmentation, 3D scene editing, and multiview depth prediction.

   @inproceedings{KunduCVPR2022PNF,
     title = {{Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation}},
     author = {Kundu, Abhijit and Genova, Kyle and Yin, Xiaoqi and Fathi, Alireza and Pantofaru, Caroline and Guibas, Leonidas and Tagliasacchi, Andrea and Dellaert, Frank and Funkhouser, Thomas},
     booktitle = {CVPR},
     year = {2022},
   }

2. 

   Virtual Multi-view Fusion for 3D Semantic Segmentation

   Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, Caroline Pantofaru.

   In ECCV 2020

   Abs arXiv Bib PDF Website

   Semantic segmentation of 3D meshes is an important problem for 3D scene understanding. In this paper we revisit the classic multiview representation of 3D meshes and study several techniques that make them effective for 3D semantic segmentation of meshes. Given a 3D mesh reconstructed from RGBD sensors, our method effectively chooses different virtual views of the 3D mesh and renders multiple 2D channels for training an effective 2D semantic segmentation model. Features from multiple per view predictions are finally fused on 3D mesh vertices to predict mesh semantic segmentation labels. Using the large scale indoor 3D semantic segmentation benchmark of ScanNet, we show that our virtual views enable more effective training of 2D semantic segmentation networks than previous multiview approaches. When the 2D per pixel predictions are aggregated on 3D surfaces, our virtual multiview fusion method is able to achieve significantly better 3D semantic segmentation results compared to all prior multiview approaches and recent 3D convolution approaches.

   @inproceedings{KunduECCV2020VirtualMVFusion,
     title = {Virtual Multi-view Fusion for 3D Semantic Segmentation},
     author = {Kundu, Abhijit and Yin, Xiaoqi and Fathi, Alireza and Ross, David and Brewington, Brian and Funkhouser, Thomas and Pantofaru, Caroline},
     booktitle = {ECCV},
     year = {2020},
     doi = {10.1007/978-3-030-58586-0_31},
     url = {https://doi.org/10.1007/978-3-030-58586-0_31},
     isbn = {978-3-030-58586-0},
   }

3. 

   3D-RCNN: Instance-level 3D Object Reconstruction via Render-and-Compare

   Abhijit Kundu, Yin Li, James M. Rehg.

   In CVPR 2018

   Abs Bib PDF Website

   We present a fast inverse-graphics framework for instance-level 3D scene understanding. We train a deep convolutional network that learns to map image regions to the full 3D shape and pose of all object instances in the image. Our method produces a compact 3D representation of the scene, which can be readily used for applications like autonomous driving. Many traditional 2D vision outputs, like instance segmentations and depth-maps, can be obtained by simply rendering our output 3D scene model. We exploit class-specific shape priors by learning a low dimensional shape-space from collections of CAD models. We present novel representations of shape and pose, that strive towards better 3D equivariance and generalization. In order to exploit rich supervisory signals in the form of 2D annotations like segmentation, we propose a differentiable Render-and-Compare loss that allows 3D shape and pose to be learned with 2D supervision. We evaluate our method on the challenging real-world datasets of Pascal3D+ and KITTI, where we achieve state-of-the-art results.

   @inproceedings{3DRCNN_CVPR2018,
     author = {Kundu, Abhijit and Li, Yin and Rehg, James M.},
     title = {3D-RCNN: Instance-level 3D Object Reconstruction via Render-and-Compare},
     booktitle = {CVPR},
     year = {2018},
     doi = {10.1109/CVPR.2018.00375},
     url = {https://doi.org/10.1109/CVPR.2018.00375},
   }

4. 

   Feature Space Optimization for Semantic Video Segmentation

   Abhijit Kundu, Vibhav Vineet, Vladlen Koltun.

   In CVPR 2016

   Abs Bib PDF Code Slides Website

   We present an approach to long-range spatio-temporal regularization in semantic video segmentation. Temporal regularization in video is challenging because both the camera and the scene may be in motion. Thus Euclidean distance in the space-time volume is not a good proxy for correspondence. We optimize the mapping of pixels to a Euclidean feature space so as to minimize distances between corresponding points. Structured prediction is performed by a dense CRF that operates on the optimized features. Experimental results demonstrate that the presented approach increases the accuracy and temporal consistency of semantic video segmentation.

   @inproceedings{KunduCVPR2016VideoFSO,
     title = {Feature Space Optimization for Semantic Video Segmentation},
     author = {Kundu, Abhijit and Vineet, Vibhav and Koltun, Vladlen},
     booktitle = {CVPR},
     year = {2016},
     doi = {10.1109/CVPR.2016.345},
     url = {https://doi.org/10.1109/CVPR.2016.345},
   }

5. 

   Joint Semantic Segmentation and 3D Reconstruction from Monocular Video

   Abhijit Kundu, Yin Li, Frank Dellaert, Fuxin Li, James M. Rehg.

   In ECCV 2014

   Abs Bib PDF Supp Website

   We present an approach for joint inference of 3D scene structure and semantic labeling for monocular video. Starting with monocular image stream, our framework produces a 3D volumetric semantic + occupancy map, which is much more useful than a series of 2D semantic label images or a sparse point cloud produced by traditional semantic segmentation and Structure from Motion(SfM) pipelines respectively. We derive a Conditional Random Field (CRF) model defined in the 3D space, that jointly infers the semantic category and occupancy for each voxel. Such a joint inference in the 3D CRF paves the way for more informed priors and constraints, which is otherwise not possible if solved separately in their traditional frameworks. We make use of class specific semantic cues that constrain the 3D structure in areas, where multiview constraints are weak. Our model comprises of higher order factors, which helps when the depth is unobservable.We also make use of class specific semantic cues to reduce either the degree of such higher order factors, or to approximately model them with unaries if possible. We demonstrate improved 3D structure and temporally consistent semantic segmentation for difficult, large scale, forward moving monocular image sequences.

   @inproceedings{KunduECCV2014JointSegRec,
     title = {Joint Semantic Segmentation and 3D Reconstruction from Monocular Video},
     author = {Kundu, Abhijit and Li, Yin and Dellaert, Frank and Li, Fuxin and Rehg, James M.},
     booktitle = {ECCV},
     year = {2014},
     doi = {10.1007/978-3-319-10599-4_45},
     url = {https://doi.org/10.1007/978-3-319-10599-4_45},
     isbn = {978-3-319-10599-4},
   }