Computational multi-view imaging with kinect

Xinchen Ye, Jingyu Yang, Hao Huang, Chunping Hou, Yao Wang

Research output: Contribution to journalArticle

Abstract

The lack of 3-D content has become a bottleneck for the advancement of three-dimensional television (3-DTV), but conventional multicamera arrays for multiview imaging are expensive to setup and cumbersome to use. This paper proposes a lightweight multiview imaging approach with Kinect, a handheld integrated depth-color camera, under the depth-image-based rendering framework. The proposed method consists of two components: depth restoration from noisy and incomplete depth measurements and view synthesis from depth-color pairs. In depth restoration, we propose a moving 2-D polynomial approximation via least squares to suppress quantization errors in the acquired depth values, and propose a progressive edge-guided trilateral filter to fill missing areas of the depth map. Edges extracted from color image are used to predict the locations of depth discontinuities in missing areas and to guide the proposed trilateral filter avoiding filtering across discontinuities. In view synthesis, we propose a low-rank matrix restoration model to inpaint disocclusion regions, fully exploiting the nonlocal correlations in images, and devise an efficient algorithm under the augmented lagrange multiplier (ALM) framework. Disocclusion areas are inpainted progressively from the boundaries of disocclusion with an estimated priority consisting of four terms: warping term, reliability term, texture term, and depth term. Experimental results show that our method restores high quality depth maps even for large missing areas, and synthesizes natural multiview images from restored depth maps. Strong 3-D visual experiences are observed when the synthesized multiview images are shown in two types of stereoscopic displays.

Original languageEnglish (US)
Article number6881679
Pages (from-to)540-544
Number of pages5
JournalIEEE Transactions on Broadcasting
Volume60
Issue number3
DOIs
StatePublished - Sep 1 2014

Fingerprint

Restoration
Color
Imaging techniques
Polynomial approximation
Lagrange multipliers
Television
Textures
Cameras
Display devices

Keywords

  • depth restoration
  • Kinect
  • matrix completion
  • Multiview imaging
  • view synthesis

ASJC Scopus subject areas

  • Media Technology
  • Electrical and Electronic Engineering

Cite this

Computational multi-view imaging with kinect. / Ye, Xinchen; Yang, Jingyu; Huang, Hao; Hou, Chunping; Wang, Yao.

In: IEEE Transactions on Broadcasting, Vol. 60, No. 3, 6881679, 01.09.2014, p. 540-544.

Research output: Contribution to journalArticle

Ye, Xinchen ; Yang, Jingyu ; Huang, Hao ; Hou, Chunping ; Wang, Yao. / Computational multi-view imaging with kinect. In: IEEE Transactions on Broadcasting. 2014 ; Vol. 60, No. 3. pp. 540-544.
@article{25b86e49056f400198017321f306d43d,
title = "Computational multi-view imaging with kinect",
abstract = "The lack of 3-D content has become a bottleneck for the advancement of three-dimensional television (3-DTV), but conventional multicamera arrays for multiview imaging are expensive to setup and cumbersome to use. This paper proposes a lightweight multiview imaging approach with Kinect, a handheld integrated depth-color camera, under the depth-image-based rendering framework. The proposed method consists of two components: depth restoration from noisy and incomplete depth measurements and view synthesis from depth-color pairs. In depth restoration, we propose a moving 2-D polynomial approximation via least squares to suppress quantization errors in the acquired depth values, and propose a progressive edge-guided trilateral filter to fill missing areas of the depth map. Edges extracted from color image are used to predict the locations of depth discontinuities in missing areas and to guide the proposed trilateral filter avoiding filtering across discontinuities. In view synthesis, we propose a low-rank matrix restoration model to inpaint disocclusion regions, fully exploiting the nonlocal correlations in images, and devise an efficient algorithm under the augmented lagrange multiplier (ALM) framework. Disocclusion areas are inpainted progressively from the boundaries of disocclusion with an estimated priority consisting of four terms: warping term, reliability term, texture term, and depth term. Experimental results show that our method restores high quality depth maps even for large missing areas, and synthesizes natural multiview images from restored depth maps. Strong 3-D visual experiences are observed when the synthesized multiview images are shown in two types of stereoscopic displays.",
keywords = "depth restoration, Kinect, matrix completion, Multiview imaging, view synthesis",
author = "Xinchen Ye and Jingyu Yang and Hao Huang and Chunping Hou and Yao Wang",
year = "2014",
month = "9",
day = "1",
doi = "10.1109/TBC.2014.2345931",
language = "English (US)",
volume = "60",
pages = "540--544",
journal = "IEEE Transactions on Broadcasting",
issn = "0018-9316",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "3",

}

TY - JOUR

T1 - Computational multi-view imaging with kinect

AU - Ye, Xinchen

AU - Yang, Jingyu

AU - Huang, Hao

AU - Hou, Chunping

AU - Wang, Yao

PY - 2014/9/1

Y1 - 2014/9/1

N2 - The lack of 3-D content has become a bottleneck for the advancement of three-dimensional television (3-DTV), but conventional multicamera arrays for multiview imaging are expensive to setup and cumbersome to use. This paper proposes a lightweight multiview imaging approach with Kinect, a handheld integrated depth-color camera, under the depth-image-based rendering framework. The proposed method consists of two components: depth restoration from noisy and incomplete depth measurements and view synthesis from depth-color pairs. In depth restoration, we propose a moving 2-D polynomial approximation via least squares to suppress quantization errors in the acquired depth values, and propose a progressive edge-guided trilateral filter to fill missing areas of the depth map. Edges extracted from color image are used to predict the locations of depth discontinuities in missing areas and to guide the proposed trilateral filter avoiding filtering across discontinuities. In view synthesis, we propose a low-rank matrix restoration model to inpaint disocclusion regions, fully exploiting the nonlocal correlations in images, and devise an efficient algorithm under the augmented lagrange multiplier (ALM) framework. Disocclusion areas are inpainted progressively from the boundaries of disocclusion with an estimated priority consisting of four terms: warping term, reliability term, texture term, and depth term. Experimental results show that our method restores high quality depth maps even for large missing areas, and synthesizes natural multiview images from restored depth maps. Strong 3-D visual experiences are observed when the synthesized multiview images are shown in two types of stereoscopic displays.

AB - The lack of 3-D content has become a bottleneck for the advancement of three-dimensional television (3-DTV), but conventional multicamera arrays for multiview imaging are expensive to setup and cumbersome to use. This paper proposes a lightweight multiview imaging approach with Kinect, a handheld integrated depth-color camera, under the depth-image-based rendering framework. The proposed method consists of two components: depth restoration from noisy and incomplete depth measurements and view synthesis from depth-color pairs. In depth restoration, we propose a moving 2-D polynomial approximation via least squares to suppress quantization errors in the acquired depth values, and propose a progressive edge-guided trilateral filter to fill missing areas of the depth map. Edges extracted from color image are used to predict the locations of depth discontinuities in missing areas and to guide the proposed trilateral filter avoiding filtering across discontinuities. In view synthesis, we propose a low-rank matrix restoration model to inpaint disocclusion regions, fully exploiting the nonlocal correlations in images, and devise an efficient algorithm under the augmented lagrange multiplier (ALM) framework. Disocclusion areas are inpainted progressively from the boundaries of disocclusion with an estimated priority consisting of four terms: warping term, reliability term, texture term, and depth term. Experimental results show that our method restores high quality depth maps even for large missing areas, and synthesizes natural multiview images from restored depth maps. Strong 3-D visual experiences are observed when the synthesized multiview images are shown in two types of stereoscopic displays.

KW - depth restoration

KW - Kinect

KW - matrix completion

KW - Multiview imaging

KW - view synthesis

UR - http://www.scopus.com/inward/record.url?scp=85027943494&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85027943494&partnerID=8YFLogxK

U2 - 10.1109/TBC.2014.2345931

DO - 10.1109/TBC.2014.2345931

M3 - Article

AN - SCOPUS:85027943494

VL - 60

SP - 540

EP - 544

JO - IEEE Transactions on Broadcasting

JF - IEEE Transactions on Broadcasting

SN - 0018-9316

IS - 3

M1 - 6881679

ER -