Understanding deep architectures using a recursive convolutional network: 2nd International Conference on Learning Representations, ICLR 2014

David Eigen, Jason Rolfe, Robert Fergus, Yann LeCun

Research output: Contribution to conferencePaper

Abstract

A key challenge in designing convolutional network models is sizing them appropriately. Many factors are involved in these decisions, including number of layers, feature maps, kernel sizes, etc. Complicating this further is the fact that each of these influence not only the numbers and dimensions of the activation units, but also the total number of parameters. In this paper we focus on assessing the independent contributions of three of these linked variables: The numbers of layers, feature maps, and parameters. To accomplish this, we employ a recursive convolutional network whose weights are tied between layers; this allows us to vary each of the three factors in a controlled setting. We find that while increasing the numbers of layers and parameters each have clear benefit, the number of feature maps (and hence dimensionality of the representation) appears ancillary, and finds most of its benefit through the introduction of more weights. Our results (i) empirically confirm the notion that adding layers alone increases computational power, within the context of convolutional layers, and (ii) suggest that precise sizing of convolutional feature map dimensions is itself of little concern; more attention should be paid to the number of parameters in these layers instead.

Original languageEnglish (US)
StatePublished - Jan 1 2014
Event2nd International Conference on Learning Representations, ICLR 2014 - Banff, Canada
Duration: Apr 14 2014Apr 16 2014

Conference

Conference2nd International Conference on Learning Representations, ICLR 2014
CountryCanada
CityBanff
Period4/14/144/16/14

Fingerprint

learning
activation
Chemical activation
Layer

ASJC Scopus subject areas

  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics
  • Education

Cite this

Eigen, D., Rolfe, J., Fergus, R., & LeCun, Y. (2014). Understanding deep architectures using a recursive convolutional network: 2nd International Conference on Learning Representations, ICLR 2014. Paper presented at 2nd International Conference on Learning Representations, ICLR 2014, Banff, Canada.

Understanding deep architectures using a recursive convolutional network : 2nd International Conference on Learning Representations, ICLR 2014. / Eigen, David; Rolfe, Jason; Fergus, Robert; LeCun, Yann.

2014. Paper presented at 2nd International Conference on Learning Representations, ICLR 2014, Banff, Canada.

Research output: Contribution to conferencePaper

Eigen, D, Rolfe, J, Fergus, R & LeCun, Y 2014, 'Understanding deep architectures using a recursive convolutional network: 2nd International Conference on Learning Representations, ICLR 2014' Paper presented at 2nd International Conference on Learning Representations, ICLR 2014, Banff, Canada, 4/14/14 - 4/16/14, .
Eigen D, Rolfe J, Fergus R, LeCun Y. Understanding deep architectures using a recursive convolutional network: 2nd International Conference on Learning Representations, ICLR 2014. 2014. Paper presented at 2nd International Conference on Learning Representations, ICLR 2014, Banff, Canada.
Eigen, David ; Rolfe, Jason ; Fergus, Robert ; LeCun, Yann. / Understanding deep architectures using a recursive convolutional network : 2nd International Conference on Learning Representations, ICLR 2014. Paper presented at 2nd International Conference on Learning Representations, ICLR 2014, Banff, Canada.
@conference{693fc38bdf7d4a97a906dd259710bd7e,
title = "Understanding deep architectures using a recursive convolutional network: 2nd International Conference on Learning Representations, ICLR 2014",
abstract = "A key challenge in designing convolutional network models is sizing them appropriately. Many factors are involved in these decisions, including number of layers, feature maps, kernel sizes, etc. Complicating this further is the fact that each of these influence not only the numbers and dimensions of the activation units, but also the total number of parameters. In this paper we focus on assessing the independent contributions of three of these linked variables: The numbers of layers, feature maps, and parameters. To accomplish this, we employ a recursive convolutional network whose weights are tied between layers; this allows us to vary each of the three factors in a controlled setting. We find that while increasing the numbers of layers and parameters each have clear benefit, the number of feature maps (and hence dimensionality of the representation) appears ancillary, and finds most of its benefit through the introduction of more weights. Our results (i) empirically confirm the notion that adding layers alone increases computational power, within the context of convolutional layers, and (ii) suggest that precise sizing of convolutional feature map dimensions is itself of little concern; more attention should be paid to the number of parameters in these layers instead.",
author = "David Eigen and Jason Rolfe and Robert Fergus and Yann LeCun",
year = "2014",
month = "1",
day = "1",
language = "English (US)",
note = "2nd International Conference on Learning Representations, ICLR 2014 ; Conference date: 14-04-2014 Through 16-04-2014",

}

TY - CONF

T1 - Understanding deep architectures using a recursive convolutional network

T2 - 2nd International Conference on Learning Representations, ICLR 2014

AU - Eigen, David

AU - Rolfe, Jason

AU - Fergus, Robert

AU - LeCun, Yann

PY - 2014/1/1

Y1 - 2014/1/1

N2 - A key challenge in designing convolutional network models is sizing them appropriately. Many factors are involved in these decisions, including number of layers, feature maps, kernel sizes, etc. Complicating this further is the fact that each of these influence not only the numbers and dimensions of the activation units, but also the total number of parameters. In this paper we focus on assessing the independent contributions of three of these linked variables: The numbers of layers, feature maps, and parameters. To accomplish this, we employ a recursive convolutional network whose weights are tied between layers; this allows us to vary each of the three factors in a controlled setting. We find that while increasing the numbers of layers and parameters each have clear benefit, the number of feature maps (and hence dimensionality of the representation) appears ancillary, and finds most of its benefit through the introduction of more weights. Our results (i) empirically confirm the notion that adding layers alone increases computational power, within the context of convolutional layers, and (ii) suggest that precise sizing of convolutional feature map dimensions is itself of little concern; more attention should be paid to the number of parameters in these layers instead.

AB - A key challenge in designing convolutional network models is sizing them appropriately. Many factors are involved in these decisions, including number of layers, feature maps, kernel sizes, etc. Complicating this further is the fact that each of these influence not only the numbers and dimensions of the activation units, but also the total number of parameters. In this paper we focus on assessing the independent contributions of three of these linked variables: The numbers of layers, feature maps, and parameters. To accomplish this, we employ a recursive convolutional network whose weights are tied between layers; this allows us to vary each of the three factors in a controlled setting. We find that while increasing the numbers of layers and parameters each have clear benefit, the number of feature maps (and hence dimensionality of the representation) appears ancillary, and finds most of its benefit through the introduction of more weights. Our results (i) empirically confirm the notion that adding layers alone increases computational power, within the context of convolutional layers, and (ii) suggest that precise sizing of convolutional feature map dimensions is itself of little concern; more attention should be paid to the number of parameters in these layers instead.

UR - http://www.scopus.com/inward/record.url?scp=84986240784&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84986240784&partnerID=8YFLogxK

M3 - Paper

AN - SCOPUS:84986240784

ER -