Accelerating data for compute clusters and grids

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Effective scaling and maximal throughput for clusters is an ongoing issue for computational workloads. We will discuss an approach to workload distribution which is data-centric, rather than process-centric. Data is moved to nodes ahead of computation. The workload management is handled as usual, except that the presence of data at nodes is used to trigger process scheduling. We will discuss how to create workflows which are data-activated and how an asynchronous pipeline can be established, allowing file serving latency to be hidden.In addition our RepliCator solution provides broadcasting to all nodes in a cluster simultaneously, substantially reducing data transfer times. This can be highly effective for throughput problems. We will present benchmark results obtained on clusters with several hundred cpus. Speedups in overall processing of factors of 2.5 to 4.5 have been observed using the combination of data-activated processing and data broadcasting provided by RepliCator data transfer management software.

Original languageEnglish (US)
Title of host publicationProceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06
DOIs
StatePublished - Dec 1 2006

Fingerprint

Data transfer
Broadcasting
Throughput
Processing
Pipelines
Scheduling

ASJC Scopus subject areas

  • Computer Science(all)

Cite this

Marchand, B. (2006). Accelerating data for compute clusters and grids. In Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06 [1188731] https://doi.org/10.1145/1188455.1188731

Accelerating data for compute clusters and grids. / Marchand, Benoit.

Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06. 2006. 1188731.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Marchand, B 2006, Accelerating data for compute clusters and grids. in Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06., 1188731. https://doi.org/10.1145/1188455.1188731
Marchand B. Accelerating data for compute clusters and grids. In Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06. 2006. 1188731 https://doi.org/10.1145/1188455.1188731
Marchand, Benoit. / Accelerating data for compute clusters and grids. Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06. 2006.
@inproceedings{2f9f57aa2a884397840e8cfcad0e3cb7,
title = "Accelerating data for compute clusters and grids",
abstract = "Effective scaling and maximal throughput for clusters is an ongoing issue for computational workloads. We will discuss an approach to workload distribution which is data-centric, rather than process-centric. Data is moved to nodes ahead of computation. The workload management is handled as usual, except that the presence of data at nodes is used to trigger process scheduling. We will discuss how to create workflows which are data-activated and how an asynchronous pipeline can be established, allowing file serving latency to be hidden.In addition our RepliCator solution provides broadcasting to all nodes in a cluster simultaneously, substantially reducing data transfer times. This can be highly effective for throughput problems. We will present benchmark results obtained on clusters with several hundred cpus. Speedups in overall processing of factors of 2.5 to 4.5 have been observed using the combination of data-activated processing and data broadcasting provided by RepliCator data transfer management software.",
author = "Benoit Marchand",
year = "2006",
month = "12",
day = "1",
doi = "10.1145/1188455.1188731",
language = "English (US)",
isbn = "0769527000",
booktitle = "Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06",

}

TY - GEN

T1 - Accelerating data for compute clusters and grids

AU - Marchand, Benoit

PY - 2006/12/1

Y1 - 2006/12/1

N2 - Effective scaling and maximal throughput for clusters is an ongoing issue for computational workloads. We will discuss an approach to workload distribution which is data-centric, rather than process-centric. Data is moved to nodes ahead of computation. The workload management is handled as usual, except that the presence of data at nodes is used to trigger process scheduling. We will discuss how to create workflows which are data-activated and how an asynchronous pipeline can be established, allowing file serving latency to be hidden.In addition our RepliCator solution provides broadcasting to all nodes in a cluster simultaneously, substantially reducing data transfer times. This can be highly effective for throughput problems. We will present benchmark results obtained on clusters with several hundred cpus. Speedups in overall processing of factors of 2.5 to 4.5 have been observed using the combination of data-activated processing and data broadcasting provided by RepliCator data transfer management software.

AB - Effective scaling and maximal throughput for clusters is an ongoing issue for computational workloads. We will discuss an approach to workload distribution which is data-centric, rather than process-centric. Data is moved to nodes ahead of computation. The workload management is handled as usual, except that the presence of data at nodes is used to trigger process scheduling. We will discuss how to create workflows which are data-activated and how an asynchronous pipeline can be established, allowing file serving latency to be hidden.In addition our RepliCator solution provides broadcasting to all nodes in a cluster simultaneously, substantially reducing data transfer times. This can be highly effective for throughput problems. We will present benchmark results obtained on clusters with several hundred cpus. Speedups in overall processing of factors of 2.5 to 4.5 have been observed using the combination of data-activated processing and data broadcasting provided by RepliCator data transfer management software.

UR - http://www.scopus.com/inward/record.url?scp=34548209952&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=34548209952&partnerID=8YFLogxK

U2 - 10.1145/1188455.1188731

DO - 10.1145/1188455.1188731

M3 - Conference contribution

AN - SCOPUS:34548209952

SN - 0769527000

SN - 9780769527000

BT - Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06

ER -