Accelerating data for compute clusters and grids

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Effective scaling and maximal throughput for clusters is an ongoing issue for computational workloads. We will discuss an approach to workload distribution which is data-centric, rather than process-centric. Data is moved to nodes ahead of computation. The workload management is handled as usual, except that the presence of data at nodes is used to trigger process scheduling. We will discuss how to create workflows which are data-activated and how an asynchronous pipeline can be established, allowing file serving latency to be hidden.In addition our RepliCator solution provides broadcasting to all nodes in a cluster simultaneously, substantially reducing data transfer times. This can be highly effective for throughput problems. We will present benchmark results obtained on clusters with several hundred cpus. Speedups in overall processing of factors of 2.5 to 4.5 have been observed using the combination of data-activated processing and data broadcasting provided by RepliCator data transfer management software.

Original languageEnglish (US)
Title of host publicationProceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06
DOIs
StatePublished - Dec 1 2006

Publication series

NameProceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06

ASJC Scopus subject areas

  • Computer Science(all)

Fingerprint Dive into the research topics of 'Accelerating data for compute clusters and grids'. Together they form a unique fingerprint.

  • Cite this

    Marchand, B. (2006). Accelerating data for compute clusters and grids. In Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06 [1188731] (Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC'06). https://doi.org/10.1145/1188455.1188731