Validation of neural spike sorting algorithms without ground-truth information

Alex H. Barnett, Jeremy F. Magland, Leslie Greengard

Research output: Contribution to journalArticle

Abstract

Background: The throughput of electrophysiological recording is growing rapidly, allowing thousands of simultaneous channels, and there is a growing variety of spike sorting algorithms designed to extract neural firing events from such data. This creates an urgent need for standardized, automatic evaluation of the quality of neural units output by such algorithms. New method: We introduce a suite of validation metrics that assess the credibility of a given automatic spike sorting algorithm applied to a given dataset. By rerunning the spike sorter two or more times, the metrics measure stability under various perturbations consistent with variations in the data itself, making no assumptions about the internal workings of the algorithm, and minimal assumptions about the noise. Results: We illustrate the new metrics on standard sorting algorithms applied to both in vivo and ex vivo recordings, including a time series with overlapping spikes. We compare the metrics to existing quality measures, and to ground-truth accuracy in simulated time series. We provide a software implementation. Comparison with existing methods: Metrics have until now relied on ground-truth, simulated data, internal algorithm variables (e.g. cluster separation), or refractory violations. By contrast, by standardizing the interface, our metrics assess the reliability of any automatic algorithm without reference to internal variables (e.g. feature space) or physiological criteria. Conclusions: Stability is a prerequisite for reproducibility of results. Such metrics could reduce the significant human labor currently spent on validation, and should form an essential part of large-scale automated spike sorting and systematic benchmarking of algorithms.

Original languageEnglish (US)
Pages (from-to)65-77
Number of pages13
JournalJournal of Neuroscience Methods
Volume264
DOIs
StatePublished - May 1 2016

Fingerprint

Benchmarking
Reproducibility of Results
Noise
Software
Datasets

Keywords

  • Algorithms
  • Automatic
  • Spike sorting
  • Stability
  • Validation

ASJC Scopus subject areas

  • Neuroscience(all)

Cite this

Validation of neural spike sorting algorithms without ground-truth information. / Barnett, Alex H.; Magland, Jeremy F.; Greengard, Leslie.

In: Journal of Neuroscience Methods, Vol. 264, 01.05.2016, p. 65-77.

Research output: Contribution to journalArticle

@article{f1c7741052fb424fadf80445f4095917,
title = "Validation of neural spike sorting algorithms without ground-truth information",
abstract = "Background: The throughput of electrophysiological recording is growing rapidly, allowing thousands of simultaneous channels, and there is a growing variety of spike sorting algorithms designed to extract neural firing events from such data. This creates an urgent need for standardized, automatic evaluation of the quality of neural units output by such algorithms. New method: We introduce a suite of validation metrics that assess the credibility of a given automatic spike sorting algorithm applied to a given dataset. By rerunning the spike sorter two or more times, the metrics measure stability under various perturbations consistent with variations in the data itself, making no assumptions about the internal workings of the algorithm, and minimal assumptions about the noise. Results: We illustrate the new metrics on standard sorting algorithms applied to both in vivo and ex vivo recordings, including a time series with overlapping spikes. We compare the metrics to existing quality measures, and to ground-truth accuracy in simulated time series. We provide a software implementation. Comparison with existing methods: Metrics have until now relied on ground-truth, simulated data, internal algorithm variables (e.g. cluster separation), or refractory violations. By contrast, by standardizing the interface, our metrics assess the reliability of any automatic algorithm without reference to internal variables (e.g. feature space) or physiological criteria. Conclusions: Stability is a prerequisite for reproducibility of results. Such metrics could reduce the significant human labor currently spent on validation, and should form an essential part of large-scale automated spike sorting and systematic benchmarking of algorithms.",
keywords = "Algorithms, Automatic, Spike sorting, Stability, Validation",
author = "Barnett, {Alex H.} and Magland, {Jeremy F.} and Leslie Greengard",
year = "2016",
month = "5",
day = "1",
doi = "10.1016/j.jneumeth.2016.02.022",
language = "English (US)",
volume = "264",
pages = "65--77",
journal = "Journal of Neuroscience Methods",
issn = "0165-0270",
publisher = "Elsevier",

}

TY - JOUR

T1 - Validation of neural spike sorting algorithms without ground-truth information

AU - Barnett, Alex H.

AU - Magland, Jeremy F.

AU - Greengard, Leslie

PY - 2016/5/1

Y1 - 2016/5/1

N2 - Background: The throughput of electrophysiological recording is growing rapidly, allowing thousands of simultaneous channels, and there is a growing variety of spike sorting algorithms designed to extract neural firing events from such data. This creates an urgent need for standardized, automatic evaluation of the quality of neural units output by such algorithms. New method: We introduce a suite of validation metrics that assess the credibility of a given automatic spike sorting algorithm applied to a given dataset. By rerunning the spike sorter two or more times, the metrics measure stability under various perturbations consistent with variations in the data itself, making no assumptions about the internal workings of the algorithm, and minimal assumptions about the noise. Results: We illustrate the new metrics on standard sorting algorithms applied to both in vivo and ex vivo recordings, including a time series with overlapping spikes. We compare the metrics to existing quality measures, and to ground-truth accuracy in simulated time series. We provide a software implementation. Comparison with existing methods: Metrics have until now relied on ground-truth, simulated data, internal algorithm variables (e.g. cluster separation), or refractory violations. By contrast, by standardizing the interface, our metrics assess the reliability of any automatic algorithm without reference to internal variables (e.g. feature space) or physiological criteria. Conclusions: Stability is a prerequisite for reproducibility of results. Such metrics could reduce the significant human labor currently spent on validation, and should form an essential part of large-scale automated spike sorting and systematic benchmarking of algorithms.

AB - Background: The throughput of electrophysiological recording is growing rapidly, allowing thousands of simultaneous channels, and there is a growing variety of spike sorting algorithms designed to extract neural firing events from such data. This creates an urgent need for standardized, automatic evaluation of the quality of neural units output by such algorithms. New method: We introduce a suite of validation metrics that assess the credibility of a given automatic spike sorting algorithm applied to a given dataset. By rerunning the spike sorter two or more times, the metrics measure stability under various perturbations consistent with variations in the data itself, making no assumptions about the internal workings of the algorithm, and minimal assumptions about the noise. Results: We illustrate the new metrics on standard sorting algorithms applied to both in vivo and ex vivo recordings, including a time series with overlapping spikes. We compare the metrics to existing quality measures, and to ground-truth accuracy in simulated time series. We provide a software implementation. Comparison with existing methods: Metrics have until now relied on ground-truth, simulated data, internal algorithm variables (e.g. cluster separation), or refractory violations. By contrast, by standardizing the interface, our metrics assess the reliability of any automatic algorithm without reference to internal variables (e.g. feature space) or physiological criteria. Conclusions: Stability is a prerequisite for reproducibility of results. Such metrics could reduce the significant human labor currently spent on validation, and should form an essential part of large-scale automated spike sorting and systematic benchmarking of algorithms.

KW - Algorithms

KW - Automatic

KW - Spike sorting

KW - Stability

KW - Validation

UR - http://www.scopus.com/inward/record.url?scp=84960327739&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84960327739&partnerID=8YFLogxK

U2 - 10.1016/j.jneumeth.2016.02.022

DO - 10.1016/j.jneumeth.2016.02.022

M3 - Article

C2 - 26930629

AN - SCOPUS:84960327739

VL - 264

SP - 65

EP - 77

JO - Journal of Neuroscience Methods

JF - Journal of Neuroscience Methods

SN - 0165-0270

ER -