A processing pipeline for high volume pulsar candidate data streams

ROBERT LYON, Benjamin Stappers, Lina Levin, Mitchell Mickaliger, Anna Scaife

Research output: Contribution to journalArticle

Abstract

Pulsar candidate analysis pipelines have historically been comprised of bespoke software systems, supporting the off-line study of data. However modern data acquisition systems are making off-line analyses impractical, as increasing candidate volumes become prohibitively expensive to retain. To maintain processing capabilities when off-line analysis becomes infeasible due to cost, requires a shift to on-line data processing. This paper makes four contributions facilitating this shift relevant to the search for radio pulsars: (i) it characterises the key components of a pulsar search candidate processing pipeline, (ii) it examines the feasibility of implementing on-line candidate filtering via existing tools, (iii) explores the problems preventing an easy transition to on-line filtering, and finally (iv) presents a new prototype filtering pipeline capable of overcoming such problems. Realised using Commercial off-the-shelf (COTS) software components, the deployable system is open source, simple, scalable, and cheap to produce. It has the potential to achieve candidate filtering design requirements for the Square Kilometre Array (SKA), illustrated via testing under simulated SKA loads.
Original languageEnglish
Article number100291
JournalAstronomy and Computing
Volume28
Early online date12 Jun 2019
DOIs
Publication statusPublished - Jul 2019

Fingerprint

pulsars
Pipelines
Processing
software
computer programs
data acquisition
shift
Open systems
shelves
radio
Data acquisition
prototypes
costs
cost
requirements
Pulsars
Testing
Costs
analysis

Keywords

  • Pulsars: general
  • Methods: data analysis
  • Methods: statistical
  • Techniques: miscellaneous

Cite this

LYON, ROBERT ; Stappers, Benjamin ; Levin, Lina ; Mickaliger, Mitchell ; Scaife, Anna. / A processing pipeline for high volume pulsar candidate data streams. In: Astronomy and Computing. 2019 ; Vol. 28.
@article{b0fd43269dbd4b34809d5c5eee46bbe2,
title = "A processing pipeline for high volume pulsar candidate data streams",
abstract = "Pulsar candidate analysis pipelines have historically been comprised of bespoke software systems, supporting the off-line study of data. However modern data acquisition systems are making off-line analyses impractical, as increasing candidate volumes become prohibitively expensive to retain. To maintain processing capabilities when off-line analysis becomes infeasible due to cost, requires a shift to on-line data processing. This paper makes four contributions facilitating this shift relevant to the search for radio pulsars: (i) it characterises the key components of a pulsar search candidate processing pipeline, (ii) it examines the feasibility of implementing on-line candidate filtering via existing tools, (iii) explores the problems preventing an easy transition to on-line filtering, and finally (iv) presents a new prototype filtering pipeline capable of overcoming such problems. Realised using Commercial off-the-shelf (COTS) software components, the deployable system is open source, simple, scalable, and cheap to produce. It has the potential to achieve candidate filtering design requirements for the Square Kilometre Array (SKA), illustrated via testing under simulated SKA loads.",
keywords = "Pulsars: general, Methods: data analysis, Methods: statistical, Techniques: miscellaneous",
author = "ROBERT LYON and Benjamin Stappers and Lina Levin and Mitchell Mickaliger and Anna Scaife",
year = "2019",
month = "7",
doi = "10.1016/j.ascom.2019.100291",
language = "English",
volume = "28",
journal = "Astronomy and Computing",
issn = "2213-1337",
publisher = "Elsevier BV",

}

A processing pipeline for high volume pulsar candidate data streams. / LYON, ROBERT; Stappers, Benjamin; Levin, Lina; Mickaliger, Mitchell; Scaife, Anna.

In: Astronomy and Computing, Vol. 28, 100291, 07.2019.

Research output: Contribution to journalArticle

TY - JOUR

T1 - A processing pipeline for high volume pulsar candidate data streams

AU - LYON, ROBERT

AU - Stappers, Benjamin

AU - Levin, Lina

AU - Mickaliger, Mitchell

AU - Scaife, Anna

PY - 2019/7

Y1 - 2019/7

N2 - Pulsar candidate analysis pipelines have historically been comprised of bespoke software systems, supporting the off-line study of data. However modern data acquisition systems are making off-line analyses impractical, as increasing candidate volumes become prohibitively expensive to retain. To maintain processing capabilities when off-line analysis becomes infeasible due to cost, requires a shift to on-line data processing. This paper makes four contributions facilitating this shift relevant to the search for radio pulsars: (i) it characterises the key components of a pulsar search candidate processing pipeline, (ii) it examines the feasibility of implementing on-line candidate filtering via existing tools, (iii) explores the problems preventing an easy transition to on-line filtering, and finally (iv) presents a new prototype filtering pipeline capable of overcoming such problems. Realised using Commercial off-the-shelf (COTS) software components, the deployable system is open source, simple, scalable, and cheap to produce. It has the potential to achieve candidate filtering design requirements for the Square Kilometre Array (SKA), illustrated via testing under simulated SKA loads.

AB - Pulsar candidate analysis pipelines have historically been comprised of bespoke software systems, supporting the off-line study of data. However modern data acquisition systems are making off-line analyses impractical, as increasing candidate volumes become prohibitively expensive to retain. To maintain processing capabilities when off-line analysis becomes infeasible due to cost, requires a shift to on-line data processing. This paper makes four contributions facilitating this shift relevant to the search for radio pulsars: (i) it characterises the key components of a pulsar search candidate processing pipeline, (ii) it examines the feasibility of implementing on-line candidate filtering via existing tools, (iii) explores the problems preventing an easy transition to on-line filtering, and finally (iv) presents a new prototype filtering pipeline capable of overcoming such problems. Realised using Commercial off-the-shelf (COTS) software components, the deployable system is open source, simple, scalable, and cheap to produce. It has the potential to achieve candidate filtering design requirements for the Square Kilometre Array (SKA), illustrated via testing under simulated SKA loads.

KW - Pulsars: general

KW - Methods: data analysis

KW - Methods: statistical

KW - Techniques: miscellaneous

U2 - 10.1016/j.ascom.2019.100291

DO - 10.1016/j.ascom.2019.100291

M3 - Article

VL - 28

JO - Astronomy and Computing

JF - Astronomy and Computing

SN - 2213-1337

M1 - 100291

ER -