The democratization of computational resources, thanks to the advent of public, private, and hybrid clouds, changed the rules in many science fields. For decades, one of the effort of computer scientists and computer engineers was the development of tools able to simplify access to high-end computational resources by computational scientists. However, nowadays any science field can be considered 'computational' as the availability of powerful, but easy to manage workflow engines, is crucial. In this work, we present DagOn∗ (Direct acyclic graph On anything), a lightweight Python library implementing a workflow engine able to execute parallel jobs represented by direct acyclic graphs on any combination of local machines, on-premise high performance computing clusters, containers, and cloud-based virtual infrastructures. We use a real-world production-level application for weather and marine forecasts to illustrate the use of this new workflow engine.

DagOn∗: Executing Direct Acyclic Graphs as Parallel Jobs on Anything

Montella R.;Di Luccio D.;
2019

Abstract

The democratization of computational resources, thanks to the advent of public, private, and hybrid clouds, changed the rules in many science fields. For decades, one of the effort of computer scientists and computer engineers was the development of tools able to simplify access to high-end computational resources by computational scientists. However, nowadays any science field can be considered 'computational' as the availability of powerful, but easy to manage workflow engines, is crucial. In this work, we present DagOn∗ (Direct acyclic graph On anything), a lightweight Python library implementing a workflow engine able to execute parallel jobs represented by direct acyclic graphs on any combination of local machines, on-premise high performance computing clusters, containers, and cloud-based virtual infrastructures. We use a real-world production-level application for weather and marine forecasts to illustrate the use of this new workflow engine.
978-1-7281-0196-5
File in questo prodotto:
Non ci sono file associati a questo prodotto.

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11367/87433
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus 16
  • ???jsp.display-item.citation.isi??? 10
social impact