Pegasus Workflow Management System Training

mardi 18 juin 2019, 08:30 → 18:00 Europe/Paris

Room 202 (CC-IN2P3)

1-day training on the Pegasus Workflow Management System

Speakers: Rafael Ferreira da Silva (ISI/USC) and Mats Rynge (ISI/USC)

Local support: Justin Bussery (CC-IN2P3) and Frédéric Suter (CC-IN2P3)

Training content:

This training targets administrators, developers, and scientists that are interested in scientific workflows. It will introduce Pegasus, one the major workflow management systems. Pegasus is being used in production to execute workflows for dozens of high-profile applications in a wide range of scientific domains. Pegasus provides the necessary abstractions for scientists to create workflows and allows for transparent execution of these workflows on a range of compute platforms including clusters, clouds, and national cyberinfrastructures. During execution, Pegasus translates an abstract resource-independent workflow into an executable workflow, determining the specific executables, data, and computational resources required for the execution. Workflow execution with Pegasus includes data management, monitoring, and failure handling, and is managed by HTCondor DAGMan. Individual workflow tasks are managed by the HTCondor workload management framework which supervises task executions on local and remote resources.

This training will include a presentation of Pegasus features through a hands-on tutorial and technical presentations on the following subjects:

• Support of Jupyter notebooks

• Data Integrity in scientific workflows

The training materials and talks will be in english.

Working environment:

TBD

Practical information:

Registration is free but mandatory.

This training session is however limited to CC-IN2P3 members, users and academic partners of CC-IN2P3, and CNRS agents. Any registration that does not satisfy this criteria could be rejected.

The training will span over two half days. It will start on Tuesday June 18, 2019 at 9AM (a welcome coffee will be available from 8:30AM) and will end on the same day around 6PM.

Note that it is not an official CNRS training. Participants thus do not have to make a training request.

For lunch, participants will have access to the administrative restaurant close to CC-IN2P3. A full meal costs about 11-12€. Cash (in euros) and credit cards are accepted.

For security reasons, you will have to present and leave a picture ID on arrival. Then you will receive a badge that you will have to keep and wear for your entire stay .  This badge will have to be returned to get your ID back when you leave.

Details about public transportation and how to access CC-IN2P3 can be found there.

 

08:30 Welcome coffee

Pegasus Tutorial

• Getting Started
• What are Scientific Workflows?
• Submitting Our First Workflow
• Command line tools for Monitoring and Debugging
• Generating the Workflow
• Information Catalogs
• Configuring Pegasus
• Recovery from Failures
• Conclusion

12:00 Lunch

Technical presentations

1 Support for Jupyter Notebooks

2 Data Integrity in Scientific Workflows

15:30 Coffee break

Open discussions