S'authentifier

WP5 monthly meeting 20 January 2020

→ Europe/Paris
Description

Meeting with all partners involved in WP5/ESAP

You have been invited to attend a VidyoConference taking place in virtual meeting room:
ASTRON Zoom Room 4 at https://astron.zoom.us

Meeting ID: 944 740 9884

Join Zoom Meeting
https://astron.zoom.us/j/9447409884

Join by web (Chrome only)
https://zoom.us/wc/join/9447409884

Dial by your location
        +31 (0) 20 241 0288 Netherlands
        +31 (0) 70 808 1008 Netherlands
Meeting ID: 944 740 9884
Find your local number: https://zoom.us/u/acRpGgV86D

Join by Skype for Business

https://astron.zoom.us/skype/9447409884

For using the full Zoom client, please download the latest client from https://astron.zoom.us/download

 

Contact
Cacher

ESCAPE WP5 Monthly Meeting notes 20 January 2020

1. Roll call

ASTRON: Zheng Meyer, Christine Werkhoven

CERN: Aristeidis Fkiaras, Riccardo Di Maria, Rizart Dona

CNRS: Enrique Garcia

CSIC: Susana Sanchez

CTAO: Matthias Fuessling

EGO: Pierre Chanial

FAIR: Raffaele Grosso

FAU: Jutta Schnabel

KIS: -

INAF: -

IFAE: Jordi Delgado Mengual

JIVE: Harro Verkouter

Nikhef: -

OU: Hugh Dickinson

RUG: Myroslav Kavatsuyk

SKAO: Rosie Bolton

UCM: Daniel Nieto

UEDIN: Stelios Voutsinas, Dave Morris

2. Last meeting minutes

-

 

3. Announcements, news and general updates

ASTRON

Each of the MVP (Minimum Viable Product) core team partners are leading the development of one or more service categories:

  • ASTRON is leading the development of the ESAP UI and API Gateway.

  • SKA and CTA lead the aspects of data lake and workload management integration.

  • Nikhef leads on the federated-AAI aspects.

  • UEDIN leads the IVOA integration and interactive data processing services.

  • INAF is leading the development of batch processing.

WP5 has been invited to give a presentation at the CS3 workshop, where the CS3MESH project will kick-off. We are also invited to the CS3MESH4EOSC Kick-off Event. Both events will take place in Copenhagen between 27-30 January.

Two web developers (1.2 FTE) are working on the ESAP UI and API Gateway development at ASTRON. We expect to have the first version ready before the developers busy week. 

CERN

  • Discussion with SWAN and Rucio this week for possible integration (still really early stages)

  • Preparing some presentations for Brussels meeting which will possibly be interesting to WP5. (Datalake usage and access showcase/tutorial)

  • Rucio workshop 10th of March at Fermilab https://indico.cern.ch/event/867913/

  • ISGC2020 March 8th in Taipei http://indico4.twgrid.org/indico/event/11/program

CSIC -

CTAO

  • CTAO is now hiring, the job advertisement can be found at: https://www.cta-observatory.org/jobs/#1579265699488-cce62d7a-7bc9 The position aims to support CTAO activities in all ESCAPE work packages (WP3, WP4, WP5) with a special focus on WP5.

  • Beyond that, we are currently coordinating the activities of the CTA partners within (and outside) ESCAPE on the topics of Rucio and DIRAC (CTA is active both in WP2 and WP5 on these topics).

FAU

Within ESCAPE: mostly activity related to WP3 - WP5 links:

  • Common phone call on distributed computing (Dirac) in tech call

  • note: ESCAPE workshop on innovative workflows 7-10 July in Salerno (tbc)

For KM3NeT: development of workflows ongoing (including Jupyter notebooks as interfaces, CWL integration), no additional hiring is planned.

INAF -

IFAE

Objective: use Big Data platforms/tools to analyze MAGIC/CTA (gamma-ray astronomy data)

 

The current status of our work is described in the following. Green as Done, Orange on Going. The rest, pending tasks

JIVE

The new, bi-directional, EVN archive is currently under analysis. A new/parallel database needs to be designed first to enable the WP5 goal of receiving and linking to derived data and updated workflows. We are investigating different pipeline schemes (Jupyter notebook, Common Workflow Language) to write our re-runnable pipeline(s) in.

KIS -

Nikhef

Working on exploring component integrations that might be of interest for the ESAP. Knowledge for design and development of large, long-lived and secure software systems is available. There will be Nikhef attendance at ISGC as well.

RUG -

OU -

SKAO

Not much to report from SKA - we still have to people actively working on WP5 activities - looking at platform technologies (K8s, LSST SW stack), using DIRAC workload management service on UK grid (under SKAtelescope VO as a testbed) and looking at ways to improve reproducibility in SKA's science data challenges - looking at open-sourcing some code for assessing submissions and at features possible through kaggle.

UEDIN

We've been experimenting with running a a Hadoop (Yarn/HDFS) / Spark cluster, with a Apache Zeppelin as the interactive analysis tool, and data stored in Parquet format to run analysis on Gaia DR2 data. Alongside this we're looking at different cloud technologies such as Kubernetes/Helm & Ansible. We're also running a JupyterHub service and working on Docker images that build environment (kernels) that allow users to access and discover data through VO tools.

 

4. WP5 Busy Week - end of March/beginning of April

We want developers to attend this meeting.

List of participants:

  • ASTRON: Nico Vermaas + Zheng Meyer-Zhao

  • CERN: Aris Fkiaras

  • SKAO: James & Rohini

  • UEDIN: Stelios Voutsinas

Date:

Zheng will set-up a doodle please let her know if she needs to invite any else.

Location:

The Netherlands, venue tbd

 

5. WP4 Tech Forum - 4-6 February

Participants from WP5

JIVE: Harro Verkouter

EGO: Pierre Chanial

UEDIN: Dave Morris

CTAO: Matthias Fuessling (not 100% sure yet)

ASTRON: Zheng Meyer-Zhao, Yan Grange, Mattia Mancini

Zheng will present the WP5 developments.

 

6. ESCAPE Progress meeting 26-27 February , Brussels, BE

The list of participants is now online available at: https://indico.in2p3.fr/event/20203/registrations/participants 

WP2/WP5 joint session topics will be discussed via email.

WP4/WP5 joint sessions will be a continuation of the WP4 Tech Forum meeting.

 

7. WP5 Tech meetings Topics

Matthias Fuessling

  • Talk/contribution by J. Colomé on a Multi-Messenger Platform prototype build up to coordinate observations, the collaboration and (to some lesser extend) the data analysis for multi-messenger use cases.

Pierre Chanial

  • ESA talk about the science platform

  • invite Microsoft representative

Rosie Bolton:

  • LSST Software Stack by James, Rohini

Stelios Voutsinas

  • I'd be interested as well in having a Zoom meeting to discuss the LSST stack as well, as I've recently been experimenting with setting up the LSP (LSST Science Platform) with K8s on Openstack.

  • Topic: Collaboration tools for MVP such as github. How do we collaborate remotely and what are the rules?

Susana Sanchez

  • MVP and how to implement a new use case

Riccardo Di Maria

  • SWAN and Rucio

8. Round table

FAU: WP3 will organize a workshop on innovative workflows 7-10 July in Salerno (tbc). We are looking for experts on ML DL please forward names to Jutta Schnabel.

UCM: Hiring is finalized, Daniel Neto will fill the post doc position and start the end of February.


 

Il y a un compte-rendu associé à cet événement. Les afficher.
    • 11:00 → 11:03
      1. Roll call 3m

      ASTRON: Zheng Meyer, Christine Werkhoven

      CERN: Aristeidis Fkiaras, Riccardo Di Maria, Rizart Dona

      CNRS: Enrique Garcia

      CSIC: Susana Sanchez

      CTAO: Matthias Fuessling

      EGO: Pierre Chanial

      FAIR: Raffaele Grosso

      FAU: Jutta Schnabel

      KIS: -

      INAF: -

      IFAE: Jordi Delgado Mengual

      JIVE: Harro Verkouter

      Nikhef: -

      OU: Hugh Dickinson

      RUG: Myroslav Kavatsuyk

      SKAO: Rosie Bolton

      UCM: Daniel Neto

      UEDIN: Stelios Voutsinas, Dave Morris

       

    • 11:03 → 11:05
      2. Last meeting minutes 2m

      Minutes of last meeting can be found at
      https://indico.in2p3.fr/event/20430

    • 11:05 → 11:15
      3. Announcements, news and general updates 10m

      Provide your information by adding minutes to the sub-contribution. You will need to have an indico account to do this.
      If you are not able to add the information please send the info to werkhoven@astron.nl. We will then add it for you.

      wp5-1.png
      wp5-2.png
      • ASTRON 1m

        Each of the MVP (Minimum Viable Product) core team partners are leading the development of one or more service categories:

        • ASTRON is leading the development of the ESAP UI and API Gateway.
        • SKA and CTA lead the aspects of data lake and workload management integration.
        • Nikhef leads on the federated-AAI aspects.
        • UEDIN leads the IVOA integration and interactive data processing services.
        • INAF is leading the development of batch processing.

        WP5 has been invited to give a presentation at the CS3 workshop, where the CS3MESH project will kick-off. We are also invited to the CS3MESH4EOSC Kick-off Event. Both events will take place in Copenhagen between 27-30 January.
         

        Two web developers (1.2 FTE) are working on the ESAP UI and API Gateway development at ASTRON. We expect to have the first version ready before the developers busy week. 

      • CERN 1m
      • CNRS-LAPP 1m
      • CSIC 1m
      • CTAO 1m

        CTAO is now hiring, the job advertisement can be found at: https://www.cta-observatory.org/jobs/#1579265699488-cce62d7a-7bc9
        The position aims to support CTAO activities in all ESCAPE work packages (WP3, WP4, WP5) with a special focus on WP5.

        Beyond that, we are currently coordinating the activities of the CTA partners within (and outside) ESCAPE on the topics of Rucio and DIRAC (CTA is active both in WP2 and WP5 on these topics)."

      • FAIR 1m
      • EGO 1m
      • FAU 1m

        Within ESCAPE: mostly activity related to WP3 - WP5 links:

        • Common phone call on distributed computing (Dirac) in tech call
        • note: ESCAPE workshop on innovative workflows 7-10 July in Salerno (tbc)

        For KM3NeT: development of workflows ongoing (including Jupyter notebooks as interfaces, CWL integration), no additional hiring is planned.

      • INAF 1m
      • IFAE 1m
        Objective: use Big Data platforms/tools to analyze MAGIC/CTA (gamma-ray astronomy data)

         

        The current status of our work is described in the following. Green as Done, Orange on Going. The rest, pending tasks
      • JIVE 1m

        The new, bi-directional, EVN archive is currently under analysis. A new/parallel database needs to be designed first to enable the WP5 goal of receiving and linking to derived data and updated workflows. We are investigating different pipeline schemes (Jupyter notebook, Common Workflow Language) to write our re-runnable pipeline(s) in.

         

      • KIS 1m
      • Nikhef 1m
      • RUG 1m
      • SKAO 1m

        (K8s, LSST SW stack), using DIRAC workload management service on UK  grid (under SKAtelescope VO as a testbed) and looking at ways to improve reproducibility in SKA's science data challenges - looking at open-sourcing some code for assessing submissions and at features possible through kaggle.

      • UCM 1m
      • UEDIN 1m

        We've been experimenting with running a a Hadoop (Yarn/HDFS) / Spark cluster, with a Apache Zeppelin as the interactive analysis tool, and data stored in Parquet format to run analysis on Gaia DR2 data. Alongside this we're looking at different cloud technologies such as Kubernetes/Helm & Ansible. We're also running a JupyterHub service and working on Docker images that build environment (kernels) that allow users to access and discover data through VO tools.

    • 11:15 → 11:25
      4. WP5 Busy Week - end of March/start of April 10m

      Discuss: preparations, goal of the week, date, locations and who needs to attend

    • 11:25 → 11:35
      5. WP4 Tech Forum - 4-6 February 10m

      https://indico.in2p3.fr/event/20005/
      What will be the input from WP5?

      Participants from WP5

      JIVE: Harro Verkouter

      EGO: Pierre Chanial

      UEDIN: Dave Morris

      CERN: Matthias Fuessling

      Zheng will attend the meeting and present the current developments.

    • 11:35 → 11:45
      6. ESCAPE Progress meeting 26-27 February , Brussels, BE 10m

      More info about the event: https://indico.in2p3.fr/event/20203/
      Who will attend from WP5 and what will we present?

      The list of participants is now online available at: https://indico.in2p3.fr/event/20203/registrations/participants

      WP2/WP5 joint session topics will be discussed via email.

      WP4/WP5 joint sessions will be a continuation of the WP4 Tech Forum meeting.

    • 11:45 → 11:50
      7. WP5 Tech meetings Topics 5m

      Decide on topics. Suggestions topics/speaker

      Matthias Fuessling

      • Talk/contribution by J. Colomé on a Multi-Messenger Platform prototype build up to coordinate observations, the collaboration and (to some lesser extend) the data analysis for multi-messenger use cases.

      Pierre Chanial

      • ESA talk about the science platform

      • Invite Microsoft representative

      Rosie Bolton:

      • LSST Software Stack by James, Rohini

      Stelios Voutsinas

      • I'd be interested as well in having a Zoom meeting to discuss the LSST stack as well, as I've recently been experimenting with setting up the LSP (LSST Science Platform) with K8s on Openstack.

      • Topic: Collaboration tools for MVP such as github. How do we collaborate remotely and what are the rules?

      Susana Sanchez

      • MVP and how to implement a new use case

      Riccardo Di Maria

      • SWAN and Rucio

       

    • 11:50 → 11:55
      8. Round table 5m

      FAU: WP3 will organize a workshop on innovative workflows 7-10 July in Salerno (tbc). We are looking for experts on ML DL please forwared names to Jutta Schnabel.

      UCM: Hiring is finalized, Daniel Neto will fill the post doc position and start the end of February.

    • 11:55 → 12:00
      9. Next meeting - Monday 17 February 2020 - 11:00 AM CET 5m

      Note: later today we will have the Tech Meeting at 16:00 CET
      More info @ https://indico.in2p3.fr/event/20431/

Fonctionne avec Indico v3.3.5