WP2 fortnightly meeting

mercredi 24 mars 2021, 11:00 → 12:30 Europe/Paris

Live Minutes

ZOOM video connection

1 News

Orateurs: Rosie Bolton (Square Kilometre Array Organisation), Xavier Espinal (CERN)

Activities

• EOSC-Future update. "last" consortium meeting on Monday, revised proposal to be sent today/tomorrow. 
• During the last month we gathered information from experiments and partner sites to know individual goals and objectives. The roadmap is quite clear and the commonality of interests is big. Strong synergies across the experiments and partner sites.
• Program of Work for ESCAPE 2nd phase second phase almost finished. Focused and very challenging. To be cristalysed on a full scale exercise by end November, the codename is DAC21 (Data and Analysis Challenge)
• WP2/WP5 joint workshop: 6th and 7tf of April 13:00 to 17:00 
• Commercial clouds, ephemeral resources, integration work ongoing: AWS and Google. Frederic from LAPP leading the activity, some preliminary results (presentation in two weeks)
• Data Lake and HPC: communication note is out [link] (credits to Tomasso, Diego, Lucia and all the INFN team)
• Full Dress Rehearsal/WP2 workshop communication note: [link]
• Synergies with CS3MESH4EOSC and LOFAR. Possibility to leverage the Data Lake infrastructure with end-user oriented and/or educational purposes through the Analysis Platforms and Notebooks Infrastructure.
• Preparation of a mini-workshop between Experiments and RUCIO, specially focused on the demands from not communities with less experience on data management, might have special needs, i.e.. metadata extension. Poll to identify possible dates to be circulated soon, Alba in charge.
• Two vCHEP contributions from CERN. Propose to add "thanks to ESCAPE GA" reference add the end of the paper and "on behalf of WP2" next to the main author(s).   

 

 

 

2 DIOS Second Phase, towards DAC21

Orateurs: Andrea Ceccanti (INFN), Paul Millar (DESY), Dr Riccardo Di Maria (CERN), Rosie Bolton (Square Kilometre Array Organisation), Xavier Espinal (CERN), Yan Grange (ASTRON, the Netherlands Institute for Radio Astronomy)

Proposal DAC21

T2.1 Plans for DAC21

----------------------------------------------------------------------------------------------------------------------------------------------

Proposal to re-gather in the Data Injector Demonstrators forum with focus on:

• Experiments needs and interests (always open to sciences/partners proposals)

  • Data registration and/or buffering injection

  • Data life-cycle aka QoS (cross-task activity with T2.2)

  • Deterministic vs. non-deterministic RSEs

  • Data/dataset size and volume (effect on namespace, tape-disk moving behaviour, etc.)

• Exploring “not-Rucio-aware site” use case

  • Data registration and/or buffering injection from non-Rucio sites (e.g. FTS)

    • Data handling and management of Telescope or Astronomy facilities,
      and addressing scenarios from Photon and Neutron sciences

  • HTTP-based storage solution → easy to use and to be adopted by communities

• Data Preparation

  • Calibration, reprocessing, formatting, etc.

  • Usually drives a conspicuous part of the computing model and data management
    (e.g. MonteCarlo campaigns, RAW→AnalysisObjectData→format-user-friendly, etc.)

  • Experiment perspective drives the activity

 

Data Injector+Access Demonstrators

Proposal for a cross-task&WP forum → Data Injector+Access Demonstrators

• Expanding fortnightly forum hosted every other Wednesday at 1100 CET

• Joint effort with T2.3 and WP5

  • Data Access and User Analysis Platform (e.g. JupyterLab-Rucio integration)

  • Caching layer and integration

  • … (T2.3 leading)

• Leveraging effort in EU-funded projects → ESCAPE+CS3MESH4EOSC

  • LOFAR and CERN have already activities and effort in both projects

  • Rucio + CERN + CS3MESH4EOSC = GSoC2021

 

Data Lake Activities

Outcome/takeaway of FDR shows room for improvements in view of DAC21

Several partners started to deploy parallel Rucio instances

• Perfect opportunity to jump on challenging activities and improvements

  • Motto should be: Try, Test, Assess, and Report

  • Experiment interests/needs as keystones

• Explore and discover new orchestration tools and phase-spaces 

  • “metadata”, “multi-VO”, “Rucio Auth schema/policies for ESCAPE”, “automatix”, “bb8”, etc.

  • Activities are already on-going

• Goal to demonstrate the full exportability of unique contributions by integrating them directly in the ESCAPE Data Lake for community-based testing

----------------------------------------------------------------------------------------------------------------------------------------------

Task 2.3

----------------------------------------------------------------------------------------------------------------------------------------------

The most important task from the T2.3 point of view is the actual integration. This touches a lot with many of the activities in the other tasks.

• Define end-to-end use cases, including cases that are part of  the “not-Rucio aware” use case mentioned in Task 2.1

  • Use those to define the ways in which data is to be accessed by software running on central processing facilities.

• AAI implementation is relevant here

  • Access to RUCIO from within jupyter lab containers

• External data access (e.g. VO -> WP4; or external archives)

----------------------------------------------------------------------------------------------------------------------------------------------

 

​​​​​​​

3 Depops hot topics

Orateur: Depops weekly chair

4 AOB