WP2 fortnightly meeting

mercredi 12 févr. 2020, 11:00 → 12:00 Europe/Paris

Connection Details

WP2 Fortnightly Meeting - 12.02.2020 – Minutes
 

Assistants: Aleem Sarwar, Andrea Ceccanti, Simone Campana, Xavier Espinal, Riccardo Di Maria, Aristeidis Fkiaras, Rizart Dona, Daniele Spiga, Diego Ciangottini, Enrico Vianello, Fabio Hernandez, Frederic Gillardo, Ghita, Gonzalo Merino, Guido Aben, Marcelo Soares, Marek Szuba, Paul Millar, Rohini, James, Stephane Jezequel, Yan Grange, Zheng

Simone

• Last meeting before Brussel meeting
• Tape-ISO certification meeting among the partners offering tape resources for the project: PIC IN2P3 CERN SARA CNAF (others?)
  • Tentative date: Wednesday, March 18th meeting in the morning TBC
• Xavier prepared a draft action plan for discussion for the next 3-to-6 months that will lead to the pilot 

Action Plan (Xavier)

LOFAR data: 

• Data injection and data distribution (Ron/Aris/SKAO)
• Workloads investigation: identify and adopt simple jobs to consume LOFAR data from the LAKE (SKAO)
• AUS storage integration (Crystal/Guido/Ron/Aris/SKAO)
• SA storage integration (Ron/Aris)
• Authorisation and data accessing: ESCAPE VO as Auth method with X509. LOFAR data should not be confidential, and everyone should have the rights to access them (not a problem for LOFAR data).
• SKA/FAIR access to particular directories while other groups in the ESCAPE VO have no access. For DESY this is already in place.
• DESY storage endpoint already has directories-Auth structure setup in some endpoints and using Tokens.
• Start with X509 as flat scenario to then move to Tokens. This should not be a showstopper, as they can be done in parallel.
• Design and plan the mapping of filenames to Rucio DOD’s.
• Frederic and Gonzalo to follow-up on this for CTA. 
• CERN and DESY take the lead to move the LHC data in the DataLake. These should be open data for both ATLAS and CMS.
• Identification of workflows ("jobs"):
  • CERN takes the lead wrt HL-LHC WFs.
  • Yan takes the lead wrt LOFAR WFs.
  • Rohini takes the lead for SKA WFs. Simulation WF in a month.
  • Frederic takes the lead for CTA WFs.
  • Timeline: end of April
• The RUCIO-DIRAC integration goes beyond the ESCAPE scope. 
  • Marcelo Soares (INFN-CNAF) will contact the right people/working-group to understand if there is an on-going effort for a RUCIO plugin to integrate DIRAC.

Integration with Compute:

• Identify the basic computing needs for different RIs, which kind of resources they need or they can make of: batch systems (which ones?), cloud resources, notebooks, HPC, interactive nodes? other? (Yan)
• Identify workflows that can run on a simple scenario: e.g. processing node, worker node on a batch system (Yan)
• Simple integration with the datalake, e.g. storage endpoint  (Yan/Aris/Riccardo/Rizart)
• WFs mean jobs that can run on resources, even starting from something simple.
• For the time being, there will be no link with WP3 WFs.
• Simone suggests to identify WFs and make sure they exist, in whenever repo they are. WFs that don't have a repo yet can be put in GitLab from WP3.
• Timeline: within one month agreed with Yan.

XCache deployment and testing

• Demonstrate the ability to deploy a simple caching layer and connection to the datalake (Riccardo/Diego/P.Musset/)
• Identify (at least) two workflows to run on (at least) two sites using the simple caching layer (Riccardo/Daniele/Diego)
  • HL-LHC: ATLAS and CMS (Riccardo/Diego/Daniele)
  • SKA/LOFAR (SKAO/others?)
  • CTA (Nadine/Stephane/PIC) 
  • LSST (Ghita/Fabio/PMusset)
• Ghita: XCache will be deployed for LSST (but not only) by a new hired person in ESCAPE.
• The new INFN-CNAF cloud platform can serve different needs.
  • This is an interesting idea for ESCAPE. Riccardo, Diego, and Daniele will follow-up, as well as others interested in this.

Implement QoS endpoints:

• Implement current QoS possibilities at the sites: Erasure Coding, N-replica, Raid with RUCIO (P.Millar/Aris)
• Define a specific RSE per QoS endpoint (some sites with multiple Rucio RSEs but different storage endpoints) (Aris)
• Demonstrate data transition through QoS 
• By policy (pre-set transition date?)
• Interactive (experiment changing the QoS “label”)
   

AOBs

• Paul Millar: WP3 meeting to understand what to present in Brussels
• Andrea: WP2/5 meeting to present AAI and to understand what to present. How to structure the Auth. Another meeting will follow.

 

 

 

 

1 News

• ESCAPE rockchat as tool for a discussion forum: https://chat.escape2020.de/home

Orateur: Simone Campana (CERN)

2 Status and summary of recent discussions. Plan.

Orateur: Xavier Espinal (CERN)

LOFAR data: 

• Data injection and data distribution
• Workloads investigation: identify and adopt simple jobs to consume LOFAR data from the LAKE 
• AUS storage integration
• SA storage integration 
   

Integration with Compute:

• Identify the basic computing needs for different RIs, which kind of resources they need or they can make of: batch systems (which ones?), cloud resources, notebooks, HPC, interactive nodes? other? 
• Identify workflows that can run on a simple scenario: e.g. processing node, worker node on a batch system, 
  • Simple integration with the datalake, e.g. storage endpoint 

XCache deployment and testing

• Demonstrate the ability to deploy of a simple caching layer and connection to the datalake 
• Identify (at least) two workflows to run on (at least) two sites using the simple caching layer 
  • HL-LHC: ATLAS and CMS?
  • SKA/LOFAR?
  • CTA?

Implement QoS endpoints:

• Implement current QoS possibilities at the sites: Erasure Coding, N-replica, Raid with RUCIO.
• Define an specific RSE per QoS endpoint (some sites with multiple Rucio RSEs but different storage endpoints)
• Demonstrate data transition through QoS:
  • By policy (pre-set transition date?)
  • Interactive (experiment changing the QoS “label”)

 

 

 

3 Quick round table of Tasks