WP2 fortnightly meeting

mercredi 30 juin 2021, 11:00 → 12:15 Europe/Paris

Xavier Espinal (CERN)

Live Minutes

ZOOM video connection

1 News

Orateurs: Rosie Bolton (Square Kilometre Array Organisation), Xavier Espinal (CERN)

• ESCAPE General Assembly extended to two days: 28 and 29 September (possibly a hybrid at CERN, covid measures allowing it) 
  • 28th September, full day for technical discussions (Ian, TC). Deeper technical understanding project progress and RI’s needs.
  • 29th morning will be dedicated to WP presentations (prototypes, technical progress with demonstrations, etc), TSPs and in the afternoon the GA presentation.
• FENIX project (https://fenix-ri.eu/) progress: agreed to start addressing common AAI by using ESCAPE/IAM as Identity provider on the FENIX AAI structure, prototype/testing data delivery using FTS3 with swift endpoints (Ian, Andrea and Rizart present at the last meeting.) 
• DAC21 Activities moving forward at good pace, preparing the scenario to start strong after the summer break. 
• WP2 meetings through the summer:
  • 14th July - Confirmed
  • 28th July - Propose to keep but need a chair
  • 11th August - Propose to Cancel
  • 25th August - Confirmed
• WP5 workshop: https://doodle.com/poll/74n7vucisds5ein3  (doodle closes 1 July)
• HDF5 meeting proposed by PANOSC (HDF5 is a file format widely used in PaNOSC and also in some of our communities): https://www.hdfgroup.org/hug/europeanhug21/ (July 7-8)
  • Abstract submitted by Paul Millar
• EGI conference 19-21st October: https://www.egi.eu/egi-conference/2021-beyond-the-horizon/call-for-abstracts/ (abstract submission deadline: tomorrow 1st of July)

 

 

2 Advancements towards the token-based authentication

Orateur: Rizart Dona (CERN)

Current Status

Storages

• CNAF-STORM accepts ESCAPE IAM issued tokens → Users encouraged to try it

• The rest  → Unknown

• Storage Providers are kindly asked to update https://wiki.escape2020.de/index.php/WP2_-_DIOS#Datalake_Status in order to reflect the actual status of the storage → First test, and if it works, indicate “Token Based Auth/Z” as “YES”

FTS3 - Pilot Instance [https://fts3-pilot.cern.ch:8446]

• Accepts ESCAPE IAM issued tokens → Users encouraged to try it

Rucio Instance

• Accepts ESCAPE IAM issued tokens

• Some technicalities still to be resolved (WiP) → Nevertheless users encouraged to try it

Testing Effort

• We need to start testing the individual components

• Storage providers can start by configuring their storage and then test with a simple GFAL2 command that they can access the storage

• We can integrate the token logic in the current testing infrastructure

  • https://github.com/ESCAPE-WP2/Utilities-and-Operations-Scripts/tree/master/gfal-sam-testing

  • https://github.com/ESCAPE-WP2/fts-analysis-datalake

  • https://github.com/ESCAPE-WP2/rucio-analysis

• Dev work is needed, contributors are encouraged to start implementing the missing parts, or create new tools to test the different layers.

User Documentation

GFAL2: https://dmc-docs.web.cern.ch/dmc-docs/developers/bearer-tokens.html

FTS3: https://fts3-docs.web.cern.ch/fts3-docs/fts-rest/docs/cli/index.html#ftsresttransfersubmit

Rucio: http://rucio.cern.ch/documentation/using-rucio-client/#open-id-connect-authentication-examples

3 Activity-One progress update: Data Injection, Replication and Processing

Orateurs: Riccardo Di Maria (CERN), Yan Grange (ASTRON, the Netherlands Institute for Radio Astronomy), frederic gillardo

ActivityOne had 3 gatherings. Minutes at: https://docs.google.com/document/d/1DFtmVtBHxPISdYKnk0iRZ-wdXGG7hEi_iuLAUDdIb1s/edit?usp=sharing .

QoS discussion and reports are being hosted during the last part of the meeting.

Agnostic scenario: only SKA showed a strong interest wrt this topic (which, however, is not excluded by KM3Net and EGO-Virgo a-priori). SKA is expected to follow up on their progress/activity/tests on the topic.

Deterministic vs not-deterministic RSEs:  almost all experiments are interested to consider and assess both scenarios.

• Description from Martin:  https://wiki.escape2020.de/index.php/WP2_-_DIOS#3.3.1_Deterministic_vs_non-deterministic_RSE
• A non-deterministic RSE will use dataset information and build the path. This has the advantage that tape systems can collocate the files of the same dataset on the same tape.

Kick-starting DL-as-a-S: JupyterLab-Rucio integration fully-flagged service. Started at CERN and proceeding to be integrated into the cluster hosting ESCAPE Rucio instance. 

 

M1 and M2 achieved with the exception of:

• Benchmarking data/dataset size and volume effect on namespace, tape-disk moving behaviour, etc.

M3 (July 31, 2021):

• Standardise DL-as-a-S in WP2 (evaluating integration in WP5), deterministic vs. non-deterministic RSEs, and agnostic scenario
• Start supporting data life-cycle aka QoS (cross-task activity with T2.2) ETA would not depend on T2.1
• External resources (cloud)

4 Short updates from DAC21 Activity Groups

Orateurs: M. Agustin Bruzzese (PIC), Andrea Ceccanti (INFN), Marek Szuba (GSI), Paul Millar (DESY), Dr Riccardo Di Maria (CERN), Rizart Dona (CERN), Rosie Bolton (Square Kilometre Array Organisation), Yan Grange (ASTRON, the Netherlands Institute for Radio Astronomy), frederic gillardo

5 Depops hot topics

Orateur: Depops weekly chair

6 AOB

Astronomy presence at Rucio community workshop (28 Sep (Tuesday) to 30 Sep (Thursday) + 01 Oct (Friday) optionally too). Encourage good ESCAPE WP2 presence in general, but in particular please get in touch with Rosie if you have an astronomy Rucio use case  / experience to present - we are planning an astronomy-specific session in the Workshop