ESCAPE QoS fortnightly meeting

Europe/Paris

Minutes for 2020-09-30 T2.2 meeting

Present

Marek Szuba
Martin Barisits
Marcelo Soares
Gareth Hughes
Paul Millar
Aleem Sarwar
Frederic GiIlardo

Apologies

Agenda


News


Round table


Streams info can be found at: https://indico.in2p3.fr/event/19937/contributions/75972/attachments/56927/75787/ESCAPE-T2.2-update.pdf

Stream A: prototyping and demonstration


Plans for demo-day
CTA
Injected data will contain metadata, describing (implicitly) which RSE is the primary recipient of this data.  The secondary copy is created by adding a Rucio rule that targets an explicit RSE.

Data ingest:
Primary RSE: 1xCHEAP-ANALYSIS + 1xSAFE
Secondary RSE: 1xSAFE

Two directories, which together give two RSEs
    /data/escape-qos-safe
    /data/escape-qos-cheap-analysis

This would be two RSEs, with the same host name and same endpoints (WebDAV, xrootd, gsiftp, etc…) but with different QoS labels.


SKA

Skipped, as nobody from SKA was here.


FAIR


Marek: both the first FAIR/CBM QoS demo and the first ingestion test had replication rules that targeted QoS.
For the former, all files were initially written to GSI-ROOT. Afterwards, rules were added requesting replicas with the target QoS tag:
for AODs, two replicas; GSI-ROOT did advertise the requested tag
for results and logs, one replica each; GSI-ROOT did not advertise requested tags
As a result, in either case FTS needed to execute a single transfer per file.
During data-ingestion test, files were uploaded to a non-QoS RSE and rules were added requesting two replicas with the target QoS; this failed because of an insufficient number of working RSE advertising the desired QoS class. Single replication worked fine.
QoS use planned for the rehearsal:
On data ingest
Log files → 1x OPPORTUNISTIC
Raw data → 2x SAFE + 1xFAST
Later on
Raw data -> 2x SAFE + 1xCHEAP-ANALYSIS.

QoS labels


Command for replication : rucio add-rule TUK:0065 1 'QOS=A'
SAFE, OPPORTUNISTIC, CHEAP-ANALYSIS, FAST

Tape storage → SAFE
Single disk copy → OPPORTUNISTIC
Erasure-encoded, RAID-6 → CHEAP-ANALYSIS
SSD → FAST

Next week: next release for Rucio. v1.23.8

Martin: might be able to update Rucio to allow QoS-based data ingestion, which would avoid the user/client choosing into which RSE the data is written.

TODO:
  Update existing RSEs to use the four QoS classes: Aleem
  Investigate splitting storage to simulate multiple QoS classes (e.g., SAFE and CHEAP-ANALYSIS), or whether to use IN2P3’s tape storage: Frederic
  Add Rucio attributes that have the site name; e.g., SITE=DESY, SITE=GSI, SITE=LAPP, ... : Aleem.

This allows us to specify Rucio rules that target a specific QoS at a specific site; e.g., QOS=SAFE&SITE=LAPP.


Stream B: engagement with experiments
Stream C: software developments

AOB
The next meeting will be on Wednesday 14th October (2020-10-14)

 
There are minutes attached to this event. Show them.
    • 2:00 PM 2:10 PM
      News 10m
      Speaker: Paul Millar (DESY)
    • 2:10 PM 2:50 PM
      Round Table 40m

      Stream A: Prototyping and demonstration

      Stream B: Engagement with experiments

      Stream C: Software development

    • 2:50 PM 3:00 PM
      AOB 10m