ESCAPE QoS fortnightly meeting


Minutes for 2020-09-30 T2.2 meeting


Marek Szuba
Martin Barisits
Marcelo Soares
Gareth Hughes
Paul Millar
Aleem Sarwar
Frederic GiIlardo




Round table

Streams info can be found at:

Stream A: prototyping and demonstration

Plans for demo-day
Injected data will contain metadata, describing (implicitly) which RSE is the primary recipient of this data.  The secondary copy is created by adding a Rucio rule that targets an explicit RSE.

Data ingest:
Secondary RSE: 1xSAFE

Two directories, which together give two RSEs

This would be two RSEs, with the same host name and same endpoints (WebDAV, xrootd, gsiftp, etc…) but with different QoS labels.


Skipped, as nobody from SKA was here.


Marek: both the first FAIR/CBM QoS demo and the first ingestion test had replication rules that targeted QoS.
For the former, all files were initially written to GSI-ROOT. Afterwards, rules were added requesting replicas with the target QoS tag:
for AODs, two replicas; GSI-ROOT did advertise the requested tag
for results and logs, one replica each; GSI-ROOT did not advertise requested tags
As a result, in either case FTS needed to execute a single transfer per file.
During data-ingestion test, files were uploaded to a non-QoS RSE and rules were added requesting two replicas with the target QoS; this failed because of an insufficient number of working RSE advertising the desired QoS class. Single replication worked fine.
QoS use planned for the rehearsal:
On data ingest
Log files → 1x OPPORTUNISTIC
Raw data → 2x SAFE + 1xFAST
Later on
Raw data -> 2x SAFE + 1xCHEAP-ANALYSIS.

QoS labels

Command for replication : rucio add-rule TUK:0065 1 'QOS=A'

Tape storage → SAFE
Single disk copy → OPPORTUNISTIC
Erasure-encoded, RAID-6 → CHEAP-ANALYSIS

Next week: next release for Rucio. v1.23.8

Martin: might be able to update Rucio to allow QoS-based data ingestion, which would avoid the user/client choosing into which RSE the data is written.

  Update existing RSEs to use the four QoS classes: Aleem
  Investigate splitting storage to simulate multiple QoS classes (e.g., SAFE and CHEAP-ANALYSIS), or whether to use IN2P3’s tape storage: Frederic
  Add Rucio attributes that have the site name; e.g., SITE=DESY, SITE=GSI, SITE=LAPP, ... : Aleem.

This allows us to specify Rucio rules that target a specific QoS at a specific site; e.g., QOS=SAFE&SITE=LAPP.

Stream B: engagement with experiments
Stream C: software developments

The next meeting will be on Wednesday 14th October (2020-10-14)

There are minutes attached to this event. Show them.
    • 2:00 PM 2:10 PM
      News 10m
      Speaker: Paul Millar (DESY)
    • 2:10 PM 2:50 PM
      Round Table 40m

      Stream A: Prototyping and demonstration

      Stream B: Engagement with experiments

      Stream C: Software development

    • 2:50 PM 3:00 PM
      AOB 10m