ESCAPE QoS fortnightly meeting
Minutes for 2020-09-30 T2.2 meeting
Present
Marek Szuba
Martin Barisits
Marcelo Soares
Gareth Hughes
Paul Millar
Aleem Sarwar
Frederic GiIlardo
Apologies
Agenda
News
Round table
Streams info can be found at: https://indico.in2p3.fr/event/19937/contributions/75972/attachments/56927/75787/ESCAPE-T2.2-update.pdf
Stream A: prototyping and demonstration
Plans for demo-day
CTA
Injected data will contain metadata, describing (implicitly) which RSE is the primary recipient of this data. The secondary copy is created by adding a Rucio rule that targets an explicit RSE.
Data ingest:
Primary RSE: 1xCHEAP-ANALYSIS + 1xSAFE
Secondary RSE: 1xSAFE
Two directories, which together give two RSEs
/data/escape-qos-safe
/data/escape-qos-cheap-analysis
This would be two RSEs, with the same host name and same endpoints (WebDAV, xrootd, gsiftp, etc…) but with different QoS labels.
SKA
Skipped, as nobody from SKA was here.
FAIR
Marek: both the first FAIR/CBM QoS demo and the first ingestion test had replication rules that targeted QoS.
For the former, all files were initially written to GSI-ROOT. Afterwards, rules were added requesting replicas with the target QoS tag:
for AODs, two replicas; GSI-ROOT did advertise the requested tag
for results and logs, one replica each; GSI-ROOT did not advertise requested tags
As a result, in either case FTS needed to execute a single transfer per file.
During data-ingestion test, files were uploaded to a non-QoS RSE and rules were added requesting two replicas with the target QoS; this failed because of an insufficient number of working RSE advertising the desired QoS class. Single replication worked fine.
QoS use planned for the rehearsal:
On data ingest
Log files → 1x OPPORTUNISTIC
Raw data → 2x SAFE + 1xFAST
Later on
Raw data -> 2x SAFE + 1xCHEAP-ANALYSIS.
QoS labels
Command for replication : rucio add-rule TUK:0065 1 'QOS=A'
SAFE, OPPORTUNISTIC, CHEAP-ANALYSIS, FAST
Tape storage → SAFE
Single disk copy → OPPORTUNISTIC
Erasure-encoded, RAID-6 → CHEAP-ANALYSIS
SSD → FAST
Next week: next release for Rucio. v1.23.8
Martin: might be able to update Rucio to allow QoS-based data ingestion, which would avoid the user/client choosing into which RSE the data is written.
TODO:
Update existing RSEs to use the four QoS classes: Aleem
Investigate splitting storage to simulate multiple QoS classes (e.g., SAFE and CHEAP-ANALYSIS), or whether to use IN2P3’s tape storage: Frederic
Add Rucio attributes that have the site name; e.g., SITE=DESY, SITE=GSI, SITE=LAPP, ... : Aleem.
This allows us to specify Rucio rules that target a specific QoS at a specific site; e.g., QOS=SAFE&SITE=LAPP.
Stream B: engagement with experiments
Stream C: software developments
AOB
The next meeting will be on Wednesday 14th October (2020-10-14)