Streams info can be found at: https://indico.in2p3.fr/event/19937/contributions/75972/attachments/56927/75787/ESCAPE-T2.2-update.pdf
Plans for demo-day
Injected data will contain metadata, describing (implicitly) which RSE is the primary recipient of this data. The secondary copy is created by adding a Rucio rule that targets an explicit RSE.
Primary RSE: 1xCHEAP-ANALYSIS + 1xSAFE
Secondary RSE: 1xSAFE
Two directories, which together give two RSEs
This would be two RSEs, with the same host name and same endpoints (WebDAV, xrootd, gsiftp, etc…) but with different QoS labels.
Skipped, as nobody from SKA was here.
Marek: both the first FAIR/CBM QoS demo and the first ingestion test had replication rules that targeted QoS.
For the former, all files were initially written to GSI-ROOT. Afterwards, rules were added requesting replicas with the target QoS tag:
for AODs, two replicas; GSI-ROOT did advertise the requested tag
for results and logs, one replica each; GSI-ROOT did not advertise requested tags
As a result, in either case FTS needed to execute a single transfer per file.
During data-ingestion test, files were uploaded to a non-QoS RSE and rules were added requesting two replicas with the target QoS; this failed because of an insufficient number of working RSE advertising the desired QoS class. Single replication worked fine.
QoS use planned for the rehearsal:
On data ingest
Log files → 1x OPPORTUNISTIC
Raw data → 2x SAFE + 1xFAST
Raw data -> 2x SAFE + 1xCHEAP-ANALYSIS.
Command for replication : rucio add-rule TUK:0065 1 'QOS=A'
SAFE, OPPORTUNISTIC, CHEAP-ANALYSIS, FAST
Tape storage → SAFE
Single disk copy → OPPORTUNISTIC
Erasure-encoded, RAID-6 → CHEAP-ANALYSIS
SSD → FAST
Next week: next release for Rucio. v1.23.8
Martin: might be able to update Rucio to allow QoS-based data ingestion, which would avoid the user/client choosing into which RSE the data is written.
Update existing RSEs to use the four QoS classes: Aleem
Investigate splitting storage to simulate multiple QoS classes (e.g., SAFE and CHEAP-ANALYSIS), or whether to use IN2P3’s tape storage: Frederic
Add Rucio attributes that have the site name; e.g., SITE=DESY, SITE=GSI, SITE=LAPP, ... : Aleem.
This allows us to specify Rucio rules that target a specific QoS at a specific site; e.g., QOS=SAFE&SITE=LAPP.
The next meeting will be on Wednesday 14th October (2020-10-14)