2.1 Datalake (Xavi)
- LOFAR data injection and distribution
- HTTP-TPC enabled on EOS instance for the Datalake.
- Start testing token based access to the Datalake from CERN HTCondor
- Activity on caching infrastructures progressing.
- Prototyping a Datalake dashboard, goal is to give an overall perspective of the datalake in action: in-flight transfers, files, sites, perfsonar, etc. and the transfer matrix for all sites.
- TPC in EOS
- Elvin wrote some documentation on XRootD and HTTP TPC
- For now only in EOSPPS, on the production instances normaly before June
2.2 QoS and orchestration (Paul)
2.3 Integration with compute (Yan)
- Computing: Have a list of questions, checked with CMS, LOFAR and got input from SURF. Next step is to share with all the ESFRI partners and ask for what their use cases would require
- LOFAR data and code: LOFAR use case has been prepared. It does ship with minimal data to make sure the readme and compilation are OK. Trying to get access to the data lake data. Also want to see if we can put our use case data in there so that we can process it. The data we want to put in is public data.
- Configuration of LOFAR test:
- Should work easily.
- Few parameters to change to adapt to a site
- Dependancy only on Singularity and Go
- Xavi requested a placeholder in the wiki for computing need of the different use case/experiment
2.4 Network (Rosie)
Meeting held 09/03/2020
Still waiting on the PerfSONAR machines at GSI to be set up (ACTION: Rizart to check up on this, Paul please to help as needed)
Rizart has been in touch with Alex Dodson at AARENT: Perth PerfSONAR box should be ready by now (due last week) but no confirmation received. (ACTION: Rizart to chase up on this)
Rizart explained how to get lightweight CERN accounts to that we can get access to the Grafana instance for the development of monitoring dashboards - this is work in progress but Rizart will create a "how to" guide so new members can get accounts and contribute to the dashboard development and ultimately, use.
Dashboards will need to be developed and ready to use by the summer, but we anticipate being early in this.
Ron, Raymond and Yan had done some data transfer tests with SARAO (Simon Ratcliffe) - some trouble with Ilifu cloud (storage was down) - we agreed an action for SKA to take responsibility for the interaction with South African colleagues to get an RSE in ESCAPE, but to include Yan in email traffic.
EOS storage at AARNET (Perth and Melbourne) has successfully been used within the SKA RAL rucio instance to transfer data from Australia to UK. Most work done by Crystal Chua, Rohini supporting with testing. Now that this works it should be much easier to also include storage in the ESCAPE rucio too. Authentication was a major challenge.
New SKA team member, Jimmy Cullen will start at SKA March 16th. We will try to schedule an intensive few days so Rizart can get Jimmy up to speed. However, travel restrictions may mean that this has to be done via videocon in the first instance, with a larger team meeting happening later once travel restrictions ease.
Rosie will doodle for a regular meeting slot once Jimmy is onboard.
2.5 AAI (Andrea)
I worked closely with Riccardo di Maria, Diego Ciangottini and Daniele Spiga
on fine-grained authorization in support of data caching and access with
XRootD. This both for GSI/VOMS-based authZ and token-based authz. XRootD VOMS
support seems fairly limited for group-based authorization, limitations were
discussed and reported by Riccardo and Diego to XRootD developers, which have
suggested an alternative version of the VOMS plugin which fulfills the
requirements for VOMS group and role-based authz on a multi-tenant XRooTD
instance.
On the token-based authz side, we exercised the multi-tenant scope-based authz
scenario using the WLCG profile against an XRootD instance and found problems
also there; Brian Bockelman has been given access to the VM to troubleshoot the
problem (details not disclosed here, ask me in chat) in the Scitokens library
code. Brian confirmed the problem and proposed a patch, which Diego tested
and fixes the problem.
So now we know how to enforce tenant separation on XRooTD with both VOMS/GSI
and token-based authz. This is interesting as it enables support for protecting
embargoed data for CMS (Diego is already doing tests on real data) and other
interested ESCAPE experiments.
As announced at the progress meeting, I will organize a webinar on ESCAPE AAI to
support integration activities in other technical WPs.
I will circulate a doodle by the end of the week to propose a date. It would be
good to record the webinar so it can be used as a reference in the future.
