Task-1 Datalake

• ESCAPE datalake pilot is getting through the consolidation phase. Datalake activities are progressing well in general terms.
• Storage and orchestration layer stable, datalake is currently populated with some real data from several experiments: LOFAR, LSST, ATLAS and CMS data (moderate volumes). Workflows and pipelines implementation to test data access is progressing and started exercising data access.
• Monitoring and dashboards being set-up: Transfer quality matrix, Storage QoS tests, Network performance monitor (PerfSonar), Job benchmarking, etc.
  • Live "view" of a datalake activity: transfers in-flight, data volume, throughputs, etc.
    • https://monit-grafana.cern.ch/d/000000420/fts-transfers-30-days?orgId=51 (accessiblity details: link)
  • Easy green/yellow/red spotting grid to spot issues on data replication. Main source of information for the operations and deployment team.
• Substantial contribution to the RUCIO development team and into the XCache (XrootD) core team in SLAC: QoS, token integration, RUCIO API, XCache authentication, etc.
• Consolidating synergies between ESCAPE and WLCG e.g. first implementation for storage QoS endpoints in the ESCAPE datalake
• Content delivery and caching taking-off outside the CERN testbed. Coordinated initiative for content delivery and caching proposed and started two weeks ago: https://indico.in2p3.fr/event/21381/
• EC projects
  • Initiative to understand possible synergies with the ARCHIVER EC-funded project to address the  ISO16363 self assessment (trustworthy digital repositories). 
  • Involvement of the ESCAPE datalake in the MECHANICS project (ICT-40-2020 call)
• Software accessibility/distribution start to be discussed as we are trying to understand how to run benchmark workflows for different sciences (coll. with OSSR). Main aim from WP2 perspective is to have "standard candles" to assess datalake performance on a regular basis (hammercloud machinery)
• We are investing some effort to have some of the services containerized for future easy deployment on e.g. K8s (XCache, RUCIO). Having this centralised would be beneficial (coll. with OSSR)
• Initiatives for getting onboard Australia and South-Africa sites slowed down a lot due to covid-19. Still I think this should be revived after summer