Minutes for 2021-03-31 T2.2 meeting
Paul Millar
Aleem Sarwar
Berkay Turk
Marcelo Soares
Lucia Morganti
Alba Vendrell
Frederic Gillardo
Martin Barisits
Marek Szuba
Round table
Streams info can be found at: https://indico.in2p3.fr/event/19937/contributions/75972/attachments/56927/75787/ESCAPE-T2.2-update.pdf
Amazon Web-service RSE
AP/ Frederic G will review the information on the wiki page
AP/ Frederic to review the CTA QoS document to see if the Amazon storage fits in the model.
AP/ Paul to contact Xavi and Rosie to check how work by Frederic on cloud resources past the M30 deliverable may be recognised.
Planning for next development phase: DAC21: Data and Analysis Challenge (November 2021) .
For injection: generating random data and injecting them into
Need to have real(-ish) data.
There is a CTA pipe -- a reconstruction pipe from RAW to high-level data
The pipeline is not yet packaged
Need to be able to retrieve data based on metadata, which is now supported by Rucio.
High-level data will be “packages” as Virtual-Observatory (VO). These will be analysed outside of the data lake. The data is really small -- is it worth putting it into the data lake?
The problem is with metadata. High-level data has complex metadata, which needs to be queried to discover the desired data (ADQL -- Astronomical Data Query Language) -- it’s unclear whether Rucio can support this.
Canary Island RSE makes sense & It would be nice to have more detail about the new RSE.
Processing data as it arrives; reprocessing data in a year’s time. Caching doesn’t really make sense.
AP/ Paul & Aleem to query other ESFRI people with details on how they wish to proceed for DAC21.
AP/ Aleem to organise a meeting with Pandey, Yan Aleem and Paul to discuss LOFAR use-case.
AP/ Paul to ping Mario about ATLAS updates.
We should now have the ability to add custom metadata to endpoints.
Rizart can grant people permission to modify values in CRIC.
AP/ Aleem will ask for access and try to inject the metadata from our wiki page.
Rizart is working on sync-ing CRIC custom metadata into Rucio as attributes.
We don’t believe this will cause any problems.
Right now, Alba has been working on the monitoring task. For QoS monitoring, there is a new panel that shows the QoS for each storage element. This needs to be updated to support filtering.
She anticipates finishing the QoS monitoring by the end of next week.
AP/ Alba to present progress at the next meeting.
AOB
The next meeting will be on 2021-04-14 at usual time 14:00- 15:00 CEST.
