To focus on technical solutions for integrating Rosetta and ESAP. The idea is that this focused technical group will make a proposal that we can then discuss with a range of stakeholders.
DISCUSSION:
Sara: Shares document
https://docs.google.com/presentation/d/1oMXnps45I-gI61imR7WC7AvH_7D38PgZM4fux523eVY/edit?usp=sharing
and introduces:
Declared scope:
focus on technical solutions for integrating Rosetta and ESAP
make a proposal to be discussed with a range of stakeholders
In general:
identify technical solutions to access computing facilities to run batch processing software
identify what we mean with “batch processing software”
Batch processing software “Identification”:
It runs unattended, does not require human intervention
it automatizes some computing operations
it starts from a specific configuration user’s provided
A concrete use case could help
Gareth: CTA use cases are mainly two:
1_ the software is already deployed in the computing facility,the user has the need to run it
2_ creating or identifying a container and sending that to the compute. That container should exist in some place like a repository, potentially in zenodo
The next use case is when data has to be taken from the data lake and then analyzed using the batch compute.
The use cases are based on a little bit of knowledge about how CTA collects data. Gamma ray telescopes point to a certain place in the sky for about 30 minutes, they generate gigabytes of data in that time and then somebody hits the end observation button. The produced file is transferred and then the telescope points somewhere else. If you analyze a source in the sky, you end up with a bunch of gigabytes sized files which will take over one, two, sometimes over 10 years. If you wanna do an analysis of this source, you need to bring data files together and analyze them. Batch in this case because the analysis flow is very poor and you need to apply the same algorithms hundred of times on many pools.
Sara: probably DIRAC is the better solution for the first use case (software already deployed in the computation infrastructure) and Rosetta is the best one when software is in a repository and it can be "enclosed" in a container and then deployed on an infrastructure.
Proposal: to discuss the computation access model keeping on a side, for the moment, authentication and authorization and data access to proceed one step at a time.
Gareth: two comments:
1. we received an e-mail from Misha and a meeting on A&A is set up
2. for accessing the data, we should keep into account the work done by CTA members in WP2 where they are trying to get DIRAC and Rucio to play together. That has been done before by the Belle II experiment, a plug-in has been written for this purpose. The development is in the standard DIRAC installation and at non-ESAP level, but when ready, it should be easier to investigate how to integrate at the platform level.
Nico: during the last meeting we identfied the paths to start software: one from the front-end, from the GUI, and goes through the URL and one is through the back-end with the job mechanism: a job should be packaged and at some point the ESAP REST API (to be written) should be invoked to call, for example, DIRAC to start the job. What is needed is to see if there are python files already doing this task and what they need from the ESAP back-end to be integrated/completed.
The code to be written is a Rosetta or DIRAC wrapper, a service connector.
Gareth: in the current DIRAC integration it is an ESAP_Concordia.py file. Could it be what it is needed?
Nico: there is not the job mechanism that receives information from the front-end APIs and connects to the service through the service connector and kick-off the job. The connections we have to make are partially defined by the metadata presented by Dave where it is a type describing the kind of process ESAP should do and that's related to the ESAP connectors we have to build in the back-end.
Klaas: containers for Rosetta for example
Stefano: Rosetta can be made to behave like BinderHub and run containers and MyBinder compatible repositories.
It is possible authentication in MyBynder?
Klaas: Jive has done something. You can log-in through Indigo-IAM and then connect to their BinderHub installation. Then probably you can go through ESAP.
Stefano: you can not share authentication between two different domains because web standards, so basically you have to re-login.
Klaas: because you are just redirected it is just a single click. DIRAC is different because it requires X509 certificates.
Stefano: in the current ESAP implementation it is nothing sending a token, it is just an URL redirect.
Klaas: Jive has a sort of configuration.
Stefano: in the web now two different domains can not share authentication. Indeed, when you use a google account to log-in in a different service, you should essentially re-login. The only way is to send a token from the back-end to MyBinder.
Nico: we have a rough solution where you create a token using IAM, propagate it to the front-end, and you copy&paste it wherever you are.
Klaas: we have a mechanism where you select your data into the shopping basket from ESAP, you have a JuphyterHub instance working somewhere else, you log into that, they are sharing the token scope. What they do is to generate the token for JupytherHub in the back-end and those can also be used for connecting to ESAP, for example. There is a configuration made by CERN, which makes available tokens inside Jupyther notebook and then you can get access from the notebook to the ESAP shopping basket to get the links to the data..
Stefano: asks clarification about the code of the interactive data analysis view in the back-end and the code in the model.
Essentials of the discussion: the authentication for the moment is left on a side to have a prototype where everyone can connect to a computational resource, specifically BinderHub, profiting to the fact that BinderHub provides a platform accessible without authentication, and then run his JupytherLab.
The code for workflows and facilities in the model is just a starting point that has to be specialized, possibly keeping into account the metadata models that Dave is working on.
Stefano: DIRAC requires back-end to back-end communication, Rosetta does not. If we are happy that the user has to be re-authenticated to Rosetta, a front-end integration can be done as a starting point. In Rosetta you can not access computation resources without authentication.
Discussion about the opportunity to enforce the authentication everywhere in ESAP.
Stefano: proposes as a first step, to make Rosetta behave as MyBinder, except that the user has to re-authenticate on Rosetta, then choosing the computing resource and hit play. This can be done in a relatively short time. For a back-end to back-end integration, we need to better define what is a computing facility. This is related to DIRAC and also to Rosetta that at the moment has not API, they have to be written. It has to be clarified also how to configure the client APIs in a multi-user environment.
Probably for the back-end APIs we need only simple functions like "submit a job", "check status", "get results", "done".
Gareth: share
https://docs.google.com/document/d/1_tuQKS-RnH_hXkEw2jVXPKjGf3PuhPKABA-mN1HrrtE/edit
description of a set of potential APIs for basic functionalities with basic example scripts.
Stefano: this use of the DIRAC client supposes to have a pre-configured environment.
Gareth: a proxy-init is needed. It has been organized the "CA batch data processing through ESAP" meeting to discuss a possible solution. There are cases of web applications running DORAC: a token is taken from the site which identify the user and then it is a general global x509 certificate which runs on the back-end and everyone uses. The specified user is identified through the token. Someone is working on that on DIRAC.
Stefano: for Rosetta, when you need access computing resources over ssh, you need two files on the host. They are the public and private keys generated by Rosetta for each user. Then the user is required to add the public key generated by Rosetta on the computing resource where he wants access. Maybe something similar can be done in DIRAC with the user certificates.
Another solution could be to spin up a container for the specific user containing the certificates and running the client.
Both Rosetta and DIRAC need some configuration parameters of the computing resource: keys (ssh or x509), the architecture of the computing resource (ARM or Intel or...), the container runtime available, the storage,
Has DIRAC different entry points? different installations? The client needs to know which server/host query.
...
All these parameters have to be included in the facility model.
Gareth: the core of DIRAC sets the configuration. Probably part of the configuration can be done installing the ESAP toolkit.
Stefano: there are a lot of black magic not visible at the moment and we need to make them explicit.
Stefano&Gareth: discuss about REST APIs, not available both in Rosetta and DIRAC.
Stefano&Nico: do we have configuration files? Or configuration in the database? Stefano uses configuration in the database, Nico uses configuration files for sensitive stuff and database for non-sensitive configuration on the server side.
Gareth: DIRAC is huge and it is difficult to have control out of the development team. To modify, at some point, you have to branch. Not convenient.
Stefano&Gareth: there will be only one DIRAC instance in ESAP.
Stefano: the current status of the facilities description in the code is far from what we need. Will mock some APIs for Rosetta in the document. For the moment there is nothing in Rosetta or in DIRAC that allows to run something back-end to back-end. There is not a mechanism to transfer "software" at the computation resource both in DIRAC and Rosetta. In Rosetta the user gives the container name and repository name and Rosetta keeps the "software" with a pull mechanism.
Nico: there is a blob containing metadata
Stefano: to put Rosetta and DIRAC on the same hat, we need to construct a common object to satisfy both.
Gareth: to normalize DIRAC and Rosetta we have to define a minimal set of functions to implement the workflow. A look to the use cases could help in this task.
Stefano: shares
https://git.astron.nl/astron-sdc/escape-wp5/workflows/metadata-examples/-/tree/test/timeseria/Timeseria
It is a try to make a concrete use case.
Agree on writing minimal common APIs.
Sara: First step is Rosetta fron-tend integration and continue working on back-end-to-back-end integration of both DIRAC and Rosetta.
