WP5 ESAP Focus Months

Europe/Paris
ZOOM

ZOOM

Zheng Meyer-Zhao (ASTRON)
Description
The goal of this event is to engage all WP5 developers to integrate the ESAP services they are working on to make the ESAP prototype ready in September. The services are, for example,

* Integration of RUCIO with ESAP Application Programming Interface (API) Gateway using RUCIO REST API to demonstrate data staging
* Searching for VO-enabled data through ESAP User Interface (UI)
* Integration with interactive data analysis tools, such as start Jupyter notebook in a cloud environment
* Integration with HPC and HTC infrastructures
* Integration with DIRAC workload management system

After the Focus Months, a first release of the ESAP minimum prototype will be available to try out to integrate other services into ESAP.
 
We will form a working group for each of the above mentioned services. Partners are free to participate in one or more working groups.
 
Current working group and participating institutes:
 
Working group Participating Institutes
Rucio CERN, ASTRON, RuG, SKAO, EGO, OU
Data discovery UI and IVOA ASTRON, UEDIN, CSIC, Paris Observatory, UCM, CTA, OU 
Interactive analysis (Jupyter Notebooks on clusters) UEDIN, SKAO, CSIC, INAF, RuG, Paris Observatory, FAIR, CTA, ASTRON, OU
Batch processing (HPC/HTC)  INAF, EGO, RuG, CTA, ASTRON
DIRAC CTA, UCM, FAU, ASTRON

AAI aspects will be covered by most of the working groups, therefore, we are not having a separate working group for this.

Please email Zheng Meyer-Zhao (meyer@astron.nl) which working group you would like to participate in.

Repositories:

ESAP Backend: https://git.astron.nl/astron-sdc/esap-api-gateway

ESAP Backend Technology: Django & Django REST Framework

ESAP Frontend: https://git.astron.nl/astron-sdc/esap-gui

ESAP Frontend Technology: ReactJS

ESCAPE WP5 Detailed Project Plan: https://projectescape.eu/sites/default/files/WP5%20D5.2.pdf

  • Communication tool
  • Working group lead
    • Rucio working group: CERN
    • Data discovery UI and IVOA working group: ASTRON
    • Interactive analysis working group: UEDIN
    • Batch processing working group: INAF
    • DIRAC working group: CTA
  • Regular meeting planning per working group (cannot overlap with each other)
    • Weekly 30 min. meeting (time to be decided by the working group)
  • Code review & commit
    • Each working group works on a separate branch of esap-api-gateway
    • In principle, each working group only works on topic-specific django application, e.g. query or rucio
    • Code needs to be reviewed within the working group before submmitting merge request to the master branch
Documentation
 
IDA working group:
 
ESAP UI & IVOA working group:
Registration
Participants
Participants
  • Catherine Boisson
  • Daniel Nieto
  • Dave Morris
  • Eoin Clerkin
  • Gareth Hughes
  • Hendrik Heinl
  • Hugh Dickinson
  • James Collinson
  • Jose Luis Contreras González
  • Jutta Schnabel
  • Kilian Schwarz
  • Maisam M. Dadkan
  • Mathieu Servillat
  • Matthias Fuessling
  • Matthieu Baumann
  • Nico Vermaas
  • Raffaele Grosso
  • Rohini Joshi
  • Sara Bertocco
  • Stefano Alberto Russo
  • Stelios Voutsinas
  • Susana Sánchez Expósito
  • Thomas Boch
  • Yan Grange
  • Zheng Meyer-Zhao
    • 13:30 14:30
      Code base introduction 1h
      • Structure
      • Query
      • Accounts
      • Rucio
      • ....
    • 14:30 14:40
      Break 10m
    • 14:40 15:40
      Use cases 1h
      • IVOA registry data discovery interface
      • ESFRI use cases
    • 15:40 15:50
      Break 10m
    • 15:50 16:30
      Discussion 40m
      • communication tool
      • identify working group lead
      • Meeting planning per working group
      • code review and commit
    • 14:00 14:30
      ESAP UI & IVOA 30m

      Weekly update of ESAP UI & IVOA working group
      https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

    • 14:00 14:30
      ESAP UI & IVOA 30m

      Weekly update of ESAP UI & IVOA working group
      https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

      NV:

      • Together with Mattia Mancini tested staging app to use Celery and RabbitMQ.
      • Made diagram of the use case provided by Stellios. Assumption is that it is all-in Python, this does not fit too much with reality. NV drew diagram of what he wrote, but with a different implementation.

       

       

      HD:

      • Prototype interface between Zooniverse and ESAP. Gave a short demo. Pretty much everything was done. Made updates to both the backend and frontend. Merge requests submitted. Waiting for NV or ZM.
      • Next steps is to do the AAI in a more integrated way. Emailed with Andrea. Also talk with NV and Zheng on how stable the front- and backend are.

      MF:

      • Trying to catch up, reading through the minutes of the last meetings.

      SS:

      • Feedback to the Google document (STV use case). Mostly checking the usefulnes of the use case from the user perspective.
        • Needs to read more carefully.
      • Think of some protocol and included a scenario based on this protocol. After writing the scenario, this has been part of the discussion already. Is SAMP used as a protocol that is relevant for the ESAP. Could be a useful option. SAMP can be useful to prevent us to reinvent the wheel.
      • Will write a scenario with information of how this could be implemented. NV could then have. GH mentions that there may need to be an application running locally that consumes the SAMP requests. STV: TOPCAT and Aladin come with their own SAMP hub. Any web page can interact with them. Https causes issues.
      • Other CSIC institute has time series data and would like to see if we can integrate timing series cataolgues as well. Will have a look at that.

      GH:
       * No update for now.

       

      STV:

      • Read through the document and the notes
      • Will have feedback on drawing Nico this week. Feedback was given during a discussion at the end of the meeting. NV took notes of the comments of STV
    • 14:00 14:30
      ESAP UI & IVOA 30m

      Weely update of ESAP UI & IVOA working group
      https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

      NV:

      • Sellios use case discussed with Stellios. Based on that discussion new diagram which is in the document.
      • Almost at the stage that ZM and NV can discuss. Backend code of HD has been merged. Frontend code has a clash with current version.
      • ZM and NV to sit together to have a look at current status and discuss how to implemet.

      GH:

      • No development

      HD:

      • Last week discussed how it would be possible to query the data resource from the backend and then compute a schema for the frontend based on what HD learned about the backend. Got it working for Zooniverse. When backend starts up it queries the database.
      • Implemented more advanced frontend. Also implemented how the data is rendered appropriately.
      • Building from the master. Needs to discuss with ZM who changed the way the context is taken into account because this breaks the current implementation.
      • Has not heared anything from Andrea Ceccanti yet.

      SS:

      • SAMP: Useful protocol to get data queries to ESAP. It is possible and SS sent around a repo that contains a demo of this. http://astrojs.github.io/sampjs/
      • Asteroseismology group. They have catalogues that could be integrated in ESAP. Has given the contact person (Jose Ramon) details on the meeting. The implementation could be somewhat similar to the Zooniverse script.
      • Alternative to using MySQL directly a protocol to access the data could be an option.
      • Will send a video of SAMP around
      • Will be on holiday shortly

      MF:

      • Nothing to report

      JRR:

      • Will be the contact point with the Asteroseismology group in Spain.
    • 11:00 12:00
      ESAP GUI update 1h

      Location: small meeting room in Sococo
      https://app.sococo.com/meet/ESCAPEWP5Meetingroom-3-3TtAZ0Z0Z85458985

    • 15:00 16:00
      Rucio integration team meeting
    • 13:30 14:00
      ESAP IDA 30m

      Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

      Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

    • 14:00 14:30
      ESAP UI & IVOA 30m

      Weekly update of ESAP UI & IVOA working group
      https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

      STV:

      Off for vacation. Not much updates. Updated the diagram with Nico.

      Nico:

      Documented use case from Susana. Split up all the databases and documented it. Played with the dynamic configuration. Merged dev branch of zooniverse to master. Added issue to ASTRON JIRA repo.

      Hugh:

      Talk to zooniverse developer to allow ESCAPE user to use zooniverse without have to type in password. Had a meeting with Nico and Zheng to discuss the next step of ESAP GUI. Added issues (tickets) to ESAP GUI gitlab. Looked at SAMP protocol. Not sure whether we need to do anything in the backend.

      STV: send SAMP request to server through HTTP, and the other way around.

      We will gilab issues for ticket tracking.

      Yan:

      Noting to report.

      Gareth:

      Nothing to report.

      Zheng:

      Going through the VO use case and Hugh's UI implementation. Will start the implementation this week.

    • 13:30 14:00
      ESAP IDA 30m

      Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

      Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

      Updatest on action from previous meeting:

      • Stelios V. setup initial list of tasks on gitlab and created branches for the IDA work on both the GUI and Gateway Gitlab projects
      • https://git.astron.nl/astron-sdc/esap-api-gateway/-/issues
      • https://git.astron.nl/astron-sdc/esap-gui/-/issues

      SKA (Rohini, James)

      • Demonstrated JupyterHub prototype with IAM authentication
      •  http://130.246.212.44/escape K8s instance works with ESCAPE IAM
      •  https://github.com/rohinijoshi06/jupyterhub-on-k8s
      •  Will be looking into REST API

      Discussion on Available Effort

      • Stelios V. will have 0.20 FTE development effort focused mostly on IDA
      • SKA will have available effort in part of Rohini & James as of September
      • Hugh Dickinson may have available dev effort, if it is focused on Zooniverse use cases
      • Zheng will be working on Rosetta 50% of her time in September of October, will have some time to work on this WG dev work, most likely in the GUI aspect

       

    • 14:00 14:30
      ESAP UI & IVOA 30m

      Weekly update of ESAP UI & IVOA working group
      https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

      Gareth:

      No update.

      Sara:

      Trying to write document to clarify use cases for Batch Processing. Will have a meeting with Matthias and ..., trying to define workflow, how to stage data and use it in computation tasks. Will share the document with Zheng in the near future.

      Yan:

      Worked on Rucio client.

      Nico:

      Working on UKIDSS use case. Created 2 new endpoints on the backend, get-services and run-query. Updated the documentation. Updated DB structure. Will be on holidays in two weeks.

      Hugh:

      Woking on Pagination, backend in Zooniverse, will also implemented in frontend. Handling the DB structure change.

      Stelios:

      Mostly working on the IDA working group.

      Zheng:

      Working on the UKIDSS use case.

       

       

    • 16:00 17:00
      Rucio ESAP integration 1h

      Update meeting

      Present:
      Rohini Joshi
      Aris Fkiaris
      Gareth Hughes
      Maisam Mohammadi-Dadkan
      Pierre Chanial
      Zheng Meyer-Zhao

      YG: Worked mostly on porting the rucio client Docker container to Singularity. T
      his also needed changes on the Dockerfile. It didn

      RJ: Implementation of the Jupyter notebook in the notebook server. Having Rucio env as one of the supported envirements. Programme increment will start in september. http://130.246.212.44/escape/

      RDM: Cluster is being upgraded. Later on, plan is to deploy a new cluster and mo
      ve the old cluster (delete it). Hopefully we can now use the DNS-resolved host n
      ame now.

      AF: hostname does resolve only to working IPs.

      MMD: Working on trying to get the Rucio jupyter lab setup.

      Sept 14 next meeting.

    • 13:30 14:00
      ESAP IDA 30m

      Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

      Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

      Updatest on action from previous meeting:

      • Stelios V. setup initial list of tasks on gitlab and created branches for the IDA work on both the GUI and Gateway Gitlab projects
      • https://git.astron.nl/astron-sdc/esap-api-gateway/-/issues
      • https://git.astron.nl/astron-sdc/esap-gui/-/issues

      SKA (Rohini, James)

      • Demonstrated JupyterHub prototype with IAM authentication
      •  http://130.246.212.44/escape K8s instance works with ESCAPE IAM
      •  https://github.com/rohinijoshi06/jupyterhub-on-k8s
      •  Will be looking into REST API

      Discussion on Available Effort

      • Stelios V. will have 0.20 FTE development effort focused mostly on IDA
      • SKA will have available effort in part of Rohini & James as of September
      • Hugh Dickinson may have available dev effort, if it is focused on Zooniverse use cases
      • Zheng will be working on Rosetta 50% of her time in September of October, will have some time to work on this WG dev work, most likely in the GUI aspect
    • 14:00 14:30
      ESAP UI & IVOA 30m

      Weekly update of ESAP UI & IVOA working group
      https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

      Yan:

      Not much to update.

      WP2 meeting with SWAN and Rucio integration. May use the SKAO JupyterHub instance to run this.

      Towards integration of WP2 and WP5.

      Mathieu:

      Not much. Catching up what has been done and following the development.

      Nico:

      1. Finished the UKIDSS use case backend with 3 API backends.

      2. All responses are wired to pagination.

      3. Documented all of it.

      4. Made Lofar service connector.

      Gareth:

      No updates. Matthias and Gareth have initiated a few meetings regarding DIRAC. Will receive information on that soon.

      Stelios:

      No updates.

      Zheng:

      Not much. Working on the frontend to use the new backends API developed by Nico. Zheng will finish it before end of this week, so a stable version of ESAP will be deployed before Nico goes on vacation.

       

       

    • 13:30 14:00
      ESAP IDA 30m

      Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

      Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

      James (SKA)

      Looking into JHub REST API and available options for extracting metadata and information

      https://docs.google.com/document/d/15Zs_OhuJ0Igfy6qiyzq4I4x4ePCRmtS6cxCuV2NzUYQ/edit#

       

      Stelios (Edinburgh)

      Also looking into JHub REST API aspects, will work with James & Rohini on sharing notes and ideas on this.

       

      Stefano 

      From Rosetta we could in theory use the Scheduler as an external microservice. 

      The interaction between Rosetta & ESAP would have to be determined, but basically we need to provide a container to be run, and a registry definition of a Compute Resource (XML/JSON) to Rosetta, with the appropriate user credentials. Rosetta then will run the Container and handle creating the tunelling.

      We would have to figure out how to then handle things like reverse proxying requests to the compute resource via the ESAP Gateway. (TBD)

       

      Actions:

      STV to add his notes to Jame's link 

       

    • 14:00 14:30
      ESAP UI & IVOA 30m

      Weekly update of ESAP UI & IVOA working group
      https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

    • 13:30 14:00
      ESAP IDA 30m

      Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

      Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

      James (SKA)

      Looking into JHub REST API and available options for extracting metadata and information

      https://docs.google.com/document/d/15Zs_OhuJ0Igfy6qiyzq4I4x4ePCRmtS6cxCuV2NzUYQ/edit#

       

      Stelios (Edinburgh)

      Also looking into JHub REST API aspects, will work with James & Rohini on sharing notes and ideas on this.

       

      Stefano 

      From Rosetta we could in theory use the Scheduler as an external microservice. 

      The interaction between Rosetta & ESAP would have to be determined, but basically we need to provide a container to be run, and a registry definition of a Compute Resource (XML/JSON) to Rosetta, with the appropriate user credentials. Rosetta then will run the Container and handle creating the tunelling.

      We would have to figure out how to then handle things like reverse proxying requests to the compute resource via the ESAP Gateway. (TBD)

       

      Actions:

      STV to add his notes to Jame's link 

    • 14:00 14:30
      ESAP UI & IVOA 30m

      Weekly update of ESAP UI & IVOA working group
      https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

    • 13:30 14:00
      ESAP IDA 30m

      Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

      Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

      • Stelios
      • Susana 
        • Planning on deploying a JHub service, which may be integrated (available as an option) in the ESAP prototype
        • Will think about how/if additional contributions can be made to this Working group 
      • Zheng
        • Will work on the UI for the ESAP IDA component
      • Gareth
        • No updates, may be working on the use of Ruscio with James & Rohini
      • James
        • Meeting with Stelios to discuss how we integrate our JHub services with the Platform
        • Will have available effort next couple months to work on integration with Ruscio
      • Matthieu
        • Most of effort in WP4,
        • VO protocols that could be useful for our use cases (SODA/Datalink)
    • 14:30 15:00
      ESAP UI & IVOA 30m

      Weekly update of ESAP UI & IVOA working group
      Join Zoom Meeting
      https://astron.zoom.us/j/98236774383

    • 13:30 14:00
      ESAP IDA 30m

      Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

      Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

    • 14:30 15:00
      ESAP UI & IVOA 30m

      Weekly update of ESAP UI & IVOA working group
      Join Zoom Meeting
      https://astron.zoom.us/j/98236774383

      Gareth:

      No updates

      Hugh:

      No updates

      Susana & Jose:

      SAMP use case. Looking at how to integrate two kinds of services into ESAP.

      Mathieu:

      No updates

      Zheng:

      Looked at the SAMP use case Susana provided.

      Yan:

      No updates

       

      SAMPJS in ESAP, doesn't support https. Susana will email Mark Taylor on this topic.