S'authentifier

WP5 Monthly meeting 20 July 2020

→ Europe/Paris
Description

Meeting with all partners involved in WP5/ESAP

Join Zoom Meeting
https://astron.zoom.us/j/97308347310?pwd=Q0xCdkQ3R2lESEtQMjcybnpQZXRzZz09

Meeting ID: 973 0834 7310
Password: Will be provided via email*

One tap mobile
+31202410288,,97308347310# Netherlands
+31207940854,,97308347310# Netherlands

Contact
    • 11:00 → 11:03
      1. Roll call 3m

      ASTRON: Michiel van Haarlem, Zheng Meyer-Zhao, Christine Werkhoven, Yan Grange

      CERN: Xavier Espinal

      CNRS:

      CSIC: Susana Sanchez, Jose Ramon

      CTAO: Gareth Hughes

      EGO:

      FAIR:

      FAU: Jutta Schnabel

      INAF: Stefano Alberto Russo

      IFAE: Jordi Delgado

      JIVE: Mark Kettenis

      KIS:

      Nikhef:

      OU: Hugh Dickenson

      RUG: Maisam Mohammadi-Dadkan

      SKAO: Rosie Bolton

      UCM: Daniel Nieto, Jose Luis Contreras

      UEDIN: Stelios Voutsinas

    • 11:03 → 11:05
      2. Last meeting minutes 2m

      Minutes of last meeting can be found at the previous meeting.
      Overview of meetings: https://indico.in2p3.fr/category/906/

    • 11:05 → 11:15
      3. Announcements, news and general updates 10m

      Provide your information by adding minutes to the sub-contribution. You will need to have an indico account to do this.
      Note: Please also mention in your updates if you are going to or were at an event to talk about ESAP.

      For the current prototype platform development (code base) go to:
      Back end: https://git.astron.nl/vermaas/esap-gateway
      ESAP GUI: https://git.astron.nl/vermaas/adex-gui

      Overview Milestones

      • 31-Jul-20: MS29 Initial science platform prototype with discovery and data staging
      • 30-Sep-20: MS30 Deployment of initial set of ESFRI software on prototype platform
      • 30-Nov-20: MS31 Second WP5 workshop to analyse prototype performance

      Overview Deliverables

      • 31-Jan-2021: D5.3 Performance assessment of initial Science Platform prototype
      • 31-Jul-2022:  D5.4 Final assessment of the performance of the Science Platform prototype and plan for deployment of production version within the EOSC.

       

      • ASTRON 1m

        - Started focus months, 13 institutes participating, 5 working groups each of which is coordinated by an institute

        - Implementing User Profile,  Rucio and IVOA connectors

        - Zheng and Nico will present the first result of ESAP this afternoon

      • CERN 1m

        Link to google doc

        Simple ESCAPE datalake data access PoC

        • The aim of this document is to illustrate a simple solution for a user to perform an analysis using data existing in the DataLake.

        • This PoC would exploit both JupiterNotebooks and RUCIO functionalities. 

        • The final goal is to allow a user to work on data without necessarily acquiring RUCIO knowledge. 

        • This will be achieved by designing, developing, and deploying a JupiterLab UI extension that allows the user to discover data and make them accessible through/on the notebook environment. 

         

        Overall architecture

         

         

        User Perspective

         

        • Users can exploit the “explore” functionality to look up for datasets, files or DIDs using wildcards. 

        • Once the desired file is found, the user can either download it locally on the notebook.

         

         

        Authentication

           

        • Users should be authenticated on the JupyterLab level and a Token should be fetched for them from IAM-ESCAPE. 

        • This Token will then be delegated to RUCIO and the actual storage sites to authorize them to fetch the files. 

        • In order to have a quickly deployed  PoC the proposal is

          • To have users authenticated only at JupyterLab level. 

          • Then, access to the storage will be performed using a shared service/robot x509 certificate and shared Rucio Service Account user credentials.



         

        Deployment and Configuration

         

        • A recent JupyterLab version with the ‘Extension’ should be packaged in a single docker container. 

        • Then, each science can deploy this container or there could be a central deployment coming from WP5. 

        • The only extra configuration required by the admins of the JupyterLab should be to place a shared X509 grid certificate under the correct path and make sure the extension is pointing to the correct ESCAPE Rucio instance.

         

        Proposed action plan to get started with the activity

         

        1. Develop the “EXTENSION” and make sure it has the functionality needed (Our GSOC Student has already started working on it)

        2. Identify a place where a dockerized service could be run

        3. Deploy the JupyterLab instance

        4. Generate a grid robot certificate to be used by the service to authenticate with Rucio and Sites

        5. Add the grid robot certificate on IAM-ESCAPE

        6. Place the certificate inside the running container (mounted volume/ Kubernetes secret)

        7. Do some configuration changes on the JupyterLab instance

        8. Access it through a web browser and try it out

        Progress already made

         

        Our GSOC Student, Muhammad Aditya Hilmy (mhilmy@hey.com) has already made significant progress in the development of the Rucio extension for Jupyter lab. 

         

        You can find his repository here

        https://github.com/didithilmy/rucio-jupyterlab

         

        He also keeps a Journal of the progress and various issues he faces here

        https://github.com/didithilmy/gsoc2020/tree/master/journal

         

        Right now there are two versions of the extension more or less working. One of them downloads the files to the local filesystem of the Notebook and the other one stages the files to a fuse mounted RSE.

         

        There are various issues still to be fixed for the 

        • Discovery of the files using wildcards

        • Authentication mechanism

        • Configuration of the extension

        But overall a satisfactory proof of concept can already be achieved.

         

        Following are some pictures from the Jupyter lab instance we got by simply running the following command. The data seen there are coming from the ESCAPE Datalake

         

        docker run -d -p 8888:8888 -e RUCIO_MODE=download -e RUCIO_BASE_URL=https://escape-rucio.cern.ch:32300/ -e RUCIO_AUTH_URL=https://escape-rucio.cern.ch:32301 -e RUCIO_DISPLAY_NAME=ESCAPE -e RUCIO_NAME=ESCAPE -e RUCIO_CA_CERT=/home/jovyan/certs/rucio_ca/ca.pem -v /home/frouk/work/secrets/test_jupyter:/home/jovyan/certs/rucio_ca -v /home/frouk/work/secrets/test_jupyter/x509up:/tmpm/x509up_u1000 didithilmy/rucio-jupyterlab:latest

      • CNRS-LAPP 1m
      • CSIC 1m
      • CTAO 1m

        Introducing Gareth Hughes. CTAO attending almost all WG for focus month. Ramping up with meetings this week.

      • EGO 1m
      • FAIR 1m
      • FAU 1m

        No dedicated WP5 activity, but preparation of software workshop WOSSL in WP3 beginning this week (indico event - still open for registration).

        Topics:

        • Software Development Lifecycle Strategies
        • Development: coding styles, versioning, maintenance, documentation, software quality standards
        • Testing: test suites and code coverage
        • efficiency optimization and benchmarking: standards and ideas
        • platform integration and metadata
        • software licenses

         

         

      • IFAE 1m

        Recent updates from the GammaHub use case:

        - We are testing connections/queries to Hadoop-Hive from the Jupyter notebooks

        - Studying how to interface Gammapy API providing access to the Hive and Hadoop DB

      • INAF 1m

        Coordinated with Zheng on how to integrate Rosetta into ESAP. Decided for a microservice-oriented integration, where the service to be integrated will be Rosetta's scheduler. Agreed to define base APIs over the summer and to try a first integration for September

      • JIVE 1m

        JIVE employees working on WP5 are all on holiday.  Before and after
        that work on CASA6 in a Jupyter environment is continuing,
        specifically adding UI elements using the same widget technology as
        used for other ESAP components."

         

      • KIS 1m
      • Nikhef 1m

        Hired a postdoc to work on the KM3Net use case. Will start after the summer.

      • RUG 1m

        We are still developing our analysis software package to add the tools which are needed to run a Deep Neural Network (DNN) algorithm for tracking neutrons in NeuLand detector.  

      • SKAO 1m
        SKA continues to work on tasks around compute provision for interactive analysis (via both Openstack and Kubernetes), ways of providing a Jupyterhub service, and some specific Software provisions to support SKA-like use cases (e.g. installing the CARTA viewer for use with ASKAP data). We are involved in the “focus month”, and attending meetings when possible. Our main work time is already committed to mid August, but thereafter we will be able to set aside resources to develop software within the WP5 working groups - we are in the working groups on “interactive analysis” and “Rucio”.
      • UCM 1m
        • This focus month UCM participates in the ESAP UI & IVOA working group.
        • We started technical discussions with Matthias Füssling, Gareth Hughes (CTAO), and Mathieu Servillat (Obs. PM), as well as with some CTAC colleagues in charge of DIRAC for CTA, directed to outline a roadmap on how to connect CTA to ESAP.
      • UEDIN 1m

        Our recent activity in WP5 has been in the busy-months:

    • 11:15 → 11:25
      4. Update Focus Months 10m

      Start: 30 June - End: September

      More info about this event is on:
      https://indico.in2p3.fr/event/21650/

      26-30 October we will discuss Focus Months developments and presentations will be given by working groups

      Working group UI & IVOA

      Coordinator: Zheng Meyer-Zhao (ASTRON)
      Google doc: https://docs.google.com/document/d/1QgFNUGWK8Uczzxd8pRi-ZPMRltaDi3JxV8kuolQ4IzM/edit?usp=sharing 
      ESAP & VO Integration
      Service description
      Use case description
      UI design and implementation
      Backend implementation
      Working group updates:
      Every Monday 14:00 – 14:30 CEST
      ESCAPE WP5 Virtual Office (sococo)

       

      Working group Interactive Analysis

      Coordinator: Stelios Voutsinas (UEDIN)
      Google doc: https://docs.google.com/document/d/1f6UAQ8ZtS37uxiikdYyTka7XWf0OOB8uukvi_qef4_E/edit?usp=sharing
      ESAP & Analysis Interface Integration
      Rosetta (developed by INAF)
      User Profile:
      List of compute resources
      List of software repositories
      Data shopping cart
      Working group updates:
      Every Monday 13:30 – 14:00 CEST
      ESCAPE WP5 Virtual Office (sococo)

       

      Working group Rucio

      Coordinator: Yan Grange (ASTRON)
      First kick-off meeting today
      Redirect user to Rucio Web UI from ESAP
      Intergrate ESAP with Rucio REST API
      Start Rucio client in Analysis Interface
      Working closely with WP2 (including AAI related topics)
      Working group updates:
      Every (?) Monday 15:00 – 15:30 CEST
      ESCAPE WP5 Virtual Office (sococo)

       

      Working group Batch Processing
      Coordinator: Sara Bertocco (INAF)
      Working group plan and updates:
      Yet to be decided

       

      Working group DIRAC
      Coordinator: Matthias Fuessling (CTA)
      Working group plan and updates:
      Yet to be decided

    • 11:25 → 11:30
      WP5 Tech meetings Topics 5m

      Tech Talks are scheduled every first and third Monday of the month @4PM CEST/CET

      Please schedule your talk via the doodle poll: https://doodle.com/poll/safnc3ep3wv7pqqw

       

      Partners still to be scheduled from September 2020:

      CERN:

      CNRS:

      CSIC: will schedule a tech talk when prototype is ready

      CTAO:

      EGO:

      FAU:

      JIVE:

      KIS:

      Nikhef:

      OU:

      RUG:

      SKAO:

      UCM:

    • 11:30 → 11:35
      Round table 5m

      Discussion about using Rocketchat or Sococo for the Focus Months. Both platforms are being used.

      In Rocketchat the working groups are working in channels (which were private - now made public).
      Pro: You can read what has been going on in the written chats and prepare detailed answers, questions.

      Sococo is the online workplace where distributed teams come to work together. You can see everyone working online and 'knock' on their door to talk (or chat/zoom).

      - If you want to join Sococo send a request to Yan or Zheng.

      Pro: It works like a physical office, you can ask a quick question. See who is working in on which topic or have a coffee.

      Con: it only works in chrome browser or IE.

       

       

       

       

    • 11:35 → 11:40
      Next meeting after summer holiday- Monday 21 September 11:00 CEST 5m

      At 4PM there will be a Tech Talk given by Zheng Meyer & Nico Vermaas (ASTRON) more info available at
      https://indico.in2p3.fr/category/907/

      Note: The Meeting ID for the tech talk is not the same anymore as for the monthly meeting ID

      https://astron.zoom.us/j/94326349405?pwd=Ukc2Qk5CeXlUaFVZekczU2x0Q2xpUT09

      Meeting ID: 943 2634 9405
      Meeting Password: Will be provided via email*.

Fonctionne avec Indico v3.3.2