Name: WP5 Monthly meeting 20 July 2020
Start: 2020-07-20T11:00:00+02:00
End: 2020-07-20T12:00:00+02:00
Location: No location set

- 1
  
  1. Roll call
  
  ASTRON: Michiel van Haarlem, Zheng Meyer-Zhao, Christine Werkhoven, Yan Grange
  
  CERN: Xavier Espinal
  
  CNRS:
  
  CSIC: Susana Sanchez, Jose Ramon
  
  CTAO: Gareth Hughes
  
  EGO:
  
  FAIR:
  
  FAU: Jutta Schnabel
  
  INAF: Stefano Alberto Russo
  
  IFAE: Jordi Delgado
  
  JIVE: Mark Kettenis
  
  KIS:
  
  Nikhef:
  
  OU: Hugh Dickenson
  
  RUG: Maisam Mohammadi-Dadkan
  
  SKAO: Rosie Bolton
  
  UCM: Daniel Nieto, Jose Luis Contreras
  
  UEDIN: Stelios Voutsinas
- 2
  
  2. Last meeting minutes
  
  Minutes of last meeting can be found at the previous meeting.
  Overview of meetings: https://indico.in2p3.fr/category/906/
- 3
  3. Announcements, news and general updates
  
  Provide your information by adding minutes to the sub-contribution. You will need to have an indico account to do this.
  Note: Please also mention in your updates if you are going to or were at an event to talk about ESAP.
  
  For the current prototype platform development (code base) go to:
  Back end: https://git.astron.nl/vermaas/esap-gateway
  ESAP GUI: https://git.astron.nl/vermaas/adex-gui
  Overview Milestones
  
  31-Jul-20: MS29 Initial science platform prototype with discovery and data staging
  
  30-Sep-20: MS30 Deployment of initial set of ESFRI software on prototype platform
  
  30-Nov-20: MS31 Second WP5 workshop to analyse prototype performance
  
  Overview Deliverables
  
  31-Jan-2021: D5.3 Performance assessment of initial Science Platform prototype
  
  31-Jul-2022: D5.4 Final assessment of the performance of the Science Platform prototype and plan for deployment of production version within the EOSC.
  - a) ASTRON
    
    - Started focus months, 13 institutes participating, 5 working groups each of which is coordinated by an institute
    
    - Implementing User Profile, Rucio and IVOA connectors
    
    - Zheng and Nico will present the first result of ESAP this afternoon
  - b) CERN
    
    Link to google doc
    
    Simple ESCAPE datalake data access PoC
    
    The aim of this document is to illustrate a simple solution for a user to perform an analysis using data existing in the DataLake.
    
    This PoC would exploit both JupiterNotebooks and RUCIO functionalities.
    
    The final goal is to allow a user to work on data without necessarily acquiring RUCIO knowledge.
    
    This will be achieved by designing, developing, and deploying a JupiterLab UI extension that allows the user to discover data and make them accessible through/on the notebook environment.
    
    Overall architecture
    
    User Perspective
    
    Users can exploit the “explore” functionality to look up for datasets, files or DIDs using wildcards.
    
    Once the desired file is found, the user can either download it locally on the notebook.
    
    Authentication
    
    Users should be authenticated on the JupyterLab level and a Token should be fetched for them from IAM-ESCAPE.
    
    This Token will then be delegated to RUCIO and the actual storage sites to authorize them to fetch the files.
    
    In order to have a quickly deployed PoC the proposal is
    
    To have users authenticated only at JupyterLab level.
    
    Then, access to the storage will be performed using a shared service/robot x509 certificate and shared Rucio Service Account user credentials.
    
    Deployment and Configuration
    
    A recent JupyterLab version with the ‘Extension’ should be packaged in a single docker container.
    
    Then, each science can deploy this container or there could be a central deployment coming from WP5.
    
    The only extra configuration required by the admins of the JupyterLab should be to place a shared X509 grid certificate under the correct path and make sure the extension is pointing to the correct ESCAPE Rucio instance.
    
    Proposed action plan to get started with the activity
    
    Develop the “EXTENSION” and make sure it has the functionality needed (Our GSOC Student has already started working on it)
    
    Identify a place where a dockerized service could be run
    
    Deploy the JupyterLab instance
    
    Generate a grid robot certificate to be used by the service to authenticate with Rucio and Sites
    
    Add the grid robot certificate on IAM-ESCAPE
    
    Place the certificate inside the running container (mounted volume/ Kubernetes secret)
    
    Do some configuration changes on the JupyterLab instance
    
    Access it through a web browser and try it out
    
    Progress already made
    
    Our GSOC Student, Muhammad Aditya Hilmy (mhilmy@hey.com) has already made significant progress in the development of the Rucio extension for Jupyter lab.
    
    You can find his repository here
    
    https://github.com/didithilmy/rucio-jupyterlab
    
    He also keeps a Journal of the progress and various issues he faces here
    
    https://github.com/didithilmy/gsoc2020/tree/master/journal
    
    Right now there are two versions of the extension more or less working. One of them downloads the files to the local filesystem of the Notebook and the other one stages the files to a fuse mounted RSE.
    
    There are various issues still to be fixed for the
    
    Discovery of the files using wildcards
    
    Authentication mechanism
    
    Configuration of the extension
    
    But overall a satisfactory proof of concept can already be achieved.
    
    Following are some pictures from the Jupyter lab instance we got by simply running the following command. The data seen there are coming from the ESCAPE Datalake
    
    docker run -d -p 8888:8888 -e RUCIO_MODE=download -e RUCIO_BASE_URL=https://escape-rucio.cern.ch:32300/ -e RUCIO_AUTH_URL=https://escape-rucio.cern.ch:32301 -e RUCIO_DISPLAY_NAME=ESCAPE -e RUCIO_NAME=ESCAPE -e RUCIO_CA_CERT=/home/jovyan/certs/rucio_ca/ca.pem -v /home/frouk/work/secrets/test_jupyter:/home/jovyan/certs/rucio_ca -v /home/frouk/work/secrets/test_jupyter/x509up:/tmpm/x509up_u1000 didithilmy/rucio-jupyterlab:latest
  - c) CNRS-LAPP
  - d) CSIC
  - e) CTAO
    
    Introducing Gareth Hughes. CTAO attending almost all WG for focus month. Ramping up with meetings this week.
  - f) EGO
  - g) FAIR
  - h) FAU
    
    No dedicated WP5 activity, but preparation of software workshop WOSSL in WP3 beginning this week (indico event - still open for registration).
    
    Topics:
    
    Software Development Lifecycle Strategies
    
    Development: coding styles, versioning, maintenance, documentation, software quality standards
    
    Testing: test suites and code coverage
    
    efficiency optimization and benchmarking: standards and ideas
    
    platform integration and metadata
    
    software licenses
  - i) IFAE
    
    Recent updates from the GammaHub use case:
    
    - We are testing connections/queries to Hadoop-Hive from the Jupyter notebooks
    
    - Studying how to interface Gammapy API providing access to the Hive and Hadoop DB
  - j) INAF
    
    Coordinated with Zheng on how to integrate Rosetta into ESAP. Decided for a microservice-oriented integration, where the service to be integrated will be Rosetta's scheduler. Agreed to define base APIs over the summer and to try a first integration for September
  - k) JIVE
    
    JIVE employees working on WP5 are all on holiday. Before and after
    that work on CASA6 in a Jupyter environment is continuing,
    specifically adding UI elements using the same widget technology as
    used for other ESAP components."
  - l) KIS
  - m) Nikhef
    
    Hired a postdoc to work on the KM3Net use case. Will start after the summer.
  - n) RUG
    
    We are still developing our analysis software package to add the tools which are needed to run a Deep Neural Network (DNN) algorithm for tracking neutrons in NeuLand detector.
  - o) SKAO
    
    SKA continues to work on tasks around compute provision for interactive analysis (via both Openstack and Kubernetes), ways of providing a Jupyterhub service, and some specific Software provisions to support SKA-like use cases (e.g. installing the CARTA viewer for use with ASKAP data). We are involved in the “focus month”, and attending meetings when possible. Our main work time is already committed to mid August, but thereafter we will be able to set aside resources to develop software within the WP5 working groups - we are in the working groups on “interactive analysis” and “Rucio”.
  - p) UCM
    
    This focus month UCM participates in the ESAP UI & IVOA working group.
    
    We started technical discussions with Matthias Füssling, Gareth Hughes (CTAO), and Mathieu Servillat (Obs. PM), as well as with some CTAC colleagues in charge of DIRAC for CTA, directed to outline a roadmap on how to connect CTA to ESAP.
  - q) UEDIN
    
    Our recent activity in WP5 has been in the busy-months:
    
    Attending weekly "busy-months" meetings
    
    Project planning and developing use cases for the "Data Discovery" working group.
    
    https://docs.google.com/document/d/1QgFNUGWK8Uczzxd8pRi-ZPMRltaDi3JxV8kuolQ4IzM
    
    Project planning for the Interactive Data Analysis working group. (WG Lead)
    
    https://docs.google.com/document/d/1f6UAQ8ZtS37uxiikdYyTka7XWf0OOB8uukvi_qef4_E
- 4
  
  4. Update Focus Months
  
  Start: 30 June - End: September
  
  More info about this event is on:
  https://indico.in2p3.fr/event/21650/
  
  26-30 October we will discuss Focus Months developments and presentations will be given by working groups
  
  Working group UI & IVOA
  
  Coordinator: Zheng Meyer-Zhao (ASTRON)
  Google doc: https://docs.google.com/document/d/1QgFNUGWK8Uczzxd8pRi-ZPMRltaDi3JxV8kuolQ4IzM/edit?usp=sharing
  ESAP & VO Integration
  Service description
  Use case description
  UI design and implementation
  Backend implementation
  Working group updates:
  Every Monday 14:00 – 14:30 CEST
  ESCAPE WP5 Virtual Office (sococo)
  
  Working group Interactive Analysis
  
  Coordinator: Stelios Voutsinas (UEDIN)
  Google doc: https://docs.google.com/document/d/1f6UAQ8ZtS37uxiikdYyTka7XWf0OOB8uukvi_qef4_E/edit?usp=sharing
  ESAP & Analysis Interface Integration
  Rosetta (developed by INAF)
  User Profile:
  List of compute resources
  List of software repositories
  Data shopping cart
  Working group updates:
  Every Monday 13:30 – 14:00 CEST
  ESCAPE WP5 Virtual Office (sococo)
  
  Working group Rucio
  
  Coordinator: Yan Grange (ASTRON)
  First kick-off meeting today
  Redirect user to Rucio Web UI from ESAP
  Intergrate ESAP with Rucio REST API
  Start Rucio client in Analysis Interface
  Working closely with WP2 (including AAI related topics)
  Working group updates:
  Every (?) Monday 15:00 – 15:30 CEST
  ESCAPE WP5 Virtual Office (sococo)
  
  Working group Batch Processing
  Coordinator: Sara Bertocco (INAF)
  Working group plan and updates:
  Yet to be decided
  
  Working group DIRAC
  Coordinator: Matthias Fuessling (CTA)
  Working group plan and updates:
  Yet to be decided
- 5
  
  WP5 Tech meetings Topics
  
  Tech Talks are scheduled every first and third Monday of the month @4PM CEST/CET
  
  Please schedule your talk via the doodle poll: https://doodle.com/poll/safnc3ep3wv7pqqw
  
  Partners still to be scheduled from September 2020:
  
  CERN:
  
  CNRS:
  
  CSIC: will schedule a tech talk when prototype is ready
  
  CTAO:
  
  EGO:
  
  FAU:
  
  JIVE:
  
  KIS:
  
  Nikhef:
  
  OU:
  
  RUG:
  
  SKAO:
  
  UCM:
- 6
  
  Round table
  
  Discussion about using Rocketchat or Sococo for the Focus Months. Both platforms are being used.
  
  In Rocketchat the working groups are working in channels (which were private - now made public).
  Pro: You can read what has been going on in the written chats and prepare detailed answers, questions.
  
  Sococo is the online workplace where distributed teams come to work together. You can see everyone working online and 'knock' on their door to talk (or chat/zoom).
  
  - If you want to join Sococo send a request to Yan or Zheng.
  
  Pro: It works like a physical office, you can ask a quick question. See who is working in on which topic or have a coffee.
  
  Con: it only works in chrome browser or IE.
- 7
  
  Next meeting after summer holiday- Monday 21 September 11:00 CEST
  
  At 4PM there will be a Tech Talk given by Zheng Meyer & Nico Vermaas (ASTRON) more info available at
  https://indico.in2p3.fr/category/907/
  
  Note: The Meeting ID for the tech talk is not the same anymore as for the monthly meeting ID
  
  https://astron.zoom.us/j/94326349405?pwd=Ukc2Qk5CeXlUaFVZekczU2x0Q2xpUT09
  
  Meeting ID: 943 2634 9405
  Meeting Password: Will be provided via email*.

WP5 Monthly meeting 20 July 2020

Simple ESCAPE datalake data access PoC