ESAP—Jupyter integration discussion

lundi 14 juin 2021, 16:00 → 17:00 Europe/Paris

Zoom

Escape ZoomRoom is inviting you to a scheduled Zoom meeting.

Meeting ID: 863 5651 9787 (https://astron.zoom.us/j/86356519787)
Passcode: (see e-mail, or ask in Rocket Chat #esap-interactive-analysis)
One tap mobile
+31207940854,,86356519787# Netherlands
+31207946519,,86356519787# Netherlands

Dial by your location
        +31 20 794 0854 Netherlands
        +31 20 794 6519 Netherlands
        +31 20 794 6520 Netherlands
        +31 20 794 7345 Netherlands
        +31 707 006 526 Netherlands
        +31 20 241 0288 Netherlands
Meeting ID: 863 5651 9787
Find your local number: https://astron.zoom.us/u/kbXjhufxPF

Join by SIP
86356519787@zoomcrc.com

Join by H.323
162.255.37.11 (US West)
162.255.36.11 (US East)
115.114.131.7 (India Mumbai)
115.114.115.7 (India Hyderabad)
213.19.144.110 (Amsterdam Netherlands)
213.244.140.110 (Germany)
103.122.166.55 (Australia Sydney)
103.122.167.55 (Australia Melbourne)
149.137.40.110 (Singapore)
64.211.144.160 (Brazil)
69.174.57.160 (Canada Toronto)
65.39.152.160 (Canada Vancouver)
207.226.132.110 (Japan Tokyo)
149.137.24.110 (Japan Osaka)
Meeting ID: 863 5651 9787

Join by Skype for Business
https://astron.zoom.us/skype/86356519787

16:00 → 17:00 Discussion

Table round

    - SKA BinderHub news
        - Planned for the next PI at SKA.
        - Integration BinderHub with the ESCAPE Jupyterhub at SKAO.
        - Will involve upgrading the K8S cluster that's being used for this work.
        - Also exploring which container registry will be used with BinderHub.
            - e.g. avoiding DockerHub rate limits.
        - Follow the “zero to BinderHub guide”.
        - Exploring what users need to have in their repo for them to be able to work with BinderHub.
        - Also tracking down use cases, e.g. the use case from IAA (HCG-16)
        - Also working to enable & respect ESCAPE IAM groups on the JupyterHub deployment as SKAO.
            - Ultimately, enables us to only allow a subset of users from the larger IAM system to log on.
        - This work starts on Wednesday, but the schedule is not fixed; expect a finish by mid August.
    - Nico has made progress on the shopping basket, expanding on the earlier work by Hugh Dickinson.
    - Pierre working on databases for ESAP.
        - So far working on the database schema.
        - Can take a query from the ESAP backend, and store it as a table.
        -  Not yet production grade.
        - Not closely integrated with Django; using FastAPI instead.
        - Should put this on the agenda for the next monthly meeting (21 June).
        - Could think about integrating the VO with these databases using some sort of federated database
        - He will pitch it next week to the monthly, then some more technical follow-up at next week's meeting.
    - [[Sara Bertocco]]:
        - Would like to play with Rucio — doesn't know which storage elements we can use.
        - Nobody on this call can help, but [[Yan Grange]] might be able to help.
        - Or there's a status & support channel on Rocket Chat.
        - WP2 should have a list of storage elements.

 

Review vision doc

    - Bertocco: Need to look at the model for pushing and pulling data in the data centre.
        - Pushing things into a data centre is harder for authentication and authorization reasons.
        - A pull model means that the site administrator can give things permission to run.
        - This only works if the ESAP Is hosted within the datacentre.
            - Needs a plugin installed inside the datacentre.
            - E.g. BinderHub.
    - Verkouter:
        - Capture metadata which describes the best place to the run IDA on some dataset.
        - Would be useful to start tracking the metadata that ESAP might need to manage this sort of process.
    - Joshi: Be aware that there are two different types of metadata — scientific, and representational of how the data is used or processed. Imagine an ESAP-global database where it can interpret particular storage URLs and understand what the closest/best place to run processing.
        - Do JupyterHub services get registered in the OSSR? If so, how is that achieved?!
- Voutsinas:
        - In earlier meetings, we decided that a hard-coded list of compute resources may be easier to manage.
        - In discussions with WP3, we're focusing entirely on the software aspect. Can follow up with WP3 about this.
        - Keep the window open to harvesting resources from both the OSSR and a local database in the long term.

Decision: track todo items in single GitLab issue tracker.