WP5 ESAP Focus Months

30 juin 2020, 09:00 → 30 nov. 2020, 18:00 Europe/Paris

ZOOM

Zheng Meyer-Zhao (ASTRON)

The goal of this event is to engage all WP5 developers to integrate the ESAP services they are working on to make the ESAP prototype ready in September. The services are, for example,

* Integration of RUCIO with ESAP Application Programming Interface (API) Gateway using RUCIO REST API to demonstrate data staging
* Searching for VO-enabled data through ESAP User Interface (UI)
* Integration with interactive data analysis tools, such as start Jupyter notebook in a cloud environment
* Integration with HPC and HTC infrastructures

* Integration with DIRAC workload management system

After the Focus Months, a first release of the ESAP minimum prototype will be available to try out to integrate other services into ESAP.

 

We will form a working group for each of the above mentioned services. Partners are free to participate in one or more working groups.

 

Current working group and participating institutes:

 

Working group	Participating Institutes
Rucio	CERN, ASTRON, RuG, SKAO, EGO, OU
Data discovery UI and IVOA	ASTRON, UEDIN, CSIC, Paris Observatory, UCM, CTA, OU
Interactive analysis (Jupyter Notebooks on clusters)	UEDIN, SKAO, CSIC, INAF, RuG, Paris Observatory, FAIR, CTA, ASTRON, OU
Batch processing (HPC/HTC)	INAF, EGO, RuG, CTA, ASTRON
DIRAC	CTA, UCM, FAU, ASTRON

AAI aspects will be covered by most of the working groups, therefore, we are not having a separate working group for this.

Please email Zheng Meyer-Zhao (meyer@astron.nl) which working group you would like to participate in.

Repositories:

ESAP Backend: https://git.astron.nl/astron-sdc/esap-api-gateway

ESAP Backend Technology: Django & Django REST Framework

ESAP Frontend: https://git.astron.nl/astron-sdc/esap-gui

ESAP Frontend Technology: ReactJS

ESCAPE WP5 Detailed Project Plan: https://projectescape.eu/sites/default/files/WP5%20D5.2.pdf

• Communication tool
  • ESCAPE RocketChat: one channel per working group
  • If you don't have an account for ESCAPE RocketChat, please register at https://indico.in2p3.fr/event/19138/
• Working group lead
  • Rucio working group: CERN
  • Data discovery UI and IVOA working group: ASTRON
  • Interactive analysis working group: UEDIN
  • Batch processing working group: INAF
  • DIRAC working group: CTA
• Regular meeting planning per working group (cannot overlap with each other)
  • Weekly 30 min. meeting (time to be decided by the working group)
• Code review & commit
  • Each working group works on a separate branch of esap-api-gateway
  • In principle, each working group only works on topic-specific django application, e.g. query or rucio
  • Code needs to be reviewed within the working group before submmitting merge request to the master branch

Documentation

 

IDA working group:

• ESAP IDA User Interface Design
• Querying JupyterHub REST API for available services
• JupyterHub on k8s

 

ESAP UI & IVOA working group:

• Use cases for VO query
• How to integrate data catalog into ESAP (slides from Hugh Dickinson)

mar. 30 juin

1 Code base introduction

• Structure
• Query
• Accounts
• Rucio
• ....

Meeting recording (pwd: meeting pwd plus "!")

14:30 Break

2 Use cases

• IVOA registry data discovery interface
• ESFRI use cases

ADEX configuration

15:40 Break

3 Discussion

• communication tool
• identify working group lead
• Meeting planning per working group
• code review and commit

lun. 20 juillet

ESAP Rucio integration meeting

Meeting notes

lun. 27 juillet

4 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

lun. 3 août

5 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

SAMP example movie

NV:

• Together with Mattia Mancini tested staging app to use Celery and RabbitMQ.
• Made diagram of the use case provided by Stellios. Assumption is that it is all-in Python, this does not fit too much with reality. NV drew diagram of what he wrote, but with a different implementation.

 

 

HD:

• Prototype interface between Zooniverse and ESAP. Gave a short demo. Pretty much everything was done. Made updates to both the backend and frontend. Merge requests submitted. Waiting for NV or ZM.
• Next steps is to do the AAI in a more integrated way. Emailed with Andrea. Also talk with NV and Zheng on how stable the front- and backend are.

MF:

• Trying to catch up, reading through the minutes of the last meetings.

SS:

• Feedback to the Google document (STV use case). Mostly checking the usefulnes of the use case from the user perspective.
  • Needs to read more carefully.
• Think of some protocol and included a scenario based on this protocol. After writing the scenario, this has been part of the discussion already. Is SAMP used as a protocol that is relevant for the ESAP. Could be a useful option. SAMP can be useful to prevent us to reinvent the wheel.
• Will write a scenario with information of how this could be implemented. NV could then have. GH mentions that there may need to be an application running locally that consumes the SAMP requests. STV: TOPCAT and Aladin come with their own SAMP hub. Any web page can interact with them. Https causes issues.
• Other CSIC institute has time series data and would like to see if we can integrate timing series cataolgues as well. Will have a look at that.

GH:
 * No update for now.

 

STV:

• Read through the document and the notes
• Will have feedback on drawing Nico this week. Feedback was given during a discussion at the end of the meeting. NV took notes of the comments of STV

lun. 10 août

6 ESAP UI & IVOA

Weely update of ESAP UI & IVOA working group
https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

NV:

• Sellios use case discussed with Stellios. Based on that discussion new diagram which is in the document.
• Almost at the stage that ZM and NV can discuss. Backend code of HD has been merged. Frontend code has a clash with current version.
• ZM and NV to sit together to have a look at current status and discuss how to implemet.

GH:

• No development

HD:

• Last week discussed how it would be possible to query the data resource from the backend and then compute a schema for the frontend based on what HD learned about the backend. Got it working for Zooniverse. When backend starts up it queries the database.
• Implemented more advanced frontend. Also implemented how the data is rendered appropriately.
• Building from the master. Needs to discuss with ZM who changed the way the context is taken into account because this breaks the current implementation.
• Has not heared anything from Andrea Ceccanti yet.

SS:

• SAMP: Useful protocol to get data queries to ESAP. It is possible and SS sent around a repo that contains a demo of this. http://astrojs.github.io/sampjs/
• Asteroseismology group. They have catalogues that could be integrated in ESAP. Has given the contact person (Jose Ramon) details on the meeting. The implementation could be somewhat similar to the Zooniverse script.
• Alternative to using MySQL directly a protocol to access the data could be an option.
• Will send a video of SAMP around
• Will be on holiday shortly

MF:

• Nothing to report

JRR:

• Will be the contact point with the Asteroseismology group in Spain.

ven. 14 août

7 ESAP GUI update

Location: small meeting room in Sococo
https://app.sococo.com/meet/ESCAPEWP5Meetingroom-3-3TtAZ0Z0Z85458985

Rucio integration team meeting

Connection details

Live minutes

lun. 17 août

8 ESAP IDA

Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

9 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

STV:

Off for vacation. Not much updates. Updated the diagram with Nico.

Nico:

Documented use case from Susana. Split up all the databases and documented it. Played with the dynamic configuration. Merged dev branch of zooniverse to master. Added issue to ASTRON JIRA repo.

Hugh:

Talk to zooniverse developer to allow ESCAPE user to use zooniverse without have to type in password. Had a meeting with Nico and Zheng to discuss the next step of ESAP GUI. Added issues (tickets) to ESAP GUI gitlab. Looked at SAMP protocol. Not sure whether we need to do anything in the backend.

STV: send SAMP request to server through HTTP, and the other way around.

We will gilab issues for ticket tracking.

Yan:

Noting to report.

Gareth:

Nothing to report.

Zheng:

Going through the VO use case and Hugh's UI implementation. Will start the implementation this week.

lun. 24 août

10 ESAP IDA

Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

Updatest on action from previous meeting:

• Stelios V. setup initial list of tasks on gitlab and created branches for the IDA work on both the GUI and Gateway Gitlab projects
• https://git.astron.nl/astron-sdc/esap-api-gateway/-/issues
• https://git.astron.nl/astron-sdc/esap-gui/-/issues

SKA (Rohini, James)

• Demonstrated JupyterHub prototype with IAM authentication
•  http://130.246.212.44/escape K8s instance works with ESCAPE IAM
•  https://github.com/rohinijoshi06/jupyterhub-on-k8s
•  Will be looking into REST API

Discussion on Available Effort

• Stelios V. will have 0.20 FTE development effort focused mostly on IDA
• SKA will have available effort in part of Rohini & James as of September
• Hugh Dickinson may have available dev effort, if it is focused on Zooniverse use cases
• Zheng will be working on Rosetta 50% of her time in September of October, will have some time to work on this WG dev work, most likely in the GUI aspect

 

11 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

Gareth:

No update.

Sara:

Trying to write document to clarify use cases for Batch Processing. Will have a meeting with Matthias and ..., trying to define workflow, how to stage data and use it in computation tasks. Will share the document with Zheng in the near future.

Yan:

Worked on Rucio client.

Nico:

Working on UKIDSS use case. Created 2 new endpoints on the backend, get-services and run-query. Updated the documentation. Updated DB structure. Will be on holidays in two weeks.

Hugh:

Woking on Pagination, backend in Zooniverse, will also implemented in frontend. Handling the DB structure change.

Stelios:

Mostly working on the IDA working group.

Zheng:

Working on the UKIDSS use case.

 

 

12 Rucio ESAP integration

Update meeting

Present:
Rohini Joshi
Aris Fkiaris
Gareth Hughes
Maisam Mohammadi-Dadkan
Pierre Chanial
Zheng Meyer-Zhao

YG: Worked mostly on porting the rucio client Docker container to Singularity. T
his also needed changes on the Dockerfile. It didn

RJ: Implementation of the Jupyter notebook in the notebook server. Having Rucio env as one of the supported envirements. Programme increment will start in september. http://130.246.212.44/escape/

RDM: Cluster is being upgraded. Later on, plan is to deploy a new cluster and mo
ve the old cluster (delete it). Hopefully we can now use the DNS-resolved host n
ame now.

AF: hostname does resolve only to working IPs.

MMD: Working on trying to get the Rucio jupyter lab setup.

Sept 14 next meeting.

lun. 31 août

13 ESAP IDA

Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

Updatest on action from previous meeting:

• Stelios V. setup initial list of tasks on gitlab and created branches for the IDA work on both the GUI and Gateway Gitlab projects
• https://git.astron.nl/astron-sdc/esap-api-gateway/-/issues
• https://git.astron.nl/astron-sdc/esap-gui/-/issues

SKA (Rohini, James)

• Demonstrated JupyterHub prototype with IAM authentication
•  http://130.246.212.44/escape K8s instance works with ESCAPE IAM
•  https://github.com/rohinijoshi06/jupyterhub-on-k8s
•  Will be looking into REST API

Discussion on Available Effort

• Stelios V. will have 0.20 FTE development effort focused mostly on IDA
• SKA will have available effort in part of Rohini & James as of September
• Hugh Dickinson may have available dev effort, if it is focused on Zooniverse use cases
• Zheng will be working on Rosetta 50% of her time in September of October, will have some time to work on this WG dev work, most likely in the GUI aspect

14 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

Yan:

Not much to update.

WP2 meeting with SWAN and Rucio integration. May use the SKAO JupyterHub instance to run this.

Towards integration of WP2 and WP5.

Mathieu:

Not much. Catching up what has been done and following the development.

Nico:

1. Finished the UKIDSS use case backend with 3 API backends.

2. All responses are wired to pagination.

3. Documented all of it.

4. Made Lofar service connector.

Gareth:

No updates. Matthias and Gareth have initiated a few meetings regarding DIRAC. Will receive information on that soon.

Stelios:

No updates.

Zheng:

Not much. Working on the frontend to use the new backends API developed by Nico. Zheng will finish it before end of this week, so a stable version of ESAP will be deployed before Nico goes on vacation.

 

 

lun. 7 septembre

15 ESAP IDA

Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

James (SKA)

Looking into JHub REST API and available options for extracting metadata and information

https://docs.google.com/document/d/15Zs_OhuJ0Igfy6qiyzq4I4x4ePCRmtS6cxCuV2NzUYQ/edit#

 

Stelios (Edinburgh)

Also looking into JHub REST API aspects, will work with James & Rohini on sharing notes and ideas on this.

 

Stefano 

From Rosetta we could in theory use the Scheduler as an external microservice. 

The interaction between Rosetta & ESAP would have to be determined, but basically we need to provide a container to be run, and a registry definition of a Compute Resource (XML/JSON) to Rosetta, with the appropriate user credentials. Rosetta then will run the Container and handle creating the tunelling.

We would have to figure out how to then handle things like reverse proxying requests to the compute resource via the ESAP Gateway. (TBD)

 

Actions:

STV to add his notes to Jame's link 

 

16 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

lun. 14 septembre

17 ESAP IDA

Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

James (SKA)

Looking into JHub REST API and available options for extracting metadata and information

https://docs.google.com/document/d/15Zs_OhuJ0Igfy6qiyzq4I4x4ePCRmtS6cxCuV2NzUYQ/edit#

 

Stelios (Edinburgh)

Also looking into JHub REST API aspects, will work with James & Rohini on sharing notes and ideas on this.

 

Stefano 

From Rosetta we could in theory use the Scheduler as an external microservice. 

The interaction between Rosetta & ESAP would have to be determined, but basically we need to provide a container to be run, and a registry definition of a Compute Resource (XML/JSON) to Rosetta, with the appropriate user credentials. Rosetta then will run the Container and handle creating the tunelling.

We would have to figure out how to then handle things like reverse proxying requests to the compute resource via the ESAP Gateway. (TBD)

 

Actions:

STV to add his notes to Jame's link 

18 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
https://app.sococo.com/meet/ESCAPEWP5Meetingroom-2

lun. 21 septembre

19 ESAP IDA

Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

• Stelios
  • Had meeting with James to share notes & ideas on using the JHub REST API
  • https://docs.google.com/document/d/15Zs_OhuJ0Igfy6qiyzq4I4x4ePCRmtS6cxCuV2NzUYQ/edit#
• Susana 
  • Planning on deploying a JHub service, which may be integrated (available as an option) in the ESAP prototype
  • Will think about how/if additional contributions can be made to this Working group 
• Zheng
  • Will work on the UI for the ESAP IDA component
• Gareth
  • No updates, may be working on the use of Ruscio with James & Rohini
• James
  • Meeting with Stelios to discuss how we integrate our JHub services with the Platform
  • Will have available effort next couple months to work on integration with Ruscio
• Matthieu
  • Most of effort in WP4,
  • VO protocols that could be useful for our use cases (SODA/Datalink)

20 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
Join Zoom Meeting
https://astron.zoom.us/j/98236774383

lun. 28 septembre

21 ESAP IDA

Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

22 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
Join Zoom Meeting
https://astron.zoom.us/j/98236774383

Gareth:

No updates

Hugh:

No updates

Susana & Jose:

SAMP use case. Looking at how to integrate two kinds of services into ESAP.

Mathieu:

No updates

Zheng:

Looked at the SAMP use case Susana provided.

Yan:

No updates

 

SAMPJS in ESAP, doesn't support https. Susana will email Mark Taylor on this topic.

lun. 5 octobre

23 ESAP IDA

Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

Meeting link: https://app.sococo.com/meet/ESCAPEWP5Interactiveanalysis

24 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
Join Zoom Meeting
https://astron.zoom.us/j/98236774383

Susana: SAMP JS email to Mark Taylor, no reply yet. Will contact other people in Spain connected to IVOA.
Gareth: No updates
Stelios: No updates
Zheng: Finalising IVOA integration use case. Working on Lofar LTA query. Plan to integrate query for data within Rucio in ESAP GUI.
Announcement planning progress meeting. Presentations from each working group.
Questions from Matthias: How CTA can contribute? 
Zheng: use cases from CTA, how to query CTA data, through Rucio or DIRAC or CTA DB?
What metadata to query can be defined by each use case. 

Matthias: CTA will work on the use case.      

lun. 12 octobre

25 ESAP IDA

Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

Join Zoom Meeting
https://astron.zoom.us/j/98236774383

Updates

James: Rob starting along with Alba on getting Rucio client working
       Working on creating a separate JHub environment for science data challenge 1

Gareth: Working with James on Rucio extension
        Some technical issues to solve
        Got it working locally

Maisam: Finalized analysis environment for an experiment
        Working on configuring JHub environenment and deploying 
        Kick off meeting last week
        Working on deploying on Groningen server
    
Hugh: Nico and Hugh had a discussion to work on many-to-many db issue
      Change to foreign key issue
      Need to push Accounts model to branch
 
Nico: Was on holidays, will be looking in troubleshooting pagination and REST APIStelios: Started working on REST Services required for the IDA. Code to be pushed to Gitlab soon

 

WP5 Presentation

STV and others (?) to present and host a discussion on IDA related work October 26th

 

Next tasks?
STV: Store some data in db and make controller to read from that

STV: Setup Github with example notebooks that the IDA can read from

26 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
Join Zoom Meeting
https://astron.zoom.us/j/98236774383

Nico:

On holidays for a months, fixing some bugs in the backend.

Gareth:

Working on the CTAO use case for WP5.

Hugh:

Pushed user profile models into zooniverse branch. Needs to use the updated requirements.

Stellios:

No updates for this working group.

Zheng:

Working together with Nico on pagination. Working on the ADEX spin-off of ESAP GUI, a show case for other RIs or institutes if they would like to use ESAP GUI as a template to build there own SAP service.

Susana's email:

Great idea. It's not clear who is going to implement it yet, but we can discuss this later. For the moment, we are focusing on reaching the WP5 milestones. More development effort will probably be available next year. Stelios has worked on SAMP integration before, so he may help with the implementation when he has more time available.

 

lun. 19 octobre

27 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
Join Zoom Meeting
https://astron.zoom.us/j/98236774383

• Hugh: Good news that Hugh figured out the DB problem. It's now fixed. 
• Nico: Struggling with pagination. Now moving to the right direction in solving it for Apertif and VO.
• Mathieu: waiting for manpower, hope to hire a postdoc soon to install the esap platform for customization.
• Jose Ramon: Working on the seismology case, will give more update on this next week.
• Stelios: Not for this working group.
• Yan: on holiday last week.
• Progress meeting: planned at 14:40 -16:00
• online collaborative programming tool 

  https://repl.it
  https://github.com/features/codespaces/

lun. 2 novembre

28 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
Join Zoom Meeting
https://astron.zoom.us/j/98236774383

lun. 9 novembre

29 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
Join Zoom Meeting
https://astron.zoom.us/j/98236774383

lun. 16 novembre

30 ESAP IDA

Weekly meeting to discuss progress on building an Interactive Data Analysis Component within ESAP.

Join Zoom Meeting
https://astron.zoom.us/j/98236774383

31 ESAP UI & IVOA

Weekly update of ESAP UI & IVOA working group
Join Zoom Meeting
https://astron.zoom.us/j/98236774383

* Rename working group to Data discovery & AAI

* Nico and Susana will look at SAMP protocol in ESAP

* How to proceed with this meeting when Focus months ends?

Zheng will create

- Data discovery & AAI working group update indico page

- IDA working group update indico page

- Weekly updates, Monday 13:30 - 14:00 IDA WG, 14:00-14:30 Data discovery & AAI WG update

- Ask Andrea Ceccanti whether he will be able to join our meeting regularly, e.g. once per month. 

Gareth:

- working on CTA use case

Hugh:

- Would like to create new repo on gitlab. Nico will create a repo for Hugh.

Nico:

- busy with ADASS. Made https working for ESAP. working on CI/CD. VO dataset.

Stelios:

- No update on this WG. Have some SAMP example, could help Nico and Susana.

Susana:

- Will provide use cases

Zheng:

- Working on the IVOA gui to prepare for the ESCAPE mid-term review.