WP5 Tech Talk: Jupyter Notebook & VO examples by Stelios Voutsinas (University of Edinburgh)

Europe/Paris
Description

Connection details

You are invited to attend the Tech Talk taking place in a virtual meeting room:

Join Zoom Meeting
https://astron.zoom.us/j/7999431526

Meeting ID: 799 943 1526

Meeting Password: Will be provided via email*.
 

One tap mobile
+31207947345,,7999431526# Netherlands
+31202410288,,7999431526# Netherlands


Recording

The recording of this tech talk (password protected) can be found here:

https://astron.zoom.us/rec/share/4uovBI_gzV5Oc53k70XlRpEaPq7feaa81yMdr_FeyU5wcwnq8MII_ToS7XQLWLtX?startTime=1591625112000


*You can sign up for the ESAP Tech Talk e-mail notifications via werkhoven@astron.nl

Contact
    • 16:00 17:00
      Jupyter Notebook & VO examples in context of the ESCAPE science analysis platform 1h

      In this tech talk we will discuss

      a) Deploying a scalable JupyterHub service
      b) Creating Docker Containers for VO discovery and access and exposing them in a JHub Service
      b) Discovering and querying VO services using Python libraries & how this can be used in the ESAP Prototype


      A brief note with a summary for a discussion item for the next WP5 meeting, regarding JHub & computing cluster integrations.

      Several of the WP5 teams are working on using similar tools and technologies to setup parallel computing platforms with JupyterHub as a user interface, and Spark, Condor as well as other tools for running batch jobs in a distributed manner. It is worth discussing and planning how we can collaborate and potentially integrate the work that is being done. One possible way of integrating may be to descibe how/and if it is possible to allow users from a single Jupyterhub deployment, to access Spark or other clusters deployed at a remote data processing center. This may be initially built on prototype services as a proof of concept project, but for a production service we will have to deal with authentication and authorization aspects, and how to federate authentication and user information between the two.

      In addition, I also described example notebook containers as a base for custom user JupyterHub images, as well as a Notebook with examples on how to use VO tools for discovery, access and visualization.

      Speaker: Stelios Voutsinas

      Summary for a discussion item for the next WP5 meeting, regarding JHub & computing cluster integrations.

      Several of the WP5 teams are working on using similar tools and technologies to setup parallel computing platforms with JupyterHub as a user interface, and Spark, Condor as well as other tools for running batch jobs in a distributed manner. It is worth discussing and planning how we can collaborate and potentially integrate the work that is being done. One possible way of integrating may be to descibe how/and if it is possible to allow users from a single Jupyterhub deployment, to access Spark or other clusters deployed at a remote data processing center. This may be initially built on prototype services as a proof of concept project, but for a production service we will have to deal with authentication and authorization aspects, and how to federate authentication and user information between the two.

      In addition, I also described example notebook containers as a base for custom user JupyterHub images, as well as a Notebook with examples on how to use VO tools for discovery, access and visualization.

      The Dockerfile can be found here:
      https://github.com/stvoutsin/escape-wp5-dev/blob/master/docker/vo/Dockerfile


      And the example notebook here:
      https://github.com/stvoutsin/jhub-notebooks/blob/master/notebooks/escape/escape_demo.ipynb


      Finally, the presentation from this talk has been upload here:
      https://github.com/stvoutsin/escape-wp5-dev/blob/master/doc/20200608-ESCAPEWP5.pdf