iRODS workshop @ CC-IN2P3

2 févr. 2009, 08:00 → 5 févr. 2009, 18:00 Europe/Paris

CC-IN2P3

Jean-Yves Nief (CC-IN2P3), Reagan Moore (RENCI)

This workshop aims to gather people from various communities who are interested in iRODS or who are already using it. This workshop will also include an iRODS tutorial on rules and micro-services, and presentations on applications of the iRODS technology. If you are interested in presenting, please contact Jean-Yves Nief ("support" button on the left hand side). Please note that the number of participants will be limited.

lun. 2 février

Talks from various projects

Presentations by projects on their applications of the iRODS data grid. We would like to encourage presentations on client APIs that have been integrated with iRODS, rule sets that have been implemented, micro-services that have been developed.

1 Introduction

Orateur: M. Jean-Yves Nief

Slides introduction.ppt

2 iRODS status.

Orateur: Dr Reagan Moore

Slides iRODS-workshop.ppt

09:45 Coffee break

3 CC-IN2P3 director speech

Orateur: Dr Dominique Boutigny

4 ARCS Data Fabric

The Australian Research Collaboration Service (ARCS) has implemented the Data Fabric using SRB. We are able to use Shibboleth authentication with the use of Short Lived Certificate Service (SLCS) to create GSI credentials. The SRB server is also modified such that we have automatic account creation for users from trusted Identity Providers (IdP). We have also built usage scripts that will keep a history if the federation’s usage. With the release of iRODS 2.0, the data fabric will now be migrated to iRODS. Initial evaluation of the migration process is now complete. We have been working with iRODS developers to enable automatic account creation in iRODS. As part of the migration process, we are moving our usage scripts written to iRODS. I will discuss our experience with the migration process. ARCS is also developing software for accessing the data fabric – Hermes and Davis. Hermes is a Java application designed to work with commons virtual file systems (commons-vfs), for which, new providers are added for SRB and iRODS. Davis is a Java servlet that uses the open standard WebDAV protocol. Client software is already part of many operating systems and this simplifies access to SRB and iRODS.

Orateur: Mlle Pauline Mak

Slides iRODS_Workshop-FINAL.ppt

5 ASPiS: integrating iRODS with Shibboleth and provenance engines

Natively, iRODS addresses authentication by means of locally managed user lists or X509 certificates. The ASPiS project is integrating iRODS with Shibboleth, to enable authentication to be devolved onto the user’s home institution, and to suppport attribute/role-based authorisation decisions within the iRODS context. The project is also integrating iRODS with external provenance engines (PASOA and KARMA), to captute provenance data for scientific processes. Both integration approaches will make use of the rule system as far as possible.

Orateur: M. Eric Liao

Slides ASPiS-iRODS_workshopIN2P3.pdf

6 Using iRODS with the EnginFrame grid portal into the GRIDA3 project

GRIDA 3 (Shared Resources Manager for Environmental Data Analysis and Applications) is an interdisciplinary project funded by the Italian Research Ministry. Primarily designed for providing a holistic description of environmental problems, the project will result in an advanced problem-solving tool for the integration, through a computing portal, of human know-how, simulation software, instrumentation and resources for data communication, storage, visualization and computation. A central part of the project is the creation of a Data Grid spanning multiple sites across federated domains, public and private: for the creation of this Data Grid we chose to use the iRODS technology. This talk will describe how we integrated iRODS with the EnginFrame Grid Portal, in order to provide applications with the ability to integrate iRODS easily and to expose to the users the access to the Data Grid resources in a intuitive and user-friendly way. In particular, we will demonstrate one of the GRIDA3 applications, showing how it can seamlessly use resources from the iRDOS data grid both as input and output, and how the user can access these resources from his web browser, upload new resources and so on.

Orateurs: M. Francesco Locunto, M. Marco Piras

Slides LyonTalk-02Feb09.odp LyonTalk-02Feb09.pdf

7 SRB usage in BioEmergences

BioEmergences is an EC funded project where biologists, mathematicians, engineers and computer scientists define, implement and systematize the production of symbolic and precise reconstruction of cells evolution starting from in vivo captures of microscope images of embryos. The computational means for the project are provided by the different partners own computing facilities for initial runs of the algorithms. Afterwards the programs are moved to run under control of a workflow management application for systematic application to multiple datasets. The talk will briefly present the project, the workflow application. Then we will explain the differnt uses of SRB in the context of this project. Finally, we will conclude with a discussion on the potential migration to IRODS.

Orateur: M. Dominique De Waleffe

Paper bioemergence-forprint.pdf

Slides bioemergence-forbeamer.pdf

12:30 Lunch

8 Using Data Grids for Long Term Digital Preservation

In this talk we describe an FP7 integrated project focussed on long-term digital preservation called SHAMAN. We focus on the issues concerned with using data grids in long-term preservation.

Orateur: Dr Adil Hasan

Slides shaman-irods.ppt

9 The Storage Abstraction Service of the SPAR project

The National Library of France (BnF) is building its distributed archiving and preservation system (SPAR) in order to preserve in the long term all the digital information collected or created. This system is based on the OAIS standard but, in order to be independant of the underlying hardware infrastructure, a Storage Abstraction Service (SAS) is used. The SAS exposes its capabilities by the way of storage units, that represent some hardware designed to satisfy a given class of service, as well as records which abstracts the possible copies. In order to implement such Service, the choice of iRods has been made. In particular, a storage unit is seen as a particular resource associated with a set of irules to comply with the said class of service. In the presentation, we will show how such elements are defined and how the multiple operations needed for long-term preservation at the storage level can be achieved through the use of iRods.

Orateur: M. Thomas Ledoux

Slides iRods-BnF.pdf iRODS-BnF.ppt

10 The Adonis research data preservation project for digital humanities in France

The aim of this project is to give to the french researchers on Humanities Sciences, a distributed data working space. On top of this, the project will be able to provide long term data preservation, to handle digital objects concepts, to provide data treatment facilities and some basic workflow mechanism. This activity is also connected to some European project as DARIAH. It will be currently based on two major software: iRods for the data manipulation at the file level and fedora-commons (version 3) at the digital object level. Two major french computing centers will be involved: the CINES for long term preservation and the CC-IN2P3 for data access. At this time, around twelve numerical resources centers, distributed in France, are identified to provide numerical data and high level data management for the overall Digital Humanities community.

Orateur: M. Thomas Kachelhoffer

Slides Adonis.ppt

11 iRODS as future data grid backend for TextGrid ?

TextGrid, which is part of the D-Grid initiative (http://www.d-grid.de), is the first project in the humanities in Germany creating a community grid for the collaborative editing, annotation, analysis, and publication of specialist text resources. The architecture of TextGrid enhances a Globus-based grid infrastructure with a specific middleware layer and an open, WebService-based service layer of specialised functionalities for textual processing. The current TextGrid data grid infrastructure is implemented by using the applicable components of Globus Toolkit 4. With regard to the coming next phase of the TextGrid project we consider the redesign of the existing storage infrastructure taking advantage of both Fedora, its Digital Object Model and the flexible, rule-based concept of iRODS.

Orateur: M. Wolfgang Pempe

Slides textgrid_irods.pdf

16:00 Coffee break

12 Collaborative data life-cycle management for petascale astronomy projects

In this brief talk, we look at the needs for Collaborative Data Life-cycle Management (CDLM), the advantages of CDLM and the concepts in iRODS that enable a large-scale CDLM infrastructure. We will highlight two astronomy projects as our usecase, namely LSST and ALMA. Both of these projects will have to manage several petabytes of data for several years from multiple independent agencies (or countries). iRODS was used by LSST in the Supercomputing 2008 HPC Storage Challenge. The objective of this talk is to introduce this problem and form a community of users who want to engineer solutions for similar large-scale data management problems.

Orateur: M. Arun Jagatheesan

Slides iRODS-CDLM.ppt

13 Enabling a robust VOSpace based on iRODS

VOSpace is the International Virtual Observatory Alliance interface to distributed storage. It is the visible side of the storage system. To make a VOSpace usable in the real life we need an efficient storage mechanism. After a few experiences we have focused on iRODS which is a new data grid software system developed by the SDSC Storage Resource Broker team and collaborators. Our first aim was to create a storage area for Aladin but also for the new CDS Portal which is under development. In a first step we have developed an Aladin plugin giving an access to the iRODS implementation (through the Jargon Java API) and in a second step the VOSpace interface has been added over iRODS. We have developed a VOSpace Explorer in Java to access and manage the files. It is possible to do the common actions on the files. If a VO Tool supports drag and drop it is also possible to interact through this way with the explorer. iRODS is easy to implement and provides a good solution to ensure the robustness of a VOSpace. The installation is simple and can be done without much manpower. It is possible to start with a small configuration and to follow the evolution of the needs. A PLASTIC compliant tool like VOSpace Explorer is useful to provide a simple access to the stored files for VO Applications. As the main conclusion of this work we think that iRODS is a very good solution for the implementation of a robust VOSpace. And for many reasons (Open source, easy to use, flexible (definition of micro-services), follows the evolution of the architecture, etc.).

Orateur: M. André Schaaff

Slides iRODS-VOSpace-02February09.pdf

14 SRB service at STFC and the road to iRODS

STFC (Science and Technology Facilities Council, UK) is a long-time user of SRB in varied projects for internal and external customers alike. In this talk we describe how SRB has been used to meet the requirements of our main customers, and how the challenges associated with running a production service in this environment have been addressed. We also outline our plans for evaluation and migration to iRODS, noting which features we need to be present in order to migrate existing services without provoking a redesign of the infrastructure beneath.

Orateurs: M. Kevin O'Neill, M. Roger Downing

Slides Lyon-SRB-iRODS-STFC.ppt

mar. 3 février

Tutorial

Given feedback on what participant needs, specific talks would be selected on the chosen topics for the tutorials. The talks will include live demos, exercises. The goal is to turn each talk into a discussion session on the subject.

15 Introduction

Participants describe how they are using or intending to use iRODS and what they would like to know about the technology. We want feedback on which items are poorly understood, and which items need additional documentations.

Orateur: Prof. Reagan Moore

Paper iRODS-refmanual.doc iRODS-refmanual.pdf

Slides iRODS-tutorial.pdf iRODS-tutorial.ppt

10:30 Coffee break

16 Hands-on demo: iRODS installation

Installation of iRODS clients and server. Connection and exercises with the iRODS test bed build for the workshop.

Orateur: Prof. Reagan Moore

Slides iRODS-install.ppt

12:30 Lunch

17 Tutorial: introduction to rules

Orateur: M. Michael Wan

Slides IntroRules.ppt

16:00 Coffee break

18 Tutorial: assessment criteria

Orateur: Prof. Reagan Moore

Paper ISO-iRODS.doc

mer. 4 février

Tutorial

Tutorial on iRODS on different topics depending upon participant needs. These topics can include: - iRODS installation, logical name spaces, iCAT metadata catalog, RDA interface to remote databases, data transfer (parallel I/O versus RBUDP), iRODS clients (Unix, web browser, Jargon), fuse client interface and performance, storage system drivers, structured information resource interface (tar files, mounted collection), rules (default rules), micro-services (default set), writing a micro-service from start to finish, HDF5 micro-services, web access micro-services, XML/XSLT micro-services, SRB to iRODS migration support, authentication (challenge-response versus GSI versus Shibboleth), planned development

19 Tutorial: advanced features, data transfer modes, structured file implementation, mounted collections, Fuse interface etc...

Orateur: M. Michael Wan

Slides irodsAdvFeature.ppt

10:30 Coffee break

20 Tutorial: using Remote Database Access (RDA) interface

12:30 Lunch

21 Tutorial: writing and debugging a micro-service

Orateur: M. Michael Wan

Slides writeMS.ppt

16:00 Coffee break

22 Tutorial: writing and debugging a micro-service

Orateur: M. Michael Wan

Example examplesMS.c

jeu. 5 février

Parallel session: Astrophysics

Parallel sessions will be held so communities of interest can build collaborations.

Minutes astrophysics-session.docx

Parallel session: Preservation environments

Parallel sessions will be held so communities of interest can build collaborations.

Minutes preservation-session.doc

10:30 Coffee break

Parallel session: Preservation environments and medical records

Parallel sessions will be held so communities of interest can build collaborations.

12:30 Lunch

Common session on GUIs, APIs, security etc...

In this session, we will talk about topics of common interests where developments have been already made or are foreseen. These topics include GUIs (Hermes, Vbrowser, JUX, Windows explorer, web browsers), APIs (Java, Perl, Python, PHP), security (Shibboleth integration, GSI), management policies, specialized micro-services (web services, XML, image processing), migration from SRB to iRODS.

23 Davies and Hermes

Orateur: Mlle Pauline Mak

Minutes Hermes-Davies.docx

24 VBrowser

Orateur: M. Tristan Glatard

Slides glatard-irodsworkshop.pdf

25 JUX

Orateur: Dr Pascal Calvat

Slides JUX.ppt

26 Discussion on APIs, SRB to iRODS migration etc...

16:00 Coffee break