Grids and Grid Applications

C. Loomis (LAL-Orsay)

EGEE Induction (Clermont-Ferrand)

March 22, 2005

Acknowledgements

Based on previous presentations by:

Dave Berry (NeSC) & David Fergusson

Contains material from:

Andrew Grimshaw (Univ. of Virginia)

Bob Jones (EGEE Tech. Director)

Mark Parsons (EPCC)

EDG Training Team

Roberto Barbera (INFN)

Ian Foster (Argonne National Laboratories)

Jeffrey Grethe (SDSC)

The National e-Science Centre

M. Petitdidier (EGAPP presentation)

O. Gervasi (EGAPP presentation)

Contents

EGEE: Enabling Grids for E-sciencE

Introduction to Grid Computing

Motivation

Expectations & Constraints

Historical Perspective

Grid Architectures

Converging Technologies

Grid Applications (e-Science)

Characteristics of e-Science

EGEE application areas

Typical Scenarios

Summary/Questions

Goal of Grid Computing

Goal in one sentence:

Allow scientists from multiple domains to use, share, and manage geographically distributed resources transparently.

Simple statement, many consequences:

Not specific to a particular application.

Jobs, policies cross administrative & political domains.

Sharing requires a means for accounting.

Transparency implies standardized services & APIs.

Access control for data and services.

Dynamic and heterogeneous resources.

Grid Actors

Users

Scientists with tasks requiring computational resources.

Virtual Organizations

People from different institutions with common goals.

Share computational resources to achieve those goals.

System Administrators

People responsible for keeping an institute's resources running.

Ensuring efficient and correct use of available resources.

Real Organizations

Institutes, funding agencies, governments, ...

Standards Bodies

OASIS, GGF, W3C, IETF, ...

Grid Vision

Grid “Middleware”

Grid Vision

Grid “Middleware”

Grid technology allows scientists:

access resources universally

interact with colleagues

analyse voluminous data

share results

Grid technology allow scientists:

access resources universally

interact with colleagues

analyze voluminous data

share results

Grid Vision

Grid “Middleware”

Includes traditional resources:

raw compute power

storage (disk, tape, ...)

network connectivity

Resources are:

heterogeneous

dynamic

Grid Vision

Grid “Middleware”

Detectors produce huge amounts of data for analysis.

Non-traditional resources:

scientific instruments

conferencing technologies

video

audio

chat

Grid Vision

Grid “Middleware”

Access to data:

data files and datasets

databases

replica metadata

application metadata

Manage data:

transfer and copy data

locate relevant data

Grid Vision

Grid “Middleware”

Services:

high-level services to facilitate use of grid

e.g. job brokering

application-specific services

e.g. portals

Grid Vision

Grid “Middleware”

What is the grid?

Middleware:

service interoperability

high-level services

Resources:

provided by participants

shared for efficient use

Scientific Motivation

Avoid reinventing the wheel:

Many computational tasks are common.

High-level, standardized services avoid duplication.

Scientists concentrate on results rather than tools.

Resource needs grow with time:

Start small for testing.

Push limits for ultimate sensitivity.

Grid APIs make finding and using additional resources easier.

Data access:

Find and access existing data more easily.

Share results for others to build upon.

Economic Motivation

Use of computing resources varies with time.

Analysis rush before major conferences.

End-of-quarter financial analyzes.

July and August holidays.

Current solutions:

Buy peak needed capacity; idle in non-peak periods.

Buy average capacity; delay results.

Grid solution:

Share resources to time-shift availability.

Buy average capacity but get timely results!

Improve reliability with automatic failover.

Grid Projects Worldwide

Access Grid

DISCOM

DOE Science Grid

Condor

ESG (Earth System Grid)

Fusion Collaboratory

Globus

GrADSoft (Grid Application

Development Software)

Grid Canada

GRIDS (Grid Research

Integration Development &

Support Center)

GriPhyN (Grid Physics

Network)

iVDGL (International Virtual

Data Grid Laboratory)

Music Grid

NASA Information Power Grid

NCSA Alliance Access Grid

AstroGrid

AVO (Astrophysical Virtual

Observatory)

Comb-e-chem

CrossGrid

DAME (Distributed Aircraft

Maintenance Environment)

DAMIEN (Distributed Applications and

Middleware for Industrial Networks)

DataTAG

Discovery Net

DutchGrid

EDG (European DataGrid)

EGSO (European Grid of Solar

Observations)

GEODISE (Grid Enabled Optimisation

& Design Search for Engineering)

GRIA (Grid Resources for

Industrial Applications)

Grid-Ireland

GridLab (Grid Application

Toolkit and Testbed)

GridPP

LCG (LHC Computing Grid)

MyGrid

NGIL (National Grid for

Learning Scotland)

NorduGrid (Nordic Testbed for Wide

Area Computing and Data Handling)

PIONIER Grid

Reality Grid

ScotGrid

ApGrid

ApBioNet

Grid Forum Korea

PRAGMA (Rim Applications and Grid Middleware Assembly)

Grid Datafarm for Petascale Data Intensive Computing

Gridbus Project

Major European Grid Projects

European Funded

European DataGrid

CrossGrid

DataTAG

LHC Computing Grid

GridLab

EUROGRID

DEISA

EGEE

National Grid Efforts

INFN Grid

NorduGrid

UK e-Science Programme

Family Tree

EDG

Globus

Condor

LCG

Alien

EGEE

...

Pan-European testbed.

Complete, functional set of services.

Significant productions demonstrated.

Subset of EDG services.

Improved robustness, scalability.

Worldwide production service.

Re-engineering:

Robustness

Standard interfaces

Expanded set of services.

Underlying Technology

Relative CPU, storage, and network capability impacts computing architecture.

Gilder’s Law
(32X in 4 yrs)

Storage Law
(16X in 4yrs)

Moore’s Law
(5X in 4yrs)

Performance per Dollar Spent

Optical Fibre
(bits per second)

Chip capacity
(numb. transistors)

Data Storage
(bits per sq. inch)

0 1 2 3 4 5

9 12 18

Doubling Time
(months)

Triumph of Light – Scientific American. George Stix, January 2001

Gilder’s Law
(32X in 4 yrs)

Storage Law
(16X in 4yrs)

Moore’s Law
(5X in 4yrs)

Performance per Dollar Spent

Optical Fibre
(bits per second)

Chip capacity
(numb. transistors)

Data Storage
(bits per sq. inch)

0 1 2 3 4 5

9 12 18

Doubling Time
(months)

Triumph of Light – Scientific American. George Stix, January 2001

Number of Years

Historical Perspective

Local Computing

All computing resources at single site.

People move to resources to work.

Remote Computing

Resources accessible from distance.

All significant resources still centralized.

Distributed Computing

Resources geographically distributed.

Specialized access; largely data transfers.

Grid Computing

Resources and services geographically distributed.

Standard interfaces; transfers of computations and data.

LCG Architecture

submit

Broker

...

Resource

Resource

InfoSys.

User

submit

publish descriptions

query

Batch-like architecture.

Stable, well-maintained resources.

Secured via Public Key Infrastructure (PKI)

Heavy support infrastructure.

Can handle large range of resources.

“Peer-to-Peer” Architecture

Peer-to-peer architecture (BOINC, XtremeWeb)

Volatile resources.

Limited security (client identifies server).

Lightweight infrastructure.

Handles limited types of resources.

App. DB

...

Resource

Resource

User

pull task

...

Service Oriented Architecture

Existing LCG system is largely service-oriented.

EGEE evolving to a clean SOA:

standard interfaces

standard technologies

Client

Service

Registry

query

location

publish description

interaction

Convergence of Technologies

Web Services

Clean, complete specification of service APIs.

Supported technology:

Good support within commercial sector.

Adequate support within open-source community.

Very active ➔ proposed standards rapidly evolving.

EGEE Service Evolution

Plain web services:

Avoid “proprietary” protocols and interfaces.

Fairly stable, will ease further evolution.

Adopt WSRF and/or WS-* standards as appropriate.

Expect user-visible changes in APIs.

e-Science

e-Science: Pushing frontiers of scientific discovery by exploiting advanced computational methods.

Use grid technology to:

Generate, curate, and analyze research data.

Develop and explore models and simulations.

Facilitate sharing of data, results, and resources.

EGEE Applications

Biomedical Applications

imaging, diagnosis, treatment

genome and protein studies

High-Energy Physics

simulation of particle interactions

analysis of detector data

Earth Science

observing terrestrial conditions

natural resources

Computational Chemistry

simulation of chemical properties

Typical Scenarios

Batch Use

Use the grid as a huge computational resource.

Simulation plays an important role in nearly all fields.

Portal Use

Use grid for load balancing of standardized applications.

Provides easy interface which hides grid complexities.

Agent Use

Centralized control and monitoring of a large production.

“Agent” jobs contact central database when started.

Interactive Use

Provide improved response time for intensive calculations.

Debugging of applications in situ.

Badly Adapted Uses

Multi-site parallel jobs:

Can't guarantee simultaneous start of all processes.

Can't easily control bandwidth and latencies between processes.

Can expose MPI-enabled site as a grid resource.

Tasks requiring real-time response.

Same as above: start up and response latencies, bandwidth.

If start up latency OK, can use “agent” model.

Summary

Grid technology attractive to many scientific endeavors:

Provides means of sharing resources to:

reduce overall hardware cost

reduce response times

improve reliability

Standardized, high-level APIs:

allow services to inter-operate effectively

allow scientists to concentrate on science rather than tools

EGEE:

improving the technology

deploying a powerful, worldwide grid

Questions

Questions?

Personal background:

Trained as a high-energy physicist.

Did doctorate on small experiment at Fermilab (near Chicago)

Worked in CDF collaboration.

Moved to Geneva and worked on LEP.

For 3.5 years, have been working on grid computing.

Integration coordinator for European DataGrid.

Chair of Project Technical Forum for EGEE.

No formal responsibilities in LCG, but there is a strong link with EGEE.

2005-03-19T10:44:00