Rubin Observatory Data Management and System Performance | Data Facilities & Multi Site Processing

11 févr. 2025, 09:00 → 13 févr. 2025, 18:00 Europe/Paris

Amphitheatre (IN2P3 computing center)

Overview

This event is organized as a 3-days long meeting to discuss a limited number of topics: 

• ComCam Data and instrument effect in DM pipelines
• Multi-Site processing with FR, UK and US.
• Final verification of DM
• Facilities papers 

Audience

The all hands meeting is an opportunity to get many of us together in one place for a few days to address topics relevant to the joint processing of Rubin data at the data facilities.

Code of Conduct

The Rubin meeting Code of Conduct applies and all participants are expected to be familiar with it at the time of registration. 

Programme

The detailed programme is under Timetable (left menu).

Slack

A dedicated Slack channel is available on the Rubin Observatory Slack instance: #mtg-df-2025. 

Remote attendance

You can attend remotely via Zoom. The details of the rooms can be found in Confluence.

Social dinner

A social dinner is scheduled for Wednesday, February 12th, at a typical French restaurant.

COVID considerations

In France, wearing a mask is no longer mandatory, but it's still strongly recommended, especially if you have symptoms or are in indoor spaces.

schematic access to CC-IN2P3

CoC Rubin Code of Conduct.pdf

mar. 11 février

Welcome: Tuesday

1 Welcome, Logistics

JTM_Logisitic.pdf

2 Meeting Goals, kick off, Code of Conduct

Orateurs: Leanne Guy (AURA / Rubin Observatory), William O'Mullane (AURA / Rubin Observatory)

Welcome

Data Facility introductions: Session 1

3 French Data Facility

Orateur: Fabio Hernandez (CC-IN2P3)

slides

4 UK Data Facility

Orateur: George Beckett (Edinburgh University)

20250211_ukdf_status.pdf

10:30 Coffee break

Data Facility introductions: Session 2

5 US Data Facility

Orateur: Adam Bolton (SLAC National Accelerator Laboratory)

US Data Facility Overview (v2).pdf

Multi Site Processing: Session 1

6 DF perspectives, risks, science impacts, mitigations

Orateurs: Adam Bolton (The University of Utah), Fabio Hernandez (CC-IN2P3), George Beckett (Edinburgh University)

7 Outstanding issues and readiness plan

Orateurs: Fabio Hernandez (CC-IN2P3), Wei Yang (SLAC National Accelerator Laboratory)

DRP Demonstrator

The Other Things for Multi-Site

12:30 Lunch

Multi Site Processing: Session 2

8 Overview of data flow / lifecycle and feedback loops; needs for inter-site data replication

Orateur: Yusra AlSayyad (Princeton University)

DRP Pipelines, Datasets, and Flow

Data Life Cycle: Session 1

9 Phasing of multi-site milestones across DP1, DP2, DR1, DR2

15:30 Coffee break

Data Life Cycle: Session 2

10 Concept of Operations / Role & Responsibilities in the data lifecycle

Orateur: Adam Bolton (SLAC National Accelerator Laboratory)

Google doc for discussion

Computing Room Visit: Visit

mer. 12 février

Welcome: Wednesday

Infrastructure: Session 1

11 Progress report on profiling execution of pipetasks

Orateurs: Antoine BERNARD (LPSC), Quentin Le Boulc'h (CC-IN2P3)

2025-02-11_Rubin_DM_Meeting_at_CC.pdf

12 Operating on-prem RSPs: separation of concerns

Orateur: Frossie Economou (Rubin Observatory)

fe_2025_lyon_onprem.pdf

10:30 Coffee break

Tools: Session 1

13 Phalanx state of the union for DFs

Orateur: Frossie Economou (Rubin Observatory)

fe_2025_lyon_phalanx.pdf

14 Status of CM tools

Orateurs: Mx Orion Eiger (SLAC National Accelerator Laboratory), M. Toby Jennings (SLAC)

slides

15 Overview & status of PanDA

Orateurs: Dr Edward Karavakis (BNL), Dr Wen Guan (BNL)

PanDA overview & status.pdf

12:30 Lunch

Tools: Session 2

16 Sasquatch architecture, purpose, and usage

Orateurs: Frossie Economou (Rubin Observatory), Kian-Tat Lim (Rubin Observatory/SLAC)

fe_2025_lyon_sasquatch.pdf

Data Life Cycle: Session 3

17 Concept of Operations / Role & Responsibilities in the application+infrastructure environment

Orateur: Adam Bolton (SLAC National Accelerator Laboratory)

ConOps R2A2 apps and infra.pdf

15:30 Coffe break

Breakouts: 1

18 Managemnt(ish) breakout on remainign construciton work

What remains to be done ..

Breakouts: 2

19 Technical discussion topics: Harware roadmaps, Zipping files for transfer, Tape archiving, Service failover

Breakouts: 3

20:00 Social Dinner

The social event will take place at the restaurant Maison Villemanzy (https://www.maison-villemanzy.com), ten minutes away from Hôtel de Ville (City Hall).

Please select your menu here: https://docs.google.com/spreadsheets/d/1J6kIwPbfupRMb2aYy7ziSNubFBmgRDt8x19lDSwSYk8/edit?usp=sharing

Directions available here:

1. Google Maps : https://maps.app.goo.gl/rvJ3kecDonjAGknCA

2. Apple Maps Transit from Hotel de Ville by metro from Hotel de Ville (City Hall) : https://maps.apple.com/?daddr=La%20Maison%20Villemanzy,%2025%20Mont%C3%A9e%20St-S%C3%A9bastien,%2069001%20Lyon,%20France&dirflg=r&saddr=H%C3%B4tel%20de%20Ville%20%E2%80%93%20Louis%20Pradel,%2069001%20Lyon,%20France

3. Apple Maps Walk from Hotel de Ville: https://maps.apple.com/?daddr=La%20Maison%20Villemanzy,%2025%20Mont%C3%A9e%20St-S%C3%A9bastien,%2069001%20Lyon,%20France&dirflg=w&saddr=H%C3%B4tel%20de%20Ville%20%E2%80%93%20Louis%20Pradel,%2069001%20Lyon,%20France

jeu. 13 février

Welcome: Thursday

Data release timeline and scope: Session 1

20 Discussion of DR1 timeline and scope options (incl interactions with DP2 & DR2)

Orateurs: Dr Leanne Guy (AURA / Rubin Observatory), William O'Mullane (AURA/LSST)

discussion slides

10:30 Coffee Break

Data Facilities Documentation: Session 1

Data Facilities Documentation: Breakout 1

Data Facilities Documentation: Breakout 2

Data Facilities Documentation: Breakout 3

12:30 Lunch

Parking lot: Breakout 1

21 2-by-3 Processing

Orateur: Mlle Jennifer Adelman-McCarthy (Fermilab)

2X3ProductionLyon.pdf

multisiteDRPYanny.pdf

22 GDPR-compliant web applications

Orateur: M. Jean-René Rouet (CC-IN2P3/CNRS)

GDPR developer's introduction.pdf

23 Questions about archiving at FrDF

Orateur: Fabio Hernandez (CC-IN2P3)

slides

Parking lot: Breakout 2

Parking lot: Breakout 3

15:30 Coffee break

Closeout: Session 1

24 Closeout, Assignments, Action Items, Plans for JTM

Close out