Rubin Observatory Data Management and System Performance | Data Facilities & Multi Site Processing

11 févr. 2025, 09:00 → 13 févr. 2025, 18:00 Europe/Paris

Amphitheatre (IN2P3 computing center)

Overview

This event is organized as a 3-days long meeting to discuss a limited number of topics: 

• ComCam Data and instrument effect in DM pipelines
• Multi-Site processing with FR, UK and US.
• Final verification of DM
• Facilities papers 

Audience

The all hands meeting is an opportunity to get many of us together in one place for a few days to address topics relevant to the joint processing of Rubin data at the data facilities.

Code of Conduct

The Rubin meeting Code of Conduct applies and all participants are expected to be familiar with it at the time of registration. 

Programme

The detailed programme is under Timetable (left menu).

Slack

A dedicated Slack channel is available on the Rubin Observatory Slack instance: #mtg-df-2025. 

Remote attendance

You can attend remotely via Zoom. The details of the rooms can be found in Confluence.

Social dinner

A social dinner is scheduled for Wednesday, February 12th, at a typical French restaurant.

COVID considerations

In France, wearing a mask is no longer mandatory, but it's still strongly recommended, especially if you have symptoms or are in indoor spaces.

schematic access to CC-IN2P3

CoC Rubin Code of Conduct.pdf

mar. 11 février ¶

09:00 → 09:30 Welcome: Tuesday¶

09:00 Welcome, Logistics¶

JTM_Logisitic.pdf

09:15 Meeting Goals, kick off, Code of Conduct¶

Orateurs: Leanne Guy (AURA / Rubin Observatory), William O'Mullane (AURA / Rubin Observatory)

Welcome

09:30 → 10:30 Data Facility introductions: Session 1¶

09:30 French Data Facility¶

Orateur: Fabio Hernandez (CC-IN2P3)

slides

10:00 UK Data Facility¶

Orateur: George Beckett (Edinburgh University)

20250211_ukdf_status.pdf

10:30 → 11:00 Coffee break

11:00 → 11:30 Data Facility introductions: Session 2¶

11:00 US Data Facility¶

Orateur: Adam Bolton (SLAC National Accelerator Laboratory)

US Data Facility Overview (v2).pdf

11:30 → 12:30 Multi Site Processing: Session 1¶

11:30 DF perspectives, risks, science impacts, mitigations¶

Orateurs: Adam Bolton (The University of Utah), Fabio Hernandez (CC-IN2P3), George Beckett (Edinburgh University)

12:00 Outstanding issues and readiness plan¶

Orateurs: Fabio Hernandez (CC-IN2P3), Wei Yang (SLAC National Accelerator Laboratory)

DRP Demonstrator

The Other Things for Multi-Site

12:30 → 14:00 Lunch

14:00 → 15:00 Multi Site Processing: Session 2¶

14:00 Overview of data flow / lifecycle and feedback loops; needs for inter-site data replication¶

Orateur: Yusra AlSayyad (Princeton University)

DRP Pipelines, Datasets, and Flow

15:00 → 15:30 Data Life Cycle: Session 1¶

15:00 Phasing of multi-site milestones across DP1, DP2, DR1, DR2¶

15:30 → 16:00 Coffee break

16:00 → 17:00 Data Life Cycle: Session 2¶

16:00 Concept of Operations / Role & Responsibilities in the data lifecycle¶

Orateur: Adam Bolton (SLAC National Accelerator Laboratory)

Google doc for discussion

17:00 → 18:00 Computing Room Visit: Visit¶

mer. 12 février ¶

09:00 → 09:10 Welcome: Wednesday¶

09:10 → 10:30 Infrastructure: Session 1¶

09:10 Progress report on profiling execution of pipetasks¶

Orateurs: Antoine BERNARD (LPSC), Quentin Le Boulc'h (CC-IN2P3)

2025-02-11_Rubin_DM_Meeting_at_CC.pdf

09:30 Operating on-prem RSPs: separation of concerns¶

Orateur: Frossie Economou (Rubin Observatory)

fe_2025_lyon_onprem.pdf

10:30 → 11:00 Coffee break

11:00 → 12:30 Tools: Session 1¶

11:00 Phalanx state of the union for DFs¶

Orateur: Frossie Economou (Rubin Observatory)

fe_2025_lyon_phalanx.pdf

11:30 Status of CM tools¶

Orateurs: Mx Orion Eiger (SLAC National Accelerator Laboratory), M. Toby Jennings (SLAC)

slides

12:00 Overview & status of PanDA¶

Orateurs: Dr Edward Karavakis (BNL), Dr Wen Guan (BNL)

PanDA overview & status.pdf

12:30 → 14:00 Lunch

14:00 → 14:30 Tools: Session 2¶

14:00 Sasquatch architecture, purpose, and usage¶

Orateurs: Frossie Economou (Rubin Observatory), Kian-Tat Lim (Rubin Observatory/SLAC)

fe_2025_lyon_sasquatch.pdf

14:30 → 15:30 Data Life Cycle: Session 3¶

14:30 Concept of Operations / Role & Responsibilities in the application+infrastructure environment¶

Orateur: Adam Bolton (SLAC National Accelerator Laboratory)

ConOps R2A2 apps and infra.pdf

15:30 → 16:00 Coffe break

16:00 → 17:00 Breakouts: 1¶

16:00 Managemnt(ish) breakout on remainign construciton work¶

What remains to be done ..

16:00 → 17:00 Breakouts: 2¶

16:00 Technical discussion topics: Harware roadmaps, Zipping files for transfer, Tape archiving, Service failover¶

16:00 → 17:00 Breakouts: 3¶

20:00 → 23:00 Social Dinner

The social event will take place at the restaurant Maison Villemanzy (https://www.maison-villemanzy.com), ten minutes away from Hôtel de Ville (City Hall).

Please select your menu here: https://docs.google.com/spreadsheets/d/1J6kIwPbfupRMb2aYy7ziSNubFBmgRDt8x19lDSwSYk8/edit?usp=sharing

Directions available here:

1. Google Maps : https://maps.app.goo.gl/rvJ3kecDonjAGknCA

2. Apple Maps Transit from Hotel de Ville by metro from Hotel de Ville (City Hall) : https://maps.apple.com/?daddr=La%20Maison%20Villemanzy,%2025%20Mont%C3%A9e%20St-S%C3%A9bastien,%2069001%20Lyon,%20France&dirflg=r&saddr=H%C3%B4tel%20de%20Ville%20%E2%80%93%20Louis%20Pradel,%2069001%20Lyon,%20France

3. Apple Maps Walk from Hotel de Ville: https://maps.apple.com/?daddr=La%20Maison%20Villemanzy,%2025%20Mont%C3%A9e%20St-S%C3%A9bastien,%2069001%20Lyon,%20France&dirflg=w&saddr=H%C3%B4tel%20de%20Ville%20%E2%80%93%20Louis%20Pradel,%2069001%20Lyon,%20France

jeu. 13 février ¶

09:00 → 09:10 Welcome: Thursday¶

09:10 → 10:30 Data release timeline and scope: Session 1¶

09:10 Discussion of DR1 timeline and scope options (incl interactions with DP2 & DR2)¶

Orateurs: Dr Leanne Guy (AURA / Rubin Observatory), William O'Mullane (AURA/LSST)

discussion slides

10:30 → 11:00 Coffee Break

11:00 → 11:30 Data Facilities Documentation: Session 1¶

11:30 → 12:30 Data Facilities Documentation: Breakout 1¶

11:30 → 12:30 Data Facilities Documentation: Breakout 2¶

11:30 → 12:30 Data Facilities Documentation: Breakout 3¶

12:30 → 14:00 Lunch

14:00 → 15:30 Parking lot: Breakout 1¶

14:00 2-by-3 Processing¶

Orateur: Mlle Jennifer Adelman-McCarthy (Fermilab)

2X3ProductionLyon.pdf

multisiteDRPYanny.pdf

14:30 GDPR-compliant web applications¶

Orateur: M. Jean-René Rouet (CC-IN2P3/CNRS)

GDPR developer's introduction.pdf

15:00 Questions about archiving at FrDF¶

Orateur: Fabio Hernandez (CC-IN2P3)

slides

14:00 → 15:30 Parking lot: Breakout 2¶

14:00 → 15:30 Parking lot: Breakout 3¶

15:30 → 16:00 Coffee break

16:00 → 17:00 Closeout: Session 1¶

16:00 Closeout, Assignments, Action Items, Plans for JTM¶

Close out