LSST school & workshop: Getting ready to do science with LSST data

Europe/Paris
Amphitheatre (IN2P3 computing center)

Amphitheatre

IN2P3 computing center

21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
Fabio Hernandez (CC-IN2P3), Réza Ansari (LAL)
Description

Overview

This event was organized as 3-days workshop devoted to discuss data processing challenges in large imaging surveys. It was preceded by a 1-day school and followed by 1-day hackathon. The whole event was held in the same location, with no parallel sessions.

Schedule:

Monday, June 12th: astronomical image processing school
Tuesday to Thursday, June 13th-15th: workshop
Friday, June 16th: hackathon

Audience

The workshop was intended for astronomers and scientists developing astronomical image processing algorithms and software, people in charge of the data processing and data management for large surveys, as well as computer scientists involved in these projects.

Experts from the LSST Data Management and DESC science working groups, as well as from other large photometric surveys such as Euclid and DES also participated.

The school was intended for young scientists, PhD students, postdocs and senior scientists not familiar with astronomical image processing. Introductory lectures will cover basics of instrument response correction, photometry, astrometry and calibration.

The hackathon was organised around the use of the LSST science pipelines (a.k.a. the stack) and associated tools.

The event was well attended by a diverse community.

 

Workshop Programme

  • Pushing the frontiers in astrometry and photometry
  • Large photometric surveys
  • LSST software stack beyond LSST
  • Computing infrastructure, data management and data bases for imaging surveys

Please refer to the Scientific Programme section for more details on the programme.

Participants willing to present their work related to one of the main subjects of the workshop 
listed above were encouraged to submit a contribution. Details available in the Call for Abstracts section.

 

Sponsors

This event is sponsored by the LSSTC Enabling Science Committee and by the institutions below:

Annotated map of the area around CC-IN2P3
Directions to CC-IN2P3
Participants
  • Alexandre Boucaud
  • Andre Schaaff
  • Augustin Guyonnet
  • Barak Zackay
  • Bui Van Tuan
  • Celine Combet
  • Christian Arnault
  • Christopher Waters
  • Cyrille Doux
  • Cyrille Rosset
  • César RICHARD
  • Darko Jevremovic
  • David Maurin
  • David ROUSSEAU
  • Dominique Boutigny
  • Dominique Fouchez
  • EMERIC LE FLOC'H
  • Eric Aubourg
  • Fabio Hernandez
  • Fabrice Jammes
  • Fritz Mueller
  • Frédéric Arenou
  • Hsin-Fang Chiang
  • Ian Sullivan
  • James Bosch
  • Jean Coupon
  • Jeff Carlin
  • Jinhyub Kim
  • Juan Pablo Reyes Gómez
  • Lance Miller
  • Laura Venuti
  • Marc Sauvage
  • Marcela HERNANDEZ
  • Mariana Penna-Lima
  • Mario Juric
  • Martin Folger
  • Masayuki Tanaka
  • Maude LE JEUNE
  • Michael Mommert
  • Mickael Rigault
  • Nicolas Chotard
  • Nicolas Regnault
  • Nima Sedaghat
  • Peter Melchior
  • Philippe Gris
  • Philippe Rosnet
  • Pierre Astier
  • Quentin Le Boulc'h
  • Robert Lupton
  • Rosaria Bonito
  • Réza Ansari
  • Sabine Elles
  • Santiago Serrano
  • SEO JIN KIM
  • Stephen Portillo
  • Thomas Puzia
  • Tim Jenness
  • Valery Brinnel
  • Vincent Pelgrims
  • William O Mullane
  • Yasna Ordenes
  • Yvan Calas
Need help? Contact the organizers
    • Registration: School Front Desk

      Front Desk

      IN2P3 computing center

    • School Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
      • 1
        Welcome and logistics Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        Orateurs: Fabio Hernandez (CC-IN2P3), Prof. Réza Ansari (LAL)
        Slides
      • 2
        Point sources and astrometry Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        Orateur: M. Pierre Astier (LPNHE)
        Transparents
        Video
      • 10:30
        Coffee and tea break 202 (IN2P3 computing center)

        202

        IN2P3 computing center

      • 3
        Point sources and astrometry (cont.) Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        Orateur: M. Pierre Astier (LPNHE)
        Video
      • 4
        Extended sources and galaxy photometry Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        Orateur: Robert Lupton (Princeton University)
        Slides
        Video
      • 12:30
        Lunch break 2nd floor (Restaurant DOMUS)

        2nd floor

        Restaurant DOMUS

      • 5
        Extended sources and galaxy photometry (cont.) Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        Video
      • 6
        Image differencing Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        Orateur: M. Dominique Fouchez (CPPM)
      • 15:30
        Coffee and tea break 202 (IN2P3 computing center)

        202

        IN2P3 computing center

      • 7
        Image differencing (cont.) Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        Orateur: M. Dominique Fouchez (CPPM)
        Video
    • Data center visit: machine rooms front desk

      front desk

      IN2P3 computing center

      Guided visit to the machine rooms in the data center.

      Président de session: Fabio Hernandez (CC-IN2P3)
    • Registration: Workshop Front Desk

      Front Desk

      IN2P3 computing center

    • Workshop: Pushing frontiers in astrometry and photometry (1) Amphitheatre

      Amphitheatre

      IN2P3 computing center

      Président de session: Prof. Réza Ansari (LAL)
      • 8
        Welcome and logistics Amphitheatre

        Amphitheatre

        IN2P3 computing center

        Orateurs: Fabio Hernandez (CC-IN2P3), Prof. Réza Ansari (LAL)
        Slides
      • 9
        Point Spread Function modelling and measurement Amphitheatre

        Amphitheatre

        IN2P3 computing center

        Accurate knowledge of the Point Spread Function (PSF) is crucial to meet the science goals of the wide area surveys of LSST and other telescopes. Increasing statistical precision in surveys leads to a requirement for increased accuracy in our knowledge of the PSF, which varies across images, with time and with the spectrum of the objects being observed – it is a challenging but interesting problem! The most demanding application is for enabling PSF correction in weak lensing surveys, and an approach for understanding the required accuracy is discussed, together with approaches for modelling the various physical effects that contribute to the overall PSF.
        Orateur: Prof. Lance Miller (Oxford University)
        Transparents
        Video
      • 10
        Efficient multi-band deblending Amphitheatre

        Amphitheatre

        IN2P3 computing center

        LSST and similarly deep surveys have to deal with ~40% of all objects being blended with others. The classical deblending approaches are often insufficient for high-quality measurements of galaxy properties. Their failures are related to simplifying assumptions about the galaxy shape or massive parameter degeneracies for complex galaxy models, and the reliance on single-band imaging. I will present a new algorithm (Melchior et al., in prep.) that is natively represented in multi-band data, i.e. it considers the SED the fundamental unit of measurement instead of per-band fluxes, and constrains the galaxy morphologies in a non-parametric fashion to avoid unstable results and model biases. The resulting constrained optimization problem is efficiently solved with a novel deterministic proximal algorithm (Moolekamp & Melchior, in prep.) with guaranteed convergence. I will show results from simulations and deep HSC imaging, and discuss the substantial gains from combining ground- and space-based imaging.
        Orateur: M. Peter Melchior (Princeton University)
        Transparents
        Video
      • 10:40
        Coffee and tea break 202 (IN2P3 computing center)

        202

        IN2P3 computing center

      • 11
        LSST DM Processing of Crowded Fields Amphitheatre

        Amphitheatre

        IN2P3 computing center

        LSST Data Management has committed to processing crowded stellar fields through our image differencing pipelines, but is not mandated to perform direct image photometry in these regions. Exactly what this means in terms of algorithms and the scientific quality of our catalogs is subtle and in some ways uncertain, and in this talk I'll explain the algorithms we're planning to run, attempt to make some guesses about the scientific quality of the outputs, and provide some information on how interested scientists can follow our progress and contribute tests and possibly algorithmic code.
        Orateur: Jim Bosch (Princeton University)
        Slides
        Video
      • 12
        Improved Point Source Detection in Crowded Fields using Probabilistic Cataloging Amphitheatre

        Amphitheatre

        IN2P3 computing center

        Cataloging is challenging in crowded fields because sources are extremely covariant with their neighbors and blending makes even the number of sources ambiguous. We present the first optical probabilistic catalog, cataloging a crowded (~0.1 sources per pixel brighter than 22nd magnitude in F606W) Sloan Digital Sky Survey r band image from M2. Probabilistic cataloging returns an ensemble of catalogs inferred from the image and thus can capture source-source covariance and deblending ambiguities. By comparing to a traditional catalog of the same image and a Hubble Space Telescope catalog of the same region, we show that our catalog ensemble better recovers sources from the image. It goes more than a magnitude deeper than the traditional catalog while having a lower false discovery rate brighter than 20th magnitude. We also present an algorithm for reducing this catalog ensemble to a condensed catalog that is similar to a traditional catalog, except it explicitly marginalizes over source-source covariances and nuisance parameters. We show that this condensed catalog has a similar completeness and false discovery rate to the catalog ensemble. Future telescopes will be more sensitive, and thus more of their images will be crowded. Probabilistic cataloging performs better than existing software in crowded fields and so should be considered when creating photometric pipelines in the Large Synoptic Survey Telescope era.
        Orateur: M. Stephen Portillo (Harvard-Smithsonian Center for Astrophysics)
    • 12:15
      Lunch break 2nd floor (Restaurant DOMUS)

      2nd floor

      Restaurant DOMUS

    • Workshop: LSST software stack beyond LSST Amphitheatre

      Amphitheatre

      IN2P3 computing center

      Président de session: M. Pierre Astier (LPNHE)
      • 13
        Hyper Suprime-Cam Subaru Strategic Program Amphitheatre

        Amphitheatre

        IN2P3 computing center

        HSC-SSP is a 3-layered imaging survey aimed at addressing major astrophysical questions such as the nature of dark matter and dark energy. The HSC data have been processed with a version of the LSST stack and a number of scientific papers have been published. I will give an overview of the survey and its fist public data release.
        Orateur: Dr Masayuki Tanaka (NAOJ)
        Slides
        Video
      • 14
        HSC data meets LSST code Amphitheatre

        Amphitheatre

        IN2P3 computing center

        The HSC is an 800 MPixel imager on the 8.2m Subaru telescope, carrying out a 1400 deg^2 week lensing survey. In many ways this survey is an LSST precursor, and we are reducing the data using prototype LSST pipelines. I shall discuss the state of the pipelines that we are using for the next data release, paying especial attention to algorithms that we know will need to be refined before we can meet LSST requirements.
        Orateur: Robert Lupton (Princeton University)
        Slides
        Video
      • 15:30
        Coffee and tea break 202 (IN2P3 computing center)

        202

        IN2P3 computing center

      • 15
        Photometry extraction and validation using the HSC pipeline of HSC+u-band data in HSC-SSP deep fields Amphitheatre

        Amphitheatre

        IN2P3 computing center

        We obtained 300 hours at CFHT to conduct a u-band follow-up (CLAUDS program, u<27) of the HSC-SSP Deep fields (grizY, r<27), over 20 deg2. The u band is primarily used to find drop-out (high redshift) galaxies and compute photometric redshifts. Therefore it is necessary to combine the two datasets in a fully consistent way. To do this we developed tools to import the CFHT stack images into the HSC pipeline framework and ran the photometry extraction using the HSC pipeline tools. In this talk I will explain how we did it and I will show some validation results.
        Orateur: Dr Jean Coupon (University of Geneva)
        Transparents
        Video
      • 16
        Reprocessing CFHT data with the LSST DM software stack Amphitheatre

        Amphitheatre

        IN2P3 computing center

        Orateur: Dominique Boutigny (LAPP)
        Slides
        Video
      • 17
        Reprocessing CFHT Deep fields with the stack Amphitheatre

        Amphitheatre

        IN2P3 computing center

        The talk will summarize the attempts made to reprocess the CFHT Deep fields with the stack. The results obtained as well as the future work will be exposed.
        Orateur: Philippe Gris (LPC Clermont-Ferrand)
        Transparents
        Video
    • Data center visit: machine rooms front desk

      front desk

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E

      Guided visit to the machine rooms in the data center.

      Président de session: Fabio Hernandez (CC-IN2P3)
    • 18
      Logistics announcements Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
      Orateur: Fabio Hernandez (CC-IN2P3)
      Slides
    • Workshop: Pushing frontiers in astrometry and photometry (2) Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
      Président de session: Robert LUPTON (Princeton University)
      • 19
        Joint astrometry Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        In order to optimally use multiple images of the same sky area, one has to accurately map the coordinates of these image onto a common coordinate system. Commonly,the images we get are significantly deeper than reference catalogs, and hence the coordinate mappings benefit from making use of common objects in the images, even if they do not appear in reference catalogs. I will describe some technicalities of this joint fit problem, the features of some implementations, and propose some goals for the LSST implementation.
        Orateur: M. Pierre Astier (LPNHE)
        Transparents
        Video
      • 20
        Creating DCR-matched templates for image differencing Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        The frequency-dependent spectral index of the atmosphere leads to spectrum-dependent distortions of sources away from zenith. While the effect of bulk refraction is easily corrected during astrometric calibration, this Differential Chromatic Refraction (DCR) introduces distortions that depend on the intrinsic spectrum of a source and vary with the parallactic angle and airmass of the observation. Past surveys have avoided DCR by restricting observations to low airmass, but LSST operations will require observations up to an airmass of 1.4 across the sky and up to 2 in some fields. With current image differencing techniques, we would have to choose between a high rate of false positive detections that would have to be culled with machine learning, or restrict template images to have similar airmasses and parallactic angles of the science image. However, the refraction of the atmosphere has been modeled for decades with increasing refinement, and we can use iterative forward modeling to estimate a sky model using that knowledge of the atmosphere to fit all observations of a patch simultaneously. We use this model to calculate new DCR-matched templates for image differencing, and find that it reduces the number of spurious source detections in DECam g-band images.
        Orateur: Ian Sullivan (University of Washington)
        Slides
        Video
      • 21
        Advances in astronomical image processing - Solving the problems of image coaddition and image subtraction Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        While co-addition and subtraction of astronomical images stand at the heart of observational astronomy, the existing solutions for them lack rigorous argumentation, are not achieving maximal sensitivity and are often slow. Moreover, there is no widespread agreement on how they should be done, and often different methods are used for different scientific applications. I am going to present rigorous solutions to these problems, deriving them from the most basic statistical principles. These solutions are proved optimal, under well defined and practically acceptable assumptions, and in many cases improve substantially the sensitivity, robustness and speed of both coaddition and subtraction. For coaddition, I will present a coadd image that is: a) sufficient for any further statistical decision or measurement on the underlying constant sky, making the entire data set redundant. b) improves both survey speed (by 5-20%) and effective spatial resolution of past and future astronomical surveys. c) improves substantially imaging through turbulence applications. d) much faster than many of the currently used coaddition solutions. For subtraction, I will present a subtraction image that is: a) optimal for transient detection under the assumption of spatially uniform noise. b) sufficient for any further statistical decision on the differences between the images, including the identification of cosmic rays and other image artifacts. c) Free of subtraction artifacts, potentially allowing (for the first time) robust automatic transient detection in real time, opening new avenues for scientific exploration. d) more sensitive than previous transient detection methods e) orders of magnitude faster than past subtraction methods.
        Orateur: M. Barak Zackay (Weizmann Institute of Science)
        Slides
        Video
      • 10:30
        Coffee and tea break 202 (IN2P3 computing center)

        202

        IN2P3 computing center

        Room adjacent to front desk

      • 22
        LSST DM stack image differencing on CFHT images Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        In an introduction part, I will present the basics of the implementation of image differencing with the LSST DM stack. Then I will present some preliminary results obtained by running this software on a subset of the CFHTLS Deep Fields images and some comparisons using the published supernova data of the SNLS collaboration.
        Orateur: M. Dominique Fouchez (CPPM)
        Slides
        Video
    • Workshop: Large photometric surveys (1) Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
      Président de session: Robert LUPTON (Princeton University)
      • 23
        Gaia, an overview
        The Gaia mission, currently in operational phase, will be described, from the current status, the organisation of the data processing, to the publication of the catalogues.
        Orateur: Dr Frédéric Arenou (CNRS/GEPI, Observatoire de Paris)
        Transparents
    • Group photo front desk

      front desk

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E

      Please go to the front desk for a group photo on the stairs at the main entrance of the building.

    • 12:30
      Lunch break 202

      202

      IN2P3 computing center

      Buffet in the room adjacent to the front desk

    • Workshop: Pushing frontiers in astrometry and photometry (3) Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
      Président de session: Prof. Mario Juric (University of Washington)
      • 24
        Dark Energy Survey - Status, Science, and Algorithmic Advancements Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        I will review the status of the Dark Energy survey, give an outlook on the upcoming results from its weak-lensing program, and describe several of the algorithmic advancements that enable high-precision measurements from ground-based imaging.
        Orateur: M. Peter Melchior (Princeton University)
        Transparents
        Video
      • 25
        Forward Global Calibration of the Dark Energy Survey [remote presentation] Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        Many scientific goals for DES require calibration of broadband grizY photometry that is stable in time and uniform over the sky to better than 1%. It is also necessary to limit systematic uncertainty in the calibrated broadband magnitudes due to the spectrum of the source. I present details on the Forward Global Calibration Method (FGCM), which combines data taken with auxiliary instrumentation with data from the survey imaging itself and models of the instrument and atmosphere to estimate the spatial- and time-dependence of the passbands of individual DES exposures. The passband of individual observations is combined with the source spectral shape to compute chromatic corrections to the standard system which is necessary to achieve sub-percent calibrations.
        Orateur: Eli Rykoff (SLAC National Accelerator Laboratory)
        Slides
        Video
      • 26
        Photometric Calibration: lessons from CFHTLS Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        For about a decade, photometric calibration has been identified as a potential limitation of current and future imaging surveys. Uncertainties on the relative (band to band) flux scale are currently the dominant contribution to the systematic error budget affecting SNIa distances. Non-uniformities of the imager response may introduce spurious correlations in the LSS analyses, via the photometric redshifts. In this talk, I review the lessons learnt from the analysis of the CFHTLS survey, and I discuss related topics of the LSST-DESC roadmap.
        Orateur: Nicolas Regnault (LPNHE)
        Transparents
        Video
      • 15:45
        Coffee and tea break 202 (IN2P3 computing center)

        202

        IN2P3 computing center

        Room adjacent to the front desk

      • 27
        PHOTOMETRYPIPELINE: An automated pipeline for calibrated photometry Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        PHOTOMETRYPIPELINE (PP) is an automated pipeline that produces calibrated photometry from imaging data through image registration, aperture photometry, photometric calibration, and target identification with only minimal human interaction. PP utilizes the widely used Source Extractor software for source identification and aperture photometry; SCAMP is used for image registration. Both image registration and photometric calibration are based on matching field stars with star catalogs, requiring catalog coverage of the respective field. A number of different astrometric and photometric catalogs can be queried online. Relying on a sufficient number of background stars for image registration and photometric calibration, PP is well-suited to analyze data from small to medium-sized telescopes. Calibrated magnitudes obtained by PP are typically accurate within <0.03 mag and astrometric accuracies are of the order of 0.3 arcsec relative to the catalogs used in the registration. The pipeline consists of an open-source software suite written in Python, can be run on Unix-based systems on a simple desktop machine, and is capable of realtime data analysis. PP has been developed for observations of moving targets, but can be used for analyzing point source observations of any kind. I will present the pipeline's implementation and discuss some current and future features.
        Orateur: Dr Michael Mommert (Northern Arizona University)
        Slides
    • 28
      HSC demonstration Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
      Orateur: Dr Masayuki Tanaka (NAOJ)
    • Data center visit: machine room Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E

      Guided visit to the machine rooms in the data center.

      Président de session: Fabio Hernandez (CC-IN2P3)
    • Social Event Restaurant Pléthore et Balthazar 72 rue Mercière 69002 LYON

      Restaurant Pléthore et Balthazar 72 rue Mercière 69002 LYON

      Restaurant
      Pléthore et Balthazar
      72 rue Mercière
      69002 LYON

      Google maps
      • 29
        Social Event
        Annotated map
        Google maps
    • 30
      Logistics announcements Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
      Orateur: Fabio Hernandez (CC-IN2P3)
      Slides
    • Workshop: Computing infrastructure and data management Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
      Président de session: Dominique Boutigny (LAPP)
      • 31
        LSST Data Management Overview Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        In this brief talk we will outline the current Data Management design and approach to handling LSST Data. DM is a multifaceted distributed system. It is also built in a distributed manner so we shall also cover the organization of the DM development work.
        Orateur: Dr William O'Mullane (AURA/LSST)
        Slides
        Video
      • 32
        What LSST Will Deliver: Images, Catalogs, Alerts, Services Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        The LSST is an integrated survey system. The observatory, telescope, camera and the data management systems will be built to conduct the LSST survey and will not support the 'PI mode' in the classical sense. Instead, the ultimate, science-enabling, deliverable of LSST will be the fully reduced data products and accompanying services. In this talk, we will present the baseline design and contents of LSST data products. "Level 1" data products will be generated continuously every observing night and include measurements such as alerts to objects that have changed brightness or position. "Level 2" data products will be made available as annual Data Releases and will include images and measurements of quantities such as positions, fluxes, and shapes, as well as variability information such as orbital parameters for moving objects and an appropriate compact description of light curves. These data will be made available through the LSST Science Platform -- a set of integrated services offered in LSST Data Access Centers. The Platform will enable efficient data access and querying, as well as execution of custom analyses and generation of added value data products. These will enable science beyond the capabilities of Level 1 and 2 deliverables (so called "Level 3").
        Orateur: Prof. Mario Juric (University of Washington)
        Transparents
        Video
      • 33
        Qserv: A distributed shared-nothing database for the LSST catalog Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        To satisfy the need to efficiently store, query, and analyze catalogs running into trillions of rows and petabytes of data, we are developing Qserv, a distributed shared-nothing SQL database query system. We describe Qserv's design, architecture, and ability to scale to LSST's data requirements. We illustrate its potential with some current test results.
        Orateur: Fritz Mueller (SLAC)
        Slides
        Video
      • 10:40
        Coffee and tea break 202 (IN2P3 computing center)

        202

        IN2P3 computing center

        Room adjacent to front desk

      • 34
        Scaling cloud for LSST catalog at IN2P3 Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        French engineers and researchers are very interested in the challenge of the LSST catalog and contribute in particular to validating Qserv and preparing its production phase. In this context, they perform integration tests, based on CFHT data, and ranging from the pixels to the execution of scientific SQL queries. In addition, they design Qserv large scale deployment and management procedure using a methodology for building modern, scalable, maintainable software-as-a-service apps. They also work closely with LSST Square team to provide continuous delivery, and automate both live distributed testing and release distribution. All these activities will be detailed during this presentation.
        Orateur: Fabrice Jammes (CNRS)
        Transparents
        Video
      • 35
        LHC experiments : Petabytes to papers Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        The LHC experiments are now routinely collecting (also simulating) and analyzing Petabytes of data each year, publishing 200 papers, signed each by 3000 people. Focussing on the ATLAS experiment, the talk will describe the organisation of the data processing that makes it possible: what balance between accuracy (of software and calibrations) and stability was reached, allowing specialized treatment of data for specific papers despite limited resources. The talk will conclude on the various challenges posed by eventual 10 fold increase of LHC luminosity and 10 fold increase of data taking rate.
        Orateur: Dr David Rousseau (LAL-Orsay)
        Transparents
        Video
      • 36
        Exploring Spark and MongoDb for LSST Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        Spark is a very promising technology offering distributed data and computing mechanisms. At LAL(Orsay) we have started to look at how the typical computing workflows used in LSST could use the Spark eco-system: How to distribute algorithms in a map-reduce approach How to format various data structures to partition them in a distributed file system Thus, a OpenStack based cluster has been configured at LAL with Spark and its various associated components, and several models are experienced to evaluate the performance and configuration parameters (memory, CPU, …) In the same context, in the process in exploring various technologies related with QServ or the catalog access techniques, we are working on two promissing technologies: MongoDB and Spark DataFrames, both offering a natural data or processing distribution approach. The method is similar for both: we exploit one limited dataset (2To) (sources and objects) and try and apply the benchmarking queries that used to be applied to QServ. The concepts, the ingestion, and the querying methods are explored, in particular looking at possible functional or performance limitations for both systems. Several platforms are used for this study: The Galactica cluster at Clermont (Petasky context) OpenStack at LAL (VirtualData context) A test cluster CCIN2P3.
        Orateur: M. Christian Arnault (CNRS)
        Transparents
        Video
    • 12:50
      Lunch break self service (DOMUS restaurant)

      self service

      DOMUS restaurant

    • Workshop: Large photometric surveys (2) Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
      Président de session: Prof. Réza Ansari (LAL)
      • 37
        Euclid - An update on the mission and its processing plan Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        The preparation of the Euclid mission is progressing, with the instruments entering the production of the Qualification models, and the data processing ground segment in its design review. I will make a brief update of where the mission stands today, and focus on the plans that are made for operation and data processing. Details will also be given on how Euclid relies on external collaborations for input data, and on the planned public data releases and their content.
        Orateur: Dr Marc Sauvage (CEA/DRF/Irfu/SAp)
        Transparents
        Video
      • 38
        The Dark Energy Survey Data Management System [remote presentation] Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        The Dark Energy Survey covering 5,000 square degrees of the South Galactic Cap along with a dedicated 30 square degree SN survey has now completed four years of observations. The Data Management group (DESDM) has been responsible for the processing, release, and curation of these data products for the DES collaboration and to make them available to the general astronomical community. I will give a brief overview of the DESDM pipelines and data management system, focusing on their evolution and performance, with an emphasis toward providing lessons that might help inform the current development effort of the LSST data management project.
        Orateur: Robert Gruendl (University of Illinois)
        Slides
        Video
      • 15:20
        Coffee and tea break 202 (IN2P3 computing center)

        202

        IN2P3 computing center

        Room adjacent to the front desk

      • 39
        Weak Lensing Data processing for the CFHTLenS and KiDS surveys [remote presentation] Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        I will discuss the data processing and data handling of the CFHTLenS and KiDS collaborations. Special emphasis will be given on our algorithms and processing methods for weak gravitational lensing studies. I will summarise the current status of our image processing pipeline THELI which is optimised to produce lensing quality data from current and future optical Wide-Field Imaging Surveys. Our software suite encompasses all necessary modules from raw images to multi-band object catalogues with photometric redshifts and galaxy shapes. The talk will be complemented with a summary of the lessons we learnt in the past 15 years to successfully implement and execute an optical Wide-Field Imaging survey.
        Orateur: Thomas Erben (Universität Bonn, Argelander-Institut für Astronomie)
        Transparents
        Video
      • 40
        Lessons from a Large Survey: The First Decade of Pan-STARRS Observations Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        This year marks the ten year anniversary of first light of the Pan-STARRS1 (PS1) telescope and 1.4 gigapixel GPC1 camera. In the subsequent decade, the Pan-STARRS Image Processing Pipeline (IPP) has continually evolved, culminating in our first data release of approximately three billion sources and one million image stacks at the end of 2016. In this presentation, I will talk about some of the challenges and solutions we have encountered in developing the current version of the IPP. This code base has enabled us to conduct a full reprocessing of the PS1 archive without impacting our ongoing asteroid search program. I will also discuss the additional data products in our future DR2 release.
        Orateur: Dr Christopher Waters (Pan-STARRS)
        Slides
        Video
      • 41
        CosmoHub demonstration [spontaneous contribution] Amphitheatre

        Amphitheatre

        IN2P3 computing center

        21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
        Orateur: Santiago Serrano (Universidad Autónoma de Barcelona)
        Slides
        Video
    • 42
      Workshop Wrap up Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
      Orateurs: Fabio Hernandez (CC-IN2P3), Prof. Réza Ansari (LAL)
      Slides
    • Data center visit: machine rooms Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E

      Guided visit to the machine rooms in the data center.

      Président de session: Fabio Hernandez (CC-IN2P3)
    • 43
      Logistics announcements Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E
      Orateurs: M. Bertrand Rigaud (USR6402), Fabio Hernandez (CC-IN2P3)
      Slides
    • Hackathon 202

      202

      IN2P3 computing center

      • 44
        What the LSST software stack can do and how to use it 202

        202

        IN2P3 computing center

        In this session, the speaker will present and demonstrate the current capabilities of the LSST software stack and its perspectives. Both the command line interface (CLI) and the application programming interface (API) will be addressed. A Python notebook-based platform is setup so that participants who want to try the software with a predefined dataset can do so. Participants are expected to bring their own personal computer configured with an operational recent web browser.
        Orateur: Jim Bosch (Princeton University)
        Slides
      • 45
        Hackathon time 202

        202

        IN2P3 computing center

      • 12:35
        Lunch break 2nd floor (Restaurant DOMUS)

        2nd floor

        Restaurant DOMUS

      • 46
        Hackathon time (cont.) 202

        202

        IN2P3 computing center

        The speakers will share their experience over several months of processing CFHT data at CC-IN2P3 using the LSST software stack.
    • Data center visit: machine rooms Amphitheatre

      Amphitheatre

      IN2P3 computing center

      21 Avenue Pierre de Coubertin CS70202 69627 VILLEURBANNE Cedex FRANCE Lat: 45°46'57.8"N Lon: 4°51'54.9"E

      Guided visit to the machine rooms in the data center.

      Président de session: Fabio Hernandez (CC-IN2P3)