LSST school & workshop: Getting ready to do science with LSST data

12 juin 2017, 08:15 → 16 juin 2017, 18:15 Europe/Paris

Amphitheatre (IN2P3 computing center)

Fabio Hernandez (CC-IN2P3), Réza Ansari (LAL)

Overview

This event was organized as 3-days workshop devoted to discuss data processing challenges in large imaging surveys. It was preceded by a 1-day school and followed by 1-day hackathon. The whole event was held in the same location, with no parallel sessions.

Schedule:

Monday, June 12th: astronomical image processing school
Tuesday to Thursday, June 13th-15th: workshop
Friday, June 16th: hackathon

Audience

The workshop was intended for astronomers and scientists developing astronomical image processing algorithms and software, people in charge of the data processing and data management for large surveys, as well as computer scientists involved in these projects.

Experts from the LSST Data Management and DESC science working groups, as well as from other large photometric surveys such as Euclid and DES also participated.

The school was intended for young scientists, PhD students, postdocs and senior scientists not familiar with astronomical image processing. Introductory lectures will cover basics of instrument response correction, photometry, astrometry and calibration.

The hackathon was organised around the use of the LSST science pipelines (a.k.a. the stack) and associated tools.

The event was well attended by a diverse community.

 

Workshop Programme

• Pushing the frontiers in astrometry and photometry
• Large photometric surveys
• LSST software stack beyond LSST
• Computing infrastructure, data management and data bases for imaging surveys

Please refer to the Scientific Programme section for more details on the programme.

Participants willing to present their work related to one of the main subjects of the workshop 
listed above were encouraged to submit a contribution. Details available in the Call for Abstracts section.

 

Sponsors

This event is sponsored by the LSSTC Enabling Science Committee and by the institutions below:

Annotated map of the area around CC-IN2P3 https://drive.google.com/open?id=14FrY1NIJXIZ7BUSL2UCZ6i6iYP8&usp=sharing

Directions to CC-IN2P3 plan_CCIN2P3_EN.pdf

lundi 12 juin

08:15 → 09:00 Registration: School

09:00 → 17:30 School

09:00 Welcome and logistics

Orateurs: Fabio Hernandez (CC-IN2P3), Prof. Réza Ansari (LAL)

Slides 2017-06-12-lyon2017-1-Mon.pdf jupyterhub_platform.pdf

09:05 Point sources and astrometry

Orateur: M. Pierre Astier (LPNHE)

Transparents course.pdf

Video https://webcast.in2p3.fr/videos-point_sources_and_astrometry

10:30 Coffee and tea break

11:00 Point sources and astrometry (cont.)

Orateur: M. Pierre Astier (LPNHE)

Video https://webcast.in2p3.fr/videos-point_sources_and_astrometry_cont

11:30 Extended sources and galaxy photometry

Orateur: Robert Lupton (Princeton University)

Slides lyon-school.pdf

Video https://webcast.in2p3.fr/videos-extended_sources_and_galaxy_photometry

12:30 Lunch break

14:00 Extended sources and galaxy photometry (cont.)

Video https://webcast.in2p3.fr/videos-extended_sources_and_galaxy_photometry_cont

15:00 Image differencing

Orateur: M. Dominique Fouchez (CPPM)

Slides

• lecture_imdiff.pdf
• school1.py
• school2.py
• startsim.txt
• startsub.txt

Video https://webcast.in2p3.fr/videos-image_differencing

15:30 Coffee and tea break

16:00 Image differencing (cont.)

Orateur: M. Dominique Fouchez (CPPM)

Video https://webcast.in2p3.fr/videos-image_differencing_cont

17:45 → 18:45 Data center visit: machine rooms

Guided visit to the machine rooms in the data center.

Président de session: Fabio Hernandez (CC-IN2P3)

mardi 13 juin

08:15 → 09:00 Registration: Workshop

09:00 → 12:15 Workshop: Pushing frontiers in astrometry and photometry (1)

Président de session: Prof. Réza Ansari (LAL)

09:00 Welcome and logistics

Orateurs: Fabio Hernandez (CC-IN2P3), Prof. Réza Ansari (LAL)

Slides 2017-06-12-lyon2017-2-Tue.pdf lsst_workshop_welcome_ansari.pdf

09:20 Point Spread Function modelling and measurement

Accurate knowledge of the Point Spread Function (PSF) is crucial to meet the science goals of the wide area surveys of LSST and other telescopes. Increasing statistical precision in surveys leads to a requirement for increased accuracy in our knowledge of the PSF, which varies across images, with time and with the spectrum of the objects being observed – it is a challenging but interesting problem! The most demanding application is for enabling PSF correction in weak lensing surveys, and an approach for understanding the required accuracy is discussed, together with approaches for modelling the various physical effects that contribute to the overall PSF.

Orateur: Prof. Lance Miller (Oxford University)

Transparents PSF_measurement.pdf

Video https://webcast.in2p3.fr/videos-psf_parameter_estimation

10:00 Efficient multi-band deblending

LSST and similarly deep surveys have to deal with ~40% of all objects being blended with others. The classical deblending approaches are often insufficient for high-quality measurements of galaxy properties. Their failures are related to simplifying assumptions about the galaxy shape or massive parameter degeneracies for complex galaxy models, and the reliance on single-band imaging. I will present a new algorithm (Melchior et al., in prep.) that is natively represented in multi-band data, i.e. it considers the SED the fundamental unit of measurement instead of per-band fluxes, and constrains the galaxy morphologies in a non-parametric fashion to avoid unstable results and model biases. The resulting constrained optimization problem is efficiently solved with a novel deterministic proximal algorithm (Moolekamp & Melchior, in prep.) with guaranteed convergence. I will show results from simulations and deep HSC imaging, and discuss the substantial gains from combining ground- and space-based imaging.

Orateur: M. Peter Melchior (Princeton University)

Transparents Deblender.pdf

Video https://webcast.in2p3.fr/videos-efficient_multiband_deblending

10:40 Coffee and tea break

11:10 LSST DM Processing of Crowded Fields

LSST Data Management has committed to processing crowded stellar fields through our image differencing pipelines, but is not mandated to perform direct image photometry in these regions. Exactly what this means in terms of algorithms and the scientific quality of our catalogs is subtle and in some ways uncertain, and in this talk I'll explain the algorithms we're planning to run, attempt to make some guesses about the scientific quality of the outputs, and provide some information on how interested scientists can follow our progress and contribute tests and possibly algorithmic code.

Orateur: Jim Bosch (Princeton University)

Slides DM_Plans_for_Crowded_Stellar_Fields.pdf

Video https://webcast.in2p3.fr/videos-lsst_dm_processing_of_crowded_fields

11:50 Improved Point Source Detection in Crowded Fields using Probabilistic Cataloging

Cataloging is challenging in crowded fields because sources are extremely covariant with their neighbors and blending makes even the number of sources ambiguous. We present the first optical probabilistic catalog, cataloging a crowded (~0.1 sources per pixel brighter than 22nd magnitude in F606W) Sloan Digital Sky Survey r band image from M2. Probabilistic cataloging returns an ensemble of catalogs inferred from the image and thus can capture source-source covariance and deblending ambiguities. By comparing to a traditional catalog of the same image and a Hubble Space Telescope catalog of the same region, we show that our catalog ensemble better recovers sources from the image. It goes more than a magnitude deeper than the traditional catalog while having a lower false discovery rate brighter than 20th magnitude. We also present an algorithm for reducing this catalog ensemble to a condensed catalog that is similar to a traditional catalog, except it explicitly marginalizes over source-source covariances and nuisance parameters. We show that this condensed catalog has a similar completeness and false discovery rate to the catalog ensemble. Future telescopes will be more sensitive, and thus more of their images will be crowded. Probabilistic cataloging performs better than existing software in crowded fields and so should be considered when creating photometric pipelines in the Large Synoptic Survey Telescope era.

Orateur: M. Stephen Portillo (Harvard-Smithsonian Center for Astrophysics)

Transparents

• catalogue_ensemble.avi
• catalogue_ensemble.mp4
• LSST_Lyon.pdf
• LSST_Lyon.pptx

Video https://webcast.in2p3.fr/videos-improved_point_source_detection_in_crowded_fields_using_probabilistic_cataloging

12:15 → 14:00 Lunch break

14:00 → 18:00 Workshop: LSST software stack beyond LSST

Président de session: M. Pierre Astier (LPNHE)

14:00 Hyper Suprime-Cam Subaru Strategic Program

HSC-SSP is a 3-layered imaging survey aimed at addressing major astrophysical questions such as the nature of dark matter and dark energy. The HSC data have been processed with a version of the LSST stack and a number of scientific papers have been published. I will give an overview of the survey and its fist public data release.

Orateur: Dr Masayuki Tanaka (NAOJ)

Slides Tanaka.pdf

Video https://webcast.in2p3.fr/videos-hyper_suprimecam_subaru_strategic_program

14:45 HSC data meets LSST code

The HSC is an 800 MPixel imager on the 8.2m Subaru telescope, carrying out a 1400 deg^2 week lensing survey. In many ways this survey is an LSST precursor, and we are reducing the data using prototype LSST pipelines. I shall discuss the state of the pipelines that we are using for the next data release, paying especial attention to algorithms that we know will need to be refined before we can meet LSST requirements.

Orateur: Robert Lupton (Princeton University)

Slides lupton.pdf

Video https://webcast.in2p3.fr/videos-hsc_data_meets_lsst_code

15:30 Coffee and tea break

16:00 Photometry extraction and validation using the HSC pipeline of HSC+u-band data in HSC-SSP deep fields

We obtained 300 hours at CFHT to conduct a u-band follow-up (CLAUDS program, u<27) of the HSC-SSP Deep fields (grizY, r<27), over 20 deg2. The u band is primarily used to find drop-out (high redshift) galaxies and compute photometric redshifts. Therefore it is necessary to combine the two datasets in a fully consistent way. To do this we developed tools to import the CFHT stack images into the HSC pipeline framework and ran the photometry extraction using the HSC pipeline tools. In this talk I will explain how we did it and I will show some validation results.

Orateur: Dr Jean Coupon (University of Geneva)

Transparents LSST_workshop_Lyon2017.pdf

Video https://webcast.in2p3.fr/videos-photometry_extraction_and_validation_using_the_hsc_pipeline_of_hsc_u_band_data_in_hscssp_deep_fields

16:40 Reprocessing CFHT data with the LSST DM software stack

Orateur: Dominique Boutigny (LAPP)

Slides LSST@Lyon-2017.pdf

Video https://webcast.in2p3.fr/videos-reprocessing_cfht_data_with_the_lsst_dm_software_stack

17:20 Reprocessing CFHT Deep fields with the stack

The talk will summarize the attempts made to reprocess the CFHT Deep fields with the stack. The results obtained as well as the future work will be exposed.

Orateur: Philippe Gris (LPC Clermont-Ferrand)

Transparents Lyon_2017.pdf

Video https://webcast.in2p3.fr/videos-reprocessing_cfht_deep_fields_with_the_stack

18:15 → 19:00 Data center visit: machine rooms

Guided visit to the machine rooms in the data center.

Président de session: Fabio Hernandez (CC-IN2P3)

mercredi 14 juin

08:55 → 09:00 Logistics announcements

Orateur: Fabio Hernandez (CC-IN2P3)

Slides 2017-06-12-lyon2017-3-Wed.pdf

09:00 → 11:30 Workshop: Pushing frontiers in astrometry and photometry (2)

Président de session: Robert LUPTON (Princeton University)

09:00 Joint astrometry

In order to optimally use multiple images of the same sky area, one has to accurately map the coordinates of these image onto a common coordinate system. Commonly,the images we get are significantly deeper than reference catalogs, and hence the coordinate mappings benefit from making use of common objects in the images, even if they do not appear in reference catalogs. I will describe some technicalities of this joint fit problem, the features of some implementations, and propose some goals for the LSST implementation.

Orateur: M. Pierre Astier (LPNHE)

Transparents joint-astrometry.pdf

Video https://webcast.in2p3.fr/videos-joint_astrometry

09:30 Creating DCR-matched templates for image differencing

The frequency-dependent spectral index of the atmosphere leads to spectrum-dependent distortions of sources away from zenith. While the effect of bulk refraction is easily corrected during astrometric calibration, this Differential Chromatic Refraction (DCR) introduces distortions that depend on the intrinsic spectrum of a source and vary with the parallactic angle and airmass of the observation. Past surveys have avoided DCR by restricting observations to low airmass, but LSST operations will require observations up to an airmass of 1.4 across the sky and up to 2 in some fields. With current image differencing techniques, we would have to choose between a high rate of false positive detections that would have to be culled with machine learning, or restrict template images to have similar airmasses and parallactic angles of the science image. However, the refraction of the atmosphere has been modeled for decades with increasing refinement, and we can use iterative forward modeling to estimate a sky model using that knowledge of the atmosphere to fit all observations of a patch simultaneously. We use this model to calculate new DCR-matched templates for image differencing, and find that it reduces the number of spurious source detections in DECam g-band images.

Orateur: Ian Sullivan (University of Washington)

Slides Sullivan_templates_Lyon_2017.pdf

Video https://webcast.in2p3.fr/videos-creating_dcr_matched_templates_for_image_differencing

10:00 Advances in astronomical image processing - Solving the problems of image coaddition and image subtraction

While co-addition and subtraction of astronomical images stand at the heart of observational astronomy, the existing solutions for them lack rigorous argumentation, are not achieving maximal sensitivity and are often slow. Moreover, there is no widespread agreement on how they should be done, and often different methods are used for different scientific applications. I am going to present rigorous solutions to these problems, deriving them from the most basic statistical principles. These solutions are proved optimal, under well defined and practically acceptable assumptions, and in many cases improve substantially the sensitivity, robustness and speed of both coaddition and subtraction. For coaddition, I will present a coadd image that is: a) sufficient for any further statistical decision or measurement on the underlying constant sky, making the entire data set redundant. b) improves both survey speed (by 5-20%) and effective spatial resolution of past and future astronomical surveys. c) improves substantially imaging through turbulence applications. d) much faster than many of the currently used coaddition solutions. For subtraction, I will present a subtraction image that is: a) optimal for transient detection under the assumption of spatially uniform noise. b) sufficient for any further statistical decision on the differences between the images, including the identification of cosmic rays and other image artifacts. c) Free of subtraction artifacts, potentially allowing (for the first time) robust automatic transient detection in real time, opening new avenues for scientific exploration. d) more sensitive than previous transient detection methods e) orders of magnitude faster than past subtraction methods.

Orateur: M. Barak Zackay (Weizmann Institute of Science)

Slides Talk_preparing_for_science_with_lsst_lyon.pdf

Video https://webcast.in2p3.fr/videos-advances_in_astronomical_image_processing__solving_the_problems_of_image_coaddition_and_image_subtraction

10:30 Coffee and tea break

Room adjacent to front desk

11:00 LSST DM stack image differencing on CFHT images

In an introduction part, I will present the basics of the implementation of image differencing with the LSST DM stack. Then I will present some preliminary results obtained by running this software on a subset of the CFHTLS Deep Fields images and some comparisons using the published supernova data of the SNLS collaboration.

Orateur: M. Dominique Fouchez (CPPM)

Slides stackimdiff.pdf

Video https://webcast.in2p3.fr/videos-lsst_dm_stack_image_differencing_on_cfht_images

11:30 → 12:10 Workshop: Large photometric surveys (1)

Président de session: Robert LUPTON (Princeton University)

11:30 Gaia, an overview

The Gaia mission, currently in operational phase, will be described, from the current status, the organisation of the data processing, to the publication of the catalogues.

Orateur: Dr Frédéric Arenou (CNRS/GEPI, Observatoire de Paris)

Transparents LSST-Gaia.pdf

12:10 → 12:30 Group photo

Please go to the front desk for a group photo on the stairs at the main entrance of the building.

12:30 → 14:00 Lunch break

Buffet in the room adjacent to the front desk

14:00 → 16:40 Workshop: Pushing frontiers in astrometry and photometry (3)

Président de session: Prof. Mario Juric (University of Washington)

14:00 Dark Energy Survey - Status, Science, and Algorithmic Advancements

I will review the status of the Dark Energy survey, give an outlook on the upcoming results from its weak-lensing program, and describe several of the algorithmic advancements that enable high-precision measurements from ground-based imaging.

Orateur: M. Peter Melchior (Princeton University)

Transparents DES.pdf

Video https://webcast.in2p3.fr/videos-dark_energy_survey__status_science_and_algorithmic_advancements

14:35 Forward Global Calibration of the Dark Energy Survey [remote presentation]

Many scientific goals for DES require calibration of broadband grizY photometry that is stable in time and uniform over the sky to better than 1%. It is also necessary to limit systematic uncertainty in the calibrated broadband magnitudes due to the spectrum of the source. I present details on the Forward Global Calibration Method (FGCM), which combines data taken with auxiliary instrumentation with data from the survey imaging itself and models of the instrument and atmosphere to estimate the spatial- and time-dependence of the passbands of individual DES exposures. The passband of individual observations is combined with the source spectral shape to compute chromatic corrections to the standard system which is necessary to achieve sub-percent calibrations.

Orateur: Eli Rykoff (SLAC National Accelerator Laboratory)

Slides rykoff_fgcm_lyon_2017.pdf

Video https://webcast.in2p3.fr/videos-forward_global_calibration_of_the_dark_energy_survey

15:10 Photometric Calibration: lessons from CFHTLS

For about a decade, photometric calibration has been identified as a potential limitation of current and future imaging surveys. Uncertainties on the relative (band to band) flux scale are currently the dominant contribution to the systematic error budget affecting SNIa distances. Non-uniformities of the imager response may introduce spurious correlations in the LSS analyses, via the photometric redshifts. In this talk, I review the lessons learnt from the analysis of the CFHTLS survey, and I discuss related topics of the LSST-DESC roadmap.

Orateur: Nicolas Regnault (LPNHE)

Transparents regnault.pdf

Video https://webcast.in2p3.fr/videos-photometric_calibration_lessons_from_cfhtls

15:45 Coffee and tea break

Room adjacent to the front desk

16:15 PHOTOMETRYPIPELINE: An automated pipeline for calibrated photometry

PHOTOMETRYPIPELINE (PP) is an automated pipeline that produces calibrated photometry from imaging data through image registration, aperture photometry, photometric calibration, and target identification with only minimal human interaction. PP utilizes the widely used Source Extractor software for source identification and aperture photometry; SCAMP is used for image registration. Both image registration and photometric calibration are based on matching field stars with star catalogs, requiring catalog coverage of the respective field. A number of different astrometric and photometric catalogs can be queried online. Relying on a sufficient number of background stars for image registration and photometric calibration, PP is well-suited to analyze data from small to medium-sized telescopes. Calibrated magnitudes obtained by PP are typically accurate within <0.03 mag and astrometric accuracies are of the order of 0.3 arcsec relative to the catalogs used in the registration. The pipeline consists of an open-source software suite written in Python, can be run on Unix-based systems on a simple desktop machine, and is capable of realtime data analysis. PP has been developed for observations of moving targets, but can be used for analyzing point source observations of any kind. I will present the pipeline's implementation and discuss some current and future features.

Orateur: Dr Michael Mommert (Northern Arizona University)

Slides mommert.pdf

16:40 → 17:00 HSC demonstration

Orateur: Dr Masayuki Tanaka (NAOJ)

17:00 → 18:00 Data center visit: machine room

Guided visit to the machine rooms in the data center.

Président de session: Fabio Hernandez (CC-IN2P3)

20:00 → 22:00 Social Event

Restaurant
Pléthore et Balthazar
72 rue Mercière
69002 LYON

Google maps https://www.google.com/maps/place/Pléthore+%26+Balthazar/@45.7617812,4.8307153,17z/data=!4m5!3m4!1s0x47f4ea55b72d2423:0x153429cb468b5afc!8m2!3d45.7617775!4d4.832904

20:00 Social Event

Annotated map Restaurant_Annotated.png

Google maps https://www.google.com/maps/place/Pléthore+%26+Balthazar/@45.7617812,4.8307153,17z/data=!4m5!3m4!1s0x47f4ea55b72d2423:0x153429cb468b5afc!8m2!3d45.7617775!4d4.832904

jeudi 15 juin

08:55 → 09:00 Logistics announcements

Orateur: Fabio Hernandez (CC-IN2P3)

Slides 2017-06-12-lyon2017-4-Thu.pdf

09:00 → 12:50 Workshop: Computing infrastructure and data management

Président de session: Dominique Boutigny (LAPP)

09:00 LSST Data Management Overview

In this brief talk we will outline the current Data Management design and approach to handling LSST Data. DM is a multifaceted distributed system. It is also built in a distributed manner so we shall also cover the organization of the DM development work.

Orateur: Dr William O'Mullane (AURA/LSST)

Slides WOM_DM_LYON_2017.pdf

Video https://webcast.in2p3.fr/videos-lsst_data_management_overview

09:30 What LSST Will Deliver: Images, Catalogs, Alerts, Services

The LSST is an integrated survey system. The observatory, telescope, camera and the data management systems will be built to conduct the LSST survey and will not support the 'PI mode' in the classical sense. Instead, the ultimate, science-enabling, deliverable of LSST will be the fully reduced data products and accompanying services. In this talk, we will present the baseline design and contents of LSST data products. "Level 1" data products will be generated continuously every observing night and include measurements such as alerts to objects that have changed brightness or position. "Level 2" data products will be made available as annual Data Releases and will include images and measurements of quantities such as positions, fluxes, and shapes, as well as variability information such as orbital parameters for moving objects and an appropriate compact description of light curves. These data will be made available through the LSST Science Platform -- a set of integrated services offered in LSST Data Access Centers. The Platform will enable efficient data access and querying, as well as execution of custom analyses and generation of added value data products. These will enable science beyond the capabilities of Level 1 and 2 deliverables (so called "Level 3").

Orateur: Prof. Mario Juric (University of Washington)

Transparents LSST-Science-Platform-Lyon2017-4.pptx

Video https://webcast.in2p3.fr/videos-what_lsst_will_deliver_images_catalogs_alerts_services

10:10 Qserv: A distributed shared-nothing database for the LSST catalog

To satisfy the need to efficiently store, query, and analyze catalogs running into trillions of rows and petabytes of data, we are developing Qserv, a distributed shared-nothing SQL database query system. We describe Qserv's design, architecture, and ability to scale to LSST's data requirements. We illustrate its potential with some current test results.

Orateur: Fritz Mueller (SLAC)

Slides Qserv-Lyon.pdf

Video https://webcast.in2p3.fr/videos-qserv_a_distributed_shared_nothing_database_for_the_lsst_catalog

10:40 Coffee and tea break

Room adjacent to front desk

11:10 Scaling cloud for LSST catalog at IN2P3

French engineers and researchers are very interested in the challenge of the LSST catalog and contribute in particular to validating Qserv and preparing its production phase. In this context, they perform integration tests, based on CFHT data, and ranging from the pixels to the execution of scientific SQL queries. In addition, they design Qserv large scale deployment and management procedure using a methodology for building modern, scalable, maintainable software-as-a-service apps. They also work closely with LSST Square team to provide continuous delivery, and automate both live distributed testing and release distribution. All these activities will be detailed during this presentation.

Orateur: Fabrice Jammes (CNRS)

Transparents Scaling_Cloud_for_LSST_catalog_at_IN2P3.pdf

Video https://webcast.in2p3.fr/videos-scaling_cloud_for_lsst_catalog_at_in2p3

11:40 LHC experiments : Petabytes to papers

The LHC experiments are now routinely collecting (also simulating) and analyzing Petabytes of data each year, publishing 200 papers, signed each by 3000 people. Focussing on the ATLAS experiment, the talk will describe the organisation of the data processing that makes it possible: what balance between accuracy (of software and calibrations) and stability was reached, allowing specialized treatment of data for specific papers despite limited resources. The talk will conclude on the various challenges posed by eventual 10 fold increase of LHC luminosity and 10 fold increase of data taking rate.

Orateur: Dr David Rousseau (LAL-Orsay)

Transparents tr170615_DavidRousseau_LHC_computing_LSSTLyon_v1.pptx.pdf

Video https://webcast.in2p3.fr/videos-lhc_experiments_petabytes_to_papers

12:20 Exploring Spark and MongoDb for LSST

Spark is a very promising technology offering distributed data and computing mechanisms. At LAL(Orsay) we have started to look at how the typical computing workflows used in LSST could use the Spark eco-system: How to distribute algorithms in a map-reduce approach How to format various data structures to partition them in a distributed file system Thus, a OpenStack based cluster has been configured at LAL with Spark and its various associated components, and several models are experienced to evaluate the performance and configuration parameters (memory, CPU, …) In the same context, in the process in exploring various technologies related with QServ or the catalog access techniques, we are working on two promissing technologies: MongoDB and Spark DataFrames, both offering a natural data or processing distribution approach. The method is similar for both: we exploit one limited dataset (2To) (sources and objects) and try and apply the benchmarking queries that used to be applied to QServ. The concepts, the ingestion, and the querying methods are explored, in particular looking at possible functional or performance limitations for both systems. Several platforms are used for this study: The Galactica cluster at Clermont (Petasky context) OpenStack at LAL (VirtualData context) A test cluster CCIN2P3.

Orateur: M. Christian Arnault (CNRS)

Transparents 20170615_SparkMongoDb.pptx

Video https://webcast.in2p3.fr/videos-exploring_spark_and_mongodb_for_lsst

12:50 → 14:00 Lunch break

14:00 → 17:10 Workshop: Large photometric surveys (2)

Président de session: Prof. Réza Ansari (LAL)

14:00 Euclid - An update on the mission and its processing plan

The preparation of the Euclid mission is progressing, with the instruments entering the production of the Qualification models, and the data processing ground segment in its design review. I will make a brief update of where the mission stands today, and focus on the plans that are made for operation and data processing. Details will also be given on how Euclid relies on external collaborations for input data, and on the planned public data releases and their content.

Orateur: Dr Marc Sauvage (CEA/DRF/Irfu/SAp)

Transparents Euclid-LSST-June2017.pdf

Video https://webcast.in2p3.fr/videos-euclid_an_update_on_the_mission_and_its_processing_plan

14:40 The Dark Energy Survey Data Management System [remote presentation]

The Dark Energy Survey covering 5,000 square degrees of the South Galactic Cap along with a dedicated 30 square degree SN survey has now completed four years of observations. The Data Management group (DESDM) has been responsible for the processing, release, and curation of these data products for the DES collaboration and to make them available to the general astronomical community. I will give a brief overview of the DESDM pipelines and data management system, focusing on their evolution and performance, with an emphasis toward providing lessons that might help inform the current development effort of the LSST data management project.

Orateur: Robert Gruendl (University of Illinois)

Slides Lyon2017_gruendl.pdf

Video https://webcast.in2p3.fr/videos-the_dark_energy_survey_data_management_system

15:20 Coffee and tea break

Room adjacent to the front desk

15:50 Weak Lensing Data processing for the CFHTLenS and KiDS surveys [remote presentation]

I will discuss the data processing and data handling of the CFHTLenS and KiDS collaborations. Special emphasis will be given on our algorithms and processing methods for weak gravitational lensing studies. I will summarise the current status of our image processing pipeline THELI which is optimised to produce lensing quality data from current and future optical Wide-Field Imaging Surveys. Our software suite encompasses all necessary modules from raw images to multi-band object catalogues with photometric redshifts and galaxy shapes. The talk will be complemented with a summary of the lessons we learnt in the past 15 years to successfully implement and execute an optical Wide-Field Imaging survey.

Orateur: Thomas Erben (Universität Bonn, Argelander-Institut für Astronomie)

Transparents Erben_LSST_2017.pdf

Video https://webcast.in2p3.fr/videos-weak_lensing_data_processing_for_the_cfhtlens_and_kids_surveys

16:20 Lessons from a Large Survey: The First Decade of Pan-STARRS Observations

This year marks the ten year anniversary of first light of the Pan-STARRS1 (PS1) telescope and 1.4 gigapixel GPC1 camera. In the subsequent decade, the Pan-STARRS Image Processing Pipeline (IPP) has continually evolved, culminating in our first data release of approximately three billion sources and one million image stacks at the end of 2016. In this presentation, I will talk about some of the challenges and solutions we have encountered in developing the current version of the IPP. This code base has enabled us to conduct a full reprocessing of the PS1 archive without impacting our ongoing asteroid search program. I will also discuss the additional data products in our future DR2 release.

Orateur: Dr Christopher Waters (Pan-STARRS)

Slides waters-lyon2017.pdf

Video https://webcast.in2p3.fr/videos-lessons_from_a_large_survey_the_first_decade_of_pan_starrs_observations

17:00 CosmoHub demonstration [spontaneous contribution]

Orateur: Santiago Serrano (Universidad Autónoma de Barcelona)

Slides CosmoHub_short.pdf

Video DemoCosmoHub.mp4 https://webcast.in2p3.fr/videos-cosmohub_demonstration_spontaneous_contribution

17:10 → 17:20 Workshop Wrap up

Orateurs: Fabio Hernandez (CC-IN2P3), Prof. Réza Ansari (LAL)

Slides Thanks.pdf

17:30 → 18:30 Data center visit: machine rooms

Guided visit to the machine rooms in the data center.

Président de session: Fabio Hernandez (CC-IN2P3)

vendredi 16 juin

09:00 → 09:05 Logistics announcements

Orateurs: M. Bertrand Rigaud (USR6402), Fabio Hernandez (CC-IN2P3)

Slides 2017-06-12-lyon2017-5-Fri.pdf jupyterhub_platform.pdf

09:05 → 17:00 Hackathon

09:10 What the LSST software stack can do and how to use it

In this session, the speaker will present and demonstrate the current capabilities of the LSST software stack and its perspectives. Both the command line interface (CLI) and the application programming interface (API) will be addressed. A Python notebook-based platform is setup so that participants who want to try the software with a predefined dataset can do so. Participants are expected to bring their own personal computer configured with an operational recent web browser.

Orateur: Jim Bosch (Princeton University)

Slides DM-Stack-Tutorial.ipynb DM-Stack-Tutorial-Updated.ipynb

11:05 Hackathon time

12:35 Lunch break

14:05 Hackathon time (cont.)

The speakers will share their experience over several months of processing CFHT data at CC-IN2P3 using the LSST software stack.

17:15 → 18:15 Data center visit: machine rooms

Guided visit to the machine rooms in the data center.

Président de session: Fabio Hernandez (CC-IN2P3)