Artificial Intelligence and the Uncertainty challenge in Fundamental Physics

27 nov. 2023, 09:00 → 1 déc. 2023, 17:00 Europe/Paris

The workshop is organised by CNRS AISSAI and CNRS IN2P3.

The integration of Artificial Intelligence (AI) into the realm of fundamental science is witnessing an unprecedented surge. However, there are specific challenges to be overcome: 

• any measurement or prediction has to be provided with a precise confidence interval
• measurements rely on numerous inputs, each with inherent uncertainties and potential inter-correlations 
• the trust in the measurement has to be communicated to peers
• an experiment complexity is between the breakout game and autonomous car: detailed, expensive, imperfect simulators exist
• each experimental device is a complex unique device, producing semi-structured data
• expensive data in the petabytes range

 

These challenges have been overcome in the past for extraordinarily complex measurements where the role of AI was not major (e.g. Higgs boson or gravitational wave discoveries). Also, Fundamental Science has decades-old big data culture.

 

This workshop aims to bring together experts from Fundamental Science, Computer Science and Statistics to exchange around the theme of Uncertainties, which has been broken down into the following themes: 

• Monday PM : Opening session, Uncertainty Quantification, uncertainty prediction
• Tuesday AM : Explainable AI, trustworthy AI
• end of Tues AM, Tuesday PM : Simulation-Based Inference
• Wednesday AM : Data frugal approaches, Data-centric AI ; Benchmarks dataset and challenges
• Wednesday PM : Fair Universe hackathon 
• Thu AM Unfolding (or de-biasing, de-blurring) 
• End of Thu AM, Thu PM Controlling uncertainties in generative models
• Fri AM Architectures (Adversarial, Bayesian, ... )
• Fri PM Closing session

 

 

Each theme will see introductory talks by leading experts, followed by contributed talks. 

 

Confirmed speakers list includes : Luca Biferale (U Roma 2), Mathias Backes (Kirchhof Institute), Sebastian Bieringer (U Hambourg), Jérôme Bobin (CEA-Saclay), Vincent Chabridon (EDF), Yuan Tuan Chou (U Washington), Vince Croft (NIKHEF-Amsterdam), Tommaso Dorigo (U Padova), Eva Govorkova (MIT), Marylou Gabrié (Ecole Polytechnique), Julien Girard-Satabin (CEA), Nathan Huetsch (U Heidelberg), Eiji Kawasaki (CEA), Gregor Kasiezcka (U Hamburg), Sabine Kraml (LPSC-Grenoble), Mikael  Kuusela (Carnegie Mellon), Alessandro Leite (LISN), Gilles Louppe (U Liège), Mark Neubauer (U Illinois), Harrison Prosper (Florida State U), Radi Radev (CERN), Pedro Rodrigues (INRIA-Grenoble), Ozgur Sahin (CEA-Saclay), Simone Scardapane (U Roma), Sofia Schweitzer (U Heidelberg), Laurens Sluyterman (U Radboud), Gaël Varoquaux (INRIA-Saclay), Emmanuel Vazquez (L2S-Saclay), Wouter Verkerke (U Amsterdam), Gordon Watts (U Washington), Christoph Weniger (U Amsterdam), Philip Windischhofer (U Chicago), Burak Yelmen (U Tartu) 

and most of the Fair Universe team : Yuan-Tang Chou (University of Washington), Wahid Bhimji (NERSC-Berkeley), Ragansu Chakkappai (IJCLab-Orsay), Yuan-Tang Chou (University of Washington), Sascha Diefenbacher (LBNL-Berkeley), Steven Farrell (NERSC-Berkeley), Elham Khoda (U Washington), David Rousseau (IJClab-Orsay), Ihsan Ullah (ChaLearn)

 

 

Also a special half-day hackathon on the Fair-Universe prototype challenge on Wednesday afternoon. The goal? Craft ML algorithms that can derive measurements with uncertainties from simulated LHC proton collisions, ensuring robustness against input uncertainties.

 

 

Registration and call for contribution

 

Registration is free (but mandatory). Registration for on-site participation (including  monday cocktail and thursday workshop dinner) remains open until the last few available slots are distributed.

 

Remote full-time or part-time participation will be possible, although the focus will be on-site. The registration form allows to specify on-site participation with some granularity. The zoom link will be distributed to participants only.

 

 

The call for contributions is closed. Talks are expected to be given in person, with few exceptions.

 

Location

The workshop will take place from Monday 2PM till Tuesday evening in Paris at SCAI, Paris (on Jussieu campus) and Wednesday, Thursday, Friday 5PM at Institut Pascal Université Paris-Saclay. 

And finally

A companion anomaly detection workshop will take place in Clermont-Ferrand 4-7th March 2024

 

All slides and recordings are available in the detailed agenda (Timetable). For recordings, we advise to right click and save,  rather than play directly. 

slack join link (participants only)

zoom link (participants only)

Poster affiche-AISSAI-Rousseau-0923-w4k.jpg affiche-AISSAI-Rousseau-0923-w4K.png

lundi 27 novembre

13:00 → 14:00 Welcome coffee : le Patio

Coffee and badges will be available in le Patio, Jussieu, just outside Amphi 25, see the map.

14:00 → 18:00 Opening session, Uncertainty Quantification: Jussieu, Amphi 25

Présidents de session: Mark Neubauer (University of Illinois at Urbana-Champaign), Vincent Alexander Croft (LIACS)

14:00 Welcome

Orateur: David Rousseau (IJCLab, CNRS/IN2P3, Université Paris-Saclay)

ai_uncertainty_mon_1_rousseau.mp4

tr202311_David_Rousseau_AIUPHYS2023_intro.pdf

14:15 Uncertainty modeling in particle physics

I will present a pedagogical introduction to uncertainty modeling in particle physics. I will mostly focus on the methods used at the Large Hadron Collider experiments, where systematic effects are explicitly parameterized in the likelihood function in terms of nuisance parameters. Accurate modeling of systematic effects is of increasing importance at the LHC as the abundant data has decreased statistical uncertainties in many measurements to be on par with systematic uncertainties. I will discuss the reasoning behind the modeling approaches commonly chosen, common challenges in the parametric modeling and in the interpretation of the corresponding uncertainties. I will conclude with the special considerations in the modeling of theoretical uncertainties, which are often incompletely defined.

Orateur: Wouter Verkerke (Nikhef/UvA)

ai_uncertainty_mon_pm_2_verkerke.mp4

verkerke_ai_hep_uncertainties_v15.pdf

15:00 Uncertainty Quantification and Machine Learning in Industry: Current Practices and Challenges in Industrial Applications for Low-Carbon Electricity Production

This talk will focus on a panel of current practices and challenges regarding both Uncertainty Quantification (UQ) and Artificial Intelligence (mainly from the Machine Learning (ML) point of view), in EDF's industrial applications, especially in the topics of risk management of industrial production assets. From our point of view, these two core topics, UQ & AI are, today, closely related to one another, especially when targeted applications are critical industrial systems (such as, e.g., nuclear reactors, dams, or wind farms). After presenting the commonly accepted methodology for UQ (and its link with ML) in numerical simulation codes, this talk will try to provide a panel of motivating applications that EDF R&D is confronted with, together with the industrial problems they arise. Then, a variety of technical and scientific challenges will be derived from these applications to introduce several research tracks we developed and pursued in the last decade. In addition to these methodological contributions, an emphasis will be put on the various open-source tools and software that have been produced as byproducts of this long-term research endeavour. Finally, a few current open questions will be discussed at the end of the talk while opening the path to future research questions and applications.

Orateur: Vincent CHABRIDON (EDF R&D)

ai_uncertainty_mon_pm_3_chabridon.mp4

Talk_VChabridon_27112023.pdf

15:30 An Introduction to Bayesian Optimization

In this talk, we delve into the foundational principles of Bayesian optimization, a method particularly well-suited for optimizing deterministic or stochastic functions, whether scalar or vectorial, especially when the evaluation of the function is computationally expensive and no gradient information is available.

Bayesian optimization is particularly relevant in the domains of Design and Analysis of Computer Experiments (DACE) and Uncertainty Quantification (UQ). In these areas, it is typically applied to minimize costs or maximize performance through complex computer simulations, including those utilizing Partial Differential Equations and finite element methods. Additionally, Bayesian optimization has become increasingly popular in Machine Learning and Artificial Intelligence for optimizing parameters in learning procedures, highlighting its versatility and effectiveness across a range of fields.

At the core of Bayesian optimization is the practice of modeling the target function using the Gaussian process framework. This modeling approach enables the construction of a sampling criterion, also known as an acquisition function. We will examine various classic sampling criteria, including the widely-used Expected Improvement (EI) criterion, and discuss the concept of Stepwise Uncertainty Reduction (SUR). Our discussion will focus on how Bayesian optimization effectively manages the balance between exploration and exploitation.

Orateur: Emmanuel Vazquez (L2S -- Paris-Saclay)

ai_uncertainty_mon_pm_4_vazquez.mp4

Colab notebook

presentation.pdf

16:00 Coffee break in the Patio

just outside Amphi 25

16:30 Uncertainty Quantification in Neural Networks: Methods and Considerations

In this talk, we delve into the complexities of uncertainty quantification for neural networks. Model predictions inherently come with uncertainties that arise from several factors: stochastic outcomes, the randomness of training data samples, and the inherent variability of the training process itself. Through the lens of a regression problem, we will unpack these factors and provide a pragmatic framework to understand and quantify uncertainty effectively. Furthermore, we will discuss various considerations and pitfalls associated with using popular approaches such as Monte Carlo dropout and Deep Ensembles.

Orateur: Laurens Sluijterman (Radboud University)

AIUCFP 27-11-2023.key

AIUCFP 27-11-2023.pdf

ai_uncertainty_mon_pm_5_sluijterman.mp4

16:50 Fair Universe Challenge Paris 2023

The Fair Universe project is building a large-compute-scale AI ecosystem for sharing datasets, training large models and hosting challenges and benchmarks. Furthermore, the project is exploiting this ecosystem for an AI challenge series focused on minimizing the effects of systematic uncertainties in High-Energy Physics (HEP), and on predicting accurate confidence intervals. This talk will describe the challenge platform we have developed that builds on the open-source benchmark ecosystem Codabench to interface it to the NERSC HPC center and its Perlmutter system with over 7000 A100 GPUs. This presentation will also mark the launch of the first of our Fair Universe public challenges hosted on this platform, the Fair Universe: HiggsML Uncertainty Challenge. There will be a hackathon during the workshop to develop the current prototype challenge, the full version of which will run in 2024. The Codabench/NERSC platform allows for hosting challenges also from other communities, and we also intend to make our benchmark designs available as templates so similar efforts can be easily launched in other domains.

Orateur: Wahid Bhimji (Lawrence Berkeley National Laboratory)

ai_uncertainty_mon_pm_6_bhimji.mp4

Fair Universe 2023 - AIUCFP - Paris.pdf

Google Slides

17:20 Metrics for Uncertainty-Aware ML Methods

Machine learning methods have managed to provide significant improvements to data analysis in a multitude of scientific fields. However, as ML finds more and more applications in science, the challenge of quantifying machine learning uncertainties moves into the forefront. This is especially notable in High Energy Physics, where high-precision measurements require precise knowledge of uncertainties. Moreover, systematic uncertainties, such as detector effects and calibration factors, are common occurrences in HEP.

This leads to a requirement for methods and approaches that are not only accurate and precise in the presence of such, imperfectly understood systematic uncertainties, but can also provide accurate estimates of the uncertainty in their prediction. Several methods have been proposed for ML uncertainty quantification, however measuring and comparing the performance of these methods is highly non-trivial.

In this talk, we present several metrics for uncertainty quantification metrics, compare their distinct advantages, and benchmark them with example uncertainty quantification challenges.

Orateur: Sascha Diefenbacher (Lawrence Berkeley National Laboratory)

ai_uncertainty_mon_pm_7_diefenbacher.mp4

Diefenbacher_Uncertainty_Metrics.pdf

18:00 → 20:00 Welcome cocktail : le Patio

The welcome cocktail will take place at the Patio, Jussieu, just outside Amphi 25, see the map.

mardi 28 novembre

09:00 → 11:00 Explainable AI: Amphi Durand, batiment Esclangon

Président de session: Dr Thomas Vuillaume (LAPP, Univ. Savoie Mont-Blanc, CNRS)

09:00 xAI in practice: current state of the art, limitations and perspectives

Explainable Artificial Intelligence (xAI) is a vibrant research field that aims to provide an insight on the decision taken by machine learning (ML) programs. The pervasiveness of AI in our societes pushed regulations (as the European AI Act) that demands transparency. We will present an overview of the field of Explainable AI, focusing on local explanations. We will present the various caveat identified by the literature, and some of our perspectives on xAI.

Orateur: Julien Girard-Satabin (CEA)

ai_uncertainty_2_tues_am_1_girard-satabin.mp4

presentation-girard.pdf

09:45 Explainable AI for Interpretability of Deep Neural Networks : the High Energy Physics perspective

Explainable AI (xAI) represents a set of processes and methods that allows human users to comprehend results created by machine learning algorithms. In the context of applications of AI to science, we need to look beyond standard metrics of prediction performance such as accuracy to ensure that AI models are robust to noise and adversarial samples, fair to biases in data populations, and generate trustworthy explanations of their predictions. A challenge is that xAI is hard to define and even harder to evaluate. There is no universal definition of what it means for an AI model to be explainable nor well-defined metrics to evaluate the "goodness" of explanations generated for AI models. Despite of these challenges, current xAI tools and methods are powerful allies for physicists. They have great utility in aiding in the interpretation deep neural networks (DNNs) and this information can be used to create better algorithms.

In this talk, I will discuss these aspects of xAI and the application of xAI methods to DNN models used in jet tagging. In our case study of jets coming from top quark decay in the high energy proton-proton collisions at the Large Hadron Collider, we use XAI to help identify which features play the most important roles in identifying the top jets, how and why feature importance varies across different XAI metrics, and how latent space representations encode information as well as correlate with physical quantities. We additionally illustrate the activity of hidden layers as Neural Activation Pattern (NAP) diagrams to understand how DNNs relay information across the layers and how this understanding can help us to make such models significantly simpler by allowing effective model re-optimization and hyperparameter tuning.

Orateur: Mark Neubauer (University of Illinois at Urbana-Champaign)

AISSAI Neubauer XAI DNN.pdf

ai_uncertainty_2_tues_am_2_neubauer.mp4

Talk on gslides

10:35 Advancing Explainable AI: Testing and Enhancing Techniques Across Multidisciplinary Use-Cases

Developing and testing methodologies for enhancing the transparency, interpretability, and explainability of AI algorithms is a pressing challenge for the application of artificial intelligence methods in fundamental physics. The Multi-disciplinary Use Cases for Convergent new Approaches to AI explainability (MUCCA) project is an innovative project that aims to address this challenge by bringing together researchers from diverse fields, each contributing complementary skills essential for comprehending AI algorithm behavior. The project centers around the investigation of a wide array of multidisciplinary use-cases, where explainable AI can play a pivotal role.  In our presentation we illustrate the MUCCA project in general, to then verticalize with respect to our ongoing research in the field of high energy physics.  We showcase its application in both high-energy physics experiment data analysis and its use in detector and real-time systems. Specifically, we delve into our exploration of various explainability methods rooted in different approaches and evaluate their effectiveness across the diverse use-cases. The outcome of our work yields a collection of potentially comprehensible and human-friendly explanations for the predictions made by our models. We conclude by highlighting limitations of existing xAI models for high-energy physics, and brainstorming ideas on how to build novel, explainable-by-design models for accelerating scientific research with AI.

Orateur: Simone Scardapane (Sapienza University of Rome, Roma, Italy)

AI and Physics Workshop.pdf

ai_uncertainty_2_tues_am_3_scardapane.mp4

talk on g slide

11:00 → 11:30 Coffee break : espace 66-65

outside building Esclangon, see the map.

11:30 → 12:30 Simulation Based Inference: Amphi Durand, batiment Esclangon

Président de session: Louis Lyons

11:30 An introduction to simulation-based inference

In this talk, we will introduce simulation-based inference and present how deep learning can be used to solve complex inverse problems commonly found in scientific disciplines. We will give an introduction and overview of the topic and present some of our recent work on the topic. We will also discuss the opportunities and challenges.

Orateur: Gilles Louppe (University of Liège)

ai_uncertainty_2_tues_am_4_louppe.mp4

Slides

ssi-2023.pdf

12:30 → 14:00 Buffet at the top of Tour Zamansky, Jussieu

Buffet with an incredible view on Paris, see the map.

14:00 → 18:00 Simulation Based Inference: Amphi Astier, bâtiment Esclangon

Présidents de session: Gordon Watts (University of Washington), Wouter Verkerke (Nikhef/UvA)

14:00 Simulation Based Inference: The Frequentist Perspective Abstract:

I give a brief introduction to the frequentist approach to simulation-based inference, which is often referred to as likelihood-free frequentist inference. The approach is illustrated with three simple examples, one from cosmology, one from particle physics, and the third from epidemiology.

Orateur: Harrison Prosper (Florida State University)

AISSAI_HBP_2023.pdf

ai_uncertainty_2_tues_pm_1_prosper.mp4

14:45 Simulation-Based Inference: Where Classical Statistics Meets Machine Learning

Simulation-based inference (SBI) refers to situations where the likelihood function cannot be readily evaluated but a simulator is available to generate data from a parametric model for any value of the unknown parameter. In recent years, a wide range of machine learning-based techniques have been developed to enable classical statistical inference in the simulation-based setting. These methods enable learning likelihood functions, posteriors, likelihood ratios, confidence sets and other inferential quantities with very high-dimensional data spaces using only simulations from the statistical model. In this talk, I will give an overview of some of our recent work in this area. I will first present our work on learning likelihood functions of otherwise intractable models for spatial data, which is one of the first uses of SBI in the context of purely statistical models. While likelihood-based inference is conceptually simple and computationally efficient, it only offers approximate coverage guarantees. To obtain rigorous uncertainty quantification, one can combine SBI with Neyman inversion. From this class of techniques, I will present Waldo, a method that uses neural predictions to form a Wald-type test statistic which is inverted to obtain guaranteed-coverage confidence sets. Waldo provides an appealing way of obtaining frequentist uncertainty quantification based on outputs of predictive models, including deep neural networks and neural posteriors. I will conclude by outlining some future research directions in SBI which revolve around challenges related to model discrepancy and high-dimensional parameter spaces.

Orateur: Mikael Kuusela (Carnegie Mellon University)

ai_uncertainty_2_tues_pm_2_kuusela.mp4

Paris_Nov_2023.pdf

15:30 Group photo, just outside the building on the left

15:35 Coffee break espace 66-65

outside building Esclangon, see the map.

16:00 Swyft: Direct marginal inference for large simulation models

As cosmology and astrophysics data advance, there is a growing demand for more detailed physical and instrumental simulation models with a multitude of uncertain parameters. Estimating the full joined posterior often becomes computationally prohibitive. Swyft is a deep learning python module that leverages the unique property of simulation-based inference to perform direct marginal inference. It enables to efficiently estimate individual parameter posteriors, perform marginal image reconstruction tasks, or do Bayesian model comparison, without access the joined posterior. I will provide a brief overview of the library and underlying algorithms, and present applications in astroparticle physics and cosmology.

Orateur: Christoph Weniger (University of Amsterdam)

231128 - ASSAI AI Uncertainties in Paris.pdf

ai_uncertainty_2_tues_pm_3_weniger.mp4

16:30 Precision-Machine Learning for the Matrix Element Method

The matrix element method is the LHC inference method of choice for limited statistics.
We present a dedicated machine learning framework, based on efficient phase-space
integration, a learned acceptance and transfer function. It is based on a choice of INN
and diffusion networks, and a transformer to solve jet combinatorics. Bayesian networks allow us to capture network uncertainties, bootstrapping allows us to estimate integration uncertainties. We showcase this setup for the CP-phase of the top Yukawa coupling in associated Higgs and single-top production.
Paper: arXiv:2310.07752

Orateur: Nathan Huetsch (Heidelberg University)

ai_uncertainty_2_tues_pm_4_huetsch.mp4

MEM_NathanHuetsch.pdf

17:00 Optimal dataset-wide inference in the presence of systematic uncertainties

Real-world datasets often comprise sets of observations that collectively constrain the parameters of an underlying model of interest. Such models typically have a hierarchical structure, where "local" parameters impact individual observations and "global" parameters influence the entire dataset. In this talk we introduce Bayesian and Frequentist approaches for optimal dataset-wide probabilistic inference in cases where the likelihood is intractable, but simulations can be realized via forward modeling. We construct neural estimators for the likelihood(-ratio) or posterior and show that explicitly accounting for the model's hierarchical structure can lead to tighter parameter constraints. We illustrate our methods using case studies from particle physics and astrophysics.

Based on: https://arxiv.org/abs/2306.12584

Orateur: Philipp Windischhofer (University of Chicago)

ai_uncertainty_2_tues_pm_5_windischhofer.mp4

windischhofer_2023_11_28_setwide_SBI.pdf

mercredi 29 novembre

09:00 → 11:00 Frugal Unmixing, Reusing, and Sampling

Président de session: Christoph Weniger (University of Amsterdam)

09:00 Introduction at Institut Pascal

tr202311_David_Rousseau_AIUPHYS2023_intro2.pdf

09:10 Data frugal machine learning approaches for unmixing problems in Physics

Unmixing problems are ubiquitous in Physics, ranging from spectral unmixing to unsupervised component separation, where elementary physical components need to be separated out from intricate observed mixtures. These problems are generally ill-posed, which mandates the design of effective regularisation to better distinguish between the sought-after components. While ML-based methods are promising, their application is very often limited to the scarcity of the training samples (e.g. very few observations, very high cost of physical simulations, etc.). We first propose using a special type of autoencoder (AE), coined interpolatory AE, to learn adapted representations for the components to be retrieved, from very few training samples. We show how such representations can be plugged into traditional solvers to tackle unmixing problems. This will be illustrated with applications in X-ray astrophysics and spectrometry in nuclear Physics.

Orateur: Jerome Bobin (CEA)

231123_AISSAI_AI_Physics.pdf

ai_uncertainty_3_wed_am_1_bobin.mp4

09:50 Publication and reuse of ML models in LHC analyses

With the increasing usage of machine learning in high energy physics analyses, the publication of the learned models in a reusable form has become a crucial question for analysis preservation and reuse. In turn, a lack of appropriate ML design and publication makes reinterpretation of analyses in terms of physics scenarios beyond those considered in the original experimental paper seriously difficult if not impossible. I will discuss recent efforts towards the preservation and reuse of ML-based LHC analyses together with guidelines for reusable ML models, which originated from the LHC Reinterpretation Forum and the 2023 PhysTeV workshop in Les Houches.

Orateur: Sabine Kraml (LPSC Grenoble, CNRS/IN2P3)

ai_uncertainty_3_wed_am_2_kraml.mp4

MLreuse_kraml_aissai.pdf

10:20 Machine Learning Assisted Sampling: Applications in Physics

This workshop gathered experts in computational methods for rare events sampling interested in machine learning methods and experts in generative models interested in applications to physical systems. In this talk I will try to summarize the main directions of research that were highlighted in the workshops: how learning can help for dimensionality reduction? which machine learning strategies can be designed to sample rare events? and how generative models can be adapted to physics-applications?

Orateur: Marylou Gabrié (Ecole Polytechnique)

ai_uncertainty_3_wed_am_3_gabrie.mp4

ASSAI_summary_MGabrie_vpdf.pdf

11:00 → 11:30 Coffee break

11:30 → 12:30 Benchmark, Datasets and Challenges

Président de session: Jerome Bobin (CEA)

11:30 Exploring Data Challenges and Leveraging Codabench: A Practical Journey with unsupervised New Physics detection at 40 MHz

This talk delves into our team's experience in orchestrating an unsupervised New Physics detection at 40 MHz, shedding light on the intricacies of design, implementation, and lessons learned.
We challenged the community to develop algorithms for detecting New Physics by reformulating the problem as an out-of-distribution detection task.
We provided datasets with four-vectors of the highest-momentum jets, electrons, and muons produced in a LHC collision event, together with the missing transverse energy, while the goal was to find a-priori unknown and rare New Physics hidden in a data sample dominated by ordinary Standard Model processes, using anomaly detection approaches.
We share insights gained from the past, highlighting the challenges faced and the innovative solutions employed to foster engaging and impactful competitions.
Furthermore, the presentation shifts focus to our recent exploration of Codabench as a versatile platform for orchestrating data challenges.
We share our firsthand experiences with Codabench, emphasizing its capabilities in simplifying the challenge setup process, fostering collaboration among participants, and streamlining the evaluation workflow.

Orateur: Ekaterina Govorkova (MIT)

ai_uncertainty_3_wed_am_4_govorkova.mp4

data_challenge.pdf

12:00 Neuroscience ML challenges using CodaBench: Decoding multi-limb trajectories from two-photon calcium imaging

In this talk, we present the ML challenge using CodaBench for the Neuroscience dataset. The ML challenge is hosted by the Accelerated AI Algorithms for Data-Driven Discovery (A3D3) Institute.

Neural decoding is the process of predicting behavior from brain signals, which is crucial for gaining insights into the functions of various brain regions and for advancing technology, such as the brain-computer interface, to aid individuals suffering from neurological injuries and diseases. Two-photon calcium imaging is a promising technique for neural decoding to record the activity of thousands of neurons in a single-cell resolution. Dadarlatlab at Purdue University organized the dataset of neural signals recorded by two-photon calcium imaging and running trajectories of mice. Decoding two-photon calcium imaging data has been challenging because the calcium signal indirectly and non-linearly represents action potential, has slow kinematics from a long decay time of calcium fluorescence, and has low sampling rates during imaging compared to natural behavior. Artificial intelligence can be a promising solution to overcome these challenges.

Orateur: Yuan-Tang Chou (University of Washington, Seattle (US))

ai_uncertainty_3_wed_am_5_chou.mp4

NeuroHackthon_FAIRUniverseWorkshop.pdf

12:30 → 14:00 Buffet at Institut Pascal

14:00 → 18:10 Fair-Universe hackathon: Fair uni

Président de session: Wahid Bhimji (Lawrence Berkeley National Laboratory)

Challenge website on CodaBench (restricted to workshop participants)

14:00 Introduction

Orateur: Wahid Bhimji (Lawrence Berkeley National Laboratory)

ai_uncertainty_3_wed_pm_0_bhimjii.mp4

Hackathon Intro - Fair Universe 2023 - AIUCFP - Paris.pdf

14:10 Overview of HiggsML Uncertainty Challenge

This afternoon's hackathon will center around the prototype "HiggsML Uncertainty Challenge" to be fully launched in 2024. This overview will describe the setup of the prototype challenge.

Orateur: Ragansu Chakkappai (IJCLab-Orsay, CNRS/IN2P3, Université Paris-Saclay)

ai_uncertainty_3_wed_pm_1_chakkappai.mp4

Overview of HiggsML Uncertainty Challenge.pdf

14:30 Walkthrough of CodaBench and submissions

Join us for an interactive workshop featuring a hands-on hackathon and insightful discussions. The session kicks off with a comprehensive tutorial on Codabench, an open-source challenge organization platform (codabench.org). During this tutorial, participants will gain familiarity with various aspects of the platform, including:
* Signing up/signing in to Codabench
* Navigating the overall structure of the platform
* Following the step-by-step getting started instructions
* Accessing sample submissions
* Utilizing the provided starting kit
* Submitting pre-existing submissions
* Exploring the leaderboard for scores
Once the hands-on tutorial concludes, participants are encouraged to apply their newfound skills by engaging in the HiggsML Uncertainty Challenge. Organizers will be onsite to provide assistance and guidance, ensuring a collaborative and enriching experience for all attendees.

Orateur: Ihsan Ullah (ChaLearn)

ai_uncertainty_3_wed_pm_2_ullah.mp4

Fair Universe_ HiggsML Uncertainty Challenge - Codabench Tutorial.pdf

15:00 Scoring and Baseline Systematic Aware method

Orateur: Sascha Diefenbacher (Lawrence Berkeley National Laboratory)

ai_uncertainty_3_wed_pm_3_diefenbacher.mp4

Uncertainty_Metrics_HEP_Challenge.pdf

15:30 Break

15:50 Hands-on hackathon and discussion

16:30 Feedback from partipants

16:50 Wrap up and next steps

Orateur: Wahid Bhimji (Lawrence Berkeley National Laboratory)

jeudi 30 novembre

09:00 → 11:40 Unfolding (de-biasing)

Présidents de session: Olaf Behnke (DESY), Dr Thomas Vuillaume (LAPP, Univ. Savoie Mont-Blanc, CNRS)

09:00 Unfolding in High Energy Physics

In high-energy physics, unfolding is a critical statistical process for interpreting experimental data that is complicated by the intrinsic ill-posedness of the problem. This complexity arises from the need to provide heuristics for statistical estimates that disentangle true physical phenomena from observational distortions. We present a typical roadmap for why, when, and how unfolding is applied in high energy physics experiments and how the treatment of uncertainties influences considerations such as the choice of algorithm and regularisation. Finally, the concept of unbinned unfolding is presented together with a description of how statistical and systematic uncertainties are typically addressed in unfolding problems, together with a discussion of how statistical modelling and AI can lead to better estimates in the future.

Orateur: Vincent Alexander Croft (LIACS)

AISSAI-unfolding.pdf

ai_uncertainty_4_thu_am_1_croft.mp4

Unfolding In HEP Slides

09:45 An unfolding method based on conditional Invertible Neural Networks (cINN) using iterative training

The unfolding of detector effects is crucial for the comparison of data to theory predictions. While traditional methods are limited to representing the data in a low number of dimensions, machine learning has enabled new unfolding techniques while retaining the full dimensionality. Generative networks like invertible neural networks (INN) enable a probabilistic unfolding, which maps individual data events to their corresponding unfolded probability distribution. The accuracy of such methods is however limited by how well the experimental data is modeled by the simulated training samples.
We introduce the iterative conditional INN (IcINN) for unfolding that adjusts for deviations between simulated training samples and data. The IcINN unfolding is first validated on toy data and then applied to pseudo-data for the $pp \rightarrow Z \gamma \gamma$ process. Additionally, we validate the probabilistic unfolding with a novel approach using the traditional transfer matrix-based methods.

The main results of this project have been published in a paper (arXiv:2212.08674: https://arxiv.org/abs/2212.08674). A second paper with a stronger focus on the probabilistic unfolding will be published prior to the conference.

Orateur: Mathias Josef Backes (Kirchhoff Institut für Physik)

AISSAI_mBackes.pdf

ai_uncertainty_4_thu_am_2_backes.mp4

10:15 Can contrastive learning de-bias my model?

Deep learning models have become ubiquitous in high-energy physics and have been successfully applied to a wide variety of tasks. Models for reconstruction are usually trained from scratch on a nominal set of simulation parameters, not taking into account variations of detector systematic uncertainties.

Following advances in contrastive learning, we present a method of pre-training a general model, that is de-biased from detector systematic uncertainties. During the pre-training phase, the model learns a representation that is invariant to simulation shifts and symmetries, by contrasting between different simulated views of the same event. Freezing the weights of the contrastive model, the extracted representation is general enough that it can be used for a variety of prediction tasks. We showcase the flexibility and efficacy of this method by training with sparse 3D neutrino liquid argon time projection chamber (LArTPC) data.

Orateur: Radi Radev (CERN)

ai_uncertainty_4_thu_am_3_radev.mp4

AIUPHYS_2023.pdf

10:45 Coffee break

11:15 Uncertainty Quantification and Anomaly Detection with Evidential Deep Learning

Evidential Deep Learning (EDL) is an uncertainty-aware deep learning approach designed to provide confidence (or epistemic uncertainty) about test data. It treats learning as an evidence acquisition process where more evidence is interpreted as increased predictive confidence. This talk will provide a brief overview of EDL for uncertainty quantification (UQ) and will discuss its connection with anomaly detection (AD). Several examples will be presented, including ongoing work in this area for HEP applications.

Orateur: Mark Neubauer (University of Illinois at Urbana-Champaign)

AISSAI Neubauer EDL UQ AD.pdf

ai_uncertainty_4_thu_am_4_neubauer.mp4

EDL gslide

11:40 → 12:25 Controlling uncertainties in generative models

Président de session: Dr Thomas Vuillaume (LAPP, Univ. Savoie Mont-Blanc, CNRS)

11:40 Potential and challenges of highly dimensional generative models

Recent progress in computer science, specifically the development normalising flows and diffusion models, has brought about a breakthrough in the fidelity of generative models in particle physics.

In this talk I will first review some of these new approaches and then discuss potential uses, considering the overall theme of uncertainties. This will allow us to discuss statistical properties, performance metrics, inherent uncertainties, surrogate models and in-situ background estimation.

Orateur: Gregor Kasieczka (Universität Hamburg)

ai_uncertainty_4_thu_am_5_kasieczka.mp4

DeepLearning_Paris_Uncertainty_2023.pdf

12:25 → 14:00 Buffet at Institut Pascal

14:00 → 18:00 Controlling uncertainties in generative models

Présidents de session: Mehmet Ozgur Sahin (CEA-Saclay), tommaso dorigo (INFN Sezione di Padova)

14:00 Machine-learning and equations-informed tools for generation and augmentation of turbulent data.

Our ability to collect data is rapidly increasing thanks to computational power and the unprecedented diversity of sensors. But how good are we at extracting, reconstructing, and understanding information from them? We present a short overview of some recent advancements for data-assimilation and modelling of turbulent multi-scale flows using both data-driven and equations-informed tools, starting from sparse and heterogeneous observations of complex fluid systems. Issues connected to validations and benchmarks in the presence of full or partial observability will be discussed. A few examples of data-generation and data- augmentation based on Generative Adversarial Learning, Diffusion Models and Nudging, for Eulerian and Lagrangian turbulence will be quantitatively discussed.

-Synthetic lagrangian turbulence by generative diffusion models
T Li, L Biferale, F Bonaccorso, MA Scarpolini, M Buzzicotti
arXiv preprint arXiv:2307.08529 (2023).
- Multi-scale reconstruction of turbulent rotating flows with proper orthogonal decomposition and generative adversarial networks
T Li, M Buzzicotti, L Biferale, F Bonaccorso, S Chen, M Wan
Journal of Fluid Mechanics 971, A3 (2023)
-Synchronization to big data: Nudging the Navier-Stokes equations for data assimilation of turbulent flows
PC Di Leoni, A Mazzino, L Biferale Physical Review X 10 (1), 011023 (2020)

Orateur: Luca Biferale (U Roma Tor Vergata)

ai_uncertainty_4_thu_pm_1_biferale.mp4

Parigi AISSAI.pdf

14:45 Generative modeling in genomics and a perspective on uncertainty quantification

In recent years, generative modeling has gained substantial momentum in genomics research thanks to increased availability of computational resources and development of deep generative models (DGMs) over the past decade. DGMs can learn the complex structure of genomic data and can be utilized for a variety of tasks such as generation of realistic artificial genomes, dimensionality reduction and prediction, with unsupervised, semi-supervised or supervised learning schemes. In this talk, I will present a background on generative models in genomics, discuss our recent work on the generation of artificial genomic data, and provide my perspective on approaches for uncertainty quantification.

Orateur: Burak Yelmen (University of Tartu)

ai_uncertainty_4_thu_pm_2_yelmen.mp4

yelmen1.pdf

15:30 Uncertainty-aware diffusion models for LHC Event Generation

Given the recent success of diffusion models in image generation, we study their applicability to generating LHC phase space distributions. We find that they achieve percent level precision comparable to INNs. Training uncertainties are quantified by developing Bayesian versions to further enhance the interpretability of our results. In this talk, diffusion models are introduced and discussed followed by a presentation of our findings.

Orateur: Sofia Palacios Schweitzer (ITP, Heidelberg University)

ai_uncertainty_4_thu_pm_3_schweitzer.mp4

Paris_SofiaPalaciosSchweitzer.pdf

16:00 Group photo

16:05 Coffee break

16:30 Data driven background estimation in HEP using generative adversarial networks

Data-driven techniques are indispensable for addressing the limitations of Monte Carlo (MC) simulations in High Energy Physics experiments, such as insufficient statistics and process mismodeling. Accurate representation of background processes is essential for achieving optimal measurement sensitivity. Traditional approaches often involve the selection of a control region to model the background, but this can introduce biases in the distribution of certain physics observables, and hence rendering them unusable in subsequent analyses. To overcome this issue, we introduced a novel method that generates physics objects that are both compatible with the region of interest and accurately represent correlations with other event properties.
To achieve this we employ conditional generative adversarial networks (GANs), leveraging their proven efficacy in various machine learning tasks. The method is illustrated by generating a new misidentified photon for the gamma+jets background of the $\mathrm{H}\to\gamma\gamma$ analysis on the CMS Open Data simulated samples. We demonstrate that the GAN is able to generate a coherent object within the region of interest and still retains correlations with other observables within the rest of the event.

Orateur: Mehmet Ozgur Sahin (Irfu, CEA Paris-Saclay)

ai_uncertainty_4_thu_pm_4_sahin.mp4

GAN_AIU23.pdf

17:00 Using an adversary trained on a control sample to control systematic errors

Machine Learning improved the sensitivity in searches for massive long-lived neutral particles decaying in the Calorimeter by over 30%. This was only after supressing a large increase in the systeamtic errors caused by the method. The largest contribution to this improvement in senstivity is the use of a Recurrant Neural Network that separates signal from standard QCD multijet background and Beam Induced Background. This classifier uses low-level data like realtive calorimeter cluster locations, tracks, and muon segments. We exploit the calorimeter cell energy deposit time as a powerful handle to reject beam induced background, which is poorly simulated by the ATLAS experiment's Monte Carlo simulation package. In addition, the beam induced background training dataset can only be drawn from data. Thus the RNN training set contains a mix of poorly simulated Monte Carlo data, and LHC collision data. A control dataset was used to train an adversary, along with signal and background samples for the RNN, simultaniously. The adversary is trained to tell the difference between collision data and simulated data, and its success is part of the main network's loss function. This dramatically reduced the systematic errors due to Monte Carlo mis-modeling. This presentatino will discuss the network design, how it was modified when the problem(s) were discovered, and its performance.

Orateur: Gordon Watts (University of Washington)

2023-11-30 Using Adversaries to Control Systematics.pdf

2023-11-30 Using Adversaries to Control Systematics.pptx

ai_uncertainty_4_thu_pm_5_watts.mp4

19:00 → 21:00 Workshop Dinner at Brass&Co

The workshop dinner is at Brass & co micro-brewery at walking distance from Institut Pascal

vendredi 1 décembre

09:00 → 12:00 Architectures

Présidents de session: David Rousseau (IJCLab, CNRS/IN2P3, Université Paris-Saclay), Gregor Kasieczka (Hamburg University)

09:00 Robustness to Uncertainties in Machine Learning Applications for HEP

In this presentation I will discuss recent trends in the handling of systematic uncertainties in HEP analysis tasks, and techniques proposed to mitigate or remove their effect in the search for optimal selection criteria and variable transformations.
The approaches discussed include nuisance-parametrized models, modified adversary losses, semi-supervised learning approaches, inference-aware techniques, and other recent developments.

Orateur: tommaso dorigo (INFN Sezione di Padova)

ai_uncertainty_5_fri_am_1_dorigo.mp4

Incorporating Systematic Uncertainties in ML v3-2.pdf

Incorporating Systematic Uncertainties in ML v3.pptx

09:45 Data Subsampling for Bayesian Neural Networks

The development of an effective Uncertainty Quantification method that computes the predictive distribution by marginalizing over Deep Neural Network parameter sets remains an important, challenging task. In this context, Markov Chain Monte Carlo algorithms do not scale well for large datasets leading to difficulties in Neural Network posterior sampling. During this talk, we'll show that a generalization of the Metropolis Hastings algorithm allows to restrict the evaluation of the likelihood to small mini-batches in a Bayesian inference context. Since it requires the computation of a so-called “noise penalty” determined by the variance of the training loss function over the mini-batches, we refer to this data subsampling strategy as Penalty Bayesian Neural Networks – PBNNs.

Orateur: Eiji Kawasaki (CEA)

ai_uncertainty_5_fri_am_2_kawasaki.mp4

Kawasaki_AISSAI_231130.pdf

10:15 Efficient Sampling from Bayesian Network Posteriors for Optimal Uncertainties

Bayesian neural networks are a key technique when including uncertainty predictions into neural network analysis, be it in classification, regression or generation. Although being an essential building block for classical Bayesian techniques, Markov Chain Monte Carlo methods are seldomly used to sample Bayesian neural network weight posteriors due to slow convergence rates in high dimensional parameter spaces. Metropolis-Hastings corrected chains exhibit two major issues: using a stochastic Metropolis-Hastings term and bad acceptance rates. We present solutions to both problems in form of a correction term to the loss objective and novel proposal distributions based on the Adam-optimizer. The combined algorithm shows fast convergence and good uncertainty estimation for physics use cases without dramatically increasing the cost of computation over gradient descent based optimization.

Orateur: Sebastian Bieringer (Hamburg University, Institute for experimental physics)

AISSAI_sbieringer.pdf

ai_uncertainty_5_fri_am_3_bieringer.mp4

10:45 Coffee break

11:15 Revisiting models and uncertainty with AI

Predictions from empirical evidence come with many sources of potential uncertainty and error. First, the specific choices of models and concepts that we tack onto the observation give a strong prism to the resulting
conclusion. Uncertainty on which functional form to use in a model, naturally results in uncertainty of conclusions. Outside of mature (post-paradigmatic) quantitative sciences such as physics, the mere choice of ingredients put the model (which quantities to measure) is open.

I will discuss how AI, or machine learning, brings a new angle to these questions, because it tackles complex observations with very flexible models. I believe that it opens new doors to scientific evidence by putting the burden on validity on model outputs, rather than ingredients.

However, a given model fitted on data should ideally express its uncertainty as a probability of the output given the input. This is particularly important in high-stakes applications such as health. I would discuss how controlling this uncertainty requires to control a quantity know as calibration, but also to go further and control the reminder, the "grouping loss", which leads to challenging estimation problems.

Orateur: Gael Varoquaux (INRIA-Saclay)

ai_uncertainty_5_fri_am_4_varoquaux.mp4

slides.pdf

12:00 → 13:30 Buffet at Institut Pascal

13:30 → 16:30 Closing session: (starts at 1:30!)

Président de session: Corentin Allaire (IJCLab, Université Paris-Saclay, CNRS/IN2P3)

13:30 Highlights from the 1st CNRS AISSAI Thematic Quarter on Causality

Causal and effect questions are the cornerstone of numerous scientific disciplines, providing a framework for formulating and comprehending them under diverse conditions. In recent years, a notable interdisciplinary effort has been to develop new methods to address causality-related challenges. The first CNRS AISSAI Thematic Quarter on Causality, held earlier this year, marked a significant interdisciplinary step in advancing our understanding of causal relationships. This talk aims to highlight some insights and methodologies gleaned during the quarter. They include the use of sigma-algebra via the Witsenhausen Intrinsic Model (WIM) and the Information Dependency Model (IDM) as an alternative approach to the traditional functional causal models and causal graphs, the axiomatization of causality through Kolmogorov's measure-theoretic axiomatization of probability, and the use of cumulant tensors for characterizing hidden common causes when learning causal graphs in linear non-Gaussian causal models. Moreover, the quarter underscored the promising role of diffusion and normalizing flow methods in capturing the underlying causal data-generating processes, offering new horizons in causal representation learning.

Orateur: Alessandro Leite (LISN, INRIA)

AISSAI_ALEITE.pdf

ai_uncertainty_5_fri_pm_1_leite.mp4

14:15 Fair Universe HiggsML Uncertainty Challenge: Lessons Learned and plans

The first FAIR Universe public challenge, FAIR Universe: HiggsML Uncertainty Challenge, will be launched before this workshop. A dedicated hackathon within the workshop will refine the existing prototype, with the full version set for release in 2024. The intricacies of devising an efficient scoring method for uncertainty-aware techniques pose a challenge, and the current approach will be discussed during the workshop. This presentation will spotlight insights gained from diverse workshop activities and explore potential approaches for forthcoming iterations of the FAIR Universe challenge.

Orateur: Elham E Khoda (University of Washington)

ai_uncertainty_5_fri_pm_2_khoda.mp4

Fair Universe HiggsML Uncertainty Challenge Lessons Learned and Plans.pdf

summary_slides_Elham

14:45 Farewell

Orateur: David Rousseau (IJCLab, CNRS/IN2P3, Université Paris-Saclay)

ai_uncertainty_5_fri_pm_3_rousseau.mp4

tr202311_David_Rousseau_AIUPHYS2023_outro.pdf