AISSAI - Heterogeneous Data and Large Representation Models in Science

30 Sept 2024, 12:30 → 3 Oct 2024, 14:00 Europe/Paris

Le Village, Auditorium (Toulouse, France)

The third workshop of the AISSAI semester on "Artificial Intelligence for the two infinites" will be held in Toulouse, France from September 30^th to October 3^rd, 2024. This workshop will be devoted to Heterogeneous Data and Large Representation Models in Science. 

---

Scientific program

In recent years, we have witnessed remarkable transformations in the AI/ML landscape. Particularly in computer vision and natural language processing, there is a notable emergence of Large Representation Models (LRMs) trained on extensive datasets, often referred to as foundation models. These LRMs possess the capability to encode information at a high level of abstraction, enabling the training of models on multimodal data such as text, image, sound, video, and more. This improvement in how models represent objects augments significantly their ability to understand and make sense of the world.

In the realm of science, we expect a similar revolution, triggered by the integration of heterogeneous multimodal scientific data from various sensor systems or sources into LRMs. Scientific data are often heterogeneous and multimodal in nature, originating from various sensors in embedded systems (robotics, aerospace), from different detector subsystems or different instruments in fundamental physics, or from different signal sources in a scientific experiment in general. The models can combine representations from neural networks with symbolic representations integrating a priori knowledge of the scientific domain.

To be more specific: The expected revolution in science will likely start with the development and deployment of algorithms that solve specific problems using heterogeneous data as inputs, as for example in the control of a robot using data from sensors of different types (cameras, infrared proximity scanners, …); or the reconstruction of elementary particles in a detector that combines diverse technologies (silicon pixels and strips, calorimeters, …). At the same time, it should become possible to use machine learning to tackle problems where the analysis of one dataset requires multiple models to collaborate, with each model being an “expert” in one aspect of the data. An example from cosmology would be the analysis of the data of the LISA gravitational wave detector (link, link). LISA will observe the superposition of signals from a large, a priori unknown number of sources of different types (galactic binaries, mergers of massive black holes, and many others). Individual models might be trained separately for each type, and then learn to collaborate on a global analysis.
In all of these initial examples, the size of the models and the resources required for training should remain modest (nowhere near, say, ChatGPT) and be readily accessible to researchers in academia.

The aim of this workshop is to bring together scientists from different fields (just to list a few: computer sciences, cosmology, human sciences, mathematics, physics, robotics, statistics, etc.) and with different profiles (experimentalists, theorists, developers) to discuss these topics at the forefront of AI/ML research, fostering collaboration and innovation in this rapidly evolving field. 

The program is planed to be a mix of high-profile guest speaker presentations and contributed talks.

This workshop will delve into a range of topics, which include but are not limited to:

• Constructing machine learning models capable of learning from diverse data types.
• Managing multimodal data from varied sources, or heterogeneous data from scientific instruments that combine multiple detector technologies, for ML applications.
• Investigating contrastive embeddings tailored for heterogeneous and multi-modal scientific data alongside shared embedding representations.
• Exploring the integration of neuro-symbolic AI and multi-level representations.
• Mathematical modeling of combined representation.
• Exploring explainability and interpretability of Large Representation Models in the scientific context.
• Embracing frugality and size management in Large Representation Models.
• Possibly on a longer timescale, exploring numerical encodings for large language representations in scientific contexts. 

---

Practicalities

The workshop will take place in the Venue Le Village, downtown Toulouse. 

Participation will be limited to 80 on-site participants.

Registrations is free (but mandatory) and includes lunches and social events. Registrations are moderated and do not constitute acceptance of participation.  A final validation for acceptance of participation in the workshop will be made by the organizers and each person who will have registered will be notified.

Talks are expected to be given in person.

---

Important dates

These dates may be subject to change.

	Registration opening	July 1st
	Abstract submission opening	July 3rd
	Abstract submission dead-line	August 15th -> September 9th
		Late abstract submission will be considered until September 15th, provided you contact us directly before September 9th
	Abstract acceptance dead-line	August 31th
	Program release	September 7th -> September 17th
	Registration closing	September 15th
		Late registration request will be considered, please contact us directly as early as possible
	Workshop start	September 30th

---

Organisers and Partners

---

 

poster_AISSAI_heterogeneous-data_2024.png

Monday 30 September

12:30 Registration & light lunch

Welcome

1 Workshop welcome

Speaker: (TBC)

2 Practicalities

Speaker: (TBC)

3 Foundation models for high energy physics

Foundation models are machine learning models designed to handle a wide range of datasets and tasks. After being pre-trained on a specific task on a specific dataset, these models can be fine-tuned for various downstream applications, including different tasks and datasets. Developing such models for physics data could significantly enhance performance in the field and substantially cut down the necessary training time and data requirements. In this talk, I will give an introduction to foundation models, provide an overview of the foundation models that exist for particle physics today, and discuss some challenges and outlooks for the future.

Speaker: Anna Hallin (Universität Hamburg)

4 PolarBERT: a Foundation Model for Neutrino Telescope Data

Neutrinos are elusive particles that require massive detectors for observation. The IceCube neutrino observatory at the South Pole is a cubic kilometer of Antarctic ice, instrumented with 5,160 digital optical modules. Its results play an essential role in both particle physics and astrophysics.
Deep learning methods, such as graph neural networks, have been successfully applied to the steady stream of heterogeneous data IceCube is receiving. In this talk, we will present a foundation model for the IceCube data. It is trained in a self-supervised way without any data labeling. We further fine-tune this pretrained model for the downstream task of directional reconstruction of neutrino events. We show that this pretrained model significantly outperforms models trained from scratch. Remarkably, the foundation model does not require any knowledge of the IceCube detector geometry or characteristics of its electronics, since it extracts all the necessary information from the raw data.

Speaker: Inar Timiryasov (Niels Bohr Institute, University of Copenhagen)

15:50 Coffee break

5 Scientific Foundation Models for Computational Fluid Dynamics: threats and opportunities

Scientific Foundation Models (SciFMs) hold the promise of accelerating numerical simulation of physical phenomena. In recent years, a myriad of SciFMs for weather forecasting have been proposed by major companies (e.g., Microsoft's ClimaX and Aurora) as well as research centers (e.g., ECMWF's AIFS). The development of SciFMs in other domains such as Computational Fluid Dynamics (CFD) has not yet reached similar maturity levels, though. In this presentation, we discuss threats and opportunities surrounding the training and deployment of SciFMs for CFD. On the one hand, weather forecasting historically benefits from open data sharing thanks to government-funded research. On the other hand, CFD community is mainly backed up by proprietary software from industry, which limits sharing of information and eventually impacts SciFM development. Still, recent initiatives in open CFD data sharing like BLASTNet give hope that open-sourced SciFMs for CFD will become available in a near future.

Speakers: Dr Fernando Gonzalez (CERFACS), Luciano Drozda (CERFACS)

6 Small thinks big: transfer learning in KM3NeT/ORCA for neutrino event reconstruction

This study explores using transformer models to analyze data from the KM3NeT/ORCA neutrino detector. Due to the current detector's size, reconstructing neutrino events is challenging. By training models on simulations for the full detector (115 detection units) and fine-tuning them on smaller configurations, significant performance improvements are achieved compared to models trained from scratch on very limited data. This approach also helps estimate the detector's sensitivity as it grows.

Speaker: Ivan Mozun

7 Automatic estimation of the wind turbine noise with recurrent neural networks

There is growing interest in the development of renewable energies, particularly wind power. However, wind turbines generate noise that can affect the sound environment of nearby residents.
This study focuses on the isolation of wind turbine noise (WTN) level from the surrounding total noise. Our method is based on a Recurrent Neural Network (RNN) Architecture that captures temporal dependencies in the acoustic signal.
This proposal is compared to Non-Negative Matrix Factorization (NMF) that has shown first promising results on a previous study on simulated sound scenes of wind turbine noise.
Our approach relies on simple RNN Vanilla conducted using an end-to-end trained model, Gated Recurrent Network (GRU), and a Long Short TermMemory (LSTM) trained from scratch and compared in the same dataset to the NMF method.

Speaker: Mr ABDELAZYZ RKHISS (Doctorant à Grenoble INP)

17:45 Free time

19:00 Welcome cocktail

Tuesday 1 October

8 Gravitational waves coming at you from all directions

In January of this year, the European Space Agency officially adopted the space-based gravitational wave detector, LISA, as a mission, to launch in 2035. LISA will open up a new band in the gravitational wave frequency spectrum, at millihertz frequencies. This band is expected to be very rich in sources, ranging from binaries of compact stars in our galaxy, to binaries involving supermassive black holes in the centres of galaxies, to stochastic backgrounds formed. In contrast to ground-based detectors, these sources will be overlapping in the LISA data in both time and frequency, posing a complex data analysis problem necessitating a simultaneous global-fit to all sources of all types. Data analysis will be further complicated by instrumental artefacts, including gaps and glitches in the data and an unknown and time-dependent instrumental noise level, and imperfect knowledge of signal models. In this talk, I will present the LISA data analysis challenge, describe the approaches that are being developed to tackle it using standard techniques, and highlight areas where novel machine learning approaches are or could be developed to improve the efficiency of the analysis.

Speaker: Jonathan Gair (AEI Potsdam)

09:45 Coffee break

9 Neural density estimation for Galactic Binaries in LISA data analysis

The future space based gravitational wave detector LISA will observe millions of galactic binaries (GBs) constantly present in the data. A small fraction of this population will be individually resolved. One of the challenging tasks will be to estimate the parameters of resolvable GBs while disentangling them from each other and from other gravitational wave sources present in the data. This problem is referred to as a global fit. A Bayesian framework is often used to infer the parameters of the sources and their number. The efficiency of the sampling techniques strongly depends on the proposals, especially in the multi-dimensional parameter space. We show how to use neural density estimators, and in particular Normalising flows, in order to build proposals which significantly improve the convergence of sampling. We also demonstrate how these methods could help in building priors based on physical models and provide an alternative way to represent the catalogue of identified sources.

Speaker: Natalia Korsakova (APC)

10 Beyond Gauss? A more accurate model for LISA astrophysical noise sources

In this talk, we explore two assumptions ubiquitous in LISA data analysis: Gaussianity and stationarity of astrophysical noise sources (i.e. arising from source confusion). I will provide an overview on characterizations and parameter estimations techniques of both properties. I will review the most recent findings on the Galactic population of double white dwarfs and the extragalactic one of extreme mass-ratio inspirals. Employing a suite of advanced statistical techniques, we aim to enhance our understanding of the noise properties and improve the overall data modeling process.

Speaker: Dr Riccardo Buscicchio (Universitá di Milano-Bicocca, Milan, IT)

11 Statistically principled learning for gravitational-wave inverse problems

An important aspect of gravitational-wave astronomy is solving inverse problems, i.e., determining the properties of astrophysical sources from their gravitational signals. This involves the construction of complex forward models for possible signals by solving the equations of general relativity, as well as the use of these forward models in data-analysis algorithms to extract and characterise actual signals in detector data. Machine learning is increasingly used to confront modern challenges in these tasks, although it faces unique hurdles such as noise-dominated data and the need for high precision in modelling. It is also crucial to clarify how any proposed learning method relates to the existing Bayesian framework for solving gravitational-wave inverse problems. In this talk, I will discuss broad strategies for developing machine-learning methods that are tailored to the needs of the field while remaining defensible on scientific rigour and principle.

Speaker: Alvin Chua (National University of Singapore)

11:40 Group picture

12:00 Lunch

12 Deep learning and the global workspace theory

.

Speaker: Rufin VanRullen (Centre de Recherche Cerveau et Cognition (CerCo), Artificial and Natural Intelligence Toulouse Institute (ANITI))

13 Semi-supervised multimodal representation learning through a global workspace

Recent deep learning models can efficiently combine inputs from different modalities (e.g., images and text) and learn to align their latent representations, or to translate signals from one domain to another (as in image captioning, or text-to-image generation). However, current approaches mainly rely on brute-force supervised training over large multimodal datasets. In contrast, humans (and other animals) can learn useful multimodal representations from only sparse experience with matched cross-modal data. Here we evaluate the capabilities of a neural network architecture inspired by the cognitive notion of a ``Global Workspace'': a shared representation for two (or more) input modalities. Each modality is processed by a specialized system (pretrained on unimodal data, and subsequently frozen). The corresponding latent representations are then encoded to and decoded from a single shared workspace. Importantly, this architecture is amenable to self-supervised training via cycle-consistency: encoding-decoding sequences should approximate the identity function. For various pairings of vision-language modalities and across two datasets of varying complexity, we show that such an architecture can be trained to align and translate between two modalities with very little need for matched data (from 4 to 7 times less than a fully supervised approach). The global workspace representation can be used advantageously for downstream classification and cross-modal retrieval tasks and for robust transfer learning. Ablation studies reveal that both the shared workspace and the self-supervised cycle-consistency training are critical to the system's performance.

Speaker: Léopold Maytié (Université Toulouse III - Paul Sabatier, Toulouse, France & Artificial and Natural Intelligence Toulouse Institute (ANITI))

14 Explaining Jet Flavour Taggers with Integrated Gradients

At the Large Hadron Collider (LHC), proton-proton collisions produce collimated streams of particles called jets created from particle decay chains. Identifying the particle that originated the jet (flavour tagging) is crucial. Modern taggers use deep learning models with features of the decay products as inputs. We show that integrated gradients reveal how these complex and opaque models use the characteristic features of the decay products to classify the particle that originated the jet.

Speaker: Scott DeGraw (University College London)

15 A graph-structured distance for heterogeneous datasets with meta variables

This talk presents a novel distance function and modeling framework for mixed-variable domains, effectively handling heterogeneous data with continuous, integer, and categorical variables, including meta variables that shape problem structure. This approach is presented in a paper that enhanced generalization and optimization in large representation models in science without partitioning data. A follow-up paper will extend this work by unifying surrogate modeling in architecture optimization, introducing graph-structured domains and partially decreed variables, with applications in green aeronautics via Bayesian optimization.

Speaker: Dr Paul SAVES (DTIS, ONERA and Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, France)

16 Challenges of heterogeneous data for building Linguistic Theory

Linguistics thrives on data, whether it stems from small highly controlled laboratory studies or from large heterogeneous datasets. Speech technology is increasingly providing new and varied tools to test linguistic theories (from sound change to second language learning) on large scale data. This, however, does not come without its challenges. In this presentation, we address one of the key challenges for testing linguistic theory (such as diverging voicing systems within language families), posed by heterogeneous data: the frequent absence of linguistically formatted metadata.

Speaker: Anisia Popescu (LISN)

16:00 Coffee break

16:30 Place holder round table

Wednesday 2 October

17 RELEO - Representation Learning for Earth Observation

This talk will introduce the RELEO (REpresentation Learning for Earth Observation) project (2024-2028), a research chair of the Artificial and Natural Intelligence Toulouse Institute (ANITI). RELEO aims at developing new self-supervised representation learning methods to produce semantically meaningful probabilistic representations from high-dimensional multi-modal EO data. The originality of the approach lies on the use of prior knowledge from physical models into Deep Learning and thus proposing advances in uncertainty estimation and interpretability. Recent results will be presented: physics-constrained deep learning for biophysical parameter retrieval from satellite optical imagery and spectro-spatio-temporal encoders for large representation models for irregular and unaligned satellite image time series.

Speaker: Jordi INGLADA (CESBIO, Université de Toulouse, CNES/CNRS/INRAe/IRD/UT3)

18 Identifying a piecewise affine signal from its nonlinear observation - application to DNA replication analysis

An important challenge in DNA replication analysis is to recover a so-called timing profile, that contains important information about the replication dynamics, from nonlinear observations. We show that this challenge can be expressed as a nonlinear sparse coding inverse problem where the unknown timing profile is assumed to be piecewise affine.

We propose a novel formalism and computational approach to harness it. In the noiseless case, we establish sufficient identifiability conditions for the timing profile,and prove that it is the solution of a non-convex optimization problem.
We propose the DNA-inverse optimization method that provably finds the global solution to the nonlinear inverse problem for noisy signals. Besides being more computationally effective than the state-of-the-art optimization, our approach automatically recovers all configurations of the replication dynamics. This is crucial for DNA replication analysis, and was not possible with previous methods

Speaker: Clara Lage (ENS de Lyon)

10:25 Coffee break

10:55 Place holder -FN - multimodal learning in cosmology

19 Galaxy detection with deep learning in radio data

Astronomical facilities generate ever-increasing data volumes, rapidly approaching the exascale. In this talk, I will introduce YOLO-CIANNA, a deep-learning object detector for astronomical images, and present results over simulated 2D continuum images and HI emission cubes from the SKAO SDCs. I will then discuss how the method could be applied to data from the SKA precursor and how we could combine heterogeneous data from other types of surveys to build an instrumental-context-aware detector.

Speaker: David Cornu (Observatoire de Paris | PSL)

12:25 Flash-talk posters

12:40 Lunch

20 Learning how to design biomolecules using a neuro-symbolic architecture

Designing requires to mix physical knowledge, experience accumulated from past designs and constraints defining design objectives.

Proteins are large biomolecules that play crucial roles in all living organisms. They are linear polymers which can be described as a sequence in a 20 letter alphabet (one for each amino acid). They can therefore be represented as discrete objects. In water, most proteins fold in a 3D structure, defining continuous atomic coordinates.

To design new proteins, we introduced an hybrid architecture that combines all above elements in a joint pairwise decomposable function over amino acid identities. Physics is represented as a force field, experience is extracted by Deep Learning from Nature's designs and design objectives represented as constraints. The resulting model is then passed to an automated reasoning prover to identify the most suitable chemical composition. The same architecture can learn how to play Sudoku from examples, w/o knowing the rules.

Speaker: Thomas Schiex (Université fédérale de Toulouse, ANITI, INRAE)

21 Salt: Multimodal, Multitask Models for the ATLAS Experiment

In High Energy Physics, experimental data can range from low level hardware information from different sub-detectors to high level reconstructed physics events. To address the need for flexible, multimodal machine learning models within the ATLAS experiment, the Salt framework based upon PyTorch and Lightning has been developed. Salt was initially developed for the identification of heavy-flavour jets but has expanded to a wider range of tasks including searches for long lived particles, jet mass regression or vertex fitting.

Speaker: Jackson Barr (UCL)

22 Graph Neural Networks for track reconstruction in the ATLAS ITk detector

High-energy physics (HEP) experiments, e.g. ATLAS and CMS, collide opposing packs of particles and characterize the collision's final state. The innermost part of a detector consists of many sensors which detect the passage of a charged-particle by measuring its energy deposit. A tracking algorithm recreates from these measurements the trajectories of all particles, a computationally intensive process which the HEP community seeks to replace with machine learning alternatives. We present in this contribution a pipeline centered on Graph Neural Network (GNN) developed for the ATLAS Inner Tracker (ITk) achieving comparable performance to the traditional analogue. We describe ML techniques employed in this pipeline, and our solutions to the unique memory and time constraints associated with processing collision data at low level. Finally, we discuss the challenge of learning heterogeneous data collected by different sensor technologies in the ITk, and our ongoing effort to address it.

Speaker: Minh-Tuan Pham (University of Wisconsin-Madison)

15:55 Poster coffee break

23 Large-scale deep-learning for weather and climate prediction

A new paradigm for weather and climate prediction has emerged recently : data-driven prediction models have achieved similar performances as standard physics-based models, thanks to an accurate (task-specific or task-agnostic) encoding of the data distribution. While these models are able to efficiently use relatively homogenous data, the next challenge to expand the capabilities of data-driven modeling is to fully exploit the vast range of atmospheric observations, characterized by spatio-temporal variations and heterogeneous outputs (point or spatial time series, vertical profiles, vertically integrated data, … ). An overview of existing LRM for weather & climate prediction will be presented, as well as early results for integrating heterogeneous data sources.

Speaker: Laure Raynaud (Météo-France)

17:20 place holder - keynote spekare - DM

18:05 Free time

19:30 Workshop diner

Thursday 3 October

09:00 Space is available for you luguages.

24 Enhancing Ultrasound Localization Microscopy (ULM) with Spatio-Temporal Deep Learning

The integration of Ultrasound Localization Microscopy (ULM) into ultrasound imaging has significantly improved resolution, providing precise insights into blood flow direction and velocity. However, despite its potential, ULM remains a complex and time-consuming technique, even as deep learning (DL) continues to drive its optimization. Current DL methods for microbubble (MB) superlocalization face challenges due to the use of high-resolution images in their networks, resulting in longer processing times compared to traditional ULM methods. Additionally, these methods often require arbitrary filtering of results before integration into tracking algorithms. To address these challenges, our study introduces a novel DL approach inspired by single-molecule localization techniques. Our 3D convolutional neural network, called 3DML-ResNet, enables fast and scalable superlocalization while providing explicit estimation of the number of MBs present in each image.

Speaker: Vassili PUSTOVALOV (Institut de recherche en informatique de Toulouse, Université Toulouse III - Paul Sabatier)

25 Preprocessing arbitrarily structured data for AI with Awkward Array

Processing heterogeneous multimodal data presents challenges. These datasets feature complex, irregular structures due to nested or variable-sized outputs from different sensors, or due to missing data values. The data are typically of mixed types, complicating the preprocessing steps required before they can be fed into algorithms like multimodal representation models. AI practitioners must manage these complexities effectively.
Awkward Array is a Python library designed to process arbitrarily structured data. Operating on an array-programming paradigm, it allows users to manipulate data using NumPy-like syntax. Awkward Array also includes GPU-accelerated kernels, enabling the preprocessing of complex data directly on modern hardware accelerators, which can significantly optimize the training process and reduce data transfer latency to the device.
We introduce the Awkward Array library and provide examples that demonstrate its usage, highlighting its potential as an AI preprocessor.

Speaker: Vangelis Kourlitis (Technical University of Munich)

26 Leveraging AI in computational physics with NVIDIA Modulus and TorchFort

NVIDIA supports the scientific community in leveraging data-driven and AI approaches in computational physics workflows. This talk will showcase how researchers use NVIDIA's open-source libraries like Modulus to integrate learning methods with scientific solvers. Specifically, it will focus on recent results that facilitate AI in-the-loop approaches and enable on-the-fly training and inference during computation thanks to the open-source library TorchFort. Finally, it will introduce a blueprint that new users can follow to replicate this in their simulation workflows, unlocking new areas of research without compromising the speed of their high-performance solvers.

Speaker: Corentin Lapeyre (NVIDIA)

11:10 Coffee break

11:40 Place holder PT / RT

12:25 Closing words / farewell

12:35 Lunch