Orateur
Description
This talk will present a summary of recent activities within Fink, an alert broker research platform. The primary focus of this contribution has been on preparing the production environment using Kubernetes as the target platform.
Software Stack Packaging
The initial goal was to package the entire software stack of Fink. This involved packaging key components (fink-broker, fink-alert-simulator, and finkctl), ensuring their readiness for deployment within a Kubernetes environment. This entailed creating Docker containers for individual components and defining necessary configurations.
Integration with Kafka, Minio-S3, and Spark for Kubernetes
Following successful software stack packaging, efforts were directed towards integrating essential technologies such as Kafka, Minio-S3, and Spark, all critical for Fink's operation. This integration facilitated seamless communication between components and streamlined the validation of the Fink software pipeline.
Implementation of an OpenStack Runner for Scalable Builds
To ensure the efficient scalability of Fink, collaboration with the IJClab team led to the implementation of a dedicated OpenStack runner for builds. This infrastructure supported effective scalability, enabling comprehensive testing of Fink with machine learning algorithms, which represent approximately 10GB of code.
Future Projects
For future endeavors, two key aspects are planned:
a. Continuous Deployment with ArgoCD: Implementation of continuous deployment using ArgoCD, a famous deployment management tool used by LSST. This will ensure seamless and secure software updates. Collaboration with IJCLab team is also planned to enhance the software release creation system.
b. Telemetry: Focus will shift towards traces and metrics monitoring, involving the setup of tools and dashboards to monitor the performance and availability of Fink.
Conclusion
During this period, several significant milestones were achieved for Fink, including software stack packaging, integration with key technologies, automated testing setup, build process optimization, and the preparation of a Kubernetes-based production environment. In 2024, further contributions will be made to the project, with an emphasis on continuous deployment and telemetry. These efforts aim to enhance the efficiency and reliability of Fink's operations on the CC-IN2P3 cloud platform.
