Choisissez le fuseau horaire
Le fuseau horaire de votre profil:
6th DIRAC project Face 2 Face meeting
→
Europe/Paris
Salle des Séminaires (LUPM)
Salle des Séminaires
LUPM
LUPM – Bâtiment 21
Université de Montpellier – Campus Triolet
Place Eugène Bataillon – CC 72
34095 Montpellier Cédex 05 FRANCE
Description
- Réunion face à face à Montpellier (LUPM)
Inscription
Participants
---------------------------------
Update projet Andrei
+ Développements
HTTPS Clients/Server framework
XML-RPC (au début) -> DISET (15 ans) -> HTTPS (aujourd'hui)
Service migration in progress but
-> still an issue with lack of support for file upload (Dirac StorageElement and SandboxStore)
+ Current implemenation 8.0 "Common BaseRequestHandler in the Tornado framework"
-> complicated but overly universal and required to fork Tornado to be compatible with M2Crypto
=> might require to find a new framework to replace Tornado to enable good streaming
*open discussion, ideas welcome*
+ HTTPS is good: single component per host, natural load balancing, container installation
but complicated logging for it requires a central service logging (ELK stack)
Acces to HTCondorCE and ARC with tokens
CloudComputingElement replaced VMDIRAC:
simpler but less universal, pragmatic approach
cloud admin not agains using Application Credentials : good news!
NB : no recent developments on HPC
Tokens for storage : work not started, a lot of thinking needed
+ EGI/FG services
running smoothly since beginning of the year : 2 M jobs
WMS history broken !
servers running DIRAC 8.0 (but one old REST interface for Complex VO)
more services to be moved to HTTPS (for token pressure)
Galaxy : discussions restarted slooowwwly
+ Projet IN2P3: 6 laboratoires, 11 ingénieurs mais 2 FTE au total en 2023
participation du CC augmente !
-> Vanessa devient responsable technique du projet IN2P3
+ DIRAC Community Workshop at the EGI conference 2023
June 2023, scheduled 10 min DIRAC service overview talk (Gino)
90 minutes training/demo sessions : goal is jupyter nb with tokens
+ CHEP 2023 : DIRAC (Federico) and CTA PS (Alice) - with proceedings
+ ISGC march 2023 : OAuth work - Andrei - proceedings being written
---------------------------------
+ Biomed Update - Sorina
Vue DIRAC : ~5 utilisateurs
mais utilisation via VIP ~ 100 utilisateurs actifs pour 1400 enregistrés
=> 80 années CPU
DMS : utilisation de 3 SE de confiance
-> difficulté de connaitre l'occupation des disques : DFC size -l
mais ne donne pas de manière sûre l'espace disponible
-> "drm -r" ne supprime plus les répertoires ?
Udocker et CVMFS
déploiement des applications
container déployé sur cvmfs, et lancer via udocker (git clone udocker)
VOMS -> EGI check-in
envisager la suite de voms = EGI check-in
mapping des sous-groupe check-in avec les groupes Dirac
---------------------------------
Production System for CTADIRAC - Alice - 11h25 -> 11h45
new YAML based interface
input/output of each transformation through MD
set parent/child, MD verifications
CWL : common workflow language, trough cwltool
run the same workflow locally and in a distributed environment
cwltool + code to translate CWL workflow into a transformation
-> more thought needed to go from TS to PS
other backends exist and are supported by different frameworks
https://www.commonwl.org/implementations/
---------------------------------
DIRAC Deployment on Kubernetes - Natthan - 11h45 + 20'
Ideas
1. create a Dockerfile with DIRAC+CTADIRAC ++
Entrypoint : script executed when running the container to launch a specific service
2. Helm Charts
Describes the Docker configuration -> one template for all services
Limitations
"Agents" (or may be "executers") cannot run on Kubernetes due to the tasks queuing framework
-> not clear if there is a significant issue
"Logs" of all HTTPS services running are regroup under the Tornado log file
-> possibility to separated logs?
Outlook for CTA
CTA has a dev K8s cluster at DESY
RUCIO is already running on the K8s cluster
CTADIRAC expected to move to K8s
---------------------------------
DIRAC-IN2P3 - Vanessa - 12h05
Ferme de tests montés en recyclant hardware des workers
Instance DIRAC EGI/FG
2 fermes : ARC-CE on Slurm pour utilisateurs + LSST
HTCondor-CE -> grille + ATLAS
Configuration CONDOR
use Tokens : needed debug but successful... but problems with "groups" not available
Configuration ARC-CE
possible to use Tokens and define groups... but to be tested
Configuration of DIRAC@CC-IN2P3
need to declare 2 CE defined, one for token and one for certificates
-> pilots running !
---------------------------------
OIDC/OAuth2 Framework - Andrei
AAI Components : Dirac Client/Service, Token Management, User Management, Connection to ressources
Workflow
user login : through "dirac-login" but transmit to multiple identity providers, through an Authorization Server
1. choose identify provider
2. Authenticate within Checkin
3. Consent Screen
4. Choose Dirac Group
5. Return to the terminal
Issue of Token to DIRAC group mapping : claims on user profiles used by IdP
-> need to add group information to the token : WIP @ Check-in
Token Manager
provides token to users : access and refresh tokens
provides token to other DIRAC components as needed (e.g. pilot user token)
User Management
moving towards complete *dynamic* user management by IdP services
one to tone correspondance of DIRAC groups and IdP claims/scope
---------------------------------
DIRAC Logs - Bertrand
Logs now in Tornardo, all to the same stream/file
-> need to push logs to Fluentd/Fluent-bit, then to ElasticSearch to Kibana
Configuration: input, output, filtres, parser
Visualisation des logs dans Kibana, e.g. sélection par Composant, recherche de texte possible
Grafana -> essais pas très concluant
Andrei mentione le problème de la nécessité de conservation des logs pendant 6 mois
Autre question de savoir comment on gère les différents niveaux de logs de manière opérationnelle
---------------------------------
Projet Européens - Gino
1. INFRA2023-TECH-01-01 - GreenDIFIT
implement resource optimisation approach in orchestration frameworks
-> 18PM CDD WP7 tests and validation
-> 3x5PM pour CPPM,IPHC, CC-IN2P3
2. PEPR Cloud - STEEL
adaptation dynamique des traitements des flux de données
-> 1 CDD IR 12PM
Il y a un compte-rendu associé à cet événement.
Les afficher.
