Réunion Dirac Face 2 Face 28 Novembre 2017
-----------------------------------------
Update Dirac
+ v6r19p7 assez mûre
gLogger based on python standard logging module
Site and CE/queue status manages by RSS
ExtraJDLParameters CREAM CE option => GPU
-> gridPP, FG et LHCb l'utilise déjà
+ on-going work
M2Crypto replacement to pyGSI
Pilot3
Cloud : OCCI native (FedCloud EGI), Pilot3
+ python 2 -> python 3
workshop dédié ?
analyse du code et planification nécessaire
expérience LSST ?
CTADIRAC -> python 3
+ v6r20 release certification
FTS3 based tools, non backward compatibility FTS2
Pilot's logging through MQ
Transactional bulk submission (job per job submission too slow)
StopSigXXX signal in the job JDL
Singularity Computing Element : Simon (UK)
-> pilot prepares jobwrapper -> starts generic container and runs application
+ PR pylint, pep8, pytest, jenkins, codecoverage etc...
+ Belle 2
CC-IN2P3 monitore la situation computing, pas encore d'engagement
Japonais utilise Dirac pour simus
projets de développements possibles : Cloud, BOINC, Data Management
+ Dirac User Workshop 22-24/25 Mai au CC-IN2P3
+ DIRAC4EGI smooth running
developments:
-> verification automatique certification VOMS
Eiscat3D file catalog extension
pilot scheduling with priorities for certain sites
multi-core vs whole node
intégration dans EGI infrastructure (GGUS, service monitoring)
formal agreement in preparation for 2017
+ FG-Dirac smooth running
pb client/server version with user proxies
MariaDB Galera cluster -> single writable node
-----------------------------------------
MariaDB, Galera cluster
+ config "finale" 1 noeud en écriture, CTA et FG-Dirac chacun sur un serveur
-> on ne touche plus à rien
-----------------------------------------
VMDIRAC
+ LUPM en attente de nouvelles "HNSciCloud"
-> voir prochaine phase de test avec ressources significatives
-> éventuellement développement plugin pour cloud IBM
+ nécessité de revoir les interfaces
car historiquement largement basée sur l'expérience de libcloud
probablement possibilité de faire une interface plus légère
(cf. CloudEndPoint.py etc...)
-> VMDIRAC3 avec code plus light
+
-----------------------------------------
HPC centers and gateways
+ HPC Center en Finlande - Federico
Test Dirac gateway = server proxy sur un gateway du centre
Federico mis à jour le gateway -> v6r19
mais pas de tests extensifs
-> test StorageElementProxy et Gateway Proxy
-----------------------------------------
Job Tags
+ Use Case CTA at CC-IN2P3 : tag SL6 and CentOS 7 queues
test show that for jobs with tags : pilots are not launched
but when jobs with no tags are submitted, pilot start and match jobs with tags correctly
-> probably a small bug to be fixed...
-> note that gridPP use tags for GPU queues
-> use *v6r19p7*
-----------------------------------------
TransformationSystem / ProductionSystem
+ refactor TransformationSystem
need to decouple "inputdataagent"
introduce concept of plugin
make it much lighter with basic concept of "data driven transformation"
+ LUPM wish to develop a prototype to use for CTA PROD4 (spring 2018)
and present concept, prototype and use case at CHEP 2018
+ pb output data have a given list of meta data
Transformation needs an input as as input data query
+ work needed on MetaQuery
-----------------------------------------
COMDIRAC
+ Andrei comments on paper daft
need to insist on the global culture of "interface"
need to detail work done on optimization of command lines via the concept of session
potentially interesting part could be "user" feedback
-----------------------------------------
Publication
+ Transformation System
Luisa works on detailed skeleton
-----------------------------------------
Conferences - faire une liste complète pour 2018
CHEP 2018
ACAT ?
ESPIE ?
ADASS ?
-----------------------------------------
Prochaine réunion en Janvier
Johan fait un doodle sur les mercredi matin.