Releases: dmwm/WMCore
WMCore 2.4.5.3 production WMAgent release
This WMAgent patch release brings in a fix for the condor2 bindings:
- Update QueryOpts.SummaryOnly call to be compatible with htcondorv2 25.0.3
It also introduces a new way to parse cvmfs releasesMap:
- Parse cvmfs releasesMap file into xml file
Complete release cycle notes can be found at: https://github.com/dmwm/WMCore/releases/tag/2.4.5 and at the previous two tag releases.
WMCore 2.4.5.2 production WMAgent release
What's Changed
This WMAgent patch release brings in a bug fix
Complete release cycle notes can be found at:
https://github.com/dmwm/WMCore/releases/tag/2.4.5
Full Changelog: 2.4.5.1...2.4.5.2
Contributors
Assets 2
WMCore 2.4.5.1 production WMAgent release
This WMAgent patch release brings in a new version of htcondor python bindings, including the migration to its API v2, as provided by the pull request #12464
Complete release cycle notes can be found at: https://github.com/dmwm/WMCore/releases/tag/2.4.5
WMCore 2.4.5 production WMAgent release
This is planned as the final WMAgent stable release, triggering the WMCore code freeze.
This release features several significant software stack updates, including the upgrade of the dmwm-base image to Debian Trixie (with new OpenSSL/Curl/libcurl libraries), an update to Oracle InstantClient 23.9, an upgrade to MariaDB 11.8.2, and the replacement of cx-Oracle with python-oracledb.
It also provides a new component called AgentWatchdog, which monitors all other worker threads and restarts unhealthy ones. This release concludes development on flexible job match-making requirements by introducing the DESIRED_ExtraMatchRequirements job classad.
Finally, important bug fixes are included, making JobSubmitter more resilient to CMSSDT failures, correcting the GPU classads data type, and refactoring stage-out timeout actions.
Thank you to everyone who contributed to this last version, and to the project overall. May WMAgent continue to meet the requirements of CMS and efficiently delivering data to the collaboration."
Release date: Oct 21, 2025.
Changes since release: 2.4.3.
WMAgent
Software stack
- New dmwm-base image based on Debian Trixie dmwm/CMSKubernetes#1643
- Improve interactive DB sessions formating for Oracle based WMAgents (Todor Ivanov) dmwm/CMSKubernetes#1674 and dmwm/CMSKubernetes#1675
- Enable python-oracledb ThickMode + sqlalchemy 1.4 (Todor Ivanov) #12451
- Add --skip-ssl to all mariadb executable functions at manage-common.sh CMSKubernetes#1666
- Update Oracle instantclient to 23.9 CMSKubernetes#1668
- Upgrade Oracle client for wmagent CMSKubernetes#1670
- Upgrading MariaDB to version 11.8.2 dmwm/CMSKubernetes#1640
Features and/or feature changes
- Rely on TagCollector caching in JobSubmitter (Alan Malta Rodrigues) #12449
- Providing a new component to watch components/worker threads: AgentWatchdog (Todor Ivanov) #12382 #12445 #12456 #12456
- WMAgent: Remove restartComponent cronjob CMSKubernetes#1671
- Add support to user-provided match requirements at job creation/execution (Alan Malta Rodrigues) #12447
- harvesting task maxpss configured in Express only (Antonio Linares) #12404
- Set Harvest memory requirements in the JobFactory (Alan Malta Rodrigues) #12458
Bug Fixes
- Revisit timeout alarm during StageOut, ensuring child process is terminated (Alan Malta Rodrigues) #12453
- Fix failed calls of AgentStatusPoller.collectAgentInfo() for down components (Todor Ivanov) #12448
- Unquote gpu related classads to be compatible with new requirements format: (Kenyi Hurtado) #12440
Enhancements
- Update GPU templates with extra match-making requirements (Alan Malta Rodrigues) #12447
WMCore 2.4.4 production central services release
This is planned to be the last WM central services stable release for WMCore code freezing. The base image has been upgraded to Debian Trixie (bringing in a new OpenSSL and Curl/libcurl library versions).
A new workflow parameter has been added, called JobExtraMatchRequirements, which can be used to propagate extra job match-making requirements from the workflow down to the job description in HTCondor pool (note that full functionality depends on WMAgent 2.4.5+).
We have also upgraded the Harvesting PSS memory watchdog default to 3GB - in sync with runtime memory request available in WMAgent 2.4.5+. Lastly, it also allows now Extension value to be used for the workflow SubRequestType parameter.
Release date: Oct 13, 2025.
Changes since release: 2.4.3.
Central Services
Software stack
- New dmwm-base image based on Debian Trixie dmwm/CMSKubernetes#1643
Features and/or feature changes
- add Extension as SubRequestType (vlimant) #12442
- Default harvesting memory/maxpss to 3GB (Antonio Linares) #12404
- Add support to JobExtraMatchRequirements task parameter (Alan Malta Rodrigues) #12417
Bug Fixes
- Make NanoAOD input data chunk policy consistent for RelVal workflows (Alan Malta Rodrigues) #12434
Enhancements
WMCore 2.4.3 production central services release
This release brings in several updates to third-party packages, such as rucio-clients, CherryPy, psutil, httplib2, etc.
We highlight major changes to MSUnmerged as well, as it now shares the same Debian+Python 3.12 base image as the other WM services, with up-to-date Gfal2 and python-gfal2 libraries and concurrent file/directory cleanup.
Lastly, several bug fixes and optimizations have been included in this release.
Release date: Aug 19, 2025.
Changes since release: 2.4.0.
WMAgent
Software stack
- Update third party python packages (Kenyi Hurtado) #12424
- Added mongomock to requirements (Dennis Lee) #12413
- Update GFAL and MSUnmerged images to Debian Bookworm + Miniconda + Py3.12 dmwm/CMSKubernetes#1627
- update CMSMonitoring version to upgrade jsonschema dependency to align with rucio-clients (Valentin Kuznetsov) #12412
- update rucio client version to 36.5.0 (Valentin Kuznetsov) #12201
Features and/or feature changes
- Replace deleteRule with updateRule with lifetime=0 (Kenyi Hurtado) #12384
- Rename GPU related parameters and split CUDACapability, following (Kenyi Hurtado) #12416
- MSUnmerged refactoring for decreased memory footprint and concurrent deletions through thread pool (Alan Malta Rodrigues) #12059
Bug Fixes
- Fix ReqMgr2 data type validation (Alan Malta Rodrigues) #12423
- Update Reqmgr workload cache for dynamic workflow parameter changes. (Todor Ivanov) #12405
- Install setuptools in the CD pipeline (Alan Malta Rodrigues) #12411
- Update CD python version to 3.12; Skip PyPi publication if same package+version has already been uploaded (Alan Malta Rodrigues) #12410
Enhancements
- Remove unused DQMCat module (Alan Malta Rodrigues) #12341
WMCore 2.4.2 production WMAgent release
This release contains a backward incompatible model for WMAgent SQL schema management, now depending on a new wmcoredb python package. In addition, it brings in several updates to third-party packages, such as rucio-clients, CherryPy, psutil, httplib2, etc.
MSUnmerged deserves to be highlighted as well, as it now shares the same Debian+Python 3.12 base image as the other WM services, with up-to-date Gfal2 and python-gfal2 libraries. Lastly, several bug fixes and optimizations have been included in this release.
Release date: Aug 15, 2025.
Changes since release: 2.4.1.
WMAgent
Software stack
- Update third party python packages (Kenyi Hurtado) #12424
- Add SQL schema wmcoredb package as a dependency (Alan Malta Rodrigues) #12408
- Added mongomock to requirements (Dennis Lee) #12413
- Update GFAL and MSUnmerged images to Debian Bookworm + Miniconda + Py3.12 dmwm/CMSKubernetes#1627
- update CMSMonitoring version to upgrade jsonschema dependency to align with rucio-clients (Valentin Kuznetsov) #12412
- update rucio client version to 36.5.0 (Valentin Kuznetsov) #12201
Features and/or feature changes
- New module and mechanism to monitor CouchDB process and replications (Valentin Kuznetsov) #12353 #12425 #12426
- Replace deleteRule with updateRule with lifetime=0 (Kenyi Hurtado) #12384
- Rename GPU related parameters and split CUDACapability, following (Kenyi Hurtado) #12416
- New model for deploying SQL schema; adopted wmcoredb in WMInit; update Oracle DAOs to use Oracle's auto-increment; etc (Alan Malta Rodrigues) #12341
- Remove Python-based CreateWMBSBase/Create/Destroy modules (Alan Malta Rodrigues) #12341
- MSUnmerged refactoring for decreased memory footprint and concurrent deletions through thread pool (Alan Malta Rodrigues) #12059
Bug Fixes
- Fix ReqMgr2 data type validation (Alan Malta Rodrigues) #12423
- Fix WMAGENT_CONFIG exports (Todor Ivanov) #12400
- Skip None checksum insertion for wmbs_file_checksums; many other bug fixes (Alan Malta Rodrigues) #12341
- Update Reqmgr workload cache for dynamic workflow parameter changes. (Todor Ivanov) #12405
- Install setuptools in the CD pipeline (Alan Malta Rodrigues) #12411
- Update CD python version to 3.12; Skip PyPi publication if same package+version has already been uploaded (Alan Malta Rodrigues) #12410
- fix bug checkProxy incompatible (Viphava280444) #12403
Enhancements
- Remove unused DQMCat module (Alan Malta Rodrigues) #12341
WMCore 2.4.1 production WMAgent release
This is the first release of WMAgent running python 3.12.
Moreover, this release includes a new way to monitor status of components and worker threads, based on psutils library and standard Linux mechanisms.
Lastly, Local WorkQueue is expected to be smaller, since the multiplies factor has been reduced from .75 to .50
Release date: Jun 18, 2025.
Changes since release: 2.3.11.1.
WMAgent
Software Stack
- python 3.12 - necessary requirements.txt upgrades (Dario Mapelli) #12336
- python 3.12 - do not use codecs (Dario Mapelli) #12338
this release also required the following changes to the dmwm/CMSKubernetes repository:
- wmcore - switch to py3.12 for pypi docker images dmwm/CMSKubernetes#1611
- cmsweb/exporters - build a new version dmwm/CMSKubernetes#1613
- dmwm-base - remove pycurl from OS && add procps: dmwm/CMSKubernetes#1622, dmwm/CMSKubernetes#1621
Features and/or feature changes
- [monitoring] add thread info status to WMAgent info pushed to CouchDB (Valentin Kuznetsov) #12302
- [monitoring] fix issues with WMagent start; and couple of major warnings reported by pylint (Valentin Kuznetsov) #12362
- [monitoring] Change look-up states for process status (Valentin Kuznetsov) #12372
- [monitoring] setup default value for cycle_time (Valentin Kuznetsov) #12372
- [monitoring] provide necessary checks to look-up valid process ID status (Valentin Kuznetsov) #12387
- Decrease site slot multiplier factor from .75 to .50 (Alan Malta Rodrigues) #12365
- Add getCodes method to fetch all dbs2go codes from its error JSON response (Valentin Kuznetsov) #12312
bug fixes
- Force cherrypy to set content-length=0 before request body if FE does not provide it (Valentin Kuznetsov) #12371
- fix issue with restartComponent.sh script (Valentin Kuznetsov) #12352
- [stageout] Default first stage out to X509 in case bearer token is not set (Alan Malta Rodrigues) #12380
- [stageout] Do not hang on unset BEARER_TOKEN_FILE env var (Alan Malta Rodrigues) #12380
- [stageout] Avoid token-based file deletion in GFAL2, unless environ is defined (Alan Malta Rodrigues) #12377
Enhancements
- Update checkProxy script to use valid domain name (Alan Malta Rodrigues) #12392
- Add Oracle DatabaseError DPI-1080 as a soft error (Alan Malta Rodrigues) #12379
- Make JobAccountant output module evaluation more verbose (Alan Malta Rodrigues) #12351
- Avoid crashing ErrorHandler on failed DNS resolution (Alan Malta Rodrigues) #12369
- [venv] Split patch files by source files edited && Apply them separately (Todor Ivanov) #12346
- [venv] Add option to chose custom built python (Todor Ivanov) #12381
WMCore 2.4.0 production central services release
This new Central Services tag mainly brings in a new version of the python runtime, python 3.12. Moreover, we increased the target job length from 8h to 12h.
Release date: Jun 17, 2025.
Changes since release: 2.3.10.3.
Central services
Software stack
- python 3.12 - necessary requirements.txt upgrades (Dario Mapelli) #12336
- python 3.12 - do not use codecs (Dario Mapelli) #12338
- switch from distribution.data_files to distribution.package_data (Valentin Kuznetsov) #12359
- Replaces nose with pynose, remove nose2 in requirements (Dennis Lee) #12390-
deployment
this release also required the following changes to the dmwm/CMSKubernetes repository:
- wmcore - switch to py3.12 for pypi docker images dmwm/CMSKubernetes#1611
- cmsweb/exporters - build a new version dmwm/CMSKubernetes#1613
- wmcore - load all CAs from k8s host dmwm/CMSKubernetes#1620
- dmwm-base - remove pycurl from OS && add procps: dmwm/CMSKubernetes#1622, dmwm/CMSKubernetes#1621
Features and/or feature changes
- Switch default job length from 8 to 12h (Alan Malta Rodrigues) #12360
Bug Fixes
- Force cherrypy to set content-length=0 before request body if FE does not provide it (Valentin Kuznetsov) #12371
Enhancements
- Further updates to the JSON templates, fix some comments and resource requirements (Alan Malta Rodrigues) #12354
- Fix bash typoed variable (Fredrik) #12364
- Yuiserver was removed in 2013 (#4495) (Fredrik) #12356
WMCore 2.3.11.1 production WMAgent release
This is the first WMAgent stable release built with the setuptools library, replacing distutils (required for the overall project compliance with PEP-625). Further on the software stack, htcondor, pycurl and PyOpenSSL libraries have been updated.
This release brings in the CPU microarchitecture job feature, some important token and stage out developments and required changes to move towards Python 3.12. Please check out the full release notes for further information on other developments, bug-fixes and optimizations.
Release date: May 2, 2025.
Changes since release: 2.3.9.2.
WMAgent
Software stack
- Upgrade htcondor python bindings to v24.0.7 (Todor Ivanov) #12344
- Update pycurl and PyOpenSSL versions (Kenyi Hurtado) #12286
- Switch WM packaging from distutils to setuptools (Valentin Kuznetsov) #12259 #12273 #12274 #12275 #12276 #12277 #12281
Features and/or feature changes
- Get rid of transactionId from the job configuration, and GlobalTagTransaction from Express specs (Kenyi Hurtado) #12331
- Performance enhancements for primary, parent and pileup data location in global workqueue (Alan Malta Rodrigues) #12330
- Addition of token support for gfal and xrootd, with refactoring of createCopyCommand and createRemoveCommand (Andrea Piccinelli) #12218
- Support nano flavours (Antonio Linares) #12235
- Add support for microarchitecture. When default microarch is not defined in XML file, use "0" as default. (Kenyi Hurtado) #12270
- python 3.11 - inspect.getargspec moved to inspect.getfullargspec (Dario Mapelli) #12289
- Replace schedd.Transaction and queue_with_itemdata() with Schedd.submit() (Kenyi Hurtado) #12287
- Introducing createDebuggingCommand in all the stageout submodules (Andrea Piccinelli) #12284
- python 3.10 - collections.(Mapping|Iterable) moved to collections.abc.(Mapping|Iterable) (Dario Mapelli) #12288
- Replaced ddm_quota with dm_weight, added deprecation warning (Dennis Lee) #12243
- Add token authentication support to job submission, but disable cms token authentication by default. (Kenyi Hurtado) #12196
Bug Fixes
- Properly HandleDidNotfound exception (Todor Ivanov) #12317
- Fix location of rel_microarchs variable in SimpleCondorPlugin (Alan Malta Rodrigues) #12305
Enhancements
- deploy wmange-venv - small changes (Dario Mapelli) #12337
- Code organization for global workqueue data location update (Alan Malta Rodrigues) #12330
- Few typos in the help message (Fredrik) #12322
- Add rucio client instances to wmagent-component-standalone (Todor Ivanov) #12318
- Refactor all JSON workflow templates (Alan Malta Rodrigues) #11949
- Add deploy-unittest-venv.sh (Todor Ivanov) #12310
- Remove unused functions (since 2015 and #5515) (Fredrik) #12300
- Remove unused double stack functions, see #12285 (Fredrik) #12292
- Remove slack from link text (Fredrik) #12268
- Fix logging typo (Fredrik) #12258