REL: 3.4.0 Release

this commit contains the entries in changelog and NEWS
update the linux package documentation
adds upgrade scripts for removed/renamed bots
and sets the version number

Author: sebix

CHANGELOG.md

@@ -6,8 +6,8 @@
 # CHANGELOG
 
 
-3.3.2 (unreleased)
-------------------
+3.4.0 Feature release (2025-03-06)
+----------------------------------
 
 ### Configuration
 
@@ -16,10 +16,6 @@
 - Python 3.8 or newer is required (PR#2541 by Sebastian Wagner).
 - `intelmq.lib.utils.list_all_bots`/`intelmqctl check`: Fix check for bot executable in $PATH by using the bot name instead of the import path (fixes #2559, PR#2564 by Sebastian Wagner).
 
-### Development
-
-### Data Format
-
 ### Bots
 #### Collectors
 - `intelmq.bots.collectors.shadowserver.collector_reports_api.py`:
@@ -30,8 +26,6 @@
   - Log the downloaded size in bytes to ease troubleshooting (PR#2554 by Sebastian Wagner).
   - Fix import for Timeout exception preventing another exception (fixes #2555, PR#2556 by Sebastian Wagner).
 - Remove `intelmq.bots.collectors.twitter` as it uses an unmaintained library and does not work any more (fixes #2346, #2441, PR#2568 by Sebastian Wagner).
-- Renamed `intelmq.bots.parser.twitter` to `intelmq.bots.parser.ioc_extractor` (PR#2568 by Sebastian Wagner).
-  - Added `intelmq.bots.parser.twitter` as a stub to load the IoC Extractor parser.
 
 #### Parsers
 - `intelmq.bots.parsers.shadowserver._config`:
@@ -40,18 +34,20 @@
   - Fix to avoid schema download if not configured #2530.
 - `intelmq.bots.parsers.misp.parser`: Replace deprecated datetime function `utcfromtimestamp` for Ubuntu 24.04 compatibility (PR#2577 by Sebastian Wagner, fixes #2576, #2571).
 - `intelmq.bots.parsers.cleanmx.parser`: Replace deprecated datetime function `utcfromtimestamp` for Ubuntu 24.04 compatibility (PR#2577 by Sebastian Wagner, fixes #2576, #2571).
+- Renamed `intelmq.bots.parsers.twitter` to `intelmq.bots.parser.ioc_extractor` (PR#2568 by Sebastian Wagner).
+  - Added `intelmq.bots.parsers.twitter` as a stub to load the IoC Extractor parser.
 
 #### Experts
 - `intelmq.bots.experts.securitytxt`:
-  - Added new bot (PR#2538 by Frank Westers and Sebastian Wagner)
-- `intelmq.bots.experts.misp`: Use `PyMISP` class instead of deprecated `ExpandedPyMISP` (PR#2532 by Radek Vyhnal)
+  - Added new bot (PR#2538 by Frank Westers and Sebastian Wagner).
+- `intelmq.bots.experts.misp`: Use `PyMISP` class instead of deprecated `ExpandedPyMISP` (PR#2532 by Radek Vyhnal).
 - `intelmq.bots.experts.fake.expert`: New expert to fake data (PR#2567 by Sebastian Wagner).
 
 #### Outputs
 - `intelmq.bots.outputs.cif3.output`:
   - The requirement can only be installed on Python version < 3.12.
   - Add a check on the Python version and exit if incompatible.
-  - Add a deprecation warning (PR#2544 by Sebastian Wagner)
+  - Add a deprecation warning (PR#2544 by Sebastian Wagner).
 - `intelmq.bots.outputs.sql.output`:
   - Treat an empty string `fields` parameter as unset parameter, fixing a crash in default configuration (PR#2548 by Sebastian Wagner, fixes #2548).
 
@@ -61,6 +57,7 @@
 - Remove empty page tutorials/intelmq-manager (PR#2562 by Sebastian Wagner).
 
 ### Packaging
+- Packages for Ubuntu 24.04 (by Sebastian Wagner, fixes #2571).
 
 ### Tests
 - Install build dependencies for `pymssql` on Python 3.8 as there are no wheels available for this Python version (PR#2542 by Sebastian Wagner).
@@ -70,11 +67,21 @@
 - Full pytest workflow: Version-independent install of postgres client, for Ubuntu 24.04 (default on GitHub now) test environment compatibility (PR#2557 by Sebastian Wagner).
 - Debian package build workflow: Use artifact upload v4 instead of v3 (PR#2565 by Sebastian Wagner).
 
-### Tools
-
-### Contrib
-
 ### Known issues
+This is short list of the most important known issues. The full list can be retrieved from [GitHub](https://github.com/certtools/intelmq/labels/bug?page=2&q=is%3Aopen+label%3Abug).
+- intelmqctl: interactive run ignores custom log level (#2563).
+- `intelmq.parsers.html_table` may not process invalid URLs in patched Python version due to changes in `urllib` (#2382).
+- Breaking changes in 'rt' 3.0 library (#2367).
+- Type error with SQL output bot's `prepare_values` returning list instead of tuple (#2255).
+- `intelmq_psql_initdb` does not work for SQLite (#2202).
+- intelmqsetup: should install a default state file (#2175).
+- Misp Expert - Crash if misp event already exist (#2170).
+- Spamhaus CERT parser uses wrong field (#2165).
+- Custom headers ignored in HTTPCollectorBot (#2150).
+- intelmqctl log: parsing syslog does not work (#2097).
+- Bash completion scripts depend on old JSON-based configuration files (#2094).
+- Bots started with IntelMQ-API/Manager stop when the webserver is restarted (#952).
+- Corrupt dump files when interrupted during writing (#870).
 
 
 3.3.1 (2024-09-03)

NEWS.md

@@ -1,5 +1,5 @@
 <!-- comment
-   SPDX-FileCopyrightText: 2015-2024 Sebastian Wagner
+   SPDX-FileCopyrightText: 2015-2025 Sebastian Wagner
    SPDX-License-Identifier: AGPL-3.0-or-later
 -->
 
@@ -10,26 +10,23 @@ This file lists all changes which have an affect on the administration of IntelM
 Please refer to the change log for a full list of changes.
 
 
-3.3.2 Bugfix release (unreleased)
----------------------------------
+3.4.0 Feature release (2025-03-05)
+----------------------------------
 
 ### Requirements
 Python 3.8 or newer is required.
 
 ## Bots
-#### CIF 3 API
+#### CIF 3 API Ouput deprecation
 The CIF 3 API Output bot is not compatible with Python version greater or equal to 3.12 and will be removed in the future due to lack of maintenance.
 See https://lists.cert.at/pipermail/intelmq-users/2024-December/000474.html for more information.
 
-### Tools
-
-### Data Format
-
-### Configuration
+#### Twitter Collector removal
+As the bot does not work anymore and uses an unmaintained library, it is removed from IntelMQ.
 
-### Libraries
-
-### Postgres databases
+#### Twitter Parser renaming
+The Twitter parser is renamed to *IoC Extractor Parser* (`intelmq.bots.parsers.ioc_extractor`).
+The previous module name is left as a stub to load the IoC Extractor parser for backwards-compatibility.
 
 
 3.3.1 Bugfix release (2024-09-03)

debian/changelog

@@ -1,8 +1,8 @@
-intelmq (3.3.2~alpha1-1) UNRELEASED; urgency=medium
+intelmq (3.4.0-1) stable; urgency=medium
 
-  * Prepare 3.3.2 release
+  * 3.4.0 Bugfix release
 
- -- Sebastian Wagner <sebix@sebix.at>  Mon, 16 Sep 2024 17:53:59 +0200
+ -- Sebastian Wagner <sebix@sebix.at>  Wed, 05 Mar 2025 21:33:14 +0100
 
 intelmq (3.3.1-1) stable; urgency=medium
 

docs/admin/installation/linux-packages.md

@@ -1,5 +1,5 @@
 <!-- comment
-   SPDX-FileCopyrightText: 2015-2024 Sebastian Wagner, Filip Pokorný
+   SPDX-FileCopyrightText: 2015-2025 Sebastian Wagner, Filip Pokorný
    SPDX-License-Identifier: AGPL-3.0-or-later
 -->
 
@@ -20,6 +20,7 @@ Native packages are currently provided for the following Linux distributions:
 - **openSUSE Tumbleweed**
 - **Ubuntu 20.04** (focal fossa)
 - **Ubuntu 22.04** (jammy jellyfish)
+- **Ubuntu 24.04** (jammy jellyfish)
 
 ### Debian 11 and 12
 
@@ -47,7 +48,7 @@ zypper refresh
 zypper install intelmq intelmq-api intelmq-manager
 ```
 
-### Ubuntu 20.04 and 22.04
+### Ubuntu 20.04, 22.04 and 24.04
 
 For Ubuntu you must enable the Universe repository which provides community-maintained free and open-source software.
 

intelmq/lib/upgrades.py

@@ -41,6 +41,7 @@
            'v320_update_turris_greylist_url',
            'v322_url_replacement',
            'v322_removed_feeds_and_bots',
+           'v340_deprecations'
            ]
 
 
@@ -952,6 +953,27 @@ def v322_removed_feeds_and_bots(configuration, harmonization, dry_run, **kwargs)
     return '\n'.join(messages) if messages else None, configuration, harmonization
 
 
+def v340_deprecations(configuration, harmonization, dry_run, **kwargs):
+    """
+    Rename twitter parser, warn on Twitter collector
+    """
+    changed = False
+    found_twitter_collector = []
+    message = None
+    for bot_id, bot in configuration.items():
+        if bot_id == 'global':
+            continue
+        if bot["module"] == "intelmq.bots.parsers.twitter.parser":
+            configuration[bot_id]["module"] = "intelmq.bots.parsers.ioc_extractor.parser"
+            changed = True
+        elif bot["module"] == "intelmq.bots.collectors.twitter.collector":
+            found_twitter_collector.append(bot_id)
+
+    if found_twitter_collector:
+        message = f"Found discontinued Twitter collector bot: {', '.join(found_twitter_collector)}"
+    return message or changed, configuration, harmonization
+
+
 UPGRADES = OrderedDict([
     ((1, 0, 0, 'dev7'), (v100_dev7_modify_syntax,)),
     ((1, 1, 0), (v110_shadowserver_feednames, v110_deprecations)),
@@ -981,7 +1003,7 @@ def v322_removed_feeds_and_bots(configuration, harmonization, dry_run, **kwargs)
     ((3, 2, 2), (v322_url_replacement, v322_removed_feeds_and_bots)),
     ((3, 3, 0), ()),
     ((3, 3, 1), ()),
-    ((3, 3, 2), ()),
+    ((3, 4, 0), (v340_deprecations, )),
 ])
 
 ALWAYS = (harmonization,)

intelmq/tests/lib/test_upgrades.py

@@ -598,6 +598,26 @@
 Found discontinued feeds collected by bots: sucuri-collector
 Remove the affected bots from the configuration."""
 
+V340_TWITTER_PARSER_IN = {
+    "global": {},
+    "twitter-parser": {
+        "module": "intelmq.bots.parsers.twitter.parser",
+    },
+}
+V340_TWITTER_PARSER_OUT = {
+    "global": {},
+    "twitter-parser": {
+        "module": "intelmq.bots.parsers.ioc_extractor.parser"
+    },
+}
+V340_TWITTER_COLLECTOR_IN = {
+    "global": {},
+    "twitter-collector": {
+        "module": "intelmq.bots.collectors.twitter.collector",
+    },
+}
+
+
 
 def generate_function(function):
     def test_function(self):
@@ -824,6 +844,18 @@ def test_v322_removed_feeds_and_bots(self):
         result = upgrades.v322_removed_feeds_and_bots(V322_DISCONTINUED_BOTS_AND_FEEDS_IN, {}, False)
         self.assertEqual(V322_DISCONTINUED_BOTS_AND_FEEDS_OUT, result[0])
 
+    def test_v340_twitter_parser(self):
+        """ Test v340_deprecations with a Twitter parser """
+        result = upgrades.v340_deprecations(V340_TWITTER_PARSER_IN, {}, False)
+        self.assertTrue(result[0])
+        self.assertEqual(V340_TWITTER_PARSER_OUT, result[1])
+
+    def test_v340_twitter_collector(self):
+        """ Test v340_deprecations with a Twitter collector """
+        result = upgrades.v340_deprecations(V340_TWITTER_COLLECTOR_IN, {}, False)
+        self.assertIn('twitter-collector', result[0])
+        self.assertEqual(V340_TWITTER_COLLECTOR_IN, result[1])
+
 
 for name in upgrades.__all__:
     setattr(TestUpgradeLib, 'test_function_%s' % name,

intelmq/version.py

@@ -1,6 +1,6 @@
-# SPDX-FileCopyrightText: 2016-2024 Sebastian Wagner
+# SPDX-FileCopyrightText: 2016-2025 Sebastian Wagner
 #
 # SPDX-License-Identifier: AGPL-3.0-or-later
 
-__version_info__ = (3, 3, 2, 'alpha1')
+__version_info__ = (3, 4, 0)
 __version__ = '.'.join(map(str, __version_info__))