Merge branch 'develop' into shodan-alert

Author: aaronkaplan

CHANGELOG.md

@@ -14,11 +14,14 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o
 --------------------------------
 
 ### Configuration
+- New parameter `stop_retry_limit` (PR#2598 by Lukas Heindl).
 
 ### Core
 - Drop support for Python 3.8 (fixes #2616, PR#2617 by Sebastian Wagner).
 - `intelmq.lib.splitreports`: Handle bot parameter `chunk_size` values empty string, due to missing parameter typing checks (PR#2604 by Sebastian Wagner).
 - `intelmq.lib.mixins.sql` Add Support for MySQL (PR#2625 by Karl-Johan Karlsson).
+- New parameter `stop_retry_limit` to gracefully handle stopping bots which take longer to shutdown (PR#2598 by Lukas Heindl, fixes #2595).
+- `intelmq.lib.datatypes`: Remove unneeded Dict39 alias (PR#2639 by Nakul Rajpal, fixes #2635)
 
 ### Development
 
@@ -27,9 +30,11 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o
 ### Bots
 #### Collectors
 - `intelmq.bots.collectors.shodan.collector_alert`: Added a new collector to query the Shodan Alert API (PR#2618 by Sebastian Wagner and Malawi CERT).
+- Remove `intelmq.bots.collectors.blueliv` as it uses an unmaintained library, does not work any more and breaks other CI tests (fixes #2593, PR#2632 by Sebastian Wagner).
 
 #### Parsers
 - `intelmq.bots.parsers.cymru.parser_cap_program`: Add mapping for TOR and ipv6-icmp protocol (PR#2621 by Mikk Margus Möll).
+- Remove `intelmq.bots.collectors.blueliv` as it is obsolete with the removed collector (PR#2632 by Sebastian Wagner).
 
 #### Experts
 - `intelmq.bots.experts.asn_lookup.expert`:
@@ -40,6 +45,7 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o
   - Use database path matching to installation type (PR#2606 by Sebastian Wagner).
   - Add new mode `random_single_value` (PR#2601 by Sebastian Wagner).
 - `intelmq.bots.experts.sieve.expert`: Test for textX dependency in self-check (PR#2605 by Sebastian Wagner).
+- `intelmq.bots.experts.trusted_introducer_lookup.expert`: Change to new TI database URL (fixes #2620, PR#2633 by Sebastian Wagner).
 
 #### Outputs
 - `intelmq.bots.outputs.smtp_batch.output`:
@@ -56,6 +62,7 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o
 ### Packaging
 - Replace `/opt/intelmq` example paths in bots with variable `VAR_STATE_PATH` for correct paths in LSB-path setups like with packages (PR#2587 by Sebastian Wagner).
 - New deb-package `intelmq-contrib` with all `contrib/` scripts and documentation (PR#2614 by Sebastian Wagner).
+- New deb-package `intelmq-autostart` containing systemd services and timers to start all enabled IntelMQ bots at boot and periodically (PR#2638 by Sebastian Wagner).
 
 ### Tests
 - `intelmq.tests.lib.test_pipeline.TestAmqp.test_acknowledge`: Skip on all Python versions when running on CI (PR#2602 by Sebastian Wagner).
@@ -67,8 +74,10 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o
 - `intelmq.bin.intelmq_psql_initdb`: Use `JSONB` type by default, Postgres supports it since version 9 (PR#2597 by Sebastian Wagner).
 - `intelmq.bin.rewrite_config_files`: Removed obsolete JSON configuration file rewriter (PR#2613 by Sebastian Wagner).
 - `intelmq/lib/bot_debugger.py`: Fix overwriting the runtime logging level by command line parameter (PR#2603 by Sebastian Wagner, fixes #2563).
+- `intelmq.bin.intelmqctl`: Fix bot log level filtering (PR#2607 by Sebastian Wagner, fixes #2596).
 
 ### Contrib
+- Bash Completion: Adapt to YAML-style runtime configuration (PR#2642 by Sebastian Wagner, fixes #2094).
 
 ### Known issues
 

Makefile

@@ -22,3 +22,6 @@ test:
 
 codestyle:
 	pycodestyle intelmq/{bots,lib,bin}
+
+licenses:
+	reuse lint

contrib/autostart/intelmq-periodic-start.timer

@@ -0,0 +1,14 @@
+# SPDX-FileCopyrightText: 2025 Institute for Common Good Technology
+# SPDX-License-Identifier: AGPL-3.0-or-later
+
+[Unit]
+Description=Periodically start all enabled IntelMQ bots
+
+[Timer]
+# run every 5 minutes
+OnUnitActiveSec=5minutes
+Unit=intelmq-start.service
+Persistent=true
+
+[Install]
+WantedBy=multi-user.target timers.target

contrib/autostart/intelmq-start.service

@@ -0,0 +1,20 @@
+# SPDX-FileCopyrightText: 2025 Institute for Common Good Technology
+# SPDX-License-Identifier: AGPL-3.0-or-later
+
+[Unit]
+Description=Start all enabled IntelMQ bots
+After=network.target
+# will be replaced by valkey in next distribution versions
+Requires=redis-server.service
+
+[Service]
+# oneshot does not work as it stops the remaining processes
+Type=simple
+ExecStart=/usr/bin/intelmq-start.sh
+User=intelmq
+Group=intelmq
+RemainAfterExit=yes
+KillMode=process
+
+[Install]
+WantedBy=multi-user.target

contrib/autostart/intelmq-start.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+# SPDX-FileCopyrightText: 2025 Institute for Common Good Technology
+# SPDX-License-Identifier: AGPL-3.0-or-later
+
+# suppress stdout output. Errors go to stderr and are kept
+
+if [ "$UID" -eq 0 ]; then
+    sudo -u intelmq intelmqctl start > /dev/null
+else
+    nohup intelmqctl start > /dev/null
+fi

contrib/bash-completion/intelmqctl

@@ -1,4 +1,4 @@
-# SPDX-FileCopyrightText: 2018 Sebastian Wagner
+# SPDX-FileCopyrightText: 2018-2020 nic.at GmbH, 2025 Institute for Common Good Technology
 #
 # SPDX-License-Identifier: AGPL-3.0-or-later
 # bash completion for intelmqctl                            -*- shell-script -*-
@@ -18,6 +18,9 @@ _intelmqctl ()
         return 0
     fi
 
+    which yq >& /dev/null
+    misses_yq=$?
+
     #echo "posice: $COMP_CWORD $COMP_WORDS";
     case $COMP_CWORD in
         1)
@@ -26,19 +29,20 @@ _intelmqctl ()
             return 0
         ;;
         2)
-            pipeline='/opt/intelmq/etc/pipeline.conf';
-            [ -f ${pipeline} ] || pipeline='/etc/intelmq/pipeline.conf';
+            runtime='/opt/intelmq/etc/runtime.yaml';
+		    [ -f ${runtime} ] || runtime='/etc/intelmq/runtime.yaml';
             case "${COMP_WORDS[1]}" in
                 start | stop | restart | status | reload | log | run | enable | disable)
-                    runtime='/opt/intelmq/etc/runtime.conf';
-		    [ -f ${runtime} ] || runtime='/etc/intelmq/runtime.conf';
-                    local bots=$(jq 'keys[]' $runtime);
+                    [[ "$misses_yq" -eq 1 ]] && return 0
+                    local bots=$(yq 'keys[]' $runtime | grep -v '^global$');
                     COMPREPLY=($(compgen -W "${bots}" -- ${cur}));
                     return 0
                 ;;
                 clear)
-                    local bots=$(jq '.[] | .["source-queue"]' $pipeline | grep -v '^null$'; jq '.[] | .["destination-queues"]'  $pipeline | grep -v '^null$' | jq '.[]');
-                    COMPREPLY=($(compgen -W "${bots}" -- ${cur}));
+                    [[ "$misses_yq" -eq 1 ]] && return 0
+                    local destination_queues=$(yq '.[] | .["parameters"] | .["destination_queues"] | .[] | .[]' $runtime);
+                    local source_queues=$(yq 'keys[]' $runtime | grep -v '^global$' | while read line; do echo "$line-queue"; done)
+                    COMPREPLY=($(compgen -W "${source_queues} ${destination_queues}" -- ${cur}));
                     return 0
                 ;;
                 list)

debian/changelog

@@ -1,4 +1,4 @@
-intelmq (3.4.1-1) UNRELEASED; urgency=medium
+intelmq (3.4.1~alpha1-1) UNRELEASED; urgency=medium
 
   * 3.4.1 Bugfix release
 

debian/compat

@@ -1 +1 @@
-9
+10

debian/control

@@ -34,7 +34,31 @@ Architecture: all
 Depends: bash-completion,
          cron,
          jq,
-         python3-dateutil (>= 2.5),
+         python3-intelmq,
+         redis-server,
+         systemd,
+         ${misc:Depends}
+Recommends: intelmq-contrib
+Description: Solution for IT security teams for collecting and processing security feeds
+ IntelMQ is a solution for IT security teams (CERTs, CSIRTs, abuse
+ departments,...) for collecting and processing security feeds (such as log
+ files) using a message queuing protocol. It's a community driven initiative
+ called IHAP (Incident Handling Automation Project) which was conceptually
+ designed by European CERTs/CSIRTs during several InfoSec events. Its main goal
+ is to give to incident responders an easy way to collect & process threat
+ intelligence thus improving the incident handling processes of CERTs.
+ .
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ SOFTWARE.
+
+Package: python3-intelmq
+Architecture: all
+Depends: python3-dateutil (>= 2.5),
          python3-dnspython (>= 2.0.0),
          python3-openssl,
          python3-psutil (>= 1.2.1),
@@ -44,12 +68,7 @@ Depends: bash-completion,
          python3-termstyle (>= 0.1.10),
          python3-tz,
          python3-importlib-metadata,
-         redis-server,
-         systemd,
-         ${misc:Depends},
-         ${sphinxdoc:Depends},
          ${python3:Depends}
-Recommends: intelmq-contrib
 Suggests: python3-geoip2 (>= 2.2.0),
           python3-imbox (>= 0.8),
           python3-psycopg2 (>= 2.5.5),
@@ -58,14 +77,8 @@ Suggests: python3-geoip2 (>= 2.2.0),
           python3-sleekxmpp (>= 1.3.1),
           python3-stomp (>= 4.1.12),
           python3-pendulum
-Description: Solution for IT security teams for collecting and processing security feeds
- IntelMQ is a solution for IT security teams (CERTs, CSIRTs, abuse
- departments,...) for collecting and processing security feeds (such as log
- files) using a message queuing protocol. It's a community driven initiative
- called IHAP (Incident Handling Automation Project) which was conceptually
- designed by European CERTs/CSIRTs during several InfoSec events. Its main goal
- is to give to incident responders an easy way to collect & process threat
- intelligence thus improving the incident handling processes of CERTs.
+Description: Python Data of IntelMQ
+ Contains the Python Libraries and Executables for IntelMQ
  .
  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
@@ -91,3 +104,20 @@ Description: Contributed scripts for IntelMQ
  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  SOFTWARE.
+
+Package: intelmq-autostart
+Architecture: all
+Depends: intelmq,
+         ${misc:Depends},
+         ${python3:Depends}
+Suggests:
+Description: Automatic start for all IntelMQ bots
+ FIXME
+ .
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ SOFTWARE.

debian/intelmq-autostart.install

@@ -0,0 +1,3 @@
+contrib/autostart/intelmq-periodic-start.timer /lib/systemd/system/
+contrib/autostart/intelmq-start.service /lib/systemd/system/
+contrib/autostart/intelmq-start.sh /usr/bin/