Skip to content

chore: [generate_all.sh] wire in fastly, oracle-oci and driftnet generators - #375

Merged
adulau merged 2 commits into
MISP:mainfrom
elhoim:warninglists/wire-orphan-generators
Aug 30, 2026
Merged

chore: [generate_all.sh] wire in fastly, oracle-oci and driftnet generators#375
adulau merged 2 commits into
MISP:mainfrom
elhoim:warninglists/wire-orphan-generators

Conversation

@elhoim

@elhoim elhoim commented Aug 29, 2026

Copy link
Copy Markdown
Member

What changed

Four lines in generate_all.sh, wiring in generators that exist in tools/ but were never called from it:

+python3 generate-oracle-oci.py
+python3 generate-fastly.py
+#python3 generate-cloudflare-top-domains.py  # requires a CLOUDFLARE_API_TOKEN env var (Cloudflare Radar API); raises KeyError: 'CLOUDFLARE_API_TOKEN' when unset
+python3 generate-driftnet.py

Why

generate_all.sh had drifted out of sync with tools/. generate-fastly.py, generate-oracle-oci.py, generate-cloudflare-top-domains.py and generate-driftnet.py were absent from it entirely, yet the fastly and oracle-oci lists are current — someone has been running them by hand. That is the same failure mode that let check-host-net and lots-project go stale for years.

Every added line was run first

A generator that fails breaks the whole batch (set -e), so nothing was wired in unverified:

script result action
generate-oracle-oci.py ran clean, oracle-oci 793 → 798 enabled
generate-fastly.py ran clean, fastly 21 entries enabled
generate-driftnet.py ran clean, produced 45 entries (list added in a separate PR) enabled
generate-cloudflare-top-domains.py KeyError: 'CLOUDFLARE_API_TOKEN' — needs a Cloudflare Radar API token added commented, with the reason inline

Deliberately not wired in, with reasons:

  • generate-akamai.py — its only source api.bgpview.io is NXDOMAIN, so it cannot run at all; adding it to a set -e batch would break the nightly run.
  • generate_alexa.py, generate-google.py, generate-stackpath.py, generate-windows-binary-hashes.py — already disabled with documented reasons.

Scope

This PR touches generate_all.sh only, and only the four lines above. Re-enabling generate_tranco.py, generate_majestic-million.py, generate-publicdns.py, generate-gptbot.py, generate-check-host-net.py and generate-lots-project.py is deliberately left to the separate PRs that fix and verify each of those generators, so this one does not conflict with them.

Verification

bash -n generate_all.sh passes; each enabled generator was executed successfully before its line was added.


🤖 Generated with Claude Code

https://claude.ai/code/session_0121gk7vaCXX9bMri7tC9XVd

…t-net/lots-project + re-enable publicdns/tranco/majestic-million/gptbot

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0121gk7vaCXX9bMri7tC9XVd
@elhoim elhoim changed the title chore: [generate_all.sh] wire in fastly/oracle-oci/driftnet/check-host-net/lots-project + re-enable publicdns/tranco/majestic-million/gptbot chore: [generate_all.sh] wire in fastly, oracle-oci and driftnet generators Aug 29, 2026
@adulau
adulau merged commit 9ae5422 into MISP:main Aug 30, 2026
4 checks passed
@elhoim
elhoim deleted the warninglists/wire-orphan-generators branch August 31, 2026 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants