add: [generate-google] merge three live sources, replacing the dead Wikipedia scrape

[elhoim]

Summary

The google warninglist generator scraped a Wikipedia page (List of Google domains) that no longer exists, and had been commented out of generate_all.sh since the list was frozen on 2025-08-21 (669 entries). This PR replaces it with a generator that merges three live, verified sources, and restores the line in generate_all.sh.

⚠️ Type change: string → hostname (please review deliberately)

This generator switches the warninglist's type from "string" (with the old leading-dot entry convention, e.g. .google.ad) to "hostname" (bare entries, e.g. google.ad). This is a user-visible, consumer-affecting change and is called out here on purpose so it gets a deliberate look, not a rubber stamp.

Why: checked directly against MISP core's matching code (MISP/MISP, 2.5 branch, app/Model/Warninglist.php):

• __evalString() (type "string") does isset($listValues[$value]) — an exact, byte-for-byte match with no dot handling. A stored entry of .google.ad can only ever match an attribute whose literal value is the string .google.ad — i.e., essentially never, since no real hostname attribute looks like that. This matches README.md's own description of string as "perfect match."
• __evalHostname() (type "hostname") is the actual suffix-walking mechanism — it walks the dot-separated suffixes of the attribute value and checks each against the list, which is what README.md documents as "hostname matching (e.g. domain matching from URL)."
• getFilteredEntries() in the same file loads hostname-type entries via strtolower(trim($v, '.')) (dot-trimmed, lowercased) but loads string-type entries completely as-is — i.e. MISP core itself treats bare/lowercase as the canonical form for hostname-type lists.
• lists/tranco/list.json is the working in-repo precedent: type: "hostname" with bare entries.

Net effect: the old leading-dot string shape (shared today by google, microsoft, and microsoft-win10-connection-endpoints) appears to already be non-functional against current MISP core. This PR fixes it for google going forward; the other two lists are out of scope here and left untouched. The type switch itself does not change which domains are in the list (verified: 7469 both before and after switching the storage shape) — only whether the list actually fires.

Sources merged

1. https://www.google.com/supported_domains — Google's own list of ccTLD variants of google.com (187 lines, one leading-dot domain per line). Authoritative, verified HTTP 200.
2. https://raw.githubusercontent.com/nickspaargaren/no-google/master/google-domains — community-maintained hosts-file style blocklist of Google service domains (~6.8k lines, 0.0.0.0 <domain> format). Verified the master branch is live (HTTP 200); main returns HTTP 404.
3. https://raw.githubusercontent.com/v2fly/domain-list-community/master/data/google — v2fly/Xray-style structured domain list (709 lines) that starts with 18 include: directives pulling in sibling files (android, blogspot, dart, fastlane, firebase, flutter, golang, google-deepmind, google-play, google-registry, google-scholar, google-trust-services, googlefcm, kaggle, opensourceinsights, polymer, v8, youtube).

Handling the tricky bits (all documented in code comments)

• include: directives are followed — these are genuinely Google-operated properties (Android, YouTube, Firebase, Google Trust Services, ...), so they're pulled in. The fetch is recursive with an explicit visited set and a MAX_INCLUDE_DEPTH cap (8) to guard against cycles/unexpected depth; verified none of the 18 first-level include files nest further (real depth is 1).
• full: / bare prefixes are both treated as domain entries (bare = suffix match, full: = exact — both map onto this list's suffix-matching hostname type).
• keyword: and regexp: entries are dropped (2 found, both inside the google-play include file) — these matching semantics (substring-anywhere, regex) cannot be represented by any type this list uses.
• Trailing @tag attributes (e.g. @cn) are stripped.
• Every hostname is normalised (lowercase, scheme/path/port/trailing-dot stripped) and validated as a plausible hostname before being kept; 11 malformed entries from source 2 (1 leading-hyphen label, 10 underscore-prefixed DNS record names like _spf.google.com) were rejected this way and are individually enumerated in the session's verification.

generate_all.sh

Removed the stale # TODO: Google page on Wikipedia does not exist anymore / passivetotal-suggestion comment lines, uncommented the invocation, and changed it from python3 generate-google.py > lists/google/list.json to a plain python3 generate-google.py — like every other generator, this script now calls write_to_file() internally instead of writing to stdout. Position in the file (right after generate-disposal.py) is unchanged. Verified with bash -n generate_all.sh.

Verification performed

• Probed all 3 source URLs with curl (status + line counts recorded above) before writing any code.
• Ran the generator; generators.log shows only INFO lines, no ERROR/WARNING.
• Confirmed via git status --porcelain that only the intended 3 files changed.
• Parsed the resulting JSON: 669 → 7469 entries (per-source and overlap counts logged).
• Programmatically asserted no entry contains a space, /, :, #, @, *, or uppercase, and none retain a leading dot.
• Ran make_list_unique.py, then jq -S + chmod -x normalisation.
• jsonschema -i lists/google/list.json schema.json — exit 0, no output.
• tools/validate_values.py — exit 0.
• bash -n generate_all.sh — exit 0.

Scope

Exactly tools/generate-google.py and generate_all.sh in this PR (lists/google/list.json is a separate PR since it's a large generated-data diff).

Update: the merge is additive

Review of an earlier revision of this branch found that 80 of the frozen list's 558 unique domains (e.g. google.af, google.ar, duck.com) appeared in none of the three sources and were not recoverable by suffix matching — a silent regression. The generator now unions the already-committed entries in before writing, normalising them through the same path as the sources:

with open(get_abspath_list_file("google")) as existing_file:
    for entry in json.load(existing_file)["list"]:
        normalised = normalise_hostname(entry)
        if normalised:
            merged.add(normalised)

So the generator only ever adds. Verified: 0 entries lost against the previous list; final count 7,549.

---

🤖 Generated with Claude Code

https://claude.ai/code/session_0121gk7vaCXX9bMri7tC9XVd