Skip to content

chg: [google] regenerate list from the three merged sources (669 -> 7,549, type string -> hostname) - #386

Merged
adulau merged 1 commit into
MISP:mainfrom
elhoim:warninglists/regenerate-google-list
Aug 30, 2026
Merged

chg: [google] regenerate list from the three merged sources (669 -> 7,549, type string -> hostname)#386
adulau merged 1 commit into
MISP:mainfrom
elhoim:warninglists/regenerate-google-list

Conversation

@elhoim

@elhoim elhoim commented Aug 29, 2026

Copy link
Copy Markdown
Member

What changed

lists/google/list.json regenerated by the new tools/generate-google.py (companion PR).

before after
raw entries 669 7,549
unique after normalisation 558 7,549
type string hostname
entry shape leading dot (.google.ad) bare (google.ad)
version 7 20260829

Zero entries lost — every one of the 558 unique domains from the frozen list is present in the new list.

⚠️ The type change is deliberate and user-visible

The list was "type": "string" with leading-dot entries. In MISP core (app/Model/Warninglist.php), __evalString() does isset($listValues[$value]) — a byte-for-byte exact match with no dot handling. A stored .google.ad can therefore only match an attribute whose literal value is the string .google.ad, which no real hostname is. The list was effectively inert.

__evalHostname() (type hostname) is the suffix-walking mechanism the README describes as "hostname matching (e.g. domain matching from URL)", and getFilteredEntries() normalises hostname-type entries with strtolower(trim($v, '.')) — MISP itself strips the leading dot for this type. lists/tranco/list.json is the in-repo precedent: type: hostname, bare entries, same matching_attributes family.

Consumers of this list will now see bare entries, and the list will actually match hostnames and their subdomains where before it matched nothing.

Sources

source raw kept
google.com/supported_domains 187 187
nickspaargaren/no-google (hosts format) 6,832 6,821
v2fly/domain-list-community data/google + 18 followed include: files 1,230 1,086
previously committed entries (unioned, never dropped) 558 558

Dropped: 11 non-hostname values from source 2 (one leading-hyphen label, ten underscore-prefixed DNS record names such as _spf.google.com), and 2 regexp: entries from source 3 that a MISP warninglist cannot express. No keyword: entries were present.

Verification

  • jsonschema -i lists/google/list.json schema.json — exit 0
  • tools/validate_values.py — exit 0
  • No entry contains whitespace, /, :, #, @, *, an uppercase character, or a leading dot (asserted programmatically over all 7,549)
  • jq -S normalised, no exec bit, version an int

🤖 Generated with Claude Code

https://claude.ai/code/session_0121gk7vaCXX9bMri7tC9XVd

…,549, type string -> hostname)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0121gk7vaCXX9bMri7tC9XVd
@adulau
adulau merged commit 1d33d42 into MISP:main Aug 30, 2026
5 checks passed
@adulau

adulau commented Aug 30, 2026

Copy link
Copy Markdown
Member

Thank you!

@elhoim
elhoim deleted the warninglists/regenerate-google-list branch August 31, 2026 18:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants