chg: [google] regenerate list from the three merged sources (669 -> 7,549, type string -> hostname) - #386
Merged
Conversation
…,549, type string -> hostname) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0121gk7vaCXX9bMri7tC9XVd
Member
|
Thank you! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
lists/google/list.jsonregenerated by the newtools/generate-google.py(companion PR).typestringhostname.google.ad)google.ad)versionZero entries lost — every one of the 558 unique domains from the frozen list is present in the new list.
typechange is deliberate and user-visibleThe list was
"type": "string"with leading-dot entries. In MISP core (app/Model/Warninglist.php),__evalString()doesisset($listValues[$value])— a byte-for-byte exact match with no dot handling. A stored.google.adcan therefore only match an attribute whose literal value is the string.google.ad, which no real hostname is. The list was effectively inert.__evalHostname()(typehostname) is the suffix-walking mechanism the README describes as "hostname matching (e.g. domain matching from URL)", andgetFilteredEntries()normalises hostname-type entries withstrtolower(trim($v, '.'))— MISP itself strips the leading dot for this type.lists/tranco/list.jsonis the in-repo precedent:type: hostname, bare entries, samematching_attributesfamily.Consumers of this list will now see bare entries, and the list will actually match hostnames and their subdomains where before it matched nothing.
Sources
data/google+ 18 followedinclude:filesDropped: 11 non-hostname values from source 2 (one leading-hyphen label, ten underscore-prefixed DNS record names such as
_spf.google.com), and 2regexp:entries from source 3 that a MISP warninglist cannot express. Nokeyword:entries were present.Verification
jsonschema -i lists/google/list.json schema.json— exit 0tools/validate_values.py— exit 0/,:,#,@,*, an uppercase character, or a leading dot (asserted programmatically over all 7,549)jq -Snormalised, no exec bit,versionan int🤖 Generated with Claude Code
https://claude.ai/code/session_0121gk7vaCXX9bMri7tC9XVd