ci: add docs discovery health check

[omarespejel]

Summary

Add a lightweight live health check for the public docs.starknet.io discovery surface used by agents, crawlers, and developer tools.

This complements the existing local llms.txt validator. The local validator checks repository format and link integrity; this workflow checks that the deployed docs site still exposes the expected discovery endpoints and headers after build, hosting, CDN, or platform changes.

What This Prevents

Examples of regressions this should catch:

• llms.txt or llms-full.txt stops being served after a docs platform or Mintlify config change.
• .well-known/llms.txt or .well-known/llms-full.txt disappears even though the root files still exist.
• robots.txt is regenerated without the docs sitemap reference.
• Homepage discovery headers stop advertising the llms.txt and llms-full.txt locations.
• A CDN/proxy behavior change breaks HEAD / or flattens repeated Link headers in a way that affects discovery.

What This Checks

• https://docs.starknet.io/llms.txt
• https://docs.starknet.io/llms-full.txt
• https://docs.starknet.io/.well-known/llms.txt
• https://docs.starknet.io/.well-known/llms-full.txt
• https://docs.starknet.io/sitemap.xml
• https://docs.starknet.io/robots.txt
• Homepage discovery headers:
  • Link with rel="llms-txt"
  • Link with rel="llms-full-txt"
  • x-llms-txt: /llms.txt

Workflow Behavior

• Runs manually via workflow_dispatch
• Runs weekly on Monday
• Runs on PRs only when the health-check script or workflow changes
• Does not run on normal docs content edits

The checker includes bounded retries, retryable HTTP status handling, a job timeout, multi-value header support, origin-only --base-url validation, and a GET fallback when HEAD / is unsupported.

Validation

Ran locally:

python3 scripts/validate_llms_txt.py
python3 scripts/check_discovery_endpoints.py

Both pass.