[Docs]: robots.txt is 404ing

[MaxwellCohen]

Describe the bug

https://vitepress.dev/robots.txt is 404ing

see

Reproduction

go to https://vitepress.dev/robots.txt

Expected behavior

see a robot.txt not a 404 page

System Info

chrome 151 on mac (doc website not vitepress app)

Additional context

No response

Validations

• Check if you're on the latest VitePress version.
• Follow our Code of Conduct
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.

[brc-dd]

I'm curious what's the point of adding one?

[MaxwellCohen]

Having a robots.txt file is a best practice and notifies crawlers where the sitemap.xml is and allows blocking the indexing of various pages (such as preview docs, which may not be needed).
I can make a PR to add one

[brc-dd]

I'm fine with adding it, but there is no reason to add it. The spec treats 404 response as "unavailable" (https://datatracker.ietf.org/doc/html/rfc9309#name-unavailable-status) and allows any compliant crawler to crawl the site. And non compliant crawlers don't care about robots.txt anyway.

notifies crawlers where the sitemap.xml is

Again, who says this is needed? All major search engines and platforms look up the conventional /sitemap.xml path automatically.

allows blocking the indexing of various pages

We don't want to restrict anything so it will mostly be just an empty Disallow.

such as preview docs

Netlify already handles that by responding with X-Robots-Tag: noindex which is supported by both Google and Bing. Also, you cannot block external domains using robots.txt. Our preview docs are on different domains, so you'll need to dynamically generate robots.txt that block everything depending on whether it's a preview deployment or prod deployment.

[MaxwellCohen]

Great, I will get a pr up after work today