Merge pull request #361 from kpumuk/pr-224-robots-rebased

Support custom robots directives

Author: kpumuk

CHANGELOG.md

@@ -3,6 +3,7 @@
 ## Unreleased
 
 - Added `truncate_array_items_at_boundaries` configuration option to preserve whole items in multi-item `title` and `keywords` arrays when truncating ([354](https://github.com/kpumuk/meta-tags/pull/354)).
+- Added support for custom robots directives via `robots`, `googlebot`, and `bingbot` hashes ([224](https://github.com/kpumuk/meta-tags/pull/224), rebased in [361](https://github.com/kpumuk/meta-tags/pull/361)).
 
 ## 2.23.0 (March 16, 2026) [☰](https://github.com/kpumuk/meta-tags/compare/v2.22.3...v2.23.0)
 

README.md

@@ -154,7 +154,7 @@ If you want to set the title and display another text, use this:
 
 ### Allowed options for `display_meta_tags` and `set_meta_tags` methods
 
-Use these options to customize the title format:
+Use these options to customize the generated tags:
 
 | Option         | Description                                                                                                         |
 | -------------- | ------------------------------------------------------------------------------------------------------------------- |
@@ -173,6 +173,9 @@ Use these options to customize the title format:
 | `:nofollow`    | Add nofollow meta tag; when true, "robots" will be used; accepts a string with a robot name or an array of strings  |
 | `:follow`      | Add follow meta tag; when true, "robots" will be used; accepts a string with a robot name or an array of strings    |
 | `:noarchive`   | Add noarchive meta tag; when true, "robots" will be used; accepts a string with a robot name or an array of strings |
+| `:robots`      | Add custom directives to the `robots` meta tag (Hash)                                                               |
+| `:googlebot`   | Add custom directives to the `googlebot` meta tag (Hash)                                                            |
+| `:bingbot`     | Add custom directives to the `bingbot` meta tag (Hash)                                                              |
 | `:canonical`   | Add canonical link tag                                                                                              |
 | `:prev`        | Add legacy prev pagination link tag                                                                                 |
 | `:next`        | Add legacy next pagination link tag                                                                                 |
@@ -410,6 +413,37 @@ set_meta_tags noindex: true, follow: true
 
 This tag will prevent search engines from indexing this specific page, but it will still allow them to crawl and index the remaining pages on your website.
 
+### Robots
+
+Use the `robots`, `googlebot`, and `bingbot` hashes when you need directives beyond `index`, `noindex`, `follow`, `nofollow`, and `noarchive`.
+
+This is useful for directives such as `max-snippet`, `max-video-preview`, and `unavailable_after`.
+
+```ruby
+set_meta_tags robots: {
+  "max-snippet" => -1,
+  "max-video-preview" => -1
+}
+# <meta name="robots" content="max-snippet:-1, max-video-preview:-1">
+
+set_meta_tags googlebot: {
+  unavailable_after: "2026-12-31"
+}
+# <meta name="googlebot" content="unavailable_after:2026-12-31">
+
+set_meta_tags bingbot: {
+  "max-image-preview" => "large"
+}
+# <meta name="bingbot" content="max-image-preview:large">
+```
+
+These hashes are merged with the existing robots helpers, so `noindex`, `nofollow`, and similar directives still render first for the same meta tag.
+
+Further reading:
+
+- [Google Search Central: robots meta tag and X-Robots-Tag](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag)
+- [Bing Webmaster Guidelines: robots meta tag support](https://www.bing.com/webmasters/help/which-robots-metatags-does-bing-support-5198d240)
+
 ### Canonical URL
 
 Canonical link elements help search engines consolidate duplicate or near-duplicate URLs under one preferred URL. They are a signal, not a guarantee.

lib/meta_tags/meta_tags_collection.rb

@@ -154,6 +154,13 @@ def extract_robots
         calculate_robots_attributes(result, attributes)
       end
 
+      [:robots, :googlebot, :bingbot].each do |bot|
+        values = extract(bot).presence
+        if values
+          result[bot.to_s].concat(values.map { |k, v| "#{k}:#{v}" })
+        end
+      end
+
       result.transform_values { |v| v.join(", ") }
     end
 

lib/meta_tags/view_helper.rb

@@ -151,6 +151,9 @@ def refresh(refresh)
     # @option default [String, Integer] :refresh (nil) meta refresh tag;
     # @option default [Hash] :open_graph ({}) add Open Graph meta tags.
     # @option default [Hash] :open_search ({}) add Open Search link tag.
+    # @option default [Hash] :robots ({}) add robots meta tags.
+    # @option default [Hash] :googlebot ({}) add googlebot meta tags.
+    # @option default [Hash] :bingbot ({}) add bingbot meta tags.
     # @return [String] HTML meta tags to render in HEAD section of the
     #   HTML document.
     # @example Render meta tags in a layout

sig/lib/meta_tags/view_helper.rbs

@@ -4,6 +4,7 @@ module MetaTags
 
     type stringish = String | (_Stringish & Object)
     type robots_value = bool | String | Array[String]
+    type meta_tags = Hash[Renderer::meta_key, Renderer::meta_value]
 
     def meta_tags: () -> MetaTagsCollection
 

spec/view_helper/robots_spec.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+require "spec_helper"
+
+RSpec.describe MetaTags::ViewHelper, "displaying robots meta tags" do
+  it "displays meta tags specified with :robots" do
+    subject.display_meta_tags(robots: {"max-snippet": -1}).tap do |meta|
+      expect(meta).to have_tag("meta", with: {content: "max-snippet:-1", name: "robots"})
+    end
+  end
+
+  it "displays meta tags specified with :googlebot" do
+    subject.display_meta_tags(googlebot: {unavailable_after: "2020-09-21"}).tap do |meta|
+      expect(meta).to have_tag("meta", with: {content: "unavailable_after:2020-09-21", name: "googlebot"})
+    end
+  end
+
+  it "displays meta tags specified with :bingbot" do
+    subject.display_meta_tags(bingbot: {"max-image-preview": "large"}).tap do |meta|
+      expect(meta).to have_tag("meta", with: {content: "max-image-preview:large", name: "bingbot"})
+    end
+  end
+
+  it "displays multiple custom robots tags in a hash" do
+    subject.display_meta_tags(robots: {"max-snippet": -1, "max-video-preview": -1}).tap do |meta|
+      expect(meta).to have_tag("meta", with: {content: "max-snippet:-1, max-video-preview:-1", name: "robots"})
+    end
+  end
+
+  it "displays custom robots tags along with noindex" do
+    subject.noindex(true)
+    subject.display_meta_tags(robots: {"max-snippet": -1, "max-video-preview": -1}).tap do |meta|
+      expect(meta).to have_tag("meta", with: {content: "noindex, max-snippet:-1, max-video-preview:-1", name: "robots"})
+    end
+  end
+
+  it "merges bot-specific directives with helper-based robots directives" do
+    subject.set_meta_tags(noindex: "googlebot", googlebot: {unavailable_after: "2026-12-31"})
+    subject.display_meta_tags.tap do |meta|
+      expect(meta).to have_tag("meta", with: {content: "noindex, unavailable_after:2026-12-31", name: "googlebot"})
+    end
+  end
+end