阻止机器人使用 htaccess 访问 WordPress 搜索 URL

Question

我在 Algolia 中使用 Wordpress，目前每天记录多达 10,000 个垃圾邮件搜索查询，这些查询似乎来自 Googlebot。

它们看起来像这样：

GET /de/%21/6/?s=+Wo+kaufen+viagra+in+wien%F0%9F%96%95%F0%9F%A4%AA%F0%9F%8E%96+www.ZavaMed.store+%F0%9F%8E%96%F0%9F%A4%AA%F0%9F%96%95+Kamagra+aus+indien+bestellen+Cialis+5mg+f%C3%BCr+die+frau+preis+im+ausland+bestellen+Viagra+150+mg+kaufen+ohne+rezept+billige HTTP/1.1" 200 33087 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.83 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "MISS:(lb2-c28-2)" "not-in-cache" "-" "66.249.66.81" "0" "0" "1" 0.693 0.690 D=684513

我正在使用robots.txt 阻止/?s= 和/search/ 页面，并且我在搜索结果页面上使用“noindex”。但垃圾邮件搜索查询不断涌入。

我想做的是使用 htaccess 来阻止 googlebot 和 bingbot 访问 /?s= 和 /search 页面，但继续允许他们访问网站上的所有其他页面。

不幸的是，我不太擅长 .htaccess 和正则表达式，经过 2-3 天的研究，这是我能想到的最好的：

# BEGIN SEARCH FORM PROTECTION
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot) [NC]
RewriteRule ^.*$ “https://example.com/?s=” [R=403,L]
# END SEARCH FORM PROTECTION

这会阻止垃圾邮件查询，但也会阻止 Google（我假设是 Bing）访问所有其他页面。

这是 robots.txt 文件：

User-agent: *
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /search/
Disallow: /*?s=
Disallow: /*?orderby=price
Disallow: /*?orderby=price-desc
Disallow: /*?orderby=rating
Disallow: /*?orderby=popularity
Disallow: /*?orderby=date
Disallow: /*?filter
Disallow: /*add-to-cart=*
Allow: /wp-content/uploads/
Allow: /wp-admin/admin-ajax.php

这里是.htaccess

# BEGIN Search Form Protection
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot) [NC]
RewriteCond %{QUERY_STRING} ^s= [OR]
RewriteCond %{REQUEST_URI} ^/search/
RewriteRule ^ - [F]
# End Search Form Protection

# BEGIN Imagify: webp file type
<IfModule mod_mime.c>
    AddType image/webp .webp
</IfModule>
# END Imagify: webp file type

# BEGIN WP Rocket v3.9.4
# Use UTF-8 encoding for anything served text/plain or text/html
AddDefaultCharset UTF-8
# Force UTF-8 for a number of file formats
<IfModule mod_mime.c>
AddCharset UTF-8 .atom .css .js .json .rss .vtt .xml
</IfModule>

# FileETag None is not enough for every server.
<IfModule mod_headers.c>
Header unset ETag
</IfModule>

# Since we’re sending far-future expires, we don’t need ETags for static content.
# developer.yahoo.com/performance/rules.html#etags
FileETag None

# Send CORS headers if browsers request them; enabled by default for images.
<IfModule mod_setenvif.c>
<IfModule mod_headers.c>
# mod_headers, y u no match by Content-Type?!
<FilesMatch "\.(cur|gif|png|jpe?g|svgz?|ico|webp)$">
SetEnvIf Origin ":" IS_CORS
Header set Access-Control-Allow-Origin "*" env=IS_CORS
</FilesMatch>
</IfModule>
</IfModule>

# Allow access to web fonts from all domains.
<FilesMatch "\.(eot|otf|tt[cf]|woff2?)$">
<IfModule mod_headers.c>
Header set Access-Control-Allow-Origin "*"
</IfModule>
</FilesMatch>

<IfModule mod_alias.c>
<FilesMatch "\.(html|htm|rtf|rtx|txt|xsd|xsl|xml)$">
<IfModule mod_headers.c>
Header set X-Powered-By "WP Rocket/3.9.4"
Header unset Pragma
Header append Cache-Control "public"
Header unset Last-Modified
</IfModule>
</FilesMatch>

<FilesMatch "\.(css|htc|js|asf|asx|wax|wmv|wmx|avi|bmp|class|divx|doc|docx|eot|exe|gif|gz|gzip|ico|jpg|jpeg|jpe|json|mdb|mid|midi|mov|qt|mp3|m4a|mp4|m4v|mpeg|mpg|mpe|mpp|otf|odb|odc|odf|odg|odp|ods|odt|ogg|pdf|png|pot|pps|ppt|pptx|ra|ram|svg|svgz|swf|tar|tif|tiff|ttf|ttc|wav|wma|wri|xla|xls|xlsx|xlt|xlw|zip)$">
<IfModule mod_headers.c>
Header unset Pragma
Header append Cache-Control "public"
</IfModule>
</FilesMatch>
</IfModule>

# Expires headers (for better cache control)
<IfModule mod_expires.c>
    ExpiresActive on
    ExpiresDefault                              "access plus 1 month"
    # cache.appcache needs re-requests in FF 3.6 (thanks Remy ~Introducing HTML5)
    ExpiresByType text/cache-manifest           "access plus 0 seconds"
    # Your document html
    ExpiresByType text/html                     "access plus 0 seconds"
    # Data
    ExpiresByType text/xml                      "access plus 0 seconds"
    ExpiresByType application/xml               "access plus 0 seconds"
    ExpiresByType application/json              "access plus 0 seconds"
    # Feed
    ExpiresByType application/rss+xml           "access plus 1 hour"
    ExpiresByType application/atom+xml          "access plus 1 hour"
    # Favicon (cannot be renamed)
    ExpiresByType image/x-icon                  "access plus 1 week"
    # Media: images, video, audio
    ExpiresByType image/gif                     "access plus 4 months"
    ExpiresByType image/png                     "access plus 4 months"
    ExpiresByType image/jpeg                    "access plus 4 months"
    ExpiresByType image/webp                    "access plus 4 months"
    ExpiresByType video/ogg                     "access plus 4 months"
    ExpiresByType audio/ogg                     "access plus 4 months"
    ExpiresByType video/mp4                     "access plus 4 months"
    ExpiresByType video/webm                    "access plus 4 months"
    # HTC files  (css3pie)
    ExpiresByType text/x-component              "access plus 1 month"
    # Webfonts
    ExpiresByType font/ttf                      "access plus 4 months"
    ExpiresByType font/otf                      "access plus 4 months"
    ExpiresByType font/woff                     "access plus 4 months"
    ExpiresByType font/woff2                    "access plus 4 months"
    ExpiresByType image/svg+xml                 "access plus 1 month"
    ExpiresByType application/vnd.ms-fontobject "access plus 1 month"
    # CSS and JavaScript
    ExpiresByType text/css                      "access plus 1 year"
    ExpiresByType application/javascript        "access plus 1 year"
</IfModule>
# Gzip compression
<IfModule mod_deflate.c>
# Active compression
SetOutputFilter DEFLATE
# Force deflate for mangled headers
<IfModule mod_setenvif.c>
<IfModule mod_headers.c>
SetEnvIfNoCase ^(Accept-EncodXng|X-cept-Encoding|X{15}|~{15}|-{15})$ ^((gzip|deflate)\s*,?\s*)+|[X~-]{4,13}$ HAVE_Accept-Encoding
RequestHeader append Accept-Encoding "gzip,deflate" env=HAVE_Accept-Encoding
# Don’t compress images and other uncompressible content
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png|rar|zip|exe|flv|mov|wma|mp3|avi|swf|mp?g|mp4|webm|webp|pdf)$ no-gzip dont-vary
</IfModule>
</IfModule>

# Compress all output labeled with one of the following MIME-types
<IfModule mod_filter.c>
AddOutputFilterByType DEFLATE application/atom+xml \
                                  application/javascript \
                                  application/json \
                                  application/rss+xml \
                                  application/vnd.ms-fontobject \
                                  application/x-font-ttf \
                                  application/xhtml+xml \
                                  application/xml \
                                  font/opentype \
                                  image/svg+xml \
                                  image/x-icon \
                                  text/css \
                                  text/html \
                                  text/plain \
                                  text/x-component \
                                  text/xml
</IfModule>
<IfModule mod_headers.c>
Header append Vary: Accept-Encoding
</IfModule>
</IfModule>

# END WP Rocket

### BEGIN GTranslate config ###
RewriteRule ^(af|sq|am|ar|hy|az|eu|be|bn|bs|bg|ca|ceb|ny|zh-CN|zh-TW|co|hr|cs|da|nl|en|eo|et|tl|fi|fr|fy|gl|ka|de|el|gu|ht|ha|haw|iw|hi|hmn|hu|is|ig|id|ga|it|ja|jw|kn|kk|km|ko|ku|ky|lo|la|lv|lt|lb|mk|mg|ms|ml|mt|mi|mr|mn|my|ne|no|ps|fa|pl|pt|pa|ro|ru|sm|gd|sr|st|sn|sd|si|sk|sl|so|es|su|sw|sv|tg|ta|te|th|tr|uk|ur|uz|vi|cy|xh|yi|yo|zu)/(af|sq|am|ar|hy|az|eu|be|bn|bs|bg|ca|ceb|ny|zh-CN|zh-TW|co|hr|cs|da|nl|en|eo|et|tl|fi|fr|fy|gl|ka|de|el|gu|ht|ha|haw|iw|hi|hmn|hu|is|ig|id|ga|it|ja|jw|kn|kk|km|ko|ku|ky|lo|la|lv|lt|lb|mk|mg|ms|ml|mt|mi|mr|mn|my|ne|no|ps|fa|pl|pt|pa|ro|ru|sm|gd|sr|st|sn|sd|si|sk|sl|so|es|su|sw|sv|tg|ta|te|th|tr|uk|ur|uz|vi|cy|xh|yi|yo|zu)/(.*)$ /$1/$3 [R=301,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(ar|zh-CN|nl|en|fr|de|it|ja|ko|pl|pt|ru|es|tr)/(.*)$ /wp-content/plugins/gtranslate/url_addon/gtranslate.php?glang=$1&gurl=$2 [L,QSA]
RewriteRule ^(af|sq|am|ar|hy|az|eu|be|bn|bs|bg|ca|ceb|ny|zh-CN|zh-TW|co|hr|cs|da|nl|en|eo|et|tl|fi|fr|fy|gl|ka|de|el|gu|ht|ha|haw|iw|hi|hmn|hu|is|ig|id|ga|it|ja|jw|kn|kk|km|ko|ku|ky|lo|la|lv|lt|lb|mk|mg|ms|ml|mt|mi|mr|mn|my|ne|no|ps|fa|pl|pt|pa|ro|ru|sm|gd|sr|st|sn|sd|si|sk|sl|so|es|su|sw|sv|tg|ta|te|th|tr|uk|ur|uz|vi|cy|xh|yi|yo|zu)$ /$1/ [R=301,L]
### END GTranslate config ###

# BEGIN WordPress
# The directives (lines) between "BEGIN WordPress" and "END WordPress" are
# dynamically generated, and should only be modified via WordPress filters.
# Any changes to the directives between these markers will be overwritten.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

Header add Access-Control-Allow-Origin "*"
Header set Access-Control-Allow-Credentials true

Answer 1

我正在使用robots.txt 阻止/?s= 和/search/ 页面

看到您的robots.txt 文件会很有趣 - Googlebot 抓取在robots.txt 中正确阻止的页面是“非常不寻常的”。例如，要阻止所述示例 URL，您需要使用以下指令：

Disallow: /*/?s=

（不仅仅是/?s=，正如您的问题所暗示的那样。）

但是，要完全阻止对这些 URL 的访问，您需要在根 .htaccess 文件的 top 附近执行以下操作（它必须在 之前 # BEGIN WordPress 部分）：

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot) [NC]
RewriteCond %{QUERY_STRING} ^s= [OR]
RewriteCond %{REQUEST_URI} ^/search/
RewriteRule ^ - [F]

以上内容将为任何包含查询字符串的 URL 提供 403 Forbidden，该查询字符串 starts s= or 其中 URL 路径 starts /search/.

F 标志是R=403 的简写。 L 标志不是必需的。

正则表达式 ^ 对于每个 URL 都是成功的（此时前面的 条件 被处理）并且 -（连字符）明确表示没有替换（在指定4xx 响应码）。

（您还在指令中使用 curly 引号来包围 substitution 参数，这在语法上是无效的。）