如何在没有 head 标签的情况下对 AJAX 加载的页面进行 noindex 索引？

Question

在客户的网站上有一系列位于不同页面上的导师弹出窗口。

http://launcheducation.com/

问题是它们不需要被编入索引，它们只是主页上的弹出窗口。如果您直接加载链接，则它们的样式不正确。它们没有正确的 html 或 head 标签，因为它们只是通过 AJAX 加载的。

因为他们没有 head 标签，所以我不能给它添加 meta 标签。我是 SEO n00b，我只知道基础知识。但我当然不认为我可以添加没有头标签的元标签。所以我将它们添加到 robots.txt 中。

我以为这会结束，但我的客户正在使用 SEOmoz 工具栏，上面说他们仍在被跟踪。我认为这根本不是问题，我认为只是工具栏没有检查robots.txt，但他不断反复提出。

我不知道该怎么做，所以我想澄清一下将链接放在 robots.txt 中是否可行。这是我的 robots.txt：

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /wp-content/themes/launcheducation/tutor-ajax/*?
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=90&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=90
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=359&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=359
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=362&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=362
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=365&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=365
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=367&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=367
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=369&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=369
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=371&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=371
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=373&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=373
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=897&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=897
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=900&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=900
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=902&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=902
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1003&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1003
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1017&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1017
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1020&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1020
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1023&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1023
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1031&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1031
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1123&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1123
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1297&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1297
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1300&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1300
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1305&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1305
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1669&linker=home
Disallow: /wp-content/themes/launcheducation/tutor-ajax/?id=1669

是的，我刚开始只是“禁止：/wp-content/themes/launcheducation/tutor-ajax/*？”当插件仍然没有确认更改时，我将它们全部添加。 robots.txt 也清楚地可以加载到根目录中：

http://launcheducation.com/robots.txt

任何帮助都将不胜感激，即使它只是确认我的陈述是正确的并且它只是不检查 robots.txt 的插件。如果我错了，当然，我想知道如何解决这个问题。

提前致谢。

Answer 1

看来您的方向是正确的。我不确定 SEOMoz 工具，但您应该使用的是 Google Webmaster Tool。在该工具中，有一项功能可让您使用 Googlebot 抓取任何给定的 URL。这将明确告诉您您的 AJAX 部分是否被 robots.txt 正确阻止

如果存在 URL 模式与您希望可抓取的内容相匹配的任何极端情况，您始终可以通过 http 标头发送 NOINDEX 命令，因为您无法添加元标记。 X-Robots-Tag http 标头用于此目的，主要用于 PDF 和图像无法呈现 HTML 的情况......它适用于您的情况。

完整规格在这里：https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag

顺便说一句，像 SEOMoz 这样的第 3 方工具非常棒，但没有什么比引擎本身提供的工具更好的了。 :)