由于存在特殊字符，无法抓取 URL

Question

尝试使用 NUTCH 1.17 进行抓取，但 URL 被拒绝， 有 ＃！在网址中 示例：xxmydomain.com/xxx/#!/xxx/abc.html

我也尝试过包含

+^/

+^#! 在我的正则表达式 urlfilter 中

Answer 1

1. 如果您特别签入regex-normalize.xml 文件 这个特定的规则文件将作为urlnormalizer-regex 插件的一部分应用。 此插件默认包含在 nutch-site.xml 中的 plugin-includes 中。

作为 URL 规范化的一部分，如果在 URLFragment 之后出现任何内容，此特定行将截断 URL

<!-- removes interpage href anchors such as site.com#location -->
<regex>
  <pattern>#.*?(\?|&amp;|$)</pattern>
  <substitution>$1</substitution>
</regex>

您可以通过评论禁用此规则。 （推荐方式） (OR) 你可以从 nutch-site.xml 的 plugin-include conf 中删除 urlnormalizer-regex。

1. 在 URL 规范化部分中还有一个 URL 片段部分被忽略的地方，即 urlnormalizer-basic

BasicURLNormalizer 用于对 URL 应用一般规范化（即删除多个立即斜杠并使用百分比编码正确编码）

    public String normalize(String urlString, String scope)
      throws MalformedURLException {
    
    if ("".equals(urlString)) // permit empty
      return urlString;

    urlString = urlString.trim(); // remove extra spaces

    URL url = new URL(urlString);

    String protocol = url.getProtocol();
    String host = url.getHost();
    int port = url.getPort();
    String file = url.getFile();

    boolean changed = false;
    boolean normalizePath = false;

    if (!urlString.startsWith(protocol)) // protocol was lowercased
      changed = true;

    if ("http".equals(protocol) || "https".equals(protocol)
        || "ftp".equals(protocol)) {

      if (host != null && url.getAuthority() != null) {
        String newHost = normalizeHostName(host);
        if (!host.equals(newHost)) {
          host = newHost;
          changed = true;
        } else if (!url.getAuthority().equals(newHost)) {
          // authority (http://<...>/) contains other elements (port, user,
          // etc.) which will likely cause a change if left away
          changed = true;
        }
      } else {
        // no host or authority: recompose the URL from components
        changed = true;
      }

      if (port == url.getDefaultPort()) { // uses default port
        port = -1; // so don't specify it
        changed = true;
      }

      normalizePath = true;
      if (file == null || "".equals(file)) {
        file = "/";
        changed = true;
        normalizePath = false; // no further path normalization required
      } else if (!file.startsWith("/")) {
        file = "/" + file;
        changed = true;
        normalizePath = false; // no further path normalization required
      }

      if (url.getRef() != null) { // remove the ref
        changed = true;
      }

    } else if (protocol.equals("file")) {
      normalizePath = true;
    }

    // properly encode characters in path/file using percent-encoding
    String file2 = unescapePath(file);
    file2 = escapePath(file2);
    if (!file.equals(file2)) {
      changed = true;
      file = file2;
    }

    if (normalizePath) {
      // check for unnecessary use of "/../", "/./", and "//"
      if (changed) {
        url = new URL(protocol, host, port, file);
      }
      file2 = getFileWithNormalizedPath(url);
      if (!file.equals(file2)) {
        changed = true;
        file = file2;
      }
    }

    if (changed) {
      url = new URL(protocol, host, port, file);
      urlString = url.toString();
    }

    return urlString;
  }

你可以从代码中看到..它完全忽略了**url.getRef**包含URLFragment的信息。

所以，我们可以做的只是简单地替换url = new URL(protocol, host, port, file);

在normalize方法结束时(line number)

url = new URL(protocol, host, port, file+"#"+url.getRef());

我是如何验证的？

scala> val url = new URL("https://www.codepublishing.com/CA/AlisoViejo/#!/AlisoViejo01/AlisoViejo01.html");
url: java.net.URL = https://www.codepublishing.com/CA/AlisoViejo/#!/AlisoViejo01/AlisoViejo01.html

scala> val protocol = url.getProtocol();
protocol: String = https

scala>     val host = url.getHost();
host: String = www.codepublishing.com

scala>     val port = url.getPort();
port: Int = -1

scala>     val file = url.getFile();
file: String = /CA/AlisoViejo/

scala> //when we construct back new url using the above information we end up loosing fragment information like shown in below

scala> new URL(protocol, host, port, file).toString
res69: String = https://www.codepublishing.com/CA/AlisoViejo/

scala> //if we use url.getRef Information in constructing url we can retain back URL fragment information

scala> //like shown below

scala> new URL(protocol, host, port, file+"#"+url.getRef).toString
res70: String = https://www.codepublishing.com/CA/AlisoViejo/#!/AlisoViejo01/AlisoViejo01.html

scala> // so we can replace the url construction object as explained above to retain url fragment information

注意：UrlFragment 将在页面内提供本地对象引用。在大多数情况下，抓取这些 URL 是没有意义的（这就是为什么 nutch 使用上述规则规范化 URL），因为 HTML 将保持不变。