在 HtmlUnit (2.33) 中模拟页面点击会给出无效或非法的选择器异常

Question

首先我应该说我根本不了解 Javascript。我正在尝试模拟对彭博超链接页面的点击。我想获取新闻项目列表（超链接），然后简单地遍历列表获取每篇文章的标题和文章文本。这是我的代码：

public List<String> getBloomNewsHtmlUnit() throws IOException {
    String searchString = "Apple";
    List<String> bloombergNewsAll = new ArrayList<>();

    WebClient webclient = new WebClient(BrowserVersion.BEST_SUPPORTED);

    HtmlPage mainpage = webclient.getPage("http://www.bloomberg.com/search?query=" + searchString);

    HtmlAnchor pageanchor = mainpage.getFirstByXPath("//*[@id=\"content\"]/div/section/section[2]/section[1]/div[2]/div[2]/article/div[1]/h1/a");

    webclient.waitForBackgroundJavaScript(50000);
    webclient.getOptions().setThrowExceptionOnScriptError(false);
    webclient.getOptions().setThrowExceptionOnFailingStatusCode(false);
    webclient.setCssErrorHandler(new SilentCssErrorHandler());

    mainpage = pageanchor.click();

    System.out.println("Main page: " + mainpage.asText());

    return bloombergNewsAll;
    //  return bloombergNewsAll;
}

这是个例外：

Sep 11, 2016 9:49:34 AM com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter runtimeError
SEVERE: runtimeError: message=[An invalid or illegal selector was specified (selector: '*,:x' error: Invalid selector: :x).] sourceName=[https://assets.bwbx.io/business/public/javascripts/application-6e1529c288.js] line=[153] lineSource=[null] lineOffset=[0]
Exception in thread "main" java.lang.RuntimeException: com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot call method "split" of undefined (https://assets.bwbx.io/business/public/javascripts/application-6e1529c288.js#79)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:284)
at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:519)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:386)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:304)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:451)
at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:436)
at com.jsoup.test.BloombergTest.getBloomNewsHtmlUnit(BloombergTest.java:71)
at com.jsoup.test.BloombergTest.main(BloombergTest.java:37)
Caused by: com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot call method "split" of undefined (https://assets.bwbx.io/business/public/javascripts/application-6e1529c288.js#79)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:921)
at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:803)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:779)
at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:975)
at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:352)
at com.gargoylesoftware.htmlunit.html.HtmlScript$2.execute(HtmlScript.java:238)
at com.gargoylesoftware.htmlunit.html.HtmlPage.initialize(HtmlPage.java:277)
... 7 more
Caused by: net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Cannot call method "split" of undefined (https://assets.bwbx.io/business/public/javascripts/application-6e1529c288.js#79)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3915)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3899)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3924)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3940)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.undefCallError(ScriptRuntime.java:3956)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThisHelper(ScriptRuntime.java:2390)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2384)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1342)
at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:413)
at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:252)
at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3264)
at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$3.doRun(JavaScriptEngine.java:794)
at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:906)
... 15 more
Java Result: 1

即使我尝试执行代码的前 4 行（不引用 HtmlAnchor），也会出现相同的错误。我在网上阅读了一些关于此错误的错误报告，但建议的解决方案似乎都不适用于我的情况：

htmlunit : An invalid or illegal selector was specified

在上面的 SOF 问题中，我将建议的 waitForBackgroundJavaScript 应用到 webclient，但这并没有解决问题。

JavaScript Exception in HtmlUnit when clicking at google result page

在这个问题中我尝试添加：

JavaScriptEngine engine = webclient.getJavaScriptEngine();
    engine.holdPosponedActions();

到代码，但错误仍然存​​在。

https://sourceforge.net/p/htmlunit/bugs/1744/

在上面的错误报告中，建议的解决方案是使用选择查询结果重新定义主页。就我而言，我尝试使用 click() 事件重新定义页面。我的代码没有走那么远，一旦我尝试定义 HtmlPage，就会抛出错误。

https://sourceforge.net/p/htmlunit/bugs/1661/

此报告建议简单地忽略警告，但在我的情况下，我遇到了一个异常（不仅仅是警告），这会阻止所需的输出。

我首先尝试使用 Jsoup 进行抓取。这工作得很好，但是当我在 Chrome 中检查它时，Jsoup 在文章文本之间提供了一些错误链接，这些链接不在原始页面上。我怀疑有一个 JS 或 Ajax 调用改变了页面 DOM。这就是我选择使用 Htmlunit 的原因。

如果我在哪里做错了得到这个错误以及如何纠正它，我将不胜感激。另外，如果有人认为可以仅使用 Jsoup 来实现我想要的，请告诉我（我刚刚读到 Jsoup 不支持 DOM 中的动态更改，因此无法单独工作）。提前致谢！

Answer 1

该异常并不一定意味着生成的页面是无用的，尽管在其他情况下可能会有所不同。您必须检查您要查找的内容的结果。

为了减少来自 javascript 引擎的错误消息的输出，您可以定义：

java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(java.util.logging.Level.OFF);

以下示例选择第一个标题，触发点击事件并抓取结果页面；为了验证，我们点击了链接，标题被打印出来：

java.util.logging.Logger.getLogger("com.gargoylesoftware").setLevel(java.util.logging.Level.OFF);

final WebClient webClient = new WebClient(BrowserVersion.CHROME);

webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setTimeout(10000);

try {
    HtmlPage page = webClient.getPage("http://www.bloomberg.com/search?query=Apple");

    System.out.println(page.getTitleText());

    ScriptResult result = page.executeJavaScript("document.querySelector(\"#content > div > section > section.search-results__content > section.content-stories > div.search-result-items > div:nth-child(1) > article > div > h1 > a\").click()");

    page = (HtmlPage)result.getNewPage();

    System.out.println(page.getTitleText());

} catch (Exception e) {
    e.printStackTrace();
} finally {
    webClient.close();
}

---

由于页面不是使用 javascript 填充的，因此您也可以完全跳过 HtmlUnit 并使用像 jsoup 这样的 html 解析器：

新闻类

class News{
    private String title;
    private String href;
    private String content="";

    public String getTitle() {
        return title;
    }

    public String getHref() {
        return href;
    }

    public String getContent() {
        return content;
    }

    public void setContent(String content) {
        this.content = content;
    }

    public News(String title, String href){
        this.title=title;
        this.href=href;
    }
}

从前两页抓取新闻的示例代码（可通过numberOfResultpages调整）：

List<News> bloombergNewsAll = new ArrayList<>();

String searchString = "Apple";
String searchUrl = "http://www.bloomberg.com/search?query=" + searchString + "&page=";
int numberOfResultpages = 2;
Document doc;

// grab title and href
for (int i = 1; i <= numberOfResultpages; i++) {
    try {
        doc = Jsoup.connect(searchUrl + i)
                .userAgent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36")
                .referrer("http://www.bloomberg.com/").get();
        Elements searchResults = doc.select("#content > div > section > section.search-results__content > section.content-stories > div.search-result-items > div > article > div > h1");
        if(searchResults.isEmpty()) break; // no more searchResults

        for (Element result : searchResults) {
            bloombergNewsAll.add(new News(result.text(), result.select("a").attr("href")));
        }
    } catch (IOException e) {
        e.printStackTrace();
    }
}

// grab content
for (News news : bloombergNewsAll) {

    try {
        doc = Jsoup.connect(news.href)
                .userAgent("Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36")
                .referrer("http://www.bloomberg.com/search?query=Apple").get();

        if(news.getHref().contains("bloomberg.com/news/videos")) continue;

        if(news.getHref().contains("bloomberg.com/news/")){
            news.setContent(doc.select("div.article-body__content").text());
        }else if(news.getHref().contains("bloomberg.com/gadfly")){
            news.setContent(doc.select("#article > div.body_ZtDFu > div.container_1KxJx").text());
        }else if(news.getHref().contains("bloomberg.com/view")){
            news.setContent(doc.select("div._31WvjDF17ltgFb1fNB1WqY").text());
        }

    } catch (IOException e) {
        e.printStackTrace();
    }
}

// do something useful with your results
for (News news : bloombergNewsAll) {
    System.out.println(news.getTitle() + "\n\t" + news.getHref() + "\n\t" + (news.getContent().length()>150 ? news.getContent().substring(0, 150) : news.getContent()));
}