HtmlAgilityPack xPath 抓取

Question

我正在尝试抓取此网站 http://www.gotickets.com/calendar.php?Display=Daily&Date=2013-03-12&EventTypeID=2&EventID=0&GenreID=159&VenueID=0&MarketAreaID=0

这就是我的兴趣

数据按层次组织如下

<div class="clr dayItem">
 <div class="clr genreHeader">Alternative Rock</div>
 <div class="clr genreEvents">
  <div class="clr dayEvent">
   <a href="/concert/muse/houston_1339329.php" title="7:00 PM Muse - Toyota Center - TX">Muse - Toyota Center - TX - 7:00 PM
   </a>
 </div>
 <div class="clr dayEvent">
  <a href="/concert/matchbox_20/pooler_1347335.php" title="7:30 PM Matchbox 20 - Johnny Mercer Theatre">Matchbox 20 - Johnny Mercer Theatre - 7:30 PM
  </a>
</div>

etc...
  </div>
 </div>

所以基本上 页面分为两列 每列都有 DayItems 其中包括基因 以及带有 href 的 dayEvents

我一直在尝试获取数据，但我对 xpath 完全陌生，并且一直在使用 Regex 直到今天

Regex 变得繁琐且过于复杂，所以我选择了 xPath

获取我使用的 DayItems：

 var cl = document.DocumentNode.SelectNodes("//*[contains(concat(' ', normalize-space(@class), ' '), ' dayItem ')]");

 foreach (var item in cl.Where(x=> x.Attributes.Any(p=>p.Value == "clr dayItem" && p.OriginalName=="class")))
      {

            /// THIS LINE FAILS
          var genre = item.SelectSingleNode("//.[contains(concat(' ', normalize-space(@class), ' '), ' genre ')]");


          Console.WriteLine(item.Name);

          foreach (var attr in item.Attributes.Select(x => x.OriginalName + ".." + x.Value))
          {


              Console.WriteLine(attr);
          }
      }

Answer 1

以下是使用 XPATH 轻松完成此任务的方法。这很容易，因为文档结构良好并且具有有意义的 CLASS 属性。

        HtmlWeb web = new HtmlWeb();
        HtmlDocument doc = web.Load("http://www.gotickets.com/calendar.php?Display=Daily&Date=2013-03-12&EventTypeID=2&EventID=0&GenreID=159&VenueID=0&MarketAreaID=0");

        foreach (HtmlNode node in doc.DocumentNode.SelectNodes("//div[@class='clr dayEvent']"))
        {
            Console.WriteLine("Event: " + node.InnerText);

            HtmlNode genre = node.SelectSingleNode("../../div[@class='clr genreHeader']");
            Console.WriteLine(" Genre:" + HtmlAgilityPack.HtmlEntity.DeEntitize(genre.InnerText));
        }

您可以将其调整为您的 Event 类。在事件文本中，它不是 HTML，因此您必须像在事件代码中那样解析它。

你可以在这里学习 XPATH：XPath Tutorial

Answer 2

这是我的工作代码，它不像我想要的那样干净，但这只是一次数据查找探险。我再也不会用这个软件了 我会很感激有人修复我的代码，使其更高效、更好，具体取决于 xpath

string html = client.DownloadString("http://www.gotickets.com/calendar.php?Display=Daily&EventTypeID=1&EventID=0&GenreID=159&VenueID=0&MarketAreaID=0" + "&Date=" + MakeDate);


      List<Event> events = new List<Event>();

        HtmlAgilityPack.HtmlDocument document = new HtmlAgilityPack.HtmlDocument();
      document.LoadHtml(html);

      var cl = document.DocumentNode.SelectNodes("//*[contains(concat(' ', normalize-space(@class), ' '), ' dayItem ')]");

      foreach (var item in cl)
      {

          var genre_text = item.InnerText.Replace("\t\t", "").Replace("\t\t\t\t", "").Replace("\t\t\t", "").Replace("\t\t", "").Replace("\t", "");

          var lines = genre_text.Split(new string[] {"\n"}, StringSplitOptions.RemoveEmptyEntries).Select(x=>  WebUtility.HtmlDecode(x)).ToArray();

          var genre = lines.Take(1).First();

          events.AddRange(lines.Skip(1).Select(f =>


              new Event(f, f.Split(new string[] { "-" }, StringSplitOptions.RemoveEmptyEntries), genre, this.Date)

              ));


      }

Event 类只是一个容器

public class Event
{
    private string OriginalString;
    private string[] p;

    public Event(string originalString, string[] parts, string genre, DateTime date)
    {
        this.OriginalString = originalString;
        this.p = parts;
        this.Genre = genre;
        this.Date = date;
        analyze(parts);
    }
 public override string ToString()
    {
        string pattern = "{0},{1},{2},{3}";
        var s = string.Format(pattern, this.Date.ToString("MMM"), this.Genre, this.Location, this.Performer);
        return s;

    }

    private void analyze(string[] parts)
    {
        if (parts.Length < 3)
        {
            throw new IndexOutOfRangeException("Length < 3 ==> " + parts.Length);
        }

        if (parts.Length > 3)
        {
            this.Performer = parts[0].Trim();
            this.Location = parts[1].Trim() + "-" + parts[2].Trim();

        }
        else
        {
            this.Performer = parts[0].Trim();
            this.Location = parts[1].Trim();

        }

    }

    public string Genre { get; set; }
    public string Performer { get; set; }
    public string Location { get; set; }
    public DateTime Date { get; set; }
}

它有效，但它是UGLYYY