使用分页从 asp.net gridview 抓取信息

Question

我正在尝试从网站上删除一些时间表。信息显示在带有分页的 GridView 中。

网址是： http://www.landmarkworldwide.com/when-and-where/register/search-results.aspx?prgid=0&pgID=270&crid=0&ctid=&sdt=0

我的问题是当我想在网格视图中刮除 #1 以外的页面时。 到目前为止我发现的最好的帖子是This One，但它不起作用，而且该主题不完整。我尝试使用 Fiddler 和 Chrome 来获取帖子数据并使用它，但我无法让它为我工作。你们能看出缺少什么吗？

这是我正在使用的代码。它在VB中，但你可以用C#回答，我会翻译-)（对不起）

    Protected Sub Page_Load(sender As Object, e As System.EventArgs) Handles Me.Load

    Dim lcUrl As String = "http://www.landmarkworldwide.com/when-and-where/register/search-results.aspx?prgid=0&pgID=270&crid=0&ctid=&sdt=0"

    ' first, request the login form to get the viewstate value
    Dim webRequest__1 As HttpWebRequest = TryCast(WebRequest.Create(lcUrl), HttpWebRequest)
    Dim responseReader As New StreamReader(webRequest__1.GetResponse().GetResponseStream())
    Dim responseData As String = responseReader.ReadToEnd()
    responseReader.Close()

    ' extract the viewstate value and build out POST data
    Dim viewState As String = ExtractViewState(responseData)


    Dim loHttp As HttpWebRequest = DirectCast(WebRequest.Create(lcUrl), HttpWebRequest)
    ' *** Send any POST data

    Dim lcPostData As String = [String].Format("__VIEWSTATE={0}&__EVENTTARGET={1}&__EVENTARGUMENT={2}", viewState, HttpUtility.UrlEncode("contentwrapper_0$maincontent_0$maincontentfullwidth_0$ucSearchResults$gvPrograms"), HttpUtility.UrlEncode("Page$3"))
    loHttp.Method = "POST"

    Dim lbPostBuffer As Byte() = System.Text.Encoding.GetEncoding(1252).GetBytes(lcPostData)
    loHttp.ContentLength = lbPostBuffer.Length
    Dim loPostData As Stream = loHttp.GetRequestStream()
    loPostData.Write(lbPostBuffer, 0, lbPostBuffer.Length)
    loPostData.Close()

    Dim loWebResponse As HttpWebResponse = DirectCast(loHttp.GetResponse(), HttpWebResponse)
    Dim enc As Encoding = System.Text.Encoding.GetEncoding(1252)

    Dim loResponseStream As New StreamReader(loWebResponse.GetResponseStream(), enc)
    Dim lcHtml As String = loResponseStream.ReadToEnd()

    loWebResponse.Close()
    loResponseStream.Close()
    Response.Write(lcHtml)

End Sub

Private Function ExtractViewState(s As String) As String
    Dim viewStateNameDelimiter As String = "__VIEWSTATE"
    Dim valueDelimiter As String = "value="""

    Dim viewStateNamePosition As Integer = s.IndexOf(viewStateNameDelimiter)
    Dim viewStateValuePosition As Integer = s.IndexOf(valueDelimiter, viewStateNamePosition)

    Dim viewStateStartPosition As Integer = viewStateValuePosition + valueDelimiter.Length
    Dim viewStateEndPosition As Integer = s.IndexOf("""", viewStateStartPosition)

    Return HttpUtility.UrlEncodeUnicode(s.Substring(viewStateStartPosition, viewStateEndPosition - viewStateStartPosition))
End Function

Answer 1

要使其工作，您需要将所有输入字段发送到页面，而不仅仅是视图状态。其他关键数据是__EVENTVALIDATION，例如您不处理它。所以：

首先，您需要在 #1 页面上进行 scrape。所以加载它并使用Html Agility Pack 将其转换为可用的结构。

然后从该结构中提取您需要发布的输入数据。从这个答案HTML Agility Pack get all input fields这里是一个关于如何做到这一点的代码。

foreach (HtmlNode input in doc.DocumentNode.SelectNodes("//input"))
{
    // use this to create the post string
    // input.Attributes["value"];        
}

然后，当您拥有成为有效帖子所需的帖子数据时，您将进入下一步。这是一个例子How to pass POST parameters to ASP.Net web request?

您也可以阅读：How to use HTML Agility pack