WinInet 只下载网页的一部分

Question

我有一个将网页下载到文本文件中的功能

#include <iostream>
#include <string>
#include <fstream>
#include <Windows.h>
#include <WinINet.h>

#pragma comment(lib, "WinINet.lib")
void Download(wstring url)
{
    std::ofstream fout(L"temp.txt");
    HINTERNET hopen = InternetOpen(L"MyAppName",
        INTERNET_OPEN_TYPE_PRECONFIG, NULL, NULL, 0);
    if (hopen)
    {
        DWORD flags = INTERNET_FLAG_DONT_CACHE;
        if (url.find(L"https://") == 0)
            flags |= INTERNET_FLAG_SECURE;
        HINTERNET hinternet = InternetOpenUrl(hopen, url.c_str(), NULL, 0, flags, 0);
        if (hinternet)
        {
            char buf[1024];
            DWORD received = 0;
            while (InternetReadFile(hinternet, buf, sizeof(buf), &received))
            {
                if (!received) break;
                fout.write(buf, received);
            }
            InternetCloseHandle(hinternet);
        }
        InternetCloseHandle(hopen);
    }
    return;
}

当我给它“https://camelcamelcamel.com/Lodge-LMS3-Miniature-Skillet/product/B000LXA9YI”作为参数时，它只是输出 https://hastebin.com/gilomexomu.xml（太大放在这里） 这会切断大部分网页。我不确定网站上是否有一些反下载脚本，或者它是否太大。

Answer 1

这不是你的代码。这是网站。而且我相信它只能提供 gzip 压缩数据。否则，它会在几 kb 的数据后爆炸。 curl 显示网站正在过早地中止传输：

$ curl https://camelcamelcamel.com/Lodge-LMS3-Miniature-Skillet/product/B000LXA9YI -o text.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 15725    0 15725    0     0   4702      0 --:--:--  0:00:03 --:--:--  4702
curl: (18) transfer closed with outstanding read data remaining

所以我做了两件事来更好地用你的代码模拟网络浏览器

1. 放置与浏览器完全相同的标头和用户代理。
2. 由于该网站似乎只想返回 gzip 编码，因此我不得不调整您的文件保存代码另存为二进制而不是文本（这会导致 Windows CRT 错误地“修复”换行符）。

然后要解码整个 HTML，我只需从 Bash 命令提示符运行它：

gunzip < temp.txt > temp_final.txt

结果是 temp_final.txt 有整个 html 响应。

以下是调整后的代码：

#include <iostream>
#include <string>
#include <fstream>
#include <Windows.h>
#include <WinINet.h>

#pragma comment(lib, "WinINet.lib")
void Download(const std::wstring& url)
{
    FILE* file = fopen("temp.txt", "wb");
    HINTERNET hopen = InternetOpen(L"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36",
        INTERNET_OPEN_TYPE_PRECONFIG, NULL, NULL, 0);
    if (hopen)
    {
        DWORD flags = INTERNET_FLAG_DONT_CACHE;
        if (url.find(L"https://") == 0)
            flags |= INTERNET_FLAG_SECURE;

        LPCWSTR headers = L"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36\r\n"
L"DNT: 1\r\n"
L"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\r\n"
L"Accept-Encoding: gzip, deflate, br\r\n"
L"Accept-Language: en-US,en;q=0.9\r\n";


        HINTERNET hinternet = InternetOpenUrl(hopen, url.c_str(), headers, 0, flags, 0);
        if (hinternet)
        {
            char buf[1024+1]={};
            DWORD received = 0;
            while (InternetReadFile(hinternet, buf, 1024, &received))
            {
                if (!received) break;

                printf("%d\n", received);

                fwrite(buf, 1, received, file);
            }
            InternetCloseHandle(hinternet);
        }
        InternetCloseHandle(hopen);
    }
    return;
}

void main()
{
    Download(L"https://camelcamelcamel.com/Lodge-LMS3-Miniature-Skillet/product/B000LXA9YI");
}

我尝试取出 Accept-Encoding 或将其设置为“身份”。结果是服务器发回半页然后中止。