php curl 返回带有内容的 http_code 404，浏览器没问题

Question

我想使用 php/curl 函数废弃此页面以获取产品信息。

http://www.easymall.my/flower/item-636507.html

但是，curl 返回 http_code = 404 但浏览器没问题。从类似的主题，我在代码中设置了 header、cookie 和 referer，仍然有 http_code 404。

网页在火狐下正常运行，但Firebug也提示404。

这可能是什么问题？？？

<?php
 $cookie_file = './cookie.txt';
 $url = "http://www.easymall.my/flower/item-636507.html";

 $ch = curl_init ( $url );
 curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
 curl_setopt($ch, CURLOPT_MAXREDIRS, 5);   
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);

 $User_Agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.9) Gecko/2008052906    Firefox/3.0';      
 $request_headers = array();
 $request_headers[] = 'User-Agent: '. $User_Agent;
 $request_headers[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8';
 curl_setopt($ch, CURLOPT_HTTPHEADER, $request_headers);
 curl_setopt ( $ch, CURLOPT_AUTOREFERER, true );     
 curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
 curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);

 curl_exec ( $ch );
 var_dump ( curl_getinfo ( $ch ) );
 if (curl_errno ( $ch )) {
  print curl_error ( $ch );
  } else {
  curl_close ( $ch );
  }
 ?>


array(22) { ["url"]=> string(46) "http://www.easymall.my/flower/item-636507.html" 
["content_type"]=> string(9) "text/html" ["http_code"]=> int(404) ["header_size"]=> 
int(279) ["request_size"]=> int(232) ["filetime"]=> int(-1) ["ssl_verify_result"]=> int(0) 
["redirect_count"]=> int(0) ["total_time"]=> float(0.188) ["namelookup_time"]=> float(0) ["connect_time"]=> float(0.016) ["pretransfer_time"]=> float(0.016) ["size_upload"]=> float(0) ["size_download"]=> float(17830) ["speed_download"]=> float(94840) ["speed_upload"]=> float(0) ["download_content_length"]=> float(-1) ["upload_content_length"]=> float(0) ["starttransfer_time"]=> float(0.11) ["redirect_time"]=> float(0) ["certinfo"]=> array(0) { } ["redirect_url"]=> string(0) "" }

Firebug 的输出：

GET item-636507.html  404 Not Found   easymall.my   17.4 KB   117.53.152.29:80

Cache-Control   no-store, no-cache, must-revalidate
Connection  Keep-Alive
Content-Type    text/html
Date    Wed, 06 Aug 2014 01:58:12 GMT
Keep-Alive  timeout=5, max=100
Server  Apache/2.4.7 (Unix) OpenSSL/1.0.1e PHP/5.5.6 mod_perl/2.0.8-dev Perl/v5.16.3
Transfer-Encoding   chunked
X-Powered-By    PHP/5.5.6
view source
Accept  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding gzip, deflate
Accept-Language en-US,en;q=0.5
Connection  keep-alive
Cookie  __utma=86241292.2105711989.1407286316.1407286316.1407290031.2; __utmc=86241292; __utmz=86241292.1407286316.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); __utmb=86241292.1.10.1407290031
Host    www.easymall.my
User-Agent  Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0

Answer 1

问题不在于您的 curl 回复。它与网站有关。它正在发送一个404 标头，如果您查看开发人员控制台的“网络”选项卡，您可以看到它。

---

您正在抓取的网站 (^{this one}) 正在发送您的 404。由他们来解决他们的问题，这不是您的代码造成的。

---

网站配置错误。该网站通过.htaccess 或http_response_code(404) 甚至header() 发送404。在不知道该代码库是什么样子的情况下，我无法确切告诉您为什么它会给您一个404，但如果它让您感到困扰，您应该与他们联系并让他们知道。