脚本将部分内容写入 csv 文件

Question

我在php 中编写了一个脚本，用于从网页中抓取titles 及其links 并将它们相应地写入一个csv 文件。由于我正在处理分页站点，因此只有最后一页的内容保留在 csv 文件中，其余部分被覆盖。我尝试了写作模式w。但是，当我使用 append a 执行相同操作时，我会在该 csv 文件中找到所有数据。

由于 appending 和 writing 数据使 csv 文件多次打开和关闭（因为我可能错误地应用了循环），脚本变得不那么高效和耗时。

我怎样才能以有效的方式做同样的事情，当然还要使用（写作）w 模式？

这是我到目前为止写的：

<?php
include "simple_html_dom.php";
$link = "https://stackoverflow.com/questions/tagged/web-scraping?page="; 

function get_content($url)
    {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        $htmlContent = curl_exec($ch);
        curl_close($ch);
        $dom = new simple_html_dom();
        $dom->load($htmlContent);
        $infile = fopen("itemfile.csv","a");
        foreach($dom->find('.question-summary') as $file){
            $itemTitle = $file->find('.question-hyperlink', 0)->innertext;
            $itemLink = $file->find('.question-hyperlink', 0)->href;
            echo "{$itemTitle},{$itemLink}<br>";
            fputcsv($infile,[$itemTitle,$itemLink]);
        }
        fclose($infile);
    }
for($i = 1; $i<10; $i++){
        get_content($link.$i);
    }
?>

Answer 1

如果您不想多次打开和关闭文件，请将打开脚本移动到您的 for-loop 之前并在之后关闭它：

function get_content($url, $inifile)
{
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    $htmlContent = curl_exec($ch);
    curl_close($ch);
    $dom = new simple_html_dom();
    $dom->load($htmlContent);
    foreach($dom->find('.question-summary') as $file){
        $itemTitle = $file->find('.question-hyperlink', 0)->innertext;
        $itemLink = $file->find('.question-hyperlink', 0)->href;
        echo "{$itemTitle},{$itemLink}<br>";
        fputcsv($infile,[$itemTitle,$itemLink]);
    }
}

$infile = fopen("itemfile.csv","w");

for($i = 1; $i<10; $i++) {
    get_content($link.$i, $inifile);
}

fclose($infile);
?>

Answer 2

我会考虑不在get_content 函数中将结果回显或写入文件。我会重写它，让它只get 内容，所以我可以以任何我喜欢的方式处理提取的数据。像这样的东西（请阅读代码 cmets）：

<?php
include "simple_html_dom.php";
$link = "https://stackoverflow.com/questions/tagged/web-scraping?page="; 

// This function does not write data to a file or print it. It only extracts data
// and returns it as an array.
function get_content($url)
    {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_BINARYTRANSFER, 1);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        $htmlContent = curl_exec($ch);
        curl_close($ch);
        $dom = new simple_html_dom();
        $dom->load($htmlContent);
        // We don't need the following line anymore
        // $infile = fopen("itemfile.csv","a");
        // We will collect extracted data in an array
        $result = [];
        foreach($dom->find('.question-summary') as $file){
            $itemTitle = $file->find('.question-hyperlink', 0)->innertext;
            $itemLink = $file->find('.question-hyperlink', 0)->href;
            $result []= [$itemTitle, $itemLink];
            // echo "{$itemTitle},{$itemLink}<br>";
            // No need to write to file, so we don't need the following as well
            // fputcsv($infile,[$itemTitle,$itemLink]);
        }
        // No files opened, so the following line is no more required
        // fclose($infile);
        // Return extracted data from this specific URL
        return $result;
    }
// Merge all results (result for each url with different page parameter
// With a little refactoring, get_content() can handle this as well
$result = [];
for($page = 1; $page < 10; $page++){
    $result = array_merge($result, get_content($link.$page));
}
// Now do whatever you want with $result. Like writing its values to a file, or print it, etc.
// You might want to write a function for this
$outputFile = fopen("itemfile.csv","a");
foreach ($result as $row) {
    fputcsv($outputFile, $row);
}
fclose($outputFile);

?>