使用带有重复检查的 PHP 将大 CSV 文件导入依赖的 MySQL 表答案

【问题标题】：Import big CSV file to dependent MySQL tables using PHP with duplicate checking使用带有重复检查的 PHP 将大 CSV 文件导入依赖的 MySQL 表
【发布时间】：2016-04-20 12:58:55
【问题描述】：

我正在阅读大约 50,000+ .csv 行并创建插入查询。插入查询值中有一个动态值。如果存在则来自 SELECT 或如果不存在则来自 INSERT 并获取插入的 id。这只是测试目的，忽略安全性不足。

$multiquery = "";
$lines = file($furl);

foreach ($lines as $line_num => $line) {
    if (($line_num + 1) % 1000 == 0) {
        include('connection.php');
    }

    $cols = split(';', $line);

    $originid = 1;
    $dest = $cols[3];
    $cost = (int)$cols[5];

    // === start: consume a lot of connections ===
    $query = "SELECT id FROM dests WHERE name = '$dest'";
    if (!$dests = mysqli_query($link, $query)) {
        die(json_encode(array("errmsg" => "Selecting existing shipdest. Error: ".mysqli_error($link))));
    }
    if (mysqli_num_rows($dests) > 0) {
        $dest = mysqli_fetch_assoc($dests);
    }
    else {
        $query = "INSERT INTO dests (name) VALUES ('$dest')";
        if (!mysqli_query($link, $query)) {
            die(json_encode(array("errmsg" => "Inserting new dest.")));
        }

        $dest['id'] = mysqli_insert_id($link);
    }
    // === end: consume a lot of connections ===

    $multiquery .= "INSERT INTO packages (id_origin, id_dest, cost) VALUES ($originid, ".$dest['id'].", ".$cost."); ";

    if (($line_num + 1) % 1000 == 0 && !mysqli_multi_query($link, $multiquery)) {
        die(json_encode(array("errmsg" => "Failed at line ".$line_num)));
    }
}

如何将该 PHP 块合并为 $multiquery 中的 id_dest 值？

【问题讨论】：

您使用 multi_query 的方式错误。它永远不会按预期工作。改为使用准备好的语句运行单独的查询。
批量插入记录，每次查询都不需要打开连接。

标签： php mysql csv mysqli mysqli-multi-query

【解决方案1】：

我最终得到了这个代码：

ini_set('max_execution_time', 900); // 15 minutes

$lines = file($furl);

$multiquery = "";
foreach ($lines as $line_num => $line) {
    $cols = split(';', $line);
    $dest = $cols[3];

    $multiquery .= "INSERT IGNORE INTO dests (name) VALUES ('$dest'); ";

    if (($line_num + 1) % 1000 == 0) {
        include('connection.php');
        if (!mysqli_multi_query($link, $multiquery)) {
            die(json_encode(array("errmsg" => "Failed at line ".$line_num." Error: ".mysqli_error($link))));
        }
        $multiquery = "";
    }
}
include('connection.php');
if (!mysqli_multi_query($link, $multiquery)) {
    die(json_encode(array("errmsg" => "Failed at line ".$line_num." Error: ".mysqli_error($link))));
}

$multiquery = "";
foreach ($lines as $line_num => $line) {
    $cols = split(';', $line);

    $originid = 1;
    $dest = $cols[3];
    $cost = (int)$cols[5];

    $multiquery .= "INSERT INTO packages (id_origin, id_dest, cost) VALUES ($originid, (SELECT id FROM dests WHERE name = '$dest'), ".$cost."); ";

    include('connection.php');
    if (!mysqli_multi_query($link, $multiquery)) {  
        die(json_encode(array("errmsg" => "Failed at line ".$line_num." Error: ".mysqli_error($link))));
    }
    $multiquery = "";
}
include('connection.php');
if (!mysqli_multi_query($link, $multiquery)) {
    die(json_encode(array("errmsg" => "Failed at line ".$line_num." Error: ".mysqli_error($link))));
}

【讨论】：