如何处理/优化数千个不同的执行 SELECT 查询？

Question

我需要在两个数据库（一个 mysql，另一个远程托管 SQL Server 数据库）之间同步数千行的特定信息。当我执行这个 php 文件时，我猜它会在几分钟后卡住/超时，所以我想知道如何解决这个问题，也许还可以优化“同步”它的方式。

代码需要做什么：

基本上，我想获取数据库中更新的每一行（=一个帐户） - 来自另一个 SQL Server 数据库的两条特定信息（= 2 SELECT 查询）。因此，我使用了一个 foreach 循环，它为每一行创建 2 个 SQL 查询，然后我将这些信息更新到这一行的 2 列中。我们讨论了需要通过这个 foreach 循环运行的大约 10k 行。

我的想法有帮助吗？

我听说过诸如 PDO Transactions 之类的东西，它应该收集所有这些查询，然后在所有 SELECT 查询的包中发送它们，但我不知道我是否正确使用它们，或者它们在这种情况下是否有帮助。

这是我当前的代码，几分钟后超时：

// DBH => MSSQL DB | DB => MySQL DB
$dbh->beginTransaction();
// Get all referral IDs which needs to be updated:
$listAccounts = "SELECT * FROM Gifting WHERE refsCompleted <= 100 ORDER BY idGifting ASC";
$ps_listAccounts = $db->prepare($listAccounts);
$ps_listAccounts->execute();

foreach($ps_listAccounts as $row) {
    $refid=$row['refId'];
    // Refsinserted
    $refsInserted = "SELECT count(username) as done FROM accounts WHERE referral='$refid'";
    $ps_refsInserted = $dbh->prepare($refsInserted);
    $ps_refsInserted->execute();
    $row = $ps_refsInserted->fetch();
    $refsInserted = $row['done'];

    // Refscompleted
    $refsCompleted = "SELECT count(username) as done FROM accounts WHERE referral='$refid' AND finished=1";
    $ps_refsCompleted = $dbh->prepare($refsCompleted);
    $ps_refsCompleted->execute();
    $row2 = $ps_refsCompleted->fetch();
    $refsCompleted = $row2['done'];

    // Update fields for local order db
    $updateGifting = "UPDATE Gifting SET refsInserted = :refsInserted, refsCompleted = :refsCompleted WHERE refId = :refId";
    $ps_updateGifting = $db->prepare($updateGifting);

    $ps_updateGifting->bindParam(':refsInserted', $refsInserted);
    $ps_updateGifting->bindParam(':refsCompleted', $refsCompleted);
    $ps_updateGifting->bindParam(':refId', $refid);
    $ps_updateGifting->execute();
    echo "$refid: $refsInserted Refs inserted / $refsCompleted Refs completed<br>";
}

$dbh->commit();

Answer 1

您可以在一个带有相关子查询的查询中完成所有这些操作：

UPDATE Gifting
SET
    refsInserted=(SELECT COUNT(USERNAME)
                    FROM accounts
                    WHERE referral=Gifting.refId),
    refsCompleted=(SELECT COUNT(USERNAME)
                    FROM accounts
                    WHERE referral=Gifting.refId
                        AND finished=1)

相关子查询本质上是使用引用父查询的子查询（查询中的查询）。因此请注意，在每个子查询中，我在每个子查询的 where 子句中引用了 Gifting.refId 列。虽然这不是最好的性能，因为这些子查询中的每一个仍然必须独立于其他查询运行，但它的性能会比您那里的性能好得多（并且可能与您将获得的一样好）。

编辑：

仅供参考。我不知道交易是否会在这里有所帮助。通常，当您有多个相互依赖的查询时使用它们，并为您提供一种在失败时回滚的方法。例如，银行交易。在插入购买之前，您不希望余额扣除一些金额。如果购买由于某种原因插入失败，您希望将更改回滚到余额。因此，在插入购买时，您开始交易，运行更新余额查询和插入购买查询，并且只有当两者都正确进入并经过验证时，您才承诺保存。

编辑2：

如果我这样做，不进行导出/导入，这就是我会做的。不过，这做出了一些假设。首先是您使用的是 mssql 2008 或更新版本，其次是推荐 ID 始终是一个数字。我还使用了一个插入数字的临时表，因为您可以使用单个查询轻松插入多行，然后运行单个更新查询来更新礼物表。此临时表遵循CREATE TABLE tempTable (refId int, done int, total int) 的结构。

//get list of referral accounts
//if you are using one column, only query for one column
$listAccounts = "SELECT DISTINCT refId FROM Gifting WHERE refsCompleted <= 100 ORDER BY idGifting ASC";
$ps_listAccounts = $db->prepare($listAccounts);
$ps_listAccounts->execute();

//loop over and get list of refIds from above.
$refIds = array();
foreach($ps_listAccounts as $row){
    $refIds[] = $row['refId'];
}


if(count($refIds) > 0){
    //implode into string for use in query below
    $refIds = implode(',',$refIds);

    //select out total count
    $totalCount = "SELECT referral, COUNT(username) AS cnt FROM accounts WHERE referral IN ($refIds) GROUP BY referral";
    $ps_totalCounts = $dbh->prepare($totalCount);
    $ps_totalCounts->execute();

    //add to array of counts
    $counts = array();

    //loop over total counts
    foreach($ps_totalCounts as $row){
        //if referral id not found, add it
        if(!isset($counts[$row['referral']])){
            $counts[$row['referral']] = array('total'=>0,'done'=>0);
        }
        //add to count
        $counts[$row['referral']]['total'] += $row['cnt'];
    }

    $doneCount = "SELECT referral, COUNT(username) AS cnt FROM accounts WHERE finished=1 AND referral IN ($refIds) GROUP BY referral";
    $ps_doneCounts = $dbh->prepare($doneCount);
    $ps_doneCounts->execute();

    //loop over total counts
    foreach($ps_totalCounts as $row){
        //if referral id not found, add it
        if(!isset($counts[$row['referral']])){
            $counts[$row['referral']] = array('total'=>0,'done'=>0);
        }
        //add to count
        $counts[$row['referral']]['done'] += $row['cnt'];
    }

    //now loop over counts and generate insert queries to a temp table.
    //I suggest using a temp table because you can insert multiple rows
    //in one query and then the update is one query.
    $sqlInsertList = array();
    foreach($count as $refId=>$count){
        $sqlInsertList[] = "({$refId}, {$count['done']}, {$count['total']})";
    }

    //clear out the temp table first so we are only inserting new rows
    $truncSql = "TRUNCATE TABLE tempTable";
    $ps_trunc = $db->prepare($truncSql);
    $ps_trunc->execute();

    //make insert sql with multiple insert rows
    $insertSql = "INSERT INTO tempTable (refId, done, total) VALUES ".implode(',',$sqlInsertList);
    //prepare sql for insert into mssql
    $ps_insert = $db->prepare($insertSql);
    $ps_insert->execute();

    //sql to update existing rows
    $updateSql = "UPDATE Gifting
                    SET refsInserted=(SELECT total FROM tempTable WHERE refId=Gifting.refId),
                        refsCompleted=(SELECT done FROM tempTable WHERE refId=Gifting.refId)
                    WHERE refId IN (SELECT refId FROM tempTable)
                        AND refsCompleted <= 100";
    $ps_update = $db->prepare($updateSql);
    $ps_update->execute();
} else {
    echo "There were no reference ids found from \$dbh";
}