【问题标题】:PHP - Import CSV file to mysql database Using LOAD DATA INFILEPHP - 使用 LOAD DATA INFILE 将 CSV 文件导入 mysql 数据库
【发布时间】:2013-09-25 17:50:38
【问题描述】:

我有一个这样的 .csv 文件数据

Date,Name,Call Type,Number,Duration,Address,PostalCode,City,State,Country,Latitude,Longitude
"Sep-18-2013 01:53:45 PM","Unknown","outgoing call",'123456',"0 Secs","null","null","null","null","null",0.0,0.0,,,
"Sep-18-2013 01:54:14 PM","Unknown","outgoing call",'1234567890',"0 Secs","null","null","null","null","null",0.0,0.0,,,
"Sep-18-2013 01:54:37 PM","Unknown","outgoing call",'14772580369',"1 Secs","null","null","null","null","null",0.0,0.0,,,

我正在使用以下代码将数据插入数据库

$sql = "LOAD DATA INFILE `detection.csv`
              INTO TABLE `calldetections`
              FIELDS TERMINATED BY '".@mysql_escape_string(",").
             "` OPTIONALLY ENCLOSED BY `".@mysql_escape_string("\"").
             "` OPTIONALLY ENCLOSED BY `".@mysql_escape_string("\'").
             "` ESCAPED BY `".@mysql_escape_string("\\").
              "` LINES TERMINATED BY `".",,,\\r\\n".
             "`IGNORE 1 LINES `"

             ."(`date`,`name`,`type`,`number`,`duration`,`addr`,`pin`,`city`,`state`,`country`,`lat`,`log`)";
      $res = @mysql_query($con,$sql); 

但没有插入任何内容;哪里错了?

【问题讨论】:

    标签: php mysql sql csv load-data-infile


    【解决方案1】:

    如果您在执行之前执行echo($sql);,您会发现查询的语法不正确,原因如下:

    1. 文件名应该用引号而不是反引号括起来,因为它是字符串文字而不是标识符。

    2. 绝对没有必要在FIELDS TERMINATED BYENCLOSED BYESCAPED BY子句中调用mysql_escape_string()来指定分隔符。

    3. 你过度使用反引号。实际上,在您的情况下,由于没有使用保留字,因此您将它们全部丢弃。它们只会增加混乱。

    4. 在 CSV 文件的第一行末尾,您必须有 ,,,,因为您将它们用作行分隔符的一部分。如果您不这样做,您不仅会跳过第一行,还会跳过包含数据的第二行。

    5. 您不能多次使用ENCLOSED BY 子句。您必须以不同的方式处理Number 字段。

    6. 查看您的示例行恕我直言,您不需要 ESCAPED BY。但是,如果您觉得需要它,请像这样使用它ESCAPED BY '\\'

    话虽如此,句法正确的语句可能看起来像这样

    LOAD DATA INFILE 'detection.csv'
    INTO TABLE calldetections
    FIELDS TERMINATED BY ','
    OPTIONALLY ENCLOSED BY '"' 
    LINES TERMINATED BY ',,,\r\n'
    IGNORE 1 LINES 
    (date, name, type, number, duration, addr, pin, city, state, country, lat, log)
    

    现在恕我直言,您需要在加载时转换相当多的字段:

    1. 如果你表中的datedatetime数据类型,那么它需要被转换,否则你会得到一个错误

      日期时间值不正确:行中“日期”列的“日期”列“2013 年 9 月 18 日 01:53:45”

    2. 您必须处理围绕 Number 字段中的值的单个 qoutes

    3. 您很可能希望将 "null" 字符串文字更改为实际的 NULL addr, pin, city, state, country

    4. 如果持续时间始终以秒为单位,那么您可以提取一个整数秒值并将其存储在您的表中,以便以后能够轻松地聚合持续时间值。

    话虽如此,该语句的有用版本应该看起来像这样

    LOAD DATA INFILE 'detection.csv'
    INTO TABLE calldetections
    FIELDS TERMINATED BY ','
    OPTIONALLY ENCLOSED BY '"' 
    LINES TERMINATED BY ',,,\r\n'
    IGNORE 1 LINES 
    (@date, name, type, @number, @duration, @addr, @pin, @city, @state, @country, lat, log)
    SET date = STR_TO_DATE(@date, '%b-%d-%Y %h:%i:%s %p'),
        number = TRIM(BOTH '\'' FROM @number),
        duration = 1 * TRIM(TRAILING 'Secs' FROM @duration),
        addr = NULLIF(@addr, 'null'),
        pin  = NULLIF(@pin, 'null'),
        city = NULLIF(@city, 'null'),
        state = NULLIF(@state, 'null'),
        country = NULLIF(@country, 'null') 
    

    下面是在我的机器上执行查询的结果

    mysql> 加载数据输入文件'/tmp/detection.csv' -> INTO TABLE 调用检测 -> 由“,”终止的字段 -> 可选地由 '"' 包围 -> 以 ',,,\n' 结尾的行 -> 忽略 1 行 ->(@日期、名称、类型、@number、@duration、@addr、@pin、@city、@state、@country、lat、log) -> 设置日期 = STR_TO_DATE(@date, '%b-%d-%Y %h:%i:%s %p'), -> number = TRIM(BOTH '\'' FROM @number), -> 持续时间 = 1 * TRIM(TRAILING 'Secs' FROM @duration), -> addr = NULLIF(@addr, 'null'), -> pin = NULLIF(@pin, 'null'), -> 城市 = NULLIF(@city, 'null'), -> 状态 = NULLIF(@state, 'null'), -> 国家 = NULLIF(@country, 'null'); 查询正常,3 行受影响(0.00 秒) 记录:3 删除:0 跳过:0 警告:0 mysql> select * from calldetections; +----------+---------+--------------+- ------------+----------+------+------+------+----- --+---------+------+------+ |日期 |姓名 |类型 |号码 |持续时间 |地址 |销 |城市|状态 |国家 |纬度 |日志 | +----------+---------+--------------+- ------------+----------+------+------+------+----- --+---------+------+------+ | 2013-09-18 13:53:45 |未知 |拨出电话 | 123456 | 0 |空 |空 |空 |空 |空 | 0.0 | 0.0 | | 2013-09-18 13:54:14 |未知 |拨出电话 | 1234567890 | 0 |空 |空 |空 |空 |空 | 0.0 | 0.0 | | 2013-09-18 13:54:37 |未知 |拨出电话 | 14772580369 | 1 |空 |空 |空 |空 |空 | 0.0 | 0.0 | +----------+---------+--------------+- ------------+----------+------+------+------+----- --+---------+------+------+ 3 行一组(0.00 秒)

    最后在 php 中将查询字符串分配给 $sql 变量应该如下所示

    $sql = "LOAD DATA INFILE 'detection.csv'
            INTO TABLE calldetections
            FIELDS TERMINATED BY ','
            OPTIONALLY ENCLOSED BY '\"' 
            LINES TERMINATED BY ',,,\\r\\n'
            IGNORE 1 LINES 
            (@date, name, type, @number, @duration, @addr, @pin, @city, @state, @country, lat, log)
            SET date = STR_TO_DATE(@date, '%b-%d-%Y %h:%i:%s %p'),
                number = TRIM(BOTH '\'' FROM @number),
                duration = 1 * TRIM(TRAILING 'Secs' FROM @duration),
                addr = NULLIF(@addr, 'null'),
                pin  = NULLIF(@pin, 'null'),
                city = NULLIF(@city, 'null'),
                state = NULLIF(@state, 'null'),
                country = NULLIF(@country, 'null') ";
    

    【讨论】:

    • 非常感谢您的详细回答。
    • @chimbu 不客气。祝你好运 :) 如果需要,请随时提出后续问题。
    • @peterm 多么棒的、清晰的答案。如果你不是一个薪水很高的老师,你应该认真考虑一下。谢谢。
    • pdo_connect(); $tt="1cs2cs1cs17409061651452795086two.csv"; $aa=","; $bb='"'; $cc='\n'; echo $query = "LOAD DATA LOCAL INFILE '".$tt."' INTO TABLE data2 FIELDS TERMINATED BY '".$aa."' 可选地由 ' 封闭".$bb."' LINES TERMINATED BY '".$cc."' IGNORE 1 LINES;"; $queryresource = $pdo_connection->query($query); ?> 我正在尝试使用与建议相同的方式插入 2 行而不是 csv 文件有 100 万行。
    • @abhinavBruteForce 您是否尝试在 php 之外运行它并查找 MySQL 错误和警告?很可能您的数据有问题(不正确/不兼容的类型、违反非空/外来约束的缺失值/值等)
    【解决方案2】:

    在数据库中1分钟内批量插入超过7000000条记录(超快速查询和计算)

        mysqli_query($cons, '
        LOAD DATA LOCAL INFILE "'.$file.'"
        INTO TABLE tablename
        FIELDS TERMINATED by \',\'
        LINES TERMINATED BY \'\n\'
        IGNORE 1 LINES
        (isbn10,isbn13,price,discount,free_stock,report,report_date)
         SET RRP = IF(discount = 0.00,price-price * 45/100,IF(discount = 0.01,price,IF(discount != 0.00,price-price * discount/100,@RRP))),
             RRP_nl = RRP * 1.44 + 8,
             RRP_bl = RRP * 1.44 + 8,
             ID = NULL
        ')or die(mysqli_error());
        $affected = (int) (mysqli_affected_rows($cons))-1; 
        $log->lwrite('Inventory.CSV to database:'. $affected.' record inserted successfully.');
    

    RRP 和 RRP_nl 和 RRP_bl 不在 csv 中,但我们是在插入之后计算出来的。

    【讨论】:

    猜你喜欢
    • 1970-01-01
    • 2011-05-28
    • 2021-10-02
    • 1970-01-01
    • 2013-05-03
    • 1970-01-01
    • 1970-01-01
    • 2015-11-22
    相关资源
    最近更新 更多