如何使用faker在laravel中更快地播种大型数据集

Question

我使用 laravel faker 和模型工厂将 25000 条记录播种到 1 个表中，并且需要将近 30 分钟才能完成，这只会延迟我的工作效率。有没有办法在更短的时间内播种如此大的数据集。大概一分钟。

这里是模型工厂

<?php

use App\Models\Client\Client;
use Faker\Generator;

$factory->define(Client::class, function (Generator $faker) {

    $employeecount  = [15,48,5,12,26,30];
    $numbers        = [0125554,0145755,0123456,8874545,4512156];
    $gross          = [25000,13500,10000,8500,5000,15000,30000];
    $gender         = ['Male','Female'];
    $marketing      = ['E-mail','SMS','Telephone'];
    $maritalstatus  = ['Single','Married','Divorced','Widow'];
    $maritaltype    = ['COP','ANC'];
    $addresstype    = ['Owner','Rent'];
    $serviceused    = ['Status Insight','Credit Union'];
    $riskstatus     = ['Low Risk','Medium Risk','High Risk'];
    $companytype    = ['Public Company','CC','Partnership','Sole Trader','PTY','TLD','Other'];



    return [
        'name'             =>$faker->firstName,
        'surname'          =>$faker->lastName,
        'idnumber'         =>$faker->unique()->ean8,
        'gender'           =>$gender[array_rand($gender)],
        'maritalstatus'    =>$maritalstatus[array_rand($maritalstatus)],
        'maritaltype'      =>$maritaltype[array_rand($maritaltype)],
        'maidenname'       =>$faker->lastName,
        'companytype'      =>$companytype[array_rand($companytype)],
        'companyname'      =>$faker->company,
        'businessnature'   =>$faker->catchPhrase,
        'vatnumber'        =>$numbers[array_rand($numbers)],
        'coregnumber'      =>$numbers[array_rand($numbers)],
        'tradingname'      =>$faker->bs,
        'employeecount'    =>$employeecount[array_rand($employeecount)],
        'employer'         =>$faker->company,
        'gross'            =>$gross[array_rand($gross)],
        'mobilenumber'     =>$numbers[array_rand($numbers)],
        'homenumber'       =>$numbers[array_rand($numbers)],
        'worknumber'       =>$numbers[array_rand($numbers)],
        'email'            =>$faker->email,
        'marketing'        =>$marketing[array_rand($marketing)],
        'physicaladd1'     =>$faker->streetAddress,
        'physicaladd2'     =>$faker->streetName,
        'physicalcity'     =>$faker->city,
        'physicalcode'     =>str_pad(rand(0, pow(10, 4) - 1), 4, '0', STR_PAD_LEFT),
        'yearsAtAddress'   =>str_pad(rand(0, pow(4, 2) - 1), 2),
        'monthsAtAddress'  =>str_pad(rand(0, pow(4, 2) - 1), 2),
        'addresstype'      =>$addresstype[array_rand($addresstype)],
        'postaladd1'       =>$faker->streetAddress,
        'postaladd2'       =>$faker->streetName,  
        'postalcity'       =>$faker->city,
        'postalcode'       =>str_pad(rand(0, pow(10, 4) - 1), 4, '0', STR_PAD_LEFT),
        'riskstatus'       =>$riskstatus[array_rand($riskstatus)],
        'noofjudgements'   =>str_pad(rand(0, pow(2, 2) - 1), 2),
        'noofdefaults'     =>str_pad(rand(0, pow(2, 2) - 1), 2),
        'dateofenquiry'    =>$faker->date($format = 'Y-m-d', $max = 'now'),
        'serviceused'      =>$serviceused[array_rand($serviceused)],

    ];

});

这是表格播种器

<?php

use App\Models\Client\Client;
use Illuminate\Database\Seeder;

class ClientTableSeeder extends Seeder
{
    /**
     * Run the database seeds.
     *
     * @return void
     */
    public function run()
    {
        factory(Client::class, 25000)->create();
    }
}

这是主播种机

<?php

use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Seeder;

/**
 * Class DatabaseSeeder.
 */

class DatabaseSeeder extends Seeder
{

    /**
     * Run the database seeds.
     *
     * @return void
     */
    public function run()
    {
        Model::unguard();

         $this->call(ClientTableSeeder::class);

        Model::reguard();
    }
}

谁有更好的解决方案？ 提前致谢。

Answer 1

尝试使用它。会有帮助的

composer 需要 fzaninotto/faker --dev

Answer 2

根据我的长期经验，在这种情况下，工厂不是要走的路。
我发现工作方式比 Factory 更快的解决方案是 Seeder。
我所做的是创建一个庞大的数据集合，例如 25000~50000 条记录。
然后使用 Collection::chunk 一次插入（而不是创建）许多（500 ~ 1000+）条记录。
最终减少查询次数，从而缩短时间。

例如

        $collectionToInsert = collect([]);
    
        $numberOfEntries = 25000;

        for ($i = 1; $i <= $numberOfEntries; $i++) {
            $collectionToInsert->push([
                'name' => $faker->firstName,
                'surname' => $faker->lastName,
                'idnumber' => $faker->unique()->ean8,
                // ...
            ]);
        }
        
        $chunks = $collectionToInsert->chunk(500);
        
         foreach ($chunks as $chunk) {
             SomeModel::insert($chunk->toArray());
         }

Answer 3

对于大数据，我们有 2 个解决方案： 1-：创建一个csv文件并使用“LOAD IN FILE”MYSQL命令。 https://www.mockaroo.com/ 可以帮助创建假 csv

2- 我用过

for($i=0 ; $i< 6000 ; $i++){
        $records = AhrefRecord::factory()->count(1000)->make();
        $chunks = $records->chunk(500);
        foreach ($chunks as $chunk) {
            AhrefRecord::insert($chunk->toArray());
        }
    }

用于制作 60m 记录并使我的插入速度提高 10 倍。只需选择块和“for 语句”取决于您的 php.ini memory_limit。

注意：对于我手动处理的关系插入： 我在 ahref_table 和 ahref_records_table 之间有关系 首先我在 ahref_table 中插入 800 条记录 然后我为第二个表（ahref_records_table）插入 60m 条记录，其中随机外部 id 介于 1 到 800 之间