【问题标题】:Artillery: Load JSON document from CSV file炮兵:从 CSV 文件加载 JSON 文档
【发布时间】:2025-12-25 22:35:11
【问题描述】:

我想要测试的 lambda 函数需要一个复杂的 json,根据我的理解,json 需要进入 csv 文件。我的问题是我尝试了各种方法从 csv 加载 json,但不断出错。我不确定这在炮兵中是否可行。我的示例 csv 如下所示。

post-data.csv

第 1 列 {“个人资料”:{“姓名”:“irfan”,“电子邮件”:“irfan@email.com”},“地址”:[“地址1”,“地址2”]} { "profile":{"name":"Tomas","email":"tomas@email.com"},"address":["address1","address2"]} { "profile":{"name":"Joel","email":"joel@email.com"},"address":["address1","address2"]}

我只有一列,因为我只想要这个 json 文档作为请求正文传递给我的 hello world lambda,这样我就可以对其进行负载测试。

这是我的炮兵脚本文件的内容。

config:
  target: "https://api-gateway-load-testing-tst-ap-southeast-2.xxxxxxxxxxx.com"
  phases:
    - 
      duration: 5
      arrivalRate: 1
  defaults:
    headers:
      x-api-key: "xxxxxxxxxxxxxxxxxxxxxxxxxxx"
      Content-Type: "application/json"
  payload:
    # path is relative to the location of the test script
    path: "post-data.csv"
    fields:
      - "column1"
    order: sequence
    delimiter: "~"
    skipHeader: true
    cast: false
  plugins:
    cloudwatch:
      namespace: "serverless-artillery-loadtest"
scenarios:
  - flow:
      - post:
          url: "/v1/hello-world"
          json:
            data: {{ column1 }}

当我在 json 中的键和值周围加上双引号时,我收到错误消息“执行任务时出错:在 1579692773908 中执行从 1579692773908 加载时遇到错误异常:炮兵以非零代码退出:”

有没有什么方法可以从 csv 加载 json,以使我的 hello world lambda 函数以以下格式接收请求正文为 json:

{“数据”:{“个人资料”:{“姓名”:“irfan”,“电子邮件”:“irfan@email.com”},“地址”:[“地址1”,“地址2”]}}

任何帮助将不胜感激。

【问题讨论】:

    标签: javascript node.js json artillery


    【解决方案1】:

    我自己通过编写自定义 JavaScript 来加载 json 有效负载而不是从 csv 文件中设法解决了这个问题。我使用 config.processor 和 beforeScenario 挂钩来定义我的自定义逻辑。

    对于可能面临类似问题的任何人,这是我的解决方案:

    script.yml

    config:
      target: "https://api-ap-southeast-2.aws.my-domain.com"
      processor: "./post-body.js"
      # Following phases test a scenario where 0 tps ramps up to 50 tps in 1 minutes, and then ramps up to 1000 tps every 30 seconds in 50 tps increment
    
      phases:
        - 
          duration: 60
          arrivalRate: 10
          rampTo: 50
        -
          duration: 30
          arrivalRate: 50
        -
          duration: 30
          arrivalRate: 100
        -
          duration: 30
          arrivalRate: 150
        -
          duration: 30
          arrivalRate: 200
        -
          duration: 30
          arrivalRate: 250
        -
          duration: 30
          arrivalRate: 300
        -
          duration: 30
          arrivalRate: 350
        -
          duration: 30
          arrivalRate: 400
        -
          duration: 30
          arrivalRate: 450
        -
          duration: 30
          arrivalRate: 500
        -
          duration: 30
          arrivalRate: 550
        -
          duration: 30
          arrivalRate: 600
        -
          duration: 30
          arrivalRate: 650
        -
          duration: 30
          arrivalRate: 700
        -
          duration: 30
          arrivalRate: 750
        -
          duration: 30
          arrivalRate: 800
        -
          duration: 30
          arrivalRate: 850
        -
          duration: 30
          arrivalRate: 900
        -
          duration: 30
          arrivalRate: 950
        -
          duration: 270
          arrivalRate: 1000
      defaults:
        headers:
          x-api-key: "fake-x-api-key"
          Content-Type: "application/json"
      plugins:
        cloudwatch:
          namespace: "my-service-name"
        influxdb:
          testName: "my-service Load Test Results"
          influx:
            host: "fake-ip-address"
            username: "fake-username"
            password: "fake-password"
            database: "influx"
    
    scenarios:
      - name: my-service-name load test with varying load
        beforeScenario: generatePostBody
        flow:
          - post:
              url: "/my-fake-endpoint"
              json:
                "{{ data }}"
    

    以下 post-body.js 包含我的自定义 JS 逻辑。我引入了一个新的 txt 文件 post-data.txt,它基本上替换了我在问题中提到的 csv 文件,以托管数千行,其中每一行都是作为 json 的请求有效负载。每次执行场景时,都会获取一个随机的 json 有效负载字符串,将其转换为 json 对象并作为 POST 请求的一部分发送。我也在使用 CloudWatch 和 InfluxDB 来输出结果。

    post-body.js

    const fs = require("fs");
    const filePath = "./post-data.txt";
    let postData;
    
    /**
     * Generates post body
     */
    const generatePostBody = async (userContext, events, done) => {
      try{
        // add variables to virtual user's context:
        if(postData === undefined || postData === null || postData.length === 0) {
          postData = await loadDataFromTxt(filePath);
        }
        const postBodyStr = postData[Math.floor(Math.random()*postData.length)];
        userContext.vars.data = JSON.parse(postBodyStr);
    
        // continue with executing the scenario:
        return done();
    
      } catch(err) {
        console.log(`Error occurred in function generatePostBody. Detail: ${err}`);
        throw(err);
      }
    }
    
    /**
     * Loads post body from csv file
     * @param {object} filePath - The path of csv file
     */
    const loadDataFromCsv = async filePath => {
      const data = [];
    
      return new Promise((resolve, reject) => {
        fs.createReadStream(filePath)
          .pipe(csv({delimiter: '||'}))
          .on("data", data => data.push(data))
          .on("end", () => {
            return resolve(data);
          })
          .on("error", error => {
            return reject(error);
          });
      });
    };
    
    /**
     * Loads post body from text file
     * @param {object} filePath - The path of text file
     */
    const loadDataFromTxt = async (path) => {
      return new Promise((resolve, reject) => {
        fs.readFile(path, 'utf8', function (err, data) {
          if (err) {
            reject(err);
          }
          resolve(data.toString().split("\n"));
        });
      });
    }
    
    // Load data from txt file once at the start of the execution
    // and save the results in a global variable
    (async () => {
      try {
      postData = await loadDataFromTxt(filePath);
      //console.log(JSON.parse(postData[0]));
      } catch (error) {
        console.log(`Error occurred in main. Detail: ${err}`);
      }
    })();
    
    module.exports.generatePostBody = generatePostBody; 
    

    post-data.txt

    { "profile":{"name":"irfan","email":"irfan@email.com"},"address":["address1","address2"]}
    { "profile":{"name":"Tomas","email":"tomas@email.com"},"address":["address1","address2"]}
    { "profile":{"name":"Joel","email":"joel@email.com"},"address":["address1","address2"]}
    

    HTH

    【讨论】: