如何通过适当的 async/await/promises 提高此代码的速度

Question

使用 Node.js，我的任务是改进我创建的代码。此代码执行 60 个 HTTP 请求并为此使用库。

处理所有 HTTP 请求并将每个请求保存到文件需要 30 秒！

据说可以在 3 秒内完成这些请求：

1.正确管理异步承诺

2。更智能的缓存

3.不使用集群

4.只添加一次开销

恐怕我不知道从哪里开始了解我到底能做什么。

所以下面的代码得到了一个包含 60 项的数组，其中每一项都是一个 HTTP 请求：

const exchanges = ccxt.exchanges

这些进入：worker = async 函数并在代码末尾：await Promise.all(workers) 等待它们完成。

我不知道从哪里开始才能真正降到 3 秒。怎样才能提高这段代码的速度？

'use strict';

const ccxt  = require ('ccxt')
    , log   = require ('ololog').noLocate // npm install ololog
    , fs    = require ('fs')

    // the numWorkers constant defines the number of concurrent workers
    // those aren't really threads in terms of the async environment
    // set this to the number of cores in your CPU * 2
    // or play with this number to find a setting that works best for you
    , numWorkers = 8

;(async () => {

    // make an array of all exchanges
    const exchanges = ccxt.exchanges

        .filter (id => ![ 'cap1', 'cap2' ].includes (id))

        // instantiate each exchange and save it to the exchanges list
        .map (id => new ccxt[id] ({
            'enableRateLimit': true,
        }))

    // the worker function for each "async thread"
    const worker = async function () {

        // while the array of all exchanges is not empty
        while (exchanges.length > 0) {

            // pop one exchange from the array
            const exchange = exchanges.pop()

            // check if it has the necessary method implemented
            if (exchange.has['fetchTickers']) {

                // try to do "the work" and handle errors if any
                try {

                    // fetch the response for all tickers from the exchange
                    const tickers = await exchange.fetchTickers()

                    // make a filename from exchange id
                    const filename = '/myproject/tickers/' + exchange.id + 'Tickers.json'

                    // save the response to a file
                    fs.writeFileSync(filename, JSON.stringify({ tickers }));

                } catch (e) { } //Error
            }
        }
    }

    // create numWorkers "threads" (they aren't really threads)
    const workers = [ ... Array (numWorkers) ].map (_ => worker ())

    // wait for all of them to execute or fail
    await Promise.all (workers)

}) ()

Answer 1

我尝试看看是否有可能做得更快。我尝试缓存所有可能需要的内存。在实际执行 .fetchTickers() 请求之前。

我设法从看起来的 15 秒缩短到了 9 秒。但是下面的代码更进一步，但我确实收到编译错误并且不确定我做错了什么。

错误是：

ReferenceError: id is not defined

id 不是在 .pushed 到 'exchangesArray' 的 'exchange' 对象中传递吗？

我正在尝试首先将交换对象放入一个数组中：

var exchangesArray = [];

然后有了这个“exchangesArray”，我尝试执行执行 fetchTickers 的函数：

'use strict';
const ccxt = require('ccxt');
const fs = require('fs');
const path = require('path');

//Cache some memories first
var exchangesArray = [];
(async () => {
  const allexchanges = ccxt.exchanges.filter((id) => !['coinmarketcap', 'theocean'].includes(id))
        .map(async (id) => {
            const Exchange = ccxt[id];
            const exchange = new Exchange({ enableRateLimit: true });
            if (exchange.has['fetchTickers']) {

                exchangesArray.push(exchange);
            }
        });

    await Promise.all(allexchanges);
})();

//Use cached memories to do the "fetchTickers()" as fast as possible
(async () => {
    const start = Date.now();

    const exchanges = exchangesArray;
    while (exchanges.length > 0) {

        // pop one exchange from the array
        const exchange = exchanges.pop()

        try {
            const tickers = await exchange.fetchTickers();
            const dumpFile = path.join(__dirname, 'exchanges', `${id}-Tickers.json`);
            await fs.promises.writeFile(dumpFile, JSON.stringify(tickers));
        } catch (e) {
            console.error(e);
        }

    }

    await Promise.all(exchanges);

    const end = Date.now();
    console.log(`Done in ${(end - start) / 1000} seconds`);
})();

Answer 2

我认为你让事情变得比他们需要的更复杂。您可以在map 回调中完成所有工作，然后使用Promise.all(promises) 等待所有操作完成。此过程确实比预期的“3 秒”（在我的情况下为 15 秒）花费的时间更长，并且产生了很多错误（例如缺少 apiToken，或未实现 fetchTickers），但这可能是我的环境的问题（我'之前从未使用过ccxt，而且我没有任何apiTokens）。

这是我想出的实现，希望它可以帮助您满足您的需求：

const ccxt = require('ccxt');
const fs = require('fs');
const path = require('path');

(async () => {
    const start = Date.now();

    const dumps = ccxt.exchanges
        .filter((id) => !['coinmarketcap', 'theocean'].includes(id))
        .map(async (id) => {
            const Exchange = ccxt[id];
            const exchange = new Exchange({enableRateLimit: true});
            if (exchange.has['fetchTickers']) {
                try {
                    const tickers = await exchange.fetchTickers();
                    const dumpFile = path.join(__dirname, 'exchanges', `${id}-Tickers.json`);
                    await fs.promises.writeFile(dumpFile, JSON.stringify(tickers));
                } catch (e) {
                    console.error(e);
                }
            }
        });

    await Promise.all(dumps);

    const end = Date.now();
    console.log(`Done in ${(end - start) / 1000} seconds`);
})();