downloadFile files allocate huge memories

[ueokande]

Package

• @kintone/rest-api-client 3.1.11

Why

The FileClinet downloads blobs into memory. This occurs high memory allocation depends on file size.

I created kintone records which have 3 attachments, each attachment have 1GiB file size. The following code allocate over 6GiB memory size.

// index.js
const { KintoneRestAPIClient } = require("@kintone/rest-api-client");

const client = new KintoneRestAPIClient({
  baseUrl: "https://my-domain.kintone.com",
  auth: {
    username: process.env.KINTONE_USERNAME,
    password: process.env.KINTONE_PASSWORD,
  },
});

(async () => {
  const { record } = await client.record.getRecord({ app: 1, id: 2 });
  const blobs = await Promise.all(
    record["Attachment"].value.map(value => client.file.downloadFile({
      fileKey: value.fileKey,
    }))
  )
  for (const blob of blobs) {
    console.log("Downloaded blob: ", blob.length);
  }

  for (const [key, value] of Object.entries(process.memoryUsage())) {
    console.log(`${key}: ${Math.round(value / 1024 / 1024 * 100) / 100} MiB`)
  }
})();

$ node index.js
Downloaded blob:  1073741824
Downloaded blob:  1073741824
Downloaded blob:  1073741824
rss: 6628.86 MiB
heapTotal: 287.4 MiB
heapUsed: 203.38 MiB
external: 5145.33 MiB
arrayBuffers: 5124.03 MiB

What

Well-known libraries and APIs returns a stream rather than buffer array. fetch() API returns a ReadableStream. It is useful to pipe other process. Of course using streams enable to download into memory.

• https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream