Suggest an edit

Improve this article

Refine the answer for “Splitting a heavy task into chunks”. Your changes go to moderation before they’re published.

Approval required

Content

What you’re changing

Title (EN)

Short answer (EN)

**The chunking principle is simple: do a small slice of work (roughly 1 to 2 ms), schedule the continuation as a macrotask so the engine can handle UI, requests and timers, and repeat until the work is done.** The key detail: `await Promise.resolve()` does not help here, because it is a microtask and no frame is painted. You need an actual macrotask: `setTimeout(fn, 0)`, `MessageChannel`, `requestAnimationFrame`, `requestIdleCallback` in the browser, or `setImmediate` in Node.js. You can slice by time budget (work for N milliseconds, then yield the frame) or by a fixed number of iterations per slice. If the computation is genuinely heavy, move it into a Web Worker or `worker_threads` instead.

```javascript
function forEachChunked(arr, fn, { chunkSize = 10_000 } = {}) {
  return new Promise((resolve) => {
    let i = 0;
    (function loop() {
      const end = Math.min(i + chunkSize, arr.length);
      for (; i < end; i++) fn(arr[i], i);
      i < arr.length ? setTimeout(loop, 0) : resolve();
    })();
  });
}
```

**Key point:** a small slice of work plus a macrotask for the continuation; microtasks never yield a frame, and truly heavy work belongs in a worker.

Shown above the full answer for quick recall.

Answer (EN)

**Chunking is the technique of cutting a heavy computation into short slices and handing control back to the event loop between them through a macrotask.** That way the browser gets to paint a frame and process input, and a Node.js server gets to serve other requests.

## Theory

### TL;DR
- Do a small slice of work, roughly 1 to 2 ms.
- Schedule the continuation as a macrotask, so the browser or Node can handle UI, requests and timers.
- Repeat until the work is finished.
- `await Promise.resolve()` does not help: it is a microtask and it does not yield a frame.
- Macrotasks: `setTimeout(fn, 0)`, `setImmediate`, `MessageChannel`, `requestAnimationFrame`, `requestIdleCallback`.
- Very heavy math or parsing is better moved into a Web Worker or `worker_threads` than chunked by hand.

### Quick example
```javascript
// Chunking by operation count: simple and reliable
function forEachChunked(arr, fn, { chunkSize = 10_000, schedule = setTimeout } = {}) {
  return new Promise((resolve) => {
    let i = 0;
    (function loop() {
      const end = Math.min(i + chunkSize, arr.length);
      for (; i < end; i++) fn(arr[i], i);
      i < arr.length ? schedule(loop, 0) : resolve();
    })();
  });
}
```

### The principle
- Do a small slice of work (for example, 1 to 2 ms).
- Schedule the continuation through a macrotask (so the browser or Node can process UI, requests, timers).
- Repeat until it is done.

> Important: `await Promise.resolve()` does not help, it is a microtask and it does not yield the frame or the UI. You need a macrotask: `setTimeout(0)`, `setImmediate`, `MessageChannel`, `requestAnimationFrame`, `requestIdleCallback`.

### A general purpose chunking utility
#### In the browser (UI friendly, 60fps)
```javascript
// Schedulers for different modes:
const scheduleTimeout = (fn) => setTimeout(fn, 0);                     // simple
const scheduleRAF = (fn) => requestAnimationFrame(fn);                 // gentle on the UI
const scheduleIdle = (fn) => requestIdleCallback(fn, { timeout: 50 }); // when the browser is idle (a fallback is mandatory)

// Time based chunker: works for about N milliseconds, then yields the loop
export function runSliced(work, { budgetMs = 4, schedule = scheduleRAF } = {}) {
  return new Promise((resolve, reject) => {
    try {
      const step = (deadlineLike) => {
        const start = performance.now();
        while (work.hasMore()) {
          work.doOne();
          const now = performance.now();
          // Over budget, so yield the frame
          if (now - start >= budgetMs) {
            schedule(step);
            return;
          }
        }
        resolve();
      };
      schedule(step);
    } catch (e) { reject(e); }
  });
}

// Usage example: processing a large array
export async function processBigArray(items) {
  let i = 0;
  const work = {
    hasMore: () => i < items.length,
    doOne: () => {
      const x = items[i];
      // ...heavy work on x...
      i++;
    }
  };
  await runSliced(work, { budgetMs: 3, schedule: scheduleRAF });
}
```

#### Node.js (so the server does not hang)
```javascript
const scheduleImmediate = (fn) => setImmediate(fn);

// Example:
async function crunch() {
  let i = 0, N = 1e9, sum = 0;
  const work = {
    hasMore: () => i < N,
    doOne: () => { sum += i % 7; i++; }
  };
  await runSlicedNode(work, { iterationsPerSlice: 100_000 });
  return sum;
}
```

### Quick recipes
#### 1) Chunking by operation count (simple and reliable)
```javascript
function forEachChunked(arr, fn, { chunkSize = 10_000, schedule = setTimeout } = {}) {
  return new Promise((resolve) => {
    let i = 0;
    (function loop() {
      const end = Math.min(i + chunkSize, arr.length);
      for (; i < end; i++) fn(arr[i], i);
      i < arr.length ? schedule(loop, 0) : resolve();
    })();
  });
}
```

#### 2) With `requestAnimationFrame` (smooth UI)
```javascript
function timeSliced(fn, { budgetMs = 4 } = {}) {
  return new Promise((resolve) => {
    function tick() {
      const start = performance.now();
      let more = true;
      while (more && performance.now() - start < budgetMs) {
        more = fn(); // fn() does a little work and returns true if something is left
      }
      more ? requestAnimationFrame(tick) : resolve();
    }
    requestAnimationFrame(tick);
  });
}
```

#### 3) Maximum gentleness, `requestIdleCallback` (where available)
```javascript
function idleSliced(fn, timeout = 50) {
  return new Promise((resolve) => {
    function step(deadline) {
      let more = true;
      while (more && (deadline.timeRemaining() > 1)) {
        more = fn();
      }
      more ? requestIdleCallback(step, { timeout }) : resolve();
    }
    requestIdleCallback(step, { timeout });
  });
}
```

#### 4) The fastest macrotask scheduler, `MessageChannel`
```javascript
function macroTaskScheduler() {
  const ch = new MessageChannel();
  let queue = [];
  ch.port1.onmessage = () => queue.shift()?.();
  return (fn) => {
    queue.push(fn);
    ch.port2.postMessage(0);
  };
}
const scheduleFast = macroTaskScheduler();
```

### Which one to pick
- **Animations and interactive UI:** `requestAnimationFrame` with a budget of 2 to 5 ms.
- **Low priority, background tasks:** `requestIdleCallback` (with a `setTimeout` fallback).
- **Simple and available everywhere:** `setTimeout(0)` or `MessageChannel`.
- **Node.js:** `setImmediate`, or slicing by iteration count.
- **Very heavy math or parsing:** better moved into a Web Worker (browser) or `worker_threads` (Node), so you do not have to slice it by hand.

### Common mistakes
Antipatterns to avoid:

- Infinite or long synchronous loops with no yielding (`while (true) {}` or a `for` over billions of iterations).
- Hoping that `await Promise.resolve()` relieves the UI, it is a microtask and no frame is painted.
- Synchronous I/O and crypto in Node.js (`readFileSync`, `pbkdf2Sync`), they block the whole process.
- Slices that are too small: scheduling a separate macrotask per element means the scheduler overhead eats the entire gain and the job stretches into minutes.
- `requestIdleCallback` with no `timeout` and no fallback: on a busy page the task may never run, and in some engines the API simply does not exist.
- Reading DOM sizes or styles inside a chunk: that forces a synchronous reflow and undoes the frame budget you were saving.

Markdown · drag & drop images · ⌘B / ⌘I shortcuts1002 words

For the reviewer

Note to the moderator (optional)

Visible only to the moderator. Helps review go faster.