Skip to content

Instantly share code, notes, and snippets.

@BioPhoton
Last active August 23, 2025 02:36
Show Gist options
  • Select an option

  • Save BioPhoton/4af1e2838cdc9f0340d72b08fca225f1 to your computer and use it in GitHub Desktop.

Select an option

Save BioPhoton/4af1e2838cdc9f0340d72b08fca225f1 to your computer and use it in GitHub Desktop.

ESLint Concurrency Benchmark Script

Overview

  • Benchmarks ESLint v9.34+ --concurrency across one or more targets
  • Runs each concurrency setting multiple times and aggregates stats
  • Prints a fastest-per-target table and a per-target breakdown (★ marks best)
  • Optional verbose mode prints ESLint commands, stderr, and one stylish preview

Defaults

  • concurrency: off,4,8,auto
  • runs: 3
  • useLocalConfig: true (uses <target>/eslint.config.js)
  • tsOnly: true (globs TS files under <target> recursively)
  • outDir: tools/eslint-perf/results

Flags

  • --targets=dir[,dir...]: Comma-separated target directories
  • --concurrency=v[,v...]: values in {off,auto,1..N}
  • --runs=N: Runs per configuration (default 3)
  • --verbose: Show ESLint commands/stderr and one stylish preview per target
  • --useLocalConfig=true|false: Pass --config <target>/eslint.config.js (default true)
  • --tsOnly=true|false: Limit to TS files under <target> recursively (default true)
  • --outDir=path: Directory for JSON + summary outputs

Examples

  • Minimal (defaults):
node --import tsx tools/eslint-perf/bench.compact.ts
  • Custom targets + concurrency:
node --import tsx tools/eslint-perf/bench.compact.ts \
  --targets=packages/lib-a,packages/lib-b \
  --concurrency=off,2,4,auto \
  --runs=5
  • Disable local config or TS-only:
node --import tsx tools/eslint-perf/bench.compact.ts \
  --useLocalConfig=false --tsOnly=false

Output

  • Raw JSON: tools/eslint-perf/results/system-info-*.json and eslint-benchmark-*.json
  • Summary: tools/eslint-perf/results/*.summary.md
  • Terminal: fastest table + per-target breakdown (★ marks best)

Output Example

> node --import tsx bench.ts --targets=packages/lib-a --concurrency=off,1,2,4,6,8,auto --runs=3 --verbose --useLocalConfig --tsOnly


📁 Target: packages/lib-a (files: 106, ts: 101, js: 5)
  🔧 Concurrency: off
    Run 1/3 ...
      $ npx eslint --config="packages/lib-a/eslint.config.js" --concurrency=off --format=json "packages/lib-a/**/*.ts"
      Time: 8.764s, Files: 101, Errors: 0, Warnings: 0
    Run 2/3 ...
    ✅ Avg: 9.49s (min 9.385s, max 9.615s, ±0.095s)
...

✅ Benchmark complete
Raw results: tools/eslint-perf/results/eslint-benchmark-2025-08-23T01-39-46-558Z.json
Summary: tools/eslint-perf/results/eslint-benchmark-2025-08-23T01-39-46-558Z.summary.txt

Fastest per target:
Target                        Best    Avg(s)    Baseline(s)   Speedup
---------------------------------------------------------------------
packages/lib-a                off     8.849     8.849         1.00x

Per-target breakdown (best marked with ★):

packages/lib-a
Concurrency   Avg(s)    StdDev    Speedup   Mark
------------------------------------------------
off           8.849     0.078     1.00x     ★
1             9.024     0.094     0.98x     
2             9.615     0.107     0.92x     
4             9.742     0.190     0.91x     
6             11.652    0.439     0.76x     
8             13.749    0.195     0.64x     
auto          9.490     0.095     0.93x     

CPU Measures

npx @push-based/cpu-prof@latest npx eslint .  --concurrency=4
/*
 * ESLint Concurrency Benchmark Suite
 * Runs ESLint with multiple concurrency settings across selected targets,
 * collects timings and quantitative data, and writes structured results.
 */
import { exec as execCb } from 'node:child_process';
import { mkdir, readdir, stat, writeFile } from 'node:fs/promises';
import os from 'node:os';
import path from 'node:path';
import { promisify } from 'node:util';

const exec = promisify(execCb);

type Concurrency = 'off' | 'auto' | `${number}`;

type BenchmarkRun = {
  target: string;
  concurrency: Concurrency;
  run: number;
  timing: {
    durationSeconds: number;
    realTimeSeconds: number | null;
    userTimeSeconds: number | null;
    sysTimeSeconds: number | null;
  };
  eslintResults: {
    filesProcessed: number;
    errors: number;
    warnings: number;
    eslintWarnings: string[];
  };
  timestamp: string;
};

type BenchmarkStats = {
  avg: number;
  min: number;
  max: number;
  stdDev: number;
  runs: number;
};

type TargetInfo = {
  target: string;
  tsFiles: number;
  jsFiles: number;
  totalFiles: number;
};

type SuiteResult = {
  target: string;
  targetInfo: TargetInfo;
  concurrency: Concurrency;
  statistics: BenchmarkStats;
  runs: BenchmarkRun[];
};

type SystemInfo = {
  timestamp: string;
  system: {
    os: string;
    osVersion: string;
    architecture: string;
    cpuCores: number;
    totalMemGb: number;
    hostname: string;
  };
  software: {
    nodeVersion: string;
    npmVersion: string;
    eslintVersion: string;
    nxVersion: string | null;
  };
};

// -----------------------------
// Config via CLI args (minimal)
// -----------------------------
const DEFAULT_TARGETS = [
  'packages/models',
  'packages/plugin-eslint',
  'packages/cli',
  'packages/core',
  'packages/utils',
];
const DEFAULT_CONCURRENCY: Concurrency[] = [
  'off',
  '1',
  '2',
  '4',
  '6',
  '8',
  'auto',
];
const DEFAULT_RUNS = 3;

function parseArgs() {
  const args = process.argv.slice(2);
  const getArg = (name: string): string | undefined => {
    const prefix = `--${name}=`;
    return args.find(a => a.startsWith(prefix))?.slice(prefix.length);
  };

  const targets =
    getArg('targets')?.split(',').filter(Boolean) ?? DEFAULT_TARGETS;
  const conc = getArg('concurrency')?.split(',').filter(Boolean) as
    | Concurrency[]
    | undefined;
  const concurrency = conc && conc.length > 0 ? conc : DEFAULT_CONCURRENCY;
  const runs = Number.parseInt(getArg('runs') ?? `${DEFAULT_RUNS}`, 10);
  const outDir =
    getArg('outDir') ?? path.join('tools', 'eslint-perf', 'results');

  const verbose = args.includes('--verbose') || getArg('verbose') === 'true';
  const patterns = getArg('patterns')
    ?.split(',')
    .map(p => p.trim())
    .filter(Boolean);

  return {
    targets,
    concurrency,
    runs,
    outDir,
    verbose,
    patterns,
  };
}

// -----------------------------
// Utilities
// -----------------------------
async function pathExists(p: string): Promise<boolean> {
  try {
    await stat(p);
    return true;
  } catch {
    return false;
  }
}

async function collectSystemInfo(): Promise<SystemInfo> {
  const cpuCores = os.cpus()?.length ?? 0;
  const totalMemGb = Math.round((os.totalmem() / 1024 ** 3) * 100) / 100;
  const nodeVersion = process.version;
  const npmVersion = (await exec('npm --version')).stdout.trim();
  const eslintVersion = (await exec('npx eslint --version')).stdout.trim();
  let nxVersion: string | null = null;
  try {
    nxVersion =
      (await exec('npx nx --version')).stdout.trim().split('\n')[0] ?? null;
  } catch {
    nxVersion = null;
  }

  return {
    timestamp: new Date().toISOString(),
    system: {
      os: os.platform(),
      osVersion: os.release(),
      architecture: os.arch(),
      cpuCores,
      totalMemGb,
      hostname: os.hostname(),
    },
    software: {
      nodeVersion,
      npmVersion,
      eslintVersion,
      nxVersion,
    },
  };
}

async function walkCountFiles(
  dir: string,
  extensions: string[],
): Promise<number> {
  let count = 0;
  async function walk(current: string): Promise<void> {
    const entries = await readdir(current, { withFileTypes: true });
    for (const entry of entries) {
      const p = path.join(current, entry.name);
      if (entry.isDirectory()) {
        if (
          entry.name === 'node_modules' ||
          entry.name === 'dist' ||
          entry.name === 'coverage'
        )
          continue;
        await walk(p);
      } else if (entry.isFile()) {
        if (extensions.some(ext => entry.name.endsWith(ext))) count += 1;
      }
    }
  }
  await walk(dir);
  return count;
}

async function analyzeTarget(target: string): Promise<TargetInfo> {
  const abs = path.resolve(target);
  const tsFiles = (await pathExists(abs))
    ? await walkCountFiles(abs, ['.ts', '.tsx'])
    : 0;
  const jsFiles = (await pathExists(abs))
    ? await walkCountFiles(abs, ['.js', '.jsx', '.mjs', '.cjs'])
    : 0;
  return {
    target,
    tsFiles,
    jsFiles,
    totalFiles: tsFiles + jsFiles,
  };
}

function buildEslintCommand(
  concurrency: Concurrency,
  options: { patterns: string[]; json: boolean; configPath?: string },
): string {
  const configArg = options.configPath
    ? ` --config=${JSON.stringify(options.configPath)}`
    : '';
  const patternArgs = options.patterns.map(p => JSON.stringify(p)).join(' ');
  const formatArg = options.json ? '--format=json' : '--format=stylish';
  return `npx eslint${configArg} --concurrency=${concurrency} ${formatArg} ${patternArgs}`;
}

async function runESLintOnce(
  target: string,
  concurrency: Concurrency,
  run: number,
  verbose = false,
  patterns: string[],
  configPath?: string,
): Promise<BenchmarkRun> {
  const start = process.hrtime.bigint();
  const cmd = buildEslintCommand(concurrency, {
    patterns,
    json: true,
    configPath,
  });
  let stdout = '';
  let stderr = '';
  try {
    const result = await exec(cmd, { maxBuffer: 1024 * 1024 * 200 });
    stdout = result.stdout;
    stderr = result.stderr;
  } catch (e: any) {
    stdout = e.stdout ?? '';
    stderr = e.stderr ?? '';
  }
  if (verbose) {
    console.log(`      $ ${cmd}`);
    if (stderr.trim()) {
      console.log('      [stderr]');
      console.log(
        stderr
          .split('\n')
          .map(l => `      ${l}`)
          .join('\n'),
      );
    }
  }
  const end = process.hrtime.bigint();
  const durationSeconds = Number(end - start) / 1_000_000_000;

  // Parse JSON output
  let filesProcessed = 0;
  let errors = 0;
  let warnings = 0;
  try {
    const json = JSON.parse(stdout) as Array<{
      errorCount: number;
      warningCount: number;
    }>;
    filesProcessed = json.length;
    for (const r of json) {
      errors += r.errorCount || 0;
      warnings += r.warningCount || 0;
    }
  } catch {
    // ignore parse errors
  }

  const eslintWarnings: string[] = [];
  if (stderr.includes('ESLintPoorConcurrencyWarning'))
    eslintWarnings.push('ESLintPoorConcurrencyWarning');

  return {
    target,
    concurrency,
    run,
    timing: {
      durationSeconds,
      realTimeSeconds: null,
      userTimeSeconds: null,
      sysTimeSeconds: null,
    },
    eslintResults: {
      filesProcessed,
      errors,
      warnings,
      eslintWarnings,
    },
    timestamp: new Date().toISOString(),
  };
}

async function printStylishOutputOnce(
  patterns: string[],
  configPath?: string,
): Promise<void> {
  const cmdStylish = buildEslintCommand('off', {
    patterns,
    json: false,
    configPath,
  });
  try {
    const { stdout, stderr } = await exec(cmdStylish, {
      maxBuffer: 1024 * 1024 * 200,
    });
    if (stdout.trim()) {
      console.log('    [stylish output]');
      console.log(
        stdout
          .split('\n')
          .slice(0, 300)
          .map(l => `    ${l}`)
          .join('\n'),
      );
    }
    if (stderr.trim()) {
      console.log('    [stylish stderr]');
      console.log(
        stderr
          .split('\n')
          .map(l => `    ${l}`)
          .join('\n'),
      );
    }
  } catch (e: any) {
    const out = e.stdout ?? '';
    const err = e.stderr ?? '';
    if (out.trim()) {
      console.log('    [stylish output]');
      console.log(
        out
          .split('\n')
          .slice(0, 300)
          .map((l: string) => `    ${l}`)
          .join('\n'),
      );
    }
    if (err.trim()) {
      console.log('    [stylish stderr]');
      console.log(
        err
          .split('\n')
          .map((l: string) => `    ${l}`)
          .join('\n'),
      );
    }
  }
}

function statsFrom(values: number[]): BenchmarkStats {
  const runs = values.length;
  const min = Math.min(...values);
  const max = Math.max(...values);
  const avg = values.reduce((a, b) => a + b, 0) / runs;
  const variance = values.reduce((acc, v) => acc + (v - avg) ** 2, 0) / runs;
  const stdDev = Math.sqrt(variance);
  return {
    runs,
    min,
    max,
    avg: Number(avg.toFixed(3)),
    stdDev: Number(stdDev.toFixed(3)),
  };
}

async function ensureDir(dir: string): Promise<void> {
  await mkdir(dir, { recursive: true });
}

async function main(): Promise<void> {
  const { targets, concurrency, runs, outDir, verbose, patterns } = parseArgs();
  await ensureDir(outDir);

  const systemInfo = await collectSystemInfo();
  const timestamp = new Date().toISOString().replace(/[:.]/g, '-');
  const resultsPath = path.join(outDir, `eslint-benchmark-${timestamp}.json`);
  const summaryPath = path.join(
    outDir,
    `eslint-benchmark-${timestamp}.summary.txt`,
  );

  const suiteResults: SuiteResult[] = [];

  // Write system info immediately
  await ensureDir(outDir);
  await writeFile(
    path.join(outDir, `system-info-${timestamp}.json`),
    JSON.stringify(systemInfo, null, 2),
  );

  for (const target of targets) {
    const absTarget = path.resolve(target);
    if (!(await pathExists(absTarget))) {
      console.warn(`Skipping missing target: ${target}`);
      continue;
    }
    const tInfo = await analyzeTarget(target);
    console.log(
      `\n📁 Target: ${target} (files: ${tInfo.totalFiles}, ts: ${tInfo.tsFiles}, js: ${tInfo.jsFiles})`,
    );
    if (verbose) {
      const resolvedPatterns =
        patterns && patterns.length > 0
          ? patterns
          : [path.join(target, '**', '*.ts')];
      const configPath = path.join(target, 'eslint.config.js');
      await printStylishOutputOnce(
        resolvedPatterns,
        (await pathExists(configPath)) ? configPath : undefined,
      );
    }

    for (const c of concurrency) {
      console.log(`  🔧 Concurrency: ${c}`);
      const runsData: BenchmarkRun[] = [];
      for (let i = 1; i <= runs; i++) {
        console.log(`    Run ${i}/${runs} ...`);
        const resolvedPatterns =
          patterns && patterns.length > 0
            ? patterns
            : [path.join(target, '**', '*.ts')];
        const configPath = path.join(target, 'eslint.config.js');
        const run = await runESLintOnce(
          target,
          c,
          i,
          verbose,
          resolvedPatterns,
          (await pathExists(configPath)) ? configPath : undefined,
        );
        runsData.push(run);
        console.log(
          `      Time: ${run.timing.durationSeconds.toFixed(3)}s, Files: ${run.eslintResults.filesProcessed}, Errors: ${run.eslintResults.errors}, Warnings: ${run.eslintResults.warnings}${run.eslintResults.eslintWarnings.length ? ' ⚠️ ' + run.eslintResults.eslintWarnings.join(',') : ''}`,
        );
      }
      const stats = statsFrom(runsData.map(r => r.timing.durationSeconds));
      suiteResults.push({
        target,
        targetInfo: tInfo,
        concurrency: c,
        statistics: stats,
        runs: runsData,
      });
      console.log(
        `    ✅ Avg: ${stats.avg}s (min ${stats.min.toFixed(3)}s, max ${stats.max.toFixed(3)}s, ±${stats.stdDev}s)`,
      );
    }
  }

  await writeFile(
    resultsPath,
    JSON.stringify({ systemInfo, results: suiteResults }, null, 2),
  );

  // Write human-friendly summary
  const lines: string[] = [];
  lines.push('ESLint Concurrency Benchmark Summary');
  lines.push(`Generated: ${new Date().toString()}`);
  lines.push('');
  lines.push('System:');
  lines.push(`- OS: ${systemInfo.system.os} ${systemInfo.system.osVersion}`);
  lines.push(`- CPU Cores: ${systemInfo.system.cpuCores}`);
  lines.push(`- Memory: ${systemInfo.system.totalMemGb} GB`);
  lines.push(`- Node: ${systemInfo.software.nodeVersion}`);
  lines.push(`- ESLint: ${systemInfo.software.eslintVersion}`);
  if (systemInfo.software.nxVersion)
    lines.push(`- Nx: ${systemInfo.software.nxVersion}`);
  lines.push('');
  for (const group of suiteResults) {
    lines.push(
      `Target: ${group.target} (files: ${group.targetInfo.totalFiles}, ts: ${group.targetInfo.tsFiles}, js: ${group.targetInfo.jsFiles})`,
    );
    lines.push(
      `  Concurrency=${group.concurrency} → Avg=${group.statistics.avg}s (min=${group.statistics.min.toFixed(3)}s, max=${group.statistics.max.toFixed(3)}s, stdDev=${group.statistics.stdDev}s)`,
    );
  }
  await writeFile(summaryPath, lines.join('\n'));

  console.log('\n✅ Benchmark complete');
  console.log(`Raw results: ${resultsPath}`);
  console.log(`Summary: ${summaryPath}`);

  // Print fastest table to console
  const targetsSet = Array.from(new Set(suiteResults.map(r => r.target)));
  const tableRows: Array<{
    target: string;
    bestConc: Concurrency;
    bestAvg: number;
    baseline: number | null;
    speedup: string;
  }> = [];
  for (const t of targetsSet) {
    const byTarget = suiteResults.filter(r => r.target === t);
    if (byTarget.length === 0) continue;
    const best = byTarget.reduce((a, b) =>
      a.statistics.avg <= b.statistics.avg ? a : b,
    );
    const baseline =
      byTarget.find(r => r.concurrency === 'off')?.statistics.avg ?? null;
    const speedupVal = baseline ? baseline / best.statistics.avg : null;
    tableRows.push({
      target: t,
      bestConc: best.concurrency,
      bestAvg: Number(best.statistics.avg.toFixed(3)),
      baseline,
      speedup: speedupVal ? `${speedupVal.toFixed(2)}x` : 'n/a',
    });
  }
  const pad = (s: string, n: number) =>
    s.length >= n ? s : s + ' '.repeat(n - s.length);
  const header = `${pad('Target', 28)}  ${pad('Best', 6)}  ${pad('Avg(s)', 8)}  ${pad('Baseline(s)', 12)}  Speedup`;
  console.log('\nFastest per target:');
  console.log(header);
  console.log('-'.repeat(header.length));
  for (const row of tableRows) {
    console.log(
      `${pad(row.target, 28)}  ${pad(String(row.bestConc), 6)}  ${pad(row.bestAvg.toFixed(3), 8)}  ${pad(row.baseline ? row.baseline.toFixed(3) : 'n/a', 12)}  ${row.speedup}`,
    );
  }

  // Per-target detailed table marking the best
  console.log('\nPer-target breakdown (best marked with ★):');
  for (const t of targetsSet) {
    const byTarget = suiteResults.filter(r => r.target === t);
    if (byTarget.length === 0) continue;
    const baseline =
      byTarget.find(r => r.concurrency === 'off')?.statistics.avg ?? null;
    const bestAvg = Math.min(...byTarget.map(r => r.statistics.avg));
    console.log(`\n${t}`);
    const h2 = `${pad('Concurrency', 12)}  ${pad('Avg(s)', 8)}  ${pad('StdDev', 8)}  ${pad('Speedup', 8)}  Mark`;
    console.log(h2);
    console.log('-'.repeat(h2.length));
    for (const c of new Set(byTarget.map(r => r.concurrency))) {
      const r = byTarget.find(x => x.concurrency === c);
      if (!r) continue;
      const isBest = Math.abs(r.statistics.avg - bestAvg) < 1e-6;
      const speed = baseline
        ? `${(baseline / r.statistics.avg).toFixed(2)}x`
        : 'n/a';
      console.log(
        `${pad(String(c), 12)}  ${pad(r.statistics.avg.toFixed(3), 8)}  ${pad(r.statistics.stdDev.toFixed(3), 8)}  ${pad(speed, 8)}  ${isBest ? '★' : ''}`,
      );
    }
  }
}

main().catch(err => {
  console.error(err);
  process.exitCode = 1;
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment