Your Node.js script works perfectly with test data. Then you feed it a real 10GB log file. Suddenly: crash. No warnings, just ENOMEM. Here's why even seasoned developers make this mistake—and the bulletproof solution.

The Root of All Evil: fs.readFile

fs.readFile is the equivalent of dumping a dump truck’s contents into your living room. It loads every single byte into RAM before you can touch it. Observe:

// Processing a 3GB database dump? Enjoy 3GB RAM usage
fs.readFile('./mega-database.sql', 'utf8', (err, data) => {
  parseSQL(data); // Hope you have 3GB to spare
});
  • CLI tools crash processing large CSVs
  • Data pipelines implode on video files
  • Background services die silently at 3AM

This isn’t “bad code”—it’s how fs.readFile operates. And it’s why your production system fails catastrophically.

Streams: The Memory Ninja Technique

Streams process data like a conveyor belt—small chunks enter, get processed, then leave memory forever. No RAM explosions:

// Process 100GB file with ~50MB memory
const stream = fs.createReadStream('./giant-dataset.csv');

stream.on('data', (chunk) => {
  analyzeChunk(chunk); // Work with 64KB-1MB pieces
});

stream.on('end', () => {
  console.log('Processed entire file without going nuclear');
});

Real-World Massacre: File Processing

The Suicide Approach (Common Mistake)

// Data import script that crashes on big files
function importUsers() {
  fs.readFile('./users.json', (err, data) => {
    JSON.parse(data).forEach(insertIntoDatabase); // 💀
  });
}

The Stream Survival Guide (Correct Way)

// Processes 50GB JSON file without memory issues
const ndjsonStream = fs.createReadStream('./users.ndjson');
const jsonParser = new TransformStream({ /* parse line-by-line */ });

ndjsonStream.pipe(jsonParser)
  .on('data', (user) => insertIntoDatabase(user));

Streams handle terabyte-scale files like they’re nothing. Your app stays responsive—no OOM crashes.

Pro Tip: Streams + Pipelines = Unstoppable

Combine streams with Node.js’ pipeline for error-proof processing:

const { pipeline } = require('stream');
const zlib = require('zlib');

// Compress 20GB log file with constant memory
pipeline(
  fs.createReadStream('./server.log'),
  zlib.createGzip(), // Compress chunk-by-chunk
  fs.createWriteStream('./server.log.gz'),
  (err) => {
    if (err) console.error('Pipeline failed:', err);
    else console.log('Compressed 20GB file like a boss');
  }
);

When to Use Which Weapon

fs.readFile (Handle With Care):

  • Configuration files (<5MB)
  • Small static assets (icons, tiny JSON)
  • Only when you absolutely need all data in memory

fs.createReadStream (Default Choice):

  • Log processing
  • Media encoding/transcoding
  • Database imports/exports
  • Any file bigger than your phone’s RAM

The Harsh Truth

fs.readFile is Node.js’ version of a loaded gun—safe in controlled environments, deadly in production. Streams aren’t “advanced” techniques; they’re essential survival skills for any serious Node.js developer.

Next time you write fs.readFile, ask: “Will this file ever grow?” If the answer is yes (and it always is), you’ve just found your memory leak. Switch to streams—before your pager goes off at midnight.