Builder Log

From Nginx + Forever to Caddy + PM2: fixing two years of self-inflicted pain

Every 60 days my Node.js server died quietly in the middle of the night. Here's the story of why, and how I finally fixed it properly.

I’ll be honest with you — this was entirely my fault.

For the better part of two years I ran a production server that crashed every 60 days like clockwork. A Node.js API serving a Flutter Web portal, sitting behind Nginx, kept alive by forever. Every two months the process would die silently in the middle of the night. No alerts. No automatic restart. Just a blank forever list and a support message from someone who couldn’t log in.

I knew something was wrong. I just kept patching instead of fixing. This is the story of finally doing it right.

What the stack looked like

Nothing exotic. Just the kind of setup you end up with after years of incremental decisions that each made sense at the time:

  • Nginx serving the Flutter Web app and handling HTTPS traffic
  • Certbot managing Let’s Encrypt certificates
  • forever keeping the Node.js process alive
  • Node.js — and this is the embarrassing part — managing its own SSL certificates

That last one is worth dwelling on. I had this in my authServer.js:

if (environment == "PROD") {
  const options = {
    key: fs.readFileSync('/etc/letsencrypt/live/portal.example.com/privkey.pem'),
    cert: fs.readFileSync('/etc/letsencrypt/live/portal.example.com/fullchain.pem')
  };
  https.createServer(options, app).listen(port, () => {
    console.log(`HTTPS Server running on port ${port}...`);
  });
}

The Node.js process was reading certificate files at startup. Which meant that every time Certbot renewed the certificates, Node.js needed to be restarted to pick them up. I had a hook script for that. The hook didn’t always work. You can probably see where this is going.

The 60-day curse

Let’s Encrypt certificates last 90 days. Certbot renews them at 60 days. So every 60 days, something would go wrong.

When I finally sat down to properly diagnose it, I found this in the cron logs:

Apr 13 00:00:01 CRON[2895952]: certbot -q renew
Apr 13 00:00:01 CRON[2895953]: /opt/certbot/bin/python ... certbot renew -q

Two Certbot instances. Running simultaneously. Every 12 hours.

Somehow, over the years, I had installed Certbot twice — once through apt (/usr/bin/certbot) and once through pip (/opt/certbot/bin/certbot). Both had active cron jobs. Both were trying to renew the same certificate. And both triggered the hook that was supposed to restart forever.

The hook itself was a mess of PATH gymnastics trying to find the right Node.js binary through nvm:

#!/bin/bash
NODE="/root/.nvm/versions/node/v21.7.3/bin/node"
FOREVER="/root/.nvm/versions/node/v21.7.3/bin/forever"
export PATH="$(dirname "$NODE"):$PATH"
# ... stop forever, start forever

When cron runs a script, it uses a minimal environment. nvm wasn’t initialised. The binary paths were sometimes wrong. The restart would fail silently, and the Node.js process — which had been stopped — would just not come back up.

Two months of uptime, then nothing. Every single time.

The fix: stop patching, start replacing

I could have kept patching. Fixed the hook, removed one Certbot, sorted out the PATH. But at some point the right answer is to step back and ask whether the tools themselves are still the right choice.

The answer here was no. So I replaced the whole thing.

Step 1 — forever → PM2

forever has been around since the early days of Node.js. It works, mostly, but it wasn’t built with modern Linux process management in mind. PM2 was.

npm install -g pm2

forever stop my-api
forever stopall

cd /root/projects/my-api
pm2 start authServer.js --name "my-api"
pm2 save
pm2 startup
# copy and run the command PM2 prints

That last command is what makes the difference. pm2 startup registers PM2 as a systemd service. If the server reboots, PM2 comes back. If the Node.js process crashes, PM2 restarts it. No hooks. No cron. No PATH issues. It just works.

┌────┬──────────┬─────────┬──────────┬────────┬──────────┐
│ id │ name     │ mode    │ status   │ uptime │ mem      │
├────┼──────────┼─────────┼──────────┼────────┼──────────┤
│ 0  │ my-api   │ fork    │ online   │ 2d 4h  │ 84.6mb   │
└────┴──────────┴─────────┴──────────┴────────┴──────────┘

Step 2 — Nginx + Certbot → Caddy

This is where things got genuinely pleasant.

Caddy is a modern web server written in Go. Its killer feature is that it manages TLS certificates automatically — no Certbot, no renewal hooks, no cron jobs, no anything. You point it at a domain, it talks to Let’s Encrypt, gets the certificate, and renews it quietly in the background forever.

My entire server configuration — Flutter Web app plus Node.js API — ended up being this:

portal.example.com {

    handle /api/* {
        uri strip_prefix /api
        reverse_proxy localhost:4000
    }

    handle {
        root * /var/www/html
        encode gzip
        try_files {path} /index.html
        file_server
    }

}

Ten lines. HTTPS, reverse proxy, gzip compression, Flutter SPA routing — all of it.

One thing I learned the hard way: the order of directives matters in Caddy. If you put root, encode, and file_server at the top level instead of inside an explicit handle {} block, Caddy processes them before the /api/* route and every API call returns your index.html. Wrap everything in handle blocks.

After starting Caddy, within seconds:

{"level":"info","msg":"certificate obtained successfully","identifier":"portal.example.com"}

No manual steps. No Certbot. No renewal configuration. Done.

Step 3 — strip SSL out of Node.js

With Caddy handling TLS, the Node.js app has no business knowing SSL exists. The traffic flow is:

Browser → HTTPS → Caddy → HTTP (internal) → Node.js

The HTTPS layer lives entirely at the Caddy boundary. Node.js just handles plain HTTP on localhost. So I deleted all of this:

// all of this can go
const https = require('https');
const options = {
  key: fs.readFileSync('/etc/letsencrypt/...'),
  cert: fs.readFileSync('/etc/letsencrypt/...')
};
if (environment == "PROD") {
  https.createServer(options, app).listen(port, ...);
} else {
  app.listen(port, ...);
}

And replaced it with this:

app.listen(port, '127.0.0.1', () => {
  console.log(`Server running on port ${port}`);
});

The '127.0.0.1' binding is important. It restricts the process to only accept connections from localhost — even if the firewall is misconfigured, nothing from the outside world can reach port 4000 directly.

You can verify it:

sudo ss -tlnp | grep 4000
# LISTEN  127.0.0.1:4000  ← correct, local only
# LISTEN  *:4000           ← exposed to the world, fix this

A CORS bug I’d been ignoring

While cleaning things up I found this lurking in the codebase:

// this throws a TypeError — you can't reassign a const
const originUrl = 'http://192.168.1.114:8080';
if (environment == "PROD") {
  originUrl = 'https://portal.example.com';
}

It had been working by accident in some environments and failing silently in others. The correct version:

const corsOrigins = {
  PROD: 'https://portal.example.com',
  DEV:  'http://192.168.1.114:8080'
};

app.use(cors({
  origin: corsOrigins[environment] || corsOrigins.DEV,
  credentials: true,
}));

Small thing, but worth catching.

Don’t forget the client

The Flutter app had this:

const String ENDPOINT_URL = "https://portal.example.com";
const String ENDPOINT_PORT = "4000";

Which meant every API request was going to https://portal.example.com:4000/login — directly to the port that was now closed to the outside world. The fix is straightforward: drop the port, add the /api prefix that Caddy uses for routing:

const String ENDPOINT_URL = "https://portal.example.com/api";
const String ENDPOINT_PORT = ""; // no longer needed

After rebuilding and deploying, everything came back to life.

What I should have done from the start

Looking back, none of this was complicated. The right architecture was always:

  1. One process manager that integrates with the OS init system (PM2 + systemd)
  2. One place for TLS — the proxy, not the application
  3. Bind to localhost for anything that shouldn’t be publicly reachable

The mistakes weren’t careless. They were the kind that accumulate when you’re moving fast and each individual decision seems reasonable in isolation. The two Certbot installations probably happened because something wasn’t working and I tried a different installation method without cleaning up the first. The SSL code in Node.js was probably adapted from a tutorial that didn’t account for running behind a proxy. The forever choice was perfectly fine in 2020.

But reasonable choices in 2020 aren’t necessarily the right choices today. At some point it’s worth sitting down and replacing the patches with something cleaner.

The final picture

BeforeAfter
Web server / proxyNginxCaddy
TLS managementCertbot ×2 + Node.jsCaddy (automatic)
Process managerforeverPM2
API port binding*:4000 (public)127.0.0.1:4000 (local only)
Config complexity~60 lines Nginx + hook scripts10 lines Caddyfile
Downtime risk on cert renewalHighNone

The server has been running since. Certificates renew themselves. PM2 restarts the API if it ever crashes. No cron jobs. No hook scripts. No middle-of-the-night surprises.

If you’re running a similar setup and you’ve been patching the same problems for longer than feels comfortable — it might be time to just replace it.