Most of your traffic isn't human: three multipliers and how to fix them
Rendering on every view, prefetching and crawlers can make a server work 10 to 30 times harder than it should. Here are the fixes.
By Hassan Ed-dyb · · 1 min read
Short answer: three silent multipliers make a server work 10 to 30 times harder than it should. Pages that render on every view, links that prefetch everything in sight, and crawlers that walk every page. Cache public pages at the edge, prefetch on hover, and block SEO-tool bots while keeping search and AI crawlers welcome.
Multiplier 1: pages that run on every view
Read a cookie while rendering a public page, for example to show the user's name in the navbar, and the whole page becomes personal. Nothing can be cached, so the server renders it for every single visit.
Multiplier 2: prefetching
Next.js loads every link in view in advance, to make clicks feel instant. On a page with a long list, one visit quietly becomes 10 to 30 server runs.
Multiplier 3: bots
SEO tools crawl every page and every ?page=N of every list, again and again. They bring you nothing, and they cost the same as real visitors.
The fixes
- —Cache public pages at the edge. Serve the same HTML to everyone and load the login state in the browser. One render then serves every visitor for 15 minutes.
- —Prefetch on hover only. It still feels instant, and you stop paying for links nobody clicks.
- —Block SEO-tool bots in robots.txt. Keep Google, Bing and AI crawlers welcome: they are how people find you.
One check before you launch
Look at the Cache-Control header on every public page. "private, no-store" means your server runs on every single view.
Users rarely break a launch. Bots and defaults do.