DDoS attack on your VPS: the first-hour runbook

First, make sure it is actually an attack

The most expensive mistake in an outage is treating the wrong thing. "The site is slow" is a symptom shared by a genuine flood, a deploy that shipped an unindexed query, a cron job that started dumping the database at the same minute every hour, a crawler that discovered your faceted search, a link that reached the front page of somewhere large, and a disk that filled up. Every one of those looks identical from a browser, and the responses are mutually exclusive: you do not want to be rate-limiting real customers because a migration forgot an index.

Three questions separate them in under a minute. Is the traffic volume genuinely abnormal, or is the traffic normal and the server slow? A flood shows up as a step change in packets or requests per second; a bad deploy shows up as normal request volume and a collapsed response time. Did anything change on your side in the last hour? Check the deploy log before the firewall — self-inflicted incidents outnumber attacks by a wide margin on small infrastructure. Is the load spread across the site, or concentrated on one path? Real floods are usually indiscriminate or aimed at the front page; an expensive endpoint being hammered by a hundred clients is closer to abuse than to a DDoS, and it has a much cheaper fix.

Answer those, then keep going. The rest of this guide assumes the answer was: volume is abnormal, nothing changed on your side, and the box is drowning.

The five-minute triage

There are only two failure modes that matter, and they need opposite responses. Either the pipe is full — packets arrive faster than your uplink or your kernel can process them, and your server is losing traffic before any of your software sees it — or the pipe is fine and the application is exhausted, because well-formed requests are arriving faster than it can answer them. Mixing them up wastes the hour: nginx tuning does nothing against a saturated link, and buying more bandwidth does nothing against a request flood.

Tell them apart by comparing two numbers. Look at the interface counters and the CPU split at the same time. If rx bytes are pinned near your port speed, if dropped or overrun counters are climbing, and if the time is going to soft interrupts rather than to your application, the flood is at layer 3 or 4 and it is a capacity problem. If bandwidth is unremarkable but the worker pool is saturated, connections are queueing, and the access log is full of plausible-looking requests, it is layer 7 and it is a filtering problem.

Then classify the layer-3/4 case one step further, because the sub-types behave differently. A SYN flood shows as tens of thousands of half-open sockets in SYN-RECV; the kernel handles this well once syncookies are on. A UDP or amplification flood shows as huge inbound volume on ports you do not even listen on — DNS, NTP, memcached, CLDAP reflections — and nothing you run can help, because the damage is done by the time the packets reach your NIC. A fragmentation or raw packet flood shows as high packets-per-second with modest bandwidth, which starves the CPU rather than the link. Write down which one you have before you touch a config file.

What you cannot fix from inside the box

This is the part most articles skip, and it is the part that decides whether your hour is productive. A firewall rule on the target does not save a saturated uplink. Your iptables DROP runs on the machine at the end of the pipe — the packet has already crossed the transit link, already consumed the bandwidth you are paying for, and already displaced a real user's packet. Dropping it locally protects your application from wasting cycles, which is worth something, and protects your bandwidth from nothing at all.

The honest ceiling for a single server is roughly the smaller of two numbers: the port speed it is connected at, and the packets per second its CPU can classify. A 1 Gbps port is full at 1 Gbps regardless of how elegant your ruleset is, and a modest flood of small packets can exhaust a couple of cores on interrupt handling long before the bandwidth number looks alarming. Past that point, the only thing that helps is a device further upstream that has more capacity than the attack and drops the traffic before it ever reaches your link. That is what scrubbing is, and it is why every machine in our fleet sits behind 1.5 Tbps of always-on mitigation rather than a bigger firewall.

The corollary matters just as much: if you have no upstream protection, your provider's response to a large volumetric attack is to null-route your address, because the alternative is degrading every other customer on that link. That is not malice, it is arithmetic — and it means the attacker wins by making you expensive rather than by breaking anything. Knowing in advance whether mitigation is included on your plan or is a paid extra you never enabled is a five-minute check that is worth doing today rather than during the incident. Ours is included on every plan, which is the only arrangement that helps at 3 a.m.

Layer 7: the flood that looks like traffic

An application-layer flood is harder because every individual request is legitimate. A well-built HTTP flood completes the TCP handshake, negotiates TLS, sends a valid GET / with a plausible user agent, and reads the response. No packet in it is malformed. What kills you is arithmetic: a request costs the attacker almost nothing to send and costs you a database query, a template render and a hundred milliseconds of a worker that is now not serving anybody else.

The tells are in your own access log, and they are usually obvious once you look for them rather than at them. Cache-busting query strings — thousands of requests to /?1234567, each one a unique URL that defeats every cache you own — are the single most common signature. A user-agent distribution with no long tail: real traffic is a messy mixture of hundreds of browser builds, and a flood is often three strings repeated a million times, or one that nobody real uses. A referrer field that is identical everywhere. Requests that skip your static assets entirely — a real browser fetches the CSS, the fonts and the images after the HTML; a flood client asks for the HTML and leaves. And a source distribution that is too flat: a botnet spread across ten thousand residential addresses each sending two requests a second looks like popularity until you notice the per-address rate is suspiciously uniform.

Then there is the variant that needs almost no traffic at all: the slow attack. A few hundred connections that open, send one header every twenty seconds and never finish will occupy every worker you have while your bandwidth graph stays flat. The fix is not a rate limit — the request rate is tiny — it is aggressive header and body timeouts, which is why they appear in the first configuration block below rather than as an afterthought.

Four controls, in order of effect per minute

Under pressure, do the highest-leverage thing first. The order below is not arbitrary; it is roughly descending in how much load each one sheds per minute of your attention.

• Serve something cheap. A micro-cache of thirty seconds in front of your application turns a thousand identical requests per second into one origin hit and 999 memory reads. It is the single biggest lever on almost every HTTP flood, it costs one directive block, and for anonymous traffic it is nearly always safe. Add proxy_cache_lock so a cache miss does not send a thundering herd to the backend.
• Cap concurrency and shorten timeouts. limit_conn per address plus tight header, body and keepalive timeouts kills slow attacks outright and stops any single client from parking your worker pool. This is the control that costs real users the least.
• Rate-limit, with a burst. limit_req with a sensible burst is precise but slower to tune, and it is the one that generates false positives if you set it from panic instead of from your own baseline. You need to know your normal requests-per-second per client before you can pick a number — which is why the post-mortem at the end of this guide matters more than it sounds.
• Block, narrowly and reluctantly. Dropping specific networks works when the sources are concentrated and does nothing when they are not. It also ages badly: every block you add during an incident is a customer you may be silently refusing in three months. Use a set with a timeout so the rules expire on their own.

Notice what is not on the list: banning single IP addresses by hand, restarting the web server repeatedly, and disabling the firewall to "see if it helps". The first is too slow to matter against a distributed source, the second throws away every warm connection you had, and the third is how an incident becomes a compromise.

Do not fight it from the machine holding your data

Every control above is worth more when it runs somewhere other than the box holding your database. If your edge is a separate node, the flood terminates on a machine whose entire job is terminating floods: it caches, rate-limits and drops with the real client address in hand, and the origin only ever sees the small, filtered remainder over a private tunnel. When the edge falls over you replace it in 15 min and lose nothing, because there is nothing on it. When the origin falls over you have an outage and a restore.

That separation also closes the bypass that makes most mitigation decorative. If the origin still has a public listener, an attacker who finds its address — through passive DNS, a certificate transparency entry, an MX record or a link preview — can aim past every control you configured and hit the application directly. This is common enough that it is worth treating as the default state of any "protected" site until proven otherwise. Building the version that holds is a guide of its own: an offshore reverse proxy with no public listener on the origin at all.

One caution about the middle of an incident: this is architecture, not first aid. Standing up an edge, moving DNS and rebuilding a tunnel while under attack is a two-hour job done badly under pressure, and the DNS change alone will not take effect for as long as your TTL says. If you have it, use it. If you do not, get through the hour with the controls you have, and build it in the calm week afterwards — which is exactly when nobody does.

The day you are the reflector, not the target

There is a second version of this incident where your server is not the victim and nobody tells you. An open resolver, an exposed NTP daemon, an unauthenticated memcached on a public interface, an SSDP or CLDAP responder inside a container — each of these answers a small spoofed request with a much larger reply, aimed at somebody else. From your side the symptoms are inverted: outbound bandwidth is high, inbound is modest, your application is fine, and the first real signal is an abuse notice or a suspended port.

The check takes a minute and belongs in the same runbook, because it is the same command you already ran during triage. ss -tulpn should list nothing bound to a public address that you did not deliberately put there, and UDP services deserve particular suspicion because they are the ones that amplify. A recursive resolver must be bound to localhost or to a tunnel address; memcached and Redis must never be reachable from the internet at all; and any container publishing a port with -p 0.0.0.0: has just punched a hole through the firewall you configured, because Docker writes its own rules ahead of yours. That last one surprises people every single time.

The same shape covers outbound floods from a machine that has already been compromised, which is the other reason a provider suddenly null-routes an address. If your outbound graph is high and your application is idle, stop reading configuration and start checking processes — that is an intrusion, not a capacity problem, and the response is to rebuild the box from a known-good backup rather than to filter it.

After it stops: the post-mortem and the standing kit

Attacks stop. Usually the attacker gets bored, sometimes the mitigation makes it pointless, occasionally it was a fixed-length booter subscription that simply expired. The temptation at that moment is to leave everything exactly as it is and go to bed, which is how a temporary rate limit becomes a permanent, forgotten, silent 429 for a whole country nine months later. Spend twenty minutes closing the loop while it is fresh.

Three artefacts are worth producing. A baseline: your normal requests per second, your normal per-client rate, your normal bandwidth at peak. Without those numbers every limit you set during the next incident is a guess, and half of them will be wrong in the direction that hurts customers. A revert list: everything you changed, with a date and a reason, so the emergency configuration does not quietly become the permanent one. A runbook that is four commands long and lives somewhere you can reach when the site is down — not on the server, and not only in your own head.

Then close the two structural gaps, because this is where the layer sits in the stack. Mitigation is what absorbs the volume, an edge is what keeps the flood away from your data, and the box behind them still has to be hardened properly and to have restorable off-site backups, because the incident after this one may not be a flood at all. A server that survives a DDoS and loses its disk a month later was never resilient — it was lucky, twice.

Step by step

1. 01

   Confirm it before you change anything

   Get one honest picture of the machine before you touch a config file. You are looking for a step change in packets or requests, and for where the time is going — an application starved of CPU by soft interrupts is a very different incident from an application waiting on a database.

   # is the box alive, and where is the time going?
   uptime                     # load average against your core count
   vmstat 1 5                 # 'in' and 'cs' high, 'id' near zero = packet work
   mpstat -P ALL 1 3          # %soft pinned on one core = interrupt saturation
   
   # is the pipe full, or just busy?
   ip -s link show eth0       # rx bytes, and the errors/dropped counters
   ethtool eth0 | grep -i speed
   
   # requests per second from your own log, minute by minute
   tail -n 20000 /var/log/nginx/access.log \
     | awk -F'[][]' '{print $2}' | cut -d: -f2-4 | uniq -c | tail -5

   Before concluding it is an attack, check your own deploy log and your cron table. Self-inflicted outages are more common than floods on a single VPS, and they look the same from outside.

2. 02

   Measure the shape of the traffic in sixty seconds

   Now classify it. Three questions: how many distinct sources, what protocol and state, and — if it is HTTP — which paths and which agents. The answers decide which control you reach for, and they take about a minute to collect.

   # top source addresses on the wire right now
   timeout 20 tcpdump -nn -i eth0 -c 20000 2>/dev/null \
     | awk '{print $3}' | rev | cut -d. -f2- | rev \
     | sort | uniq -c | sort -rn | head -20
   
   # TCP state census: a wall of SYN-RECV is a SYN flood
   ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn
   
   # layer 7: talkers, paths, agents over the last 50k requests
   L=/var/log/nginx/access.log
   tail -n 50000 $L | awk '{print $1}' | sort | uniq -c | sort -rn | head -20
   tail -n 50000 $L | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
   tail -n 50000 $L | cut -d'"' -f6   | sort | uniq -c | sort -rn | head -10

   Read the result against the tells: unique query strings on one path, a user-agent list with no long tail, no requests for your static assets, or a per-address rate that is oddly uniform. If bandwidth is high on ports you do not listen on, stop here — that is a volumetric flood and step six is the only step that matters.

3. 03

   Serve something cheap, and cap the connections

   Highest leverage first. A thirty-second micro-cache collapses a flood of identical anonymous requests into one origin hit, and tight timeouts kill slow attacks that a rate limit cannot see. Put both in the http block, then reload rather than restart so you keep your warm connections.

   # /etc/nginx/nginx.conf — http block
   limit_conn_zone $binary_remote_addr zone=perip:10m;
   limit_req_zone  $binary_remote_addr zone=flood:20m rate=10r/s;
   
   client_header_timeout 10s;
   client_body_timeout   10s;
   send_timeout          10s;
   keepalive_timeout     20s;
   reset_timedout_connection on;
   
   proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=hot:64m
                    max_size=2g inactive=10m use_temp_path=off;

   # the server block — cap concurrency, serve the cached copy
   limit_conn perip 20;
   
   location / {
       proxy_cache hot;
       proxy_cache_valid 200 301 302 30s;
       proxy_cache_lock on;
       proxy_cache_use_stale error timeout updating
                             http_500 http_502 http_503 http_504;
       add_header X-Cache $upstream_cache_status;
       proxy_pass http://127.0.0.1:8080;
   }

   nginx -t && systemctl reload nginx
   curl -sI https://example.com/ | grep -i x-cache   # want: HIT on the second call

4. 04

   Rate-limit the real client, not your own proxy

   If anything sits in front of nginx, every request arrives from one address and a per-address limit will throttle the proxy instead of the attacker — or ban it outright and take the site down while you are defending it. Trust the forwarded header from the proxy address only, never from the internet, then apply the limit.

   # /etc/nginx/conf.d/realip.conf — the tunnel or edge address only
   set_real_ip_from 10.66.0.1;
   real_ip_header   X-Forwarded-For;
   real_ip_recursive off;

   # burst absorbs bursty humans; nodelay keeps the page fast for them
   location / {
       limit_req zone=flood burst=20 nodelay;
       limit_req_status 429;
   }
   
   # the expensive paths get a much tighter bucket of their own
   location ~ ^/(search|login|register|api/) {
       limit_req zone=flood burst=5;
       limit_req_status 429;
   }

   Then watch what you just did: tail -f /var/log/nginx/error.log | grep limiting. If the addresses being limited look like your customers, the rate is too low — raise it. A limit that blocks real users is an outage you caused yourself.

5. 05

   Block narrowly, and give every block an expiry

   Only worth doing when step two showed concentrated sources. Use a set, not a thousand rules — an ipset lookup is constant time, a long iptables chain is walked for every packet and becomes its own denial of service. Give every entry a timeout so today's emergency does not become next year's silent block list.

   ipset create flood hash:net timeout 3600 -exist
   iptables -I INPUT -m set --match-set flood src -j DROP
   
   # feed it from the census: /24s you actually verified, not guesses
   for n in 203.0.113.0/24 198.51.100.0/24; do ipset add flood $n -exist; done
   
   # SYN flood: let the kernel do the part it is good at
   sysctl -w net.ipv4.tcp_syncookies=1
   sysctl -w net.ipv4.tcp_max_syn_backlog=8192
   sysctl -w net.core.somaxconn=8192
   
   ipset list flood | head -20      # keep a copy of this for the post-mortem

   Resist geo-blocking a whole country unless you can name the customers you are cutting off. And never ufw disable to test a theory: an unfirewalled box under active attack is how a bandwidth incident turns into a breach.

6. 06

   Escalate to the layer that can actually absorb it

   If the interface counters say the link is saturated, or drops are climbing while your CPU is idle, you have reached the ceiling of anything you can do on the server. Confirm that reading, then escalate rather than keep tuning.

   # drops in the stack itself — second column is 'dropped'
   awk '{print strtonum("0x" $2)}' /proc/net/softnet_stat | paste -sd+ | bc
   
   # interface-level loss against link speed
   ip -s link show eth0 | sed -n '3,6p'

   With upstream scrubbing in place there is usually nothing to do: detection is always-on and the flood is dropped in the network before it reaches your port — on our fleet that is 1.5 Tbps of capacity sitting in front of every plan, so the incident is often visible only as a graph after the fact. If the attack is a well-formed HTTP flood rather than a volumetric one, that is the case for an L7 shield, because request floods are indistinguishable from users at the packet level and have to be judged higher up. If you have no mitigation at all, your realistic options are to move behind an edge that does, or to wait — and to plan the first one before the next attack.

7. 07

   Close the loop: verify, revert, then write it down

   Verify from outside the machine, not from a shell on it. Then undo the emergency measures deliberately, keep the ones that were always a good idea, and record the numbers so the next incident starts from a baseline instead of a guess.

   # from somewhere else entirely: is the site healthy for a normal user?
   curl -s -o /dev/null -w 'code=%{http_code} ttfb=%{time_starttransfer}s\n' \
        https://example.com/
   
   # did you leave a limit that is biting real people?
   grep -c 'limiting requests' /var/log/nginx/error.log
   
   # what is still blocked, and when does it expire?
   ipset list flood | head -30

   Keep the cache, the timeouts and the syncookies — those are permanent improvements. Roll back the aggressive rate limits to your measured baseline plus headroom, and let the ipset entries age out. Then write the four commands from steps one and two into a runbook stored somewhere that is not this server, next to your normal requests-per-second and your normal peak bandwidth. The next attack is a shorter incident purely because those two numbers exist.