Server logs on a VPS: the data you decided not to keep

The request you answered is now a record you hold

Everything else in this series points outward. Hide the origin, encrypt the disk, pick the jurisdiction, stop somebody else from keeping a log of you. This one points at the machine itself, because a server that answers requests writes down who made them — by default, in several places at once, with a retention nobody chose.

Look at what a stock image is holding a fortnight after deploy. /var/log/nginx/access.log has one line per request: address, timestamp, path, referrer, user agent, rotated daily and kept for fourteen days. /var/log/auth.log has every SSH session, including the source address of every one of yours. The journal has the same events again, plus whatever your services printed to stderr. If Docker is running, each container has a JSON log file which, out of the box, has no size limit at all. If fail2ban is running, it has a log of every address it ever banned and a SQLite database saying the same thing, and the addresses in both are complete.

None of that is malicious and most of it is genuinely useful — for about an hour after something goes wrong. The problem is the shape: full fidelity, kept a long time, by accident rather than by decision. The question worth asking about each of those files is not "is this sensitive?" but "what would I actually do with the line I wrote three weeks ago?" For the overwhelming majority of lines on the overwhelming majority of servers the answer is nothing, and a record you will never read is pure liability — to a compromise, to a backup that outlives the machine, to whoever eventually asks for it.

Data minimisation is the formal name for the fix and it is the least controversial idea in data protection: collect what the job needs, keep it while the job needs it, then stop. What follows applies that to a box you actually run, on the assumption you have already done the first-hour hardening pass and there is something worth protecting on the machine.

Six logs, and the two that name people

Before changing anything, know the inventory. A small Debian or Ubuntu VPS running a web service typically writes six streams, and they overlap more than people expect — the same event often lands in three files with three different retentions.

The nginx access log is the one that names your visitors. One line per request, with the client address, the exact path, the referrer and a user-agent string detailed enough to be a weak fingerprint on its own. The nginx error log is the one people forget: it records client: 203.0.113.9 on every upstream timeout, every 403, every malformed request — and unlike the access log, its format is fixed and cannot be templated.

The auth log — /var/log/auth.log on Debian family, /var/log/secure on RHEL family — names you. Every accepted publickey line carries your source address and the fingerprint of the key that opened the session. Anyone reading a month of it learns which networks you administer from, at what hours, and how many distinct keys you hold. On an anonymously-owned box that file is often more revealing than anything your visitors generated.

The journal holds a copy of most of the above plus every unit's stdout and stderr, and on a default install it is allowed to grow to 10% of the filesystem, capped at 4 GB, before it starts dropping the oldest entries. On any disk of 40 GB or more that ceiling is the full 4 GB, which at the volumes a small server produces is many months.

Application logs are the wildcard. A framework in debug mode logs full URLs including query strings, and query strings routinely carry session tokens, password-reset links and search terms. PHP-FPM can be configured to write its own access log, which duplicates the request line nginx already wrote, from a file your nginx settings never touch.

Container logs are the quiet one. Docker's default json-file driver has no rotation unless you configure it, so /var/lib/docker/containers/*/*-json.log keeps everything the container has said since it was created. It is a common way to discover that a "14-day" retention policy is actually holding eleven months, and it is the same class of surprise as the firewall rules Docker writes behind ufw.

Of the six, two carry identifiers about human beings that are worth minimising on purpose: the nginx access log (your visitors) and the auth log (you). The rest mostly need a size cap and a shorter clock.

Truncate at write time, not at rotation

The instinct is to keep logging normally and clean up later — a nightly cron that rewrites yesterday's file with the addresses stripped. Do not build that. A scrubbing job means the raw addresses genuinely existed on disk for up to a day, and during that day they were snapshotted by your backup run, possibly copied by your host's block-level snapshot, and left in whatever the filesystem did with the old blocks. Worse, the job is a moving part: it fails silently the week the disk fills, and nothing tells you that yesterday's file is still complete.

The only reduction you can actually rely on is the one that happens before the line is written. In nginx that is a map block evaluated at log time, which rewrites the address into a new variable, and a log_format that uses the new variable instead of $remote_addr. The full address is never serialised. There is nothing to clean up afterwards, nothing to schedule, and nothing to get wrong on the day you are not watching.

# /etc/nginx/conf.d/00-privacy-log.conf  --  http context, loaded before the sites
map $remote_addr $ip_trunc {
    # IPv4: keep the /24, zero the host part
    ~(?<v4>\d+\.\d+\.\d+)\.     "${v4}.0";
    # IPv6: keep the first two groups, drop the rest
    ~(?<v6>[^:]+:[^:]+):          "${v6}::";
    # anything the two patterns cannot parse -- including compressed forms
    # like ::1 -- falls through here, i.e. fails closed rather than open
    default                       "0.0.0.0";
}

log_format privacy '$ip_trunc - - [$time_local] "$request" $status '
                   '$body_bytes_sent "$http_referer" "$http_user_agent" '
                   'rid=$request_id rt=$request_time';

Three details decide whether this actually works. First, the map must live in the http context — inside a server block nginx refuses to start. Dropping it in conf.d/ with a name that sorts early is the simplest way to be sure.

Second, if you sit behind Cloudflare or any other edge, check what $remote_addr contains. The realip module replaces it with the real client address early in request processing, and keeps the connecting address in $realip_remote_addr. That ordering is in your favour: the map runs at log time, so it truncates the real visitor rather than the edge. But it also means that if you have not configured realip, you are truncating Cloudflare's address and learning nothing, while the true address sits in a header.

Third, audit the rest of the format string. Truncating $remote_addr accomplishes precisely nothing if the line still ends with "$http_x_forwarded_for" or carries $http_cf_connecting_ip — and a great many stock and control-panel formats include one of those. The full address is in the request headers; it only stays out of the file if you leave every header that carries it out of the format.

Note what replaced it: $request_id, a random 32-character hex string nginx generates per request. That is the pivot for the next section.

What truncation breaks, and the three jobs a log actually does

Someone always objects that anonymised logs are useless, and they are half right — because "the logs" is three unrelated jobs wearing one filename, and only one of them needs the address.

Job one: block whoever is hammering you right now. This needs the complete address and needs it within seconds. It does not need it tomorrow. fail2ban is the usual tool and it genuinely cannot work off a truncated file — banning 203.0.113.0 bans one innocent host and leaves the attacker connected. But this job is satisfied by a file that exists for a day, or by not using a file at all: nginx's own limit_req and limit_conn hold their state in shared memory, act in microseconds rather than on fail2ban's polling interval, and write nothing to disk.

Job two: work out why that request returned a 500. This needs correlation, not identity. A request ID threaded from nginx into the application ties the access line, the upstream error and the application's stack trace to one event — which is what you were actually trying to do when you reached for the address. In practice the ID is better: it survives a client on a mobile network whose address changes mid-session, and it does not go stale when four visitors share one CGNAT address.

Job three: understand traffic over time. Volume, status-code mix, which paths are hot, whether the crawler is out of control. A truncated address is fine here, and the /24 that survives truncation is enough to see that one network is responsible for 40% of your requests.

So the design is not "log less" but split by clock: a full-fidelity stream that lives for a day and feeds the blocking tools, and a truncated stream that lives as long as you want statistics. Both are written by nginx at the same moment, so there is no processing step in between and no window where the wrong thing is on disk for longer than intended.

# inside the server block, or in a snippet included by it
access_log  /var/log/nginx/access.log       privacy;   # truncated, keep for weeks
access_log  /var/log/nginx/security.log     secip;     # full address, keep for a day

# static assets are pure noise in a privacy log -- drop them entirely
location ~* \.(?:css|js|svg|png|jpe?g|webp|woff2?|ico)$ {
    access_log off;
    expires 30d;
}

The secip format is a one-liner next to the other one — $remote_addr, the timestamp, the request and the status, nothing else. It is the only file on the box where a complete visitor address is allowed to sit, which makes its retention a single decision in a single place rather than a property you have to reason about across six files.

journald, auth.log, and the trail that leads back to you

Visitor privacy is the part that gets written about. The administrator's trail is the part that matters on a box whose whole point is that your name is not attached to it, and it is almost entirely in two places.

/var/log/auth.log logs an Accepted publickey line for every session you open, with your source address and your key fingerprint. Over a month it is a schedule of your working habits and a list of the networks you use. If you always connect through the same tunnel, that is one repeated address and relatively dull. If you connect from wherever you happen to be, it is a travel history.

The journal holds the same events plus everything your units printed, and its defaults are generous: SystemMaxUse= is 10% of the filesystem, and MaxRetentionSec= is unset, which means there is no time limit at all — only a size one. On a quiet server that combination retains months.

There is a real trade here and it deserves to be stated rather than hand-waved. The logs that describe you are the same logs that tell you how somebody got in. Set Storage=volatile and the journal lives in RAM only, disappearing at reboot — genuinely private, and genuinely useless the morning you find a process you did not start, because your attacker's first reboot erased the evidence. For most people the sane middle is persistent storage with a hard cap and a short clock: long enough to investigate an incident you notice within a week, short enough that the file is not a diary.

Two implementation notes that catch people. Debian and Ubuntu images differ on whether rsyslog is installed; if /var/log/auth.log exists on your box then rsyslog is writing it, and journald's caps do not govern that file at all — it is logrotate's job. And ForwardToSyslog= is what feeds rsyslog from the journal, so turning it off on a box that has both stops you keeping two copies of everything under two different retention policies.

Your retention policy is whatever your backups say it is

This is the one that undoes all the careful work above, and it is invisible unless you go looking.

Say logrotate keeps fourteen days of nginx logs, and you are happy with that. Now add the off-site backup you were right to set up: a daily Borg or restic run, with a retention of seven daily, four weekly and six monthly archives. Each of those archives contains /var/log as it stood on the day it ran. The oldest monthly archive is six months old and holds the fourteen days of logs that were current then. Your effective log retention is not fourteen days. It is six months, in an encrypted repository you cannot grep without restoring, on a second machine, in a different country.

There are exactly two honest resolutions. Exclude the log directories from the backup — they are the one thing on a server you can nearly always rebuild or do without, and a restore that omits /var/log is not a worse restore. Or include them deliberately and accept that your real retention is the repository's, in which case say so in whatever policy you publish, because the alternative is a stated policy your own infrastructure contradicts.

# exclude logs from the backup, and prove it took
borg create --stats                       \
    --exclude '/var/log'                  \
    --exclude '/var/lib/docker/containers' \
    ::'{hostname}-{now:%Y-%m-%d}' /

# the check that matters: can you still find an address in the newest archive?
borg list ::"$(borg list --short --last 1)" | grep -c '^var/log/' || echo "clean"

While you are in this frame of mind, two neighbours of the same problem. Your host's snapshots are not yours. A hypervisor-level snapshot captures the disk as it was, including logs you rotated away since, and it lives on the provider's storage under the provider's retention — which is one more reason the addresses should never have been written in full rather than a reason to do anything clever afterwards.

And do not reach for shred on a VPS. Overwriting a file on a thin-provisioned virtual disk, over a copy-on-write filesystem, on top of an SSD that remaps blocks for wear-levelling, does not reliably overwrite the physical cells that held it. Secure deletion on rented, virtualised storage is theatre. The reduction that works is the one you made at write time; everything after that is a best effort you cannot verify. Encrypting the volume changes this equation — but it changes it before the data is written, which is the same lesson again.

The copies you do not control

Minimisation on your own box is one layer of several, and being clear about the others is what stops this from becoming a false sense of completeness.

Your hosting network sees the flow record. Source, destination, ports, bytes, timing — for every connection in and out of the machine, whether or not you log anything. No configuration on the server changes that. It is a large part of why the jurisdiction the box sits in is a real variable and not a marketing one; what is legally required of the network differs enormously by country.

Your CDN or edge logs at the edge. If Cloudflare terminates TLS for you, it has the request line and the client address before your server is involved, on its own retention schedule, subject to its own legal process. Truncating your origin log does not reach backwards through it. Running an edge you own is the version of this you can actually configure — and the logging rules in this guide apply to the edge box first, since that is where the untruncated addresses arrive.

Error trackers and analytics ship it off the box for you. Sentry and most of its peers attach the client IP to every event by default; the setting is usually called something like send_default_pii and it is worth checking rather than assuming. Any hosted analytics is, by construction, a third-party copy of the access log you just spent an afternoon truncating.

Mail is the leakiest of all. If anything on the box sends mail, the headers carry the sending host and address, and every relay in the path keeps a copy of the envelope with timestamps. A contact form that emails you is a log you do not administer.

None of this makes the local work pointless — the local copy is the one that gets seized with the machine, exfiltrated in a breach, or handed over by you. It is simply the one layer you fully control, and treating it as the whole picture is the mistake.

Minimisation by design, not deletion on notice

Worth being precise here, because the two get conflated and the difference is the whole difference between standard engineering practice and something you should not do.

Deciding in advance what your service collects and how long it keeps it is ordinary, documented, encouraged practice. Under the GDPR it is two of the core principles — data minimisation and storage limitation — and a shorter log retention is a control auditors ask for, not one they object to. There is no general obligation on a website operator or a hosting customer in the EU to retain traffic logs; the blanket-retention directive that once implied otherwise was struck down by the Court of Justice in 2014, and the national laws that survive it mostly bind telecommunications providers, not people who run a web server. The United States has no general retention mandate for site operators either.

Destroying specific records after you have been put on notice about them is a completely different act. A preservation request, a litigation hold, a court order, or a police enquiry changes what you may do with the data that exists at that moment, and "my retention policy deleted it" is not a defence if you accelerated the deletion because of the notice. Nothing in this guide is about that. A retention policy is something you set on a quiet Tuesday and then leave alone; if it only gets shortened when something happens, it was not a policy.

The same distinction runs through the rest of this site: there is a real line between privacy engineering and the "bulletproof" posture that markets itself as immunity. Designing a service that never accumulates a visitor history sits comfortably on the right side of it, in the same box as encrypting your disks and not asking users for an email address you do not need.

Two practical consequences. If you publish a privacy policy, make its retention figures match what is actually on the disk — a stated fourteen days and a real six months in a backup repository is the kind of gap that turns a good-faith operator into a bad-faith one on paper. And write the decision down for yourself, in a comment at the top of the logrotate file if nowhere else, because the person who has to justify these numbers in eighteen months is you, and they will not remember why the number was seven. None of the above is legal advice; if you operate somewhere with sector-specific retention duties, check them against your own situation before you shorten anything.

Step by step

1. 01

   Find out what the box is already keeping

   Do not configure anything until you have measured. The point of this pass is to find the file you forgot about, which on most machines is either a container log or an application log nobody has looked at since the deploy.

   # biggest log files anywhere on the box, largest last
   sudo du -ah /var/log /var/lib/docker/containers 2>/dev/null \
     | sort -h | tail -20
   
   # how much disk the journal holds, and how far back it goes
   journalctl --disk-usage
   journalctl --output=short-iso | head -1
   
   # how old is the oldest nginx line still on disk?
   zcat -f /var/log/nginx/access.log* 2>/dev/null | head -1

   Then look at one line from each file and ask what it identifies. The command below counts how many distinct complete addresses are currently retrievable from your web logs — it is usually the number that makes the case for the rest of this guide.

   zcat -f /var/log/nginx/access.log* 2>/dev/null \
     | grep -oE '^([0-9]{1,3}\.){3}[0-9]{1,3}' | sort -u | wc -l

   Write down what you find. You will run these same commands at the end to prove the change took effect.

2. 02

   Truncate the client address before nginx writes the line

   Create the map and the two formats in a file loaded into the http context. On Debian and Ubuntu, /etc/nginx/conf.d/ is included from nginx.conf before the site configs, which is exactly where this belongs.

   sudo tee /etc/nginx/conf.d/00-privacy-log.conf > /dev/null <<'EOF'
   # The client address is reduced here, at log time, and never written in full
   # to the long-retention file. $realip_remote_addr still holds the connecting
   # address if you need it while debugging a proxy problem.
   map $remote_addr $ip_trunc {
       ~(?<v4>\d+\.\d+\.\d+)\.     "${v4}.0";
       ~(?<v6>[^:]+:[^:]+):          "${v6}::";
       # anything the two patterns cannot parse -- including compressed forms
       # like ::1 -- falls through here, i.e. fails closed rather than open
       default                       "0.0.0.0";
   }
   
   # Long retention: no full address, no forwarded-for header, request id instead.
   log_format privacy '$ip_trunc - - [$time_local] "$request" $status '
                      '$body_bytes_sent "$http_referer" "$http_user_agent" '
                      'rid=$request_id rt=$request_time';
   
   # Short retention: the one file allowed to hold a complete address.
   log_format secip   '$remote_addr [$time_local] "$request" $status';
   EOF
   
   sudo nginx -t && sudo systemctl reload nginx

   Now point the site at them. In your server block, replace the existing access_log line with the pair, and switch off logging for static assets while you are there.

   access_log  /var/log/nginx/access.log    privacy;
   access_log  /var/log/nginx/security.log  secip;

   Reload, load a page, and read the result. The first field should end in .0, and the line should carry a rid= value.

   sudo nginx -t && sudo systemctl reload nginx
   curl -s -o /dev/null https://your-domain.example/
   sudo tail -1 /var/log/nginx/access.log

   If the address is still complete, the server block is overriding the format somewhere further down — grep -rn access_log /etc/nginx/ finds the line that wins.

3. 03

   Keep full addresses only where something acts on them

   If fail2ban is running, it is currently reading the file you just truncated. Point it at the security log instead, and give that log a one-day life so the full addresses expire on their own.

   sudo tee /etc/fail2ban/jail.d/nginx-privacy.local > /dev/null <<'EOF'
   [nginx-http-auth]
   enabled  = true
   logpath  = /var/log/nginx/error.log
   
   [nginx-botsearch]
   enabled  = true
   logpath  = /var/log/nginx/security.log
   maxretry = 6
   findtime = 10m
   bantime  = 1h
   EOF
   
   sudo fail2ban-client reload

   Then deal with fail2ban's own memory, which most people never touch: it keeps its own log of bans under the default rotate 4 weekly, and a SQLite database of every ban it has issued. dbpurgeage is what expires rows out of that database — set it to something close to your longest ban rather than leaving it at the default.

   sudo tee /etc/fail2ban/fail2ban.d/retention.local > /dev/null <<'EOF'
   [Definition]
   dbpurgeage = 2d
   loglevel   = NOTICE
   EOF
   
   sudo systemctl restart fail2ban

   Better still, for plain floods, do the work in nginx where nothing is written down at all. limit_req holds its counters in shared memory, responds in microseconds instead of on a polling interval, and leaves no record of who was throttled — see the first-hour DDoS runbook for sizing the zones under real load.

   # http context: keyed on the truncated address, so nothing complete is
   # held in memory either. 10m of shared state is plenty for a small site.
   limit_req_zone $ip_trunc zone=perip:10m rate=20r/s;
   
   # in the location you want protected
   limit_req zone=perip burst=40 nodelay;

   Keying the zone on $ip_trunc rather than $binary_remote_addr is a deliberate trade: the limit now applies to a whole /24 at once, so a busy office behind one block shares a budget. For a small site that is usually fine and occasionally an improvement; if it is not, key the zone on the full address — a rate-limiter's state lives in shared memory and is never written to disk, so it is not part of what you are minimising here.

4. 04

   Cap the journal and stop the second copy

   journald takes a drop-in, which survives package upgrades in a way that editing journald.conf does not. The values below keep about a week — enough to investigate something you notice on Monday that started on Friday — inside a hard 200 MB ceiling.

   sudo mkdir -p /etc/systemd/journald.conf.d
   sudo tee /etc/systemd/journald.conf.d/retention.conf > /dev/null <<'EOF'
   [Journal]
   Storage=persistent
   SystemMaxUse=200M
   SystemMaxFileSize=20M
   MaxRetentionSec=7day
   MaxFileSec=1day
   ForwardToSyslog=no
   EOF
   
   sudo systemctl restart systemd-journald
   journalctl --disk-usage

   The restart applies the size caps immediately; the retention clock is enforced as new files are rotated, so an existing oversized journal shrinks on the next rotation rather than instantly. journalctl --vacuum-time=7d forces it now if you want the disk back today.

   ForwardToSyslog=no matters on any image that ships rsyslog: without it, every journal entry is also appended to /var/log/syslog, under logrotate's schedule rather than journald's, and you have two copies with two different expiry dates. Check which situation you are in before assuming:

   systemctl is-active rsyslog 2>/dev/null || echo "rsyslog not running"
   ls -la /var/log/auth.log /var/log/syslog 2>/dev/null

   If those files exist, rsyslog owns them and step six is where their retention gets set. If they do not, the journal is the only copy and you have just capped it.

5. 05

   Stop the application from re-logging what you removed

   Nothing above touches your application, and an application in the wrong mode will cheerfully write the full address, the full URL and the session token into a file of its own. Three things to check.

   PHP-FPM ships an access.log directive in its pool config, commented out by default but enabled by a great many control panels. If it is on, it is a second copy of every request line, from a file your nginx work never touched.

   grep -rn '^access.log' /etc/php/*/fpm/pool.d/ || echo "fpm access log off"

   Your framework's log level decides whether query strings and request bodies land on disk. Debug mode in most frameworks logs the full URL, and a password-reset link is a full URL. Set the production level and confirm it is actually the one loaded, rather than the one in the file you think is being read.

   Docker's default driver never rotates. Fix it at the daemon level so every future container inherits the cap. Note that this applies to containers created after the restart — existing ones keep their current, unbounded file until they are recreated.

   sudo tee /etc/docker/daemon.json > /dev/null <<'EOF'
   {
     "log-driver": "json-file",
     "log-opts": { "max-size": "10m", "max-file": "3" }
   }
   EOF
   
   sudo systemctl restart docker
   docker inspect --format '{{.HostConfig.LogConfig}}' $(docker ps -q) 2>/dev/null

   If a container is already holding a large file, recreating it with docker compose up -d --force-recreate is what actually truncates the history — restarting alone keeps the same log file.

6. 06

   Set retention deliberately, in one place per file

   logrotate is where the clock lives for everything rsyslog and nginx write. Edit the shipped stanza rather than adding a second one: two stanzas naming the same path make logrotate fail with a duplicate-entry error and stop rotating that file entirely, which is the most common way a retention change quietly becomes an infinite one.

   # check for duplicates BEFORE editing, then again after
   sudo logrotate --debug /etc/logrotate.conf 2>&1 | grep -i 'duplicate\|error'

   For nginx, the two files want different clocks: the truncated one can live for weeks, the one holding complete addresses should not survive the day. Add a separate stanza for the security log — a different path, so no duplicate — and shorten the shipped one.

   sudo tee /etc/logrotate.d/nginx-security > /dev/null <<'EOF'
   # The only file on this box that holds complete client addresses.
   # One day, uncompressed so fail2ban can read it. Do not lengthen without
   # a reason you would be happy to write down here.
   /var/log/nginx/security.log {
       daily
       rotate 1
       maxage 1
       missingok
       notifempty
       nocompress
       create 0640 www-data adm
       sharedscripts
       postrotate
           [ -f /var/run/nginx.pid ] && kill -USR1 `cat /var/run/nginx.pid`
       endscript
   }
   EOF
   
   sudo sed -i 's/^\trotate 14$/\trotate 7/' /etc/logrotate.d/nginx
   sudo logrotate --debug /etc/logrotate.d/nginx-security

   Do the same arithmetic for /etc/logrotate.d/rsyslog if rsyslog is installed — its defaults keep four weeks of auth.log, which is four weeks of your own SSH sessions. And run the debug pass one final time: it prints exactly which files it would rotate and delete, which is the only way to confirm the numbers you just typed are the numbers in force.

7. 07

   Prove it, including through the backup

   Re-run the measurements from step one. The count of distinct complete addresses in the long-retention log should now stop growing, and after one rotation cycle it should be zero.

   # should print 0 once the pre-change files have rotated out
   zcat -f /var/log/nginx/access.log* 2>/dev/null \
     | grep -oE '^([0-9]{1,3}\.){3}[0-9]{1,3}' \
     | grep -v '\.0$' | sort -u | wc -l
   
   # and the journal should now be bounded
   journalctl --disk-usage

   Then the check almost nobody runs: look inside the backup. A repository that still carries /var/log is holding the fidelity and the retention you just spent an afternoon removing, and it will keep holding it for as long as your oldest archive survives.

   borg list | head -3          # borg lists oldest first
   borg list ::"$(borg list --short | head -1)" 2>/dev/null \
     | grep -c '^var/log/' || echo "no logs in oldest archive"

   If the count is non-zero, either add the exclusion from earlier and let the old archives age out, or prune them deliberately. Until one of those happens, your real retention is the repository's — which is the single most useful sentence in this guide, and the easiest to forget.

   Finally, write the numbers down where the next person will find them: the retention you chose, the reason, and the date. A comment at the top of the logrotate file is enough. The configuration above is a decision, and a decision nobody can reconstruct in a year becomes a default again.