Why Backups Fail (and How the 3-2-1 Rule Fixes It)
Most data-loss incidents are not caused by exotic failures. They come from the boring stuff: a rm -rf in the wrong directory, a corrupted database after a power event, ransomware that encrypts everything it can reach, or a single disk that quietly dies. The common thread is that the "backup" was either nonexistent, stored on the same machine, or never actually tested.
The 3-2-1 rule is the industry-standard answer, and it is deliberately simple:
- 3 copies of your data (the live copy plus two backups)
- 2 different types of storage media or technologies
- 1 copy kept off-site
If you follow it honestly, no single failure — hardware, human error, or malware — can take out all three copies at once. This guide shows how to implement 3-2-1 on real Linux servers using snapshots, object storage, and the restic backup tool, with an emphasis on keeping your off-site copy inside Saudi Arabia for PDPL data-residency compliance.
Mapping 3-2-1 to Real Infrastructure
Here is a concrete layout for a typical web application or business email workload:
| Copy | What it is | Media / type | Location |
|---|---|---|---|
| 1 (live) | Production server disk | NVMe/SSD | Primary cloud region |
| 2 (backup) | Server snapshot | Block-storage image | Same provider, separate storage layer |
| 3 (backup) | restic repository |
Object storage (S3-compatible) | Off-site bucket, in-Kingdom |
This satisfies all three numbers: three copies, two distinct technologies (block snapshot vs. object storage), and one off-site. Crucially, the off-site copy lives in a different storage system, so a fault in the production block layer cannot corrupt it.
Step 1: Snapshots — Fast but Not Enough
A snapshot captures the entire server volume at a point in time. It is the fastest way to recover from a bad deploy or a botched upgrade because you can roll the whole machine back in minutes.
On a Skyline cloud server or VPS, you can take a manual snapshot from the control panel before any risky change, and schedule automatic ones. The important caveat: a snapshot stored on the same provider is copy #2, not your off-site copy. It protects against your mistakes, not against an account-level or regional incident. Never treat snapshots as your only backup.
Step 2: Application-Consistent Database Dumps
Snapshots are crash-consistent, not necessarily application-consistent. For databases, take a proper logical dump so you can restore a single table without rolling back the whole server.
For MySQL/MariaDB:
mysqldump --single-transaction --quick --routines --triggers \
--all-databases | gzip > /var/backups/db-$(date +%F).sql.gz
For PostgreSQL:
pg_dumpall -U postgres | gzip > /var/backups/pg-$(date +%F).sql.gz
The --single-transaction flag gives you a consistent snapshot of InnoDB tables without locking writes. Write these dumps into a directory that your off-site backup job will pick up.
Step 3: The Off-Site Copy with restic and Object Storage
This is the copy that makes the strategy real. restic is an excellent open-source backup tool: it deduplicates, compresses, and encrypts client-side before anything leaves your server. That last point matters — your data is encrypted before it touches the network, so the storage provider never sees plaintext.
Install and initialize against an S3-compatible object-storage bucket:
# Install restic (Debian/Ubuntu)
sudo apt-get update && sudo apt-get install -y restic
# Point restic at your in-Kingdom object storage bucket
export RESTIC_REPOSITORY="s3:https://objects.example-region.com/my-backup-bucket"
export AWS_ACCESS_KEY_ID="<your-access-key>"
export AWS_SECRET_ACCESS_KEY="<your-secret-key>"
export RESTIC_PASSWORD="<a-long-random-passphrase>"
# Initialize the repository once
restic init
Now run your first backup, including the database dumps and the web root:
restic backup /var/www /etc /var/backups
restic only uploads changed blocks after the first run, so subsequent backups are fast and cheap.
Automate and prune
Add a daily cron job and a retention policy so the repository does not grow forever:
# /etc/cron.d/restic-backup
30 2 * * * root /usr/local/bin/restic-backup.sh
#!/usr/bin/env bash
# /usr/local/bin/restic-backup.sh
set -euo pipefail
source /root/.restic-env # exports the variables above
restic backup /var/www /etc /var/backups --tag daily
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --prune
This keeps 7 daily, 4 weekly, and 6 monthly restore points — a sensible default that balances coverage against object-storage cost.
Step 4: Harden Against Ransomware
A backup that an attacker can delete is not a backup. Ransomware increasingly targets backup repositories. Two practical defenses:
- Separate credentials: the object-storage key used by the server should ideally be append/write-only, not allowed to delete. Run
restic prunefrom a separate, more trusted context. - Immutability / object lock: enable write-once-read-many (WORM) retention on the bucket where supported, so backups cannot be overwritten or deleted before a set period.
This turns your off-site copy into a true last line of defense.
Step 5: Test Restores — the Step Everyone Skips
An untested backup is a hope, not a plan. Schedule a recurring restore drill:
# List available snapshots
restic snapshots
# Restore a specific snapshot into a scratch directory
restic restore <snapshot-id> --target /tmp/restore-test
# Verify repository integrity
restic check
Confirm the restored database actually imports and the application starts. Put a calendar reminder to do this monthly. The first time you discover a backup is unrecoverable should never be during a real outage.
Why In-Kingdom Matters
If you process personal data of people in Saudi Arabia, the PDPL and NCA guidance push you toward keeping data — including backups — inside the Kingdom unless you have a lawful basis for transfer. Your off-site copy is still personal data. Hosting it on Skyline Cloud gives you in-Kingdom data residency, S3-compatible object storage for restic, and Arabic-speaking support when you are mid-incident and need answers fast.
Get Started
You can stand up a complete 3-2-1 setup — a cloud server, snapshots, and an in-Kingdom object-storage bucket — in minutes. Create your Skyline Cloud account and build a backup strategy you can actually rely on.
Comments
0 total · 0 threads