Automated Linux Directory Backup to S3 with Failure Alerts

Backing up critical Linux directories is essential for system reliability, and automating this process reduces human error. In this guide, I will walk you through how I automated backup of /var/log to Amazon S3, added SNS email notifications for failures, and resolved common issues encountered in production.

---

Step 1: Prerequisites

Before starting, ensure the following:

• Linux server (EC2 instance) with sudo access.

• AWS CLI installed and configured.

• IAM role attached to EC2 with:

  • AmazonS3FullAccess or bucket-level S3 permissions.

  • AmazonSNSFullAccess.

• Basic knowledge of shell scripting.

---

Step 2: Create S3 Bucket

Create a dedicated S3 bucket for backups:

aws s3 mb s3://linux-backup-bucket-123

---

Step 3: Set Up SNS for Failure Alerts

1. Create an SNS topic

aws sns create-topic --name backup-failure-topic

Save the TopicArn from the output.

2. Subscribe an email to the topic

aws sns subscribe \
  --topic-arn <TopicArn> \
  --protocol email \
  --notification-endpoint your-email@example.com

Confirm the subscription from your inbox.

---

Step 4: Create Backup Script

Create a script /opt/s3_backup.sh:

sudo vi /opt/s3_backup.sh

Script Contents:

#!/bin/bash

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

SOURCE_DIR="/var/log"
BACKUP_DIR="/tmp/backups"
BACKUP_NAME="backup_$(date +%F_%H-%M-%S).tar.gz"
S3_BUCKET="s3://linux-backup-bucket-123"
SNS_TOPIC_ARN="<Your_SNS_TopicArn>"

mkdir -p "$BACKUP_DIR"

# Create backup
/bin/tar --ignore-failed-read \
         --warning=no-file-changed \
         -czf "\(BACKUP_DIR/\)BACKUP_NAME" "$SOURCE_DIR"

TAR_EXIT_CODE=$?

# Handle warnings vs failure
if [ "$TAR_EXIT_CODE" -gt 1 ]; then
  /usr/bin/aws sns publish \
    --topic-arn "$SNS_TOPIC_ARN" \
    --message "Backup FAILED during tar (exit code \(TAR_EXIT_CODE) on \)(hostname)"
  exit 1
fi

# Upload to S3 recursively
/usr/bin/aws s3 cp "\(BACKUP_DIR" "\)S3_BUCKET/" --recursive

if [ $? -ne 0 ]; then
  /usr/bin/aws sns publish \
    --topic-arn "$SNS_TOPIC_ARN" \
    --message "Backup FAILED during S3 upload on $(hostname)"
  exit 1
fi

Make the script executable:

sudo chmod +x /opt/s3_backup.sh

---

Step 5: Manual Test

Run the script manually to verify:

sudo /opt/s3_backup.sh

Check S3 to confirm the backup file exists.

---

Step 6: Schedule Backup with Cron

Add a cron job to automate the backup. Use root crontab:

sudo crontab -e

* * * * * /opt/s3_backup.sh >> /var/log/s3_backup.log 2>&1

---

Step 7: Common Problems and Their Solutions

Problem 1: Permission Denied

Error:

tar: /var/log/audit: Cannot open: Permission denied

Solution:
Run the script as root to access all system logs.

---

Problem 2: File Changed as We Read It

Error:

tar: file changed as we read it

Solution:
Use --ignore-failed-read --warning=no-file-changed and handle tar exit codes:

• 0 → success

• 1 → warnings (acceptable)

• >1 → real failures

---

Problem 3: Cron Job Failing and SNS Emails

• Cron jobs run in a limited PATH, causing tar or aws commands to fail.

• Logs /var/log/cron or /var/log/messages may not exist on systemd-based Linux.

Solution:

• Use absolute paths for commands:

/bin/tar
/usr/bin/aws

• Set PATH at the top of the script:

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

• Check cron logs via journald:

sudo journalctl -u crond --no-pager | tail -n 50

• ---

Step 8: Verify Cron Execution

sudo tail -n 50 /var/log/s3_backup.log

• Check backups in S3.

• Confirm no failure emails unless a real error occurs.

---

Step 9: Key Learnings

• Always handle tar exit codes correctly when backing up live logs.

• Cron runs in a minimal environment, so absolute paths and PATH settings are necessary.

• Automating S3 backup with SNS notifications ensures proactive alerts for failures.

• Systemd journald changes logging paths; /var/log/cron may not exist.

---

Conclusion

With this setup, you now have:

• Fully automated Linux directory backups to S3.

• Recursive uploads for multiple backups.

• Failure notifications via SNS.

• Production-ready cron automation.

This approach is robust, scalable, and easily extendable for more directories or servers.