FM-KED-004 — Backup Failure Due to Excessive Backup Size

Severity: S1 — Critical
Recovery Class: B — Standard Recovery
Covered by Monthly Support: Yes

Description

Automatic daily backups fail because the generated database dump exceeds available disk capacity on the backup worker or application node.

This condition usually indicates uncontrolled growth of historical data and must be addressed immediately to restore backup continuity.

Typical Symptoms

• Daily backup job not completed or failing repeatedly
• Backup files missing or incomplete
• Disk space exhaustion on backup or worker node
• Alerts indicating failed or skipped backup runs

Root Cause

• Excessive volume of historical records retained in the database
• Insufficient disk capacity on the worker node where
  • the Authorizer VM is deployed, or
  • the Finmars Community Edition is installed

Diagnostic Checklist

Verify Backup Job Status

• Confirm backup job execution logs
• Identify failure reason and timestamps

Check Disk Space on Backup Node

df -h

Estimate Database Dump Size

pg_dump --format=custom --file=/tmp/test.dump db_name
ls -lh /tmp/test.dump

Recovery Options

Choose one or a combination, depending on business requirements.

Option 1: Remove Obsolete Historical Records

• Identify historical tables with excessive row counts
• Confirm data retention requirements
• Delete or archive obsolete records
• Re-run backup after cleanup

⚠️ Data deletion is irreversible and must be explicitly approved.

Option 2: Increase Disk Capacity on Backup Worker Node

• Extend disk size of the worker node
• Ensure sufficient free space for full backup generation
• Re-run the backup job and verify completion

This option preserves all historical data.

Preventive Notes

• Define and enforce data retention policies
• Monitor backup file sizes over time
• Monitor free disk space on backup and worker nodes
• Periodically validate backup completion, not only existence

Responsibility Boundary

Finmars SCSA provides diagnostics, recommendations, and operational guidance.
Infrastructure changes such as disk resizing may depend on customer approval or cloud provider action.