Appearance
Troubleshooting: Backup & Recovery
Resolve backup failures, missing backups, and restore errors for platform and tenant recovery.
Backup not running
| Check | Action |
|---|---|
Backup:Enabled | Must be true in settings |
| Scheduler service | API host must be running continuously |
IntervalHours | Check schedule in admin portal |
| Disk space | Local path must have free space |
| S3 credentials | Verify access key and bucket permissions |
Platform admin: Admin Portal → Backups → Settings
Backup stuck in Running
| Cause | Resolution |
|---|---|
| API crashed mid-backup | Restart API — stale runs marked failed after 15 min |
| Large database | Allow more time; check server resources |
| pg_dump hung | Verify PostgreSQL connectivity |
| S3 upload slow | Check network bandwidth |
Check backup detail for error message after status changes to Failed.
Backup failed
| Error | Fix |
|---|---|
pg_dump not found | Set PgDumpPath or PgBinDirectory in config |
| Permission denied (DB) | Connection string needs dump privileges |
| S3 access denied | Fix IAM policy for bucket |
| Disk full | Clear old backups or expand storage |
See Platform Backups.
No backups visible (tenant owner)
| Cause | Fix |
|---|---|
| Platform backup never completed | Contact platform admin |
| Retention expired | Older backups auto-deleted |
| Tenant created after backup | Wait for next scheduled run |
| Wrong account | Must be tenant owner |
API: GET /tenant/backups
Restore failed
| Error | Fix |
|---|---|
pg_restore not found | Configure PgRestorePath on API server |
| Confirmation mismatch | Type exact confirmation phrase |
| Tenant not in manifest | Backup may predate tenant; use newer backup |
| Active connections | Restore drops and recreates database |
| Insufficient permissions | API needs PostgreSQL superuser or CREATEDB |
Restore took too long
| Factor | Guidance |
|---|---|
| Database size | Large tenants may take 30+ minutes |
| Do not interrupt | Wait for status to change |
| Monitor API logs | Watch for pg_restore progress |
Data missing after restore
| Scenario | Explanation |
|---|---|
| Data after backup date | Expected — restore is point-in-time |
| Users can't login | Passwords restored to backup state |
| Subscriptions wrong | Tenant restore doesn't fix platform billing |
| Uploads missing | Check uploads archive in manifest |
S3-specific issues
| Issue | Fix |
|---|---|
| Backup not in bucket | Check Prefix path in S3 console |
| Cannot download | Verify IAM read permissions |
| MinIO compatibility | Set ForcePathStyle: true and ServiceUrl |
Testing backups
| Test | Frequency |
|---|---|
| Verify backup completes | Daily (automated) |
| Download and inspect manifest | Weekly |
| Test tenant restore on staging | Quarterly |
| Full DR drill | Annually |
Emergency recovery
| Disaster | Action |
|---|---|
| Single tenant corruption | Single-tenant restore from latest backup |
| Platform server loss | Deploy new API + restore FullPlatform from S3 |
| Accidental delete | Restore from backup before delete timestamp |
DANGER
Full platform restore replaces ALL tenant data. Coordinate with all labs before proceeding.