maintenance.md

Table Maintenance

0.13.0: expire_snapshots previously raised NotImplementedError whenever it had work to do — it called Table.expire_snapshots, which is not an attribute of PyIceberg's Table. It is now implemented on the real API (Table.maintenance.expire_snapshots) and returns the list of expired snapshot ids. remove_orphan_files is now dry-run by default; pass dry_run=False to actually delete.

Iceberg tables require periodic maintenance to ensure optimal performance and manage storage costs. IceFrame provides simple methods for common maintenance tasks.

Expiring Snapshots

Remove old table snapshots to free up space and keep metadata size manageable.

# Remove snapshots older than 7 days, keeping at least the last 1
ice.expire_snapshots("my_table", older_than_days=7, retain_last=1)

Removing Orphan Files

Clean up data files that are no longer referenced by any snapshot (e.g., from failed writes).

# Remove orphan files older than 3 days
ice.remove_orphan_files("my_table", older_than_days=3)

Compacting Data Files

Combine small data files into larger ones to improve read performance (compaction).

# Compact files to target size of 512 MB
ice.compact_data_files("my_table", target_file_size_mb=512)

Tip

Run compaction regularly on tables with frequent small updates (streaming ingestion).

Best Practices

1. Schedule Maintenance: Run these operations periodically (e.g., daily or weekly) via a scheduler like Airflow.
2. Order of Operations:
   1. expire_snapshots
   2. remove_orphan_files
   3. compact_data_files