Checkpointing & Recovery

This guide shows how to save and restore evolution state for long-running optimizations.

Why Checkpointing?

  • Resume interrupted runs: Continue after crashes or shutdowns
  • Experiment branching: Try different strategies from the same point
  • Progress monitoring: Analyze intermediate states
  • Resource limits: Work within time-limited environments

Basic Checkpointing

Creating Checkpoints

use fugue_evo::prelude::*;
use std::path::PathBuf;

// Create checkpoint manager
let checkpoint_dir = PathBuf::from("./checkpoints");
let mut manager = CheckpointManager::new(&checkpoint_dir, "my_evolution")
    .every(50)    // Save every 50 generations
    .keep(3);     // Keep last 3 checkpoints

// In evolution loop
for gen in 0..max_generations {
    // ... evolution step ...

    if manager.should_save(gen + 1) {
        let individuals: Vec<Individual<RealVector>> = population.iter().cloned().collect();
        let checkpoint = Checkpoint::new(gen + 1, individuals)
            .with_evaluations((gen + 1) * population_size);

        manager.save(&checkpoint)?;
        println!("Saved checkpoint at generation {}", gen + 1);
    }
}

Loading Checkpoints

use fugue_evo::checkpoint::load_checkpoint;

// Load specific checkpoint
let checkpoint: Checkpoint<RealVector> = load_checkpoint("./checkpoints/my_evolution_gen_100.ckpt")?;

println!("Loaded generation: {}", checkpoint.generation);
println!("Population size: {}", checkpoint.population.len());
println!("Evaluations: {}", checkpoint.evaluations);

// Reconstruct population
let mut population: Population<RealVector, f64> =
    Population::with_capacity(checkpoint.population.len());
for ind in checkpoint.population {
    population.push(ind);
}

Complete Example

//! Checkpointing and Reproducible Recovery
//!
//! This example demonstrates how to save and restore evolution state using
//! checkpoints so that a long-running optimization can be interrupted and
//! resumed **bit-identically** - i.e. a run that is checkpointed, restored, and
//! continued reaches the *exact same* result as a run that was never
//! interrupted.
//!
//! The key ingredient (EV-02) is capturing the RNG state alongside the
//! population. Reproducible resume requires a snapshot-able ChaCha RNG
//! (`ChaCha8Rng`/`ChaCha12Rng`/`ChaCha20Rng`); generic generators such as
//! `StdRng`/`ThreadRng` cannot be serialized and therefore cannot reproduce the
//! stochastic trajectory.
//!
//! Note that the whole example is expressed against the **library** resume API:
//! [`SimpleGA::init_run`]/[`SimpleGA::step_generation`] drive an incremental
//! run, [`SimpleGA::checkpoint_run`] snapshots it (population + RNG),
//! [`save_checkpoint`]/[`load_checkpoint`] round-trip it through disk, and
//! [`SimpleGA::run_from_checkpoint`] resumes it — no hand-rolled generation loop.

use fugue_evo::prelude::*;
use rand::SeedableRng;
use rand_chacha::ChaCha8Rng;
use std::path::{Path, PathBuf};

const DIM: usize = 10;
const POP_SIZE: usize = 100;
const SEED: u64 = 42;
const TOTAL_GENERATIONS: usize = 20;
const CHECKPOINT_AT: usize = 10;

type SphereGa = SimpleGA<
    RealVector,
    f64,
    TournamentSelection,
    SbxCrossover,
    PolynomialMutation,
    Sphere,
    MaxGenerations,
>;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    println!("=== Checkpointing and Reproducible Recovery ===\n");

    let checkpoint_dir = PathBuf::from("/tmp/fugue_evo_checkpoints");
    if checkpoint_dir.exists() {
        std::fs::remove_dir_all(&checkpoint_dir)?;
    }
    std::fs::create_dir_all(&checkpoint_dir)?;

    // (1) Straight run: TOTAL_GENERATIONS with no interruption.
    let straight_best = run_straight(TOTAL_GENERATIONS);
    println!("Straight run ({TOTAL_GENERATIONS} gens):            best = {straight_best:.12}");

    // (2) Interrupted run: CHECKPOINT_AT gens, checkpoint (population + RNG) to
    //     disk, restore, then continue for the remaining gens — all via the
    //     library resume API.
    let resumed_best = run_with_checkpoint(&checkpoint_dir, TOTAL_GENERATIONS, CHECKPOINT_AT)?;
    println!(
        "Resumed run ({CHECKPOINT_AT} + {} gens via disk): best = {resumed_best:.12}",
        TOTAL_GENERATIONS - CHECKPOINT_AT
    );

    // (3) The two must be bit-identical.
    println!();
    if straight_best.to_bits() == resumed_best.to_bits() {
        println!("SUCCESS: resumed run is bit-identical to the uninterrupted run.");
    } else {
        return Err(format!(
            "reproducibility broken: straight={straight_best} resumed={resumed_best}"
        )
        .into());
    }

    if checkpoint_dir.exists() {
        std::fs::remove_dir_all(&checkpoint_dir)?;
        println!("\nCheckpoint directory cleaned up.");
    }

    Ok(())
}

/// Build the GA used by both runs. `real_valued()` fixes the genome/fitness
/// types and installs tournament selection, SBX crossover, and polynomial
/// mutation as defaults (no turbofish); elitism mirrors a typical config.
fn build_ga(total: usize) -> SphereGa {
    SimpleGABuilder::real_valued()
        .population_size(POP_SIZE)
        .bounds(MultiBounds::symmetric(5.12, DIM))
        .fitness(Sphere::new(DIM))
        .elitism(true)
        .elite_count(1)
        .max_generations(total)
        .build()
        .expect("build GA")
}

/// Run `generations` generations start-to-finish via the incremental API and
/// return the best fitness.
fn run_straight(generations: usize) -> f64 {
    let ga = build_ga(generations);
    let mut rng = ChaCha8Rng::seed_from_u64(SEED);
    let mut state = ga.init_run(&mut rng).expect("init_run");
    while ga.step_generation(&mut state, &mut rng).expect("step") {}
    ga.finish_run(state).best_fitness
}

/// Run `checkpoint_at` generations, persist a checkpoint (population + RNG) to
/// disk via the library, then load it back and continue to `total` generations
/// with [`SimpleGA::run_from_checkpoint`]. Returns the final best fitness.
fn run_with_checkpoint(
    checkpoint_dir: &Path,
    total: usize,
    checkpoint_at: usize,
) -> Result<f64, Box<dyn std::error::Error>> {
    let ga = build_ga(total);

    // --- Phase 1: run up to the checkpoint ---
    let mut rng = ChaCha8Rng::seed_from_u64(SEED);
    let mut state = ga.init_run(&mut rng)?;
    for _ in 0..checkpoint_at {
        ga.step_generation(&mut state, &mut rng)?;
    }

    // Snapshot the in-progress run — the algorithm captures the population,
    // best, evaluations, statistics AND the ChaCha RNG state for us.
    let checkpoint = ga.checkpoint_run(&state, &rng)?;
    let path = checkpoint_dir.join("resume.ckpt");
    save_checkpoint(&checkpoint, &path, CheckpointFormat::Binary)?;
    println!("Saved checkpoint at generation {checkpoint_at} -> {path:?}");

    // --- Phase 2: simulate a restart and resume from disk ---
    let checkpoint: Checkpoint<RealVector> = load_checkpoint(&path)?;
    let resumed = ga.run_from_checkpoint::<ChaCha8Rng>(&checkpoint)?;

    Ok(resumed.best_fitness)
}

Source: examples/checkpointing.rs

Running the Example

cargo run --example checkpointing

Checkpoint Manager Options

Save Frequency

// Every N generations
CheckpointManager::new(&dir, "name").every(50);

// Only at specific generations
CheckpointManager::new(&dir, "name").at_generations(&[100, 200, 500]);

Retention Policy

// Keep last N checkpoints
CheckpointManager::new(&dir, "name").keep(3);

// Keep all checkpoints
CheckpointManager::new(&dir, "name").keep_all();

// Custom retention
CheckpointManager::new(&dir, "name").keep_every(100); // Keep every 100th

Custom Naming

// Default: name_gen_N.ckpt
let manager = CheckpointManager::new(&dir, "experiment_1");
// Creates: experiment_1_gen_50.ckpt, experiment_1_gen_100.ckpt, etc.

Checkpoint Contents

The Checkpoint struct stores:

pub struct Checkpoint<G: EvolutionaryGenome> {
    /// Current generation number
    pub generation: usize,

    /// Full population with fitness values
    pub population: Vec<Individual<G>>,

    /// Total fitness evaluations so far
    pub evaluations: usize,

    /// Optional metadata
    pub metadata: Option<CheckpointMetadata>,
}

Adding Metadata

let checkpoint = Checkpoint::new(gen, individuals)
    .with_evaluations(evaluations)
    .with_metadata(CheckpointMetadata {
        timestamp: chrono::Utc::now(),
        best_fitness: population.best().map(|b| *b.fitness_value()),
        config: serde_json::to_string(&config).ok(),
    });

Resume Strategy

Find Latest Checkpoint

fn find_latest_checkpoint(dir: &Path, prefix: &str) -> Option<PathBuf> {
    std::fs::read_dir(dir)
        .ok()?
        .filter_map(|e| e.ok())
        .filter(|e| {
            e.path()
                .file_name()
                .and_then(|n| n.to_str())
                .map(|n| n.starts_with(prefix) && n.ends_with(".ckpt"))
                .unwrap_or(false)
        })
        .max_by_key(|e| e.path())
        .map(|e| e.path())
}

Resume or Start Fresh

fn run_evolution(checkpoint_dir: &Path) -> Result<(), Box<dyn Error>> {
    let latest = find_latest_checkpoint(checkpoint_dir, "my_evolution");

    let (mut population, start_gen) = if let Some(path) = latest {
        println!("Resuming from {:?}", path);
        let ckpt: Checkpoint<RealVector> = load_checkpoint(&path)?;
        let pop = reconstruct_population(ckpt.population);
        (pop, ckpt.generation)
    } else {
        println!("Starting fresh");
        let pop = Population::random(100, &bounds, &mut rng);
        (pop, 0)
    };

    // Continue evolution from start_gen
    for gen in start_gen..max_generations {
        // ... evolution ...
    }

    Ok(())
}

Saving Algorithm State

For algorithms with internal state (like CMA-ES):

#[derive(Serialize, Deserialize)]
struct CmaEsCheckpoint {
    generation: usize,
    mean: Vec<f64>,
    sigma: f64,
    covariance: Vec<Vec<f64>>,
    // ... other CMA-ES state
}

impl CmaEsCheckpoint {
    fn from_cmaes(cmaes: &CmaEs) -> Self {
        Self {
            generation: cmaes.state.generation,
            mean: cmaes.state.mean.clone(),
            sigma: cmaes.state.sigma,
            covariance: cmaes.state.covariance.clone(),
        }
    }

    fn restore(&self) -> CmaEs {
        let mut cmaes = CmaEs::new(self.mean.clone(), self.sigma);
        cmaes.state.generation = self.generation;
        cmaes.state.covariance = self.covariance.clone();
        cmaes
    }
}

Error Handling

match load_checkpoint::<RealVector>(&path) {
    Ok(checkpoint) => {
        println!("Loaded successfully");
    }
    Err(CheckpointError::FileNotFound(path)) => {
        println!("Checkpoint not found: {:?}", path);
    }
    Err(CheckpointError::DeserializationFailed(err)) => {
        println!("Corrupted checkpoint: {}", err);
    }
    Err(e) => {
        println!("Unknown error: {}", e);
    }
}

Best Practices

1. Checkpoint Frequently Enough

// For long runs, checkpoint every ~5-10% of expected runtime
let interval = max_generations / 20;
manager.every(interval);

2. Verify Checkpoints

// After saving, verify it can be loaded
manager.save(&checkpoint)?;
let verified: Checkpoint<RealVector> = load_checkpoint(&manager.latest_path())?;
assert_eq!(verified.generation, checkpoint.generation);

3. Include Random State

For reproducible resumption, save RNG state:

use rand::SeedableRng;

#[derive(Serialize, Deserialize)]
struct FullCheckpoint<G> {
    evolution: Checkpoint<G>,
    rng_seed: u64, // Or full RNG state
}

4. Use Atomic Writes

Prevent corruption from interrupted saves:

// Write to temp file, then rename
let temp_path = path.with_extension("tmp");
write_checkpoint(&checkpoint, &temp_path)?;
std::fs::rename(temp_path, path)?;

Feature Flag

Checkpointing requires the checkpoint feature:

[dependencies]
fugue-evo = { version = "0.1", features = ["checkpoint"] }

Next Steps