28 min read

Overview

Every test a developer writes starts with a sequence they imagined. The bugs that reach mainnet are in the sequences they did not. Each instruction may succeed on its own, but the order is what breaks the program.

No test suite covers that by hand. A program with 20 instructions has 3.2 million possible five-step orderings before a single amount or account varies. Fuzz testing turns the job around. Instead of writing out sequences and checking what each one returns, the developer writes down the rule the program must never break, and the fuzzer goes looking for a sequence that breaks it.

Crucible is an open-source, coverage-guided invariant fuzzer for Solana programs. It takes the actions a user can perform and the properties that must always hold, then generates millions of action sequences, watches which code paths each one reaches so it can steer toward logic nothing has exercised yet, and reports any sequence that breaks a property, reduced to the shortest version that still does.

info

Crucible was created and is maintained by Asymmetric Research and licensed under MIT. This guide uses version 0.3.0-alpha.1.

Crucible is also slated to ship inside the Anchor CLI in Anchor 2.0, where the same workflow runs as anchor fuzz init, anchor fuzz run, and anchor fuzz show with no separate install. That work lives in the anchor-fuzzing fork today. This guide uses the standalone crucible CLI, which works on any Anchor version and on non-Anchor programs.

This guide builds a complete harness for an escrow program with a seeded boundary bug, finds the bug, minimizes it to the shortest sequence that still reproduces it, and verifies the fix.

TL;DR

• Crucible is a coverage-guided invariant fuzzer for Solana programs. It works on any compiled .so, with or without Anchor.
• A harness has three parts: a fixture with setup(), actions the fuzzer sequences and mutates, and an invariant checked after every action with fuzz_assert_*! macros.
• This guide fuzzes a timelock escrow with an off-by-one slot check. The fuzzer finds it in under a second and crucible tmin strips the crash down to the few actions that matter.
• Stateful mode keeps a pool of live program states so deep sequences are reached once and reused, and it ran at roughly 66,000 single-action iterations per second on four cores in this guide.

What You Will Do

• Install the Crucible CLI and build a timelock escrow program with a seeded boundary bug
• Walk through a fuzz harness with a fixture, four actions, and one invariant
• Run the fuzzer and read its coverage and finding output
• Minimize and replay the crash down to the actions that matter
• Fix the bug, verify with a clean run, and switch to stateful mode for higher throughput
• Scaffold a harness for your own Anchor program with crucible init

What You Will Need

This guide assumes familiarity with Rust and with writing or testing an Anchor program. Prerequisites:

• Experience writing Solana Programs with Anchor
• Familiarity with LiteSVM, the in-process Solana Virtual Machine (SVM) that Crucible executes transactions in
• An understanding of fuzz testing, covered in How to Fuzz Test Programs with Trident
• A Quicknode account (optional) for cloning mainnet-beta accounts into a harness, covered in the last section

The following tools must be installed. Platform-tools v1.52 is a hard minimum, because earlier versions ship a rustc that cannot build Crucible's dependency tree. It needs no separate install: cargo build-sbf --tools-version v1.52 downloads it on first use.

Dependency	Version
Solana CLI	4.2.2
platform-tools	v1.52
Rust	1.89.0
crucible-fuzz-cli	0.3.0-alpha.1
Anchor CLI (own-program section only)	1.0.0

What Is Invariant Fuzzing?

Invariant fuzzing is a testing technique where the developer states properties that must always hold for a program (a vault can never pay out more than it holds, a pool's shares can never outnumber the tokens backing them) and lets a fuzzer search for an input sequence that breaks one. The developer writes the property once. The fuzzer generates the sequences.

Coverage-guided fuzzing is a technique where the fuzzer records which code paths each input reaches and prioritizes inputs that reach paths nothing has hit before. Crucible adds that guidance to the search. Every transaction runs inside LiteSVM with bytecode tracing turned on. Solana programs compile to Solana BPF (sBPF) bytecode, and the trace lets Crucible see every branch the program took. Each one of those branches is called an edge, and Crucible's goal is to reach edges it has not reached before.

Crucible keeps a sequence that reaches a new edge in the corpus (the saved set of inputs worth mutating further) and discards a sequence that reaches nothing new. Over time the corpus fills with inputs that reach deeper and rarer code paths, the ones a sequence bug depends on.

How Crucible Works

A Crucible test has three parts:

1. Setup: A fixture's setup() method loads the program, creates accounts, and initializes state. Crucible runs it once, snapshots the result, and clones the snapshot for each iteration.
2. Actions: Methods on the fixture prefixed with action_. Each usually wraps one transaction. Crucible discovers them automatically and treats their parameters as typed fuzz inputs, bounded by optional #[range(..)] attributes.
3. Invariants: A function marked #[invariant_test] that runs after every action in a sequence. Crucible records any fuzz_assert_*! that fails as a crash, along with the exact action sequence that produced it.

Between runs, Crucible changes the order of the harness's actions and the values passed to them. It reorders, adds, and drops actions to build new sequences, and it favors the values that break programs, such as zero, the maximum, and the edges of any declared range. Because it works with the harness's actions instead of random bytes, every sequence it builds is one the program could actually receive.

Stateless vs. Stateful Mode

Crucible has two execution modes that share one harness. Stateless mode, the default, clones the post-setup snapshot and runs a full mutated action sequence on every iteration. Stateful mode keeps a pool of live program states, picks one, applies a single mutated action, and saves any state that reaches new code back into the pool.

	Stateless (default)	Stateful (--stateful)
Per iteration	Clone the post-setup snapshot, run a full mutated sequence	Pick a live state from a pool, apply one mutated action
Coverage feedback	Per sequence	Per state; states that reach new code are saved back into the pool
Reaching deep states	Re-executes the whole chain every iteration	Resumes where prior iterations left off
Throughput	Thousands of sequences per second	Tens of thousands of single actions per second; one iteration is one action, not a full sequence
Memory	Flat	Grows with the pool as coverage grows
Limits	8 actions per sequence (--max-actions)	Chain depth (--max-depth), pool size 256,000 (--pool-size)

Stateless mode suits harness development, because its output is easier to read. Stateful mode suits long runs.

Crucible vs. Trident

Crucible vs. Trident comes down to guidance strategy and program support. Trident is the other widely used Solana fuzzer, and it makes different design choices.

	Crucible	Trident
Guidance	Coverage-guided via sBPF edge tracing	Manually guided: you define instruction flows and input ranges
Program support	Any compiled program, including native and Pinocchio programs; Interface Definition Language (IDL) optional	Anchor programs with an IDL
Type bindings	Anchor, Codama, or Shank IDL via declare_fuzz_program!, or the program crate directly, or raw_call()	Generated from the Anchor IDL by trident init
Mutation	Reorders your actions and favors edge-case values	Random selection from developer-defined flows and ranges
Invariant checks	#[invariant_test] runs after every action	assert! inside flows or in the #[end] method
Execution modes	Stateless and stateful (state pool)	Iteration loop with #[init], #[flow], and #[end]
Crash tooling	crucible show, --replay, crucible tmin minimization	Crash seed replay with trident fuzz debug, metrics dashboard
Maintainer	Asymmetric Research	Ackee Blockchain Security

Choose Crucible when you want coverage feedback to drive the search, need to fuzz a non-Anchor or native program, or plan long stateful runs. Choose Trident when you want to hand-specify instruction flows for an Anchor program and prefer its metrics dashboard. Both run inside an in-process SVM, both let you write invariants, and both replay crashes.

Install Crucible CLI

Crucible ships as a Cargo workspace. Clone the repository and install the CLI crate. The same clone contains the escrow example used in the rest of this guide, so keep it.

git clone https://github.com/asymmetric-research/crucible
cd crucible
cargo install --path crates/crucible-fuzz-cli

Verify the installation:

crucible --help

Expected output:

Solana smart contract fuzzing framework

Usage: crucible [OPTIONS] <COMMAND>

Commands:
  init  Create a new fuzz harness for a program
  run   Run a fuzz test
  list  List available fuzz tests
  show  View/replay crashes
  tmin  Minimize a crash to smallest reproducing action sequence
  cmin  Minimize corpus to smallest set preserving coverage
  help  Print this message or the help of the given subcommand(s)

Options:
  -C, --harness-dir <HARNESS_DIR>  Use this directory instead of ./fuzz/<program_name>/
  -h, --help                       Print help

The Escrow Program

The program and its harness both ship inside the Crucible repository you just cloned, at examples/escrow. It is a timelock escrow written in Anchor. One vault Program Derived Address (PDA) per depositor holds lamports, and three instructions move them:

• deposit: The depositor adds lamports to their vault.
• withdraw: The depositor takes lamports back, allowed only before unlock_slot.
• claim: The beneficiary takes everything, allowed only at or after unlock_slot.

A withdraw and a claim should never both be allowed at the same slot. The program ships with a seeded bug in withdraw. The slot check uses <= instead of <, so at exactly slot == unlock_slot both the depositor and the beneficiary can drain the vault, and whoever lands first wins. The relevant part of withdraw:

examples/escrow/programs/escrow/src/lib.rs

pub fn withdraw(ctx: Context<Withdraw>, amount: u64) -> Result<()> {
    let clock = Clock::get()?;
    let vault = &mut ctx.accounts.vault;
    // BUG: should be `<` (strictly before unlock). Using `<=` lets the depositor
    // drain the vault at the exact unlock slot, racing the beneficiary's claim.
    require!(
        clock.slot <= vault.unlock_slot,
        EscrowError::AlreadyUnlocked
    );
    require!(
        amount > 0 && amount <= vault.amount,
        EscrowError::InvalidAmount
    );
    // ... transfer lamports back to the depositor
    Ok(())
}

A unit test that checks "withdraw fails after unlock" passes with this code because it tests unlock_slot + 1. The bug only shows up at one exact slot, and only after a deposit. Hitting it by hand means guessing that exact slot. Crucible gets there by trying many slot values and keeping the ones that reach code it has not run yet.

Build the Program

From the examples/escrow directory, build the program with cargo build-sbf. Crucible fuzzes the compiled .so, so this step produces the artifact the harness loads.

cd examples/escrow
cargo build-sbf --tools-version v1.52 --manifest-path programs/escrow/Cargo.toml

Confirm the binary exists:

ls target/deploy/escrow.so

Any time you change the program, rebuild before fuzzing again. The harness loads whatever binary is at that path.

Anatomy of a Crucible Harness

The harness is a standalone Cargo package in fuzz/escrow/. Keeping it out of the program's workspace avoids Solana dependency conflicts between the program and the fuzzer.

fuzz/escrow/
├── Cargo.toml        # Standalone workspace, one [[bin]], one feature per test
├── src/main.rs       # Fixture, actions, and invariant
└── crashes/          # Written by the fuzzer
    └── invariant_escrow/
        ├── crash_<id>            # Raw input
        └── crash_<id>.meta.json  # Decoded action sequence

Configure Cargo.toml

Each fuzz test needs its own Cargo feature, and the test name you pass to crucible run must match that feature name exactly. This example depends on the Crucible crates by path because it lives inside the Crucible repository, and on the escrow program crate directly so it can reuse the program's own instruction and accounts types.

fuzz/escrow/Cargo.toml

[package]
name = "escrow_fuzz"
version = "0.1.0"
edition = "2021"
rust-version = "1.89"

[workspace]

[dependencies]
crucible-fuzzer = { path = "../../../../crates/crucible-fuzzer" }
crucible-test-context = { path = "../../../../crates/crucible-test-context" }
anchor-lang = "1.0.1"
# Required by the #[fuzz_fixture] and #[invariant_test] macros and the LibAFL
# runner they expand into. Crucible is built on LibAFL, the Rust fuzzing library.
arbitrary = { version = "1", features = ["derive"] }
ctrlc = "3.4"
libafl = { version = "0.15.1", features = ["std", "cli", "prelude"] }
libafl_bolts = { version = "0.15.1", features = ["std"] }
escrow = { path = "../../programs/escrow", features = ["no-entrypoint"] }
solana-keypair = "3.0"
solana-pubkey = "3.0"
solana-signer = "3.0"

[[bin]]
name = "invariant_test"
path = "src/main.rs"

[features]
invariant_escrow = []

When you fuzz your own program, crucible init generates this file with published crate versions instead of path dependencies. The Fuzz Your Own Program section covers that workflow.

The harness itself is a single file. The next four sections walk through src/main.rs in order: the fixture struct, setup(), the actions, and the invariant. Together they are the complete file.

Define the Fixture

The fixture holds everything an action or invariant needs: the TestContext that wraps LiteSVM, keypairs, derived addresses, and any bookkeeping the invariant reads. It must derive Clone because Crucible clones it for every iteration. Wrap keypairs in Rc to keep that clone cheap.

fuzz/escrow/src/main.rs

use crucible_fuzzer::anchor_lang::system_program;
use crucible_fuzzer::*;
use escrow::*;
use solana_keypair::Keypair;
use solana_pubkey::Pubkey;
use solana_signer::Signer;
use std::rc::Rc;

const INITIAL_BALANCE: u64 = 10_000_000_000;
const UNLOCK_DELAY: u64 = 10;

#[derive(Clone)]
struct EscrowFixture {
    ctx: TestContext,
    program_id: Pubkey,
    depositor: Rc<Keypair>,
    beneficiary: Rc<Keypair>,
    vault_pda: Pubkey,
    unlock_slot: u64,
    // Bookkeeping the invariant reads: the slot at which each successful
    // withdraw was submitted.
    successful_withdraw_slots: Vec<u64>,
}

Write the Setup Method

#[fuzz_fixture] marks the impl block that Crucible scans. Inside it, setup() is required. It runs once, and Crucible snapshots the resulting state. Here it loads the compiled program, funds two accounts, derives the vault PDA the same way the program does, and calls initialize with an unlock slot set 10 slots ahead.

The ctx.program(..).call(..).accounts(..).signers(..).send() chain is the TestContext builder. It assembles the instruction, signs it, submits it, and returns the result as a TxOutcome. Every call here uses unwrap() so that a broken setup crashes immediately instead of producing a harness that tests nothing.

fuzz/escrow/src/main.rs

#[fuzz_fixture]
impl EscrowFixture {
    pub fn setup() -> Self {
        let mut ctx = TestContext::new();
        let program_id = Pubkey::new_from_array(ID.to_bytes());
        ctx.add_program(&program_id, "../../target/deploy/escrow.so")
            .unwrap();

        let depositor = Rc::new(Keypair::new());
        let beneficiary = Rc::new(Keypair::new());
        for kp in [&depositor, &beneficiary] {
            ctx.create_account()
                .pubkey(kp.pubkey())
                .lamports(INITIAL_BALANCE)
                .owner(system_program::ID)
                .create()
                .unwrap();
        }

        let (vault_pda, _) = Pubkey::find_program_address(
            &[b"vault", depositor.pubkey().as_ref()],
            &program_id,
        );
        let unlock_slot = ctx.slot() + UNLOCK_DELAY;

        ctx.program(program_id)
            .call(instruction::Initialize {
                beneficiary: beneficiary.pubkey(),
                unlock_slot,
            })
            .accounts(accounts::Initialize {
                vault: vault_pda,
                depositor: depositor.pubkey(),
                system_program: system_program::ID,
            })
            .signers(&[&*depositor])
            .send()
            .unwrap();

        Self {
            ctx,
            program_id,
            depositor,
            beneficiary,
            vault_pda,
            unlock_slot,
            successful_withdraw_slots: Vec::new(),
        }
    }

Write the Actions

Any public method on the fixture whose name starts with action_ becomes a fuzzable action. Its parameters become fuzz inputs. A #[range(start..end)] attribute bounds a parameter, and Crucible favors the endpoints of that range.

An action's return value feeds the ok: statistic and the -> OK markers in crash traces. Returning bool is the simplest choice. Actions can also return () for implicit success or Result<()> for ?-style error propagation. Actions should never panic on a failed transaction, because failures are expected. The fuzzer is supposed to try things that do not work.

The fourth action controls time. warp_to_slot moves LiteSVM's clock so the fuzzer can reach and cross unlock_slot.

fuzz/escrow/src/main.rs

    pub fn action_deposit(&mut self, #[range(1..1_000_000)] amount: u64) -> bool {
        self.ctx
            .program(self.program_id)
            .call(instruction::Deposit { amount })
            .accounts(accounts::Deposit {
                vault: self.vault_pda,
                depositor: self.depositor.pubkey(),
                system_program: system_program::ID,
            })
            .signers(&[&*self.depositor])
            .send()
            .map(|o| o.is_success())
            .unwrap_or(false)
    }

    pub fn action_withdraw(&mut self, #[range(1..1_000_000)] amount: u64) -> bool {
        let pre_slot = self.ctx.slot();
        let success = self
            .ctx
            .program(self.program_id)
            .call(instruction::Withdraw { amount })
            .accounts(accounts::Withdraw {
                vault: self.vault_pda,
                depositor: self.depositor.pubkey(),
            })
            .signers(&[&*self.depositor])
            .send()
            .map(|o| o.is_success())
            .unwrap_or(false);
        // Record the slot of every withdraw the program accepted.
        if success {
            self.successful_withdraw_slots.push(pre_slot);
        }
        success
    }

    pub fn action_claim(&mut self) -> bool {
        self.ctx
            .program(self.program_id)
            .call(instruction::Claim {})
            .accounts(accounts::Claim {
                vault: self.vault_pda,
                depositor: self.depositor.pubkey(),
                beneficiary: self.beneficiary.pubkey(),
            })
            .signers(&[&*self.beneficiary])
            .send()
            .map(|o| o.is_success())
            .unwrap_or(false)
    }

    pub fn action_advance_slots(&mut self, #[range(0..15)] slots: u64) -> bool {
        let new_slot = self.ctx.slot() + slots;
        self.ctx.warp_to_slot(new_slot);
        true
    }
}

Write the Invariant

The #[invariant_test] function runs after every action in a sequence. It receives the fixture and checks one thing. Every withdraw the program accepted must have happened before unlock_slot, never on it.

Use the fuzz_assert_*! macros, never plain assert!. A plain assert! panics and kills the fuzzer process. The fuzz_assert_*! macros record the violation, save the crash file, and let the run continue.

Available macros:

Macro	Asserts
fuzz_assert!(cond)	cond is true
fuzz_assert_eq!(a, b)	a equals b
fuzz_assert_ne!(a, b)	a does not equal b
fuzz_assert_lt!(a, b)	a is less than b
fuzz_assert_le!(a, b)	a is less than or equal to b
fuzz_assert_gt!(a, b)	a is greater than b
fuzz_assert_ge!(a, b)	a is greater than or equal to b
fuzz_assert_approx_eq!(a, b, delta)	a and b differ by no more than delta

fuzz/escrow/src/main.rs

// Time-guard invariant: every successful withdraw must have happened strictly
// before unlock. The seeded bug uses `slot <= unlock_slot`, which lets a
// withdraw succeed at the exact unlock slot.
#[invariant_test]
fn invariant_escrow(fixture: &mut EscrowFixture) {
    for &s in &fixture.successful_withdraw_slots {
        fuzz_assert_lt!(
            s,
            fixture.unlock_slot,
            "withdraw at slot {} should have been rejected (unlock_slot = {})",
            s,
            fixture.unlock_slot
        );
    }
}

Notice that the check never recalculates the unlock slot. It compares against the value set in setup(). Copying the program's own math into an invariant is a common mistake, because then a bug in that math shows up on both sides of the comparison and the check passes no matter what.

Run the Fuzzer

From examples/escrow, run the invariant_escrow test. The first run compiles the harness, which takes a few minutes. Use --release for anything beyond a smoke test. --stop-on-crash exits at the first violation so the output stays short.

crucible run escrow invariant_escrow --release --stop-on-crash

Expected output:

[FUZZ_PULSE] run time: 0s, clients: 1, corpus: 25, crashes: 0, executions: 288, exec/sec: 0.000, edges: 419/3512 (11.9%), branches: 390/1756 (22.2%), actions/exec: 4.6, ok: 788/1337 (58.9%), discovered: 4/4 actions
[FUZZ_FINDING] crash:crash_7ec16ec2b3199e23 summary:withdraw at slot 10 should have been rejected (unlock_slot = 10)

=== FUZZ SEQUENCE (5 executed, 3 skipped) ===
  1. deposit(amount=610171) -> OK
  2. advance_slots(slots=0) -> OK
  3. advance_slots(slots=0) -> OK
  4. advance_slots(slots=10) -> OK
  5. withdraw(amount=257) -> OK [VIOLATION]
  ... 3 action(s) not executed (stopped on violation)
================================
[FUZZ] First crash found. Exiting (--stop-on-crash).

The bug surfaced in under a second, before the first one-second pulse had a rate to report. Your run will show a different sequence, different amounts, and a different crash ID, because every run draws new random values. Crucible prints a [FUZZ_PULSE] line about once a second. Its fields tell you whether the harness is working:

• corpus: Inputs kept because they reached new coverage. It should grow early and plateau later.
• edges: Unique sBPF edges hit out of the program's total. The absolute percentage stays low because much of the .so is Anchor error handling and runtime code the harness never reaches, so watch whether the number grows and then plateaus rather than the raw value. A number that never moves means actions are failing before they reach program logic.
• actions/exec: Average sequence length.
• ok: Successful actions over total. A ratio near zero means a blocker in setup or account wiring.
• discovered: How many of the harness's actions the fuzzer has exercised at least once.

The [FUZZ_FINDING] line names the crash, and the sequence below it shows exactly how the fuzzer got there: deposit, warp forward 10 slots, withdraw. The declared #[range(0..15)] allows a 10-slot hop, which is what lands the clock on unlock_slot.

Minimize and Replay the Crash

Crash artifacts land in fuzz/escrow/crashes/invariant_escrow/. List them with crucible show:

crucible show escrow

=== Crashes for escrow (1 total) ===

  1. crash_7ec16ec2b3199e23 (2026-09-11T14:29:44Z, test: invariant_escrow, 5 actions)

The recorded sequence includes noise. Two advance_slots(slots=0) calls do nothing. crucible tmin removes every action that is not needed to reproduce the violation and overwrites the crash file in place.

crucible tmin escrow invariant_escrow crash_7ec16ec2b3199e23 --release

[TMIN] Minimizing: crash_7ec16ec2b3199e23

[TMIN] [1/1] crash_7ec16ec2b3199e23
[TMIN] 8 actions → 3 actions (5 removed, 2 passes)

[TMIN] Done. 1 crashes in 0.0s (860.0/s)

show listed the crash as 5 actions while tmin starts from 8. show counts the actions that ran. tmin works on the full recorded input, including the three that never ran because the fuzzer stopped on the violation.

View the minimized sequence:

crucible show escrow crash_7ec16ec2b3199e23

=== Crash: crash_7ec16ec2b3199e23 ===
Test: invariant_escrow
Timestamp: 2026-09-11T14:29:59Z
Iteration: 0

=== Action Sequence (3 actions) ===
  1. deposit(amount=610171) -> OK
  2. advance_slots(slots=10) -> OK
  3. withdraw(amount=257) -> OK
================================

Three actions is the smallest reproducer for this run. Yours may minimize to four if the fuzzer reached slot 10 in two hops, such as two advance_slots(slots=5) calls, since both are needed. Add --replay to re-execute the crash against the current binary and confirm it still reproduces:

crucible show escrow crash_7ec16ec2b3199e23 --replay

[REPLAY] Input size: 34 bytes
[FUZZ_FINDING] crash:crash_ace9da4ff38e51d2 reproduces:true summary:withdraw at slot 10 should have been rejected (unlock_slot = 10)

=== FUZZ SEQUENCE (3 executed, 0 skipped) ===
  1. deposit(amount=610171) -> OK
  2. advance_slots(slots=10) -> OK
  3. withdraw(amount=257) -> OK [VIOLATION]
================================
[REPLAY] SUCCESS: Crash reproduced!

Replay writes its own finding under a fresh ID, which is why crash_ace9da4ff38e51d2 differs from the crash you asked for. reproduces:true is the field that matters.

The companion crash_<id>.meta.json file holds the same sequence as JSON, with each action's name, parameters, and success flag, so you can turn a finding into a regression test.

Fix the Bug and Verify

Open programs/escrow/src/lib.rs and change the withdraw slot check from <= to <:

examples/escrow/programs/escrow/src/lib.rs

    require!(
        clock.slot < vault.unlock_slot,
        EscrowError::AlreadyUnlocked
    );

Rebuild the program, clear the old crash artifacts, and run the fuzzer for a fixed 20 seconds:

cargo build-sbf --tools-version v1.52 --manifest-path programs/escrow/Cargo.toml
rm -rf fuzz/escrow/crashes
crucible run escrow invariant_escrow --release --timeout 20

[FUZZ_PULSE] run time: 15s, clients: 1, corpus: 26, crashes: 0, executions: 54862, exec/sec: 3.656k, edges: 419/3512 (11.9%), branches: 390/1756 (22.2%), actions/exec: 4.9, ok: 208065/268283 (77.6%), discovered: 4/4 actions
[FUZZ] Timeout reached (20s). Exiting gracefully.

The run reports crashes: 0 at roughly 3,600 sequences per second on a single core. About 22% of actions now fail, which is correct, because the program rejects withdraws at or after the unlock slot and claims before it. Coverage reached the same 419 edges as the buggy build, so the fuzzer still exercises the withdraw path.

Run in Stateful Mode

Stateless mode re-executes a full sequence from the setup snapshot on every iteration. For the escrow, that is cheap, but a lending protocol harness that needs a dozen setup actions before anything interesting happens pays that cost every time. Stateful mode keeps a pool of saved program states, picks one, and applies a single new action to it. States that reach new code go back in the pool.

Change the withdraw check back to <= so there is a bug to find, rebuild, and run stateful on four cores for 15 seconds:

cargo build-sbf --tools-version v1.52 --manifest-path programs/escrow/Cargo.toml
crucible run escrow invariant_escrow --release --stateful --cores 4 --timeout 15

[FUZZ_PULSE] [00:14] iter: 926630, iter/sec: 66223, pool: 117/250k (0.0%), crashes: 16, ok: 704330/909561 (77.4%), discovered: 4/4 actions, edges: 419/3512 (11.9%), branches: 389/1756, workers: 4
[STATEFUL] Timeout reached (15s). Exiting.

Roughly 66,000 iterations per second across four workers on the machine used for this guide, against about 3,600 sequences per second in single-core stateless mode. Those are different units. A stateful iteration is one action, while a stateless execution is a sequence of about five, so the raw gain is smaller than the two numbers suggest. The real win is that deep states are reached once and reused instead of replayed from the snapshot every time. Your numbers will differ. The pool field shows how many distinct program states are live. Crashes in stateful mode record the full action chain from the root state to the violation, so crucible tmin and crucible show work the same way.

Fuzz Your Own Program

The escrow harness borrowed the program crate for its instruction types. For your own Anchor program, crucible init scaffolds a harness that instead generates types from the IDL at compile time, so the harness never depends on the program crate or its Solana version.

From your Anchor project root, build the program so the IDL and binary exist, then scaffold:

anchor build
crucible init my_program

This creates fuzz/my_program/ with a Cargo.toml pinned to the CLI's crate versions, a rust-toolchain.toml, an idls/ directory, and a src/main.rs skeleton. Then copy your IDL into idls/. Anchor 0.30 and later IDLs work as they are. Older ones need anchor idl convert first.

cp target/idl/my_program.json fuzz/my_program/idls/my_program.json
# Anchor 0.29 or earlier:
# anchor idl convert target/idl/my_program.json -o fuzz/my_program/idls/my_program.json

The generated Cargo.toml declares the published Crucible crates and one feature per test (other dependencies trimmed here):

fuzz/my_program/Cargo.toml

[dependencies]
crucible-fuzzer = "0.3.0-alpha.1"
crucible-test-context = "0.3.0-alpha.1"
crucible-idl-gen = "0.3.0-alpha.1"
anchor-lang = "1.0.1"
# ... solana-*, libafl, and utility crates

[[bin]]
name = "invariant_test"
path = "src/main.rs"

[features]
invariant_test = []

The generated main.rs uses declare_fuzz_program! to turn the IDL into instruction::*, accounts::*, state::*, and an ID constant. Fill in the TODO blocks with the same three parts you saw in the escrow harness.

fuzz/my_program/src/main.rs

use crucible_fuzzer::*;
use anchor_lang::prelude::*;
use solana_keypair::Keypair;
use solana_signer::Signer;
use solana_pubkey::Pubkey;
use anchor_lang::system_program;
use std::rc::Rc;

// Generate types from IDL (no crate dependency - avoids version conflicts)
crucible_idl_gen::declare_fuzz_program!("idls/my_program.json");

use my_program::instruction;
use my_program::accounts;

#[derive(Clone)]
struct MyProgram {
    ctx: TestContext,
    program_id: Pubkey,
    admin: Rc<Keypair>,
    // TODO: Add your state here (users, accounts, etc.)
}

#[fuzz_fixture]
impl MyProgram {
    /// Called ONCE to setup initial state (programs + accounts)
    pub fn setup() -> Self {
        let mut ctx = TestContext::new();
        let program_id = my_program::ID;

        // Load program binary (built separately from fuzz harness)
        ctx.add_program(&program_id, "../../target/deploy/my_program.so").unwrap();

        // Create admin account
        let admin = Rc::new(Keypair::new());
        ctx.create_account()
            .pubkey(admin.pubkey())
            .lamports(100_000_000_000)
            .owner(system_program::ID)
            .create()
            .unwrap();

        // TODO: Initialize your program state here

        Self { ctx, program_id, admin }
    }

    /// ACTIONS - Define actions that the fuzzer can call
    pub fn action_noop(&mut self) {
        // Placeholder - replace with real actions
    }

    // TODO: Add your actions here
}

#[invariant_test]
fn invariant_test(fixture: &mut MyProgram) {
    // TODO: Add invariant checks that should hold after every action
}

Run it the same way, with a short timeout while you iterate:

crucible run my_program invariant_test --release --timeout 10

With only action_noop in place, this reports edges: 0 and stops finding new inputs after a handful of executions. That is expected. Coverage starts moving once you add real actions.

A few practices from Asymmetric Research's harness guide:

• Let setup crash on the first problem. Print the outcome of each setup transaction and panic on failure, as the escrow setup() does with unwrap().
• Return early when an action cannot work. If borrow cannot succeed without collateral, check the state and return before sending the transaction. Guaranteed failures waste iterations and flatten the ok ratio.
• Read onchain state in invariants. Deserialize accounts with ctx.read_anchor_account::<T>(&pubkey) and assert relationships between them. Local counters drift out of sync with account data.
• Watch edges and repeated errors. Coverage that stops growing while the same Custom(6xxx) error repeats means an action is blocked by a config limit, a missing account, or a PDA seed encoding that differs from what you assumed. The IDL does not tell you whether a seed is encoded as bytes or as a string. Read the program source.
• Work without an IDL. Build the Instruction yourself and submit it with ctx.raw_call(instruction).signers(..).send(). Coverage tracing works the same, because it comes from the sBPF trace, not from the IDL.

Beyond the Basics

Three flags go further than a plain run.

Probe for Missing Account Checks

--mutate-accounts turns on Crucible's constraint-check engine, which probes for the account constraints a program forgot to enforce. After an instruction succeeds, the engine replays it with one account deliberately wrong (a different owner, a cleared signer, a spoofed PDA, a token account whose mint does not match the mint passed in, or the same address passed twice) and reports a labeled finding if the transaction still succeeds.

Run it after coverage stops growing, because it slows the fuzzer down. Run it against the fixed build. On the buggy build, every result is the seeded invariant violation and any probe finding is buried under hundreds of those.

The stateful section left the bug in place and wrote its crashes to disk, so change the withdraw check back to <, rebuild, and clear the old crash artifacts first. Otherwise crucible show lists the stateful run's invariant violations alongside any probe findings, and the two look identical.

cargo build-sbf --tools-version v1.52 --manifest-path programs/escrow/Cargo.toml
rm -rf fuzz/escrow/crashes
crucible run escrow invariant_escrow --release --mutate-accounts --timeout 60

Findings are ordinary crashes, so crucible show and crucible tmin work on them. Expect some triage. An owner check on an account nobody can tamper with, or a swap between two accounts the program is happy to treat as interchangeable, will show up as a finding. Dismiss those by hand.

Generate Coverage Reports

--coverage writes an LCOV file (the line-coverage format that tools such as genhtml read) to ./coverage/coverage.lcov. Without debug symbols the line numbers are sBPF program counters, which is enough to track growth over time. The recommended flow is to fuzz at full speed into a corpus first, then replay the corpus once with coverage enabled:

crucible run escrow invariant_escrow --release --timeout 300 --corpus-out ./corpus
crucible run escrow invariant_escrow --release --coverage --corpus-in ./corpus

The second command replays the saved inputs once, reports the same edge count the fuzzer reached, and writes the LCOV file. For source-level reports that highlight untested Rust lines, rebuild the program with opt-level = 1, debug = 2, and strip = false under [profile.release], then pass the unstripped binary to the replay with --symbols. Passing --symbols against the default build prints a warning that the binary has no DWARF debug symbols and falls back to bytecode-level output. The Crucible coverage guide has the full recipe, including the platform-tools version it needs.

Fork Mainnet Accounts into a Harness

Real bugs often depend on real state: a lending market's actual reserve configuration, an oracle account, a pool with millions in liquidity. TestContext::clone_from_rpc pulls live accounts from any Solana RPC endpoint into the local SVM during setup(), caches them on disk under .fuzz-cache/accounts/, and serves later runs from the cache without network calls.

Enable the rpc-clone feature on crucible-test-context in the harness Cargo.toml:

fuzz/<program>/Cargo.toml

crucible-test-context = { version = "0.3.0-alpha.1", features = ["rpc-clone"] }

The feature pulls in solana-rpc-client, and a fresh resolve of that tree picks solana-reward-info 6.3.0, which moved to wincode 0.6 while the Solana RPC client crates still derive against wincode 0.5. Pin it back until they catch up:

cargo update -p solana-reward-info --precise 6.2.0

For the escrow harness in this guide, delete the harness lockfile and regenerate it before pinning. The committed fuzz/escrow/Cargo.lock holds spl-token-group-interface at 0.7.1 for anchor-spl 1.0.1, and the solana-account-decoder that arrives with the feature requires 0.7.2. A fresh resolve bumps anchor-spl to 1.2.0 and satisfies both.

cd fuzz/escrow
rm Cargo.lock
cargo generate-lockfile
cargo update -p solana-reward-info --precise 6.2.0

Create a Solana mainnet-beta endpoint from the Quicknode dashboard and export it as an environment variable so it never lands in source control:

export QUICKNODE_ENDPOINT=https://your-endpoint-name.solana-mainnet.quiknode.pro/your-token/

The harness reads that variable during setup. This fragment goes at the top of setup(), right after the existing let mut ctx = TestContext::new(); line. Because setup() returns Self, it uses expect rather than ?. Replace program_id, reserve_a, reserve_b, and oracle with the addresses your program uses.

// In setup(), after `let mut ctx = TestContext::new();` and before creating
// your own accounts.
let rpc_url = std::env::var("QUICKNODE_ENDPOINT")
    .expect("set QUICKNODE_ENDPOINT to a Solana mainnet-beta RPC endpoint");

let mut cloner = ctx.clone_from_rpc(&rpc_url);

// Clone the deployed program (handles the BPF Upgradeable Loader accounts)
cloner.clone_account(&program_id).expect("clone program");

// Batch clone the accounts your actions touch: PDAs, token accounts, oracles
cloner
    .clone_accounts(&[reserve_a, reserve_b, oracle])
    .expect("clone accounts");

Call .force_refresh() on the cloner when you want current data instead of the cached copy, and add .fuzz-cache/ to .gitignore.

Frequently Asked Questions

What is Crucible?

Crucible is an open-source, coverage-guided invariant fuzzing framework for Solana programs built by Asymmetric Research on LibAFL and LiteSVM. You define a fixture with setup, actions the fuzzer can call, and invariants that must hold after every action. Crucible mutates typed action sequences, tracks sBPF edge coverage from the compiled program without instrumentation, and saves any sequence that violates an invariant as a minimizable, replayable crash.

Does Crucible require Anchor?

No. Crucible fuzzes any compiled Solana program binary, including native and Pinocchio programs, because coverage comes from tracing sBPF execution rather than from source instrumentation. Typed instruction bindings can be generated from an Anchor, Codama, or Shank IDL with declare_fuzz_program!, taken from the program crate directly, or skipped entirely by building an Instruction by hand and submitting it with raw_call().

How is Crucible different from Trident?

Crucible is coverage-guided. It keeps action sequences that reach new sBPF edges and mutates them further, and it supports stateless and stateful execution modes. Trident is manually guided, so you define instruction flows and input ranges, and it randomly selects among them each iteration. Crucible works on any compiled program with an optional IDL; Trident targets Anchor programs and derives its types from the Anchor IDL. Both let you write invariants and reproduce failures, and both run against an in-process SVM rather than a validator.

When should I use stateful mode instead of stateless mode?

Use stateless mode (the default) while writing and debugging a harness, because each iteration is a complete sequence from a fresh snapshot and the output is easy to read. Switch to stateful mode with --stateful for long runs, or for programs that need many setup actions before anything interesting happens. Stateful mode keeps a pool of saved program states and applies one new action to one of them per iteration, which reached roughly 66,000 single-action iterations per second on four cores in this guide and reuses deep states instead of replaying them. Memory use grows with the pool, and --pool-size caps it.

Can I fuzz against real mainnet state?

Yes. Enable the rpc-clone feature on crucible-test-context and call ctx.clone_from_rpc(url) in setup() with a Solana RPC endpoint such as a Quicknode mainnet endpoint. Clone the program plus the PDAs, token accounts, and oracles your actions touch. Accounts are cached in .fuzz-cache/accounts/ after the first run, so fuzzing itself never makes RPC calls, and .force_refresh() refetches when you need current state.

How do I run Crucible in continuous integration (CI)?

Build the program, then run crucible run <program> <test> --release --timeout <seconds> with a fixed budget, optionally adding --seed for a reproducible run and --stop-on-crash to exit early. The run command returns exit code 0 whether or not it found a violation, so fail the job by checking whether the crashes/<test>/ directory contains any files afterward. Commit the crash .meta.json files from real findings and replay them with crucible show <program> <crash_id> --replay as regression tests.

Wrapping Up

You installed Crucible, read a complete harness for a timelock escrow, and watched the fuzzer find an off-by-one slot check in a Solana program in under a second that a conventional test would have missed. You minimized the finding to its shortest reproducer, replayed it, fixed the program, and confirmed a clean run, then saw stateful mode push the same harness past 66,000 iterations per second. You also have the scaffold and the checklist for pointing Crucible at your own program.

The pattern is the core of Solana fuzz testing and carries over to any program. Write down what must always be true, give the fuzzer the actions a user could take, and let it search the orderings you cannot enumerate by hand. Start with the property your users already trust without checking, such as a vault never paying out more than it holds.

Resources

• Crucible GitHub Repository
• Introducing Crucible: An Invariant Fuzzing Framework for Solana
• Crucible Product Page
• Crucible Documentation: Writing Tests
• Crucible Documentation: Harness Guide
• Crucible Documentation: CLI Reference
• How to Test Solana Programs with LiteSVM
• How to Fuzz Test Programs with Trident
• Sealevel Attacks: Common Security Exploits and Protections