Skip to main content

Blockchain Data Backfilling – Streams

Updated on
Jul 20, 2026

Backfilling is the process of retrieving historical blockchain data to populate databases, analyze past trends, or audit transactions. Whether you need to index an entire chain from the Genesis block or just catch up on the last 24 hours of activity, retrieving historical data efficiently is a critical infrastructure challenge.

Streams makes backfilling simple with one-click templates, server-side filtering, batching and compression options, and guaranteed delivery to your preferred destination. Instead of writing complex scripts to poll RPC endpoints, you configure a Stream, set your block range, optionally apply filters to extract only the data you need, and Streams pushes the data to your destination (e.g., Webhook, S3, PostgreSQL, Azure Storage, etc.) reliably and at scale.

Why Use Streams for Backfilling?​

Streams handles the infrastructure side such as retries, block ordering, and error handling, letting you focus exclusively on your application's data logic.

FeatureDescription
FiltersYou can process and transform data server-side before delivery using filters. Use decodeEVMReceipts for hex decoding, shape custom payloads, and debug with console.log().
Batching & CompressionYou can set dataset_batch_size (e.g., 10-100) to batch multiple blocks per request. You can also enable compression: "gzip" in destination config.
Key-Value StoreYou can store watchlists, ABIs, or config values and access them in filters via qnLib methods. You can also manage values via REST API.
REST APIYou can programmatically create, update, pause, and delete Streams using the REST API.
MultichainEach Stream targets one chain/network. You can run multiple Streams in parallel for multi-chain backfills. See Supported Chains.
Real-Time TransitionYou can omit end_range to continue streaming after backfill completes. You can use elastic_batch_enabled to auto-reduce batch size at tip and keep_distance_from_tip to reduce reorg frequency.

Estimating Backfill Costs​

Backfilling historical blockchain data consumes API credits based on the number of blocks processed, using network dataset multipliers. Streams uses the same shared API credit pool as RPC.

Use the API Credits Calculator to estimate your backfill costs before starting. The calculator shows:


  • Total API credits needed for backfilling available historical blocks
  • Credits per block based on your selected dataset and network
  • Which networks support backfilling (Solana networks, for example, do not currently support backfill)

How to Backfill Data​


  1. Select your chain and network. See the supported chains for Streams.

  2. Define your block range. You can choose to start from the Genesis block or a specific block height. For the end block, you can set it to a specific height or choose continuous streaming to keep receiving new blocks.

  3. Select your dataset. Streams provide different datasets such as Block, Block with Receipts, Transactions, Logs, etc. Choose the one that fits your use case. See Backfill Data by Ecosystem below for chain-specific datasets.

  4. Apply filters (optional). Use server-side filtering to narrow down the data you want to receive.

  5. Choose your destination. Configure where you want the data to be sent, such as Webhooks, S3, PostgreSQL, etc. Check out the Destinations documentation for more details.

  6. Check the connection and send a test payload in the Stream configuration page to ensure everything is set up correctly.

  7. Start the Stream.

Not sure which settings to choose?

See our Tips for Backfilling below for recommendations on batching, compression, and performance optimization.

Backfill Data by Ecosystem​

Streams can be used to backfill data for any ecosystem, including Ethereum, Bitcoin, Solana, and more. Since they have different data structures and formats, select your ecosystem below for chain-specific datasets, example filters, and responses.


Ethereum and EVM Chains​

Streams support many EVM chains, including Ethereum, Base, Arbitrum, and BNB Smart Chain. All EVM chains share a similar architecture and data structure, so filters and payload formats are generally the same across all chains, while some chain-specific differences may exist.

Decoding EVM Data​

When working with EVM-compatible chains, you can verify and parse data more easily using the decodeEVMReceipts function. This utility transforms raw hex data into human-readable formats by taking raw transaction receipts and your contract ABIs as inputs.

The decoding process automatically:


  • Matches event signatures in transaction logs with the provided ABIs.
  • Decodes parameters according to their types (addresses, integers, strings, etc.).
  • Returns structured data with named parameters in a decodedLogs object.

Learn more about this function: Decoding EVM Data

Available Data Sources​

For EVM chains such as Ethereum, Base, Arbitrum, and BNB Smart Chain, you can use the following datasets to backfill data.

For the full JSON specification and details of each dataset, please refer to the Data Sources documentation.

Data SourceDescription
BlockAn array of block objects as returned by eth_getBlockByNumber
Block with ReceiptsAn array of objects containing a composite dataset with block and receipts as returned by eth_getBlockByNumber and eth_getBlockReceipts
TransactionsAn array of arrays of transaction objects, as they appear in the transactions array of block data
LogsAn array of arrays of log objects, as they appear within the logs array in transaction receipts
ReceiptsAn array of arrays, each containing receipt objects as returned by eth_getBlockReceipts
Traces (debug_trace)An array of arrays of trace data as returned by debug_traceBlock
Traces (trace_block)An array of arrays of trace data as returned by trace_block
Block with Receipts + debug_traceAn array of objects containing a composite dataset with block, receipts, and traces from debug_traceBlock
Block with Receipts + trace_blockAn array of objects containing a composite dataset with block, receipts, and traces from trace_block
note

Data sources availability varies by chain. Some datasets (specifically Traces) may not be supported on all EVM networks. Check the Data Sources page for the current support matrix.

Example: ERC-20 Token Transfers​

The sample function below filters a block of transactions to identify and decode standard ERC-20 token transfers by checking the input data for the transfer method signature.


// Chain: Ethereum
// Dataset: Transactions
// Test with block: 23977403

type Payload struct {
    Data     [][]Tx                 `json:"data"`
    Metadata map[string]interface{} `json:"metadata"`
}
type Tx struct {
    Hash        string `json:"hash"`
    From        string `json:"from"`
    To          string `json:"to"`
    Input       string `json:"input"`
    BlockNumber string `json:"blockNumber"`
}

func Filter(qn *qn.QNContext, payload Payload) interface{} {
    // The standard ERC-20 transfer(address,uint256) method signature
    const transferMethodID = "0xa9059cbb"

    var filteredTransactions []map[string]interface{}

    // Loop through all blocks in the batch (data is [][]tx: outer = block, inner = txs)
    for _, transactions := range payload.Data {
        for _, tx := range transactions {
            // Ensure the input carries a full transfer(address,uint256) call:
            // "0x" + 4-byte selector + two 32-byte words = 138 hex chars.
            if !strings.HasPrefix(tx.Input, transferMethodID) || len(tx.Input) < 138 {
                continue
            }

            // Decode the 'to' address (skip selector + left-padding)
            toAddress := "0x" + tx.Input[34:74]
            // Decode the 'value' (amount) from the trailing 32-byte word.
            // uint256 can exceed int64, so convert hex → decimal without math/big.
            amount := hexToDecimal(tx.Input[74:])

            filteredTransactions = append(filteredTransactions, map[string]interface{}{
                "txHash":        tx.Hash,
                "fromAddress":   tx.From,
                "toAddress":     toAddress,
                "amount":        amount,
                "tokenContract": tx.To,
                "blockNumber":   tx.BlockNumber,
            })
        }
    }

    // Return nil if no transfers were found — skips delivery to your destination
    // (API credits are still consumed based on blocks processed).
    if len(filteredTransactions) == 0 {
        return nil
    }
    return map[string]interface{}{"transactions": filteredTransactions}
}

// hexToDecimal converts an arbitrary-length hex string (with or without a "0x"
// prefix) to its decimal string form, safe for full uint256 values.
func hexToDecimal(hexStr string) string {
    hexStr = strings.TrimPrefix(strings.ToLower(hexStr), "0x")
    hexStr = strings.TrimLeft(hexStr, "0")
    if hexStr == "" {
        return "0"
    }

    // digits holds the decimal value little-endian (least-significant first).
    digits := []int{0}
    for _, c := range hexStr {
        v, err := strconv.ParseInt(string(c), 16, 32)
        if err != nil {
            return "0"
        }
        carry := int(v)
        for i := 0; i < len(digits); i++ {
            cur := digits[i]*16 + carry
            digits[i] = cur % 10
            carry = cur / 10
        }
        for carry > 0 {
            digits = append(digits, carry%10)
            carry /= 10
        }
    }

    var sb strings.Builder
    for i := len(digits) - 1; i >= 0; i-- {
        sb.WriteByte(byte('0' + digits[i]))
    }
    return sb.String()
}

Additional Resources​

Tips for Backfilling​

Batching for Faster Backfills​

By default, Streams delivers one block at a time. For historical backfills, increasing the batch size (e.g., 10 or 100 blocks per delivery) reduces overhead and speeds up ingestion. Adjust this in your Stream settings based on your destination's capacity.

Consider using Elastic Batch for automatic batch size adjustment if you continue to real-time streaming.

Performance Optimization​

Combining Filtering (to stream only the data you want) with Compression (to optimize data transfer) improves performance during large-scale backfills.


tip

If your filter returns null, no data is delivered to your destination (reducing unnecessary payloads). Note: Streams consumes API credits based on blocks processed, regardless of filtering.

Debugging Filters​

Log from your filter function to debug during development. Logs appear in the Logs tab next to the Results tab in the Stream Filter Editor.

In JavaScript, use console.log(). In Go, use fmt.Println — its output is captured in the Logs tab as an info-level log, and it formats objects (maps, slices, structs) into a string automatically, so you can log the payload directly.

The Go example below assumes a Payload struct is declared in your filter (see the examples above). Referencing an undefined type in the Filter signature causes the interpreter to report a misleading constant definition loop error, so make sure the struct is defined.


func Filter(qn *qn.QNContext, payload Payload) interface{} {
    fmt.Println("Data:", payload.Data)
    fmt.Println("Metadata:", payload.Metadata)

    // ... rest of your filter
}

Testing with Raw Data​

When writing a new filter, you can download the raw data for a specific test block from the Stream configuration UI. This helps you understand the exact payload structure before writing filter logic.

Security and Authentication​

Streams includes a security token with each delivery that you can use to verify requests originated from Quicknode. You can also configure custom headers in your destination settings. For webhook destinations, see How to Validate Incoming Streams Webhook Messages for implementation details.


Last updated on
Share this doc