Skip to main content

Blockchain Data Backfilling

Updated on
Dec 17, 2025

Overview​

Backfilling is the process of retrieving historical blockchain data to populate databases, analyze past trends, or audit transactions. Whether you need to index an entire chain from the Genesis block or just catch up on the last 24 hours of activity, retrieving historical data efficiently is a critical infrastructure challenge.

Streams makes backfilling simple with one-click templates, server-side filtering, batching and compression options, and guaranteed delivery to your preferred destination. Instead of writing complex scripts to poll RPC endpoints, you configure a Stream, set your block range, optionally apply filters to extract only the data you need, and Streams pushes the data to your destination (e.g., Webhook, S3, PostgreSQL, Azure Storage, etc.) reliably and at scale.

Why Use Streams for Backfilling?​

Streams handles the infrastructure side such as retries, block ordering, and error handling, letting you focus exclusively on your application’s data logic.

Server-Side Filtering and Transformation​

Filtering allows you to process data before it leaves Quicknode, ensuring you only receive the data you need.

  • Custom Payloads: Beyond simple filtering, you can modify the data structure to match your schema. For example, you can return only transaction hashes instead of full transaction objects, or use built-in helper functions like decodeEVMReceipts to transform raw hex data into human-readable formats.

  • Templates: Jumpstart your logic with pre-built templates for common use cases available directly in the Stream filter editor. You can also use one-click templates to quickly create a filter for a specific use case.

  • Rapid Development: Use console.log() for debugging, and download raw data for specific blocks to understand payload structures while building your filter logic.

  • Key-Value Store Integration: Use the Key-Value Store to store and manage extensive lists or key-value sets, then reference them within Streams filters to efficiently evaluate, update, or match against blockchain data. You can also retrieve stored values via REST for use outside of Streams.

Optimized Data Delivery (Batching & Compression)​

While optional, enabling batching and compression is highly recommended for backfills to improve speed and reduce costs.

  • Batching: Instead of one HTTP request per block, Streams can send multiple blocks in a single payload. This reduces HTTP overhead and significantly increases throughput.

  • Compression: You can enable Gzip compression for your payloads. This lowers bandwidth usage, which directly reduces costs, and speeds up data transfer for data-heavy blocks.

Learn how to reduce costs with Streams compression: Reduce Streams Costs with Data Compression

Scalability & Automation​

Backfilling often requires indexing multiple chains simultaneously. While a single Stream targets one chain, the architecture is designed to scale horizontally.

  • Multichain Support: You can run multiple Streams to backfill ranges of historical blocks on Ethereum, Solana, Bitcoin, or any other supported chains.

  • Automation: You can use the Quicknode Streams REST API to create and manage Streams programmatically or duplicate an existing Stream in the UI.

Transition to Real-Time​

Streams can also be used to retrieve real-time data after the backfill is complete. You can configure your Stream to start from a specific block range and then continue streaming new blocks as they are produced, ensuring you always have up-to-date data without needing to switch tools or create a new Stream.

To ensure a smooth transition from historical indexing to live monitoring, Streams offers several advanced features:

  • Elastic Batch: When enabled along with batching, it automatically reduces the batch size to 1 to prioritize low latency as you approach the latest block.

  • Latest Block (Slot) Delay: Once synced with the tip, you can set a delay (e.g., 2 blocks/slots) to reduce reorg risks for real-time data.

  • Reorg Handling: In addition to the above, Streams can automatically detects chain reorganizations and sends correction payloads, ensuring your database remains consistent with the canonical chain.

How to Backfill Data with Streams​


  1. Select your chain and network. See the supported chains for Streams.

  2. Define your block range. You can choose to start from the Genesis block or a specific block height. For the end block, you can set it to a specific height or choose continuous streaming to keep receiving new blocks.

  3. Select your dataset. Streams provide different datasets such as Block, Block with Receipts, Transactions, Logs, etc. Choose the one that fits your use case. See Backfill Data by Ecosystem below for chain-specific datasets.

  4. Apply filters (optional). Use server-side filtering to narrow down the data you want to receive.

  5. Choose your destination. Configure where you want the data to be sent, such as Webhooks, S3, PostgreSQL, etc. Check out the Streams Destinations documentation for more details.

  6. Check the connection and send a test payload in the Stream configuration page to ensure everything is set up correctly.

  7. Start the Stream.

Note: Not sure which settings to choose? See our Tips for Backfilling with Streams below for our tips and recommendations.

Backfill Data by Ecosystem​

Streams can be used to backfill data for any ecosystem, including Ethereum, Bitcoin, Solana, and more. Since they might have different data structures and formats, here are some examples of how you can use Streams to backfill data for different ecosystems.


Backfill Data for Ethereum and EVM Chains​

Streams support many EVM chains, including Ethereum, Base, Arbitrum, and BNB Smart Chain. All EVM chains share a similar architecture and data structure, so filters and payload formats are generally the same across all chains, while some chain-specific differences may exist.

Decoding EVM Data​

When working with EVM-compatible chains, you can verify and parse data more easily using the decodeEVMReceipts function. This utility transforms raw hex data into human-readable formats by taking raw transaction receipts and your contract ABIs as inputs.

The decoding process automatically:

  • Matches event signatures in transaction logs with the provided ABIs.
  • Decodes parameters according to their types (addresses, integers, strings, etc.).
  • Returns structured data with named parameters in a decodedLogs object.

Learn more about this function: Decoding EVM Data Filters

Available Data Sources​

For EVM chains such as Ethereum, Base, Arbitrum, and BNB Smart Chain, you can use the following datasets to backfill data.

For the full JSON specification and details of each dataset, please refer to the Streams Data Sources documentation.

Data SourceDescription
BlockAn array of block objects as returned by eth_getBlockByNumber.
Block with ReceiptsAn array of objects containing a composite dataset with block and receipts as returned by eth_getBlockByNumber and eth_getBlockReceipts.
TransactionsAn array of arrays of transaction objects, as they appear in the transactions array of block data.
LogsAn array of arrays of log objects, as they appear within the logs array in transaction receipts.
ReceiptsAn array of arrays, each containing receipt objects as returned by eth_getBlockReceipts.
Traces (debug_trace)An array of arrays of trace data as returned by debug_traceBlock.
Traces (trace_block)An array of arrays of trace data as returned by trace_block.
Block with Receipts + debug_traceAn array of objects containing a composite dataset with block, receipts, and traces from debug_traceBlock.
Block with Receipts + trace_blockAn array of objects containing a composite dataset with block, receipts, and traces from trace_block.

Note: Data sources availability varies by chain. Some datasets (specifically Traces) may not be supported on all EVM networks. Check the Streams Data Sources page for the current support matrix.

Example Filter and Response​

The sample function below filters a block of transactions to identify and decode standard ERC-20 token transfers by checking the input data for the transfer method signature.

Example filter function:

// Chain: Ethereum
// Dataset: Transactions
// Test with block: 23977403

function main(payload) {
    const filteredTransactions = [];

    // The standard ERC-20 transfer(address,uint256) method signature
    const transferMethodId = "0xa9059cbb";

    // Loop through all blocks in the batch
    for (const transactions of payload.data) {
        for (const transaction of transactions) {
            // Ensure the transaction object and input data are valid
            if (typeof transaction === 'object' && transaction !== null && typeof transaction.input === 'string') {

                // Check if the transaction input starts with the transfer method ID
                if (transaction.input.startsWith(transferMethodId)) {

                    // Decode the 'to' address (skipping method ID and padding)
                    const toAddress = "0x" + transaction.input.substr(34, 40);
                    // Decode the 'value' (amount) from the subsequent hex data
                    const value = BigInt("0x" + transaction.input.substr(74));

                    filteredTransactions.push({
                        txHash: transaction.hash,
                        fromAddress: transaction.from,
                        toAddress: toAddress,
                        amount: value.toString(),
                        tokenContract: transaction.to,
                        blockNumber: transaction.blockNumber,
                    });
                }
            }
        }
    }

    // Return the filtered list, or null if no transfers were found (to save costs)
    return filteredTransactions.length > 0 ? { transactions: filteredTransactions } : null;
}

Example response:

{
  "transactions": [
    {
      "amount": "80007920",
      "blockNumber": "0x16dddbb",
      "fromAddress": "0xa80f9793051cd1f428ad61b276d431f30dd59b6a",
      "toAddress": "0xabd22d07c199a56bafa5e2add3cde1127bc98292",
      "tokenContract": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48",
      "txHash": "0x013a9cdd1de2ade97a1d79178fe59f2025e41495db050aab0cc706cd2be3b2fe"
    },
    {
      "amount": "71839000",
      "blockNumber": "0x16dddbb",
      "fromAddress": "0x828ee64b59f33e6c3a6b8d4ad8298aeb65421445",
      "toAddress": "0xbb03a0b5159d985c304ab183b80e884ae38c9c89",
      "tokenContract": "0xdac17f958d2ee523a2206206994597c13d831ec7",
      "txHash": "0x2ae18c8918de4744df1d93729e75f89f402723630982224654d18cae047ec74b"
    },
    // ...
    ]
}

Additional Resources​


Tips for Backfilling with Streams​

Debugging Filters​

Use console.log() in your filter function to debug during development. Logs appear in the Logs tab next to the Results tab in the Stream Filter Editor.

function main(payload) {
  const {
    data,
    metadata,
  } = payload;

  console.log("Data:", data)
  console.log("Metadata:", metadata)

  // ... rest of your filter
}

Testing with Raw Data​

When writing a new filter, you can download the raw data for a specific test block from the Stream configuration UI. This helps you understand the exact payload structure before writing filter logic.

Batching for Faster Backfills​

By default, Streams delivers one block at a time. For historical backfills, increasing the batch size (e.g., 10 or 100 blocks per delivery) reduces overhead and speeds up ingestion. Adjust this in your Stream settings based on your destination's capacity.

Consider using Elastic Batch for automatic batch size adjustment if you continue to real-time streaming.

Cost Optimization​

Combining Filtering (to remove unwanted data) with Compression (to shrink payload size) is the most effective way to manage costs during large-scale backfills.

Additional Resources​


TIP: If your filter returns null, you are not charged for that block.

Security and Authentication​

When backfilling, ensure your destination endpoints are secured. This protects your data during transit and prevents unauthorized access. Streams supports custom headers and includes a security token with each delivery that you can use to verify the request originated from Quicknode.

Additional Resources​



Last updated on
Share this doc