メインコンテンツへスキップ

Blockchain Data Backfilling

更新日：
2026年6月4日

概要​

Backfilling is the process of retrieving historical blockchain data to populate databases, analyze past trends, or audit transactions. Whether you need to index an entire chain from the Genesis block or just catch up on the last 24 hours of activity, retrieving historical data efficiently is a critical infrastructure challenge.

Streams makes backfilling simple with one-click templates, server-side filtering, batching and compression options, and guaranteed delivery to your preferred destination. Instead of writing complex scripts to poll RPC endpoints, you configure a Stream, set your block range, optionally apply filters to extract only the data you need, and Streams pushes the data to your destination (e.g., Webhook, S3, PostgreSQL, Azure Storage, etc.) reliably and at scale.

Why Use Streams for Backfilling?​

Streams handles the infrastructure side such as retries, block ordering, and error handling, letting you focus exclusively on your application's data logic.

Estimating Backfill Costs​

Backfilling historical blockchain data consumes API credits based on the number of blocks processed, using network dataset multipliers. Streams uses the same shared API credit pool as RPC.

Use the API Credits Calculator to estimate your backfill costs before starting. The calculator shows:

  • Total API credits needed for backfilling available historical blocks
  • Credits per block based on your selected dataset and network
  • Which networks support backfilling (Solana networks, for example, do not currently support backfill)

Server-Side Filtering and Transformation​

Filtering allows you to process data before it leaves Quicknode, ensuring you only receive the data you need.

  • Custom Payloads: Beyond simple filtering, you can modify the data structure to match your schema. For example, you can return only transaction hashes instead of full transaction objects, or use built-in helper functions like decodeEVMReceipts to transform raw hex data into human-readable formats.

  • Templates: Jumpstart your logic with pre-built templates for common use cases available directly in the Stream filter editor. You can also use one-click templates to quickly create a filter for a specific use case.

  • Rapid Development: Use console.log() for debugging, and download raw data for specific blocks to understand payload structures while building your filter logic.

  • Key-Value Store Integration: Use the Key-Value Store to store and manage extensive lists or key-value sets, then reference them within Streams filters to efficiently evaluate, update, or match against blockchain data. You can also retrieve stored values via REST for use outside of Streams.

Optimized Data Delivery (Batching & Compression)​

While optional, enabling batching and compression is highly recommended for backfills to improve speed and reduce bandwidth usage for your destination systems.

  • Batching: Instead of one HTTP request per block, Streams can send multiple blocks in a single payload. This reduces HTTP overhead and significantly increases throughput.

  • Compression: You can enable Gzip compression for your payloads. This lowers bandwidth usage for your destination systems and speeds up data transfer for data-heavy blocks.

Scalability & Automation​

Backfilling often requires indexing multiple chains simultaneously. While a single Stream targets one chain, the architecture is designed to scale horizontally.

  • Multichain Support: You can run multiple Streams to backfill ranges of historical blocks on Ethereum, Solana, Bitcoin, or any other supported chains.

  • Automation: You can use the Quicknode Streams REST API to create and manage Streams programmatically or duplicate an existing Stream in the UI.

Transition to Real-Time​

Streams can also be used to retrieve real-time data after the backfill is complete. You can configure your Stream to start from a specific block range and then continue streaming new blocks as they are produced, ensuring you always have up-to-date data without needing to switch tools or create a new Stream.

To ensure a smooth transition from historical indexing to live monitoring, Streams offers several advanced features:

  • Elastic Batch: When enabled along with batching, it automatically reduces the batch size to 1 to prioritize low latency as you approach the latest block.

  • Latest Block (Slot) Delay: Once synced with the tip, you can set a delay (e.g., 2 blocks/slots) to reduce reorg risks for real-time data.

  • Reorg Handling: In addition to the above, Streams can automatically detect chain reorganizations and sends correction payloads, ensuring your database remains consistent with the canonical chain.

How to Backfill Data with Streams​


  1. Select your chain and network. See the supported chains for Streams.

  2. Define your block range. You can choose to start from the Genesis block or a specific block height. For the end block, you can set it to a specific height or choose continuous streaming to keep receiving new blocks.

  3. Select your dataset. Streams provide different datasets such as Block, Block with Receipts, Transactions, Logs, etc. Choose the one that fits your use case. See Backfill Data by Ecosystem below for chain-specific datasets.

  4. Apply filters (optional). Use server-side filtering to narrow down the data you want to receive.

  5. Choose your destination. Configure where you want the data to be sent, such as Webhooks, S3, PostgreSQL, etc. Check out the Streams Destinations documentation for more details.

  6. Check the connection and send a test payload in the Stream configuration page to ensure everything is set up correctly.

  7. Start the Stream.

Note: Not sure which settings to choose? See our Tips for Backfilling with Streams below for our tips and recommendations.

Backfill Data by Ecosystem​

Streams can be used to backfill data for any ecosystem, including Ethereum, Bitcoin, Solana, and more. Since they might have different data structures and formats, here are some examples of how you can use Streams to backfill data for different ecosystems.


Backfill Data for Ethereum and EVM Chains​

Streams support many EVM chains, including Ethereum, Base, Arbitrum, and BNB Smart Chain. All EVM chains share a similar architecture and data structure, so filters and payload formats are generally the same across all chains, while some chain-specific differences may exist.

Decoding EVM Data​

When working with EVM-compatible chains, you can verify and parse data more easily using the decodeEVMReceipts function. This utility transforms raw hex data into human-readable formats by taking raw transaction receipts and your contract ABIs as inputs.

The decoding process automatically:

  • Matches event signatures in transaction logs with the provided ABIs.
  • Decodes parameters according to their types (addresses, integers, strings, etc.).
  • Returns structured data with named parameters in a decodedLogs object.

Learn more about this function: Decoding EVM Data Filters

Available Data Sources​

For EVM chains such as Ethereum, Base, Arbitrum, and BNB Smart Chain, you can use the following datasets to backfill data.

For the full JSON specification and details of each dataset, please refer to the Streams Data Sources documentation.

データソース説明
ブロックAn 配列 of block objects as returned by eth_getBlockByNumber.
Block with ReceiptsAn 配列 of objects containing a composite dataset with ブロック そして receipts as returned by eth_getBlockByNumber そして eth_getBlockReceipts.
取引An 配列 of arrays of transaction objects, as they appear in the 取引 array of block data.
ログAn 配列 of arrays of log objects, as they appear within the ログ array in transaction receipts.
ReceiptsAn 配列 of arrays, each containing receipt objects as returned by eth_getBlockReceipts.
Traces (debug_trace)An 配列 of arrays of trace data as returned by debug_traceBlock.
Traces (trace_block)An 配列 of arrays of trace data as returned by trace_block.
Block with Receipts + debug_traceAn 配列 of objects containing a composite dataset with ブロック, receipts、および traces 出典： debug_traceBlock.
Block with Receipts + trace_blockAn 配列 of objects containing a composite dataset with ブロック, receipts、および traces 出典： trace_block.

Note: Data sources availability varies by chain. Some datasets (specifically Traces) may not be supported on all EVM networks. Check the Streams Data Sources page for the current support matrix.

Example Filter and Response​

The sample function below filters a block of transactions to identify and decode standard ERC-20 token transfers by checking the input data for the 転送 method signature.

Example filter function:

// Chain: Ethereum
// Dataset: Transactions
// Test with block: 23977403

function main(payload) {
    const filteredTransactions = [];

    // The standard ERC-20 transfer(address,uint256) method signature
    const transferMethodId = "0xa9059cbb";

    // Loop through all blocks in the batch
    for (const transactions of payload.data) {
        for (const transaction of transactions) {
            // Ensure the transaction object and input data are valid
            if (typeof transaction === 'object' && transaction !== null && typeof transaction.input === 'string') {

                // Check if the transaction input starts with the transfer method ID
                if (transaction.input.startsWith(transferMethodId)) {

                    // Decode the 'to' address (skipping method ID and padding)
                    const toAddress = "0x" + transaction.input.substr(34, 40);
                    // Decode the 'value' (amount) from the subsequent hex data
                    const value = BigInt("0x" + transaction.input.substr(74));

                    filteredTransactions.push({
                        txHash: transaction.hash,
                        fromAddress: transaction.from,
                        toAddress: toAddress,
                        amount: value.toString(),
                        tokenContract: transaction.to,
                        blockNumber: transaction.blockNumber,
                    });
                }
            }
        }
    }

    // Return the filtered list, or null if no transfers were found (skips delivery to your destination; API credits are still consumed based on blocks processed)
    return filteredTransactions.length > 0 ? { transactions: filteredTransactions } : null;
}

Example response:

{
  "transactions": [
    {
      "amount": "80007920",
      "blockNumber": "0x16dddbb",
      "fromAddress": "0xa80f9793051cd1f428ad61b276d431f30dd59b6a",
      "toAddress": "0xabd22d07c199a56bafa5e2add3cde1127bc98292",
      "tokenContract": "0xa0b86991c6218b36c1d19d4a2e9eb0ce3606eb48",
      "txHash": "0x013a9cdd1de2ade97a1d79178fe59f2025e41495db050aab0cc706cd2be3b2fe"
    },
    {
      "amount": "71839000",
      "blockNumber": "0x16dddbb",
      "fromAddress": "0x828ee64b59f33e6c3a6b8d4ad8298aeb65421445",
      "toAddress": "0xbb03a0b5159d985c304ab183b80e884ae38c9c89",
      "tokenContract": "0xdac17f958d2ee523a2206206994597c13d831ec7",
      "txHash": "0x2ae18c8918de4744df1d93729e75f89f402723630982224654d18cae047ec74b"
    },
    // ...
    ]
}

その他の参考資料​


Tips for Backfilling with Streams​

Debugging Filters​

使用方法 console.log() in your filter function to debug during development. Logs appear in the ログ tab next to the 結果 tab in the Stream Filter Editor.

function main(payload) {
  const {
    data,
    metadata,
  } = payload;

  console.log("Data:", data)
  console.log("Metadata:", metadata)

  // ... rest of your filter
}

Testing with Raw Data​

When writing a new filter, you can download the raw data for a specific test block from the Stream configuration UI. This helps you understand the exact payload structure before writing filter logic.

Batching for Faster Backfills​

By default, Streams delivers one block at a time. For historical backfills, increasing the batch size (e.g., 10 or 100 blocks per delivery) reduces overhead and speeds up ingestion. Adjust this in your Stream settings based on your destination's capacity.

Consider using Elastic Batch for automatic batch size adjustment if you continue to real-time streaming.

パフォーマンスの最適化​

Combining Filtering (to stream only the data you want) with Compression (to optimize data transfer) improves performance during large-scale backfills.

その他の参考資料​


TIP: If your filter returns null, no data is delivered to your destination (reducing unnecessary payloads). Note: Streams consumes API credits based on blocks processed, regardless of filtering.

Security and Authentication​

When backfilling, ensure your destination endpoints are secured. This protects your data during transit and prevents unauthorized access. Streams supports custom headers and includes a security token with each delivery that you can use to verify the request originated from Quicknode.

その他の参考資料​



最終更新日 on
このドキュメントを共有する