Filters enable you to customize and filter your streams payload using JavaScript (ECMAScript) code. With the help of filters, you can match specific patterns and rules against the data from your stream, and personalize the data sent to your destination. This feature is in beta and can be configured within the QuickNode developer portal, and through the Streams REST API.

Understanding Filters

In Streams, a filter_function is an optional configuration option for your stream. This function must be named main and it modifies the data out of the stream before it's sent to its destination. Using filters, you can precisely control the data you receive, ensuring you only pay for and process the data you need.

Benefits of Filters

• Cost Efficiency: Only receive and pay for the data you actually need, minimizing unnecessary data transmission and costs.
• Customizability: Implement custom filter functions to match specific patterns or criteria, offering great flexibility in data handling.
• Enhanced Data Relevance: Filters ensure that you receive data that is directly relevant to your needs, increasing the overall usefulness of the data.
• Modify the Payload Before Streaming: Customize the payload from your stream before it is sent to your Streams destination.

Example Filter Functions

Below are examples of how you might define a filter function to target data from Streams.

Please note:

• Your filter must be named main.
• Your filter function must return an object.
• Your filter can access streamData, streamName, streamRegion, streamNetwork, and streamDataset.

Return Hash and Block Number

This filter works with the block dataset.

function main(data) {
    var numberDecimal = parseInt(data.streamData.number, 16);
    var filteredData = {
        hash: data.streamData.hash,
        number: numberDecimal
    };
    return filteredData;
}

Get Receipts for ERC20 Transfers

This filter works with the block_with_receipts dataset.

function main(data) {
    try {
        var data = data.streamData;
        var filteredReceipts = [];

        data.receipts.forEach(receipt => {
            let relevantLogs = receipt.logs.filter(log =>
                log.topics[0] === "0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef" &&
                log.topics.length === 3
            );
            if (relevantLogs.length > 0) {
                filteredReceipts.push(receipt);
            }
        });

        return {
            totalReceipts: data.receipts.length,
            filteredCount: filteredReceipts.length,
            receipts: filteredReceipts
        };
    } catch (e) {
        return {error: e.message};
    }
}

Get Transactions and Receipts for Specific Addresses

This filter works with the block_with_receipts dataset.

function main(data) {
    try {
        var data = data.streamData;

        const addresses = [
            "0x56220b7e25c7d0885159915cdebf5819f2090f57",
            "0x351e1b4079cf180971025a3b35dadea1d809de26",
            "0xa61551e4e455edebaa7c59f006a1d2956d46eecc"
        ];
        var addressSet = new Set(addresses.map(address => address.toLowerCase()));
        var paddedAddressSet = new Set(addresses.map(address => '0x' + address.toLowerCase().slice(2).padStart(64, '0')));

        var matchingTransactions = [];
        var matchingReceipts = [];

        data.block.transactions.forEach(transaction => {
            let transactionMatches = (transaction.from && addressSet.has(transaction.from.toLowerCase())) || 
                        (transaction.to && addressSet.has(transaction.to.toLowerCase()));

            if (transactionMatches) {
                matchingTransactions.push(transaction);
            }
        });

        data.receipts.forEach(receipt => {
            let receiptMatches = receipt.logs && receipt.logs.some(log =>
                log.topics && log.topics.length > 1 && 
                (paddedAddressSet.has(log.topics[1]) || (log.topics.length > 2 && paddedAddressSet.has(log.topics[2])))
            );
            if (receiptMatches) {
                matchingReceipts.push(receipt);
            }
        });

        if (matchingTransactions.length === 0 && matchingReceipts.length === 0) {
            return null;
        }

        return {
            transactions: matchingTransactions,
            receipts: matchingReceipts
        };
    } catch (e) {
        return {error: e.message};
    }
}

How to Use Filters

To apply filters to your stream, specify the filter_function configuration option when creating a stream using the Streams REST API and in the "Stream payload" step of the Streams configuration wizard. You can then define your custom filter function to match specific criteria or patterns according to your requirements.

To omit sending payloads when your filter doesn't match data within a specific block, you can add conditional logic to return null instead of an empty result.

Using Key-Value Store with Streams filters

Key-Value Store can be accessed and managed seamlessly within a Streams filter. You can create new lists and key-value sets, add and remove items from your lists, update key-value sets, retrieve set value, and check if data from the streaming dataset matches an item on your list. To learn more about Key-Value Store, please visit the Key-Value Store documentation

Available Key-Value Store functions inside your Streams filter

Lists

• qnUpsertList: Creates or updates a new list in Key-Value Store.
• qnGetList: Retrieves all item from a specific list in Key-Value Store.
• qnGetAllLists: Retrieves all lists in Key-Value Store.
• qnAddListItem: Adds an item to a specific list in Key-Value Store.
• qnRemoveListItem: Removes an item from a specific list in Key-Value Store.
• qnContainsListItem: Checks if an item exists in a specific list in Key-Value Store.
• qnDeleteList: Deletes a specific list in Key-Value Store.

Example filter code for lists

The filter code below can be used within Streams to demonstrate ways that Key-Value Store lists can be used within your Streams filter. You can deploy this example with one click here.

function main() {
    // List of results for each operation
    let results = {};

    try {
        // Create a new list
        results.createList = qnUpsertList("list_docs_example", {
            add_items: ["item1", "item2"],
        });

        // Update a list
        results.qnUpsertList = qnUpsertList("list_docs_example", {
            add_items: ["item3"],
            remove_items: ["item1"],
        });

        // Get a list
        results.qnGetList = qnGetList("list_docs_example");

        // Get all lists
        results.qnGetAllLists = qnGetAllLists();

        // Add item to the list
        results.qnAddListItem4 = qnAddListItem("list_docs_example", "item4");
        results.qnAddListItem5 = qnAddListItem("list_docs_example", "item5");

        // Remove item from the list
        results.qnRemoveListItem4 = qnRemoveListItem("list_docs_example", "item4");

        // Get a list after changes
        results.qnGetListAfterChanges = qnGetList("list_docs_example");

        // Check if an item is in the list
        results.qnContainsListItem2 = qnContainsListItem("list_docs_example", "item2");
        results.qnContainsListItem1 = qnContainsListItem("list_docs_example", "item1");

        // Delete the list
        results.qnDeleteList = qnDeleteList("list_docs_example");
    } catch (error) {
        results.error = error.message;
    }

    return results;
}

Sets

• qnAddSet: Creates a key-value set in Key-Value Store.
• qnBulkSets: Creates and removes bulk key-value sets in Key-Value Store.
• qnDeleteSet: Deletes a key-value set from Key-Value Store.
• qnGetSet: Retrieves the value of a specific key from Key-Value Store.
• qnListAllSets: List all keys for sets in Key-Value Store.

Example filter code for Key-Value Store sets

The filter code below can be used within Streams to demonstrate ways that Key-Value Store sets can be used within your Streams filter. You can deploy this example with one click here.

function main() {
    // List of results for each operation
    let results = {};

    try {
        // Create a set
        results.qnAddSet = qnAddSet("set_docs_example", "value1");

        // Get a set
        results.qnGetSet = qnGetSet("set_docs_example");

        // Get all sets
        results.qnListAllSets = qnListAllSets();

        // Bulk add/remove sets
        results.qnBulkSets = qnBulkSets({
            add_sets: {
                "set_docs_example2": "value1",
                "set_docs_example3": "value2"
            },
            delete_sets: ["set_docs_example1"]
        });

        // Get all sets after bulk
        results.qnListAllSets2 = qnListAllSets();

        // Delete key values
        results.qnDeleteSet2 = qnDeleteSet("set_docs_example2");
        results.qnDeleteSet3 = qnDeleteSet("set_docs_example3");

    } catch (error) {
        results.error = error.message;
    }

    return results;
}

Note

The filter function must be named main and return an object containing the modified data.