How to Fetch Transaction History on Ethereum using Web3.py

June 06, 2022

Overview

If you are building on Ethereum, you may run into scenarios where you need to fetch transaction history for a given address. Unfortunately, the current design for Ethereum lacks an easy way to retrieve this data. For this reason, we are dedicating this guide to explaining the different ways in which you can retrieve transaction history on Ethereum. We will demonstrate one of these ways with Python. Take a look at the prerequisites for what you'll need to follow along: 

What You Will Need

Methods to Retrieve Transactions for a Given Address

To retrieve transactions for a given address, we can do any of the following: 

  • Iterate through each block: Since there is no specific RPC method we can call to obtain a list of transaction data for a given address, we will need to iterate through the blocks of the blockchain ourselves and record each instance where we detect a transaction related to our address. This method can be time-consuming and can consume a lot of RPC calls if you need to start from the beginning of the blockchain (genesis block). Additionally, we should request this data from an Archive node (which contains all the historical data of the blockchain since the genesis block).

  • Use an indexing service: APIs such as the Etherscan API allow you to fetch transactions for a given address. However, these come with limitations as you will need to make different requests for different types of transactions you are looking for (e.g., external vs. internal transactions).

  • Use an open-sourced Indexer: This repository has set up an indexing database for users to be able to query this type of transaction data. Although it takes a bit longer to set up, it will be easier to analyze and manage once you have the data.

Access an Ethereum Node with QuickNode

If we're interested in getting all blocks from genesis, we'll need to communicate with an Archive node. At the current time of this writing, the full state of the Ethereum blockchain takes around 10,000 GB of storage (~11TB). Luckily, we don't need to manage this infrastructure ourselves, and we can use QuickNode to boot up an Archive node.

To access an Ethereum node, navigate to the endpoints page and click the Create an Endpoint button. Then, select the chain of your choice. For this example, we are using the Ethereum Mainnet, but you can also use other EVM-related chains (i.e., Fantom, Polygon, Avalanche, Optimism).

QuickNode Select Blockchain Tab


If you're looking to get transaction data from the very beginning of the Ethereum chain, select the Archive node add-on:

QuickNode Configure Add-ons Tab


Note: If you're looking to include transactions where your address was included in a trace call, QuickNode provides a quick and easy way to enable this functionality on your blockchain node via the Trace Mode add-on.

Once your node is ready, copy the HTTP URL as you will need it in the following section.

Fetching Transaction History using Web3.py

Now that we have our Archive node ready, we will move on to creating the Python script. Open your terminal and run the following commands to create a project directory:

mkdir eth-transactions-web3py && cd eth-transactions-web3py && echo > transaction_finder.py

Next, we will install the required dependencies:

pip install web3

For simplicity, this demonstration will only be searching from the current block and the trailing 100 blocks. Additionally, we'll only check if the to address matches the one set in our blockchain_address variable. In your transaction_finder.py file, add the following code:

fetching transaction history using web3 py

Copy
#import dependencies
import pickle
from web3 import Web3, HTTPProvider

#instantiate a web3 remote provider
w3 = Web3(HTTPProvider('YOUR_QUICKNODE_HTTP_ENDPOINT'))

#request the latest block number
ending_blocknumber = w3.eth.blockNumber

#latest block number minus 100 blocks
starting_blocknumber = ending_blocknumber - 100 

#filter through blocks and look for transactions involving this address
blockchain_address = "INPUT_A_VALID_ETHEREUM_ADDRESS"

#create an empty dictionary we will add transaction data to
tx_dictionary = {}

def getTransactions(start, end, address):
    '''This function takes three inputs, a starting block number, ending block number
    and an Ethereum address. The function loops over the transactions in each block and
    checks if the address in the to field matches the one we set in the blockchain_address.
    Additionally, it will write the found transactions to a pickle file for quickly serializing and de-serializing
    a Python object.'''
    print(f"Started filtering through block number {start} to {end} for transactions involving the address - {address}...")
    for x in range(start, end):
        block = w3.eth.getBlock(x, True)
        for transaction in block.transactions:
            if transaction['to'] == address or transaction['from'] == address:
                with open("transactions.pkl", "wb") as f:
                    hashStr = transaction['hash'].hex()
                    tx_dictionary[hashStr] = transaction
                    pickle.dump(tx_dictionary, f)
                f.close()
    print(f"Finished searching blocks {start} through {end} and found {len(tx_dictionary)} transactions")
    

getTransactions(starting_blocknumber, ending_blocknumber, blockchain_address)

Note that the code above stores the transaction data in a pickle file. The advantage of storing the data in pickle format is that it can serialize any Python object without handling additional code. Plus, we can then easily read, analyze and clean up the results without making another request to the blockchain for the same information.

Remember to add your RPC URL to the w3 variable and also update the blockchain_address to the address you're searching historical transactions for. When you're finished with that, save the file and run the script by running the command python3 transaction_finder.py

Terminal output from transactionfinder.py script


If the output in your terminal is similar to the screenshot above, congrats! 🎉

To read the transactions.pkl file's contents, you can create another file called transaction_reader.py and add the following code snippet:

fetching transaction history using web3 py

Copy
import pickle

data = pickle.load( open("transactions.pkl", "rb" ))
print(data)

When running the transaction_reader.py script, you should get something similar to the screenshot below, where the key is the transaction hash, and the value contains the transaction details.

Terminal output from transaction_reader.py script


Important things to note: The script above only detects transactions with your address in the to or from field. There are scenarios where a transaction can trigger smaller actions that will still modify the internal state of the EVM. The data related to these actions are logged internally within the transaction and are classified as a trace.

If you're looking for transactions related to NFTs, you can use our NFT API to pull transaction history and key collection information about NFTs without having to sort through individual smart contracts.

Conclusion

There you have it! We just demonstrated how to fetch transactions for a given address on Ethereum! If you find this useful, check out some of our guides here. Subscribe to our newsletter for more articles and guides. Feel free to reach out to us via Twitter if you have any feedback. You can always chat with us on our Discord community server, featuring some of the coolest developers you'll ever meet 😊

Related articles 54

How to Send a Transaction On Solana Using JavaScript
Apr 13, 2022

Hello reader! Today is an exhilarating day because we are going on an expedition to the Solana Blockchain. Solana is an up-and-coming blockchain seeking to improve upon the current ecosystem's solutions to the complex problem of providing a secure, scalable, decentralized...

Continue reading
How to do a non-custodial transaction with QuickNode
Apr 12, 2022

Private keys are one of the most sensitive pieces of data when it comes to cryptography and the blockchain. However, there has always been debate/confusion about choosing between custodial wallets (where the wallet provider has custody of the user’s private key) and...

Continue reading
How to connect to Ethereum using .NET (Nethereum)
Apr 12, 2022

Dotnet or .NET is very popular for the development of desktop applications, most Windows desktop applications are built using .NET, and it also contributes largely to web application’s tech stack. In this guide, let’s see how we can connect to Ethereum using .NET and

Continue reading
How to Set Up a Near Project from Scratch
Jan 27, 2022

In this tutorial we will look at how we can setup a basic NEAR project from scratch, installing and configuring dependencies and customizing the project to work well with AssemblyScript.We will first start by initializing our project with a package.json file using...

Continue reading
Como crear y lanzar un ERC-721 (NFT)
Dec 29, 2021

Coleccionables digitales que son compatibles con ERC-721 se han vuelto muy populares desde el lanzamiento de Cryptokitties y han ganado adopción masiva en los últimos meses. Esta guía cubrirá la parte de creación y lanzamiento...

Continue reading
How to connect to Ethereum network using Java / Web3j
Apr 12, 2022

We can say that Java is one of the most versatile languages out there, and it continues to be relevant in today's time. Java is so popular because of its massive user base and use cases. In this guide/tutorial, we'll learn how to connect to the Ethereum Blockchain network...

Continue reading
How to integrate IPFS with Ethereum
Apr 12, 2022

It can be costly to store massive files on a blockchain mainnet, and this is where decentralized file storing systems like IPFS can come in handy. Sometimes, NFTs use IPFS as well. In this guide, we’ll cover how we can integrate IPFS with...

Continue reading
How to Connect to the Ethereum Network using Ruby
Jun 13, 2022

The Ruby programming language has a huge fanbase. Ruby was developed by its creator with an intention to invent a language developers can enjoy learning and using. Ruby has been largely accepted by developers all around the world since its launch, in fact, the biggest...

Continue reading
How to connect to Ethereum network with ethers.js
Apr 12, 2022

When someone thinks of developing a dApp the first tool that comes to their mind is web3.js which is pretty common because of its popularity in the community and wide use cases, dApp development has been consistently growing and there are a lot of developers who want to...

Continue reading
How to Mint an NFT on Solana
Apr 12, 2022

Updated at: April 10, 2022Welcome to another QuickNode guide on Solana - the up-and-coming blockchain that seeks to solve the scalability issues of Ethereum. We will be walking through step-by-step how to create an NFT on Solana. NFT, short for Non Fungible Token,...

Continue reading
The Web3 Developer Stack
Apr 12, 2022

A developer stack is a bag of technologies a developer possesses. For example, MEAN (MongoDB, Express.js, AngularJS/Angular, and Node.js) and MERN (MongoDB, Express.js, React, and Node.js) are common web developer stacks. Similarly, today we will learn more about the web3...

Continue reading
How to deploy a smart contract with Brownie
Apr 12, 2022

Python is one of the most versatile programming languages; from researchers running their test models to developers using it in heavy production environments, it has use cases in every possible technical field. In today's guide, we will learn about Brownie, a Python-based...

Continue reading
How to Get All Tokens Held by a Wallet in Solana
Jun 24, 2022

Hello readers! To kick off Solana Summer and the current whitelist meta, we thought it would be helpful to dig into all of the token accounts you and your users have using the getParsedProgramAccounts method. This tool is convenient for querying different...

Continue reading
Introduction to Scaffold-ETH 🏗
Dec 29, 2021

Developing applications involves juggling several moving pieces like front-ends, back-ends, and databases. But developing a decentralized application on a blockchain adds a few more elements like smart contracts and nodes that allow you to connect to the...

Continue reading
Como crear un NFT en SOLANA
Dec 29, 2021

¡Hola querido lector! Bienvenidos a una nueva guía de Solana.Solana es una blockchain que promete mucho a la hora de intentar resolver los problemas de escalabilidad que podemos apreciar en otras blockchains, como Ethereum por...

Continue reading
How to Get Transaction Logs on Solana
Jun 28, 2022

Ever need to pull all the transactions associated with a Wallet? Want to see all of the mint transactions associated with a Candy Machine? Or maybe see transaction history of an NFT? Solana's getSignaturesForAddress method is a versatile tool that makes...

Continue reading
How to Send an EIP-1559 Transaction
Apr 12, 2022

While Ethereum has been trying to scale, it has encountered some gas price issues. Many layer 2 solutions and sidechains sprang into existence to solve this problem, but Ethereum is the main chain, and at some point, it has to be improved. EIP-1559 was introduced to...

Continue reading
How to Create an Address in Solana using JavaScript
Apr 12, 2022

Hello reader! Welcome to QuickNode's first Solana guide. Solana is an up-and-coming blockchain that seeks to solve the scalability issues that Ethereum has been handling. You will walk through step-by-step how to create a Solana address using the @solana/web3.js...

Continue reading
How to create your own DAO with Aragon
Apr 12, 2022

Blockchain provides us with the power of decentralization. Decentralization means the transfer of power to users/members rather than having a single centralized authority governing everything; it enables various use cases in finance, governance, voting, fundraising, etc....

Continue reading
How to Connect to Terra with JavaScript using Terra.js
Apr 12, 2022

Stablecoins have been bridging the gap between traditional currencies and blockchains. Stablecoins offer stable price tokens pegged by a reserve asset which is often a fiat current like USD, EUR, GBP. The Terra protocol provides a framework to work with stablecoins. This...

Continue reading
How to connect to Ethereum network using Go
Apr 12, 2022

Go helps you make faster scalable backends and this guide will show you how to connect your backend to Ethereum (and make it even faster, more reliable, and globally accessible, all thanks to QuickNode’s global infrastructure). What is...

Continue reading
How To Fork Ethereum Mainnet with Hardhat
Apr 12, 2022

Forking the chain at an older block of the blockchain is helpful if you want to simulate the blockchain’s state at that block; Hardhat has this functionality built in. In this guide, let’s go through the process of forking the Ethereum Mainnet at an older...

Continue reading
How to connect to Ethereum using PHP
Apr 12, 2022

PHP is a very popular choice among developers and has a vast community due to its long presence in web development. In this guide, we’ll cover how to connect to Ethereum with PHP using the web3.php...

Continue reading
How to use Subspace with QuickNode
Apr 12, 2022

In this guide, we'll understand a bit about reactive development and how to use Subspace with QuickNode.JavaScript is the programming language behind most of the internet apps and websites. JavaScript today has become one of the most used programming languages,...

Continue reading
How to Connect Your Dapp With MetaMask Using Ethers.js
Dec 29, 2021

In our dApp, we will have a simple react user interface that has a material button asking the user to connect to MetaMask. And if they do not have an account, they can create one or log in to their account. They will then view their wallet balance and address displayed on...

Continue reading
How to generate a new Ethereum address in Go
Dec 29, 2021

Golang is very popular among backend developers for building infrastructures and microservices. Go is a procedural programming language. Developed in 2007 by Robert Griesemer, Rob Pike, and Ken Thompson at Google, then launched in 2009 as...

Continue reading
How to generate a new Ethereum address in Python
Dec 29, 2021

Python is one of the most versatile programming languages out there with an abundance of use cases; We can build many applications with Python from client-side to back end. In this guide, we will cover creating an Ethereum address in Python using the

Continue reading
How to Lazy Mint an NFT on Rarible with Rarepress
Apr 12, 2022

NFTs are great for creators to monetize their artwork and for people to get ownership of an item. But since gas prices are usually high given the highly in-demand space on Ethereum, minting an NFT or an NFT collection can become costly for a creator. Lazy minting solves...

Continue reading