Dmitry

February 29, 2024 — 6 min read

Get started with Streams today

https://www.producthunt.com/posts/streams?utm_source=badge-featured&utm_medium=badge&utm_souce=badge-streams

In an era where data rivals currency in value, navigating the complexities of web3 data has emerged as a major innovation roadblock. Developers and enterprises alike have grappled with the daunting task of extracting blockchain insights, often hitting barriers of complexity, operational challenges, and the opaque economics of on-chain data mining.

Leveraging our extensive experience in overcoming infrastructure challenges and responding to feedback from a wide range of innovators, we have decided to bridge the gap between data tools and web3 with a portfolio of ETL products.

Our first solution, Streams, is designed to make blockchain data seamlessly available for conventional tools used for data, embodying our vision to streamline, simplify, and secure web3 data integration.

As we introduce Streams, we're not just presenting a product but laying a foundation for our new ETL pipeline vision for web3 data. Streams embody QuickNode's dedication to innovation, efficiency, and making blockchain data universally accessible and actionable.

Announcing Streams

Streams are a foundational layer of our ETL vision for blockchain data. It connects data sources consisting of 17+ chains and networks, 5+ essential datasets with multiple destinations, including webhooks, S3-compatible storage, PostgreSQL and Snowflake available during Public Beta.

Complexity is not just a barrier to entry; it's a continual source of inefficiency. Users often struggle with the methods, parameters, and ever-changing schemas, leading to frustration and wasted effort.

To dismantle these barriers, we introduce an intuitive, user-friendly interface that simplifies integrating and managing blockchain data. In less than 6 clicks you will get streamed essential web3 datasets in seconds.

Equipping users with preset datasets eliminates the guesswork and technical overhead associated with data retrieval, making the first step into blockchain data straightforward.

Exactly-Once, Guaranteed Delivery

Streams revolutionize web3 data infrastructure by acting as an exactly once guaranteed delivery service. This means you can seamlessly integrate Streams with your data lake in a few clicks, ensuring reliable data extraction and loading without the hassle and significant engineering time to manage retries, corrupt data, or missing blocks.

Streams eliminates common obstacles and developer pain points with endpoint infrastructure, such as:

• Rescaling your ingestion layer depending on chain dynamics — Streams will provision and scale compute, memory, and network resources without ongoing administration.

• Corrupt or missing data impacting your analysis — As a guaranteed delivery service, we mean it: every block, receipt, or trace will be delivered exactly-once in the order of the dataset finality.

Forget the cumbersome process of provisioning workers to pull data from blockchain endpoints and having to monitor their progress.

Real-Time Data Consistency

Navigating the complexities of real-time blockchain data—such as varying finality times across different datasets (like blocks, receipts, traces) or navigating through blockchain reorganizations—often results in data inconsistency. Streams guarantee the delivery of consistent, real-time data, empowering you to create live dashboards without reliability concerns.

Efficient Historical Data Handling

For historical data analysis or backfills, Streams allows configuring large data batches, setting precise data ranges, and fine-tuning your destination. This capability enables you to select the most optimal setup and throughput for your chosen destination, significantly simplifying your data pipelines.

Comprehensive and Transparent User Experience

Logs offer transparency into the operational aspects of ingestion and retries, while metrics provide insights into performance and reliability.

Financial management is as crucial as data management. Our platform offers transparent billing and usage tracking, empowering users with the information they need to manage their costs effectively. This transparency ensures that users can allocate their resources wisely, avoiding unexpected expenses and optimizing their investment in web3 data infrastructure.

Pricing and Feature Availability

Throughout its Beta stage, Streams will follow a transparent and scalable pricing model designed to meet the diverse needs of our clients, from startups to large-scale data engineering teams.

At the core, our pricing is based on the volume of data you stream, measured in gigabytes, ensuring costs directly correlate with your usage.

Starting from the Build Plan, which is perfect for prototyping and basic data access, we scale up to the Enterprise Plan, which is designed for high-volume, real-time direct data integration.

Pricing Examples

To illustrate our pricing in action, consider the following scenarios of use for Streams.

As such these estimates offer a starting point for understanding your potential costs:

Streams currently supports: 👇

Chains

1. Arbitrum

2. Arbitrum Nova

3. Avalanche C-Chain

4. Base

5. Bera

6. Blast

7. BNB Smart Chain

8. Celo

9. Ethereum

10. Fantom

11. Gnosis

12. Immutable X

13. Optimism

14. Polygon

15. Polygon zkEVM

16. Scroll

17. ZKSync

Datasets

• Blocks

• Transactions

• Receipts

• Logs

• Blocks with Receipts

Destinations

• Webhooks

• S3-compatible storage

• PostgreSQL

• Snowflake

See the full list of features and chains on our docs website.

A Complete Platform for Blockchain Data and ETL Expertise

Functions, also launching soon, can be combined with Streams to radically solve any blockchain data need; use common programming languages—including Javascript, Python, Go, or Ruby—to efficiently transform payloads from Streams, enrich them with contract data by calling QuickNode endpoints, and much more.

Building upon an already robust blockchain data product portfolio, including Core RPC and QuickAlerts, the addition of Streams and Functions ensures QuickNode is the trusted solution provider for all your ETL needs.

How To Access Streams

Streams will change the way you use blockchain data.

Ready to see for yourself?

Streams are now available on your QuickNode dashboard for accounts with the Build plan and higher. We value your input and invite you to contribute to our evolving roadmap. Get started with Streams today and discover the impact on your project.

For guidance and more information, our team is just a click away.