S3-compatible storage destinations let you archive stream data to object storage services like Amazon S3, Google Cloud Storage, Backblaze B2, and other S3-compatible providers.

Why use S3?

• Large Data Storage: S3 offers virtually unlimited storage, making it suitable for handling massive amounts of blockchain data
• Data Durability and Reliability: High durability and secure storage options ensure data is safely stored and readily available
• Cost-Effective: For substantial data volumes, S3 can be more cost-effective due to its pricing model based on storage and access
• Analytics Integration: Data stored in S3 can be seamlessly integrated with various analytics tools and data lakes
• Scalability: S3 scales automatically to accommodate data growth

Configuration

Follow the Quickstart Guide to create a stream. When you reach the destination step, select S3-Compatible Storage and configure the following fields:

Field	Description
Endpoint	S3 endpoint URL. For Amazon S3, leave as default. For GCP, use storage.googleapis.com. Use your provider's endpoint for other services.
Region	AWS region for the bucket (optional). Example: us-east-1
File compression	Available on all paid plans. Use None for human-readable files, or Gzip for bandwidth optimization in production.
Access key ID	Your unique identifier for storage access (found in your storage service's IAM section). Note: GCP calls this 'Access Key' in the Interoperability settings.
Secret access key	The corresponding secret key for authentication. Note: GCP calls this 'Secret' in the Interoperability settings.
Bucket name	Name of the S3 bucket that your credentials have write permissions to.
Prefix	Optional path prefix for organizing objects in your bucket, similar to folders. See AWS docs on using prefixes.
File type	Output format for the data. Currently Streams supports JSON as the primary format.
Retry wait period	Time in seconds to wait between retry attempts when a write to S3 fails (default: 1).
Pause stream after	Number of failed retries before the stream pauses. When retries are exhausted, the stream terminates but can be resumed once delivery issues are resolved (default: 3).
Use SSL	Enable SSL/TLS for secure connections. Required by most S3 services - you'll see an error if the service requires SSL and this is not selected.

Click Check Connection to verify your credentials and bucket access before creating the stream.

S3 Transfer Acceleration

Recommended for Backfills

For optimal performance, especially during backfills, we strongly recommend enabling S3 Transfer Acceleration on your S3 bucket. Without acceleration, S3 can become a significant bottleneck in your data pipeline, particularly when processing large amounts of historical data.

To enable Transfer Acceleration:

1. Go to your S3 bucket properties
2. Scroll down to Transfer Acceleration
3. Click Enable and save the changes

The acceleration endpoint will be in the format:

https://<bucket-name>.s3-accelerate.amazonaws.com

Sequential Processing

important

Streams processes blocks sequentially and will not proceed to the next block or batch until receiving confirmation that the current block or batch was successfully written to S3. This ensures data consistency but means your S3 bucket must be able to handle the write operations within a reasonable timeframe.

You can use any S3-compatible service such as:

• AWS S3
• GCP Cloud Storage
• Backblaze B2 Cloud Storage

Azure Blob Storage

Using Azure Blob with Streams S3-compatible storage requires you to set up and run a proxy: s3proxy. Alternatively, use the native Azure Blob Storage destination.

AWS S3 Setup

AWS Permissions

When setting up Quicknode Streams to upload data directly to an AWS S3 bucket, it's essential to configure the correct permissions for the IAM policy applied to the credentials you're using with Streams.

Steps to Create IAM Policy for S3 Access

Step 1: Log in to AWS Management Console

1. Open the AWS Management Console
2. In the navigation bar, click on Services and select IAM

Step 2: Create a New IAM Policy

1. In the IAM Dashboard, click on Policies in the left navigation pane
2. Click on the Create policy button
3. Select the JSON tab to define the policy

Step 3: Define the IAM Policy

Copy and paste the following JSON policy document. Replace your-bucket-name with the name of your S3 bucket:

• GetBucketLocation — Allows retrieval of the bucket's region
• PutObject — Allows uploading objects to the specified S3 bucket
• DeleteObject — Allows Streams to delete objects in the bucket (necessary to enable overwriting of existing objects)

S3 Versioning

Consider enabling versioning on your S3 bucket. This way, old versions of objects are retained even when overwritten.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket-name/*"
      ]
    }
  ]
}

Step 4: Review and Create Policy

1. After pasting the JSON, click on the Review policy button
2. Provide a Name and Description for the policy
3. Click on Create policy

Step 5: Attach Policy to IAM User

1. Go to Users in the IAM Dashboard
2. Select the user that will be uploading to S3
3. In the Permissions tab, click on Add permissions
4. Choose Attach policies directly
5. Search for and select the policy you just created
6. Click on Next: Review and then Add permissions

For more details, see the AWS IAM documentation or contact Quicknode support.

Google Cloud Storage Setup

GCP Cloud Storage is S3-compatible through interoperability mode. You'll need to create HMAC keys to authenticate.

Creating HMAC Keys

In GCP, users can generate access keys in the Cloud Storage settings under the Interoperability tab. There are two options:

Option 1: Service Account HMAC (Recommended for Production)

Recommended for production workloads. This option allows the creation of access keys tied to your organization's Cloud Platform service accounts, ensuring continuity and reducing administrative oversight. Learn more.

1. Go to Cloud Storage → Settings → Interoperability
2. Click Create a key for a service account
3. Select or create a service account with Storage Object Admin permissions
4. Save the Access Key and Secret

Option 2: User Account HMAC

This allows authentication with access keys tied to your user account. This is less recommended for production, as it ties access directly to individual user accounts. Learn more.

1. Go to Cloud Storage → Settings → Interoperability
2. Click Create a key for another service account or use your user account
3. Save the Access Key and Secret

GCP Configuration

Use the following settings for GCP:

• Endpoint: storage.googleapis.com
• Access key ID: Your HMAC access key
• Secret access key: Your HMAC secret

Click Check Connection to verify your credentials and bucket access before creating the stream.

---

Other Destinations

Webhook

Deliver stream data to your HTTP endpoint in real-time.

Azure Blob Storage

Microsoft Azure cloud storage with enterprise features.

PostgreSQL

Store in a relational database for SQL queries and joins.

Kafka

Stream to Apache Kafka for high-throughput pipelines.

Support

💬

Discord CommunityGet help from the community and Quicknode team

↗

🎧

Direct SupportGet personalized help from our support team

↗