Performance Consideration for High-Volume Data Streams

If using Solana gRPC on Node.js for very large quantities of data, it is possible to overwhelm a single CPU. If you're building systems that need to process all transactions from busy programs or track multiple programs simultaneously, consider the following high performance alternatives:

• Using multiple threads. Node.js has two modules: cluster (multi-process, one thread per process) and worker (single process, multiple threads). Either will solve the problem of node overloading a single core, but cluster is simpler as each process is memory isolated. Divide the Solana gRPC subscriptions between multiple instances of cluster or worker to distribute the load.

• Using Solana gRPC on Rust (see also Solana gRPC on Rust guide)

• Using Solana gRPC on Go (see also Solana gRPC on Go guide).

Solana gRPC is available on Mainnet and Devnet for Scale and Business customers. The example below uses Mainnet. For Devnet, use your Solana Devnet endpoint hostname and token; see Endpoint and Token Configuration.

Overview

TypeScript is a strongly-typed superset of JavaScript, offering improved developer experience and tooling support. This document provides a step-by-step process for setting up a TypeScript environment for interacting with Solana gRPC (Yellowstone-compatible Geyser gRPC), including setting up a project, configuring dependencies, and implementing authentication mechanisms.

Authentication Required for TypeScript

Creates a client in order to communicate with Solana gRPC. It uses an endpoint URL and an authentication token for secure access.

import Client from '@triton-one/yellowstone-grpc';

// For HTTP Provider URL: https://example-guide-demo.solana-mainnet.quiknode.pro/123456789/

const ENDPOINT = 'https://example-guide-demo.solana-mainnet.quiknode.pro:443';
const TOKEN = '123456789';
const client = new Client(ENDPOINT, TOKEN, {
  grpcDefaultCompressionAlgorithm: 0, // 0 = gzip, 1 = zstd
});
await client.connect();

The token passed to the Client constructor is sent as the x-token metadata header on every request.

gzip and zstd are selected with the grpcDefaultCompressionAlgorithm channel option. See Compression for per-language configuration.

Note:

To learn how to split your Solana gRPC-enabled URL into endpoint and token, refer to this section - Endpoint and Token Configuration

Setting up TypeScript

To run our TypeScript code examples, you'll need to have Node.js installed, as TypeScript is built on top of JavaScript. Ensure you also have TypeScript installed globally by running:

npm install -g typescript

Install Yellowstone gRPC Node Package

npm install @triton-one/yellowstone-grpc

These examples target @triton-one/yellowstone-grpc v7.0.1 or later. Reading bankId or block footers needs v8.0.0 or later, and block footer certificates need v9.0.0 or later.

Decode Block Footer Certificates

This example subscribes to block footers with includeCertificates set and reads the highest-known finalization certificate carried by each footer. The certificate’s slot can be earlier than the footer’s slot; the footer’s bankId identifies the carrying bank, not the certified block. blockFinalCert carries only finalAggregate when the block finalized on the fast path, and both finalAggregate and notarAggregate on the slow path. signerBitmap marks each validator in the epoch that signed, so counting its marks gives the number of signers.

import Client, {
  CommitmentLevel,
  SubscribeRequest,
  SubscribeUpdate,
} from '@triton-one/yellowstone-grpc';

// For HTTP Provider URL: https://example-guide-demo.solana-mainnet.quiknode.pro/123456789/
const ENDPOINT = 'https://example-guide-demo.solana-mainnet.quiknode.pro:443';
const TOKEN = '123456789';

// The bitmap starts with a version byte and a little-endian u16 bit count.
// Version 0 uses one bit per validator. Version 1 packs five base-3 digits
// per byte, where a nonzero digit marks a signer.
function countSigners(bitmap: Uint8Array): number {
  const version = bitmap[0];
  const bitCount = bitmap[1] | (bitmap[2] << 8);
  const payload = bitmap.subarray(3);
  let signers = 0;
  for (let i = 0; i < bitCount; i++) {
    if (version === 0) {
      signers += (payload[i >> 3] >> (i & 7)) & 1;
    } else if (version === 1) {
      signers += Math.floor(payload[Math.floor(i / 5)] / 3 ** (i % 5)) % 3 ? 1 : 0;
    } else {
      throw new Error(`Unknown signer bitmap version: ${version}`);
    }
  }
  return signers;
}

async function main() {
  const client = new Client(ENDPOINT, TOKEN, {
    grpcDefaultCompressionAlgorithm: 0, // 0 = gzip, 1 = zstd
  });
  await client.connect();
  const stream = await client.subscribe();

  const request: SubscribeRequest = {
    accounts: {},
    slots: {},
    transactions: {},
    transactionsStatus: {},
    blocks: {},
    blocksMeta: {},
    entry: {},
    blockFooter: {
      footers: { includeCertificates: true },
    },
    commitment: CommitmentLevel.FINALIZED,
    accountsDataSlice: [],
  };

  stream.on('data', (update: SubscribeUpdate) => {
    const footer = update.blockFooter;
    const cert = footer?.blockFinalCert;
    if (!footer || !cert?.finalAggregate) return;

    const finalization = cert.notarAggregate ? 'slow' : 'fast';
    const signers = countSigners(cert.finalAggregate.signerBitmap);
    console.log(
      `certified slot ${cert.slot}: ${finalization} finalization, ${signers} signers (carried by footer slot ${footer.slot}, bank ${footer.bankId})`
    );
  });
  stream.on('error', (error) => console.error(error));

  await new Promise<void>((resolve, reject) => {
    stream.write(request, (err: Error | null | undefined) => (err ? reject(err) : resolve()));
  });
}

main();

Additional Resources

For more information about working with Solana gRPC in TypeScript, refer to Monitor Solana Programs with Solana gRPC (TypeScript). This guide provides an in-depth overview of setting up gRPC endpoints, managing authentication tokens, and understanding key concepts like commitment levels.