Skip to content

Latest commit

 

History

History
130 lines (107 loc) · 5.98 KB

README.md

File metadata and controls

130 lines (107 loc) · 5.98 KB
Raw

Welcome to the Lite RPC QUIC Forward Proxy Sub-Project!

Project Info

Purpose

This component (quic-forward-proxy) can be optionally used along with one or multiple Lite RPC micro-service instances to optimized and simplify sending transactions to Solana TPUs.

Benefits:

  • the quic-forward-proxy can batch transactions from multiple source and send them efficiently to the Solana validator
  • the quic-forward-proxy can be optionally configured with a validator identity keypair:
    • connections to TPU will be privileged (staked connections)
    • keypair can be kept in one place while the Lite RPC instances can benefit from the staked connection

Configuration

Prepare: choose a proxy port (e.g. 11111) and bind address of appropriate (minimal) network interface (e.g. 127.0.0.1)

  1. run quic proxy 
    # unstaked
solana-lite-rpc-quic-forward-proxy --proxy-listen-addr 127.0.0.1:11111
# staked
solana-lite-rpc-quic-forward-proxy --proxy-listen-addr 127.0.0.1:11111 --identity-keypair /pathto/validator-keypair.json
  2. run lite-rpc 
    lite-rpc --experimental-quic-proxy-addr 127.0.0.1:11111

Architecture Overview

 +------------+          +------------+          +------------+          +------------+
 |            |          |            |          |            |          |            |
 |   client   | ---1---> |  lite-rpc  | ---2---> |   proxy    | ---3---> |  validator |
 |            |          |            |          |            |          |            |
 +------------+          +------------+          +------------+          +------------+
 
 1. rpc request
 2. tpu forward proxy request (QUIC): transactions, tpu address and tpu identity
 3. tpu call (QUIC), transactions:
 
 * client: RPC client to lite-rpc
 * proxy: QUIC forward proxy service (one instance)
 * lite-rpc: N lite-rpc services 
 * validator: solana validator (TPU) according to the leader schedule

Local Development / Testing

Rust Integration Test

Use integrated testing in quic_proxy_tpu_integrationtest.rs for fast feedback.

Local Setup With Solana Test Validator

  1. run test-validator (tested with 1.16.1) 
    RUST_LOG="error,solana_streamer::nonblocking::quic=debug" solana-test-validator --log
  2. run quic proxy 
    # unstaked
RUST_LOG=debug cargo run --bin solana-lite-rpc-quic-forward-proxy -- --proxy-listen-addr 0.0.0.0:11111
# staked
RUST_LOG=debug cargo run --bin solana-lite-rpc-quic-forward-proxy -- --proxy-listen-addr 0.0.0.0:11111 --identity-keypair /pathto-test-ledger/validator-keypair.json
  3. run lite-rpc 
    RUST_LOG=debug cargo run --bin lite-rpc -- --quic-proxy-addr 127.0.0.1:11111
  4. run rust bench tool in lite-rpc 
    cd bench; cargo run -- --tx-count=10

Implementation Details

  • proxy expects clients to send transactions via QUIC using the proxy request format:
    • array of transactions (signature and raw bytes)
    • array of tpu target nodes (address:port and identity public key)
  • the proxy is designed to be light-weight and stateless (no persistence)
  • note: only one instance of the proxy should talk to the TPU nodes at a time to be able to correctly comply with the validator quic policy
  • outbound connections (to TPU):
    • proxy tries to maintain active quic connections to recent TPU nodes using transparent reconnect
    • proxy will maintain multiple quic connection per TPU according to the Solana validator quic policy
    • proxy will use lightweight quic streams to send the transactions
  • inbound traffic (from Lite RPC)
    • client-proxy-communication is done via QUIC using a custom wire format
    • proxy supports only quic ATM but that could be extended to support other protocols
    • proxy should perform client authentication by TLS (see issue)
  • proxy uses a single queue (channel) for buffering the transactions from any inbound connection
  • TPU selection / Leader Schedule
    • the proxy will not perform any TPU selection; the TPU target nodes MUST be selected by the client (Lite RPC) and not by the proxy
    • pitfall: the TPU target node list might become stale if the transactions are not sent out fast enough

Example Output from Solana Validator:

(note: the peer type is Staked)

[2023-06-26T15:16:18.430602000Z INFO  solana_streamer::nonblocking::quic] Got a connection 127.0.0.1:8058
[2023-06-26T15:16:18.430633000Z DEBUG solana_streamer::nonblocking::quic] Peer public key is EPLzGRhibYmZ7qysF9BiPmSTRaL8GiLhrQdFTfL8h2fy
[2023-06-26T15:16:18.430839000Z DEBUG solana_streamer::nonblocking::quic] Peer type: Staked, stake 999999997717120, total stake 999999997717120, max streams 2048 receive_window Ok(12320) from peer 127.0.0.1:8058
[2023-06-26T15:16:18.430850000Z DEBUG solana_streamer::nonblocking::quic] quic new connection 127.0.0.1:8058 streams: 0 connections: 1
[2023-06-26T15:16:18.430854000Z DEBUG solana_streamer::nonblocking::quic] stream error: ApplicationClosed(ApplicationClose { error_code: 0, reason: b"done" })

Solana Validator QUIC Policy

TPU has complex logic to assign connection capacity to TPU clients (see Solana quic.rs)

  • it purges connections
  • it limits number of parallel quic streams
  • it considers stake vs unstaked connections (validator identity signature in QUIC TLS certificate, see Solana get_remote_pubkey+get_pubkey_from_tls_certificate)
  • it keeps connections on a per peer address / peer identity basis
  • ...

License & Copyright

Copyright (c) 2022 Blockworks Foundation

Licensed under the AGPL-3.0 license