Architecture
The SP1 Cluster is the official multi-GPU prover service implementation for generating SP1 proofs on the Succinct Prover Network. It can coordinate proof generation across tens to hundreds of GPU nodes.
It bids on requests from the Prover Network, proves assigned requests, and fulfills them back to the Prover Network.
Components
The cluster has several key components:
API
The API is a simple gRPC and HTTP server built using Tonic, sqlx, and axum that serves as an entrypoint to the cluster for submitting new proofs and querying pending requests. It tracks all proof requests in a Postgres DB. If the coordinator process crashes, the coordinator is able to restart any pending proofs upon rebooting based on the state stored in the DB.
The fulfiller connects to the API server in order to sync the state of the Prover Network with the cluster.
Coordinator
The coordinator is a gRPC server built using Tonic that coordinates task assignment. When tasks are assigned to workers, they are sent through a one-directional gRPC stream to that worker for minimal latency. The coordinator constantly polls the API server for new or canceled proof requests and updates the in-memory state accordingly. If a worker node crashes or a task fails for a retryable reason, the coordinator will automatically reassign tasks to another worker node.
There are multiple AssignmentPolicy implementations that can be used to determine how tasks are assigned to workers. The default policy is Balanced which balances worker utilization across all active proofs.
Node
The node is the actual process that runs on each worker machine which handles proving tasks. There is a Cargo feature flag gpu which uses sp1-gpu-prover, Succinct's GPU prover library, to accelerate proving tasks. The node connects to the coordinator service to receive tasks to work on, update task/proof status, and possibly create new tasks.
The node runs tasks of several types:
| Task Type | Worker Type | Description | Inputs | Outputs |
|---|---|---|---|---|
| Controller | CPU | Start the proving process, coordinate its tasks, and assemble the final proof. More detail can be found in the Controller section. | Program, Stdin, Proof Type | Final Proof |
| Core Execute | CPU | Execute the program during proving and submit Prove Shard tasks from its trace, precompile, and memory data. | Program, Stdin, Common Input | Trace Data, Execution Output |
| Setup Vkey | GPU | Generate a verification key for a program. (similar to running prover.setup(&elf) locally). The verification key is a prerequisite to Prove Shard so this task is done on GPU as early and quickly as possible. | Program | Verification Key |
| Prove Shard | GPU | Generate a shard proof from core execution trace data, precompile events, or memory data. | Program, Common Input, Trace Data | Shard Proof |
| Recursion Reduce | GPU | Reduce a batch of shard or recursion proofs into one recursion proof. | Recursion Proofs | Recursion Proof |
| Recursion Deferred | GPU | Process a deferred proof (an SP1 Compressed proof being verified by the program in the VM) into a proof that can be compressed with other recursion proofs. | Deferred Proof Input | Recursion Proof |
| Shrink + Wrap | GPU | Compress a recursion proof into a smaller proof that can be verified in the Groth16/Plonk wrapper. | Recursion Proof | Shrink + Wrap Proof |
| Groth16/Plonk | CPU | Generate a Groth16/Plonk proof from the result of Shrink + Wrap. | Shrink + Wrap Proof | Groth16/Plonk Proof |
| ExecuteOnly | CPU | Execute the program without generating a proof. Used for simulation and gas estimation. | Program, Stdin | Execution Result |
| UtilVkeyMapController | CPU | Coordinate verification key generation across multiple chunks for large programs. | Program | Coordination State |
| UtilVkeyMapChunk | GPU | Generate a portion of the verification key map in parallel with other chunks. | Program, Chunk Index | Partial Vkey Map |
Bidder
The bidder watches the Prover Network for new proof requests. Before bidding, it checks whether the cluster has enough proving throughput and CPU-worker capacity to complete a request before its deadline. It counts proofs already assigned to the cluster. It also counts bids that wait for auction settlement. If the bidder wins, the request is assigned to the prover and the fulfiller submits it to the cluster.
See Bidder for configuration details.
Fulfiller
The fulfiller queries the Prover Network and Cluster API in a loop in order to find any new proofs assigned to the fulfiller's address in the Prover Network and add them to the cluster. It also takes completed proofs from the cluster and fulfills them in the Prover Network.
Artifact Store
The cluster uses an artifact store to store intermediate data used in the proving process. Artifacts can vary in size from <1 MB to >500 MB. Currently the cluster supports Redis and S3 as artifact stores.
Controller Details
The controller task is created at the beginning of a proof request and orchestrates the entire proving process of a single proof request. Several Tokio tasks are spawned in parallel to handle data movement and task creation in order to minimize the end to end proving latency and maximize cluster throughput.
Setup Vkey
The first thing that happens in the controller is a Setup Vkey task being created which will run on a GPU worker. The task generates the vkey and uploads it to the artifact store. Then, the vkey is downloaded in the controller where some small computation happens to create the common proving input, which is uploaded and provided to every Prove Shard task.
Core Execute
During proving, the controller submits a Core Execute task to a CPU worker. The task executes the program and submits Prove Shard tasks from the resulting core trace data. It also submits Prove Shard tasks for precompile events and memory initialization and finalization data.
Prove Shard Tasks
As the Core Execute task produces trace data, it uploads each record and submits a Prove Shard task to a GPU worker. Each record contains one of the following:
- Core execution trace data
- Precompile event data
- Memory initialization and finalization data
Each Prove Shard task generates a shard proof and reports it to the controller.
Precompile Data Processing
When a core execution shard is proven, its precompile events are separated and uploaded to the artifact store.
The Core Execute task collects these events and packs them into precompile shards. When it has enough events of one type, or no core execution shards remain, it submits a Prove Shard task for the precompile shard.
Deferred Leaves
If the program is verifying any SP1 proofs using proof aggregation, these can be processed immediately and put into a Recursion Deferred task for each proof. These tasks then are sent to the recursion thread where they're awaited and compressed into a final recursion proof.
Recursion Thread
The recursion thread waits for Prove Shard, Recursion Reduce, and Recursion Deferred tasks to complete. It creates Recursion Reduce tasks in batches until one final recursion proof remains.
Groth16/Plonk
Finally, once a final compressed proof is achieved, it is wrapped into a Groth16/Plonk proof (in a new task) if applicable.