Skip to main content

Frequently Asked Questions

Network​

1. How profitable is running a prover?​

Profitability depends on hardware, electricity costs, and competition on the network. The minimal implementation provided is not expected to be competitive for generating significant revenue. It serves primarily as a reference for developers building optimized proving implementations.

2. How much $PROVE do I need to stake?​

You need to stake at least 1000 $PROVE tokens to be eligible to bid for proofs on the network. However, the staking requirements may change over time.

3. What happens if I can't complete a proof in time?​

If your prover accepts a request but fails to deliver a proof before the deadline, a portion of your stake may be slashed as a penalty. This ensures the reliability of the network.

Slashing is not currently enabled.

4. What happens if my prover goes offline?​

If your prover goes offline, it won't be able to bid for or complete proofs. There's no penalty for simply being offline, but if your prover was in the middle of generating a proof and fails to deliver it before the deadline, a portion of your stake may be slashed as a penalty. This ensures the reliability of the network.

Slashing is not currently enabled.

5. How can I unstake my $PROVE tokens?​

You can unstake your tokens through the frontend or by directly interacting with the SuccinctStaking contract. Note that there may be a cooldown period before tokens can be withdrawn after unstaking.

Cluster​

6. Why does a single Fibonacci proof stop getting faster when I add more GPUs?​

The CPU worker that executes the program limits how many GPUs one proof can use. One proof keeps roughly 70 to 90 NVIDIA L4 GPUs busy. Faster GPUs reach the limit at a lower count.

The CPU worker runs the program and divides the work into shards for the GPUs. Eventually, the GPUs prove shards as fast as the CPU worker can produce them. Adding more GPUs cannot make that proof finish sooner.

A CPU with faster single-core performance can produce shards faster and raise the limit.

Extra GPUs can still increase total cluster throughput. Run several proofs at the same time to measure that gain.

7. Why is my GPU VRAM usage so low?​

SP1 proof generation depends more on compute speed and memory bandwidth than VRAM capacity. Low VRAM use does not always mean the GPU is underused.

For SP1 v6, use a GPU with at least 24 GB of VRAM. More VRAM does not always improve performance.

8. How can I tell if I am getting good performance out of my cluster? The CPU/GPU utilization seems low.​

Ensuring the cluster is performant is a complex problem. There are many factors that could affect performance.

Measuring performance​

Firstly, it's important to get an accurate measurement of the resulting performance from your cluster. You can do this using the cluster CLI:

docker run --rm -it --network=infra_default ghcr.io/succinctlabs/sp1-cluster:base-latest \
  /cli bench fibonacci 300 -c 2 \
  --cluster-rpc http://api:50051 \
  --redis-nodes redis://:redispassword@redis:6379/0

You should increase --count based on how many GPUs you have. Start with # GPUs divided by the per-proof GPU limit from FAQ #6 (roughly 80 for L4s), and try increasing it by 1 until aggregate throughput does not improve anymore.

This should output something like:

2025-09-10T21:03:55.029021Z  INFO crates/common/src/logger.rs:110: logging initialized    
2025-09-10T21:03:55.029688Z  INFO bin/cli/src/commands/bench.rs:71: Running 1x Fibonacci Compressed for 5 million cycles...
...
2025-09-10T21:03:56.373132Z  INFO bin/cli/src/commands/bench.rs:150: base_id: cli_1757538236373
2025-09-10T21:04:02.425794Z  INFO bin/cli/src/commands/bench.rs:215: Completed after 5.969402833s
2025-09-10T21:04:02.425818Z  INFO bin/cli/src/commands/bench.rs:217: Total Cycles: 5000000 | Aggregate MHz: 0.84

The output Aggregate MHz is the aggregate proving throughput of your cluster observed from the test.

In our experience, we tend to observe ~2 million PGU/s of proving throughput per GPU node for higher performance GPUs (ex. RTX 4090/5090) and closer to 1m PGU/s for less powerful GPUs (ex. L4). Other factors such as GPU bandwidth / PCIe lane count, CPU cores/clock speed, CPU memory speed and network bandwidth can also affect overall performance.

Potential bottlenecks​

There are a few likely bottlenecks that can affect performance:

  • Uploading/downloading circuit artifacts takes too much time (>300ms)
    • You can verify whether this is the case by inspecting the CPU/GPU node logs for download/upload span times (ex. download: close time.busy=10ms time.idle=1200ms) or by setting up distributed tracing and inspecting the traces for unusually large spans.
    • Redis nodes could be bound by compute, memory, or networking. You can scale the artifact store load horizontally by simply setting up additional Redis instances and adding them (comma-separated) to the REDIS_NODES env var. (ex. REDIS_NODES=redis://:redispassword@redis1:6379/0,redis://:redispassword@redis2:6379/0). Also ensure that persistence is disabled for your Redis instance (persistence.enabled: false or REDIS_AOF_ENABLED=no) as it is not needed. You can also try tuning other Redis parameters such as io-threads and hz.
    • You should not use S3 as the artifact store unless you are running in AWS due to the increased data transfer cost and latency.
  • GPU nodes are not tuned properly, meaning the node is not able to provide enough work to the GPU itself to keep it busy.
    • Check the worker's computed worker max_weight startup log. See FAQ #13 before changing WORKER_MAX_WEIGHT_OVERRIDE.
    • You can measure this by setting up Prometheus metrics and the provided Grafana dashboard and inspecting the "Active GPUs" chart while a proof runs for several minutes. This chart has "Estimated Active GPUs" which reflects total GPU utilization over time. A low value relative to # of total GPUs means some GPUs were idle for a majority of the interval. You can also measure GPU utilization using nvidia-smi or similar tools.
  • Network operations between the coordinator and nodes has too much latency.
    • These should all be running on the same internal network, ideally with at least 10Gbps bandwidth.
  • There are not enough CPU nodes to run proofs in parallel.
    • Compare each CPU worker's queued task weight against its computed worker max_weight startup log (see FAQ #13). A full budget means tasks queue behind it.

Miscellaneous issues​

  • WORKER_TYPE=ALL requires the GPU-enabled node image and makes the worker eligible for both CPU and GPU tasks. Use WORKER_TYPE=CPU with the CPU node image; setting it to ALL causes the node to exit during startup.

9. Why are my Groth16/Plonk proofs timing out / failing?​

If your prover is consistently failing or timing out on Groth16 or Plonk proofs, it's likely that circuit artifacts were not downloaded properly or your CPU node is not performant enough.

Artifacts setup​

Check your CPU node logs for messages like the following:

2025-08-12T00:44:40.415405Z  INFO task: /app/crates/worker/src/tasks/finalize.rs:55: Waiting for circuit artifacts to be ready     proof_id="28cf80a36990e85a064da2d9ccf82a7dfc7aeecd57a95aeb3353143f14b5102c" task_id="task_01k2dtppn8fe9v7az1mek4rnvg" otel.name="PLONK_WRAP"
2025-08-12T00:44:45.415810Z  INFO task: /app/crates/worker/src/tasks/finalize.rs:55: Waiting for circuit artifacts to be ready     proof_id="28cf80a36990e85a064da2d9ccf82a7dfc7aeecd57a95aeb3353143f14b5102c" task_id="task_01k2dtppn8fe9v7az1mek4rnvg" otel.name="PLONK_WRAP"
2025-08-12T00:44:50.417412Z  INFO task: /app/crates/worker/src/tasks/finalize.rs:55: Waiting for circuit artifacts to be ready     proof_id="28cf80a36990e85a064da2d9ccf82a7dfc7aeecd57a95aeb3353143f14b5102c" task_id="task_01k2dtppn8fe9v7az1mek4rnvg" otel.name="PLONK_WRAP"

This suggests that the CPU worker could not download the circuit artifacts. A mounted cache directory might not be writable. Current images store circuit artifacts under /var/cache/sp1/circuits. Make sure the mounted directory is writable by UID and GID 1001:1001. Look for logs like these when the CPU worker starts:

2025-08-12T00:54:29.242040Z  INFO bin/node/src/main.rs:207: worker type: Cpu
2025-08-12T00:54:29.242178Z  INFO bin/node/src/main.rs:215: downloading circuit artifacts
thread 'tokio-runtime-worker' panicked at /usr/local/cargo/git/checkouts/sp1-9091391fc1cd5ab7/6544380/crates/sdk/src/install.rs:85:41:
failed to create build directory: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
2025-08-12T00:54:29.244128Z  INFO bin/node/src/main.rs:123: Not creating circuits dir: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }
2025-08-12T00:54:29.244158Z  INFO bin/node/src/main.rs:133: Not creating temp dir: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }

When the artifacts are correctly setup, you'll see a message like circuit artifacts ready after 3 min.

CPU wrap proof is too slow​

If circuit artifacts are ready but Groth16 or Plonk proofs still time out, check the CPU worker's max_weight and CPU resources. A wrap task must fit on one worker. More CPU cores can reduce wrap time. If timeouts continue, you can:

  • Increase BIDDER_BUFFER_SEC for more general slack, or increase the relevant wrap-time estimate (BIDDER_GROTH16_BUFFER_SEC or BIDDER_PLONK_BUFFER_SEC) to make bidding more conservative.
  • Temporarily disable a mode: set BIDDER_GROTH16_ENABLED=false or BIDDER_PLONK_ENABLED=false to avoid bidding on that proof type.

10. Why is my Redis node using so much disk space?​

Make sure persistence is disabled for Redis (persistence.enabled: false for Helmchart or REDIS_AOF_ENABLED=no for docker image) as it is not needed. This should prevent Redis from using any disk space. You can also try tuning other Redis parameters such as io-threads and hz.

10a. My long-running proofs fail on missing artifacts​

Stdin and intermediate artifacts expire from Redis 6 hours after they are written, set by ARTIFACT_TIMEOUT_SECONDS. A proof still running past that window fails when a task reads an artifact that has already expired. Program artifacts are exempt and never expire.

The expiry is applied by whichever process writes the artifact, so raise the value on every service that touches the artifact store, not just one. Longer retention raises steady-state Redis memory — size your Redis for it.

10b. Why does my CPU worker accept no tasks or fail with "Program simulation failed"?​

CPU workers use /dev/shm during program simulation. Docker provides only 64 MB by default.

Without an override, Docker's 64 MB default makes the worker report zero capacity and accept no tasks. WORKER_MAX_WEIGHT_OVERRIDE lets the worker accept tasks (FAQ #13), but it does not increase /dev/shm. Simulation can then crash when shared memory fills. On SP1 v6 that crash reports Child native executor crashed, details: CrashDetails { signal: 7, ... } — signal 7 is SIGBUS.

Set shm_size on the CPU/mixed service in your Compose file to match the container's memory limit — the provided docker-compose.yml already does this. On Kubernetes, mount a memory-backed emptyDir at /dev/shm; the bundled cpu-node chart does this by default.

11. What's the optimal configuration for my cluster hardware?​

For optimal performance, we recommend the following:

  • CPU nodes
    • Use at least 32 GB of RAM per concurrent proof, and at least 96 GB for a CPU worker that runs Plonk wraps. Configure worker capacity.
    • Use a high CPU core count for Groth16 and Plonk wraps.
  • GPU nodes
    • RTX 5090 and 4090 GPUs perform best. Use GPUs with at least 24 GB of VRAM. See FAQ #7 for more VRAM guidance. Use one PCIe x16 connection per GPU for best memory throughput.
    • For non-VRAM: DDR5 RAM is recommended, and 32 GB per GPU is recommended.
    • Give each GPU at least four CPU cores. Faster GPUs need more CPU cores to stay busy.
  • 10 Gbps internal networking recommended.
  • If running more than 8 GPUs, multiple Redis nodes are recommended. (See FAQ #8)

12. How should I configure the bidder?​

The bidder estimates whether the cluster can finish each open request before its deadline, and bids only when it can. Assigned proofs and unsettled bids count as committed work, so admitted requests cannot overload the cluster.

Set BIDDER_THROUGHPUT_MGAS to the cluster's measured sustained throughput. Set BIDDER_CPU_WORKER_MAX_WEIGHTS to each CPU worker's max_weight (see FAQ #13).

The bidder guide documents every variable, its default, and the bid-price rules.

13. How do worker weights control cluster capacity?​

Workers use a weight budget to limit how much work they run at once. max_weight is that budget. Each task has a weight that estimates the memory it needs, in GiB. A worker accepts tasks until their total weight reaches max_weight.

By default, max_weight is the smaller of the worker's available RAM and /dev/shm. /dev/shm is shared memory used during program simulation, so it can limit the worker even when RAM is still available. WORKER_MAX_WEIGHT_OVERRIDE changes the admission limit but does not add RAM or /dev/shm. Setting it too high can cause out-of-memory errors or simulation failures.

The bidder uses the same capacity when it decides whether to accept new requests. Set BIDDER_CPU_WORKER_MAX_WEIGHTS to the max_weight reported by each CPU worker. For two workers that each report 96, use BIDDER_CPU_WORKER_MAX_WEIGHTS=96,96.