The generation of zk-SNARKs quickly became a bottleneck, so the expensive parts (FFT and MultiExponentiation) were implemented on the GPU using OpenCL and CUDA. Because the core operations supporting proof generation are highly parallelizable, we take advantage of modern general-purpose GPU computing to off-loading them from the CPU.

This allows a much higher throughput, while also creating economic efficiency. By moving the parallelizable work to relatively inexpensive parallel processors, we keep main memory and CPU free for the highly-sequential and memory-intensive workloads used to create the data miners must prove.

Even though Batch Verification helped, we needed faster verification, so we implemented SnarkPack. This allows us to aggregate many zk-SNARKs into a single combined proof. Not only does this optimization reduce verification time by a factor of more than 10x at scale — it also reduces chain bandwidth by reducing the average bytes-per-proof which must be submitted to the chain.

In order to accomplish this, we built on the research on the Inner Product Argument — and collaborated with the authors to extend it to support our needs without requiring a new trusted setup. We accomplished this by adapting the techniques to securely apply using two existing Powers of Tau trusted setups. This is a great example of how we have historically had to pick our way through obstacles to the practical realization of groundbreaking scale.