SuperSnaps: Snarkpack for Snap deals

[cryptonemo]

We have determined that it's possible to Aggregate the ProveReplicaUpdates proofs in a manner consistent with the original Snarkpack proposal (https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0013.md) where ProveCommitSector can now be ProveCommitSectorAggregated.

In plain terms, it's possible to Snap data into many Sectors at once and then aggregate those proofs such that when the proposed ProveReplicaUpdatesAggregated is used, all of those Sectors can be activated at once. This feature would be optional and is not required to be used.

We believe this would help reduce the cost of ProveReplicaUpdates compared to how it is operating on chain today, but are curious to hear what the wider community thinks.

If we could get a thumbs up, we are in the process of drafting a FIP for this and would like to see if there is sufficient interest in using something like this.

[LaurenSpiegel]

Will this be addressed in the actual fip? protocol/CryptoEconLab#74

[jennijuju]

@scotthconner I think there is some misunderstanding.
tl'dr: The supersnap code path would be totally independent of the ProveCommit code paths which is where the batch balancer logic lives.

The reason why I recommended option 1 is because https://github.com/filecoin-project/FIPs/blob/master/FIPS/fip-0008.md was also introduced to optimize sector onboarding gas usage, and it was introduced WITHOUT the batch balancer. I think the same apply to SuperSnap - on its own, it's a gas usage saving optimization for SPs.

I dont think the batch balancer complexity is necessary and agree its somewhat broken (@Kubuxu is working on an FIP that will explains why in detail) - and thats yet another reason im recommending option 1 and I dont think it will complicates the protocol as the batch balancer code is fairy independent. As you can see here, the batch balancer fees are calculated here https://github.com/filecoin-project/builtin-actors/blob/a77674d65d42dc3547ed2753d5e8c729971920ef/actors/miner/src/monies.rs#L299-L321 and being called in provecommitaggregation message and so on https://github.com/filecoin-project/builtin-actors/blob/a77674d65d42dc3547ed2753d5e8c729971920ef/actors/miner/src/lib.rs#L941.
However, ProveReplicateUpdate is its own message, and we will simply not introduce aggregate_fee to it.

[jennijuju]

Just to elaborate more on what I meant in option 1: Introduce the aggregation without interact with batch balancer (always incentivized to aggregate with lower gas usage despite the base fee)

So the batch balancer would continue to do its thing, but in cases for snarkpack it would skip it?

Snarkpack is a term for a technology used in ProveCommitAggregate and @cryptonemo is proposing to apply it to ProveReplicaUpdate (snap). We could simply make SuperSnaps work without the batch balancer (like FIP-0008). PRU/snap deal is when a SP add data into an existing sector. AFAIK The new construction CNL is working on #587 are not relevant to adding data into sectors, but only storage onboarding.

seems like we are introducing special cases into the code/protocol spec that will further complicate the gas mechanics and make it more difficult to predict.

Not really, the proposal is to add a new functionality which does not carry old outdated technical debt (explained above).

[scotthconner]

Got it. I didn't have a clear picture how integrated the batch balancer was. If I understand you correctly, its isolated enough that introducing snark/snap as a parallel messaging it doesn't really do any harm (and also sounds like there is precedence for that). If that's the case, I'm not certain what batch balancer has to do with it, which I think was your point.

@anorth I misunderstood the dependency. Can you clarify crisply why you think this work is dependent on the batch balancer work? It doesn't seem to be consensus that it is.

[anorth]

The batch balancer does not optimise gas usage, more like the opposite. It increases costs (and protocol revenue) when proof aggregation is used. Its intention is to charge a fee when scalable methods like proof aggregation would otherwise consume sub-linear gas. I agree with Jenn that it's technically easy to either invoke it here or not, but the question is whether we should. Note that the batch balancer was subsequently added to the method introduced by FIP-0008, i.e. "should" have been there all along.

While I think the batch balancer is bad at doing its job (hence work on an alternative), it is doing an important job of ensuring network burn roughly proportional to utility provided to SPs. If we introduce proof aggregation methods without an associated fee, the network will lose this burn. SPs will enjoy this (it reduces their short-term costs) and it may be very difficult to subsequently introduce a fee to restore the balance. The construction Cryptonet are working on from #587 is relevant to adding data into sectors and will mostly likely apply here.

The best path is to replace the batch balancer with a mechanism that is better at doing that job, and then (or simultaneously) implement snap proof aggregation. If a new fee mechanism is not available, the next best path is to invoke the batch balancer for snap proof aggregation too, thus ensuring protocol burn stays in line with utility. Then when we replace the batch balancer with a better mechanism, it will be replace here too. I don't expect any of this to be made more or less difficult due to taking this path. The path of introducing aggregation without a fee could easily be a local minima from which it is governance-wise very difficult to restore network fees.

[AxCortesCubero]

Just to add CEL's position is pretty much what @anorth has described, and we are working together with Cryptonet on this mechanism that should be better at doing the batch balancer's job.

Any proposed optimization in gas usage will have an impact on economics. Whenever some optimization is proposed, everyone at CEL gets worried that it is not the right time and will throw the balance of the circulating supply dynamics out of whack. What we are working on is a mechanism to decouple these network revenue/circulating supply dynamics from gas usage, by charging directly for storage proofs, instead of for their gas usage. Once this is in place, everyone can feel free to optimize SP gas usage everywhere without worrying about the impact on economics.

[snadrus]

IF we could defer registering CC sectors until their data is snapped in, is it possible for that proof to be aggregated here?
Ref: FipD-603 (Approach 1)

[cryptonemo]

Didn't realize this wasn't linked here: #838

[mtrisic]

Hey, just caught wind of this proposal and wanted to share some thoughts 😊 . The integration of Snarkpack for Snap deals, as outlined, seems like a forward-thinking enhancement, albeit not immediately critical. Its potential for reducing the cost of ProveReplicaUpdates is intriguing, especially as we look towards the future of Filecoin's scalability and efficiency. (yeah, I believe that snapping is more efficient, especially once you seal CC remotely).

While it might not be a top priority at this very moment, the future broader adoption of "NiPoRep" suggests that the ability to aggregate proofs and snap sectors en masse could become increasingly important.

It's definitely something that could gain more importance as we advance...

[rvagg]

What's the status of this FIP? Has it gone stale, or is just looking for a champion to push it forward? It seems like there's a risk of already invested code getting too far out of date to be easily salvageable.

https://github.com/filecoin-project/bellperson/tree/feat-ipp2/src/groth16/aggregate

https://github.com/filecoin-project/rust-fil-proofs/tree/supersnaps

[cryptonemo]

The FIP was accepted a while back and the implementation (in proofs) is nearly complete and has been tested (via API and FFI). But only the implementation in rust-fil-proofs was merged into master and the other repos need some updating and integration with actors before end to end testing, etc. We have some of that work in the public repos (api/ffi).

[rvagg]

The reason this came up for me was that I was working in adjacent code and note that there's a very clean stub in builtin-actors waiting for this to be implemented in the proof system, so the lift to get it working in there probably isn't huge.