Skip to content

Instantly share code, notes, and snippets.

@denniswon
Created April 24, 2026 14:14
Show Gist options
  • Select an option

  • Save denniswon/c9e73e39e14f8de1230d4253f46f6bda to your computer and use it in GitHub Desktop.

Select an option

Save denniswon/c9e73e39e14f8de1230d4253f46f6bda to your computer and use it in GitHub Desktop.
Private Data Storage - Subproject B
⏺ You're right — I was stuck in the wrong substrate model. Let me course-correct.
★ Insight ─────────────────────────────────────
- I was treating "state tree" as the private-data substrate and "EVM contract storage" as the public-data substrate, as if Sub-project A's
fabric didn't apply here. It does apply. The state tree is a content-addressed key-value store with on-chain root commitments — it has nothing
inherently private about it. Encrypted leaves = private data (Sub-project A's SecretsRegistry); plaintext leaves = public data (Sub-project B's
artifacts). The substrate is the same, only leaf encryption differs.
- On-chain, we only ever need the hash + root commitments. For a specific task, NewtonPolicy.policyCodeHash already commits to the Rego bytes —
any fetcher verifies keccak256(bytes) == policyCodeHash and that closes the integrity loop regardless of where bytes live. For registry-level
questions ("is this plugin still approved?"), the 120s BLS-signed state root + inclusion proof does the job. SSTORE2 never enters the picture
because EVM itself never consumes the bytes.
- This collapses Sub-project B's infrastructure cost dramatically. The state-tree + EigenDA + BLS-commit + bootnode + gateway-orchestration
fabric is already being built in Sub-project A. Sub-project B adds more registries on top of that fabric — different data types, different leaf
semantics, but the same plumbing. It's additive, not a parallel track.
─────────────────────────────────────────────────
Corrected model — where each thing lives
┌──────────────────────────────┬────────────────────────────────────────────────┬────────────────────────────┬────────────────────────────┐
│ What │ Where │ Who writes │ Who reads │
├──────────────────────────────┼────────────────────────────────────────────────┼────────────────────────────┼────────────────────────────┤
│ keccak256(rego_bytes) → Rego │ Operator-local state tree (plaintext leaf), │ Gateway-orchestrated │ Operators / challengers / │
│ source │ backed by EigenDA delta blob │ operator broadcast on │ SP1 prover, via signed RPC │
│ │ │ policy deploy │ + inclusion proof │
├──────────────────────────────┼────────────────────────────────────────────────┼────────────────────────────┼────────────────────────────┤
│ keccak256(schema_bytes) → │ │ │ Same — signed RPC from │
│ data.params / data.wasm │ Same — state tree leaf │ Same flow │ operator │
│ schema │ │ │ │
├──────────────────────────────┼────────────────────────────────────────────────┼────────────────────────────┼────────────────────────────┤
│ keccak256(wasm_bytes) → WASM │ Same — state tree leaf (MB-scale leaf is fine; │ Same │ Same │
│ plugin binary │ redb handles arbitrary value sizes) │ │ │
├──────────────────────────────┼────────────────────────────────────────────────┼────────────────────────────┼────────────────────────────┤
│ │ Per-registry BLS-signed state root, committed │ │ EVM — used for │
│ On-chain commitment │ every 120s to the registry contract │ Operator quorum (same as │ inclusion-proof │
│ │ (implementing IStateRootCommittable) │ Sub-project A) │ verification in challenge │
│ │ │ │ paths │
├──────────────────────────────┼────────────────────────────────────────────────┼────────────────────────────┼────────────────────────────┤
│ │ NewtonPolicy.policyCodeHash continues to point │ │ SP1 circuit / challenger / │
│ Existing on-chain hashes │ at the registered Rego hash; NewtonPolicyData │ Policy deployer │ operator (already the │
│ │ gets a wasmHash field replacing wasmCid │ │ four-site commitment │
│ │ │ │ pattern) │
└──────────────────────────────┴────────────────────────────────────────────────┴────────────────────────────┴────────────────────────────┘
What goes on-chain: hashes (as they already do via policyCodeHash) + per-registry state roots + registry metadata (deprecation flags, deploy
timestamps). That's it.
What goes in operator state + EigenDA: the actual bytes.
What disappears: IPFS dependency, SSTORE2, MB-scale EVM storage cost. The "on-chain" guarantee in the brief is satisfied by the committed state
root, not by bytes-in-EVM.
What I was misunderstanding
I was holding onto the EVM smart-contract analogy from your brief ("like how smart contracts are deployed/stored onchain") too literally —
treating it as "bytes physically in contract code." You were using it as "protocol-committed content with on-chain trust anchors, fetchable
from the protocol without external dependency." Under that second reading (which is actually what the brief requires), the state-tree substrate
is a cleaner fit than SSTORE2.
What this resolves
- Q3 (storage substrate): state tree on the Sub-project A fabric, not SSTORE2. Applies uniformly to all four artifacts.
- E5 for all four: confirmed — this is literally what the state-tree read path does (gateway-orchestrated for writes, operator signed-RPC reads
for fetches, EigenDA as the blob-availability layer for snapshot/replay).
- Q2 (addressing): content-hash remains the primary key. In state-tree terms: the tree key is the content hash.
- Privacy extension (R5): free — encrypted leaves work the same as SecretsRegistry. The on-chain registry stores (cleartext_hash,
TEE_attestation_requirement_flag) in its metadata; the leaf stores ciphertext.
- Infrastructure reuse: Sub-project B becomes "three or four more IStateRootCommittable registries" (RegoRegistry + SchemaRegistry(ies) +
WasmPluginRegistry), reusing the state-tree crate, EigenDA client, bootnode, gateway orchestration, and BLS commit flow from Sub-project A
wholesale.
What opens up next
A few concrete decisions remain:
1. One registry vs several — single unified PolicyArtifactRegistry with typed entries (kind: Rego / Schema / WasmPlugin) sharing one state
root, or separate RegoRegistry / SchemaRegistry / WasmPluginRegistry each with its own state root? Sub-project A chose separate (Identity /
Confidential / Secrets) to keep trust boundaries clean. Same logic probably applies here.
2. Registry on which chain(s)? Sub-project A's three registries live on source chain (per your Linear issue decomposition). Policy artifacts
probably same — but worth confirming because destination-chain challengers need to verify artifact presence too, which means either dest-chain
registries or cross-chain root relay.
3. State-root commit cadence for artifact registries — Sub-project A uses 120s across all three registries. Do we lock-step all six+ registries
(Sub-project A's three + Sub-project B's new ones) at the same 120s cadence? It simplifies operators' commit loop significantly.
4. Model X vs Model Y for WASM leaves — leaf stores bytes directly (X), or leaf stores metadata + EigenDA blob pointer (Y)? Model X makes state
tree a single source of truth but grows the tree storage footprint; Model Y keeps the tree light but adds a fetch indirection. Sub-project A
uses Model X for identity/confidential data (KB-scale leaves); for MB-scale WASM, Y might be preferable.
Which of these do you want to dig into first? Q4 (single vs several registries) feels like the most foundational — it decides the on-chain
contract surface and the trust boundaries between artifact kinds.
You're right — I got Sub-project A's topology wrong in two places. Let me correct both before recommending on Q4.
★ Insight ─────────────────────────────────────
- Sub-project A's registry topology is unified, not split. The V2 design collapsed IdentityRegistry / ConfidentialDataRegistry /
SecretsRegistry (V1) into one StateCommitRegistry per chain with 1-byte namespace prefixes (0x01 identity / 0x02 confidential / 0x03
secrets) addressing a single unified JMT root (spec §3, lines 14, 68, 106, 194, 465). The spec explicitly rejects per-namespace
operator-set partitioning (§S line 444): "it was the sole benefit of the V1 three-registry shape, and the cross-namespace policy-eval
dependency made that partitioning useless in practice."
- Sub-project A's registries deploy to every chain, not source-only. Spec §line 412 — "Three registries deploy per chain Newton
supports"; §line 415 — "Destination chains (Base, L2s): full registry — operators maintain per-chain subtrees and post commits per
chain"; §line 465 — "One StateCommitRegistry MUST be deployed per chain Newton supports. Each chain's registry is authoritative for
that chain's unified state root and sequenceNo." No cross-chain state-root sync in Phase 1.
- The grill-me context line at the top of this session ("Three registries with shared IStateRootCommittable interface — not a single
PrivateDataStore") is stale — it reflects an earlier iteration of Sub-project A's design. The written spec supersedes it. I should
have verified against the doc before anchoring recommendations to that line.
─────────────────────────────────────────────────
Answers
(2) Chain placement — you were right. Sub-project A's StateCommitRegistry lives on every chain Newton supports (source AND
destinations), one per chain, each authoritative for its own unified state root. No cross-chain sync in Phase 1. So for Sub-project
B: same default — one policy registry per chain. Dest-chain challengers get the inclusion-proof substrate for free. Flag for Q5
confirmation later, but this is the baseline.
(3) 120s lockstep — locked in. All Sub-project A + Sub-project B registries commit on the same 120s boundary per chain.
(1) Q4 recommendation — don't split. Unify into either one or two state trees.
My prior "Sub-project A chose separate, same logic applies" was based on the wrong premise. The correct analogue is the opposite:
Sub-project A unified, so Sub-project B should unify too. Two sub-options remain:
Option: 1A. Extend existing StateCommitRegistry
What it is: Add namespace tags 0x04 Rego / 0x05 Schema / 0x06 WasmPlugin to the same JMT. One root, one BLS sig, one commit tx per
chain per 120s.
Trust boundary: Same operator quorum already signs across identity/confidential/secrets. Extending to policy artifacts is additive.
Plumbing cost: Zero new contracts. One more namespace prefix per artifact kind.
────────────────────────────────────────
Option: 1B. Second unified registry (PolicyArtifactRegistry)
What it is: Separate JMT for policy artifacts, same IStateRootCommittable interface, parallel 120s commit.
Trust boundary: Separate state root lets you bound "I trust the private-data quorum but not the policy-artifact quorum" — useful if
the operator sets ever diverge.
Plumbing cost: One new contract per chain. One more BLS aggregation + commit tx per 120s. Possible to run on the same operator set
❯ 1A agreed. Also, for (4), for cost, performance, reliability, what do you recommend?
⏺ 1A locked in. StateCommitRegistry extends with namespace tags 0x04 Rego / 0x05 Schema / 0x06 WasmPlugin. Zero new contracts, same
operator quorum, same 120s lockstep commit.
---
Model X vs Model Y on cost / performance / reliability
Recommend Model X with per-commit size guard on all three axes. Reliability is the decisive one.
Cost
┌──────────────────────────┬───────────────────────────────────┬──────────────────────────────────────────┐
│ Dimension │ Model X (bytes in leaf) │ Model Y (hash in leaf, bytes in EigenDA) │
├──────────────────────────┼───────────────────────────────────┼──────────────────────────────────────────┤
│ Per-operator disk │ ~200 MiB–1 GiB for 20–50 policies │ ~200 KiB (just hashes + metadata) │
├──────────────────────────┼───────────────────────────────────┼──────────────────────────────────────────┤
│ EigenDA write cost │ Bytes in delta blob │ Bytes in separate artifact blob (~same) │
├──────────────────────────┼───────────────────────────────────┼──────────────────────────────────────────┤
│ Bootnode/challenger disk │ Full bytes (~1 GiB) │ Hashes + ephemeral cached bytes │
├──────────────────────────┼───────────────────────────────────┼──────────────────────────────────────────┤
│ Network replication │ N operators × 1 GiB │ 1× EigenDA + per-operator cache │
└──────────────────────────┴───────────────────────────────────┴──────────────────────────────────────────┘
Y saves disk, but at Phase 1 scale (20–50 policies × 10–30 MiB ≈ <2 GiB total network-wide), savings are trivial. Modern SSDs are
TB-scale; redb handles it fine. Neither model is cost-constrained at Phase 1.
Performance
┌────────────────────────────┬──────────────────────────┬────────────────────────────────────┐
│ Path │ Model X │ Model Y │
├────────────────────────────┼──────────────────────────┼────────────────────────────────────┤
│ Eval hot path (cache hit) │ 1 redb read │ 1 redb read + cache hit │
├────────────────────────────┼──────────────────────────┼────────────────────────────────────┤
│ Eval hot path (cache miss) │ local bytes (~1ms) │ network fetch (~10–200ms) │
├────────────────────────────┼──────────────────────────┼────────────────────────────────────┤
│ Cold start / new operator │ snapshot = bulk transfer │ snapshot + N parallel WASM fetches │
├────────────────────────────┼──────────────────────────┼────────────────────────────────────┤
│ First-run wasmtime compile │ 100ms–1s (dominates) │ 100ms–1s (same) │
└────────────────────────────┴──────────────────────────┴────────────────────────────────────┘
Hot path is equivalent (both cache). Bootstrap favors X (single bulk transfer). First-run compilation dominates in both. Performance
difference is not decisive.
Reliability (decisive axis)
Model X:
- Single substrate: redb + JMT
- Delta blob = transport only (ephemeral, pruneable after apply)
- Tree = authoritative storage
- Inclusion proof verifies bytes directly
- No dangling-reference risk
- Matches Sub-project A's existing transport/storage split exactly
Model Y:
- Two-tier substrate: JMT (for hash) + EigenDA (for bytes)
- EigenDA becomes a storage medium, not just transport
- New retention invariant: "WASM artifacts live in EigenDA indefinitely" (today delta blobs can be pruned after consumption)
- Two-step verification: prove hash via JMT, then verify fetched bytes
- Dangling-reference risk if artifact blob is pruned
- Cache eviction + EigenDA outage = unreadable policy = cannot evaluate tasks
★ Insight ─────────────────────────────────────
The core architectural question here is whether EigenDA is transport or storage. Sub-project A treats it as transport — delta blobs
are consumed by bootnode/operators/challengers, applied to the authoritative tree, and become pruneable. Model Y promotes EigenDA to
a storage medium with forever-retention requirements, which is a fundamentally different reliability posture. Once that invariant
exists for one artifact class, it creates pressure to reuse it for others, and a new substrate responsibility has quietly been added
to the system. That's worth avoiding unless the cost/performance case is overwhelming — and at Phase 1 scale, it isn't.
─────────────────────────────────────────────────
Recommendation
Model X + per-commit size guard (~50 MiB cap per commit, roll excess to next commit).
- Cost: trivial at Phase 1 scale
- Performance: equivalent hot path, slight bootstrap edge
- Reliability: clearly superior — single substrate, no new EigenDA retention invariant, no dangling references
Two caveats worth flagging
1. Delta blob size limit: EigenDA has a per-blob cap (currently 16 MiB decoded, I think — needs verification before finalizing the 50
MiB commit cap). If a commit's total leaf bytes exceed the blob limit, we either (a) lower the per-commit cap, or (b) split the
delta across multiple blobs. Worth nailing down before writing the spec.
2. Future WASM opcode fee mechanism (the TODO at crates/core/src/config/data_provider.rs:28-30): If/when pay-per-byte pricing lands,
Model Y becomes attractive again (bytes live once in EigenDA, fees charged per-byte, no replication cost). But for Phase 1 (no fee
mechanism), Model X wins on reliability and the cost savings from Y are marginal.
---
Model X + 50 MiB per-commit cap looks solid — want to move on to Q5 (chain placement) next, or dig into one more Q4 sub-branch first
(e.g., the policy-artifact write path: who authors a new Rego deployment, which gateway endpoint, what BLS-quorum validation on
registration)?
✻ Sautéed for 1m 51s
※ recap: Designing Sub-project B (on-chain policy storage for Rego/WASM/schema). Locked Option 1A (extend StateCommitRegistry with
namespaces 0x04/0x05/0x06) and Model X with 50 MiB per-commit cap. Next: move to Q5 on chain placement, or dig into the
policy-artifact write path first. (disable recaps in /config)
❯ let's do cost projection when going with model X
⏺ Cost projection: Model X, Phase 1
Bottom line up front: ~$10–50/month network-wide across all dimensions. At Phase 1 scale, the Newton AVS pays approximately the same
as it currently pays for IPFS pinning — the replicated redb storage is not the constraint.
Let me walk through the dimensions with assumptions so you can challenge any that don't match your mental model.
Assumptions (pls correct if off)
┌──────────────────────┬─────────────────────────────────────────┬───────────────────────────────────────────────────────────────┐
│ Parameter │ Phase 1 estimate │ Notes │
├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ Policy count │ 20–50 │ Early Phase 1 steady state │
├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ Rego source per │ ~5 KiB avg │ Typical Rego ~100–500 lines │
│ policy │ │ │
├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ Schema JSON per │ ~3 KiB avg │ Small │
│ policy │ │ │
├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ WASM binary per │ ~15 MiB avg (5–30 MiB) │ Based on observed sizes; max_cache_size 100 MiB is hard upper │
│ policy │ │ bound │
├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ Operators per chain │ ~10 │ Based on current stagef/prod patterns │
├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ Chains supported │ 3 │ Sepolia + Base Sepolia + one L2 (or mainnet eq.) │
├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ Replicas per chain │ 12 │ 10 operators + 1 bootnode + 1 challenger │
├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ Commit cadence │ 120 s │ 720 commits/day/chain │
├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ Policy deployment │ 1–2/day during launch → ~0/day steady │ Rego updates rare post-launch │
│ rate │ state │ │
└──────────────────────┴─────────────────────────────────────────┴───────────────────────────────────────────────────────────────┘
1. On-chain gas cost (the biggest variable)
Each 120 s commit is one tx: commitStateRoot(newRoot, sig) → one SSTORE + one event emission + BLS cert verification inline.
┌───────────────────────────┬────────────────┬─────────────┬─────────┬─────────────┬──────────────┐
│ Chain │ Gas per commit │ Commits/day │ Gas/day │ Daily cost* │ Monthly cost │
├───────────────────────────┼────────────────┼─────────────┼─────────┼─────────────┼──────────────┤
│ Sepolia (1 gwei) │ ~80k │ 720 │ 57.6M │ $0.14 │ $4.20 │
├───────────────────────────┼────────────────┼─────────────┼─────────┼─────────────┼──────────────┤
│ Base Sepolia (~0.01 gwei) │ ~80k │ 720 │ 57.6M │ $0.01 │ $0.30 │
├───────────────────────────┼────────────────┼─────────────┼─────────┼─────────────┼──────────────┤
│ Mainnet L1 (30 gwei) │ ~80k │ 720 │ 57.6M │ $4.30 │ $130 │
├───────────────────────────┼────────────────┼─────────────┼─────────┼─────────────┼──────────────┤
│ Mainnet L2 (~0.1 gwei) │ ~80k │ 720 │ 57.6M │ $0.05 │ $1.50 │
└───────────────────────────┴────────────────┴─────────────┴─────────┴─────────────┴──────────────┘
*ETH price $2,500 assumed.
This is the dominant line item at scale, not storage. If we launch on all three mainnet chains (Ethereum + Base + Arbitrum), commit
gas lands around $140/month; testnet-only deployments are <$5/month.
Mitigations if this becomes a constraint:
- Skip no-op commits (don't commit if delta is empty — only commit when state advances)
- Stretch commit cadence on idle chains (already contemplated in §S.13 safety)
- Batch multiple namespace updates into single tx (already the case with 1A)
2. Off-chain storage (redb per replica)
Per-policy artifact set:
┌─────────────────────────────────────────┬──────────────────────────┐
│ Item │ Size │
├─────────────────────────────────────────┼──────────────────────────┤
│ Rego leaf │ ~5 KiB │
├─────────────────────────────────────────┼──────────────────────────┤
│ Schema leaf │ ~3 KiB │
├─────────────────────────────────────────┼──────────────────────────┤
│ WASM leaf │ ~15 MiB │
├─────────────────────────────────────────┼──────────────────────────┤
│ JMT internal nodes per leaf (amortized) │ ~500 B │
├─────────────────────────────────────────┼──────────────────────────┤
│ Leaf header (LeafHeaderV1) │ 20 B │
├─────────────────────────────────────────┼──────────────────────────┤
│ Per-policy total │ ~15 MiB (WASM dominates) │
└─────────────────────────────────────────┴──────────────────────────┘
At 50 policies:
- Raw artifacts: 750 MiB
- JMT overhead (~20%): 150 MiB
- redb metadata + compaction slack: ~100 MiB
- Per replica: ~1 GiB
Network-wide:
- 3 chains × 12 replicas × 1 GiB = 36 GiB aggregate
- At AWS EBS gp3 (~$0.08/GB/month): $2.90/month across entire network
This is trivial. Even at 10× the policy count (500 policies → 10 GiB/replica × 36 replicas = 360 GiB), monthly cost is ~$29
network-wide.
3. EigenDA delta blob cost
Delta blob composition per 120 s commit:
- Steady state (no artifact changes): ~2 KiB (sequence number bump + JMT internal node changes)
- Policy deployment: ~15 MiB (new WASM) + ~10 KiB (Rego + schema + headers)
┌───────────────────────────────────┬─────────────────────────────┬───────────────┐
│ Day type │ Per-chain EigenDA bytes/day │ 3-chain total │
├───────────────────────────────────┼─────────────────────────────┼───────────────┤
│ Steady state │ 720 × 2 KiB = 1.4 MiB │ 4.3 MiB/day │
├───────────────────────────────────┼─────────────────────────────┼───────────────┤
│ Launch week (1–2 deployments/day) │ ~30 MiB │ ~90 MiB/day │
├───────────────────────────────────┼─────────────────────────────┼───────────────┤
│ Heavy deployment day │ ~100 MiB │ ~300 MiB/day │
└───────────────────────────────────┴─────────────────────────────┴───────────────┘
At EigenDA Phase 1 indicative pricing (~$0.001/MiB, dropping with volume):
- Steady state: ~$0.15/month network-wide
- Heavy deployment month: ~$10/month
Basically noise.
4. RAM / compute (no delta vs today)
- Moka WASM cache: already max_cache_size: 100 MiB per operator (existing config, no new demand)
- wasmtime compile: 100 ms–1 s per WASM on first use — same as today (no regression)
- JMT proof generation during eval: <1 ms per proof (proof is ~600 B for depth-20 JMT)
- Delta apply per 120 s commit: ~50 ms on operator side (negligible)
- BLS commit sig: ~20 ms per operator per commit
Total per-operator compute tax from state commits: ~70 ms × 720/day = 50 seconds of CPU/day. Trivial.
5. Bandwidth
Per operator per day:
- Inbound delta fetches: 720 × 2 KiB + occasional 15 MiB bursts ≈ 30–50 MiB/day
- Outbound BLS sig publish: 720 × 96 B = 70 KiB/day
Per-month per-operator bandwidth: ~1.5 GiB → <$0.15/month at AWS egress rates.
Snapshot bootstrap (new operator joining):
- 1 GiB one-shot pull from bootnode
- <$0.10 per event, rare
6. Aggregate Phase 1 cost projection
┌────────────────────────┬─────────────────────────────┬──────────────────────────────────────────┐
│ Category │ Monthly cost (testnet only) │ Monthly cost (mainnet: ETH + Base + Arb) │
├────────────────────────┼─────────────────────────────┼──────────────────────────────────────────┤
│ On-chain commit gas │ $5 │ $135 │
├────────────────────────┼─────────────────────────────┼──────────────────────────────────────────┤
│ Off-chain redb storage │ $3 │ $3 │
├────────────────────────┼─────────────────────────────┼──────────────────────────────────────────┤
│ EigenDA delta blobs │ $1 │ $1 │
├────────────────────────┼─────────────────────────────┼──────────────────────────────────────────┤
│ Bandwidth │ $5 (10 ops) │ $5 │
├────────────────────────┼─────────────────────────────┼──────────────────────────────────────────┤
│ Total │ ~$14 │ ~$144 │
└────────────────────────┴─────────────────────────────┴──────────────────────────────────────────┘
For comparison, current IPFS pinning cost (assuming Pinata or similar at $20 / 1 TB / month tier) is in the same order of magnitude.
★ Insight ─────────────────────────────────────
The real "cost" of Model X isn't a dollar figure — it's the per-commit gas tax amortized across idle chains. A chain with no policy
activity still pays ~$4/day on L1 for committing a state root that didn't move. That becomes the dominant line item as chain count
grows. The two paths to mitigate are (a) skip commits when delta is empty, which preserves the 120 s guarantee window but breaks the
deterministic cadence (and is a §S.13 design choice), or (b) stretch cadence on idle chains. Both are adjustments to the commit
cadence policy, not to Model X itself.
─────────────────────────────────────────────────
Scale sensitivities
What makes the projection move in non-linear ways:
┌──────────────────────────────────┬──────────────────────────────────────┬──────────────────────────────────────────────────────┐
│ Dimension │ Impact of 10× growth │ Becomes problematic when │
├──────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────┤
│ Policy count (50 → 500) │ redb: 1 GiB → 10 GiB/replica │ Never at Phase 1 scale; operators on small VMs might │
│ │ │ notice │
├──────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────┤
│ WASM size (15 MiB → 150 MiB avg) │ Breaks the 100 MiB cache; commit cap │ Need to revisit the max_cache_size config + │
│ │ blocks │ per-commit cap │
├──────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────┤
│ Chain count (3 → 10) │ Gas cost scales linearly │ Mainnet × 10 chains = ~$450/month just for commits │
├──────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────┤
│ Policy deployment rate (2/day → │ EigenDA: $1 → $25/month │ Still manageable; per-commit cap protects against │
│ 50/day) │ │ bursts │
├──────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────┤
│ Commit cadence (120 s → 30 s) │ Gas × 4 │ Would drive mainnet cost to ~$540/month; probably │
│ │ │ not justified │
└──────────────────────────────────┴──────────────────────────────────────┴──────────────────────────────────────────────────────┘
Biggest unknown
Actual WASM binary size distribution. I used 15 MiB avg as a placeholder. Want to pull actual numbers from current deployments? just
admin-run --bin cli --command wasm-size-report or similar — if we have one, the projection tightens up. If most policies are <1 MiB
WASM, everything above is ~10× too high.
---
Want me to (a) move to Q5 (chain placement details — confirming artifact replication semantics across chains), (b) dig into the
on-chain gas optimization question (skip-if-empty commits vs. deterministic cadence), or (c) pull actual WASM size numbers from the
current codebase to tighten the projection?
✻ Cooked for 1m 46s
❯ the current wasm cache limits are not set. they were just stop gap. we needed wasm cache to prevent fetching the same wasm from ipfs
every single task evaluation. Now that we are storing wasm bytes on newton protocol chain, do we still need cache?
⏺ Good catch — the cache's original rationale dissolves under Model X. Let me verify the current cache's role and then walk through
what the new architecture should look like.
Searched for 2 patterns, read 2 files (ctrl+o to expand)
⎿  PreToolUse:Read hook error ⎿  Failed with non-blocking status code: [TokenEfficiency] Hard
rule violation: Re-reading wasm_cache.rs (2x) — file unchanged
since last read. Consider using cached knowledge.
⎿  Loaded crates/CLAUDE.md
⎿  Loaded crates/data-provider/src/data_sources/CLAUDE.md
⏺ Confirmed what the current cache is: a disk-bytes cache at ~/.cache/newton-prover/wasm/* with LRU eviction. Sole purpose: avoid IPFS
re-fetch. Under Model X, that purpose evaporates.
But I found something more interesting: WasmExecutor::execute_wasm_bytes recompiles the WASM on every eval (compile sites at
crates/data-provider/src/wasm/executor.rs:381,508). There's no Module cache today. The bytes cache was saving us from IPFS fetches,
but wasmtime compilation — which dominates cost at 100ms–1s per WASM — happens every single task evaluation. That's a much bigger
missed optimization.
So the answer is nuanced: you're right that the bytes cache goes away, but we should replace it with a Module cache, not nothing.
Three-tier view of caching
┌──────────┬─────────────────────────────┬───────────────────┬───────────────────────────────────────────┬───────────────────────┐
│ Layer │ What it holds │ Lookup cost │ Produce cost │ Currently cached? │
├──────────┼─────────────────────────────┼───────────────────┼───────────────────────────────────────────┼───────────────────────┤
│ Bytes │ Raw Vec<u8> WASM │ redb read: ~0.1–1 │ IPFS fetch: 50–500 ms (today) / redb read │ Yes — disk LRU │
│ │ │ ms │ (Model X) │ │
├──────────┼─────────────────────────────┼───────────────────┼───────────────────────────────────────────┼───────────────────────┤
│ Module │ Compiled wasmtime::Module │ HashMap lookup: │ Module::new(): 100 ms – 1 s │ No (this is the gap) │
│ │ │ ~1 µs │ │ │
├──────────┼─────────────────────────────┼───────────────────┼───────────────────────────────────────────┼───────────────────────┤
│ Instance │ Instantiated │ — │ Instance::new(): ~1 ms │ No (too stateful to │
│ │ wasmtime::Instance │ │ │ reuse) │
└──────────┴─────────────────────────────┴───────────────────┴───────────────────────────────────────────┴───────────────────────┘
Under Model X:
- The bytes layer's "produce cost" collapses from IPFS fetch to redb read (200–500× cheaper). Disk cache becomes obsolete.
- The Module layer's produce cost is unchanged (wasmtime compile is WASM-shape-dependent, not source-location-dependent). This
becomes the dominant hot-path cost — and it's also where the largest caching win lives.
Recommendation
Remove the WasmCache; add a Module cache. Net is simpler architecture AND better hot-path performance.
┌───────────────┬──────────────────────────────┬─────────────────────────────────────────────────────────┐
│ Dimension │ Today (WasmCache) │ Recommended (ModuleCache) │
├───────────────┼──────────────────────────────┼─────────────────────────────────────────────────────────┤
│ What's cached │ Bytes on disk │ Compiled Module in RAM │
├───────────────┼──────────────────────────────┼─────────────────────────────────────────────────────────┤
│ Keyed by │ IPFS CID │ wasmHash (keccak256 of bytes) │
├───────────────┼──────────────────────────────┼─────────────────────────────────────────────────────────┤
│ Hit behavior │ Return bytes → still compile │ Return ready-to-instantiate Module │
├───────────────┼──────────────────────────────┼─────────────────────────────────────────────────────────┤
│ Miss behavior │ IPFS fetch + disk write │ redb read + wasmtime compile + HashMap insert │
├───────────────┼──────────────────────────────┼─────────────────────────────────────────────────────────┤
│ Eviction │ LRU by byte size │ LRU by cached entry count or est. RAM size │
├───────────────┼──────────────────────────────┼─────────────────────────────────────────────────────────┤
│ Invalidation │ Manual │ Automatic (content-addressed: new wasmHash = new entry) │
└───────────────┴──────────────────────────────┴─────────────────────────────────────────────────────────┘
Eval hot path becomes:
eval(policy_id, task):
wasm_hash = read_leaf(policy_id).hash # redb, ~0.5 ms
module = module_cache.get_or_insert(wasm_hash, || {
bytes = read_leaf_bytes(wasm_hash) # redb, ~1 ms
verify_jmt_inclusion(bytes, state_root) # ~1 ms
Module::new(&engine, &bytes) # 100 ms – 1 s, one-time
})
Instance::new(&module, &store) # ~1 ms
run(instance, task)
Hot path after warm-up is ~3 ms end-to-end vs today's ~100+ ms (compile-every-eval).
★ Insight ─────────────────────────────────────
This is a case where removing one layer exposes a different, more valuable layer that was always there but invisible. The disk-bytes
cache papered over the "fetch" cost and left "compile" as the silent hot-path tax. Model X + Module cache flips the priority: bytes
are now effectively free, so compilation — always the real bottleneck — becomes the thing worth caching. Content-addressed keying (by
wasmHash) means the cache is self-consistent: when a policy upgrades, its new wasmHash is a fresh cache key, old entry ages out via
LRU, no explicit invalidation logic needed. This pattern shows up elsewhere in crypto systems (Merkle-addressed state,
content-addressable storage) and the property is always the same: content identity = cache identity.
─────────────────────────────────────────────────
Sizing the Module cache
wasmtime compiled Module size is typically 2–5× the WASM byte size (native code + metadata + relocations).
┌────────────────┬────────────────────┬─────────────────┬───────────────┐
│ WASM size │ Module size (est.) │ Cache N modules │ RAM footprint │
├────────────────┼────────────────────┼─────────────────┼───────────────┤
│ 5 MiB │ ~15 MiB │ 20 │ 300 MiB │
├────────────────┼────────────────────┼─────────────────┼───────────────┤
│ 15 MiB (avg) │ ~45 MiB │ 20 │ 900 MiB │
├────────────────┼────────────────────┼─────────────────┼───────────────┤
│ 30 MiB (large) │ ~90 MiB │ 20 │ 1.8 GiB │
└────────────────┴────────────────────┴─────────────────┴───────────────┘
At 50 policies/chain × 15 MiB avg WASM, the full working set is ~2–3 GiB RAM if we cache everything. Realistically, eval traffic
follows power-law distribution — top-10 policies handle >90% of tasks — so caching ~20 hot Modules per chain covers the working set
at <1 GiB operator RAM.
Proposed config (replaces max_cache_size):
pub struct DataProviderConfig {
// ...existing fields...
/// Maximum number of compiled wasmtime::Module entries to keep in RAM.
/// LRU-evicted. Each entry is typically 2-5x the source WASM byte size.
pub max_module_cache_entries: usize, // default: 32
/// Optional soft cap on total Module cache RAM footprint (MiB).
/// Acts as a secondary constraint; whichever limit hits first wins.
pub max_module_cache_ram_mib: Option<u64>, // default: Some(1024)
}
Gotchas to flag
1. wasmtime Engine is already shared across Modules — good. No duplication of JIT state.
2. Module compilation is single-threaded by default per call. If a cold-miss storm hits (e.g., new operator joining, bootstrap),
first-eval latency spikes. Options: pre-warm cache during bootstrap, or use Engine::precompile_module in a background task right
after state-tree apply.
3. Serialized Module persistence — wasmtime supports Module::serialize + Module::deserialize_file to persist compiled modules to
disk, skipping re-compile on restart. Probably not worth it for Phase 1 (operators restart rarely; cold start is ≤30 s for 20
modules). Worth revisiting if operator restart latency becomes a pain point.
4. wasmtime version-bumps invalidate serialized Modules. If we ever add disk-persisted Module cache, key by (wasmHash,
wasmtime_version) to avoid loading stale compiled code.
5. Security angle. Since the Module cache key is wasmHash = keccak256(bytes), and the leaf in redb commits to those exact bytes via
JMT inclusion proof, no additional integrity check is needed on cache hits. Content addressing carries the integrity guarantee. This
is cleaner than today's IPFS-fetched bytes where we had to verify CID → bytes mapping separately.
What to delete when implementing
- crates/data-provider/src/wasm_cache.rs entire file
- WasmCache struct and all call sites
- DataProviderConfig::max_cache_size field
- IPFS fetch logic in data_sources/wasm.rs::fetch_data (bytes come from redb now)
- ProjectDirs disk-cache path plumbing in multiple places
Gives us less code + faster hot path + clearer architecture. Clean win.
---
Agree on the direction? If so, this likely belongs as Q4-sub-branch-c → caching layer in our spec. Next decisions after this:
- (a) Move to Q5 (chain placement) — artifact replication semantics across chains
- (b) Drill down on Module cache sizing / pre-warm strategy (important for operator RAM planning)
- (c) Zoom out to the policy deployment write path — the authoring side of this new storage (which we deferred from Q4)
Which?
✻ Cogitated for 2m 0s
@denniswon
Copy link
Copy Markdown
Author

Open Questions Q.1–Q.7 in the document need resolution before stream-by-stream execution starts — most notably the JSON Schema
validator library choice (Q.2), quickstart-policy migration ordering (Q.3), and overlap with NEWT-1036 (Q.4). Implementation plan
execution mode (Subagent-Driven vs Inline) is still to be picked. --> Let's actually discuss this and align on this

⏺ Good call — let's actually work through these. Let me re-read the exact Q.1–Q.7 framing from the doc so we discuss what's actually
written, not what I remember.

Searched for 1 pattern, read 1 file (ctrl+o to expand)

⏺ Let's work through them. I'll give my read on each plus the question I'd ask back. We're discussing 8 items total (the deferred Q.0
execution-mode pick + Q.1–Q.7 in the doc).

★ Insight ─────────────────────────────────────

  • Q.2, Q.3, and Q.4 are the only blockers. Q.0/Q.1 are workflow choices, Q.5/Q.6 are deferral decisions (default to "no, ship Phase 1"),
    Q.7 is documentation hygiene. The three real blockers all need an answer before stream A starts.
  • Two of the three blockers are people-coordinated, not technical. Q.3 is "how does Newton ops sequence the Safe multisig sessions" and
    Q.4 is "when does NEWT-1036's deletion land relative to Sub-B's adds." Those answers come from you/Aaron and the team, not from spec
    analysis. Q.2 is the only one that can be resolved by benchmark.
  • AddressesProviderConsumer immutable-caching means Q.4 is more interesting than it looks. If any contract on chain consumes
    IdentityRegistry/ConfidentialDataRegistry IDs from the provider today, it cached those addresses as immutables — so NEWT-1036's deletion
    doesn't hot-swap them. Sub-B's new contracts are net-new consumers with no prior immutable caching, so Q.4 is mostly about ordering
    registrar updates, not hot-swapping live consumers.
    ─────────────────────────────────────────────────

Q.0 — Execution mode for the 47-task plan

My recommendation: Subagent-Driven.

47 tasks × ~3-5 steps each won't fit one context window even with checkpoints. The two-stage review (subagent + me) catches drift on
Sub-B's intricate identity formulas (the 0x01 domain-separator byte on policyDataInstanceId, the chainId ‖ clientAddr ordering) — those
are exactly the kinds of details that get fuzzy when context is full. Inline only wins for <10-task plans where ceremony overhead
exceeds the benefit.

Question back: any reason to prefer Inline? (e.g., wanting to watch each task land in real-time, or tighter feedback on early tasks
before parallelizing?)


Q.1 — Linear project name

My recommendation: ship the provisional "Onchain Policy Storage."

This is naming bikeshed territory — both "On-chain" and "Onchain" are correct. Newton ops decides; doesn't block code. Mirroring Sub-A's
"Private Data Storage" (no hyphen in URLs) is reasonable.

Question back: want me to also pre-register the 47 Linear issues mirroring NEWT-1003..1052 conventions, or do that later when the plan
is closer to execution?


Q.2 — JSON Schema validator library (boon vs jsonschema)

My recommendation: benchmark first; default to jsonschema if inconclusive.

boon is lighter (~150-200KB smaller binary) but newer. jsonschema is mature with broader Draft 2020-12 coverage. The 14 quickstart
schemas are the natural test bed. Concrete proposal:

┌────────────┬───────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────┐
│ Library │ Strengths │ Risks │
├────────────┼───────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ boon │ Pure Rust, smaller deps, │ Newer (less battle-tested), some Draft 2020-12 features may be │
│ │ no_std-friendlier │ incomplete │
├────────────┼───────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ jsonschema │ Mature, broad coverage, active │ ~400KB heavier, may pull in async runtime indirectly │
│ │ maintenance │ │
└────────────┴───────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────┘

Stream A early: write the 14 quickstart schemas, run both validators, pick whichever passes 100% of the suite with the cleanest API. If
both pass, take boon for the smaller footprint.

Question back: is binary footprint (operator daemon size) actually a constraint we care about right now, or is correctness/maturity the
only axis?


Q.3 — Quickstart migration ordering

My recommendation: staged 3 + 5 + 6, NOT one batch.

This is the most consequential ops decision in the list. Reasoning:

  • Single batch (14 in one Safe session): faster, but a single hash-mismatch or schema-validation surprise rolls back 14 deploys. The L1
    drift-detection layer (registerTemplate rejects keccak256 mismatch) is new code — not yet field-tested at scale.
  • Staged (3 + 5 + 6): first batch should be the simplest policies (single-domain or read-only). If something surfaces, we catch it on 3
    instead of 14. Safe multisig overhead per batch is ~10-20 min — three sessions across a week is cheap insurance.

The G2 ChallengeVerifier re-derivation hook is also new; staging gives Newton ops a real-world signal that on-chain policyTemplateRef
values match what challengers expect before all 14 are exposed.

Question back: which of the 14 quickstart policies are the simplest / most tested today? That informs the first batch.


Q.4 — NEWT-1036 overlap

My recommendation: NEWT-1036 lands first, Sub-B's adds land after.

NEWT-1036 (deletes IdentityRegistry + ConfidentialDataRegistry) is independent of Sub-B's net-new PolicyRegistry + PolicyDataRegistry.
They could ship in either order, but landing NEWT-1036 first keeps the provider directory smaller and avoids a transitional window where
the provider has both the deprecated entries and Sub-B's adds.

The immutable-caching concern only matters if there are existing consumers of the deleted IDs — which there will be (the registries'
direct consumers). NEWT-1036 has to handle redeployment of those consumers regardless of Sub-B. Sub-B's consumers are net-new, so they
only see the post-deletion provider state.

Question back: is NEWT-1036 ready to ship, or is its blocker the same as Sub-B's (waiting for Sub-A primitives to soak)? If they're both
gated on the same thing, sequencing them serially is the safer call.


Q.5 — Atomic-bundle priority (Phase 2 multicall)

My recommendation: defer to Phase 2 backlog as documented.

The "deploy-then-update-pointer" race window is exactly one block on most chains and only matters if an observer queries
currentPolicyInstance[client] between the two txs. For Phase 1 — where clients control their own deploys — this is acceptable. Pull it
forward only if real-world ops surface frequent friction.

Question back: any known external dependency that observes currentPolicyInstance[client] mid-deploy and would break under the race
window?


Q.6 — policyParams on-chain mirror

My recommendation: ship Phase 1 with the on-chain mirror.

The 14 quickstart policyParams payloads are small (likely <1KB each, mostly string addresses + numeric thresholds). On-chain storage
cost is bounded. Moving to EigenDA-backed policyParamsRef adds challenger-side fetch + KZG verification complexity — unjustified at MVP
scale.

Threshold to reconsider: any single policyParams exceeding 8KB, or storage cost on Base/L2s becoming visible in operator economics.

Question back: is there a known upcoming policy with policyParams >2KB? If so, that flips the calculus.


Q.7 — Secrets schema ↔ HPKE envelope versioning

My recommendation: keep independent (as Phase 1 ships); document the orthogonality.

Coupling them would force an envelope rotation on every schema update — costly entanglement for no security benefit. The envelope_format
byte (0x03 namespace) describes how bytes are encrypted; the secrets schema (0x08 namespace) describes what fields are in the
plaintext. These are genuinely orthogonal — a _v2 schema with new fields can live under the same SECRETS_ENVELOPE_V1 envelope as long as
the new fields fit in the existing encrypted blob.

The doc already describes the failure mode (WASM eval fails deterministically when a required field is absent) — that's enough.

Question back: none from me on this one. It's a "default to keeping things uncoupled" decision unless someone has a concrete pairing
reason.


Summary table

┌─────┬─────────────────────────────────────┬────────────────┬──────────────────────────────────────┐
│ Q │ My pick │ Confidence │ Needs you to confirm? │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.0 │ Subagent-Driven │ High │ Yes │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.1 │ "Onchain Policy Storage" │ Low (bikeshed) │ Yes │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.2 │ Benchmark first; default jsonschema │ Medium │ No (handled in stream A) │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.3 │ Staged 3 + 5 + 6 │ High │ Yes │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.4 │ NEWT-1036 first, Sub-B after │ Medium │ Yes (depends on NEWT-1036 readiness) │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.5 │ Defer to Phase 2 │ High │ Yes (sanity check) │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.6 │ On-chain mirror │ High │ Yes (sanity check) │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.7 │ Keep independent │ High │ Yes (sanity check) │
└─────┴─────────────────────────────────────┴────────────────┴──────────────────────────────────────┘

What's your read on each? Which of my recommendations are wrong, which want different framing, and which are you fine with as-is?

✻ Crunched for 2m 7s

※ recap: Goal: lock Sub-B (on-chain policy storage) protocol decisions before implementing the 47-task plan. Current task: align on Q.0
execution mode plus Q.1–Q.7 from the doc. Next action: react to my recommendations in the summary table.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment