-
-
Save denniswon/c9e73e39e14f8de1230d4253f46f6bda to your computer and use it in GitHub Desktop.
| ⏺ You're right — I was stuck in the wrong substrate model. Let me course-correct. | |
| ★ Insight ───────────────────────────────────── | |
| - I was treating "state tree" as the private-data substrate and "EVM contract storage" as the public-data substrate, as if Sub-project A's | |
| fabric didn't apply here. It does apply. The state tree is a content-addressed key-value store with on-chain root commitments — it has nothing | |
| inherently private about it. Encrypted leaves = private data (Sub-project A's SecretsRegistry); plaintext leaves = public data (Sub-project B's | |
| artifacts). The substrate is the same, only leaf encryption differs. | |
| - On-chain, we only ever need the hash + root commitments. For a specific task, NewtonPolicy.policyCodeHash already commits to the Rego bytes — | |
| any fetcher verifies keccak256(bytes) == policyCodeHash and that closes the integrity loop regardless of where bytes live. For registry-level | |
| questions ("is this plugin still approved?"), the 120s BLS-signed state root + inclusion proof does the job. SSTORE2 never enters the picture | |
| because EVM itself never consumes the bytes. | |
| - This collapses Sub-project B's infrastructure cost dramatically. The state-tree + EigenDA + BLS-commit + bootnode + gateway-orchestration | |
| fabric is already being built in Sub-project A. Sub-project B adds more registries on top of that fabric — different data types, different leaf | |
| semantics, but the same plumbing. It's additive, not a parallel track. | |
| ───────────────────────────────────────────────── | |
| Corrected model — where each thing lives | |
| ┌──────────────────────────────┬────────────────────────────────────────────────┬────────────────────────────┬────────────────────────────┐ | |
| │ What │ Where │ Who writes │ Who reads │ | |
| ├──────────────────────────────┼────────────────────────────────────────────────┼────────────────────────────┼────────────────────────────┤ | |
| │ keccak256(rego_bytes) → Rego │ Operator-local state tree (plaintext leaf), │ Gateway-orchestrated │ Operators / challengers / │ | |
| │ source │ backed by EigenDA delta blob │ operator broadcast on │ SP1 prover, via signed RPC │ | |
| │ │ │ policy deploy │ + inclusion proof │ | |
| ├──────────────────────────────┼────────────────────────────────────────────────┼────────────────────────────┼────────────────────────────┤ | |
| │ keccak256(schema_bytes) → │ │ │ Same — signed RPC from │ | |
| │ data.params / data.wasm │ Same — state tree leaf │ Same flow │ operator │ | |
| │ schema │ │ │ │ | |
| ├──────────────────────────────┼────────────────────────────────────────────────┼────────────────────────────┼────────────────────────────┤ | |
| │ keccak256(wasm_bytes) → WASM │ Same — state tree leaf (MB-scale leaf is fine; │ Same │ Same │ | |
| │ plugin binary │ redb handles arbitrary value sizes) │ │ │ | |
| ├──────────────────────────────┼────────────────────────────────────────────────┼────────────────────────────┼────────────────────────────┤ | |
| │ │ Per-registry BLS-signed state root, committed │ │ EVM — used for │ | |
| │ On-chain commitment │ every 120s to the registry contract │ Operator quorum (same as │ inclusion-proof │ | |
| │ │ (implementing IStateRootCommittable) │ Sub-project A) │ verification in challenge │ | |
| │ │ │ │ paths │ | |
| ├──────────────────────────────┼────────────────────────────────────────────────┼────────────────────────────┼────────────────────────────┤ | |
| │ │ NewtonPolicy.policyCodeHash continues to point │ │ SP1 circuit / challenger / │ | |
| │ Existing on-chain hashes │ at the registered Rego hash; NewtonPolicyData │ Policy deployer │ operator (already the │ | |
| │ │ gets a wasmHash field replacing wasmCid │ │ four-site commitment │ | |
| │ │ │ │ pattern) │ | |
| └──────────────────────────────┴────────────────────────────────────────────────┴────────────────────────────┴────────────────────────────┘ | |
| What goes on-chain: hashes (as they already do via policyCodeHash) + per-registry state roots + registry metadata (deprecation flags, deploy | |
| timestamps). That's it. | |
| What goes in operator state + EigenDA: the actual bytes. | |
| What disappears: IPFS dependency, SSTORE2, MB-scale EVM storage cost. The "on-chain" guarantee in the brief is satisfied by the committed state | |
| root, not by bytes-in-EVM. | |
| What I was misunderstanding | |
| I was holding onto the EVM smart-contract analogy from your brief ("like how smart contracts are deployed/stored onchain") too literally — | |
| treating it as "bytes physically in contract code." You were using it as "protocol-committed content with on-chain trust anchors, fetchable | |
| from the protocol without external dependency." Under that second reading (which is actually what the brief requires), the state-tree substrate | |
| is a cleaner fit than SSTORE2. | |
| What this resolves | |
| - Q3 (storage substrate): state tree on the Sub-project A fabric, not SSTORE2. Applies uniformly to all four artifacts. | |
| - E5 for all four: confirmed — this is literally what the state-tree read path does (gateway-orchestrated for writes, operator signed-RPC reads | |
| for fetches, EigenDA as the blob-availability layer for snapshot/replay). | |
| - Q2 (addressing): content-hash remains the primary key. In state-tree terms: the tree key is the content hash. | |
| - Privacy extension (R5): free — encrypted leaves work the same as SecretsRegistry. The on-chain registry stores (cleartext_hash, | |
| TEE_attestation_requirement_flag) in its metadata; the leaf stores ciphertext. | |
| - Infrastructure reuse: Sub-project B becomes "three or four more IStateRootCommittable registries" (RegoRegistry + SchemaRegistry(ies) + | |
| WasmPluginRegistry), reusing the state-tree crate, EigenDA client, bootnode, gateway orchestration, and BLS commit flow from Sub-project A | |
| wholesale. | |
| What opens up next | |
| A few concrete decisions remain: | |
| 1. One registry vs several — single unified PolicyArtifactRegistry with typed entries (kind: Rego / Schema / WasmPlugin) sharing one state | |
| root, or separate RegoRegistry / SchemaRegistry / WasmPluginRegistry each with its own state root? Sub-project A chose separate (Identity / | |
| Confidential / Secrets) to keep trust boundaries clean. Same logic probably applies here. | |
| 2. Registry on which chain(s)? Sub-project A's three registries live on source chain (per your Linear issue decomposition). Policy artifacts | |
| probably same — but worth confirming because destination-chain challengers need to verify artifact presence too, which means either dest-chain | |
| registries or cross-chain root relay. | |
| 3. State-root commit cadence for artifact registries — Sub-project A uses 120s across all three registries. Do we lock-step all six+ registries | |
| (Sub-project A's three + Sub-project B's new ones) at the same 120s cadence? It simplifies operators' commit loop significantly. | |
| 4. Model X vs Model Y for WASM leaves — leaf stores bytes directly (X), or leaf stores metadata + EigenDA blob pointer (Y)? Model X makes state | |
| tree a single source of truth but grows the tree storage footprint; Model Y keeps the tree light but adds a fetch indirection. Sub-project A | |
| uses Model X for identity/confidential data (KB-scale leaves); for MB-scale WASM, Y might be preferable. | |
| Which of these do you want to dig into first? Q4 (single vs several registries) feels like the most foundational — it decides the on-chain | |
| contract surface and the trust boundaries between artifact kinds. | |
| You're right — I got Sub-project A's topology wrong in two places. Let me correct both before recommending on Q4. | |
| ★ Insight ───────────────────────────────────── | |
| - Sub-project A's registry topology is unified, not split. The V2 design collapsed IdentityRegistry / ConfidentialDataRegistry / | |
| SecretsRegistry (V1) into one StateCommitRegistry per chain with 1-byte namespace prefixes (0x01 identity / 0x02 confidential / 0x03 | |
| secrets) addressing a single unified JMT root (spec §3, lines 14, 68, 106, 194, 465). The spec explicitly rejects per-namespace | |
| operator-set partitioning (§S line 444): "it was the sole benefit of the V1 three-registry shape, and the cross-namespace policy-eval | |
| dependency made that partitioning useless in practice." | |
| - Sub-project A's registries deploy to every chain, not source-only. Spec §line 412 — "Three registries deploy per chain Newton | |
| supports"; §line 415 — "Destination chains (Base, L2s): full registry — operators maintain per-chain subtrees and post commits per | |
| chain"; §line 465 — "One StateCommitRegistry MUST be deployed per chain Newton supports. Each chain's registry is authoritative for | |
| that chain's unified state root and sequenceNo." No cross-chain state-root sync in Phase 1. | |
| - The grill-me context line at the top of this session ("Three registries with shared IStateRootCommittable interface — not a single | |
| PrivateDataStore") is stale — it reflects an earlier iteration of Sub-project A's design. The written spec supersedes it. I should | |
| have verified against the doc before anchoring recommendations to that line. | |
| ───────────────────────────────────────────────── | |
| Answers | |
| (2) Chain placement — you were right. Sub-project A's StateCommitRegistry lives on every chain Newton supports (source AND | |
| destinations), one per chain, each authoritative for its own unified state root. No cross-chain sync in Phase 1. So for Sub-project | |
| B: same default — one policy registry per chain. Dest-chain challengers get the inclusion-proof substrate for free. Flag for Q5 | |
| confirmation later, but this is the baseline. | |
| (3) 120s lockstep — locked in. All Sub-project A + Sub-project B registries commit on the same 120s boundary per chain. | |
| (1) Q4 recommendation — don't split. Unify into either one or two state trees. | |
| My prior "Sub-project A chose separate, same logic applies" was based on the wrong premise. The correct analogue is the opposite: | |
| Sub-project A unified, so Sub-project B should unify too. Two sub-options remain: | |
| Option: 1A. Extend existing StateCommitRegistry | |
| What it is: Add namespace tags 0x04 Rego / 0x05 Schema / 0x06 WasmPlugin to the same JMT. One root, one BLS sig, one commit tx per | |
| chain per 120s. | |
| Trust boundary: Same operator quorum already signs across identity/confidential/secrets. Extending to policy artifacts is additive. | |
| Plumbing cost: Zero new contracts. One more namespace prefix per artifact kind. | |
| ──────────────────────────────────────── | |
| Option: 1B. Second unified registry (PolicyArtifactRegistry) | |
| What it is: Separate JMT for policy artifacts, same IStateRootCommittable interface, parallel 120s commit. | |
| Trust boundary: Separate state root lets you bound "I trust the private-data quorum but not the policy-artifact quorum" — useful if | |
| the operator sets ever diverge. | |
| Plumbing cost: One new contract per chain. One more BLS aggregation + commit tx per 120s. Possible to run on the same operator set | |
| ❯ 1A agreed. Also, for (4), for cost, performance, reliability, what do you recommend? | |
| ⏺ 1A locked in. StateCommitRegistry extends with namespace tags 0x04 Rego / 0x05 Schema / 0x06 WasmPlugin. Zero new contracts, same | |
| operator quorum, same 120s lockstep commit. | |
| --- | |
| Model X vs Model Y on cost / performance / reliability | |
| Recommend Model X with per-commit size guard on all three axes. Reliability is the decisive one. | |
| Cost | |
| ┌──────────────────────────┬───────────────────────────────────┬──────────────────────────────────────────┐ | |
| │ Dimension │ Model X (bytes in leaf) │ Model Y (hash in leaf, bytes in EigenDA) │ | |
| ├──────────────────────────┼───────────────────────────────────┼──────────────────────────────────────────┤ | |
| │ Per-operator disk │ ~200 MiB–1 GiB for 20–50 policies │ ~200 KiB (just hashes + metadata) │ | |
| ├──────────────────────────┼───────────────────────────────────┼──────────────────────────────────────────┤ | |
| │ EigenDA write cost │ Bytes in delta blob │ Bytes in separate artifact blob (~same) │ | |
| ├──────────────────────────┼───────────────────────────────────┼──────────────────────────────────────────┤ | |
| │ Bootnode/challenger disk │ Full bytes (~1 GiB) │ Hashes + ephemeral cached bytes │ | |
| ├──────────────────────────┼───────────────────────────────────┼──────────────────────────────────────────┤ | |
| │ Network replication │ N operators × 1 GiB │ 1× EigenDA + per-operator cache │ | |
| └──────────────────────────┴───────────────────────────────────┴──────────────────────────────────────────┘ | |
| Y saves disk, but at Phase 1 scale (20–50 policies × 10–30 MiB ≈ <2 GiB total network-wide), savings are trivial. Modern SSDs are | |
| TB-scale; redb handles it fine. Neither model is cost-constrained at Phase 1. | |
| Performance | |
| ┌────────────────────────────┬──────────────────────────┬────────────────────────────────────┐ | |
| │ Path │ Model X │ Model Y │ | |
| ├────────────────────────────┼──────────────────────────┼────────────────────────────────────┤ | |
| │ Eval hot path (cache hit) │ 1 redb read │ 1 redb read + cache hit │ | |
| ├────────────────────────────┼──────────────────────────┼────────────────────────────────────┤ | |
| │ Eval hot path (cache miss) │ local bytes (~1ms) │ network fetch (~10–200ms) │ | |
| ├────────────────────────────┼──────────────────────────┼────────────────────────────────────┤ | |
| │ Cold start / new operator │ snapshot = bulk transfer │ snapshot + N parallel WASM fetches │ | |
| ├────────────────────────────┼──────────────────────────┼────────────────────────────────────┤ | |
| │ First-run wasmtime compile │ 100ms–1s (dominates) │ 100ms–1s (same) │ | |
| └────────────────────────────┴──────────────────────────┴────────────────────────────────────┘ | |
| Hot path is equivalent (both cache). Bootstrap favors X (single bulk transfer). First-run compilation dominates in both. Performance | |
| difference is not decisive. | |
| Reliability (decisive axis) | |
| Model X: | |
| - Single substrate: redb + JMT | |
| - Delta blob = transport only (ephemeral, pruneable after apply) | |
| - Tree = authoritative storage | |
| - Inclusion proof verifies bytes directly | |
| - No dangling-reference risk | |
| - Matches Sub-project A's existing transport/storage split exactly | |
| Model Y: | |
| - Two-tier substrate: JMT (for hash) + EigenDA (for bytes) | |
| - EigenDA becomes a storage medium, not just transport | |
| - New retention invariant: "WASM artifacts live in EigenDA indefinitely" (today delta blobs can be pruned after consumption) | |
| - Two-step verification: prove hash via JMT, then verify fetched bytes | |
| - Dangling-reference risk if artifact blob is pruned | |
| - Cache eviction + EigenDA outage = unreadable policy = cannot evaluate tasks | |
| ★ Insight ───────────────────────────────────── | |
| The core architectural question here is whether EigenDA is transport or storage. Sub-project A treats it as transport — delta blobs | |
| are consumed by bootnode/operators/challengers, applied to the authoritative tree, and become pruneable. Model Y promotes EigenDA to | |
| a storage medium with forever-retention requirements, which is a fundamentally different reliability posture. Once that invariant | |
| exists for one artifact class, it creates pressure to reuse it for others, and a new substrate responsibility has quietly been added | |
| to the system. That's worth avoiding unless the cost/performance case is overwhelming — and at Phase 1 scale, it isn't. | |
| ───────────────────────────────────────────────── | |
| Recommendation | |
| Model X + per-commit size guard (~50 MiB cap per commit, roll excess to next commit). | |
| - Cost: trivial at Phase 1 scale | |
| - Performance: equivalent hot path, slight bootstrap edge | |
| - Reliability: clearly superior — single substrate, no new EigenDA retention invariant, no dangling references | |
| Two caveats worth flagging | |
| 1. Delta blob size limit: EigenDA has a per-blob cap (currently 16 MiB decoded, I think — needs verification before finalizing the 50 | |
| MiB commit cap). If a commit's total leaf bytes exceed the blob limit, we either (a) lower the per-commit cap, or (b) split the | |
| delta across multiple blobs. Worth nailing down before writing the spec. | |
| 2. Future WASM opcode fee mechanism (the TODO at crates/core/src/config/data_provider.rs:28-30): If/when pay-per-byte pricing lands, | |
| Model Y becomes attractive again (bytes live once in EigenDA, fees charged per-byte, no replication cost). But for Phase 1 (no fee | |
| mechanism), Model X wins on reliability and the cost savings from Y are marginal. | |
| --- | |
| Model X + 50 MiB per-commit cap looks solid — want to move on to Q5 (chain placement) next, or dig into one more Q4 sub-branch first | |
| (e.g., the policy-artifact write path: who authors a new Rego deployment, which gateway endpoint, what BLS-quorum validation on | |
| registration)? | |
| ✻ Sautéed for 1m 51s | |
| ※ recap: Designing Sub-project B (on-chain policy storage for Rego/WASM/schema). Locked Option 1A (extend StateCommitRegistry with | |
| namespaces 0x04/0x05/0x06) and Model X with 50 MiB per-commit cap. Next: move to Q5 on chain placement, or dig into the | |
| policy-artifact write path first. (disable recaps in /config) | |
| ❯ let's do cost projection when going with model X | |
| ⏺ Cost projection: Model X, Phase 1 | |
| Bottom line up front: ~$10–50/month network-wide across all dimensions. At Phase 1 scale, the Newton AVS pays approximately the same | |
| as it currently pays for IPFS pinning — the replicated redb storage is not the constraint. | |
| Let me walk through the dimensions with assumptions so you can challenge any that don't match your mental model. | |
| Assumptions (pls correct if off) | |
| ┌──────────────────────┬─────────────────────────────────────────┬───────────────────────────────────────────────────────────────┐ | |
| │ Parameter │ Phase 1 estimate │ Notes │ | |
| ├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤ | |
| │ Policy count │ 20–50 │ Early Phase 1 steady state │ | |
| ├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤ | |
| │ Rego source per │ ~5 KiB avg │ Typical Rego ~100–500 lines │ | |
| │ policy │ │ │ | |
| ├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤ | |
| │ Schema JSON per │ ~3 KiB avg │ Small │ | |
| │ policy │ │ │ | |
| ├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤ | |
| │ WASM binary per │ ~15 MiB avg (5–30 MiB) │ Based on observed sizes; max_cache_size 100 MiB is hard upper │ | |
| │ policy │ │ bound │ | |
| ├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤ | |
| │ Operators per chain │ ~10 │ Based on current stagef/prod patterns │ | |
| ├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤ | |
| │ Chains supported │ 3 │ Sepolia + Base Sepolia + one L2 (or mainnet eq.) │ | |
| ├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤ | |
| │ Replicas per chain │ 12 │ 10 operators + 1 bootnode + 1 challenger │ | |
| ├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤ | |
| │ Commit cadence │ 120 s │ 720 commits/day/chain │ | |
| ├──────────────────────┼─────────────────────────────────────────┼───────────────────────────────────────────────────────────────┤ | |
| │ Policy deployment │ 1–2/day during launch → ~0/day steady │ Rego updates rare post-launch │ | |
| │ rate │ state │ │ | |
| └──────────────────────┴─────────────────────────────────────────┴───────────────────────────────────────────────────────────────┘ | |
| 1. On-chain gas cost (the biggest variable) | |
| Each 120 s commit is one tx: commitStateRoot(newRoot, sig) → one SSTORE + one event emission + BLS cert verification inline. | |
| ┌───────────────────────────┬────────────────┬─────────────┬─────────┬─────────────┬──────────────┐ | |
| │ Chain │ Gas per commit │ Commits/day │ Gas/day │ Daily cost* │ Monthly cost │ | |
| ├───────────────────────────┼────────────────┼─────────────┼─────────┼─────────────┼──────────────┤ | |
| │ Sepolia (1 gwei) │ ~80k │ 720 │ 57.6M │ $0.14 │ $4.20 │ | |
| ├───────────────────────────┼────────────────┼─────────────┼─────────┼─────────────┼──────────────┤ | |
| │ Base Sepolia (~0.01 gwei) │ ~80k │ 720 │ 57.6M │ $0.01 │ $0.30 │ | |
| ├───────────────────────────┼────────────────┼─────────────┼─────────┼─────────────┼──────────────┤ | |
| │ Mainnet L1 (30 gwei) │ ~80k │ 720 │ 57.6M │ $4.30 │ $130 │ | |
| ├───────────────────────────┼────────────────┼─────────────┼─────────┼─────────────┼──────────────┤ | |
| │ Mainnet L2 (~0.1 gwei) │ ~80k │ 720 │ 57.6M │ $0.05 │ $1.50 │ | |
| └───────────────────────────┴────────────────┴─────────────┴─────────┴─────────────┴──────────────┘ | |
| *ETH price $2,500 assumed. | |
| This is the dominant line item at scale, not storage. If we launch on all three mainnet chains (Ethereum + Base + Arbitrum), commit | |
| gas lands around $140/month; testnet-only deployments are <$5/month. | |
| Mitigations if this becomes a constraint: | |
| - Skip no-op commits (don't commit if delta is empty — only commit when state advances) | |
| - Stretch commit cadence on idle chains (already contemplated in §S.13 safety) | |
| - Batch multiple namespace updates into single tx (already the case with 1A) | |
| 2. Off-chain storage (redb per replica) | |
| Per-policy artifact set: | |
| ┌─────────────────────────────────────────┬──────────────────────────┐ | |
| │ Item │ Size │ | |
| ├─────────────────────────────────────────┼──────────────────────────┤ | |
| │ Rego leaf │ ~5 KiB │ | |
| ├─────────────────────────────────────────┼──────────────────────────┤ | |
| │ Schema leaf │ ~3 KiB │ | |
| ├─────────────────────────────────────────┼──────────────────────────┤ | |
| │ WASM leaf │ ~15 MiB │ | |
| ├─────────────────────────────────────────┼──────────────────────────┤ | |
| │ JMT internal nodes per leaf (amortized) │ ~500 B │ | |
| ├─────────────────────────────────────────┼──────────────────────────┤ | |
| │ Leaf header (LeafHeaderV1) │ 20 B │ | |
| ├─────────────────────────────────────────┼──────────────────────────┤ | |
| │ Per-policy total │ ~15 MiB (WASM dominates) │ | |
| └─────────────────────────────────────────┴──────────────────────────┘ | |
| At 50 policies: | |
| - Raw artifacts: 750 MiB | |
| - JMT overhead (~20%): 150 MiB | |
| - redb metadata + compaction slack: ~100 MiB | |
| - Per replica: ~1 GiB | |
| Network-wide: | |
| - 3 chains × 12 replicas × 1 GiB = 36 GiB aggregate | |
| - At AWS EBS gp3 (~$0.08/GB/month): $2.90/month across entire network | |
| This is trivial. Even at 10× the policy count (500 policies → 10 GiB/replica × 36 replicas = 360 GiB), monthly cost is ~$29 | |
| network-wide. | |
| 3. EigenDA delta blob cost | |
| Delta blob composition per 120 s commit: | |
| - Steady state (no artifact changes): ~2 KiB (sequence number bump + JMT internal node changes) | |
| - Policy deployment: ~15 MiB (new WASM) + ~10 KiB (Rego + schema + headers) | |
| ┌───────────────────────────────────┬─────────────────────────────┬───────────────┐ | |
| │ Day type │ Per-chain EigenDA bytes/day │ 3-chain total │ | |
| ├───────────────────────────────────┼─────────────────────────────┼───────────────┤ | |
| │ Steady state │ 720 × 2 KiB = 1.4 MiB │ 4.3 MiB/day │ | |
| ├───────────────────────────────────┼─────────────────────────────┼───────────────┤ | |
| │ Launch week (1–2 deployments/day) │ ~30 MiB │ ~90 MiB/day │ | |
| ├───────────────────────────────────┼─────────────────────────────┼───────────────┤ | |
| │ Heavy deployment day │ ~100 MiB │ ~300 MiB/day │ | |
| └───────────────────────────────────┴─────────────────────────────┴───────────────┘ | |
| At EigenDA Phase 1 indicative pricing (~$0.001/MiB, dropping with volume): | |
| - Steady state: ~$0.15/month network-wide | |
| - Heavy deployment month: ~$10/month | |
| Basically noise. | |
| 4. RAM / compute (no delta vs today) | |
| - Moka WASM cache: already max_cache_size: 100 MiB per operator (existing config, no new demand) | |
| - wasmtime compile: 100 ms–1 s per WASM on first use — same as today (no regression) | |
| - JMT proof generation during eval: <1 ms per proof (proof is ~600 B for depth-20 JMT) | |
| - Delta apply per 120 s commit: ~50 ms on operator side (negligible) | |
| - BLS commit sig: ~20 ms per operator per commit | |
| Total per-operator compute tax from state commits: ~70 ms × 720/day = 50 seconds of CPU/day. Trivial. | |
| 5. Bandwidth | |
| Per operator per day: | |
| - Inbound delta fetches: 720 × 2 KiB + occasional 15 MiB bursts ≈ 30–50 MiB/day | |
| - Outbound BLS sig publish: 720 × 96 B = 70 KiB/day | |
| Per-month per-operator bandwidth: ~1.5 GiB → <$0.15/month at AWS egress rates. | |
| Snapshot bootstrap (new operator joining): | |
| - 1 GiB one-shot pull from bootnode | |
| - <$0.10 per event, rare | |
| 6. Aggregate Phase 1 cost projection | |
| ┌────────────────────────┬─────────────────────────────┬──────────────────────────────────────────┐ | |
| │ Category │ Monthly cost (testnet only) │ Monthly cost (mainnet: ETH + Base + Arb) │ | |
| ├────────────────────────┼─────────────────────────────┼──────────────────────────────────────────┤ | |
| │ On-chain commit gas │ $5 │ $135 │ | |
| ├────────────────────────┼─────────────────────────────┼──────────────────────────────────────────┤ | |
| │ Off-chain redb storage │ $3 │ $3 │ | |
| ├────────────────────────┼─────────────────────────────┼──────────────────────────────────────────┤ | |
| │ EigenDA delta blobs │ $1 │ $1 │ | |
| ├────────────────────────┼─────────────────────────────┼──────────────────────────────────────────┤ | |
| │ Bandwidth │ $5 (10 ops) │ $5 │ | |
| ├────────────────────────┼─────────────────────────────┼──────────────────────────────────────────┤ | |
| │ Total │ ~$14 │ ~$144 │ | |
| └────────────────────────┴─────────────────────────────┴──────────────────────────────────────────┘ | |
| For comparison, current IPFS pinning cost (assuming Pinata or similar at $20 / 1 TB / month tier) is in the same order of magnitude. | |
| ★ Insight ───────────────────────────────────── | |
| The real "cost" of Model X isn't a dollar figure — it's the per-commit gas tax amortized across idle chains. A chain with no policy | |
| activity still pays ~$4/day on L1 for committing a state root that didn't move. That becomes the dominant line item as chain count | |
| grows. The two paths to mitigate are (a) skip commits when delta is empty, which preserves the 120 s guarantee window but breaks the | |
| deterministic cadence (and is a §S.13 design choice), or (b) stretch cadence on idle chains. Both are adjustments to the commit | |
| cadence policy, not to Model X itself. | |
| ───────────────────────────────────────────────── | |
| Scale sensitivities | |
| What makes the projection move in non-linear ways: | |
| ┌──────────────────────────────────┬──────────────────────────────────────┬──────────────────────────────────────────────────────┐ | |
| │ Dimension │ Impact of 10× growth │ Becomes problematic when │ | |
| ├──────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────┤ | |
| │ Policy count (50 → 500) │ redb: 1 GiB → 10 GiB/replica │ Never at Phase 1 scale; operators on small VMs might │ | |
| │ │ │ notice │ | |
| ├──────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────┤ | |
| │ WASM size (15 MiB → 150 MiB avg) │ Breaks the 100 MiB cache; commit cap │ Need to revisit the max_cache_size config + │ | |
| │ │ blocks │ per-commit cap │ | |
| ├──────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────┤ | |
| │ Chain count (3 → 10) │ Gas cost scales linearly │ Mainnet × 10 chains = ~$450/month just for commits │ | |
| ├──────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────┤ | |
| │ Policy deployment rate (2/day → │ EigenDA: $1 → $25/month │ Still manageable; per-commit cap protects against │ | |
| │ 50/day) │ │ bursts │ | |
| ├──────────────────────────────────┼──────────────────────────────────────┼──────────────────────────────────────────────────────┤ | |
| │ Commit cadence (120 s → 30 s) │ Gas × 4 │ Would drive mainnet cost to ~$540/month; probably │ | |
| │ │ │ not justified │ | |
| └──────────────────────────────────┴──────────────────────────────────────┴──────────────────────────────────────────────────────┘ | |
| Biggest unknown | |
| Actual WASM binary size distribution. I used 15 MiB avg as a placeholder. Want to pull actual numbers from current deployments? just | |
| admin-run --bin cli --command wasm-size-report or similar — if we have one, the projection tightens up. If most policies are <1 MiB | |
| WASM, everything above is ~10× too high. | |
| --- | |
| Want me to (a) move to Q5 (chain placement details — confirming artifact replication semantics across chains), (b) dig into the | |
| on-chain gas optimization question (skip-if-empty commits vs. deterministic cadence), or (c) pull actual WASM size numbers from the | |
| current codebase to tighten the projection? | |
| ✻ Cooked for 1m 46s | |
| ❯ the current wasm cache limits are not set. they were just stop gap. we needed wasm cache to prevent fetching the same wasm from ipfs | |
| every single task evaluation. Now that we are storing wasm bytes on newton protocol chain, do we still need cache? | |
| ⏺ Good catch — the cache's original rationale dissolves under Model X. Let me verify the current cache's role and then walk through | |
| what the new architecture should look like. | |
| Searched for 2 patterns, read 2 files (ctrl+o to expand) | |
| ⎿ PreToolUse:Read hook error ⎿ Failed with non-blocking status code: [TokenEfficiency] Hard | |
| rule violation: Re-reading wasm_cache.rs (2x) — file unchanged | |
| since last read. Consider using cached knowledge. | |
| ⎿ Loaded crates/CLAUDE.md | |
| ⎿ Loaded crates/data-provider/src/data_sources/CLAUDE.md | |
| ⏺ Confirmed what the current cache is: a disk-bytes cache at ~/.cache/newton-prover/wasm/* with LRU eviction. Sole purpose: avoid IPFS | |
| re-fetch. Under Model X, that purpose evaporates. | |
| But I found something more interesting: WasmExecutor::execute_wasm_bytes recompiles the WASM on every eval (compile sites at | |
| crates/data-provider/src/wasm/executor.rs:381,508). There's no Module cache today. The bytes cache was saving us from IPFS fetches, | |
| but wasmtime compilation — which dominates cost at 100ms–1s per WASM — happens every single task evaluation. That's a much bigger | |
| missed optimization. | |
| So the answer is nuanced: you're right that the bytes cache goes away, but we should replace it with a Module cache, not nothing. | |
| Three-tier view of caching | |
| ┌──────────┬─────────────────────────────┬───────────────────┬───────────────────────────────────────────┬───────────────────────┐ | |
| │ Layer │ What it holds │ Lookup cost │ Produce cost │ Currently cached? │ | |
| ├──────────┼─────────────────────────────┼───────────────────┼───────────────────────────────────────────┼───────────────────────┤ | |
| │ Bytes │ Raw Vec<u8> WASM │ redb read: ~0.1–1 │ IPFS fetch: 50–500 ms (today) / redb read │ Yes — disk LRU │ | |
| │ │ │ ms │ (Model X) │ │ | |
| ├──────────┼─────────────────────────────┼───────────────────┼───────────────────────────────────────────┼───────────────────────┤ | |
| │ Module │ Compiled wasmtime::Module │ HashMap lookup: │ Module::new(): 100 ms – 1 s │ No (this is the gap) │ | |
| │ │ │ ~1 µs │ │ │ | |
| ├──────────┼─────────────────────────────┼───────────────────┼───────────────────────────────────────────┼───────────────────────┤ | |
| │ Instance │ Instantiated │ — │ Instance::new(): ~1 ms │ No (too stateful to │ | |
| │ │ wasmtime::Instance │ │ │ reuse) │ | |
| └──────────┴─────────────────────────────┴───────────────────┴───────────────────────────────────────────┴───────────────────────┘ | |
| Under Model X: | |
| - The bytes layer's "produce cost" collapses from IPFS fetch to redb read (200–500× cheaper). Disk cache becomes obsolete. | |
| - The Module layer's produce cost is unchanged (wasmtime compile is WASM-shape-dependent, not source-location-dependent). This | |
| becomes the dominant hot-path cost — and it's also where the largest caching win lives. | |
| Recommendation | |
| Remove the WasmCache; add a Module cache. Net is simpler architecture AND better hot-path performance. | |
| ┌───────────────┬──────────────────────────────┬─────────────────────────────────────────────────────────┐ | |
| │ Dimension │ Today (WasmCache) │ Recommended (ModuleCache) │ | |
| ├───────────────┼──────────────────────────────┼─────────────────────────────────────────────────────────┤ | |
| │ What's cached │ Bytes on disk │ Compiled Module in RAM │ | |
| ├───────────────┼──────────────────────────────┼─────────────────────────────────────────────────────────┤ | |
| │ Keyed by │ IPFS CID │ wasmHash (keccak256 of bytes) │ | |
| ├───────────────┼──────────────────────────────┼─────────────────────────────────────────────────────────┤ | |
| │ Hit behavior │ Return bytes → still compile │ Return ready-to-instantiate Module │ | |
| ├───────────────┼──────────────────────────────┼─────────────────────────────────────────────────────────┤ | |
| │ Miss behavior │ IPFS fetch + disk write │ redb read + wasmtime compile + HashMap insert │ | |
| ├───────────────┼──────────────────────────────┼─────────────────────────────────────────────────────────┤ | |
| │ Eviction │ LRU by byte size │ LRU by cached entry count or est. RAM size │ | |
| ├───────────────┼──────────────────────────────┼─────────────────────────────────────────────────────────┤ | |
| │ Invalidation │ Manual │ Automatic (content-addressed: new wasmHash = new entry) │ | |
| └───────────────┴──────────────────────────────┴─────────────────────────────────────────────────────────┘ | |
| Eval hot path becomes: | |
| eval(policy_id, task): | |
| wasm_hash = read_leaf(policy_id).hash # redb, ~0.5 ms | |
| module = module_cache.get_or_insert(wasm_hash, || { | |
| bytes = read_leaf_bytes(wasm_hash) # redb, ~1 ms | |
| verify_jmt_inclusion(bytes, state_root) # ~1 ms | |
| Module::new(&engine, &bytes) # 100 ms – 1 s, one-time | |
| }) | |
| Instance::new(&module, &store) # ~1 ms | |
| run(instance, task) | |
| Hot path after warm-up is ~3 ms end-to-end vs today's ~100+ ms (compile-every-eval). | |
| ★ Insight ───────────────────────────────────── | |
| This is a case where removing one layer exposes a different, more valuable layer that was always there but invisible. The disk-bytes | |
| cache papered over the "fetch" cost and left "compile" as the silent hot-path tax. Model X + Module cache flips the priority: bytes | |
| are now effectively free, so compilation — always the real bottleneck — becomes the thing worth caching. Content-addressed keying (by | |
| wasmHash) means the cache is self-consistent: when a policy upgrades, its new wasmHash is a fresh cache key, old entry ages out via | |
| LRU, no explicit invalidation logic needed. This pattern shows up elsewhere in crypto systems (Merkle-addressed state, | |
| content-addressable storage) and the property is always the same: content identity = cache identity. | |
| ───────────────────────────────────────────────── | |
| Sizing the Module cache | |
| wasmtime compiled Module size is typically 2–5× the WASM byte size (native code + metadata + relocations). | |
| ┌────────────────┬────────────────────┬─────────────────┬───────────────┐ | |
| │ WASM size │ Module size (est.) │ Cache N modules │ RAM footprint │ | |
| ├────────────────┼────────────────────┼─────────────────┼───────────────┤ | |
| │ 5 MiB │ ~15 MiB │ 20 │ 300 MiB │ | |
| ├────────────────┼────────────────────┼─────────────────┼───────────────┤ | |
| │ 15 MiB (avg) │ ~45 MiB │ 20 │ 900 MiB │ | |
| ├────────────────┼────────────────────┼─────────────────┼───────────────┤ | |
| │ 30 MiB (large) │ ~90 MiB │ 20 │ 1.8 GiB │ | |
| └────────────────┴────────────────────┴─────────────────┴───────────────┘ | |
| At 50 policies/chain × 15 MiB avg WASM, the full working set is ~2–3 GiB RAM if we cache everything. Realistically, eval traffic | |
| follows power-law distribution — top-10 policies handle >90% of tasks — so caching ~20 hot Modules per chain covers the working set | |
| at <1 GiB operator RAM. | |
| Proposed config (replaces max_cache_size): | |
| pub struct DataProviderConfig { | |
| // ...existing fields... | |
| /// Maximum number of compiled wasmtime::Module entries to keep in RAM. | |
| /// LRU-evicted. Each entry is typically 2-5x the source WASM byte size. | |
| pub max_module_cache_entries: usize, // default: 32 | |
| /// Optional soft cap on total Module cache RAM footprint (MiB). | |
| /// Acts as a secondary constraint; whichever limit hits first wins. | |
| pub max_module_cache_ram_mib: Option<u64>, // default: Some(1024) | |
| } | |
| Gotchas to flag | |
| 1. wasmtime Engine is already shared across Modules — good. No duplication of JIT state. | |
| 2. Module compilation is single-threaded by default per call. If a cold-miss storm hits (e.g., new operator joining, bootstrap), | |
| first-eval latency spikes. Options: pre-warm cache during bootstrap, or use Engine::precompile_module in a background task right | |
| after state-tree apply. | |
| 3. Serialized Module persistence — wasmtime supports Module::serialize + Module::deserialize_file to persist compiled modules to | |
| disk, skipping re-compile on restart. Probably not worth it for Phase 1 (operators restart rarely; cold start is ≤30 s for 20 | |
| modules). Worth revisiting if operator restart latency becomes a pain point. | |
| 4. wasmtime version-bumps invalidate serialized Modules. If we ever add disk-persisted Module cache, key by (wasmHash, | |
| wasmtime_version) to avoid loading stale compiled code. | |
| 5. Security angle. Since the Module cache key is wasmHash = keccak256(bytes), and the leaf in redb commits to those exact bytes via | |
| JMT inclusion proof, no additional integrity check is needed on cache hits. Content addressing carries the integrity guarantee. This | |
| is cleaner than today's IPFS-fetched bytes where we had to verify CID → bytes mapping separately. | |
| What to delete when implementing | |
| - crates/data-provider/src/wasm_cache.rs entire file | |
| - WasmCache struct and all call sites | |
| - DataProviderConfig::max_cache_size field | |
| - IPFS fetch logic in data_sources/wasm.rs::fetch_data (bytes come from redb now) | |
| - ProjectDirs disk-cache path plumbing in multiple places | |
| Gives us less code + faster hot path + clearer architecture. Clean win. | |
| --- | |
| Agree on the direction? If so, this likely belongs as Q4-sub-branch-c → caching layer in our spec. Next decisions after this: | |
| - (a) Move to Q5 (chain placement) — artifact replication semantics across chains | |
| - (b) Drill down on Module cache sizing / pre-warm strategy (important for operator RAM planning) | |
| - (c) Zoom out to the policy deployment write path — the authoring side of this new storage (which we deferred from Q4) | |
| Which? | |
| ✻ Cogitated for 2m 0s |
all looks good. One nit. instead of policy "plan" command, can we name it like policy "describe"? 1. expireAfter refers to how long
the policy evaluation result is valid for, not how long the policy is valid for. Whether testnet or mainnet, expireAfter should be
larger than 0. 2. (a) 3. What do you mean? clarify. The dev needs to be able to set up the dev's deployer signer wallet for the cli.
(and also their gateway API key for other commands). Either env var or direct params.
⏺ All three answers locked. Let me capture them and renaming, then close Q8 and pivot to Q10.
Q8 finalization
┌────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────────────────────┐
│ Sub-decision │ Status │
├────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┤
│ Q8.1 CLI is canonical interface, multi-step, schema-driven │ LOCKED │
├────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┤
│ Q8.2 Breaking redesign, no legacy carry-over │ LOCKED │
├────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┤
│ Q8.3 Any-order idempotent (Option B); atomic bundle = Phase 2 │ LOCKED │
├────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┤
│ Q8.4 policy describe (read-only inspection of dependency DAG, kubectl-style) │ LOCKED — replaces my │
│ │ proposed plan │
├────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┤
│ Q8.5 expireAfter > 0 enforced at deploy time on every chain │ LOCKED │
├────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┤
│ Q8.6 Param mutation = new instance with new ID; no "update-params" syntactic sugar in Phase 1 │ LOCKED │
├────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────────────────────┤
│ Q8.7 --client and signer wallet are distinct concepts; signer + gateway API key configured │ LOCKED │
│ globally via env var or flag, not per-command │ │
└────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────────────────────┘
★ Insight ─────────────────────────────────────
You're right to correct me on expireAfter. I conflated two lifetimes that the existing Newton model already separates: deployment
lifetime (how long the on-chain instance struct lives — "forever" in Phase 1, no unregistration) vs attestation lifetime (how long an
evaluation result remains valid before re-evaluation is required — the per-task expiration = block.number + expireAfter). The
existing AttestationExpired selector in error-selectors.md fires precisely because of expireAfter=0 misconfigurations creating
dead-on-arrival attestations. So enforcing expireAfter > 0 is a guardrail against a known footgun, not a new policy decision — same
rule applies to testnet and mainnet because the attestation lifecycle is identical on both.
─────────────────────────────────────────────────
★ Insight ─────────────────────────────────────
On signer vs --client separation: this is genuinely two roles, and conflating them would have been a security regression. The signer
is "who pays for the tx and authenticates the deployment action"; the client is "who owns the resulting instance and whose params are
bound to its ID." A Newton-internal deployer or a tooling team often signs on behalf of a customer wallet. Defaulting --client to
signer would have silently swapped ownership in those flows. The kubectl/Terraform model is the right precedent: signer/credentials
are global session config, resource ownership is explicit per-resource. Your separation of "signer key (deployer wallet)" vs "API key
(gateway auth)" further matches Newton's reality where some commands (template register, instance deploy) need on-chain signing and
others (task creation, evaluation) need gateway auth — they're authenticating to different systems.
─────────────────────────────────────────────────
CLI configuration model (per Q8.7)
Two independent credential lanes, each settable via env or flag, with config file as fallback:
┌────────────────┬─────────────────────────────┬────────────────────────┬─────────────────┬─────────────────────────────────────┐
│ Credential │ Purpose │ Env var │ CLI flag │ Config file │
├────────────────┼─────────────────────────────┼────────────────────────┼─────────────────┼─────────────────────────────────────┤
│ Deployer │ Sign on-chain deploy txs │ NEWTON_SIGNER_KEY │ --signer-key │ ~/.newton/config.toml → signer_key │
│ signer key │ │ │ 0x... │ = "..." │
├────────────────┼─────────────────────────────┼────────────────────────┼─────────────────┼─────────────────────────────────────┤
│ Gateway API │ Auth to gateway for task │ NEWTON_GATEWAY_API_KEY │ --api-key ... │ ~/.newton/config.toml → │
│ key │ create, eval, read │ │ │ gateway_api_key = "..." │
├────────────────┼─────────────────────────────┼────────────────────────┼─────────────────┼─────────────────────────────────────┤
│ Chain RPC URL │ Submit on-chain txs to │ NEWTON_RPC_URL │ --rpc-url ... │ ~/.newton/config.toml → rpc_url = │
│ │ │ │ │ "..." │
├────────────────┼─────────────────────────────┼────────────────────────┼─────────────────┼─────────────────────────────────────┤
│ Gateway URL │ Address of gateway for │ NEWTON_GATEWAY_URL │ --gateway ... │ ~/.newton/config.toml → gateway_url │
│ │ non-chain commands │ │ │ = "..." │
└────────────────┴─────────────────────────────┴────────────────────────┴─────────────────┴─────────────────────────────────────┘
Resolution order (highest precedence first): CLI flag → env var → config file → error.
Adds a newton-cli config family for setup ergonomics:
newton-cli config init # interactive: writes ~/.newton/config.toml
newton-cli config set signer-key 0x...
newton-cli config set gateway-api-key sk_...
newton-cli config show # masks secrets in output
newton-cli config use-profile # multi-env: dev / staging / mainnet profiles
★ Insight ─────────────────────────────────────
The use-profile command pays for itself fast in the testnet→mainnet transition. Newton has Sepolia, Base Sepolia, and mainnets; a dev
iterating against testnet is one fat-fingered tx away from spending real funds when they switch to mainnet. Profiles let them keep
mainnet credentials in a separate file (or keychain) and require an explicit use-profile mainnet to even target it. This is the same
pattern as aws --profile prod, gcloud config configurations, kubectl config use-context. Cheap to add; expensive bug to fix once it's
bitten someone.
─────────────────────────────────────────────────
Updated dev workflow with the locked CLI
Smallest deploy:
export NEWTON_SIGNER_KEY=0x...
export NEWTON_RPC_URL=https://...
newton-cli policy template register --rego ./policy.rego
newton-cli policy instance deploy
--template
--client 0xclientAddr \ # explicit, not defaulted
--no-params
--expire-after 150 # required, > 0 enforced on-chain
Inspecting state:
newton-cli policy describe --policy-instance
Walks dependency DAG, shows what's registered/deployed/missing.
Mutating params (per Q8.6 — new instance, no syntactic sugar):
Old instance:
newton-cli policy-data instance show 0xold
templateId: 0xt, params: {max_limit: 100}, paramsHash: 0xh1, instanceId: 0xold
Deploy new instance with updated params:
newton-cli policy-data instance deploy --template 0xt --client 0xc --params ./new-params.json
paramsHash: 0xh2, instanceId: 0xnew
Re-deploy policy instance pointing at the new data instance:
newton-cli policy instance deploy --template 0xpt --client 0xc --params ./pp.json --data-instances 0xnew
The dev (or their tooling) is responsible for migrating off-chain references from 0xold to the new policy instance ID. We don't add a
"current instance pointer" indirection on-chain.
★ Insight ─────────────────────────────────────
The "no on-chain mutable pointer" choice has a non-obvious downstream benefit: the off-chain client systems (wallets, dApps) become
the single source of truth for "which instance is currently in use." This forces those systems to handle the migration explicitly,
which is healthier than a contract-level redirect that lets stale clients keep hitting old instances forever. It also keeps the
on-chain surface minimal — every instance entry is immutable except for expireAfter (which is per-task and carried in attestations,
not on-chain mutation), making the registry behavior much easier to reason about during audits and challenges. The trade-off: a
migration UX cost for the dev. Phase 2 governance can revisit if migration friction becomes a real complaint.
─────────────────────────────────────────────────
Q8 — CLOSED
All sub-decisions locked. Open items deferred to spec drafting:
- Exact TOML schema for ~/.newton/config.toml
- Profile encryption-at-rest (do we shell out to OS keychain or read plaintext from disk?) — Phase 2 hardening
- describe output format (table vs JSON-with---output json flag) — implementation detail
- Idempotency semantics if a template register is called concurrently from two CLIs (likely on-chain-level: second tx becomes a no-op
observing existing template; CLI surfaces "already registered" message)
Q10 — Governance & lifecycle (final substantive Q)
This is largely Phase 2, but I want to confirm the Phase 1 baseline and what gets explicitly deferred. Five governance dimensions:
Q10.1 Template authorship & ownership
- Phase 1 default (proposed): template author is recorded in the registry struct (the address author field already locked), but Phase
1 grants no special powers to the author — they can't deprecate, can't update, can't transfer. - Phase 2: add author-can-deprecate + author-can-transfer-authorship.
Q10.2 Template deprecation / unregistration
- Phase 1 (per U1, already locked): templates persist forever; no deprecate or unregister function.
- Phase 2: add a deprecated boolean that prevents new instance deployments referencing this template, but doesn't break existing
instances.
Q10.3 Registry permissioning toggle authority
- Phase 1 (already locked in Q5 sub-decision): templateRegistrationPermissionless and instanceDeploymentPermissionless toggles owned
by registry owner (multisig). - Open: who is the registry owner on testnet (Newton deployer EOA?) and mainnet (Safe multisig per existing pattern in
transfer-ownership.sh)?
Q10.4 Permissioned-mode allowlist mechanism
- When a registry is in permissioned mode (e.g., mainnet default), how does an author/client get added?
- Phase 1 proposal: OperatorRegistry-style: addTemplateAuthor(address) and addInstanceDeployer(address) owner-only functions.
- Or: re-use existing OperatorRegistry allowlist? (probably no — different role)
Q10.5 Schema evolution (policyParamsSchema v2)
- A template is content-addressed by its inputs including the schema bytes. So "evolving the schema" means a new policyTemplateId.
There's no in-place schema upgrade. - Phase 1 proposal: template authors who want to evolve register a new template with a new schema; clients migrate at their pace. No
special governance machinery. - Phase 2: add a policyTemplateLineage mapping for human-readable "v2 of v1" tracking — purely advisory metadata.
Most of these have obvious Phase 1 defaults. The two that benefit from your input:
Q10.3 (registry owner identity): Should the new PolicyRegistry and PolicyDataRegistry follow the existing Safe multisig pattern
(SAFE_ADDRESS=0x... ./contracts/deploy/transfer-ownership.sh) on mainnet, with the deployer EOA owning during testnet? This is the
established Newton pattern and I'd recommend yes — same transfer-ownership.sh flow, no new governance surface.
Q10.4 (permissioned-mode allowlist): Two allowlist roles (template authors + instance deployers, matching the two permission
toggles), or a single combined allowlist? I'd recommend two separate allowlists because the threat models differ — a malicious
template author can ship dangerous Rego logic, while a malicious instance deployer can only mis-configure params on already-vetted
templates. Two allowlists let mainnet have stricter author gating with looser deployer access.
Confirm Q10.3=Safe-on-mainnet-EOA-on-testnet and Q10.4=two-separate-allowlists, plus any pushback on Q10.1, Q10.2, Q10.5? Then I'll
close Q10 and we move to spec drafting.
✻ Sautéed for 1m 25s
❯ one qusetion for Q8. for policy template register, it is multichain/chain agnostic right? But instance deploy is chains specific?
how should we handle rpc url / chain id configurations for the cli for devs?
⏺ Sharp question — this distinction has real DX consequences, so worth pinning down explicitly.
Q8.8 Chain semantics: templates vs instances
Templates: chain-agnostic ID, chain-specific registration.
The policyTemplateId = keccak256(rego_bytes ‖ policyParamsSchema_bytes ‖ canonical(policyDataTemplateRefs)) formula has zero chainId
input. So the same template content produces the same ID on every chain. But the PolicyRegistry contract is deployed per chain, so
the on-chain row exists per chain.
Instances: fully chain-specific.
The locked formula policyInstanceId = keccak256(chainId ‖ clientAddr ‖ policyTemplateId ‖ paramsHash) includes chainId, so the same
template + same client + same params → different instance IDs on different chains. This is correct: the same client logically wants
distinct behavior on Sepolia vs Mainnet (different params, different secrets, different expireAfter values for differing block
times).
★ Insight ─────────────────────────────────────
Three layers, three different chain semantics. EigenDA is chain-agnostic: artifact bytes live in one DA layer regardless of how many
EVM chains we deploy on. Templates have chain-agnostic IDs but chain-specific on-chain registrations — same policyTemplateId on every
chain, but each chain's registry needs its own row pointing at the same EigenDA blob ref. Instances are fully chain-specific. This
three-layer separation is actually a nice property: it means uploading bytes is a one-shot operation (not N-times for N chains), and
the dev never has to worry about content drift across chains because content is deduplicated at the DA layer. The cost is just N
on-chain registerTemplate calls — cheap because they only store ~3 blob refs each.
─────────────────────────────────────────────────
CLI configuration model for chains
Per-profile config in ~/.newton/config.toml with a [chains] table and a default-chain pointer:
[profiles.testnet]
network = "testnet"
signer_key = "0x..."
gateway_api_key = "..."
default_chain = "sepolia" # optional; if absent, --chain becomes required
[profiles.testnet.chains.sepolia]
chain_id = 11155111
rpc_url = "https://sepolia.infura.io/v3/..."
[profiles.testnet.chains.base-sepolia]
chain_id = 84532
rpc_url = "https://sepolia.base.org"
[profiles.testnet.chains.op-sepolia]
chain_id = 11155420
rpc_url = "https://sepolia.optimism.io"
[profiles.mainnet]
network = "mainnet"
signer_key = "0x..."
default_chain = "ethereum"
[profiles.mainnet.chains.ethereum]
chain_id = 1
rpc_url = "https://..."
... etc
Resolution precedence (highest first):
- CLI flag: --chain sepolia or --chain-id 11155111 or --rpc-url https://...
- Env var: NEWTON_CHAIN=sepolia, NEWTON_RPC_URL=...
- Profile default_chain (resolves to the named entry under [profiles.X.chains.Y])
- Error: "specify --chain or set default_chain in profile"
This mirrors the existing Newton NetworkMode pattern (per agent-guide.md — gateway/operator support --network local|testnet|mainnet),
keeping CLI semantics consistent with the gateway and operator binaries.
Command-level chain semantics (which commands need a chain)
┌─────────────────────────────────┬──────────────────────────────────┬──────────────────────────────────────────────────────────┐
│ Command │ Chain semantics │ Why │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ policy template compute-id │ Chain-agnostic (no flag needed) │ Pure local hash computation │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ policy-data template compute-id │ Chain-agnostic │ Same │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ policy template register │ Chain-required (multi-chain via │ Writes to per-chain registry; same EigenDA upload feeds │
│ │ --chains) │ all chains │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ policy-data template register │ Chain-required (multi-chain via │ Same │
│ │ --chains) │ │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ policy template show │ Chain-required or --all-chains │ Reads on-chain registration row; with --all-chains shows │
│ │ │ registration matrix │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ policy template list │ Chain-required │ Reads per-chain registry │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ policy instance deploy │ Strictly single-chain │ Instance ID embeds chainId; ambiguity = bug │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ policy instance show │ Chain-required │ Reads on-chain instance row │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ policy instance list --client │ Chain-required │ Reads per-chain registry │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ policy-data instance deploy │ Strictly single-chain │ Same as policy instance │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ policy-data instance │ Strictly single-chain │ Secrets land in per-chain state-tree namespace 0x03 │
│ set-secrets │ │ │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ policy describe │ Chain-required │ DAG walk is per-chain │
│ --policy-instance │ │ │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ policy describe │ Chain-required or --all-chains │ Templates can be cross-chain │
│ --policy-template │ │ │
├─────────────────────────────────┼──────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ config init / set / show │ Chain-agnostic │ Local config edit │
└─────────────────────────────────┴──────────────────────────────────┴──────────────────────────────────────────────────────────┘
Strictly single-chain commands deliberately reject --chains / --all-chains to prevent the dev from accidentally deploying the same
instance config across chains and producing N different instance IDs they then have to track. If they want multi-chain instance
deploys, the right pattern is to script the loop themselves with explicit per-chain params (since expireAfter and possibly params
should differ across chains).
Multi-chain template register: concrete flow
newton-cli policy template register
--rego ./policy.rego
--params-schema ./params.schema.json
--data-templates 0xabc
--chains sepolia,base-sepolia,op-sepolia
CLI internal sequence:
- Compute policyTemplateId locally — deterministic, identical for all target chains
- Upload Rego bytes to EigenDA → blob ref 0xR (one upload, shared by all chains)
- Upload schema bytes to EigenDA → blob ref 0xS (one upload)
- For each chain in --chains:
- Connect to [profiles.X.chains.Y] RPC
- Query PolicyRegistry.templates(policyTemplateId) — if registered, log "skipped (already registered)"
- Otherwise submit registerTemplate(blobRefR, blobRefS, [0xabc]) tx - Print summary table: chain → registered/skipped/failed
newton-cli policy template register --rego ./policy.rego --chains sepolia,base-sepolia,op-sepolia
Computed policyTemplateId: 0x4f2a...
Uploaded Rego to EigenDA: blob ref 0xR
Skipped policyParamsSchema upload (empty)
Registering on 3 chains:
sepolia : registered (tx 0xabc...)
base-sepolia : already registered (skipped)
op-sepolia : registered (tx 0xdef...)
★ Insight ─────────────────────────────────────
The "already registered" idempotency is what makes multi-chain register low-anxiety — you can re-run the command after a failed run
and only the missing chains get hit. Combined with content-addressed IDs, this turns template registration into something close to
declarative state management: "I want this template to exist on these chains" is the user's intent, and the CLI converges to that
state regardless of starting condition. It also means a CI pipeline can blindly re-run register on every deploy without producing
duplicate state — same property as kubectl apply.
─────────────────────────────────────────────────
Cross-chain template propagation — explicitly NOT in Phase 1
Worth flagging: there's no automatic cross-chain sync of template registrations (à la transporter-style propagation). If a template
is registered on Sepolia but not Base Sepolia, an instance deploy on Base Sepolia will revert with TemplateNotRegistered (or
equivalent typed error). The dev must explicitly register on every chain they intend to deploy instances on.
This matches the lazy-load fetch model from Q6: each chain's PolicyRegistry row is independently authoritative, no cross-chain trust
assumption.
Phase 2 could add a template propagate --from sepolia --to base-sepolia helper, or even auto-propagation through transporter, but
it's strictly an ergonomics improvement — not a correctness requirement.
Q8 — closing for real this time
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬────────┐
│ Sub-decision │ Status │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ Q8.1 CLI canonical, multi-step, schema-driven │ LOCKED │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ Q8.2 Breaking redesign │ LOCKED │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ Q8.3 Any-order idempotent (Option B); atomic = Phase 2 │ LOCKED │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ Q8.4 policy describe (kubectl-style) │ LOCKED │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ Q8.5 expireAfter > 0 enforced on every chain │ LOCKED │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ Q8.6 Param mutation = new instance with new ID │ LOCKED │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ Q8.7 Signer + API key configured globally; --client distinct, no signer-default │ LOCKED │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ Q8.8 Templates chain-agnostic ID + chain-specific registration; instances strictly single-chain; multi-chain via │ LOCKED │
│ --chains only on template ops │ │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────┘
Now back to Q10 — same five sub-decisions as before. Most are obvious Phase 1 deferrals. The two needing your input are still:
Q10.3: PolicyRegistry and PolicyDataRegistry follow existing Newton ownership pattern — Safe multisig on mainnet (via
transfer-ownership.sh), deployer EOA on testnet. Confirm?
Q10.4: When a registry is in permissioned mode, two separate allowlists (templateAuthors, instanceDeployers) rather than one combined
list — different threat models warrant different access. Confirm?
❯ Q10 all agreed. including 3 and 4
⏺ All 10 grilling questions closed. Locking Q10 and presenting the unified design summary.
Q10 Governance & lifecycle — LOCKED
┌─────────────────────────────────────────────────────┬────────┬─────────────────────────────────────────────────────────────────┐
│ Sub-decision │ Status │ Rationale │
├─────────────────────────────────────────────────────┼────────┼─────────────────────────────────────────────────────────────────┤
│ Q10.1 Template author field recorded but no special │ LOCKED │ Author identity preserved on-chain for future Phase 2 expansion │
│ powers in Phase 1 │ │ (deprecate / transfer / update) without contract migration │
├─────────────────────────────────────────────────────┼────────┼─────────────────────────────────────────────────────────────────┤
│ Q10.2 No deprecation / unregistration in Phase 1 │ LOCKED │ Templates and instances persist forever; matches PR #570 │
│ (per U1) │ │ Sub-project A pattern │
├─────────────────────────────────────────────────────┼────────┼─────────────────────────────────────────────────────────────────┤
│ Q10.3 PolicyRegistry + PolicyDataRegistry follow │ │ Re-uses contracts/deploy/transfer-ownership.sh flow; zero new │
│ existing Newton ownership: Safe multisig on │ LOCKED │ governance surface │
│ mainnet, deployer EOA on testnet │ │ │
├─────────────────────────────────────────────────────┼────────┼─────────────────────────────────────────────────────────────────┤
│ Q10.4 Two separate allowlists (templateAuthors, │ │ Different threat surfaces — author ships logic, deployer ships │
│ instanceDeployers) when registry is in permissioned │ LOCKED │ config │
│ mode │ │ │
├─────────────────────────────────────────────────────┼────────┼─────────────────────────────────────────────────────────────────┤
│ Q10.5 Schema evolution = new template │ │ Falls out of content-addressing; clients migrate at their own │
│ (content-addressed); no in-place upgrade; lineage │ LOCKED │ pace │
│ tracking deferred to Phase 2 │ │ │
└─────────────────────────────────────────────────────┴────────┴─────────────────────────────────────────────────────────────────┘
★ Insight ─────────────────────────────────────
The Phase 1 governance posture here is deliberately minimal: store enough metadata to enable future governance (author,
registeredAt), but expose zero governance verbs (deprecate, transfer, update) on the contract surface. This keeps the Phase 1 attack
surface small and audit-friendly, while leaving room for Phase 2 to add governance via upgrade without a data migration. The pattern
matches how Newton's existing registries evolved — OperatorRegistry started as a simple allowlist and accreted governance hooks over
time as concrete needs emerged. Premature governance machinery in Phase 1 would have to be defended even when no one is using it.
─────────────────────────────────────────────────
★ Insight ─────────────────────────────────────
Q10.4's two-allowlist split is worth a second look because it affects on-chain storage shape. A combined single allowlist with a role
bitmap (mapping(address => uint8 roles) where bit 0 = author, bit 1 = deployer) is cheaper for storage and lets one address hold
both roles with a single SSTORE. But two separate mapping(address => bool) mappings are simpler to reason about, easier to audit, and
match the existing OperatorRegistry shape on the codebase. For Phase 1 with low-volume governance writes, the cost difference is
negligible — go with the two-mapping approach for code clarity. Phase 2 hardening can revisit if storage cost ever matters.
─────────────────────────────────────────────────
All 10 grilling Qs — unified design at a glance
┌─────────────────────────┬──────────────────────────────────────────────────────────┬───────────────────────────────────────────┐
│ Q │ Decision │ Key constraint │
├─────────────────────────┼──────────────────────────────────────────────────────────┼───────────────────────────────────────────┤
│ Q1 Scope │ Validium pivot: bytes via EigenDA, only commitments │ Matches Sub-project A pattern │
│ │ on-chain │ │
├─────────────────────────┼──────────────────────────────────────────────────────────┼───────────────────────────────────────────┤
│ Q2 Per-chain registries │ Each chain has own PolicyRegistry + PolicyDataRegistry │ Instances bind chainId in their ID │
├─────────────────────────┼──────────────────────────────────────────────────────────┼───────────────────────────────────────────┤
│ Q3 Asymmetric identity │ │ │
│ (R2 with │ Templates: content-addressed, immutable. Instances: │ OOP class/instance distinction │
│ template/instance │ (chainId, clientAddr, templateId, paramsHash) │ │
│ split) │ │ │
├─────────────────────────┼──────────────────────────────────────────────────────────┼───────────────────────────────────────────┤
│ Q4 paramsHash trust │ paramsHash in instance ID + included in attestation │ Three-layer drift defense │
│ │ digest │ (deploy/eval/challenge) │
├─────────────────────────┼──────────────────────────────────────────────────────────┼───────────────────────────────────────────┤
│ Q5 Topology + DA + │ EigenDA primary, three schemas (params public, secrets │ │
│ schemas + naming + │ private), 50 MiB cap, default permissionless on testnet │ State-tree namespaces 0x04-0x08 │
│ permissions │ / permissioned on mainnet │ │
├─────────────────────────┼──────────────────────────────────────────────────────────┼───────────────────────────────────────────┤
│ │ Five tiers: enclave RAM → daemon RAM → daemon disk → │ Content-addressing eliminates race │
│ Q6 Cache topology │ EigenDA → bootnode. Lazy fetch. │ conditions │
│ │ Verify-on-fill-trust-on-hit │ │
├─────────────────────────┼──────────────────────────────────────────────────────────┼───────────────────────────────────────────┤
│ │ Two-tier challenge: digest reconstruction (wrong-bytes) │ │
│ Q7 Verification surface │ + SP1 fraud proof (wrong-result). 11-class mismatch │ Slashing primitive: │
│ │ matrix. Permissionless challenges; gas-only cost; │ ChallengeLib.slashOperator(operator, 10) │
│ │ cryptographic spam protection │ │
├─────────────────────────┼──────────────────────────────────────────────────────────┼───────────────────────────────────────────┤
│ │ CLI-canonical, breaking redesign, any-order idempotent. │ │
│ Q8 Deployment flow │ policy describe for DAG inspection. Templates │ expireAfter > 0 enforced everywhere; │
│ │ chain-agnostic ID + multi-chain register; instances │ param mutation = new instance │
│ │ strictly single-chain. Profiles + per-chain config │ │
├─────────────────────────┼──────────────────────────────────────────────────────────┼───────────────────────────────────────────┤
│ Q9 Migration │ Clean fresh start (M1); no legacy NewtonPolicy / │ Existing policy.sol files removed │
│ │ NewtonPolicyData support │ entirely │
├─────────────────────────┼──────────────────────────────────────────────────────────┼───────────────────────────────────────────┤
│ │ Author recorded but no powers in Phase 1. Safe multisig │ │
│ Q10 Governance │ on mainnet. Two separate allowlists. Schema evolution = │ All real governance deferred to Phase 2 │
│ │ new template │ │
└─────────────────────────┴──────────────────────────────────────────────────────────┴───────────────────────────────────────────┘
Code surface deltas (entering / leaving the codebase)
Removed from contracts/src/:
- NewtonPolicy.sol (factory + per-policy contract → replaced by template/instance pattern)
- NewtonPolicyData.sol (same)
- NewtonPolicyFactory.sol (atomic multi-artifact deploy moves to optional PolicyDeployer factory in Phase 2)
Added to contracts/src/:
- PolicyRegistry.sol (templates + instances, two mappings)
- PolicyDataRegistry.sol (same shape)
- Phase 2: PolicyDeployer.sol (atomic bundle factory)
Added to crates/:
- crates/cli/src/policy/ — full new CLI surface (template, instance, describe, config)
- crates/data-provider/src/cache/ — five-tier cache implementation
- crates/core/src/policy/identity.rs — local ID computation (policy_template_id, policy_instance_id, etc.)
Modified in crates/:
- crates/data-provider/data-provider.toml — gain max_per_window_cache_size = 52428800 (50 MiB)
- crates/operator/src/policy.rs — replace IPFS fetcher with state-tree + DA reader using policyTemplateId / policyDataTemplateId
lookups - crates/chainio/src/avs/reader.rs — gain read_policy_template, read_policy_instance, etc.
- crates/chainio/src/avs/writer.rs — gain register_policy_template, deploy_policy_instance, etc.
Cross-project integration points (Sub-project A dependency):
- State-tree namespaces 0x04-0x08 (Sub-project B writes)
- 0x03 secrets namespace (Sub-project B's set-secrets delegates to A's signed-write)
- 120s state-commit cadence (affects how soon set-secrets is observable to operators)
Where we go next
The grill-me phase is complete. Three options for the next step, each appropriate at this stage:
Option 1 — Direct to spec drafting (docs/superpowers/specs/2026-04-22-onchain-policy-storage-design.md). I'd structure the spec as:
Overview / Motivation / Detailed Design (per-Q sections in the order Q1→Q10) / CLI surface / Code-surface delta / Sub-project A
dependencies / Phase 2 backlog / Open questions for review. This gets a written artifact you can audit holistically before committing
to implementation.
Option 2 — Consistency dry-run first. Walk through 3-5 hostile scenarios end-to-end (e.g., "operator deploys instance, fetches data,
evaluates, challenger files wrong-result challenge — every step traces to a locked decision") and confirm we have no missing
primitives. Catches design holes that don't show up when each Q is examined in isolation. Adds ~30 minutes before spec drafting but
reduces spec-rewrite risk.
Option 3 — Linear ticket scaffolding first. Sub-project A's Phase 1 MVP was decomposed across 50 NEWT-1003..1052 tickets; doing the
same for Sub-project B before the spec lands gives the spec concrete implementation targets and ordering. Risk: doing this without
the spec written can over-commit to specifics that change during spec review.
My recommendation is Option 2 → Option 1. Half an hour of hostile-scenario walkthrough is a cheap insurance policy on a 10-question
design, and the scenarios you generate become valuable test cases for the spec's "Threat model" section. Option 3 is best done last,
once the spec is at least in draft.
Which direction do you want?
✻ Worked for 1m 26s
※ recap: Designing Sub-project B (on-chain policy storage migration) via grill-me interview; all 10 questions now locked. Next action
is your pick between hostile-scenario consistency dry-run, direct spec drafting, or Linear ticket scaffolding.
Scenario A — Happy path: full deploy + eval + attestation
Setup: Dev "Bob" wants to deploy a policy with one data dependency on Sepolia. Policy: max_amount rule. Data: kyc_status lookup with
API key secret.
#: 1
Action: Bob runs policy-data template register --wasm ./kyc.wasm --params-schema ./pp.json --secrets-schema ./ss.json --chains sepolia
Locked Q: Q5, Q8.8
Verdict: OK
────────────────────────────────────────
#: 2
Action: CLI computes policyDataTemplateId = keccak256(wasm ‖ pdp_schema ‖ secrets_schema), uploads bytes to EigenDA, queries chain (no
existing row), submits registerTemplate tx
Locked Q: Q3, Q5
Verdict: OK
────────────────────────────────────────
#: 3
Action: Bob runs policy template register --rego ./rule.rego --params-schema ./pp.json --data-templates 0xKYC --chains sepolia
Locked Q: Q3, Q8.8
Verdict: OK
────────────────────────────────────────
#: 4
Action: Bob runs policy-data instance deploy --template 0xKYC --client 0xBob --params '{"region":"us"}' --expire-after 150 --chain
sepolia
Locked Q: Q3, Q8.5, Q8.7
Verdict: OK
────────────────────────────────────────
#: 5
Action: CLI fetches policyDataParamsSchema from EigenDA, validates params locally, computes paramsHash, computes policyDataInstanceId =
keccak256(11155111 ‖ 0xBob ‖ 0xKYC ‖ paramsHash), submits deployInstance tx
Locked Q: Q3, Q4, Q8.5
Verdict: OK
────────────────────────────────────────
#: 6
Action: Bob runs policy-data instance set-secrets --instance 0xDataInst --secrets '{"api_key":"sk_..."}' --chain sepolia
Locked Q: Q5 + cross-project Sub-A 0x03 namespace
Verdict: See gap
────────────────────────────────────────
#: 7
Action: CLI fetches secretsSchema from EigenDA, validates locally, fetches each operator's HPKE pubkey, fan-out HPKE seal per operator,
submits to state-tree namespace 0x03 via Sub-A signed-write API
Locked Q: Q5, Q6 (TEE individual mode for secrets)
Verdict: GAP G1
────────────────────────────────────────
#: 8
Action: Bob runs policy instance deploy --template 0xPol --client 0xBob --params '{"max_amount":1000}' --data-instances 0xDataInst
--expire-after 150 --chain sepolia
Locked Q: Q3
Verdict: OK
────────────────────────────────────────
#: 9
Action: Eval: Gateway routes task to operator. Operator looks up policyInstanceId in PolicyRegistry, dereferences template, fetches
metadata via 5-tier cache cascade, dereferences each policyDataInstanceRefs[i], fetches WASM, reads encrypted secrets from state-tree
0x03, decrypts in enclave, evaluates Rego with policyParams + policyDataParams + decrypted secrets
Locked Q: Q6, Q5
Verdict: OK
────────────────────────────────────────
#: 10
Action: Operator constructs attestation digest including chainId, policyInstanceId, policyTemplateId, all artifactHashes, paramsHash,
referenceBlock, expirationBlock, per-data-instance entries; BLS signs; returns to gateway
Locked Q: Q4, Q7
Verdict: OK
────────────────────────────────────────
#: 11
Action: Gateway aggregates BLS signatures, calls respondToTask, attestation lands on-chain
Locked Q: Existing Newton
Verdict: OK
Gap G1 (step 7): We locked "secrets encrypted per-operator individual mode" in Q6, but did NOT lock how the CLI discovers operator-set
membership at fan-out time. Three options:
- (a) CLI calls a gateway endpoint gateway_getSecretsRecipients() that returns current operator set + their pubkeys
- (b) CLI reads OperatorRegistry directly + queries each operator's newt_getSecretsPublicKey via socket
- (c) Dedicated operator-key-cache service
Recommend (a) — gateway already aggregates this view; reusing it ensures CLI's view matches what operators will use during evaluation.
★ Insight ─────────────────────────────────────
The set-secrets fan-out is structurally identical to the existing privacy-task encryption fan-out in crates/core/src/crypto/privacy.rs.
Same code path (HPKE per operator + state-tree write) works here. The real question is where the operator set comes from. Reusing the
gateway endpoint avoids a class of bug where CLI's view of operators diverges from the gateway's — a divergence that could produce
ciphertexts no operator can decrypt, silently breaking evaluation.
─────────────────────────────────────────────────
Scenario B — Wrong-bytes challenge
Setup: Operator returns an attestation but secretly fetched a malicious WASM that produces a permissive answer instead of the real
kyc.wasm. Challenger detects via digest reconstruction.
┌─────┬─────────────────────────────────────────────────────────────────────────┬────────────────────────────────────────┬─────────┐
│ # │ Action │ Locked Q │ Verdict │
├─────┼─────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────┼─────────┤
│ 1 │ Operator gets task at block N. Reads policyInstanceId on-chain at block │ Q6 │ OK │
│ │ N. Resolves policyTemplateId and policyDataInstanceIds. So far honest. │ │ │
├─────┼─────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────┼─────────┤
│ │ Operator's malicious daemon serves fake WASM bytes. The │ │ │
│ 2 │ policyDataTemplateId it commits in the digest is the ORIGINAL template │ Q3 (template ID = keccak256 of bytes) │ OK │
│ │ ID (because that's what's in the on-chain row at block N) │ │ │
├─────┼─────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────┼─────────┤
│ │ Operator commits artifactHash_kyc_wasm = keccak256(fake_wasm_bytes) in │ │ │
│ 3 │ attestation digest. This is the operator's own claim — they can put │ Q4, Q7 │ GAP G2 │
│ │ anything here. │ │ │
├─────┼─────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────┼─────────┤
│ │ Challenger reads attestation. Pulls digest. Reads on-chain at │ │ │
│ 4 │ referenceBlock = N: PolicyRegistry.instances(0xInst).policyTemplateId = │ Q7 │ OK │
│ │ 0xT, PolicyDataRegistry.templates(0xDataT).wasmBlobRef. │ │ │
├─────┼─────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────┼─────────┤
│ │ Challenger fetches WASM from EigenDA blob ref 0xR, computes │ │ │
│ 5 │ keccak256(real_wasm_bytes) = 0xRealHash. Compares to │ Q7 │ OK │
│ │ artifactHash_kyc_wasm in digest. Mismatch detected. │ │ │
├─────┼─────────────────────────────────────────────────────────────────────────┼────────────────────────────────────────┼─────────┤
│ │ Challenger calls raiseAndResolveChallenge with mismatch proof. On-chain │ Q7.5 │ │
│ 6 │ verifier re-fetches wasmBlobRef, re-computes hash, compares to │ (ChallengeLib.slashOperator(operator, │ OK │
│ │ digest's artifactHash. Slashes operator 10%. │ 10)) │ │
└─────┴─────────────────────────────────────────────────────────────────────────┴────────────────────────────────────────┴─────────┘
Gap G2 (step 3): We locked that the digest contains artifactHashes, but did NOT explicitly lock that the on-chain ChallengeVerifier
re-derives expected artifactHashes from the on-chain blob refs at referenceBlock as the comparison source of truth.
If left implicit, an implementer could mistakenly accept the operator's claimed hashes without re-derivation, defeating the whole
detection mechanism.
Decision needed: Spec must mandate the verifier protocol as four discrete steps:
- Challenger fetches DA bytes, computes hashes locally
- Challenger submits hashes as part of challenge proof
- On-chain verifier re-fetches on-chain blob refs at referenceBlock
- Verifier requires submitted hash matches re-derived hash
★ Insight ─────────────────────────────────────
This is the scaffolding that makes digest-reconstruction challenges actually work. Without on-chain re-derivation at referenceBlock,
the digest is just a self-attestation by the operator — anyone can forge a self-attestation. The on-chain verifier comparing
reconstructed-from-DA hashes against reconstructed-from-on-chain-refs is what closes the loop. This is also what lessons.md "Multi-site
commitment" warns about: every verifier site must agree on the same bytes-level hash, derived independently from its own authoritative
source.
─────────────────────────────────────────────────
Scenario C — Multi-chain template drift
Setup: Bob registered policy template 0xT on Sepolia. Wants to deploy an instance on Base Sepolia, where template is NOT registered.
┌─────┬───────────────────────────────────────────────────────────────────────────────────────────────────┬─────────┬──────────────┐
│ # │ Action │ Locked │ Verdict │
│ │ │ Q │ │
├─────┼───────────────────────────────────────────────────────────────────────────────────────────────────┼─────────┼──────────────┤
│ 1 │ Bob runs policy instance deploy --template 0xT --client 0xBob --chain base-sepolia ... │ Q8.8 │ OK │
├─────┼───────────────────────────────────────────────────────────────────────────────────────────────────┼─────────┼──────────────┤
│ 2 │ CLI connects to base-sepolia RPC. Reads PolicyRegistry.templates(0xT) — returns zero/empty struct │ Q8.8 │ GAP G3 │
│ │ (template not registered on this chain). │ │ │
├─────┼───────────────────────────────────────────────────────────────────────────────────────────────────┼─────────┼──────────────┤
│ 3 │ CLI must fail-fast OR auto-propagate. Decision pending. │ — │ DECISION │
│ │ │ │ NEEDED │
└─────┴───────────────────────────────────────────────────────────────────────────────────────────────────┴─────────┴──────────────┘
Gap G3 (steps 2-3): We locked "no automatic cross-chain propagation in Phase 1," but didn't decide whether CLI does client-side
pre-flight checks before submitting on-chain deployInstance tx. Two options:
- (a) Pre-flight check in CLI: Before submitting deployInstance, CLI reads PolicyRegistry.templates(templateId) on the target chain. If
empty, fail-fast with helpful error. No tx submitted. - (b) Let on-chain revert handle it: Submit deployInstance blind; on-chain code reverts; CLI surfaces typed error.
Recommend (a) — dev's mental model is "the CLI tells me what's wrong before I spend money." Saves a wasted tx and gives faster
feedback.
★ Insight ─────────────────────────────────────
This question generalizes beyond just template-missing-on-chain. The CLI also needs to pre-flight: (i) is expireAfter > 0?, (ii) does
params validate against schema?, (iii) is signer authorized in permissioned mode?, (iv) is computed instanceId already deployed
(idempotent no-op)?, etc. All of these are "things the on-chain contract will check anyway" — but doing them client-side is the
difference between a 5-second feedback loop and a 30-second tx-mining-then-revert loop. Worth locking as a general CLI principle:
"every deploy command pre-flights every on-chain assertion before submitting." This becomes a meaningful spec section: "Pre-flight
assertions per command."
─────────────────────────────────────────────────
Scenario D — Operator-set churn between attestation and challenge
Setup: Task at block N. Operator set at block N: {Op1, Op2, Op3}. Op1 returns malicious response. Block N+50: Op1 deregisters from
OperatorRegistry. Block N+80: Challenger detects fraud and tries to file challenge.
┌─────┬──────────────────────────────────────────────────────────────────────────────┬────────────────────────┬────────────────────┐
│ # │ Action │ Locked Q │ Verdict │
├─────┼──────────────────────────────────────────────────────────────────────────────┼────────────────────────┼────────────────────┤
│ 1 │ Task created at block N. BLS aggregator builds signature with operator set │ Existing Newton │ OK │
│ │ as of block N. Attestation lands on-chain. │ (taskCreatedBlock) │ │
├─────┼──────────────────────────────────────────────────────────────────────────────┼────────────────────────┼────────────────────┤
│ 2 │ Block N+50: Op1 deregisters. OperatorRegistry updated. │ EigenLayer middleware │ OK │
├─────┼──────────────────────────────────────────────────────────────────────────────┼────────────────────────┼────────────────────┤
│ 3 │ Block N+80: Challenger files challenge. Calls ChallengeVerifier. │ Q7 │ OK │
├─────┼──────────────────────────────────────────────────────────────────────────────┼────────────────────────┼────────────────────┤
│ │ ChallengeVerifier re-reads template/instance state at referenceBlock = N │ │ │
│ 4 │ (per Q7.4: content-addressed immutability). For │ Q7.4 │ OK │
│ │ PolicyRegistry/PolicyDataRegistry, registeredAt <= N still holds, so │ │ │
│ │ template/instance state is intact. │ │ │
├─────┼──────────────────────────────────────────────────────────────────────────────┼────────────────────────┼────────────────────┤
│ │ ChallengeVerifier slashes Op1. EigenLayer middleware checks "is Op1 │ Existing Newton + │ │
│ 5 │ currently slashable?" — they deregistered at N+50, but stake may still be in │ EigenLayer slashing │ OK │
│ │ 7-day withdrawal-delay window. │ │ │
├─────┼──────────────────────────────────────────────────────────────────────────────┼────────────────────────┼────────────────────┤
│ │ If Op1 has fully withdrawn (past 7-day delay), slashing fails with │ │ Known Newton │
│ 6 │ OperatorNotSlashable (selector 0xebbff497). │ error-selectors.md │ limitation, not │
│ │ │ │ Sub-B specific │
└─────┴──────────────────────────────────────────────────────────────────────────────┴────────────────────────┴────────────────────┘
No new gap. This scenario stresses operator-set-churn risk that already exists in Newton today; Sub-project B inherits the same
protection without making it worse. The scenario does confirm that content-addressed immutability (Q3, Q7.4) is doing real work — we
don't worry about template state changing between block N and challenge block N+80.
★ Insight ─────────────────────────────────────
Content-addressed immutability is what makes historical reads at referenceBlock cheap — we don't need to snapshot template state at
block N because it's literally identical to template state at any later block (same policyTemplateId always maps to same content). This
is a property the spec should call out explicitly: "All PolicyRegistry and PolicyDataRegistry rows are write-once-with-identity. Once
templates[0xT] is populated, subsequent calls revert as no-op. Therefore reading templates[0xT] at any block ≥ registeredAt returns
identical data." This eliminates an entire class of historical-read bugs that Newton's cache layer has hit before (per lessons.md
"Operator registry cache silently serves wrong-block data").
─────────────────────────────────────────────────
Consistency dry-run summary
Walked 4 scenarios. Surfaced 3 gaps that need explicit decisions before spec drafting:
┌─────┬──────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Gap │ Location │ Recommendation │
├─────┼──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ G1 │ Scenario A │ CLI uses gateway endpoint gateway_getSecretsRecipients() to discover operator-set + secrets pubkeys for │
│ │ step 7 │ HPKE fan-out │
├─────┼──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ G2 │ Scenario B │ On-chain ChallengeVerifier MUST re-derive artifactHashes from on-chain blob refs at referenceBlock and │
│ │ step 3 │ require match with submitted-by-challenger hashes — codify as 4-step verifier protocol in spec │
├─────┼──────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ G3 │ Scenario C │ CLI pre-flights ALL on-chain assertions before submitting deploy txs (template-exists, expireAfter>0, │
│ │ step 3 │ params-validates, signer-authorized, instance-not-already-deployed) │
└─────┴──────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Confirmed strong: Q7 challenge mechanism (digest reconstruction + on-chain re-derivation), Q3 template/instance content-addressing, Q5
per-chain registry isolation, Q6 cache cascade with verify-on-fill.
Net assessment: Design is consistent. The 3 gaps are filling-in-details, not structural issues. None require revisiting locked
decisions. All resolved in the spec's "Detailed protocol" section without changing contract surface.
Confirm G1, G2, G3 with the recommended directions, and we move to spec drafting (Option 1)? Or want to walk additional scenarios
first?
✻ Baked for 1m 41s
※ recap: We're designing Sub-project B (on-chain policy storage architecture) and just finished a 4-scenario consistency walk that
surfaced 3 gaps with recommended fixes. Next: confirm G1, G2, G3 to proceed to spec drafting.
❯ G1: in general for privacy, there are 2 modes - 1) shared operator HPKE encryption key --> this is what is returned from the operators
2) threshold decryption with PSS sharing. But for policy client secrets for data wasms specifically im not sure, but I think we ONLY
use the shared operator HPKE encryption/decryption because Secrets are needed for data wasm execution that happens on "Prepare" phase
of consensus, even when the threshold mode turns on for the protocol and for other privacy features (identity data, confidential data,
ephemeral privacy) we (will) use threshold decryption. First help me confirm my thinking here, and help me "re-evaluate" this design
of having to using the "shared operator HPKE encryption key" for policy client secrets specifically. For G1 question specifically, it
only returns 1 public key (whether non-threshold mode or threshold mode) to the client to encrypt against before uploading the
encrypted data. G2: confirm, agreed. G3: yes pre-flights
⏺ Before I re-evaluate, let me verify your thinking against the actual code — there's a meaningful difference between "fan-out N
envelopes per operator" and "single envelope to one shared key" that the spec needs to get right.
I have the full picture. Let me give you the codebase reality check, then re-evaluate the design.
G1 codebase reality check
★ Insight ─────────────────────────────────────
- HPKE key derivation is per-operator, not shared. derive_hpke_keypair_from_ecdsa (crates/core/src/crypto/envelope.rs:103) chains HKDF
off each operator's own ECDSA secret. Two operators with different ECDSA keys → two different HPKE keypairs. - Gateway returns 1 pubkey, but it's broadcast_first_success — whichever operator answers first wins. So "1 pubkey returned" is true at
the wire, but it's one operator's individual key, not a group-shared key. - Comment vs reality mismatch: the gateway's doc-comment says "WASM secrets must be encrypted per-operator so operators can decrypt
independently" (line 1669) — implying fan-out — but the CLI seals exactly one envelope to one pubkey
(bin/newton-cli/src/commands/secrets.rs:131). No fan-out happens. So today's flow only works end-to-end if the operator that returned
its pubkey is also the operator that decrypts at Prepare.
─────────────────────────────────────────────────
Confirming your mental model — partially
┌───────────────────────────────────┬───────────────────────────────────────────────────────────────┬─────────────────────────────┐
│ Your assertion │ Codebase says │ Verdict │
├───────────────────────────────────┼───────────────────────────────────────────────────────────────┼─────────────────────────────┤
│ Privacy data has 2 modes (shared │ Yes — get_privacy_public_key returns threshold MPK if DKG │ ✅ confirmed │
│ HPKE / threshold PSS) │ active, else operator HPKE │ │
├───────────────────────────────────┼───────────────────────────────────────────────────────────────┼─────────────────────────────┤
│ Secrets must use non-threshold │ lessons.md "Encryption domain must match the phase that │ ✅ confirmed │
│ (Prepare-phase constraint) │ decrypts" + the gateway comment at line 1670 │ │
├───────────────────────────────────┼───────────────────────────────────────────────────────────────┼─────────────────────────────┤
│ Secrets endpoint returns "1 │ Wire-level: yes — single pubkey JSON. Semantically: it's one │
│ public key" │ operator's individual key │ "shared" semantics │
├───────────────────────────────────┼───────────────────────────────────────────────────────────────┼─────────────────────────────┤
│ The single key is "shared across │ No. Each operator derives its own HPKE keypair from its own │ ❌ contradicted by code │
│ operators" │ ECDSA secret. No bootstrap or shared derivation. │ │
└───────────────────────────────────┴───────────────────────────────────────────────────────────────┴─────────────────────────────┘
What this means for the current implementation
Today's flow has a hidden assumption: one operator decrypts all secrets for a given (policy_client, policy_data) pair. With N>1
operators, the gateway's broadcast_first_success is non-deterministic, and only the operator whose pubkey was returned can decrypt.
Other operators in the quorum cannot independently re-evaluate that WASM during Prepare. This works in devnet (1 operator) and likely
works today because:
- newt_storeEncryptedSecrets validates schema via the same operator the gateway routed to (secrets.rs:71), so validation succeeds.
- At eval time, if the same operator is contacted, decryption succeeds.
- If a different operator is contacted, decryption fails silently — and quorum likely degrades or the whole task fails.
This is the architectural debt your "re-evaluate" question is poking at.
Re-evaluation: four options for Sub-project B secrets encryption
Option A — Status quo: per-operator individual key, single-envelope storage
CLI ─[get pubkey]→ gateway ─[broadcast_first_success]→ operator_X.hpke_pk
CLI seals 1 envelope to operator_X.hpke_pk
Store 1 envelope in Postgres + EigenDA
At eval: any operator decrypts → only operator_X succeeds
┌────────────────────────────┬────────────────────────────────────────────────────────────┐
│ Aspect │ Verdict │
├────────────────────────────┼────────────────────────────────────────────────────────────┤
│ Multi-operator correctness │ ❌ Broken — non-X operators cannot decrypt │
├────────────────────────────┼────────────────────────────────────────────────────────────┤
│ DA storage cost │ 1× envelope │
├────────────────────────────┼────────────────────────────────────────────────────────────┤
│ Phase-1 viability │ OK only if 1 operator per policy or single-operator devnet │
├────────────────────────────┼────────────────────────────────────────────────────────────┤
│ TEE dependency │ None │
└────────────────────────────┴────────────────────────────────────────────────────────────┘
Option B — Per-operator fan-out: N pubkeys, N envelopes
CLI ─[get pubkeys]→ gateway returns [op_1.pk, op_2.pk, ..., op_N.pk]
CLI seals N envelopes (one per operator)
Store N envelopes in Postgres + EigenDA, indexed by operator_id
At eval: each operator decrypts its own envelope independently
┌────────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────┐
│ Aspect │ Verdict │
├────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Multi-operator correctness │ ✅ Each operator has its own envelope │
├────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ DA storage cost │ N× envelope (linear blowup) │
├────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Operator-set churn │ ❌ Adding new operator → must re-fan-out by client (no key for new op until re-upload) │
├────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Schema validation │ ✅ Any operator validates its own envelope │
├────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ TEE dependency │ None │
├────────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────┤
│ Existing code shape │ Matches doc-comments saying "per-operator" │
└────────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────┘
Option C — Shared HPKE keypair via TEE-attested derivation (your conceptual target)
All operators run Nitro Enclave with same PCR0
Enclave derives HPKE keypair from (PCR0 ‖ epoch_id ‖ domain_separator)
All operators share the SAME HPKE pubkey (deterministic from PCR0)
CLI ─[get pubkey]→ gateway returns the shared pubkey
CLI seals 1 envelope, stores 1 envelope
Any operator decrypts (private key recoverable inside attested enclave)
┌──────────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Aspect │ Verdict │
├──────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Multi-operator │ ✅ All operators have the same key │
│ correctness │ │
├──────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ DA storage cost │ 1× envelope │
├──────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Operator-set churn │ ✅ New operator with same PCR0 inherits the key automatically │
├──────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ TEE dependency │ ❌ Requires Nitro Enclaves bootstrapped + PCR0 whitelist enforced; today operators run without TEE in │
│ │ most environments │
├──────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Epoch rotation │ ✅ Bump epoch_id → new key → re-encrypt cycle │
├──────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Phase-1 viability │ Blocked until TEE rollout (Phase 1b in privacy roadmap) │
├──────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Match to user's framing │ ✅ "1 pubkey, shared, threshold-or-not" matches │
└──────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Option D — Threshold for secrets too (conceptual)
Use DKG MPK for secrets, just like privacy data
At eval: threshold partial DH combine to decrypt
┌───────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Aspect │ Verdict │
├───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Phase ordering │ ❌ Secrets needed in Prepare; partial DHs from peers not collected until Commit. Encrypting to MPK deadlocks │
│ │ the consensus pipeline. │
├───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ lessons.md │ Explicitly forbidden by "Encryption domain must match the phase that decrypts" │
│ guidance │ │
├───────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Verdict │ Hard no — this is the design bug newt_getSecretsPublicKey was created to prevent │
└───────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Recommendation for spec
Phase 1 (with Sub-project B): Option C with an Option-A escape hatch.
┌─────────────────────┬──────────────────────────────────────┬────────────────────────────────────────────────────────────────────┐
│ Phase │ Encryption model │ Rationale │
├─────────────────────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ Phase 1 MVP (no │ Option A — accept │ TEE rollout is downstream; Sub-project B should not block on it │
│ TEE) │ single-operator-decrypts limitation │ │
├─────────────────────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ Phase 1b (Nitro │ Option C — shared HPKE from PCR0 + │ Once enclave version registry is enforced (already exists at │
│ Enclaves attested) │ epoch │ EnclaveVersionRegistry), this is the natural derivation point │
├─────────────────────┼──────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ Never │ Option D — threshold for secrets │ Architectural contradiction with Prepare/Commit phasing │
└─────────────────────┴──────────────────────────────────────┴────────────────────────────────────────────────────────────────────┘
Why not Option B (fan-out): the operator-set-churn problem is severe. Every time an operator joins or leaves, every secrets-holding
client must re-upload N envelopes. This couples client UX to the AVS roster and bloats EigenDA storage linearly. Option C cleanly
avoids this by anchoring the key to the attested code identity (PCR0), not the operator-set membership.
Proposed G1 lock for the spec
▎ G1 (LOCKED — Option C target, Option A interim):
▎
▎ - Encryption surface: newt_getSecretsPublicKey returns exactly 1 X25519 HPKE pubkey to the client. Wire shape identical to
▎ newt_getPrivacyPublicKey (same JSON, same suite string).
▎ - Key derivation:
▎ - Phase 1a (no TEE): operator-individual HPKE pubkey via broadcast_first_success. Documented limitation: only the responding
▎ operator can decrypt; clients should treat this as devnet/single-operator-per-policy mode. Status-quo behavior; no code change in
▎ Sub-project B.
▎ - Phase 1b (TEE active): shared HPKE pubkey deterministically derived inside the Nitro Enclave from (PCR0 ‖ epoch_id ‖
▎ "newton-secrets-shared-v1"). Every operator with the whitelisted PCR0 derives the same keypair. Gateway returns this shared pubkey
▎ from any operator. Anchored to EnclaveVersionRegistry for governance.
▎ - Decryption: Always Prepare-phase, always non-threshold. Never routed through DKG/PSS — this is enforced by separate RPC endpoints
▎ (newt_getSecretsPublicKey vs newt_getPrivacyPublicKey).
▎ - Domain separation: secrets HPKE derivation salt is distinct from any other HPKE/Ed25519 derivation (per lessons.md "HKDF derivation
▎ chains must use domain-separated salts").
▎ - Spec note: Sub-project B does not change Phase 1a behavior; it only documents the migration target and ensures the secretsSchema
▎ (state-tree namespace 0x08) is referenced by the same envelope shape under both phases.
Where this leaves the dry-run
┌────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────┐
│ Gap │ State │
├────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ G1 — secrets encryption model │ ✅ LOCKED (Option C target, Option A interim, see above) │
├────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ G2 — verifier re-derives artifactHashes from on-chain refs │ ✅ confirmed │
├────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────┤
│ G3 — CLI pre-flights all on-chain assertions before deploy │ ✅ confirmed │
└────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────┘
All three gaps closed. Ready to pivot to spec drafting at docs/superpowers/specs/2026-04-22-onchain-policy-storage-design.md.
Want me to start the draft now? The structure I have in mind: §1 motivation, §2 scope (Sub-A/B/C delineation), §3 identity formulas +
namespaces, §4 registries + storage, §5 deployment flow + CLI, §6 eval/verification surface, §7 governance, §8 migration, §9 G1/G2/G3
invariants, §10 open questions for Phase 2.
✻ Worked for 2m 42s
※ recap: Designing Sub-project B (on-chain policy storage) spec; just locked G1 secrets encryption as Option C target with Option A
interim, closing the consistency dry-run. Next: draft the spec at docs/superpowers/specs/2026-04-22-onchain-policy-storage-design.md.
❯ Are you sure? validate your analysis above by looking into the code @integration-tests/src/multichain_routing.rs:253~261
⏺ Reading the test file at the specified lines now.
Read 1 file (ctrl+o to expand)
⎿ Loaded integration-tests/src/CLAUDE.md
⏺ You're right — I missed critical evidence. Let me verify what I got wrong before correcting the analysis.
Searched for 2 patterns (ctrl+o to expand)
⏺ Reading the production loader path now.
Searched for 1 pattern, read 2 files (ctrl+o to expand)
⏺ You're correct — I had the production model backwards. Apologies for the bad analysis. Let me show what I actually missed and revise.
The correction
★ Insight ─────────────────────────────────────
- The HPKE key is NOT derived from each operator's ECDSA secret in production. It's loaded from a separate config field
service.encryption.private_key (crates/operator/src/config.rs:252-255), set via env OPERATOR__ENCRYPTION__PRIVATE_KEY, and the same
value is provisioned to every operator out-of-band. - derive_hpke_keypair_from_raw is the production path (builder.rs:488); derive_hpke_keypair_from_ecdsa is only used in test modules
(core.rs:1634, 1663, 1687 — all behind #[cfg(test)]). The two functions share machinery but the "from_ecdsa" name is
historical/test-only — production starts from a generic 32-byte secret, not the signer's ECDSA key. - The test comment at multichain_routing.rs:253-260 is the smoking gun: "both operators must load the same key for the envelope to
decrypt" — this is enforced by the test harness setting OPERATOR_HPKE_KEY to a single hardcoded value for every operator. - PR #524 moved the HPKE key from gateway → operators but preserved the "all parties hold the same key" property. It changed who owns
it, not how it's shared.
─────────────────────────────────────────────────
What the codebase actually does (corrected)
Out-of-band: operator deployer sets OPERATOR__ENCRYPTION__PRIVATE_KEY = X (32 bytes)
every operator receives the same X
Startup: operator.builder.rs:483 reads X
derive_hpke_keypair_from_raw(X) → (sk_shared, pk_shared)
all operators end up with sk_shared / pk_shared (identical)
get_public: gateway broadcast_first_success → any operator returns pk_shared
wire shape is "1 pubkey", semantically it IS a shared key
Encrypt: CLI seals 1 envelope to pk_shared
Storage: 1 envelope in Postgres + EigenDA
Decrypt: ANY operator can decrypt with sk_shared during Prepare
This is exactly your mental model. Your framing — "ONLY shared HPKE for secrets, only 1 pubkey returned" — describes production
accurately. My earlier "operators have different keys, fan-out is needed" claim was wrong; I conflated the test-only ECDSA-derivation
helper with the production path.
What I had wrong, line by line
┌──────────────────────────────────────────────┬───────────────────────────────────────────────────────────────────────────────────┐
│ My earlier claim │ Reality │
├──────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤
│ "Each operator derives its own HPKE keypair │ False — operators load a shared 32-byte secret from config, derivation is │
│ from its own ECDSA secret" │ identical │
├──────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤
│ "Two operators → two different HPKE │ False — by construction all operators share the same keypair │
│ keypairs" │ │
├──────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤
│ "Option B (per-operator fan-out, N │ False — there is no fan-out today; the architecture has always been │
│ envelopes) is the natural multi-operator │ single-envelope-shared-key │
│ path" │ │
├──────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤
│ "Option A breaks in multi-operator quorum" │ False — the deployment contract is "all operators load same │
│ │ encryption.private_key," and the multichain E2E proves it works │
├──────────────────────────────────────────────┼───────────────────────────────────────────────────────────────────────────────────┤
│ "Option C (TEE-derived shared key) is an │ Partially right but mis-framed — it's a distribution-mechanism evolution, not an │
│ architectural shift" │ encryption-model shift. The encryption model is already shared-key. │
└──────────────────────────────────────────────┴───────────────────────────────────────────────────────────────────────────────────┘
Re-evaluation, corrected
Three options for the spec
┌────────┬──────────────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────┐
│ Option │ Distribution of shared HPKE secret │ Status │
├────────┼──────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ A │ Out-of-band env var OPERATOR__ENCRYPTION__PRIVATE_KEY, manually provisioned │ Current production │
│ │ to every operator │ │
├────────┼──────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ B │ Bootstrap-protocol distribution (operator joins → gateway/bootnode hands key │ Reduces ops burden, still trust-rooted │
│ │ encrypted to operator's ECDSA pubkey) │ in gateway/bootnode │
├────────┼──────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ C │ TEE-attested derivation: sk_shared = HKDF(PCR0 ‖ epoch_id ‖ │ Removes out-of-band trust, ties key │
│ │ "newton-secrets-shared-v1") inside Nitro Enclave with whitelisted PCR0 │ identity to attested code identity │
└────────┴──────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────┘
The encryption model is the same in all three. Only the who-holds-what-and-how-they-got-it changes.
Key trade-offs
┌──────────────────────┬───────────────────────────────────────┬───────────────────────────┬──────────────────────────────────────┐
│ Aspect │ Option A (today) │ Option B (bootstrap) │ Option C (TEE-derived) │
├──────────────────────┼───────────────────────────────────────┼───────────────────────────┼──────────────────────────────────────┤
│ Operator-set churn │ ❌ Manual re-provision per join │ ✅ Auto on join │ ✅ Auto via PCR0 match │
├──────────────────────┼───────────────────────────────────────┼───────────────────────────┼──────────────────────────────────────┤
│ Trust root │ Whoever has env-var access │ Gateway/bootnode operator │ Attested enclave code │
├──────────────────────┼───────────────────────────────────────┼───────────────────────────┼──────────────────────────────────────┤
│ Key rotation │ Coordinated env-var swap + restart │ Coordinated bootstrap │ Bump epoch_id, re-derive │
│ │ │ re-run │ │
├──────────────────────┼───────────────────────────────────────┼───────────────────────────┼──────────────────────────────────────┤
│ Compromise blast │ Compromise of any operator → full │ Same as A │ Compromise requires breaking enclave │
│ radius │ secret leak │ │ │
├──────────────────────┼───────────────────────────────────────┼───────────────────────────┼──────────────────────────────────────┤
│ Phase-1 viability │ ✅ Already shipping │ Possible but new code │ Blocked on Nitro Enclaves rollout │
│ │ │ │ (Phase 1b) │
├──────────────────────┼───────────────────────────────────────┼───────────────────────────┼──────────────────────────────────────┤
│ Sub-project B impact │ None (Sub-B doesn't touch this) │ None (orthogonal) │ None (orthogonal) │
└──────────────────────┴───────────────────────────────────────┴───────────────────────────┴──────────────────────────────────────┘
Implication for Sub-project B spec
G1 is largely a clarification, not a design change. Sub-project B is about policy bytes + schemas on-chain; the secrets encryption
model is a Sub-A / privacy-layer concern that already works. The spec should:
- Document that secretsSchema (state-tree namespace 0x08) is referenced by policy_data_template and validated at upload time by the
operator that decrypts the envelope. - Note that the encryption model is shared-HPKE-key (production: out-of-band distribution, future: TEE-derived) and that Sub-B does
not modify it. - Not include any operator-key fan-out logic, because there isn't any.
- Defer the Option A → C migration to its own design (privacy-layer roadmap, not policy storage).
Revised G1 lock
▎ G1 (LOCKED — clarification only, no Sub-B change):
▎
▎ - Encryption surface for policyDataInstance secrets: newt_getSecretsPublicKey returns exactly 1 X25519 HPKE pubkey, identical across
▎ all operators by construction. The shared HPKE secret is provisioned out-of-band today via OPERATOR__ENCRYPTION__PRIVATE_KEY; this is
▎ unchanged by Sub-project B.
▎ - Domain separation: secrets HPKE key is distinct from the threshold MPK (newt_getPrivacyPublicKey). Clients use the secrets endpoint
▎ for newt_storeEncryptedSecrets, the privacy endpoint for everything else (per lessons.md "Encryption domain must match the phase
▎ that decrypts").
▎ - Schema validation: at newt_storeEncryptedSecrets upload time, gateway delegates HPKE decrypt + JSON-schema validation to an
▎ operator (gateway/src/rpc/api/secrets.rs:65-89). The operator fetches secretsSchema from state-tree namespace 0x08 (Sub-B addition)
▎ and validates plaintext shape. Any operator can do this because the key is shared.
▎ - Storage: 1 encrypted envelope per (chain_id, policy_client, policyDataInstance) tuple in Postgres + EigenDA. No fan-out.
▎ - Out of scope for Sub-B: the migration from out-of-band distribution → TEE-derived shared key (Option A → C). That's a privacy-layer
▎ evolution to track separately.
Where this leaves the dry-run (corrected)
┌───────────────────────────────────────────────────────────────────┬──────────────────────────────────────────────────────────────┐
│ Gap │ State │
├───────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ G1 — secrets encryption model │ ✅ LOCKED — shared HPKE key, 1 envelope, no Sub-B change. │
│ │ Just document. │
├───────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ G2 — verifier re-derives artifactHashes from on-chain refs at │ ✅ confirmed │
│ referenceBlock │ │
├───────────────────────────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ G3 — CLI pre-flights all on-chain assertions before deploy │ ✅ confirmed │
└───────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────┘
All three closed. Sub-B spec gets simpler — no new key-mgmt mechanics, just a schema namespace addition and an honest pointer at the
existing privacy-layer model.
Want me to draft the spec at docs/superpowers/specs/2026-04-22-onchain-policy-storage-design.md now? I can also append a one-paragraph
"future evolution" note flagging the Option A → C migration as a separate workstream.
✻ Crunched for 2m 3s
Spec is intact at 510 lines / 31.4K. Now doing the self-review pass for placeholders, consistency, scope, and ambiguity.
★ Insight ─────────────────────────────────────
- A spec self-review is fundamentally different from a content review — it asks "does the document say enough to implement" rather than
"is the design right". The four review axes (placeholders, consistency, scope, ambiguity) map to four classes of post-acceptance bugs:
TODO drift, contradiction-driven rework, scope creep, and "we thought we agreed" reinterpretation. - The spec uses a deliberate two-tier identity model where templates are content-addressed but instances are chain-bound. This means
policyTemplateId is reproducible from local files alone — a property the CLI exploits via compute-id for offline scripting and the
contract exploits via deterministic dedup. Catching where the spec under-specifies this dual semantics is a high-value review target. - Reference-section accuracy matters more than people credit: when citations drift (off-by-one line numbers, ranges that span a whole
file), readers lose the ability to verify the spec's claims against the code. Each citation in this spec is a load-bearing claim about
the existing system.
─────────────────────────────────────────────────
Self-review findings
MUST FIX — substantive gaps
A. currentInstanceIds shape for PolicyDataRegistry is undefined (lines 178, 407)
Line 178 says "clients can have multiple policy-data instances per policy" and uses currentInstanceIds (plural). The Solidity
pseudocode at lines 167-176 shows only PolicyRegistry with clientCurrent: mapping(address => bytes32). The PD parallel layout is not
shown. Implementer ambiguity: is it mapping(address => mapping(bytes32 policyId => bytes32[])), or mapping(address => bytes32[]), or
keyed by something else?
B. expireAfter field is referenced but never declared (line 393)
▎ "registry rejects expireAfter == 0 at deployInstance time"
But PolicyInstance struct shape isn't shown anywhere. The implementer cannot tell from the spec whether expireAfter lives on the
instance, on the template, or in params.
C. policyDataTemplateRefs storage is implied but not declared (line 82, 90)
The formula canonical(policyDataTemplateRefs) is part of policyTemplateId. For policy describe to walk the DAG (line 228-231), the
template must store these references on-chain explicitly — but PolicyTemplate struct shape isn't shown.
D. Contract behavior on duplicate registerTemplate is unspecified (lines 190 vs 271)
- Line 271 says the CLI short-circuits before submitting on duplicate
- Line 190 says "the second and third calls succeed even if a different deployer ran the first because the deterministic ID
short-circuits to a no-op"
But the contract behavior is undefined — does registerTemplate revert on duplicate, no-op, or update? Critical for retry semantics if a
CLI pre-flight is bypassed (admin tool, direct contract interaction).
SHOULD FIX — clarity
E. Hashing parenthetical is confusing (line 89)
▎ "arguments are encoded with abi.encodePacked for hashing only (commitments stored as raw bytes, never abi.encode'd at registry-write
▎ time)"
The "stored as raw bytes" phrase is ambiguous — is it talking about the registry storing bytes32 results, or storing the input
rego_bytes? Re-read suggests it means the on-chain commitment is the 32-byte hash, not abi-encoded. Worth re-stating without the
parenthetical.
F. Namespace tag terminology drifts (lines 109 vs 441)
- Line 109: "Leaf-key prefix bytes are namespace tags (per Sub-A's LeafHeaderV1)"
- Line 441: "header.namespace distinguishes 0x04–0x08"
Sub-A uses leaf_key[0] as the 1-byte namespace prefix. Pick one term and use it consistently — LeafHeaderV1.namespace field vs
leaf_key[0] prefix are not the same shape.
G. show-secrets metadata semantics undefined (line 226)
▎ "show-secrets # show metadata only"
What metadata? Upload timestamp? Schema reference? Envelope size? Encryption epoch? The implementer needs the field list.
H. CID-vs-KZG integrity abstraction unclear (line 396-398)
The spec says "Sub-B replaces IPFS CID integrity verification with EigenDA blob KZG-commitment verification" — but SHA-256 multihash
and KZG pairing are categorically different checks. Is the operator hot path wrapped behind a trait that abstracts both? Or are they
two distinct code paths gated by the feature flag?
I. Off-by-one in code citation (line 504)
▎ "integration-tests/src/multichain_routing.rs:253-260"
Actual range was 253-261. Same kind of issue may be in the secrets.rs:1-120 citation (line 503) — that's a whole-file range, not a
function range.
J. secretsSchema validation locality (lines 449, 462)
Line 449 says validation "runs inside any operator the gateway delegates to" — does "any" mean exactly one, or all decrypters? Given
the shared key model, one is sufficient cryptographically; redundancy is a defense-in-depth choice that should be explicit.
K. Client owner role description misses plural case (line 407)
▎ "Point their client's currentInstanceId to a deployed instance they own"
For PD, it's currentInstanceIds. The role description should accommodate both registries.
NICE TO HAVE — style
L. L4 hit rate qualitative outlier (line 331)
L0–L3 have target percentages or "Tail"; L4 has "Tail emergency" with no target. Would be more uniform to say "fallback only — no SLA"
or pick a numeric target.
M. Single-snapshot mentions only 5 artifacts (line 389)
"rego bytes, WASM bytes, and all three schemas" — but PD instance params are also read at eval time. Implicit but worth listing for
completeness.
N. Multi-chain example forces an order despite "any-order" Q8 lock (lines 297-303)
The example registers PD templates first, then policy templates referencing them. Q8.3 says any-order works thanks to pre-flight. A
one-line note ("Order is illustrative; pre-flight permits any order") would prevent reader confusion.
O. "Off-chain documentation" hand-wave (line 419-420)
▎ "Off-chain documentation can mark templates as discouraged"
What channel? Newton docs site? GitHub README? A specific runbook? Either point to the channel or remove the line.
ALL CLEAR
- No TODO / TBD / XXX markers
- No patterns
- Identity formulas are consistent across §Architecture and §Verifier re-derivation
- Namespace map (0x04–0x08) is consistent across §State-tree namespaces, §Three schemas, §Cross-project integration
- Q-decision table matches the body sections it summarizes
- Scope (in/out/non-goals) is unambiguous
- Risks table covers the major failure surfaces
Recommendation
Substantive fixes A–D should land before user review. The SHOULD-FIX items E–K are individually small but together drive ~15 lines of
edits. NICE-TO-HAVE items L–O are polish.
Want me to apply A–D + E–K now, or stage them for your review pass first?
✻ Worked for 2m 27s
※ recap: Drafting the Sub-project B on-chain policy storage spec; finished writing it to
docs/superpowers/specs/2026-04-22-onchain-policy-storage-design.md and ran a self-review surfacing four substantive gaps. Next action
is your call on whether to apply the fixes now or after your review.
❯ yes for A: currentInstanceIds refers to the "deployed policy data instances" not templates right? conceptually, policy ref (template)
to policy Data refs (template) are mapped, and each poilcy client (address) has policy instance id (policy registry) + policy data
instance ids (policy data registry)? confirm my understanding and correct me if im wrong. B. i need to confirm, but think expire After
is set at the policy/policy data "instance" level? So first, yes, let's apply fixes for A-D + E-K, and for the rest, let's review /
discuss as well
⏺ Confirming your mental model first, then applying fixes.
Your mental model — correct, with one refinement
▎ "currentInstanceIds refers to the 'deployed policy data instances' not templates"
✅ Correct. currentInstanceIds (plural) tracks policy-data instances the client has deployed, not templates.
▎ "conceptually, policy ref (template) to policy Data refs (template) are mapped, and each policy client (address) has policy instance
▎ id (policy registry) + policy data instance ids (policy data registry)"
✅ Correct. The relationship is:
policyTemplate ──declares──→ policyDataTemplateRefs = [pd1, pd2, pd3] (1-to-N at template level)
policyInstance ──references──→ policyTemplate (client deploys 1 policy instance)
policyDataInstance ──references──→ policyDataTemplate (client deploys 1 PDI per referenced PD template)
One refinement on the keying: the per-client PD instance map should be keyed by (client, policyDataTemplateId), not a flat bytes32[]
array. So:
// PolicyRegistry — one policy instance per client
mapping(address client => bytes32 instanceId) public currentPolicyInstance;
// PolicyDataRegistry — one PD instance per (client, PD template) slot
mapping(address client => mapping(bytes32 policyDataTemplateId => bytes32 instanceId))
public currentPolicyDataInstance;
Why mapping not array: the operator's eval loop walks the policy template's policyDataTemplateRefs and asks the registry "for client C,
which PD instance fills slot pd1?". A keyed mapping gives O(1) per slot and decouples client-side instance ordering from policy-author
template-ref ordering. An array would force ordering coupling and make swapping a single PD instance harder.
▎ "expireAfter is set at the policy/policy data 'instance' level"
✅ Correct. It's per-instance because two clients deploying against the same template can choose different validity windows. This
matches today's NewtonPolicy.PolicyConfig.expireAfter.
★ Insight ─────────────────────────────────────
- The template-vs-instance keying decision matters for upgrade ergonomics: mapping-keyed-by-templateId means a client can swap their
pd1 instance independently of pd2, which lets clients hot-swap a single data domain (e.g., "use a fresh sanctions list") without
redeploying their policy. Array-keyed would force re-pointing the entire policy any time one PD changes. - Per-instance expireAfter means challengers must read the instance's value (not the template's) at the dispute block. This bakes into
the verifier re-derivation flow at G2: the verifier must read both the template (for hash commitments) AND the instance (for
expireAfter) from the registry at the task's reference block. - A subtle implication of currentPolicyDataInstance[client][templateId] being a mapping: an operator evaluating client C's policy must
verify that every PD template the policy requires has a corresponding instance pointed by the client. A missing slot is a deploy-time
invariant violation that pre-flight (G3) must catch — otherwise eval-time will see a zero-instance lookup and need to fail loudly.
─────────────────────────────────────────────────
Now applying fixes A–D + E–K.
All substantive fixes applied. Spec is now 587 lines, internally consistent across struct names and pointer references.
★ Insight ─────────────────────────────────────
- The bigger structural change here was making currentPolicyDataInstance a mapping(address => mapping(bytes32 => bytes32)) instead of
an array. This bakes a key design property into the data model: clients deploy PD instances against template IDs, and the operator's
eval loop walks the policy template's policyDataTemplateRefs to look up instances by slot — never by index. That's how schema evolution
stays clean: rotating one PD template doesn't shift array indices for unrelated PD slots. - The duplicate-registration revert vs. silent no-op decision is a small choice with a big consequence for tooling. Reverting forces
every caller (CLI, admin scripts, Foundry deploy) to handle the case explicitly. A silent no-op would let buggy deploy scripts believe
they registered something fresh when they actually re-keyed against an existing template — a source of "phantom deployment" bugs. - expireAfter living on the instance, not the template, also means challenger verification at G3 needs to read the instance state at
the dispute block. The spec's verifier pseudocode already does both reads — but make sure the implementation plan keeps these two reads
atomic across the same RPC call to avoid race conditions on registry upgrades.
─────────────────────────────────────────────────
Summary of fixes applied (A–D + E–K)
┌─────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Fix │ What changed │
├─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ A │ Added full PolicyRegistry and PolicyDataRegistry struct definitions; currentPolicyDataInstance keyed by (client, │
│ │ policyDataTemplateId) mapping │
├─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ B │ expireAfter: uint64 declared on both PolicyInstance and PolicyDataInstance │
├─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ C │ policyDataTemplateRefs: bytes32[] declared on PolicyTemplate (sorted) │
├─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ D │ Added ### Duplicate registration semantics section: TemplateAlreadyRegistered and InstanceAlreadyDeployed reverts; CLI │
│ │ short-circuits via G3 │
├─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ E │ Hashing parenthetical rewritten to clarify raw-vs-hash storage │
├─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ F │ Unified to leaf_key[0] convention; dropped LeafHeaderV1.namespace references │
├─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ G │ show-secrets metadata enumerated: epoch_id, schema ref, ciphertext size, upload timestamp, uploader │
├─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ H │ Added ArtifactVerifier trait abstraction; documented SHA2-256 (IPFS) vs KZG (EigenDA) paths │
├─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ I │ Citation ranges tightened: secrets.rs:65-89, multichain_routing.rs:253-261, added handler/mod.rs:1665-1708 │
├─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ J │ secretsSchema validation runs in single operator (cryptographically sufficient); defense-in-depth multi-validator │
│ │ deferred to Phase 1b │
├─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ K │ Client owner row updated to mention currentPolicyInstance + currentPolicyDataInstance[templateId] + setSecrets │
├─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ (bonus) │ Narrative pointer language at line 94 updated to match new field names │
└─────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Remaining items for discussion (L–O)
These are smaller-stakes items I held back per your instruction. Each is a one-line decision:
L. L4 cache tier hit rate is qualitative, not numeric
Current: L0=95%+, L1=99%+, L2=99.5%+, L3="Tail", L4="Tail emergency".
Options:
- Leave qualitative — operationally honest (we don't have measured L4 rates)
- Specify "fallback only — no SLA" for L3 and L4 — clearer that these are edge cases
- Pick numbers (e.g., L3 99.99%, L4 99.999%) — gives a target to design against
My take: option 2 — the L4 path is genuinely best-effort, and committing to numbers without measurement is the kind of fiction that
breaks in prod review.
M. Single-snapshot section omits PD instance params
Spec says "rego bytes, WASM bytes, and all three schemas read from the same snapshot". Should also list PD instance params (which are
stored in EigenDA blobs but referenced from the registry — also need to read at the snapshot block).
My take: add it — listing all five artifact classes (rego, WASM, 3 schemas, instance params) makes the invariant exhaustive rather than
approximate.
N. Multi-chain example forces an order despite "any-order" Q8 lock
Example registers PD templates first, then policy templates that reference them. Q8 says any-order works thanks to pre-flight.
Options:
- Leave as-is and add a one-line note: "Order is illustrative; pre-flight permits any order"
- Reorder the example to register policy template first to demonstrate the any-order property
- Both — note + reordered
My take: option 1 — the dependency-first order is the default mental model for the reader, and a single sentence is clearer than
reverse-order pedagogy.
O. "Off-chain documentation" hand-wave for deprecation
Spec says: "Off-chain documentation can mark templates as discouraged." No location specified.
Options:
- Drop the line entirely — Phase 1 has no deprecation, full stop
- Point at a real channel (Newton docs site? docs/POLICY_DEPRECATION.md? The runbook?)
- Keep as-is — leaves room for ad-hoc docs
My take: option 1 — the line is currently a hint at a process that doesn't exist yet, which is exactly the kind of phrase that becomes
a future "we said we'd document this somewhere but didn't" issue.
Pick L/M/N/O choices and I'll apply them. Or if you'd rather hold them and review the full spec end-to-end first, I can stage them.
✻ Brewed for 5m 39s
※ recap: Drafted Sub-project B on-chain policy storage spec at docs/superpowers/specs/2026-04-22-onchain-policy-storage-design.md and
just applied substantive fixes A through K from my self-review. Next action is your decision on the four remaining items L through O
before user review.
⏺ ★ Insight ─────────────────────────────────────
Recall search confirmed no prior Sub-B entries — the closest hits are Sub-A "Private Data Storage" memories. Sub-B needs its own memory
file and tagged recall entries; this isn't an update of an existing record. Local memory dir has project_private_data_storage.md for
Sub-A (52.9K, 5 days old), so the natural pattern is a sibling project_onchain_policy_storage.md for Sub-B with a parallel entry in
MEMORY.md.
Session search recovered all Q1-Q10 + G1-G3 context, so nothing was lost to compaction.
─────────────────────────────────────────────────
Preview — what I'll save across all 3 memory systems
Project/Topic: Newton Private Data Storage — Sub-project B (On-chain Policy + Policy Data Migration)
Linear: TBD — no Linear project created yet (Sub-A's Private Storage Layer project covers Sub-A only). Sub-B Linear decomposition is a
follow-up task.
Summary (2-3 sentences)
Completed architectural design and implementation plan for Sub-project B, which migrates policy Rego bytecode + policy-data WASM + 3
schemas from IPFS-only addressing to on-chain commitment + EigenDA bytes (validium pivot). Locked 10 architectural decisions Q1-Q10 plus
3 consistency-dry-run gaps G1-G3 (shared HPKE clarification, ChallengeVerifier re-derivation, CLI pre-flight checks). Final 3,860-line
implementation plan saved with 47 TDD tasks across 6 streams (A:12, B:9, C:9, D:11, E:6, X:4); awaiting execution-mode choice.
Key Decisions
- Validium pivot LOCKED (Q1) — bytes via EigenDA, only commitments on-chain. State-tree namespaces 0x04-0x08 multiplex through
leaf_key[0] prefix in Sub-A's unified JMT. - Two-tier identity formulas (Q2):
- policyTemplateId = keccak256(rego ‖ policyParamsSchema ‖ canonical(policyDataTemplateRefs)) (sorted refs)
- policyDataTemplateId = keccak256(wasm ‖ policyDataParamsSchema ‖ secretsSchema)
- policyInstanceId = keccak256(chainId ‖ clientAddr ‖ policyTemplateId ‖ paramsHash)
- policyDataInstanceId = keccak256(0x01 ‖ chainId ‖ clientAddr ‖ policyDataTemplateId ‖ paramsHash) — domain separator byte - Three schemas (Q3) — policyParamsSchema (0x05), policyDataParamsSchema (0x07), secretsSchema (0x08); JSON Schema Draft 2020-12.
- Template/Instance split (Q4) — Templates immutable + content-addressed (Newton-vetted allowlist); instances mutable + client-owned.
currentPolicyInstance[client] (single) and currentPolicyDataInstance[client][templateId] (per-slot mapping). - Three-layer drift defense (Q5) — L1 deploy-time, L2 eval-time (single-snapshot invariant), L3 challenge-time (re-derivation).
- Five-tier cache (Q6) — L0 enclave RAM, L1 daemon RAM, L2 daemon disk, L3 EigenDA fallback, L4 bootnode fallback.
- CLI surface (Q7) — policy template/instance, policy-data template/instance, set-secrets, describe, with profile config 4-lane
resolver. - Per-client mapping shape (Q8) — single + per-slot mapping (not array).
- Duplicate registration (Q9) — TemplateAlreadyRegistered / InstanceAlreadyDeployed reverts; CLI G3 pre-flight short-circuits.
- Allowlists (Q10) — templateAuthors + instanceDeployers via AllowlistGuard mixin (Safe multisig per chain).
- G1 (gap fix) — Production reuses ONE shared HPKE keypair via OPERATOR__ENCRYPTION__PRIVATE_KEY; Sub-B only adds secretsSchema
namespace 0x08. - G2 (gap fix) — ChallengeVerifier re-reads on-chain policyTemplateRef / policyDataTemplateRef at task's reference block; never trusts
challenger-supplied hash. - G3 (gap fix) — Every CLI deploy command pre-flights all on-chain assertions before broadcasting any tx.
- expireAfter per-instance — confirmed at instance level, > 0 enforced at deployInstance.
- Atomicity — Phase 1 ships any-order (Option B); atomic bundle multicall (Option A) deferred to Phase 2.
Implementation Progress
- Spec: docs/superpowers/specs/2026-04-22-onchain-policy-storage-design.md (586 lines, gitignored, accepted)
- Plan: docs/superpowers/plans/2026-04-22-onchain-policy-storage-plan.md (3,860 lines, 127K, gitignored, complete with self-review)
- 47 TDD tasks across 6 streams:
- A. Foundation (12 tasks) — namespaces 0x04-0x08, artifact-verifier crate (IPFS+EigenDA), policy-id crate (template/instance ID
formulas), schema-validator crate - B. Contracts (9 tasks) — IPolicyRegistry/IPolicyDataRegistry, AllowlistGuard mixin, ChallengeVerifier G2 re-derivation, deployment +
bindings - C. CLI (9 tasks) — profile config, G3 pre-flight, policy template/instance, policy-data template/instance, set-secrets, describe
- D. Service refactors (11 tasks) — chainio writers/readers, LayeredArtifactCache, pre-warm subscriber, WASM compile cache, schema
cache, single-snapshot invariant, gateway secretsSchema validation, challenger G2 binding, data-provider hot path - E. E2E (6 tasks) — feature flag, deploy/eval/challenge/secrets-validation scenarios, 14 quickstart policy migration
- X. Cross-cutting (4 tasks) — metrics, docs, Phase 1.5 IPFS-default-off, Phase 2 IPFS deletion
- A. Foundation (12 tasks) — namespaces 0x04-0x08, artifact-verifier crate (IPFS+EigenDA), policy-id crate (template/instance ID
- Self-review inlined in plan: spec coverage table, placeholder scan, type consistency table, risk callouts (B7 reference-block stashing
deferred, pre-warm subscriber bootstrapping)
Learnings
- Sub-A primitives reuse pattern — Sub-B reuses every Sub-A primitive unchanged (StateCommitRegistry, IStateRootCommittable, JMT state
tree, EigenDaClient, BootnodeServer, NamespaceId enum). Net-new is contracts, CLI, and service refactors only — no new infrastructure
crates. - ArtifactVerifier trait abstracts SHA2-256 (IPFS) vs KZG (EigenDA) — both return keccak256(bytes) as canonical commitment. Decouples
on-chain commitment from off-chain availability layer. - Stream parallelism — Streams A and B are fully independent (no shared symbols), can fan out 2 concurrent subagents in week 1. Stream C
blocks on A; Stream D blocks on A+B; Stream E blocks on D. - expireAfter > 0 enforcement — moved from policy implementation to deployInstance revert (ExpireAfterZero); avoids the historical
AttestationExpired failure mode where expireAfter=0 produced expiration == reference_block DOA attestations. - Three-layer drift defense lesson generalized — applies to any class of artifact where on-chain commitment must bind cryptographically
to off-chain bytes. Sub-A applies it to private data; Sub-B applies it to policy artifacts. Both reuse the same ChallengeVerifier
re-derivation pattern.
Current Issues / Limitations
- B7 reference-block stashing deferred — Phase 1 reads current instance from registries since clients deploy a new instance per config
change. If a client races setCurrent mid-task, mitigation is task-creation-time instanceId snapshot + verifier reads immutable instance.
Phase 1.5 hardening. - Pre-warm subscriber bootstrapping — A new operator joining mid-stream must replay all TemplateRegistered events from genesis via
from_block backfill before live subscription. May take minutes at high event volume. - No Linear decomposition yet — 47 tasks not yet captured as Linear issues. Sub-A's pattern was 50 issues across NEWT-1003..1052 created
post-spec; Sub-B's equivalent is pending. - docs/superpowers/ is gitignored — both spec and plan are local-only. Cannot be committed; teammates need this memory file + recall
entries to track Sub-B context.
Remaining TODOs
- Choose execution mode — Subagent-Driven (recommended for parallel A+B streams) vs Inline. User must pick before plan execution
begins. - Decompose into Linear issues — Sub-A pattern: 47 tasks → ~47 issues with crate-shaped titles, tags, depends-on/blocks, acceptance
criteria. Mirror NEWT-1003..1052 conventions. - Begin Stream A Task A1 — Extend NamespaceId with PolicyRego (0x04), PolicyParamsSchema (0x05), PolicyDataWasm (0x06),
PolicyDataParamsSchema (0x07), SecretsSchema (0x08). - Begin Stream B Task B1 in parallel — IPolicyRegistry interface with BlobReference Solidity type.
- Phase 1.5 hardening — instanceId snapshot at task creation; switch use_ipfs_resolver default to false.
- Phase 2 cleanup — delete IPFS resolver code; consider atomic bundle multicall (Option A).
Aligned Directions (discussed, not implemented)
- Sub-A primitive reuse — Sub-B will share Sub-A's StateCommitRegistry, IStateRootCommittable, JMT, EigenDaClient, BootnodeServer,
NamespaceId infrastructure unchanged. - Three-layer drift defense as protocol pattern — both Sub-A and Sub-B implement it; future on-chain-bytes work should follow the same
L1-deploy / L2-eval / L3-challenge pattern. - Five-tier cache as standard for content-addressed artifacts — LayeredArtifactCache<L0,L1,L2,L3,L4> reusable for any future
committed-bytes class.
Open Questions
- Linear project name for Sub-B — likely Onchain Policy Storage or similar; not yet created.
- Atomic bundle multicall priority — Phase 2 deferral assumes atomic deploy is not blocking for Phase 1 launches. Re-evaluate after
first Phase 1 client onboards. - Quickstart policy migration ordering — 14 quickstart policies; should they migrate one batch or staggered? Sub-A pattern was one batch
via deploy script.
Relevant Links
- Spec (gitignored): docs/superpowers/specs/2026-04-22-onchain-policy-storage-design.md
- Plan (gitignored): docs/superpowers/plans/2026-04-22-onchain-policy-storage-plan.md
- Sub-A spec (PR #570 merged): docs/PRIVATE_DATA_STORAGE.md
- Sub-A plan (gitignored): docs/superpowers/plans/2026-04-20-private-data-migration-plan.md
- Sub-A memory: project_private_data_storage.md (sibling file in this dir)
Anything to add, correct, or remove before I save this across all 3 systems?
✻ Cooked for 3m 5s
25 tasks (17 done, 1 in progress, 7 open)
◼ Transition to writing-plans skill for implementation plan
◻ Propose 2-3 approaches with trade-offs
◻ Present design sections for approval
◻ Apply 10-item delta report (scope C)
◻ Verification pass 2
… +3 pending, 17 completed
Open Questions Q.1–Q.7 in the document need resolution before stream-by-stream execution starts — most notably the JSON Schema
validator library choice (Q.2), quickstart-policy migration ordering (Q.3), and overlap with NEWT-1036 (Q.4). Implementation plan
execution mode (Subagent-Driven vs Inline) is still to be picked. --> Let's actually discuss this and align on this
⏺ Good call — let's actually work through these. Let me re-read the exact Q.1–Q.7 framing from the doc so we discuss what's actually
written, not what I remember.
Searched for 1 pattern, read 1 file (ctrl+o to expand)
⏺ Let's work through them. I'll give my read on each plus the question I'd ask back. We're discussing 8 items total (the deferred Q.0
execution-mode pick + Q.1–Q.7 in the doc).
★ Insight ─────────────────────────────────────
- Q.2, Q.3, and Q.4 are the only blockers. Q.0/Q.1 are workflow choices, Q.5/Q.6 are deferral decisions (default to "no, ship Phase 1"),
Q.7 is documentation hygiene. The three real blockers all need an answer before stream A starts. - Two of the three blockers are people-coordinated, not technical. Q.3 is "how does Newton ops sequence the Safe multisig sessions" and
Q.4 is "when does NEWT-1036's deletion land relative to Sub-B's adds." Those answers come from you/Aaron and the team, not from spec
analysis. Q.2 is the only one that can be resolved by benchmark. - AddressesProviderConsumer immutable-caching means Q.4 is more interesting than it looks. If any contract on chain consumes
IdentityRegistry/ConfidentialDataRegistry IDs from the provider today, it cached those addresses as immutables — so NEWT-1036's deletion
doesn't hot-swap them. Sub-B's new contracts are net-new consumers with no prior immutable caching, so Q.4 is mostly about ordering
registrar updates, not hot-swapping live consumers.
─────────────────────────────────────────────────
Q.0 — Execution mode for the 47-task plan
My recommendation: Subagent-Driven.
47 tasks × ~3-5 steps each won't fit one context window even with checkpoints. The two-stage review (subagent + me) catches drift on
Sub-B's intricate identity formulas (the 0x01 domain-separator byte on policyDataInstanceId, the chainId ‖ clientAddr ordering) — those
are exactly the kinds of details that get fuzzy when context is full. Inline only wins for <10-task plans where ceremony overhead
exceeds the benefit.
Question back: any reason to prefer Inline? (e.g., wanting to watch each task land in real-time, or tighter feedback on early tasks
before parallelizing?)
Q.1 — Linear project name
My recommendation: ship the provisional "Onchain Policy Storage."
This is naming bikeshed territory — both "On-chain" and "Onchain" are correct. Newton ops decides; doesn't block code. Mirroring Sub-A's
"Private Data Storage" (no hyphen in URLs) is reasonable.
Question back: want me to also pre-register the 47 Linear issues mirroring NEWT-1003..1052 conventions, or do that later when the plan
is closer to execution?
Q.2 — JSON Schema validator library (boon vs jsonschema)
My recommendation: benchmark first; default to jsonschema if inconclusive.
boon is lighter (~150-200KB smaller binary) but newer. jsonschema is mature with broader Draft 2020-12 coverage. The 14 quickstart
schemas are the natural test bed. Concrete proposal:
┌────────────┬───────────────────────────────────────────┬──────────────────────────────────────────────────────────────────────────┐
│ Library │ Strengths │ Risks │
├────────────┼───────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ boon │ Pure Rust, smaller deps, │ Newer (less battle-tested), some Draft 2020-12 features may be │
│ │ no_std-friendlier │ incomplete │
├────────────┼───────────────────────────────────────────┼──────────────────────────────────────────────────────────────────────────┤
│ jsonschema │ Mature, broad coverage, active │ ~400KB heavier, may pull in async runtime indirectly │
│ │ maintenance │ │
└────────────┴───────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────┘
Stream A early: write the 14 quickstart schemas, run both validators, pick whichever passes 100% of the suite with the cleanest API. If
both pass, take boon for the smaller footprint.
Question back: is binary footprint (operator daemon size) actually a constraint we care about right now, or is correctness/maturity the
only axis?
Q.3 — Quickstart migration ordering
My recommendation: staged 3 + 5 + 6, NOT one batch.
This is the most consequential ops decision in the list. Reasoning:
- Single batch (14 in one Safe session): faster, but a single hash-mismatch or schema-validation surprise rolls back 14 deploys. The L1
drift-detection layer (registerTemplate rejects keccak256 mismatch) is new code — not yet field-tested at scale. - Staged (3 + 5 + 6): first batch should be the simplest policies (single-domain or read-only). If something surfaces, we catch it on 3
instead of 14. Safe multisig overhead per batch is ~10-20 min — three sessions across a week is cheap insurance.
The G2 ChallengeVerifier re-derivation hook is also new; staging gives Newton ops a real-world signal that on-chain policyTemplateRef
values match what challengers expect before all 14 are exposed.
Question back: which of the 14 quickstart policies are the simplest / most tested today? That informs the first batch.
Q.4 — NEWT-1036 overlap
My recommendation: NEWT-1036 lands first, Sub-B's adds land after.
NEWT-1036 (deletes IdentityRegistry + ConfidentialDataRegistry) is independent of Sub-B's net-new PolicyRegistry + PolicyDataRegistry.
They could ship in either order, but landing NEWT-1036 first keeps the provider directory smaller and avoids a transitional window where
the provider has both the deprecated entries and Sub-B's adds.
The immutable-caching concern only matters if there are existing consumers of the deleted IDs — which there will be (the registries'
direct consumers). NEWT-1036 has to handle redeployment of those consumers regardless of Sub-B. Sub-B's consumers are net-new, so they
only see the post-deletion provider state.
Question back: is NEWT-1036 ready to ship, or is its blocker the same as Sub-B's (waiting for Sub-A primitives to soak)? If they're both
gated on the same thing, sequencing them serially is the safer call.
Q.5 — Atomic-bundle priority (Phase 2 multicall)
My recommendation: defer to Phase 2 backlog as documented.
The "deploy-then-update-pointer" race window is exactly one block on most chains and only matters if an observer queries
currentPolicyInstance[client] between the two txs. For Phase 1 — where clients control their own deploys — this is acceptable. Pull it
forward only if real-world ops surface frequent friction.
Question back: any known external dependency that observes currentPolicyInstance[client] mid-deploy and would break under the race
window?
Q.6 — policyParams on-chain mirror
My recommendation: ship Phase 1 with the on-chain mirror.
The 14 quickstart policyParams payloads are small (likely <1KB each, mostly string addresses + numeric thresholds). On-chain storage
cost is bounded. Moving to EigenDA-backed policyParamsRef adds challenger-side fetch + KZG verification complexity — unjustified at MVP
scale.
Threshold to reconsider: any single policyParams exceeding 8KB, or storage cost on Base/L2s becoming visible in operator economics.
Question back: is there a known upcoming policy with policyParams >2KB? If so, that flips the calculus.
Q.7 — Secrets schema ↔ HPKE envelope versioning
My recommendation: keep independent (as Phase 1 ships); document the orthogonality.
Coupling them would force an envelope rotation on every schema update — costly entanglement for no security benefit. The envelope_format
byte (0x03 namespace) describes how bytes are encrypted; the secrets schema (0x08 namespace) describes what fields are in the
plaintext. These are genuinely orthogonal — a _v2 schema with new fields can live under the same SECRETS_ENVELOPE_V1 envelope as long as
the new fields fit in the existing encrypted blob.
The doc already describes the failure mode (WASM eval fails deterministically when a required field is absent) — that's enough.
Question back: none from me on this one. It's a "default to keeping things uncoupled" decision unless someone has a concrete pairing
reason.
Summary table
┌─────┬─────────────────────────────────────┬────────────────┬──────────────────────────────────────┐
│ Q │ My pick │ Confidence │ Needs you to confirm? │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.0 │ Subagent-Driven │ High │ Yes │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.1 │ "Onchain Policy Storage" │ Low (bikeshed) │ Yes │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.2 │ Benchmark first; default jsonschema │ Medium │ No (handled in stream A) │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.3 │ Staged 3 + 5 + 6 │ High │ Yes │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.4 │ NEWT-1036 first, Sub-B after │ Medium │ Yes (depends on NEWT-1036 readiness) │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.5 │ Defer to Phase 2 │ High │ Yes (sanity check) │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.6 │ On-chain mirror │ High │ Yes (sanity check) │
├─────┼─────────────────────────────────────┼────────────────┼──────────────────────────────────────┤
│ Q.7 │ Keep independent │ High │ Yes (sanity check) │
└─────┴─────────────────────────────────────┴────────────────┴──────────────────────────────────────┘
What's your read on each? Which of my recommendations are wrong, which want different framing, and which are you fine with as-is?
✻ Crunched for 2m 7s
※ recap: Goal: lock Sub-B (on-chain policy storage) protocol decisions before implementing the 47-task plan. Current task: align on Q.0
execution mode plus Q.1–Q.7 from the doc. Next action: react to my recommendations in the summary table.
※ recap: We're locking Sub-project B's architecture via grill-me interview; Q7 (challenger verification) is now corrected and locked,
confirming challengers don't stake in Newton. Next: decide Q8.3 atomicity — strict
PolicyDeployerfactory vs idempotent independentregistration.
❯ 1. CLI. dev use cli to register a new policy template. Or if there is an existing policy template they want to use, they just deploy
instance with policy params. But if policy has some policy data, then in order to actually use it / simulate it, for policy data
that requires client secrets, they need to also set secrets for each policy data required. Depending on whether policy has params
schema or polciy data has params schema/secrets schema, there might be multiple steps that the dev needs to take step by step to
fully set up policy/policy data for their policy clients. 2. either update, or extend , etc. Basically, this update will be breaking
change for the cli commands, and we should design the cli to be most optimally designed for the new architecture. (no legacy
consideration) 3. register them in any order is okay to start with. Atomic experience is a nice to have, but not required. TLDR
Option B, but Option A as nice to have dev experience improvements.
⏺ Locking Q8 with all three sub-decisions captured. The substantive output is the CLI command surface — let me draft it concretely so
you can confirm or push back on specifics.
Q8 Deployment flow — LOCKED
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬────────┐
│ Sub-decision │ Status │
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ Q8.1 CLI is the canonical deployment interface; multi-step setup driven by which schemas are non-empty │ LOCKED │
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ Q8.2 Breaking change for CLI; redesign command surface around template/instance model with no legacy carry-over │ LOCKED │
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────┤
│ Q8.3 Any-order, idempotent registration (Option B); atomic PolicyDeployer bundle is Phase 2 nice-to-have │ LOCKED │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────┘
Proposed CLI command surface (Sub-project B)
Naming pattern: newton-cli — read top-down: domain → entity-type → action. Replaces existing newton-cli policy
deploy etc. entirely.
Templates (immutable, content-addressed)
newton-cli policy template register
--rego ./policy.rego
--params-schema ./params.schema.json \ # optional: empty schema = no params
--data-templates 0xabc...,0xdef... # optional: list of policyDataTemplateIds
newton-cli policy template show
newton-cli policy template list [--author 0x...]
newton-cli policy template compute-id ./policy.rego ./params.schema.json --data-templates ...
newton-cli policy-data template register
--wasm ./data.wasm
--params-schema ./pdp.schema.json \ # optional
--secrets-schema ./secrets.schema.json # optional
newton-cli policy-data template show
newton-cli policy-data template list [--author 0x...]
newton-cli policy-data template compute-id ./data.wasm ./pdp.schema.json ./secrets.schema.json
Instances (per-client, mutable config)
newton-cli policy instance deploy
--template
--client 0xclientAddr
--params ./params.json \ # required if template has non-empty schema; else --no-params
--data-instances 0xinst1,0xinst2 \ # required: must match arity of template's dataTemplateRefs
--expire-after 150
newton-cli policy instance show
newton-cli policy instance list --client 0x...
newton-cli policy-data instance deploy
--template
--client 0xclientAddr
--params ./params.json \ # required if template has non-empty schema
--expire-after 150
newton-cli policy-data instance show
newton-cli policy-data instance list --client 0x...
Secrets (per-instance, encrypted, mutable)
newton-cli policy-data instance set-secrets
--instance
--secrets ./secrets.json # required only if template's secretsSchema is non-empty
newton-cli policy-data instance show-secrets
Shows: ciphertext envelope per operator + secretsSchema (NOT plaintext)
Plan / status (the "what do I still need to do" command)
newton-cli policy plan --policy-instance
Output:
Policy template : REGISTERED
Data template : REGISTERED
Data template : MISSING ← register required
Data instance : DEPLOYED
Data instance : DEPLOYED but secrets MISSING ← run set-secrets
Policy instance : NOT DEPLOYED
★ Insight ─────────────────────────────────────
The plan command is the antidote to multi-step DX pain. Instead of forcing the dev to run six commands and remember which schemas are
non-empty, plan walks the dependency DAG from a target instance back to roots and prints "what's done, what's missing, what's next."
This is the same pattern as kubectl describe or Terraform's plan — it reads beats writing for intermediate steps. It also gives us a
stepping-stone toward Option A (the atomic bundle) in Phase 2: plan --apply could just sequence the missing operations. The plan
command is cheap because everything is content-addressed — the CLI computes IDs locally and queries on-chain only for "is this ID
registered yet."
─────────────────────────────────────────────────
Step-by-step dev workflow (the dependency DAG)
For the maximally-complex case (new policy + new data templates + non-empty schemas + non-empty secrets), here's the strict
dependency order. The CLI lets you run these in any order, but they must all complete before the policy instance is usable.
┌─────┬──────────────────────────┬───────────────────────────────────────────────────┬──────────────────────────────────────────┐
│ # │ Operation │ Required when │ Notes │
├─────┼──────────────────────────┼───────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ 1a │ policy-data template │ Each unique data WASM that isn't already on-chain │ Idempotent; same content → same ID │
│ │ register │ │ │
├─────┼──────────────────────────┼───────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ 1b │ policy template register │ New Rego logic │ Idempotent; references step-1a IDs │
├─────┼──────────────────────────┼───────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ 2a │ policy-data instance │ Each data dependency (one per template, per │ One per element in policy template's │
│ │ deploy │ client, per param config) │ dataTemplateRefs │
├─────┼──────────────────────────┼───────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ 2b │ policy-data instance │ Each data instance whose template has non-empty │ CLI fan-outs HPKE envelopes per operator │
│ │ set-secrets │ secretsSchema │ │
├─────┼──────────────────────────┼───────────────────────────────────────────────────┼──────────────────────────────────────────┤
│ 3 │ policy instance deploy │ Final step, references all step-2a IDs │ All step-2 must be complete │
└─────┴──────────────────────────┴───────────────────────────────────────────────────┴──────────────────────────────────────────┘
When schemas are empty, steps collapse:
┌──────────────────────────────┬──────────────────────────────────────────────┐
│ Schema state │ Required for instance deploy │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ policyParamsSchema empty │ --no-params flag, no params.json file needed │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ policyDataParamsSchema empty │ Same --no-params for data instance │
├──────────────────────────────┼──────────────────────────────────────────────┤
│ secretsSchema empty │ set-secrets step is skipped entirely │
└──────────────────────────────┴──────────────────────────────────────────────┘
Smallest possible deploy (zero-config policy, no data dependencies):
newton-cli policy template register --rego ./policy.rego
newton-cli policy instance deploy --template --client 0x... --no-params
Two commands. That's the floor.
Largest realistic deploy (3 data dependencies, all schemas non-empty):
One-time per unique template (skip if already on-chain):
newton-cli policy-data template register --wasm ./d1.wasm --params-schema ./d1p.json --secrets-schema ./d1s.json
newton-cli policy-data template register --wasm ./d2.wasm --params-schema ./d2p.json --secrets-schema ./d2s.json
newton-cli policy-data template register --wasm ./d3.wasm --params-schema ./d3p.json
newton-cli policy template register --rego ./pol.rego --params-schema ./pp.json --data-templates 0xd1,0xd2,0xd3
Per-client setup:
newton-cli policy-data instance deploy --template 0xd1 --client 0xc --params ./d1-config.json
newton-cli policy-data instance deploy --template 0xd2 --client 0xc --params ./d2-config.json
newton-cli policy-data instance deploy --template 0xd3 --client 0xc --no-params
newton-cli policy-data instance set-secrets --instance 0xi1 --secrets ./d1-secrets.json
newton-cli policy-data instance set-secrets --instance 0xi2 --secrets ./d2-secrets.json
d3 has empty secretsSchema; no set-secrets needed
newton-cli policy instance deploy --template 0xpt --client 0xc --params ./pol-config.json --data-instances 0xi1,0xi2,0xi3
10 commands max. Realistic average for a non-trivial policy: 5-7.
What the CLI does silently for the dev
These transparent steps eliminate per-command friction:
┌──────────────────────────────────────────────┬────────────────────────────────────┬───────────────────────────────────────────┐
│ Step │ Where │ Why it matters │
├──────────────────────────────────────────────┼────────────────────────────────────┼───────────────────────────────────────────┤
│ Read file, compute content hash │ Local │ Dev doesn't compute IDs themselves │
├──────────────────────────────────────────────┼────────────────────────────────────┼───────────────────────────────────────────┤
│ Query on-chain "is this ID registered?" │ RPC │ Skip re-registration silently │
│ │ │ (idempotency) │
├──────────────────────────────────────────────┼────────────────────────────────────┼───────────────────────────────────────────┤
│ Upload bytes to EigenDA, get blob ref │ EigenDA client │ Dev never sees blob refs │
├──────────────────────────────────────────────┼────────────────────────────────────┼───────────────────────────────────────────┤
│ Validate params.json against │ Local (after fetching schema from │ Catches errors before tx submission │
│ policyParamsSchema │ DA) │ │
├──────────────────────────────────────────────┼────────────────────────────────────┼───────────────────────────────────────────┤
│ HPKE-encrypt secrets per operator │ Local │ Dev sees plaintext only on their machine │
├──────────────────────────────────────────────┼────────────────────────────────────┼───────────────────────────────────────────┤
│ Compute policyInstanceId deterministically │ Local │ Dev can --dry-run to preview ID │
├──────────────────────────────────────────────┼────────────────────────────────────┼───────────────────────────────────────────┤
│ Submit on-chain tx │ RPC │ Single tx per command (Option B) │
└──────────────────────────────────────────────┴────────────────────────────────────┴───────────────────────────────────────────┘
★ Insight ─────────────────────────────────────
The reason --dry-run is critical for this CLI (more so than for the existing one): with content-addressed instance IDs, the dev needs
the ID before the deploy happens, because they need to reference it in the next step (policy instance deploy --data-instances ).
So every deploy and register command supports --dry-run that prints the ID it would produce, and the actual tx is --commit. This
lets a dev script the whole bundle as ID1=$(... --dry-run); ID2=$(... --dry-run); ... --commit before committing anything to chain.
It also lays the groundwork for Option A: a policy bundle apply command can compute every ID upfront, encode all txs, and submit them
sequentially (or, with PolicyDeployer, atomically in Phase 2).
─────────────────────────────────────────────────
Cross-project dependency: secrets storage
The policy-data instance set-secrets command writes encrypted bytes to the state tree under namespace 0x03 (Sub-project A's "secrets"
namespace). The CLI delegates to Sub-project A's signed-write API — Sub-project B's contracts don't store ciphertext directly.
Implication for spec: Sub-project B's docs/superpowers/specs/2026-04-22-onchain-policy-storage-design.md will reference Sub-project
A's secrets-write API as a hard dependency. Suggest adding a "Cross-project integration" section listing:
Open sub-questions on Q8 (or close it?)
A few residual decisions I'd like a yes/no on before closing Q8 entirely:
Newton hits the expireAfter=0 footgun (per error-selectors.md AttestationExpired); we should require non-zero on mainnet at minimum.
migrate references, or (b) we add an update-params command that creates a new instance under the hood and emits a migration event? My
preference is (a) — clean, content-addressed, no special-casing. (b) is friendlier UX but adds a "current instance pointer"
indirection.
a different address and loses ownership of the instance.
If you confirm 1=non-zero-required-on-mainnet, 2=Option (a), 3=yes-default-to-signer, I'll close Q8 and move to Q10 (governance &
lifecycle — largely Phase 2 deferral, should be quick).
✻ Churned for 1m 55s