Skip to content

Instantly share code, notes, and snippets.

@jordotech
Last active March 26, 2026 22:12
Show Gist options
  • Select an option

  • Save jordotech/96e62bf0fd8842fb74c8638345a5fba1 to your computer and use it in GitHub Desktop.

Select an option

Save jordotech/96e62bf0fd8842fb74c8638345a5fba1 to your computer and use it in GitHub Desktop.
EY Application-Level S3 Encryption - Technical Overview

Application-Level S3 Encryption: Technical Overview

Prepared for: EY Information Security Team Date: March 2026 Status: Proposed (LOE Review)


Executive Summary

This document describes a proposed application-level envelope encryption layer for all files stored in EY-designated S3 buckets. This layer ensures that files are stored as ciphertext and are unreadable outside the Capitol.ai application — even to platform operators with direct AWS storage access. Decryption occurs on-the-fly within the application at the moment of retrieval.

This proposal was designed to address three specific EY requirements:

# Requirement How This Solution Addresses It
1 Capitol admins should only see ciphertext — not raw file content — when accessing storage directly Files are encrypted with per-file keys before storage. Direct S3 access returns only ciphertext. Decryption requires a key unwrap operation through Azure Key Vault, which only the application performs.
2 Key rotation — EY's policy mandates key rotation every 2 years Supported via Azure Key Vault key versioning. A zero-downtime re-encryption process migrates files to the new key version (see Key Rotation section).
3 Key revocation — data must become completely inaccessible when EY revokes keys By design. If the KEK is revoked in Azure Key Vault, no wrapped DEKs can be unwrapped, and all encrypted files become permanently unreadable. This is an irreversible security guarantee under EY's control.

Security Architecture

This is the second encryption layer in a defense-in-depth architecture, complementing the existing XKS hardware-backed storage encryption.

graph TB
    subgraph "Layer 1: XKS Storage Encryption"
        S3["S3 Bucket"] -->|encrypted at rest| KMS["AWS KMS<br/>(Custom Key Store)"]
        KMS -->|key operations via| XKS["XKS Proxy"]
        XKS -->|wrapping key stored in| AKV1["Azure Key Vault<br/>(EY-controlled)"]
    end

    subgraph "Layer 2: Application-Level Encryption (Proposed)"
        APP["Application"] -->|envelope encryption| CIPHER["Ciphertext in S3"]
        APP -->|key agreement via| AKV2["Azure Key Vault<br/>(EY-controlled)"]
    end

    style AKV1 fill:#0066cc,color:#fff
    style AKV2 fill:#0066cc,color:#fff
    style CIPHER fill:#cc3300,color:#fff
Loading
Layer What It Protects Against Key Custody
XKS Encryption Unauthorized physical/infrastructure access to storage Azure Key Vault (EY-controlled)
App-Level Encryption (proposed) Direct storage access by platform operators with AWS credentials Azure Key Vault (EY-controlled)

Threat Model

Scenario: A platform operator with AWS console or CLI access attempts to read EY files directly from S3, bypassing the application.

  • Layer 1 (XKS) does not prevent this — AWS KMS decrypts transparently for any principal with kms:Decrypt permission.
  • Layer 2 (App-Level Encryption) prevents this — files are stored as ciphertext with a per-file encryption key that can only be unwrapped via Azure Key Vault. Direct S3 access returns only ciphertext.

Encryption Scheme: Envelope Encryption with Forward Secrecy

Each file is encrypted with a unique Data Encryption Key (DEK). The DEK is protected using ECDH-ES (Elliptic Curve Diffie-Hellman Ephemeral-Static) key agreement with the Key Encryption Key (KEK) in Azure Key Vault. This provides forward secrecy — if the long-term KEK is compromised in the future, previously encrypted files remain protected because each wrapping operation uses a unique ephemeral key that is discarded after use.

Key Hierarchy

graph LR
    EPH["Ephemeral EC Key<br/>(generated per file,<br/>discarded after wrap)"] -->|ECDH key agreement with| KEK["KEK<br/>(Azure Key Vault)<br/>EC P-384"]
    EPH -->|derives| WK["Wrapping Key<br/>(AES-256, ephemeral)"]
    WK -->|wraps| DEK["DEK<br/>(per-file)<br/>AES-256"]
    DEK -->|encrypts| FILE["File Content<br/>(ciphertext in S3)"]

    style KEK fill:#0066cc,color:#fff
    style EPH fill:#996699,color:#fff
    style WK fill:#996699,color:#fff
    style DEK fill:#339966,color:#fff
    style FILE fill:#cc3300,color:#fff
Loading
Component Algorithm Location Lifecycle
KEK (Key Encryption Key) EC P-384 Azure Key Vault Long-lived, EY-managed, rotatable every 2 years
Ephemeral EC Key EC P-384 (ECDH-ES) Generated in application memory Per-file, discarded immediately after key agreement
Wrapping Key AES-256 (derived via ECDH) Derived in memory, never stored Per-file, ephemeral
DEK (Data Encryption Key) AES-256-GCM Generated per file, stored wrapped in S3 metadata Per-file, never stored in plaintext
IV (Initialization Vector) 96-bit random Stored in S3 object metadata Per-file, unique

Forward secrecy guarantee: The ephemeral EC private key is generated in memory, used once for ECDH key agreement, and immediately discarded. Only the ephemeral public key is stored in S3 metadata (needed for unwrapping). Even if the KEK private key in Azure Key Vault were compromised at a future date, an attacker would also need the ephemeral private keys — which no longer exist.

Write Path (File Upload)

sequenceDiagram
    participant User
    participant App as Application
    participant AKV as Azure Key Vault
    participant S3

    User->>App: Upload file
    App->>App: Generate random DEK (AES-256)
    App->>App: Generate ephemeral EC key pair
    App->>AKV: ECDH key agreement (ephemeral pub + KEK)
    AKV-->>App: Shared secret
    App->>App: Derive wrapping key (HKDF)
    App->>App: Wrap DEK with wrapping key (AES-KW)
    App->>App: Discard ephemeral private key
    App->>App: Encrypt file with DEK (AES-256-GCM)
    App->>S3: Store ciphertext + metadata
    Note over S3: Metadata: wrapped DEK,<br/>ephemeral public key, IV
    App-->>User: Upload confirmed
Loading

Read Path (File Download)

sequenceDiagram
    participant User
    participant App as Application
    participant AKV as Azure Key Vault
    participant S3

    User->>App: Request file download
    App->>S3: Retrieve ciphertext + metadata
    S3-->>App: Ciphertext, wrapped DEK, ephemeral public key, IV
    App->>AKV: ECDH key agreement (ephemeral pub + KEK)
    AKV-->>App: Shared secret
    App->>App: Derive wrapping key (HKDF)
    App->>App: Unwrap DEK
    App->>App: Decrypt ciphertext (AES-256-GCM)
    App->>App: Verify authentication tag (integrity check)
    App-->>User: Stream decrypted file
Loading

Large File Handling

Files exceeding 50 MB use a chunked streaming encryption scheme. Each chunk is independently encrypted and authenticated, enabling streaming decryption without buffering the entire file in memory.

graph LR
    subgraph "Chunked Encryption (files > 50 MB)"
        C1["Chunk 1<br/>+ Auth Tag"] --> C2["Chunk 2<br/>+ Auth Tag"]
        C2 --> C3["Chunk 3<br/>+ Auth Tag"]
        C3 --> CN["Chunk N<br/>+ Final Tag"]
    end

    DEK["Shared DEK<br/>(per-file)"] -.->|encrypts each chunk| C1
    DEK -.-> C2
    DEK -.-> C3
    DEK -.-> CN

    style DEK fill:#339966,color:#fff
Loading

Each chunk carries its own authentication tag, ensuring:

  • Tamper detection at the chunk level
  • Streaming decryption without full-file buffering
  • Truncation detection via a final sentinel tag

Scope of Changes

Services Requiring Modification

Three application services interact with EY file storage:

Service Role Changes Required
Platform API File uploads, downloads, browser previews Encrypt on upload, decrypt on download, replace direct download links with authenticated proxy endpoints
Search Indexing Service Parses uploaded documents for vector search Decrypt before parsing, re-encrypt if storing intermediate results
Workflow Engine Stores and retrieves workflow-generated files and outputs Encrypt/decrypt for all file I/O operations

Shared Encryption Library

A single encryption library will be shared across all three services, providing:

  • Envelope encryption and decryption (AES-256-GCM)
  • ECDH-ES key agreement with Azure Key Vault for forward-secret key wrapping
  • DEK caching (5-minute TTL) to minimize Key Vault API calls
  • Chunked streaming for large files
  • Automatic bypass for non-EY organizations (zero impact to other tenants)

S3 Buckets in Scope

Bucket Purpose Content
Document uploads User-uploaded files (PDFs, documents, spreadsheets)
Workflow files Files attached to AI workflows
Workflow outputs AI-generated reports and analysis results

Encryption Decision Flow

The encryption layer activates only for EY organizations. Other tenants are completely unaffected.

flowchart TD
    REQ["File Operation"] --> CHECK{"Organization has<br/>encryption enabled?"}
    CHECK -->|No| PLAIN["Standard S3 operation<br/>(no change)"]
    CHECK -->|Yes| ENC["Use envelope encryption"]
    ENC --> WRITE{"Operation type?"}
    WRITE -->|Upload| W["Generate DEK → ECDH wrap → Encrypt → Store"]
    WRITE -->|Download| R["Fetch ciphertext → ECDH unwrap → Decrypt → Stream"]

    style PLAIN fill:#339966,color:#fff
    style W fill:#0066cc,color:#fff
    style R fill:#0066cc,color:#fff
Loading

Key Rotation

EY's key rotation policy (every 2 years) is fully supported with zero downtime.

How It Works

sequenceDiagram
    participant EY as EY Admin
    participant AKV as Azure Key Vault
    participant App as Application
    participant S3

    EY->>AKV: Create new KEK version (v2)
    Note over AKV: v1 remains active for reads
    EY->>App: Trigger re-encryption

    loop For each encrypted file
        App->>S3: Read ciphertext + metadata
        App->>AKV: Unwrap DEK using KEK v1
        AKV-->>App: Plaintext DEK
        App->>AKV: Re-wrap DEK using KEK v2 (ECDH-ES)
        AKV-->>App: New wrapped DEK
        App->>S3: Update metadata with new wrapped DEK
        Note over S3: File content unchanged —<br/>only metadata updated
    end

    EY->>AKV: Disable KEK v1 (optional)
Loading

Key properties of rotation:

Property Detail
Downtime required None — reads and writes continue during rotation
File re-encryption Not required — only the DEK wrapping is updated, not the file ciphertext itself
Duration Depends on file count. Metadata-only updates are fast (~1,000 files/minute)
Rollback Safe — old KEK version can remain enabled until rotation is verified complete
Concurrent access Supported — the application checks KEK version in metadata and uses the correct version for unwrapping

Rotation Process

  1. EY creates a new KEK version in Azure Key Vault (v1 → v2)
  2. Background re-wrap job runs: for each file, unwraps DEK with old KEK version, re-wraps with new KEK version, updates S3 metadata
  3. Verification: audit confirms all files reference the new KEK version
  4. EY disables old KEK version once re-wrap is complete (optional — old version can remain as a safety net)

Since only the wrapped DEK metadata changes (not the file ciphertext), this process is fast and does not require re-uploading or re-encrypting file contents.


Alternative Considered: Full Migration to Azure

We evaluated migrating EY's storage infrastructure entirely to Azure (Azure Blob Storage + Azure Key Vault native encryption) as an alternative approach.

Factor App-Level Encryption (Recommended) Full Azure Migration
Implementation risk Low — additive change to existing system High — requires re-architecting storage, networking, and service layers
Operational complexity Minimal — same infrastructure, new encryption layer Significant — dual-cloud operations, new monitoring, new incident response
Timeline 3-5 weeks 3-6 months (estimated)
Impact to other tenants None — encryption is per-organization Potential — infrastructure changes affect shared components
Security guarantee Equivalent — ciphertext at rest, key custody in Azure Key Vault Equivalent — Azure-native encryption with Azure Key Vault
Forward secrecy Yes (ECDH-ES) Depends on Azure Blob encryption mode

Recommendation: The application-level encryption approach delivers the same security guarantees with significantly lower risk and faster delivery. A full Azure migration introduces operational complexity (dual-cloud infrastructure, new deployment pipelines, cross-cloud networking) without meaningfully improving the security posture for file encryption.


Download URL Changes

Standard pre-signed S3 URLs return raw bytes from storage. Since files are now stored as ciphertext, direct S3 URLs would return encrypted (unreadable) content.

Solution: Browser file previews and downloads are served through an authenticated proxy endpoint. The application verifies the user's identity, decrypts the file on-the-fly, and streams the plaintext to the browser. Download URLs are signed with time-limited tokens (configurable expiry, default 1 hour).


Data Model Changes

Three new fields on the organization record control encryption behavior:

Field Purpose
Encryption enabled Boolean flag to activate app-level encryption for this organization
Key name Identifies which Azure Key Vault key to use for this organization's KEK
Key Vault URL Azure Key Vault endpoint for this organization

This per-organization model supports:

  • Independent key management per customer
  • Gradual rollout (enable per organization)
  • Immediate disable for rollback (flag flip)

Migration Strategy

New Files

All files uploaded after enablement are automatically encrypted. No user action required.

Existing Files

A background migration process encrypts existing files in place:

  1. Read existing plaintext file from S3
  2. Encrypt using the organization's KEK via ECDH-ES key agreement
  3. Write ciphertext back to S3 with encryption metadata
  4. Verify round-trip integrity (decrypt and compare hash)

Migration runs at controlled throughput to avoid service impact. Estimated duration depends on total file volume.


Rollback Plan

If issues arise, rollback is immediate and non-destructive:

  1. Disable encryption flag on the organization record (instant — new operations bypass encryption)
  2. Run reverse migration to decrypt existing files back to plaintext (background process)
  3. Revert application code if needed (standard deployment rollback)

No data is lost at any stage. The wrapped DEKs in S3 metadata allow decryption at any time as long as the KEK exists in Azure Key Vault.


Failure Modes

Scenario Impact Mitigation
Azure Key Vault unreachable New uploads fail; downloads of uncached files fail DEK cache (5-min TTL) allows continued reads for recently accessed files. Uploads queue and retry.
Partial write failure Incomplete ciphertext in S3 Atomic write pattern: write to temporary key, rename on success. Failed writes are cleaned up.
KEK rotation in progress Files may reference old or new KEK version Application checks KEK version in metadata and uses the correct version. Both versions remain active during rotation.
KEK revocation All encrypted files become permanently unreadable By design — this is the security guarantee. KEK revocation is an irreversible emergency measure under EY's control.

Level of Effort

Phase Description Estimated Duration
Phase 1 Shared encryption library (ECDH-ES + AES-256-GCM) + unit tests 1 week
Phase 2 Platform API integration (uploads, downloads, proxy endpoints) 1 week
Phase 3 Search indexing + workflow engine integration 1 week
Phase 4 End-to-end testing, key rotation tooling, migration tooling, staging validation 1-2 weeks
Total 3-5 weeks

Rollout Sequence

Deployment follows a staged regional rollout:

graph LR
    EU["EU Region<br/>(primary)"] -->|validate| APAC["APAC Region"]
    APAC -->|validate| US["US Region"]

    style EU fill:#0066cc,color:#fff
    style APAC fill:#339966,color:#fff
    style US fill:#339966,color:#fff
Loading

Each region is validated before proceeding to the next. Rollback is available at every stage.


Open Questions

  1. APAC latency: Azure Key Vault is currently in a European region. Key operations from APAC may have higher latency (~200-400ms per unwrap). Should a regional Key Vault replica be provisioned?
  2. Key rotation cadence: Confirmed as every 2 years per EY policy. Should the re-wrap job be triggered manually by EY or run automatically on a schedule?
  3. Audit logging: Should all key operations (wrap/unwrap) be logged to a dedicated audit stream accessible to EY?
  4. File size limits: Are there expected maximum file sizes that would affect chunked encryption design?

Summary

This proposal adds a cryptographic guarantee that EY files stored in S3 are unreadable without application-mediated decryption. The ECDH-ES key agreement scheme provides forward secrecy, ensuring that even a future compromise of the long-term key does not expose previously encrypted files. Key custody remains in Azure Key Vault under EY's control, with full support for EY's key rotation and revocation policies.

Combined with the existing XKS storage encryption, this creates a two-layer defense-in-depth architecture where no single point of compromise exposes file contents.

Next steps: Review this proposal, address open questions, and confirm timeline for implementation kickoff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment