# AGENTS.MD – Meta Wearables Device Access Toolkit (DAT) iOS 0.2

> Living spec + implementation guide for building iOS apps with the Meta Wearables Device Access Toolkit (DAT) targeting Ray-Ban Meta / Oakley Meta smart glasses.

---

## 0. Goals of this document

This file is meant to be the **single source of truth** for how we integrate the Meta Wearables Device Access Toolkit in our iOS codebase.

It should help any AI agent / human developer to:

1. Understand the **capabilities and limits** of the SDK.
2. Follow a **consistent architecture** (registration → permissions → device selection → sessions → streaming / photo / audio).
3. Reuse **common patterns** (session management, error handling, device selectors, photo capture, etc.).
4. Quickly find **API details** (types, enums, methods) without re-opening the docs.

Whenever the SDK changes, **update this file first**.

---

## 1. High-level overview

### 1.1 What DAT does

The Wearables Device Access Toolkit (DAT) lets an iOS app connect to supported Meta smart glasses and use them as **remote sensors & outputs**:

- **Camera**: live video streaming + still photo capture.
- **Microphones**: voice input via glasses mic over Bluetooth.
- **Speakers**: audio playback through glasses speakers.

All application logic runs on the **phone**. Glasses provide sensor data and audio I/O; there is **no app runtime on the glasses**.

### 1.2 Supported devices (currently)

- Ray-Ban Meta (Gen 1)
- Ray-Ban Meta (Gen 2)
- Oakley Meta HSTN

Future devices may be added; the SDK exposes `DeviceType` / compatibility utilities.

### 1.3 Core concepts

- **Registration**: one-time handshake between our app and the Meta AI companion app. After registration, our app appears in the Meta AI "App Connections" list.
- **Permissions**: camera permission is granted per-app but confirmed per-device through Meta AI. Microphone/speaker access is handled via iOS audio + Bluetooth permissions.
- **Devices**: AI glasses linked to the Meta AI app and visible through DAT.
- **Device selector**: object deciding **which device** a given session targets (auto or specific).
- **Sessions**:
  - **StreamSession**: media session for live video + optional photo capture.
  - **DeviceStateSession**: session focused on tracking device state.
- **Publishers / async streams**: provide state updates, frames, errors, photo data, registration state, device lists.

---

## 2. iOS integration checklist

### 2.1 Prerequisites

- Xcode project with a **registered bundle identifier**.
- Access to Meta Wearables DAT GitHub repo (`meta-wearables-dat-ios`).
- Meta AI companion app installed on device and compatible glasses paired.

### 2.2 Info.plist configuration

We must configure:

1. **Custom URL scheme** for callbacks from Meta AI.
2. **Query scheme** to allow `fb-viewapp`.
3. **External accessory protocol** for Meta wearables.
4. **Background modes** for Bluetooth / external accessory.
5. **Bluetooth usage description**.
6. **MWDAT dictionary** for `AppLinkURLScheme` + `MetaAppID`.

Template (adjust `myexampleapp` and `MetaAppID`):

```xml
<!-- Custom URL scheme for callbacks from Meta AI -->
<key>CFBundleURLTypes</key>
<array>
  <dict>
    <key>CFBundleTypeRole</key>
    <string>Editor</string>
    <key>CFBundleURLName</key>
    <string>$(PRODUCT_BUNDLE_IDENTIFIER)</string>
    <key>CFBundleURLSchemes</key>
    <array>
      <string>myexampleapp</string>
    </array>
  </dict>
</array>

<!-- Allow Meta AI (fb-viewapp) to call the app -->
<key>LSApplicationQueriesSchemes</key>
<array>
  <string>fb-viewapp</string>
</array>

<!-- External Accessory protocol for Meta Wearables -->
<key>UISupportedExternalAccessoryProtocols</key>
<array>
  <string>com.meta.ar.wearable</string>
</array>

<!-- Background modes for Bluetooth and external accessories -->
<key>UIBackgroundModes</key>
<array>
  <string>bluetooth-peripheral</string>
  <string>external-accessory</string>
</array>

<key>NSBluetoothAlwaysUsageDescription</key>
<string>Needed to connect to Meta Wearables</string>

<!-- Wearables Device Access Toolkit configuration -->
<key>MWDAT</key>
<dict>
  <key>AppLinkURLScheme</key>
  <string>myexampleapp://</string>
  <key>MetaAppID</key>
  <string>0</string> <!-- 0 for dev mode, real ID for production -->
</dict>
```

> If Info.plist is preprocessed, the `://` suffix may get stripped unless the `-traditional-cpp` flag is used (Apple TN2175).

### 2.3 Add SDK via Swift Package Manager

1. Xcode → **File > Add Package Dependencies…**
2. URL: `https://github.com/facebook/meta-wearables-dat-ios/`
3. Pick a tagged release (e.g. `v0.2.0.x`).
4. Add package to all targets that require wearables access.
5. In Swift files using the SDK, import the specific modules:

```swift
import MWDATCore       // Core features, registration, device discovery
import MWDATCamera     // Camera streaming, StreamSession
import MWDATMockDevice // (Debug only) Mock device support
```

### 2.4 Configure SDK on launch

Call **once** during app startup:

```swift
func configureWearables() {
  do {
    try Wearables.configure()
  } catch {
    assertionFailure("Failed to configure Wearables SDK: \(error)")
  }
}
```

### 2.5 URL handling for Meta AI callbacks

When Meta AI returns to the app (registration, permissions), we must forward the URL to the SDK.

**SwiftUI (WindowGroup level):**

```swift
.onOpenURL { url in
    guard let components = URLComponents(url: url, resolvingAgainstBaseURL: false),
          components.queryItems?.contains(where: { $0.name == "metaWearablesAction" }) == true else {
        return
    }
    
    Task {
        do {
            _ = try await Wearables.shared.handleUrl(url)
        } catch {
            print("Handle URL failed: \(error)")
        }
    }
}
```

**UIKit (AppDelegate / SceneDelegate):**

```swift
func handleWearablesCallback(url: URL) async throws {
  _ = try await Wearables.shared.handleUrl(url)
}
```

---

## 3. Registration, devices & permissions

### 3.1 Registration

Registration binds our app to the user’s glasses via Meta AI:

- One-time per install / account.
- User is deep-linked to Meta AI to confirm, then sent back via callback URL.
- After registration, our app appears in Meta AI **App Connections** list.
- User can unregister in Meta AI; we can also start unregistration in-app.

API:

```swift
func startRegistration() throws {
  try Wearables.shared.startRegistration()
}

func startUnregistration() throws {
  try Wearables.shared.startUnregistration()
}
```

Observe registration state asynchronously:

```swift
let wearables = Wearables.shared

Task {
  for await state in wearables.registrationStateStream() {
    // Update registration UI (connected / not connected / in progress)
  }
}
```

### 3.2 Device discovery

Devices (glasses) are discovered via `devicesStream()`:

```swift
Task {
  for await devices in wearables.devicesStream() {
    // devices: [Device]
    // Update our device list UI / selection
  }
}
```

`Device` represents a pair of glasses.

Key members:

- `identifier: DeviceIdentifier` — unique ID.
- `name: String` — human-readable device name (may be empty).
- `nameOrId() -> String` — name or identifier fallback.
- `deviceType() -> DeviceType` — e.g. Ray-Ban Meta.
- `compatibility() -> Compatibility` — compatibility with this DAT version.
- `linkState` — connection state.
- `addCompatibilityListener(_:)` — listen for compatibility changes.
- `addLinkStateListener(_:)` — listen for link state changes.

### 3.3 Permissions

**Camera permission** is managed by DAT through Meta AI:

- Granted at app level but confirmed per device.
- Flow is handled in Meta AI, with options like **Allow once** / **Allow always** / **Deny**.

API pattern:

```swift
var cameraStatus: PermissionStatus = .denied

cameraStatus = try await wearables.checkPermissionStatus(.camera)

if cameraStatus != .granted {
  cameraStatus = try await wearables.requestPermission(.camera)
}
```

Notes:

- If user denies, we must handle gracefully (disable features, show explanation).
- With multiple devices, permission is **granted** if any linked device has approved.

**Microphone / speaker** access uses Bluetooth HFP/A2DP and is governed by iOS audio permissions and routing (not a DAT permission type). See audio section.

---

## 4. Device selection & state sessions

### 4.1 Device selectors

All session-based operations (e.g. `StreamSession`) use a **DeviceSelector**.

#### `AutoDeviceSelector`

```swift
final class AutoDeviceSelector: DeviceSelector {
  public init(wearables: WearablesInterface)

  var activeDevice: DeviceIdentifier?

  public func activeDeviceStream() -> AnyAsyncSequence<DeviceIdentifier?>
}
```

Behavior:

- Automatically selects the “best” device.
- Picks the first connected device from the devices list.
- If no device is connected, falls back to the first device.
- `activeDeviceStream()` updates whenever the device list changes.

Use when we want **smart default selection** with minimal UI friction.

#### `SpecificDeviceSelector`

```swift
class SpecificDeviceSelector: DeviceSelector {
  public init(device: DeviceIdentifier)

  var activeDevice: DeviceIdentifier? { get }

  public func activeDeviceStream() -> AnyAsyncSequence<DeviceIdentifier?>
}
```

Behavior:

- Always targets a **specific** device (by identifier).
- `activeDeviceStream()` yields that device and then completes.

Use when user explicitly selects a device in UI.

### 4.2 DeviceStateSession

`DeviceStateSession` monitors state changes for the selected device.

```swift
final class DeviceStateSession {
  public init(deviceSelector: DeviceSelector)

  var state: /* device state type – see full docs */

  public func start()
  public func stop()
}
```

Use this when we need ongoing device state monitoring separate from media streaming.

---

## 5. StreamSession – video streaming & photo capture

### 5.1 Overview

`StreamSession` manages a **media streaming session** from glasses:

- Live video frames from camera.
- On-demand still photo capture during streaming.
- Real-time state updates.
- Error reporting.

```swift
final class StreamSession {

  // Constructors
  public init(deviceSelector: DeviceSelector)
  public init(streamSessionConfig: StreamSessionConfig,
              deviceSelector: DeviceSelector)

  // Properties
  var state: StreamSessionState { get }
  var streamSessionConfig: StreamSessionConfig { get }
  var statePublisher: /* Publisher<StreamSessionState> */ { get }
  var videoFramePublisher: /* Publisher<VideoFrame> */ { get }
  var photoDataPublisher: /* Publisher<PhotoData> */ { get }
  var errorPublisher: /* Publisher<StreamSessionError> */ { get }

  // Methods
  public func start()
  public func stop()
  public func capturePhoto(format: PhotoCaptureFormat) -> Bool
}
```

New sessions start in `.stopped` state.

### 5.2 StreamSessionConfig

Configuration for a streaming session:

```swift
struct StreamSessionConfig {

  var videoCodec: VideoCodec
  var resolution: StreamingResolution
  var frameRate: UInt

  public init(videoCodec: VideoCodec,
              resolution: StreamingResolution,
              frameRate: UInt)

  public init() // defaults: raw codec, medium resolution, 30 FPS
}
```

### 5.3 StreamingResolution

Valid 9:16 resolutions:

```swift
enum StreamingResolution: CaseIterable {
  case high   // 720 x 1280
  case medium // ~504 x 896
  case low    // 360 x 640

  var videoFrameSize: VideoFrameSize { get }
}
```

Internally the SDK may use additional steps (576x1024, 540x960, 432x768) for laddering.

### 5.4 VideoCodec

```swift
enum VideoCodec {
  case raw // raw decompressed frames
}
```

Currently only `.raw` is exposed.

### 5.5 StreamSessionState

```swift
enum StreamSessionState {
  case stopping
  case stopped
  case waitingForDevice
  case starting
  case streaming
  case paused
}
```

Transitions:

- `.stopped → .waitingForDevice` (no compatible device at `start()` time).
- `.stopped → .starting → .streaming` (device available).
- Any → `.stopping → .stopped` (on `stop()` or error).
- `.streaming ↔ .paused` (device / system pauses and resumes session).

### 5.6 StreamSessionError

```swift
enum StreamSessionError: Error, Equatable {
  case internalError
  case deviceNotFound(DeviceIdentifier)
  case deviceNotConnected(DeviceIdentifier)
  case timeout
  case videoStreamingError
  case audioStreamingError
  case permissionDenied
}
```

Emitted via `errorPublisher` on various failures (connection, permissions, timeouts, internal issues).

**User-Friendly Error Mapping Example:**

```swift
private func formatStreamingError(_ error: StreamSessionError) -> String {
  switch error {
  case .internalError:
    return "An internal error occurred. Please try again."
  case .deviceNotFound:
    return "Device not found. Please ensure your device is connected."
  case .deviceNotConnected:
    return "Device not connected. Please check your connection and try again."
  case .timeout:
    return "The operation timed out. Please try again."
  case .videoStreamingError:
    return "Video streaming failed. Please try again."
  case .audioStreamingError:
    return "Audio streaming failed. Please try again."
  case .permissionDenied:
    return "Camera permission denied. Please grant permission in Settings."
  @unknown default:
    return "An unknown streaming error occurred."
  }
}
```

### 5.7 VideoFrame & VideoFrameSize

```swift
struct VideoFrame: Sendable {
  var sampleBuffer: CMSampleBuffer { get } // read-only

  /// Converts to UIImage safely.
  public func makeUIImage() -> sending UIImage?
}

struct VideoFrameSize {
  public init(width: UInt, height: UInt)

  var width: UInt
  var height: UInt
}
```

**Important**: `sampleBuffer` is shared; do **not** mutate attachments, timing, or pixel buffer. Use `makeUIImage()` for safe conversion.

### 5.8 PhotoCaptureFormat & PhotoData

```swift
enum PhotoCaptureFormat: Sendable {
  case heic
  case jpeg
}

struct PhotoData: Sendable {
  public init(data: Data, format: PhotoCaptureFormat)

  let data: Data
  let format: PhotoCaptureFormat
}
```

Usage with `StreamSession`:

- Call `capturePhoto(format:)` while streaming.
- If it returns `true`, subscribe to `photoDataPublisher` for `PhotoData`.
- Streaming pauses during capture and resumes automatically after.

### 5.9 Example: start stream & display frames

```swift
let wearables = Wearables.shared
let deviceSelector = AutoDeviceSelector(wearables: wearables)

let config = StreamSessionConfig(
  videoCodec: .raw,
  resolution: .low,
  frameRate: 24
)

let session = StreamSession(
  streamSessionConfig: config,
  deviceSelector: deviceSelector
)

// Track loading state
var hasReceivedFirstFrame = false

let stateToken = session.statePublisher.listen { state in
  Task { @MainActor in
    // Update UI (e.g. show streaming / paused / stopped)
  }
}

let frameToken = session.videoFramePublisher.listen { frame in
  guard let image = frame.makeUIImage() else { return }
  Task { @MainActor in
    // Display the image in a preview surface
    // Set hasReceivedFirstFrame = true
  }
}

Task {
  await session.start()
}
```

### 5.10 Example: photo capture

```swift
let accepted = session.capturePhoto(format: .jpeg)

if !accepted {
  // No active session, capture already in progress, or capture failed
}

let photoToken = session.photoDataPublisher.listen { photoData in
  let data = photoData.data
  if let image = UIImage(data: data) {
    // Save, share, or process the image
  }
}
```

### 5.11 Bandwidth & quality laddering

- Streaming uses **Bluetooth Classic**.
- SDK automatically manages bandwidth:
  - First lowers resolution (High → Medium → Low).
  - Then reduces frame rate (e.g. 30 → 24 → 15 FPS, never below 15).
- Effective image quality may be lower than configured resolution because of adaptive per-frame compression.
- Requesting lower resolution and/or frame rate can yield better perceived quality with fewer artifacts.

---

## 6. Audio: microphones & speakers

### 6.1 Bluetooth profiles

Glasses audio uses:

- **A2DP** — high-quality, output-only audio (media).
- **HFP** — two-way audio (mic + speaker) for voice interactions.

DAT shares mic/speaker access with the system Bluetooth stack; we control it via `AVAudioSession`.

### 6.2 AVAudioSession configuration (iOS)

Typical setup for HFP:

```swift
let audioSession = AVAudioSession.sharedInstance()
try audioSession.setCategory(.playAndRecord,
                             mode: .default,
                             options: [.allowBluetooth])
try audioSession.setActive(true,
                           options: .notifyOthersOnDeactivation)
```

Notes:

- Use `.playAndRecord` category for input + output.
- Combine with `.allowBluetooth` and/or `.allowBluetoothA2DP` as appropriate.
- Listen for `AVAudioSession.routeChangeNotification` to handle device connect/disconnect.

### 6.3 Coordinating audio with streaming

When using HFP and streaming at the same time:

- Ensure HFP audio session is fully configured **before** starting a `StreamSession` that uses audio.
- Avoid fixed-sleep hacks in real code; prefer state-based detection (route change events).

Pseudo:

```swift
func startStreamSessionWithAudio() async {
  // 1. Set up HFP
  startAudioSession()

  // 2. Wait for audio route to be ready (replace with real logic)
  try? await Task.sleep(nanoseconds: 2 * NSEC_PER_SEC)

  // 3. Start streaming
  await streamSession.start()
}
```

### 6.4 Practical tips

- For voice-centric features, favor stable HFP routing; keep video resolution/FPS conservative.
- For visual-centric features, be aware that HFP can reduce audio quality; test UX carefully.
- Log route changes and `StreamSessionError.audioStreamingError` for diagnostics.

---

## 7. Session lifecycle & device behavior

### 7.1 Sessions vs transactions

- **Device sessions** (what we use via `StreamSession`): long-lived, sensor/output access.
- **Transactions**: short, system-owned interactions (notifications, “Hey Meta” triggers) – not controlled by us.

### 7.2 Device-driven state changes

Session state can change when:

- User opens another experience via system gesture.
- Another app or system component starts a device session.
- Glasses are removed, folded, or moved out of range → Bluetooth disconnect.
- User removes our app from Meta AI companion app.
- Meta AI ↔ glasses connectivity drops.

We **must not** guess the cause; just react to state (`StreamSessionState`, device link state, availability).

### 7.3 Device availability

- Hinge position and wear detection affect connectivity but are not exposed directly.
- Key observable effects:
  - Closing hinges → Bluetooth disconnect, active streams stop, sessions go to `.stopped`.
  - Opening hinges → Bluetooth may reconnect, but **sessions do not auto-restart**. We must explicitly start sessions once device is available and user initiates.

### 7.4 Lifecycle checklist

- Subscribe to **registrationStateStream** and **devicesStream**.
- Before starting streaming:
  - Verify **registered**.
  - Verify camera permission **granted**.
  - Verify at least one compatible device **available**.
- React to `StreamSessionState`:
  - `.streaming`: show active UI; process frames.
  - `.paused`: keep connection; pause heavy processing; show paused UI.
  - `.stopped`: release resources; show idle UI; allow restart.
- Handle `StreamSessionError` values (especially `.permissionDenied`, `.deviceNotConnected`, `.deviceNotFound`, `.timeout`).

---

## 8. Recommended architecture (for agents & humans)

Use an **agent-based modular architecture** so both humans and AI tools can work safely.

### 8.1 Modules / agents

1. **WearablesBootstrapAgent**
   - Calls `Wearables.configure()` at startup.
   - Checks environment (Info.plist, MetaAppID).
   - Sets global flags (dev mode vs production).

2. **RegistrationAgent**
   - Wraps `startRegistration()`, `startUnregistration()`.
   - Handles callback URL → `handleWearablesCallback(url:)`.
   - Exposes a simplified `RegistrationState` to rest of the app.

3. **DeviceAgent**
   - Wraps `devicesStream()`.
   - Maintains list of `Device` objects and selection logic.
   - Creates `AutoDeviceSelector` and `SpecificDeviceSelector` as needed.

4. **PermissionsAgent**
   - Wraps camera permission APIs.
   - Exposes computed permission state and helper methods:
     - `ensureCameraPermission()` → `Bool`
     - `requireCameraPermission { ... }`

5. **SessionAgent**
   - Owns `StreamSession` instances.
   - Manages `StreamSessionConfig`.
   - Subscribes to `statePublisher` and `errorPublisher`.
   - Provides high-level states: `.idle`, `.starting`, `.running`, `.paused`, `.stopping`, `.error`.

6. **CameraAgent**
   - Subscribes to `videoFramePublisher` and `photoDataPublisher`.
   - Handles conversion to `UIImage` or CV/ML inputs.
   - Encapsulates capture flows (trigger, save, share).

7. **AudioAgent**
   - Configures `AVAudioSession`.
   - Manages Bluetooth routing, route changes.
   - Provides high-level methods for starting/stopping audio use.

8. **UIAgent**
   - Connects agents to SwiftUI / UIKit views.
   - Owns view models modeling the whole lifecycle:
     - Not registered → needs registration
     - Registered but no permission
     - Permission granted but no device
     - Device ready, session not started
     - Session active / paused / stopped / error

9. **MockAgent**
   - Uses Mock Device Kit when available.
   - Simulates devices, streams, errors.
   - Enables automated tests and offline dev.

### 8.2 Coding guidelines

- Keep direct SDK calls inside agents; UI interacts with our abstractions only.
- Prefer async/await and structured concurrency over nested callbacks.
- Treat publishers / async streams as authoritative sources of truth.
- Always handle `StreamSessionError` & permission failures gracefully.
- Log key events (state changes, errors, route changes) for telemetry & debugging.

### 8.3 Mock Device Implementation (Debug)

For development without physical devices, `MWDATMockDevice` allows simulating connection, state, and media.

**Setup in Debug Build:**

```swift
#if DEBUG
import MWDATMockDevice

class MockManager {
    let mockDeviceKit = MockDeviceKit.shared
    var mockDevice: MockDevice?

    func pairMockDevice() {
        // Creates and pairs a simulated Ray-Ban Meta device
        let device = mockDeviceKit.pairRaybanMeta()
        self.mockDevice = device
        
        // Simulate physical state to allow streaming
        device.powerOn()
        device.unfold()
        device.don() // "Put on" the glasses
    }
    
    func setupMockMedia(videoURL: URL, imageURL: URL) async {
        guard let cameraKit = (mockDevice as? MockDisplaylessGlasses)?.getCameraKit() else { return }
        
        // Feed a local video file as the camera stream
        await cameraKit.setCameraFeed(fileURL: videoURL)
        
        // Set the image to be returned when capturePhoto() is called
        await cameraKit.setCapturedImage(fileURL: imageURL)
    }
}
#endif
```

**Mock Controls:**
- `powerOn() / powerOff()`
- `fold() / unfold()`: Closing hinges stops the stream.
- `don() / doff()`: Wearing state affects auto-pause behavior.

### 8.4 Common UI Patterns

**Video Rendering in SwiftUI:**

```swift
struct StreamView: View {
    let videoFrame: UIImage
    
    var body: some View {
        GeometryReader { geometry in
            Image(uiImage: videoFrame)
                .resizable()
                .aspectRatio(contentMode: .fill)
                .frame(width: geometry.size.width, height: geometry.size.height)
                .clipped()
        }
        .edgesIgnoringSafeArea(.all)
    }
}
```

**Loading State:**
Use a boolean flag `hasReceivedFirstFrame` (set to true on the first video frame emission) to toggle between a loading spinner and the video view. This prevents showing a black screen while the connection initializes.

---

## 9. Edge cases & pitfalls

- **Permission denial loop**: if user keeps denying camera access, avoid spamming prompts; instead surface instructions to enable permissions in Meta AI.
- **Multiple devices**: don’t assume only one; ensure selectors and UI handle multiple glasses.
- **Session conflicts**: only one session per device; handle failures when another app already has a session.
- **Bluetooth instability**: expect transient disconnects; implement reconnection / recovery strategies.
- **Capture without streaming**: `capturePhoto` returns `false` if no active session or capture in progress.
- **Mutating `sampleBuffer`**: never mutate `VideoFrame.sampleBuffer`.
- **Quality expectations**: communicate potential quality degradation due to bandwidth to stakeholders/UX.

---

## 10. Maintenance rules for AGENTS.MD

Whenever we:

- Upgrade the DAT SDK version.
- Add new capabilities (codecs, resolutions, devices).
- Change our architecture around sessions, permissions, audio.

We must:

1. Update all relevant API signatures & enums **here**.
2. Add sections for any new major features.
3. Mark removed/changed APIs as deprecated and document migration paths.

This keeps AI agents and human developers aligned on the current integration contract.