AGENTS.md Critique Report

What worked well

• Clear project purpose: The opening line immediately tells me this is Pulumi IaC for DataRobot resources, and lists the specific resource types (agents, LLM blueprints, etc.)
• Explicit "do not modify" list: Calling out uv.lock, Pulumi.yaml, and .vscode/ prevents common mistakes
• Command reference with context: The commands section shows where to run commands from (project root) and what each does
• Critical constraints section: Auth requirements, plugin versioning, and rate limiting are clearly flagged as gotchas
• Resource naming convention: The {Component} {Type} pattern is explicit with an example
• Feature flag location: Clear that feature_flags/ contains YAML toggles for optional components
• Adding new resources workflow: 4-step process is concrete and actionable

What was unclear or missing

1. What is "deploy-dev" and where is it?

The doc mentions dr task run infra:start # alias for deploy-dev (backend only) and "update patterns when adding resources" in the URN targeting section, but:

• I don't see deploy-dev in the commands list
• The relationship between infra:start and deploy-dev is unclear
• What does "backend only" mean in this context?

Suggested fix: Add to Commands section:

dr task run infra:deploy-dev  # selective deploy (backend resources only, uses URN patterns)

2. What are "sibling dependencies" and where are they?

The doc says "Resources reference outputs from agent/, mcp_server/, fastapi_server/, frontend_web/ builds" but:

• Are these directories at the project root level (siblings to infra/)?
• What commands deploy those first?
• What happens if I run infra:up without deploying them?

Suggested fix: Add to Critical constraints:

- **Sibling dependencies**: This infra depends on build artifacts from sibling directories (`../agent/`, `../mcp_server/`, `../fastapi_server/`, `../frontend_web/`). Run `dr task run build` from project root before deploying, or use `infra:up-yes` which handles dependencies automatically.

3. How do I run tests?

There's a tests/ directory with test_agent.py, but no command for running tests. The "development loop" section mentions infra:lint but not testing.

Suggested fix: Add to Commands section:

dr task run infra:test        # run pytest suite

4. What is dr and how do I get it?

All commands use dr task run but there's no explanation of what dr is or how to install it.

Suggested fix: Add before Commands section:

## Prerequisites

- **DataRobot CLI (`dr`)**: Install via `pip install datarobot-cli` or see [link]
- **Pulumi CLI**: Install from https://www.pulumi.com/docs/get-started/install/
- **Authentication**: Run `dr auth` to configure credentials before any deploy commands

5. What does mcp_server_user_params.py actually do?

The description "wire custom MCP runtime params from mcp_server/user-metadata.yaml" is vague. Is this a mapping? A parser? When would I modify it?

Suggested fix: In "Modify these" section:

- `infra/mcp_server_user_params.py` — parses and transforms MCP runtime parameters from `../mcp_server/user-metadata.yaml` into Pulumi resource inputs. Modify when adding new MCP configuration fields.

6. What's in the "... (1 more)" file in infra/?

The directory listing shows infra/ has one more file not listed. This creates uncertainty.

Suggested fix: Either list all files or explain:

infra/
  __init__.py
  agent.py
  fastapi_server.py
  frontend_web.py
  libllm.py
  llm.py
  mcp_server.py
  mcp_server_user_params.py
  utils.py  # (or whatever it is)

7. What are "URN patterns" and how do I update them?

The doc says "update patterns when adding resources" but doesn't explain what a URN pattern looks like or where in Taskfile.yaml to add them.

Suggested fix: Add to "Adding new resources" section:

3. If selective deploy needed, add URN pattern to `deploy-dev` task in `Taskfile.yaml`:
   ```yaml
   # Example: to target only agent resources
   - pulumi up --target "urn:pulumi:*::*::datarobot:*:Agent::*"

### 8. **What's the difference between `configurations/` patterns?**
Five different LLM configuration files are listed but no guidance on when to use which.

**Suggested fix**: Add to Conventions section:
```markdown
- **Configuration patterns**: 
  - `llm/blueprint_with_external_llm.py` — use when integrating external LLM providers
  - `llm/blueprint_with_llm_gateway.py` — use when routing through DataRobot LLM Gateway
  - `llm/deployed_llm.py` — use for already-deployed LLM references
  - `llm/gateway_direct.py` — use for direct gateway access without blueprints
  - `llm/registered_model.py` — use for custom registered models

Commands or instructions that need fixing

1. infra:init and infra:select syntax is ambiguous

dr task run infra:init -- <stack>   # create new stack
dr task run infra:select -- <stack> # switch stack

The -- separator suggests these are passing arguments to Pulumi, but it's unclear if <stack> is a literal placeholder or if I should replace it.

Suggested fix:

dr task run infra:init -- dev       # create new stack (example: 'dev')
dr task run infra:select -- prod    # switch stack (example: 'prod')

2. Missing stack context

No explanation of what stacks are or why I'd create/switch them.

Suggested fix: Add after Commands section:

## Stacks

Pulumi stacks represent different environments (dev, staging, prod). Each stack maintains separate state. Create a stack before first deploy:

```shell
dr task run infra:init -- dev
dr task run infra:select -- dev
dr task run infra:up

## Suggested additions

### 1. **Troubleshooting section**
```markdown
## Troubleshooting

**"No valid credential providers found"**
→ Run `dr auth` to configure DataRobot credentials

**"Plugin not found: datarobot v0.10.27"**
→ Set `export PULUMI_SKIP_UPDATE_CHECK=1` and re-run `infra:install`

**"Resource not found" errors on deploy**
→ Run `dr task run build` from project root to build sibling dependencies

**Type errors in configurations/**
→ Expected; `configurations/` is excluded from mypy (see pyproject.toml)

2. Quick start section

## Quick Start (First Time Setup)

```shell
# From project root
dr auth                          # configure credentials
dr task run infra:install        # install dependencies
dr task run infra:init -- dev    # create dev stack
dr task run infra:select -- dev  # select dev stack
dr task run build                # build sibling dependencies
dr task run infra:up             # deploy infrastructure

### 3. **File purpose summary table**
```markdown
## File Reference

| File | Purpose | Modify? |
|------|---------|---------|
| `__main__.py` | Pulumi entry point, orchestrates all resources | Yes |
| `infra/agent.py` | DataRobot agent resource definitions | Yes |
| `infra/llm.py` | LLM blueprint and model resources | Yes |
| `infra/libllm.py` | Shared LLM utilities | Yes |
| `infra/mcp_server.py` | MCP server deployment resources | Yes |
| `infra/fastapi_server.py` | FastAPI backend resources | Yes |
| `infra/frontend_web.py` | Frontend deployment resources | Yes |
| `Pulumi.yaml` | Project metadata | No |
| `uv.lock` | Dependency lock file | No |

4. Example of adding a new resource

## Example: Adding a Custom Model Resource

1. Create `infra/custom_model.py`:
   ```python
   import pulumi_datarobot as dr
   
   def create_custom_model():
       return dr.CustomModel("Custom Model Resource",
           name="my-custom-model",
           # ... config
       )

2. Import in __main__.py:

   from infra.custom_model import create_custom_model
   
   custom_model = create_custom_model()
   pulumi.export("custom_model_id", custom_model.id)

3. Add URN pattern to Taskfile.yaml under deploy-dev:

   - --target "urn:pulumi:*::*::datarobot:*:CustomModel::*"

4. Deploy:

   dr task run infra:up

## Overall verdict

**NEEDS WORK** — The document provides good structural orientation and conventions, but has several critical gaps around dependencies, testing, prerequisites, and command context that would force a new agent to grep the codebase or make incorrect assumptions.