AGENTS.md Critique Report

What worked well

• Clear "Do NOT modify" warnings: The explicit callouts for llm() method, public/, docker_context/, and runtime infrastructure files are excellent. These are bolded and marked as CRITICAL, which would prevent a new agent from breaking deployment.

• Concrete code pattern: The implementation pattern section provides actual Python code showing class structure, required naming conventions, and method signatures. This is immediately actionable.

• Command reference with context: Commands are clearly listed with their purpose, and the note to run from project root is helpful. The post-deployment validation example is a nice touch.

• Tool addition workflow: The 3-step process for adding tools is clear and includes the dependency sync step.

• Critical constraints section: Having a dedicated section that reiterates the most dangerous gotchas (llm() method, class name, dependencies) is good redundancy for an AI agent that might skim.

What was unclear or missing

1. What does this agent actually DO?

The opening says "Implements conversational workflows with tool calling" but gives no hint about the domain, use case, or what tools/capabilities it currently has.

Suggested fix: Add after the first paragraph:

This agent currently implements [describe what it does - e.g., "a customer support assistant with database query capabilities" or "a data analysis agent with plotting tools"]. See `agent/myagent.py` for the current tool set and workflow logic.

2. What is create_agent() and where does it come from?

The pattern shows create_agent(self.llm(), tools=self.tools, system_prompt=...) but never explains what this helper is, where it's imported from, or what it returns.

Suggested fix: Add to the implementation pattern section:

# Import from datarobot_genai.langgraph
from datarobot_genai.langgraph import create_agent

# Returns a callable node that combines LLM + tools + system prompt

3. What is MessagesState and StateGraph?

These types appear in the workflow property but are never explained or shown how to import.

Suggested fix: Add imports to the pattern:

from langgraph.graph import StateGraph
from langgraph.graph.message import MessagesState

4. Where do tools go and what should they look like?

"Create tool in agent/ directory" - but as what? A function? A class? A separate file? What's the import pattern?

Suggested fix: Expand the "Adding tools" section:

## Adding tools

1. Create tool in `agent/` directory:
   - Simple tools: add as decorated functions in `myagent.py`
   - Complex tools: create separate file like `agent/my_tool.py`
   - Use `@tool` decorator from `langchain_core.tools`
   
   Example:
   ```python
   from langchain_core.tools import tool
   
   @tool
   def my_tool(query: str) -> str:
       """Tool description for LLM."""
       return result

2. Import and add to MyAgent.tools property:

   @property
   def tools(self) -> list:
       return [my_tool, other_tool]

3. If new packages needed: add to pyproject.toml dependencies, run dr task run agent:install

### 5. **What goes in `agent/config.py`?**
It's listed as a file to modify for "agent configuration" but there's no explanation of what configuration options exist or what format they take.

**Suggested fix**: Add:
```markdown
### `agent/config.py`
Contains configuration dataclasses/constants:
- Model names and parameters
- System prompts or prompt fragments
- Tool-specific settings (API keys loaded from env, timeouts, etc.)
- Workflow behavior flags

6. What test patterns should be followed?

The tests directory is mentioned but there's no guidance on how to write tests for agents, what fixtures exist, or what patterns to follow.

Suggested fix: Add:

## Testing

- Use fixtures from `tests/conftest.py` (see file for available fixtures)
- Test async workflows with `@pytest.mark.asyncio`
- Mock LLM calls to avoid external dependencies
- Example test structure:
  ```python
  async def test_my_feature(agent_instance):
      result = await agent_instance.workflow.ainvoke(...)
      assert result["messages"][-1].content == expected

### 7. **What's the relationship between `dev.py`, `cli.py`, and `custom.py`?**
They're marked "do not modify" but no explanation of what they do or why they exist.

**Suggested fix**: Add to "Do NOT modify" section:
```markdown
- `dev.py` — Chainlit dev server entrypoint
- `cli.py` — Command-line interface for testing deployments
- `custom.py` — DataRobot custom model runtime wrapper

8. What does "deployment" mean in this context?

The doc mentions "deployable to DataRobot" and "Post-deployment validation" but never explains the deployment model or lifecycle.

Suggested fix: Add a "Deployment" section:

## Deployment

This agent deploys to DataRobot as a custom model. The deployment process:
1. Build: `dr task run agent:create-docker-context` generates deployment artifacts
2. Deploy: [via DataRobot UI/API - link to docs]
3. Validate: Use `agent:cli -- execute-deployment` to test deployed endpoint

The `llm()` method handles authentication to DataRobot's LLM Gateway in production.

Commands or instructions that need fixing

1. Missing agent:create-docker-context in commands section

The command is mentioned in "Do NOT modify" and "Critical constraints" but not in the "Commands" section where it should be documented.

Suggested fix: Add to Commands section:

dr task run agent:create-docker-context  # generate deployment artifacts (before deploying)

2. Ambiguous CLI command syntax

The -- <args> pattern is shown but no concrete examples of what args are available.

Suggested fix: Expand the CLI section:

dr task run agent:cli -- execute-deployment --user_prompt "test" --deployment_id <id>
dr task run agent:cli -- --help  # see all available commands

Suggested additions

1. Quick start section

Add at the top, right after "Key directories/files":

## Quick start

First time setup:
```shell
# Install dependencies
dr task run agent:install

# Run tests to verify setup
dr task run agent:test

# Start local dev server
dr task run agent:dev
# Visit http://localhost:8000 to interact with agent

Making changes:

1. Edit agent/myagent.py (workflow, tools, prompts)
2. Run dr task run agent:test to verify
3. Test interactively with dr task run agent:dev
4. Lint before committing: dr task run agent:lint

### 2. **Troubleshooting section**

```markdown
## Troubleshooting

**"Module not found" errors**: Run `dr task run agent:install` after pulling changes

**Tests failing after adding tool**: Ensure tool is imported and added to `MyAgent.tools` property

**Dev server not reloading**: Restart the server; some config changes require full restart

**Deployment auth errors**: Never modify the `llm()` method - it handles DataRobot authentication

3. File modification decision tree

## What file should I modify?

- **Adding/changing agent behavior, tools, or prompts** → `agent/myagent.py`
- **Changing model name, system settings** → `agent/config.py`
- **Adding tests** → `tests/test_*.py`
- **Adding dependencies** → `pyproject.toml` (then run `agent:install`)
- **Everything else** → Probably shouldn't modify; check with team

Overall verdict

NEEDS WORK — The document prevents catastrophic mistakes (good "do not touch" warnings) but lacks enough context for a new agent to work confidently without extensive codebase exploration. Key gaps: missing imports, unclear tool patterns, no explanation of what the agent does, and insufficient testing guidance.