evals-mcp-server

Paste it into your assistant. It asks for the manifest, the server wiring and the skills, and for an honest account of what this record does not say. Read that second file first.

Draft an Agent Plugin (agent-plugins.org, specification 1.1.0) that wraps the
MCP server described below, so that somebody could install one thing and have
an assistant that knows when and how to use it.

An Agent Plugin is one installable unit: a `plugin.json` manifest, an
`mcp.json` that wires up the servers it needs, and a `skills/` directory
where each skill is a folder holding a `SKILL.md`. Hand back every file in
full, each under its own path, ready to save.

1. Write `plugin.json` with `$schema` exactly `https://agent-plugins.org/schemas/1.1.0/plugin.schema.json`. The name is
   1 to 64 characters of a-z, 0-9, `-` and `.`, alphanumeric at both ends, with
   no `--` and no `..` in it.

2. Write `mcp.json` wiring THIS server exactly as its record declares it. A
   remote keeps the URL and the transport type as written. A package keeps the
   registry, the identifier and the version as written. Do not invent a command,
   a port, a flag or an argument that is not in the record.

3. Every secret stays an input. No key, token, password or connection string
   belongs in either file. Declare what has to be supplied, name it, and say what
   it is for.

4. Do not invent tools. The record lists the tools it lists, and if it lists
   none then the honest plugin says the tool list was not published rather than
   guessing one from the description.

5. Skills are jobs, not tools. Write one skill per thing somebody would actually
   ask for, and inside each one say when to reach for this server, what a good
   result looks like, and what to do when it comes back empty. A skill per tool
   is a manual page with a different filename.

6. Say where it stops. Name what this plugin will not do — what it has no tool
   for, what needs a person, and what it must not be pointed at. A plugin with no
   stated edge reads as one with no edge.

7. Keep the author's own words for the description. If you would rather say it
   differently, say yours somewhere else and leave theirs where it is.

8. Record which version of the server you wrapped, and where the record came
   from, at the top of `plugin.json`'s description or in the readme. A plugin
   nobody can trace back to a version is one nobody can update.

Produce a second file alongside them, `LIMITS.md`, and treat it as the more
important of the two. The plugin is for whoever installs it. This is for
whoever has to decide whether installing it is a good idea, and that is
usually a different person who will never read the manifest.

It has three parts.

**What this is built from.** One paragraph: whose server it is, what the
record says it does, which version, and the fact that the record is all you
had. Say plainly that nobody ran it.

**What the record does not say.** One entry per gap. Whether the tool list was
published. What the server does with what it reads. What it costs. Whether it
writes anything anywhere. What credentials it will ask for and what those
credentials can reach. An unanswered question stays an unanswered question:
do not fill one in from the description or from what similar servers usually do.

**What a person has to check before trusting it.** The specific things
somebody should verify for themselves, in the order that would stop them
soonest if the answer is bad.

Write it in plain English, and do not soften it. A plugin drafted from a
directory record is a starting point to argue with, not something to install
into anything that matters.

The server record follows, exactly as the public index holds it. It is
everything I have: nobody has run this server, called a tool on it, or checked
that the address answers.

```json
{
  "server": {
    "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
    "name": "io.github.cyanheads/evals-mcp-server",
    "description": "Author verifiable eval records through a draft→review→revise→submit loop with enforced graders.",
    "repository": {
      "url": "https://github.com/cyanheads/evals-mcp-server",
      "source": "github"
    },
    "version": "0.1.3",
    "packages": [
      {
        "registryType": "npm",
        "registryBaseUrl": "https://registry.npmjs.org",
        "identifier": "@cyanheads/evals-mcp-server",
        "version": "0.1.3",
        "runtimeHint": "node",
        "transport": {
          "type": "stdio"
        },
        "packageArguments": [
          {
            "value": "run",
            "type": "positional"
          },
          {
            "value": "start:stdio",
            "type": "positional"
          }
        ],
        "environmentVariables": [
          {
            "description": "Root folder for record JSON. The store manages drafts/, submitted/, and exports/ subdirs under it.",
            "format": "string",
            "default": "./evals-data",
            "name": "EVALS_DATA_DIR"
          },
          {
            "description": "When 'true', evals_submit_draft requests human confirmation before finalizing.",
            "format": "string",
            "default": "false",
            "name": "EVALS_REQUIRE_CONFIRMATION"
          },
          {
            "description": "Default metadata.license applied when a draft omits one.",
            "format": "string",
            "name": "EVALS_DEFAULT_LICENSE"
          },
          {
            "description": "Directory of framework-written tool-call captures; when set, captures EvalsIDs resolve to full dumps.",
            "format": "string",
            "name": "EVALS_CAPTURE_DIR"
          },
          {
            "description": "Sets the minimum log level for output (e.g., 'debug', 'info', 'warn').",
            "format": "string",
            "default": "info",
            "name": "MCP_LOG_LEVEL"
          }
        ]
      },
      {
        "registryType": "npm",
        "registryBaseUrl": "https://registry.npmjs.org",
        "identifier": "@cyanheads/evals-mcp-server",
        "version": "0.1.3",
        "runtimeHint": "node",
        "transport": {
          "type": "streamable-http",
          "url": "http://localhost:3010/mcp"
        },
        "packageArguments": [
          {
            "value": "run",
            "type": "positional"
          },
          {
            "value": "start:http",
            "type": "positional"
          }
        ],
        "environmentVariables": [
          {
            "description": "The hostname for the HTTP server.",
            "format": "string",
            "default": "127.0.0.1",
            "name": "MCP_HTTP_HOST"
          },
          {
            "description": "The port to run the HTTP server on.",
            "format": "string",
            "default": "3010",
            "name": "MCP_HTTP_PORT"
          },
          {
            "description": "The endpoint path for the MCP server.",
            "format": "string",
            "default": "/mcp",
            "name": "MCP_HTTP_ENDPOINT_PATH"
          },
          {
            "description": "Authentication mode to use: 'none', 'jwt', or 'oauth'.",
            "format": "string",
            "default": "none",
            "name": "MCP_AUTH_MODE"
          },
          {
            "description": "Root folder for record JSON. The store manages drafts/, submitted/, and exports/ subdirs under it.",
            "format": "string",
            "default": "./evals-data",
            "name": "EVALS_DATA_DIR"
          },
          {
            "description": "When 'true', evals_submit_draft requests human confirmation before finalizing.",
            "format": "string",
            "default": "false",
            "name": "EVALS_REQUIRE_CONFIRMATION"
          },
          {
            "description": "Default metadata.license applied when a draft omits one.",
            "format": "string",
            "name": "EVALS_DEFAULT_LICENSE"
          },
          {
            "description": "Directory of framework-written tool-call captures; when set, captures EvalsIDs resolve to full dumps.",
            "format": "string",
            "name": "EVALS_CAPTURE_DIR"
          },
          {
            "description": "Sets the minimum log level for output (e.g., 'debug', 'info', 'warn').",
            "format": "string",
            "default": "info",
            "name": "MCP_LOG_LEVEL"
          }
        ]
      }
    ]
  },
  "_meta": {
    "io.modelcontextprotocol.registry/official": {
      "status": "active",
      "statusChangedAt": "2026-08-22T19:46:40.006971Z",
      "publishedAt": "2026-08-22T19:46:40.006971Z",
      "updatedAt": "2026-08-22T19:46:40.006971Z",
      "isLatest": true
    }
  }
}
```

The server is listed as "evals-mcp-server".