AI Integration

Blockstudio ships a pre-built context file designed for LLM coding assistants. The file contains the complete v7 documentation and all JSON schemas in a single blockstudio-llm.txt that any AI tool can consume.

What's included

The context file combines two sources into roughly 48k tokens:

• Documentation: every page from the v7 docs, stripped of formatting and navigation, with code examples preserved.
• JSON schemas: the block, settings, page, and extension schemas. Repeated definitions (like field types shared between blocks and extensions) are deduplicated to save tokens.

How it's built

The file is assembled at build time by npm run build:llm, which runs scripts/build-llm.ts. The script:

1. Walks the docs directory tree using each folder's meta.json to determine page order.
2. Strips MDX frontmatter, JSX components, link URLs, and bold/italic markers from each page.
3. Reads the JSON schemas from includes/schemas/, trims the block schema to only the blockstudio key, and deduplicates field type definitions shared between the block and extension schemas.
4. Writes the combined output to includes/llm/blockstudio-llm.txt.

How to use it

1. Enable the ai/enableContextGeneration setting in your blockstudio.json:

{
  "ai": {
    "enableContextGeneration": true
  }
}

2. The context file is now available at your-site.com/blockstudio-llm.txt.

3. Point your AI tool to that URL:

   • Cursor: add the URL as a doc in your project settings.
   • Claude Code: reference the URL or download the file and add it to your project context.
   • GitHub Copilot: include the file in your repository or reference it in your instructions.

Any tool that accepts a URL or text file as context will work. The file is static and does not include site-specific data like your registered blocks or current settings.

Evals

The Blockstudio repository includes an eval suite for development purposes. It is not part of the plugin itself and is not shipped to users. The suite verifies that LLMs given the context file can generate correct Blockstudio code across all framework domains. It requires the Claude Code CLI to be installed locally.

Each eval case sends a prompt to the CLI with the context file as a system prompt, then runs deterministic scorers against the output (JSON schema validation, PHP syntax checks, structural checks).

Running evals

These commands are run from the repository root during development:

# Run all 81 cases
npm run eval

# Run a specific domain
npm run eval:blocks
npm run eval:templates
npm run eval:pages
npm run eval:patterns
npm run eval:config
npm run eval:extensions
npm run eval:hooks

# Run a single case
npm run eval:blocks -- --case=5

# Use a different model
npm run eval:blocks -- --model=opus

Results print to the console and are written in full to evals/results/.

Domains

Scorers

All scoring is deterministic (no LLM-as-judge):

• JsonParse: all JSON code blocks parse successfully.
• JsonSchema: block.json, page.json, or pattern.json validates against the corresponding JSON schema.
• PhpSyntax: matching PHP tags, correct variable usage ($a/$b), no ACF patterns.
• TemplateVars: expected strings appear in template output.
• HookPresence: correct filter/action names and add_filter calls present.
• Structure checks: field types, field IDs, conditions, switch, storage, extension structure, config keys, pattern properties.

Adding a case

Each domain has an evals/{domain}.eval.ts file that exports an array of cases. A case is a prompt string and an array of scorer functions:

{
  name: "My new case",
  prompt: 'Create a Blockstudio block called "acme/example" with ...',
  scorers: [
    jsonParse(),
    jsonSchema("block"),
    hasFieldTypes("text", "number"),
    hasFieldIds("title", "count"),
  ],
}

Scorer functions are in evals/scorers/. Each takes the raw LLM output string and returns { name, score, details }.