CI Integration

Score MCP tool definitions on every pull request. Catch quality regressions before they ship.

Quick start

Add this workflow to your repository. It runs on every PR that modifies MCP tool definitions.

name: ToolRank Score
on:
  pull_request:
    paths:
      - '**/mcp.json'
      - '**/tools.json'
      - '**/server.json'

jobs:
  score:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Score tool definitions
        run: |
          pip install httpx
          curl -sL https://raw.githubusercontent.com/imhiroki/toolrank/v1.0.0/packages/scoring/toolrank_score.py -o toolrank_score.py
          python3 -c "
          import json, sys
          from toolrank_score import score_server, to_json
          with open('your-tools.json') as f:
              tools = json.load(f).get('tools', [])
          result = to_json(score_server('ci-check', tools))
          print(f'ToolRank Score: {result["average_score"]}/100')
          for t in result['tools']:
              for i in t['issues']:
                  print(f'  [{i["severity"]}] {t["name"]}: {i["message"]}')
          if result['average_score'] < 70:
              print('::error::Score below 70')
              sys.exit(1)
          "

Configuration

Minimum score: Change 70 to your desired threshold. We recommend 70 (Preferred level) as a starting point.

Tool file path: Replace your-tools.json with the path to your MCP tool definition file.

Fail on low score: Remove sys.exit(1) to make the check informational only (no PR blocking).

Version pinning: The example pins to v1.0.0. Replace with a specific tag or commit SHA for reproducible builds. Using main will always pull the latest scoring logic, which may change.

Score levels

Level	Score	CI recommendation
Dominant	85-100	Ship with confidence
Preferred	70-84	Good baseline. Recommended minimum.
Selectable	50-69	Needs improvement before shipping
Below 50	0-49	Block merge. Tools are invisible to agents.

Trust Gates v2

Go beyond scoring — gate deployments based on real trust signals. Three stages, each checking progressively deeper quality criteria.

Stage 1: Pre-merge

Runs on every PR. Blocks merge if spec quality is below threshold.

mode: pre-merge · checks: spec score ≥ min-score

Stage 2: Pre-release

Runs before release tags. Verifies agents actually choose this tool.

mode: pre-release · checks: spec score + selection win rate ≥ min-selection-rate

Stage 3: Post-deploy

Runs after deployment. Ensures trust tier hasn't expired or regressed.

mode: post-deploy · checks: trust level ≥ 1

Pre-release gate example:

name: ToolRank Pre-Release Gate
on:
  push:
    tags: ['v*']

jobs:
  trust-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: imhiroki/toolrank-action@v2
        with:
          mode: pre-release
          min-score: 80
          min-selection-rate: 60
          server-id: ${{ secrets.TOOLRANK_SERVER_ID }}

Trust API

Trust Status: GET mcp.toolrank.dev/api/trust-status/SERVER_ID — full trust profile for a server

Trusted List: GET mcp.toolrank.dev/api/trusted-list?min_trust=2 — verified servers for registries

Trust Badge: GET mcp.toolrank.dev/api/trust-badge/SERVER_ID.svg — embeddable trust tier badge

Other integrations

MCP Server: npx @toolrank/mcp-server — score from Claude Code, Cursor, or any MCP client

Python: from toolrank_score import score_server — embed scoring in your own scripts

HTTP API: POST https://mcp.toolrank.dev/mcp — JSON-RPC endpoint for remote scoring

Scoring engine: open source · Score version: v2.0.0 · Governance