MCP Server Management

This guide covers how to create, configure, and manage MCP (Model Context Protocol) servers in muster.

Overview

MCP servers provide structured access to tools and resources for AI assistants. muster supports three types of MCP servers:

• Stdio servers: Execute as local processes with configurable command lines
• Streamable HTTP servers: Connect to external MCP servers via HTTP
• SSE servers: Connect to external MCP servers via Server-Sent Events

Goal

Add a new MCP server to extend muster's tool capabilities.

Prerequisites

• muster control plane running
• MCP server binary available
• Understanding of the tool's requirements

Creating MCP Servers

Stdio Command Servers

Create a stdio server that runs as a local process:

1. Create MCP server configuration

# example-server.yaml
apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: example-tool
  namespace: default
spec:
  type: localCommand
  command: ["mcp-example-tool"]
  autoStart: true
  env:
    TOOL_CONFIG: "/path/to/config"
    LOG_LEVEL: "info"
  description: "Example MCP server providing custom tools"

1. Register the server

muster create mcpserver example-server.yaml

1. Verify server status

muster get mcpserver example-tool

1. Test tool availability

muster agent --repl
# In REPL:
list tools
# Or filter tools to see ones from this server:
filter tools example

Verification

• Server shows status "running"
• Tools from the server appear in tool listings
• Tools can be executed successfully

Configure Auto-Start Behavior

Goal

Control when MCP servers start automatically.

Steps

1. Enable auto-start (start with muster)

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: auto-start-server
  namespace: default
spec:
  autoStart: true
  # ... other configuration

1. Disable auto-start (manual control)

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: manual-server
  namespace: default
spec:
  autoStart: false
  # ... other configuration

1. Apply configuration

muster create mcpserver server-config.yaml

1. Manual server control

# Check server status
muster get mcpserver example-tool

# List all servers
muster list mcpserver

# Check server availability
muster check mcpserver example-tool

Monitor MCP Server Health

Goal

Set up monitoring and health checks for MCP servers.

Steps

1. Check server status

# List all servers with status
muster list mcpserver

# Get detailed server info
muster get mcpserver example-tool

1. Test server communication

# Check if server is available
muster check mcpserver example-tool

1. Set up health monitoring

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: monitored-server
  namespace: default
spec:
  type: localCommand
  command: ["mcp-example-tool"]
  healthCheck:
    enabled: true
    interval: "30s"
    timeout: "10s"
    command: ["health-check"]

1. Configure alerting (if monitoring system available)

muster exports logs, traces, and metrics via OpenTelemetry (OTLP). Point the standard OTEL_EXPORTER_OTLP_* environment variables at your collector and build alerts there; there is no muster metrics command.

Troubleshoot Server Startup Issues

Goal

Diagnose and fix common MCP server startup problems.

Advanced Configuration

Environment Variables

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: filesystem-tools
spec:
  description: "File system operations"
  toolPrefix: "fs"
  type: stdio
  autoStart: true
  command: "npx"
  args: ["@modelcontextprotocol/server-filesystem", "/workspace"]
  env:
    DEBUG: "1"
    LOG_LEVEL: "info"

Remote Servers

Connect to external MCP servers:

Streamable HTTP Transport

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: remote-api
spec:
  description: "Remote API tools"
  toolPrefix: "api"
  type: streamable-http
  url: "https://api.example.com/mcp"
  timeout: 60
  headers:
    Authorization: "Bearer your-token-here"

Server-Sent Events (SSE) Transport

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: sse-server
spec:
  description: "SSE MCP server"
  toolPrefix: "sse"
  type: sse
  url: "https://sse.example.com/mcp"
  timeout: 90
  headers:
    Authorization: "Bearer your-token-here"

SSO Authentication

muster supports Single Sign-On (SSO) for MCP servers, allowing users to authenticate once and access multiple servers without separate authentication flows.

SSO Mechanisms

muster supports two SSO mechanisms:

Mechanism	What You Do	What Happens	Configuration
Token Forwarding	Authenticate once to muster	muster forwards its ID token to downstream servers	auth.forwardToken: true
Token Exchange	Authenticate once to muster	muster exchanges its token for one valid on the remote IdP	auth.tokenExchange config

Token Forwarding (Recommended for Trusted Servers)

When Token Forwarding is enabled, muster forwards its ID token to the downstream MCP server. This provides seamless SSO without requiring users to authenticate to each server individually.

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: mcp-kubernetes
spec:
  description: "Kubernetes MCP with SSO"
  toolPrefix: "k8s"
  type: streamable-http
  url: "https://mcp-kubernetes.example.com/mcp"
  auth:
    forwardToken: true  # Enable SSO via token forwarding
    # Specify audiences required by downstream server (e.g., for Kubernetes OIDC)
    requiredAudiences:
      - "dex-k8s-authenticator"

How it works: 1. User runs muster auth login to authenticate to muster 2. muster requests tokens with all requiredAudiences from the IdP via cross-client scopes 3. On first MCP request, muster proactively connects to all SSO-enabled servers using the multi-audience token 4. User can immediately access SSO servers without additional authentication 5. The CLI shows the SSO type for each server: mcp-kubernetes Connected [SSO: Forwarded]

Requirements: - The downstream MCP server must trust muster's OAuth client ID - Both muster and the downstream server must use the same identity provider (issuer) - For Kubernetes OIDC auth, the IdP must support cross-client authentication (audience:server:client_id:* scopes)

Important: Required audiences are collected at muster startup and during user authentication. If you add or modify MCPServers with requiredAudiences after users have authenticated, those users must re-authenticate (muster auth logout followed by muster auth login) to obtain tokens with the new audiences.

Security Note: Access control for requiredAudiences is enforced at two levels: 1. Kubernetes RBAC: Only users with permissions to create/modify MCPServer CRDs can configure requiredAudiences 2. IdP Cross-Client Configuration: The identity provider (e.g., Dex) must be configured to allow cross-client authentication for the specified audiences. Unauthorized audience requests will be rejected by the IdP.

Token Exchange (Cross-Cluster SSO)

When clusters have separate Identity Providers, muster can use RFC 8693 Token Exchange to obtain a token valid on the remote cluster's IdP. This enables cross-cluster SSO without requiring shared trust.

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: remote-cluster-mcp
spec:
  description: "MCP on remote cluster with Token Exchange"
  type: streamable-http
  url: "https://mcp.remote-cluster.example.com/mcp"
  auth:
    tokenExchange:
      enabled: true
      tokenEndpoint: "https://dex.remote-cluster.example.com/token"
      audience: "mcp-server"

How it works: 1. User authenticates to muster 2. When accessing the remote server, muster exchanges its token at the remote IdP 3. Remote IdP validates the token and issues a new one valid for that cluster 4. muster uses the exchanged token for downstream requests

Checking SSO Status

Use muster auth status to see which servers are using SSO:

$ muster auth status

muster: authenticated
  Endpoint: https://muster.example.com
  Expires:  in 23 hours

MCP Servers:
  mcp-kubernetes  Connected [SSO: Forwarded]
  remote-cluster  Connected [SSO: Exchanged]
  isolated-server Not authenticated          Run: muster auth login --server isolated-server

Troubleshooting SSO

SSO server not connecting automatically: - Verify forwardToken: true is set in the MCPServer spec - Check that the downstream server trusts muster's OAuth client ID - Run with --debug to see detailed SSO connection logs

AWS SigV4 Request Signing

An AWS-hosted MCP server accepts no bearer token. It authenticates each request by an AWS Signature Version 4 signature. Set auth.type: sigv4 to make muster sign every request that it sends to such a server.

This is not SSO. The signature carries muster's own machine identity, not the identity of the user who made the call, so all users of the server share one AWS identity. CloudTrail records muster, not a named engineer.

auth.sigv4 is only valid with type: streamable-http. muster rejects it together with forwardToken, tokenExchange or authorizationServer, because none of them apply to a machine identity. These rules hold in both Kubernetes and filesystem mode.

A connection failure reports itself as such. A 401 from a SigV4 server is not "Auth Required" — there is no login flow to send a user to — so muster keeps the server in Failed and retries with backoff. Check the signing region, the assumed role and its policy.

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: aws-root
spec:
  type: streamable-http
  url: "https://aws-mcp.eu-central-1.api.aws/mcp"
  timeout: 120
  auth:
    type: sigv4
    sigv4:
      region: eu-central-1
  meta:
    AWS_REGION: eu-central-1

Fields:

Field	Required	Purpose
auth.sigv4.region	yes	The signing region. It must match the region in url, because the endpoint checks the credential scope of the signature.
auth.sigv4.service	no	The signing service name. It defaults to the first hostname label of url, so aws-mcp.eu-central-1.api.aws signs as aws-mcp.
auth.sigv4.roleArn	no	An IAM role that muster assumes before it signs. Leave it empty to sign as muster's own identity.
meta	see below	Entries merged into params._meta of every request that carries params. It is not a SigV4 field: see Request metadata.

muster gets its base credentials from the default AWS credential chain. In Kubernetes that means IRSA: the pod identity webhook injects AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE, and the chain exchanges the projected token for credentials. Set roleArn to chain one more hop from there, which is how one muster reaches many accounts. Each account gets its own MCPServer, and a shared family puts them all behind one tool name with an account selector.

Set meta for the AWS-hosted server. It takes the region that it operates in from params._meta.AWS_REGION. This region is a different value from the signing region, even when the two strings match: the signing region belongs to the endpoint, and the operating region belongs to the resources that the call reads. A value that a caller puts in _meta itself wins over the one in meta.

Set it even though calls succeed without it. The backend falls back to its own region rather than failing, so a missing entry does not raise an error — it returns a correct-looking answer about the wrong region. Measured against aws-mcp.eu-central-1.api.aws: the same query for CloudWatch log groups returned nothing with no meta, and the one log group that exists in the account with meta: {AWS_REGION: eu-north-1}. An agent reads the first result as "there are none".

Request metadata

meta is a remote-server field, not a SigV4 field. muster merges its entries into the params._meta object of every outbound JSON-RPC request that carries params. Use it for a backend that reads call-scoped configuration from the MCP metadata field instead of from tool arguments.

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: regional-tools
spec:
  type: streamable-http
  url: "https://mcp.example.com/mcp"
  meta:
    AWS_REGION: eu-central-1

Rules:

• The merge applies to type: streamable-http and type: sse, with any auth type and with none. A server that needs a login keeps its entries after the login: the per-session connection carries them too.
• An entry that the request already has in _meta wins, so a caller can override one per call.
• A request without params is left untouched. initialize does carry params, so the handshake gets the entries as well.
• type: stdio rejects the field. A stdio server speaks over a pipe, so no HTTP transport can inject the entries, and muster refuses the definition instead of accepting the map and dropping it.

Using the CLI

Creating Servers via CLI

Create a stdio server:

muster create mcpserver filesystem-tools \
  --type stdio \
  --command "npx" \
  --args "@modelcontextprotocol/server-filesystem,/workspace" \
  --auto-start \
  --tool-prefix fs \
  --description "File system operations"

Create a streamable HTTP server:

muster create mcpserver remote-api \
  --type streamable-http \
  --url "https://api.example.com/mcp" \
  --timeout 60 \
  --tool-prefix api \
  --description "Remote API tools"

Create an SSE server:

muster create mcpserver sse-server \
  --type sse \
  --url "https://sse.example.com/mcp" \
  --timeout 90 \
  --tool-prefix sse \
  --description "SSE MCP server"

Listing Servers

muster list mcpserver

Getting Server Details

muster get mcpserver filesystem-tools

Updating Servers

# Update stdio server
muster update mcpserver filesystem-tools \
  --auto-start=false \
  --description "Updated file system tools"

# Update remote server
muster update mcpserver remote-api \
  --url "https://new-api.example.com/mcp" \
  --timeout 120

Deleting Servers

muster delete mcpserver filesystem-tools

Configuration Best Practices

Stdio Servers

• Use absolute paths for commands when possible
• Set appropriate environment variables for configuration
• Enable auto-start for critical servers
• Use descriptive tool prefixes to avoid conflicts

Remote Servers (Streamable HTTP and SSE)

• Use HTTPS endpoints when possible for security
• Set appropriate timeouts based on server response times
• Test connectivity before deploying to production
• Monitor server availability and health
• Include necessary authentication headers

Tool Prefixes

• Use short but descriptive prefixes (e.g., k8s, git, fs)
• Avoid generic prefixes like tools or server
• Be consistent across related servers

spec:
  toolPrefix: "custom"  # Tools will be prefixed as "x_custom_*"

Troubleshooting

Stdio Server Issues

Command not found:

# Check if the command is available
which npx
npm install -g @modelcontextprotocol/server-filesystem

# Verify the server definition
muster get mcpserver filesystem-tools

Permission errors:

# Check file permissions
ls -la /workspace
chmod +x /path/to/mcp-server

# Run with appropriate user
sudo -u mcpuser muster start mcpserver filesystem-tools

Remote Server Issues

Connection timeouts:

# Test connectivity
curl -v https://api.example.com/mcp

# Increase timeout
muster update mcpserver remote-api --timeout 120

Transport errors:

# Check server type and endpoint
muster get mcpserver remote-api

# For SSE servers, ensure endpoint supports Server-Sent Events
# For HTTP servers, ensure endpoint supports streaming HTTP

Authentication errors:

# Update headers for authentication
muster update mcpserver remote-api \
  --header "Authorization=Bearer new-token"

Upstream 5xx and reconnect backoff:

A remote server whose connection attempt fails for a transient reason (connection refused, DNS, timeout, HTTP 5xx) is retried automatically. The wait between attempts starts at 30 s, doubles on every consecutive failure and is capped at 2 minutes; from the third failure on the server is reported as Failed (unreachable in core_service_status). The orchestrator checks every 30 s whether an attempt is due, so a server whose upstream has recovered is back within the cap plus one tick.

# Why did the last attempt fail and when is the next one?
kubectl get mcpserver remote-api -o jsonpath='{.status.lastError}{"\n"}{.status.lastFailureHTTPStatus}{"\n"}{.status.nextRetryAfter}{"\n"}'

# The same from the events: HTTP status and scheduled retry per attempt
muster events --resource-type MCPServer --resource-name remote-api

status.lastFailureHTTPStatus is set when the endpoint answered -- 504 from a gateway or tunnel in front of a healthy server, 503 from the server itself -- and absent when nothing answered at all. Three environment variables on the muster process tune the schedule: MUSTER_MCPSERVER_INITIAL_BACKOFF (default 30s), MUSTER_MCPSERVER_MAX_BACKOFF (default 2m) and MUSTER_ORCHESTRATOR_RETRY_INTERVAL (default 30s), each a Go duration.

A connected server is also probed with an MCP ping every MUSTER_ORCHESTRATOR_HEALTH_CHECK_INTERVAL (default 30s, a Go duration), each probe bounded by the server's spec.timeout. Three failed probes in a row close the client and put the server on the schedule above with a reconnect due at once; metadata.consecutiveHealthCheckFailures in core_service_status shows the running count. Servers served per session (forwardToken, tokenExchange, OAuth) have no shared client and are not probed.

Advanced Configuration

Environment Variables for Stdio Servers

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: custom-tools
spec:
  type: stdio
  command: "python"
  args: ["-m", "my_mcp_server"]
  env:
    PYTHONPATH: "/usr/local/lib/python3.9/site-packages"
    API_KEY: "your-api-key"
    DEBUG: "true"
    LOG_LEVEL: "info"

Custom Headers for Remote Servers

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: authenticated-api
spec:
  type: streamable-http
  url: "https://secure-api.example.com/mcp"
  headers:
    Authorization: "Bearer jwt-token-here"
    X-API-Version: "v2"
    Content-Type: "application/json"
  timeout: 45

Monitoring and Health Checks

Check server status:

# List all servers with status
muster list mcpserver

# Get detailed server information
muster get mcpserver <server-name>

# Check if server is available
muster check mcpserver <server-name>

Inspect server state and logs:

# Server status and any startup error
muster get mcpserver <server-name> -o yaml
muster list mcpserver --all --verbose

# Aggregator logs (servers log through it) — there is no `muster logs`
muster serve --debug

Integration Examples

With Cursor/VS Code

Configure Cursor to use muster MCP servers:

{
  "mcpServers": {
    "muster": {
      "command": "muster",
      "args": ["standalone"]
    }
  }
}

With Other AI Assistants

Most MCP-compatible assistants can connect to muster's aggregator endpoint at http://localhost:8090/mcp.

Configuration Best Practices

1. Naming Conventions

• Use descriptive names: git-tools, k8s-cluster-prod
• Include environment in name for multi-env setups
• Avoid special characters and spaces

2. Configuration Management

• Store configurations in version control
• Use environment-specific overlays
• Document required environment variables

3. Monitoring

• Always enable health checks for production
• Set appropriate timeouts
• Monitor resource usage

4. Security

• Run with minimal required permissions
• Use read-only filesystems where possible
• Regularly update server binaries

Related Documentation

• Configuration Reference - Detailed configuration options
• HTTP endpoints - The endpoints the aggregator serves
• CRD Reference - Kubernetes CRD schema
• Architecture - How MCP servers fit into muster
• Server Configuration Schema
• Troubleshooting Guide