MCP Server Management

This guide covers how to create, configure, and manage MCP (Model Context Protocol) servers in muster.

Overview

MCP servers provide structured access to tools and resources for AI assistants. muster supports three types of MCP servers:

• Stdio servers: Execute as local processes with configurable command lines
• Streamable HTTP servers: Connect to external MCP servers via HTTP
• SSE servers: Connect to external MCP servers via Server-Sent Events

Goal

Add a new MCP server to extend muster's tool capabilities.

Prerequisites

• muster control plane running
• MCP server binary available
• Understanding of the tool's requirements

Creating MCP Servers

Stdio Command Servers

Create a stdio server that runs as a local process:

1. Create MCP server configuration

# example-server.yaml
apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: example-tool
  namespace: default
spec:
  type: localCommand
  command: ["mcp-example-tool"]
  autoStart: true
  env:
    TOOL_CONFIG: "/path/to/config"
    LOG_LEVEL: "info"
  description: "Example MCP server providing custom tools"

1. Register the server

muster create mcpserver example-server.yaml

1. Verify server status

muster get mcpserver example-tool

1. Test tool availability

muster agent --repl
# In REPL:
list tools
# Or filter tools to see ones from this server:
filter tools example

Verification

• Server shows status "running"
• Tools from the server appear in tool listings
• Tools can be executed successfully

Configure Auto-Start Behavior

Goal

Control when MCP servers start automatically.

Steps

1. Enable auto-start (start with muster)

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: auto-start-server
  namespace: default
spec:
  autoStart: true
  # ... other configuration

1. Disable auto-start (manual control)

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: manual-server
  namespace: default
spec:
  autoStart: false
  # ... other configuration

1. Apply configuration

muster create mcpserver server-config.yaml

1. Manual server control

# Check server status
muster get mcpserver example-tool

# List all servers
muster list mcpserver

# Check server availability
muster check mcpserver example-tool

Monitor MCP Server Health

Goal

Set up monitoring and health checks for MCP servers.

Steps

1. Check server status

# List all servers with status
muster list mcpserver

# Get detailed server info
muster get mcpserver example-tool

1. Test server communication

# Check if server is available
muster check mcpserver example-tool

1. Set up health monitoring

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: monitored-server
  namespace: default
spec:
  type: localCommand
  command: ["mcp-example-tool"]
  healthCheck:
    enabled: true
    interval: "30s"
    timeout: "10s"
    command: ["health-check"]

1. Configure alerting (if monitoring system available)

muster exports logs, traces, and metrics via OpenTelemetry (OTLP). Point the standard OTEL_EXPORTER_OTLP_* environment variables at your collector and build alerts there; there is no muster metrics command.

Troubleshoot Server Startup Issues

Goal

Diagnose and fix common MCP server startup problems.

Advanced Configuration

Environment Variables

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: filesystem-tools
spec:
  description: "File system operations"
  toolPrefix: "fs"
  type: stdio
  autoStart: true
  command: "npx"
  args: ["@modelcontextprotocol/server-filesystem", "/workspace"]
  env:
    DEBUG: "1"
    LOG_LEVEL: "info"

Remote Servers

Connect to external MCP servers:

Streamable HTTP Transport

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: remote-api
spec:
  description: "Remote API tools"
  toolPrefix: "api"
  type: streamable-http
  url: "https://api.example.com/mcp"
  timeout: 60
  headersFrom:
    - name: Authorization
      valueFrom:
        type: Secret
        name: remote-api-token
        key: authorization

Server-Sent Events (SSE) Transport

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: sse-server
spec:
  description: "SSE MCP server"
  toolPrefix: "sse"
  type: sse
  url: "https://sse.example.com/mcp"
  timeout: 90
  headersFrom:
    - name: Authorization
      valueFrom:
        type: Secret
        name: sse-server-token
        key: authorization

Credentials come from a Secret

Everyone who can read an MCPServer can read its headers, url and env, so a credential goes in a Secret in the MCPServer's own namespace and the server refers to it in headersFrom. muster reads the value each time it connects; it never shows the value in the MCPServer, in core_service_* or in core_mcpserver_* output.

muster's own connection to the server reads the Secret at the connect only. A rotated value reaches it at the next connect: set restartRequestedAt, or wait for a reconnect. A person's session connection reads the Secret each time it connects.

kubectl create secret generic remote-api-token \
  --namespace <mcpserver-namespace> \
  --from-literal=authorization="Bearer <token>"

muster refuses a server with a value in an Authorization, Proxy-Authorization, Cookie or X-API-Key header, with a user name or password in its url, or with env on a remote server. A server written before this rule reconciles to Failed and names the field; move the value to a Secret. In filesystem mode muster has no Secret reader: there headersFrom fails the connect, and headers may still carry a credential.

SSO Authentication

muster supports Single Sign-On (SSO) for MCP servers, allowing users to authenticate once and access multiple servers without separate authentication flows.

SSO Mechanisms

muster supports two SSO mechanisms:

Mechanism	What You Do	What Happens	Configuration
Token Forwarding	Authenticate once to muster	muster forwards its ID token to downstream servers	auth.forwardToken: true
Token Exchange	Authenticate once to muster	muster exchanges its token for one valid on the remote IdP	auth.tokenExchange config

Token Forwarding (Recommended for Trusted Servers)

When Token Forwarding is enabled, muster forwards its ID token to the downstream MCP server. This provides seamless SSO without requiring users to authenticate to each server individually.

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: mcp-kubernetes
spec:
  description: "Kubernetes MCP with SSO"
  toolPrefix: "k8s"
  type: streamable-http
  url: "https://mcp-kubernetes.example.com/mcp"
  auth:
    forwardToken: true  # Enable SSO via token forwarding
    # Specify audiences required by downstream server (e.g., for Kubernetes OIDC)
    requiredAudiences:
      - "dex-k8s-authenticator"

How it works: 1. User runs muster auth login to authenticate to muster 2. muster requests tokens with all requiredAudiences from the IdP via cross-client scopes 3. On first MCP request, muster proactively connects to all SSO-enabled servers using the multi-audience token 4. User can immediately access SSO servers without additional authentication 5. The CLI shows the SSO type for each server: mcp-kubernetes Connected [SSO: Forwarded]

Requirements: - The downstream MCP server must trust muster's OAuth client ID - Both muster and the downstream server must use the same identity provider (issuer) - For Kubernetes OIDC auth, the IdP must support cross-client authentication (audience:server:client_id:* scopes)

Important: Required audiences are collected at muster startup and during user authentication. If you add or modify MCPServers with requiredAudiences after users have authenticated, those users must re-authenticate (muster auth logout followed by muster auth login) to obtain tokens with the new audiences.

Security Note: Access control for requiredAudiences is enforced at two levels: 1. Kubernetes RBAC: Only users with permissions to create/modify MCPServer CRDs can configure requiredAudiences 2. IdP Cross-Client Configuration: The identity provider (e.g., Dex) must be configured to allow cross-client authentication for the specified audiences. Unauthorized audience requests will be rejected by the IdP.

Token Exchange (Cross-Cluster SSO)

When clusters have separate Identity Providers, muster can use RFC 8693 Token Exchange to obtain a token valid on the remote cluster's IdP. This enables cross-cluster SSO without requiring shared trust.

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: remote-cluster-mcp
spec:
  description: "MCP on remote cluster with Token Exchange"
  type: streamable-http
  url: "https://mcp.remote-cluster.example.com/mcp"
  auth:
    tokenExchange:
      enabled: true
      tokenEndpoint: "https://dex.remote-cluster.example.com/token"
      audience: "mcp-server"

How it works: 1. User authenticates to muster 2. When accessing the remote server, muster exchanges its token at the remote IdP 3. Remote IdP validates the token and issues a new one valid for that cluster 4. muster uses the exchanged token for downstream requests

Checking SSO Status

Use muster auth status to see which servers are using SSO:

$ muster auth status

muster: authenticated
  Endpoint: https://muster.example.com
  Expires:  in 23 hours

MCP Servers:
  mcp-kubernetes  Connected [SSO: Forwarded]
  remote-cluster  Connected [SSO: Exchanged]
  isolated-server Not authenticated          Run: muster auth login --server isolated-server

Troubleshooting SSO

SSO server not connecting automatically: - Verify forwardToken: true is set in the MCPServer spec - Check that the downstream server trusts muster's OAuth client ID - Run with --debug to see detailed SSO connection logs

AWS SigV4 Request Signing

An AWS-hosted MCP server accepts no bearer token. It authenticates each request by an AWS Signature Version 4 signature. Set auth.type: sigv4 to make muster sign every request that it sends to such a server.

This is not SSO. The signature carries muster's own machine identity, not the identity of the user who made the call, so all users of the server share one AWS identity. CloudTrail records muster, not a named engineer.

auth.sigv4 is only valid with type: streamable-http. muster rejects it together with forwardToken, tokenExchange or authorizationServer, because none of them apply to a machine identity. These rules hold in both Kubernetes and filesystem mode.

A connection failure reports itself as such. A 401 from a SigV4 server is not "Auth Required" — there is no login flow to send a user to — so muster keeps the server in Failed and retries with backoff. Check the signing region, the assumed role and its policy.

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: aws-root
spec:
  type: streamable-http
  url: "https://aws-mcp.eu-central-1.api.aws/mcp"
  timeout: 120
  auth:
    type: sigv4
    sigv4:
      region: eu-central-1
  meta:
    AWS_REGION: eu-central-1

Fields:

Field	Required	Purpose
auth.sigv4.region	yes	The signing region. It must match the region in url, because the endpoint checks the credential scope of the signature.
auth.sigv4.service	no	The signing service name. It defaults to the first hostname label of url, so aws-mcp.eu-central-1.api.aws signs as aws-mcp.
auth.sigv4.roleArn	no	An IAM role that muster assumes before it signs. Leave it empty to sign as muster's own identity.
meta	see below	Entries merged into params._meta of every request that carries params. It is not a SigV4 field: see Request metadata.

muster gets its base credentials from the default AWS credential chain. In Kubernetes that means IRSA: the pod identity webhook injects AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE, and the chain exchanges the projected token for credentials. Set roleArn to chain one more hop from there, which is how one muster reaches many accounts. Each account gets its own MCPServer, and a shared family puts them all behind one tool name with an account selector.

Set meta for the AWS-hosted server. It takes the region that it operates in from params._meta.AWS_REGION. This region is a different value from the signing region, even when the two strings match: the signing region belongs to the endpoint, and the operating region belongs to the resources that the call reads. A value that a caller puts in _meta itself wins over the one in meta.

Set it even though calls succeed without it. The backend falls back to its own region rather than failing, so a missing entry does not raise an error — it returns a correct-looking answer about the wrong region. Measured against aws-mcp.eu-central-1.api.aws: the same query for CloudWatch log groups returned nothing with no meta, and the one log group that exists in the account with meta: {AWS_REGION: eu-north-1}. An agent reads the first result as "there are none".

Request metadata

meta is a remote-server field, not a SigV4 field. muster merges its entries into the params._meta object of every outbound JSON-RPC request that carries params. Use it for a backend that reads call-scoped configuration from the MCP metadata field instead of from tool arguments.

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: regional-tools
spec:
  type: streamable-http
  url: "https://mcp.example.com/mcp"
  meta:
    AWS_REGION: eu-central-1

Rules:

• The merge applies to type: streamable-http and type: sse, with any auth type and with none. A server that needs a login keeps its entries after the login: the per-session connection carries them too.
• An entry that the request already has in _meta wins, so a caller can override one per call.
• A request without params is left untouched. initialize does carry params, so the handshake gets the entries as well.
• type: stdio rejects the field. A stdio server speaks over a pipe, so no HTTP transport can inject the entries, and muster refuses the definition instead of accepting the map and dropping it.

Display name

UIs such as the developer portal show an MCPServer by the ui.giantswarm.io/display-name annotation when it is set, and by metadata.name otherwise.

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: kagi-search
  annotations:
    ui.giantswarm.io/display-name: "Kagi Search"
spec:
  type: streamable-http
  url: "https://mcp.example.com/kagi"

Through the core tools, pass displayName to core_mcpserver_create or core_mcpserver_update, or --display-name to muster create mcpserver. Surrounding whitespace is trimmed, and on create a blank value sets no annotation. On update, omitting displayName keeps the annotation and an empty or blank value removes it. core_mcpserver_get and core_mcpserver_list return it as displayName.

Using the CLI

Creating Servers via CLI

Create a stdio server:

muster create mcpserver filesystem-tools \
  --type stdio \
  --command "npx" \
  --args "@modelcontextprotocol/server-filesystem,/workspace" \
  --auto-start \
  --tool-prefix fs \
  --description "File system operations"

Create a streamable HTTP server:

muster create mcpserver remote-api \
  --type streamable-http \
  --url "https://api.example.com/mcp" \
  --timeout 60 \
  --tool-prefix api \
  --display-name "Remote API" \
  --description "Remote API tools"

Create an SSE server:

muster create mcpserver sse-server \
  --type sse \
  --url "https://sse.example.com/mcp" \
  --timeout 90 \
  --tool-prefix sse \
  --description "SSE MCP server"

Listing Servers

muster list mcpserver

Getting Server Details

muster get mcpserver filesystem-tools

Updating Servers

# Update stdio server
muster update mcpserver filesystem-tools \
  --auto-start=false \
  --description "Updated file system tools"

# Update remote server
muster update mcpserver remote-api \
  --url "https://new-api.example.com/mcp" \
  --timeout 120

Deleting Servers

muster delete mcpserver filesystem-tools

Configuration Best Practices

Stdio Servers

• Use absolute paths for commands when possible
• Set appropriate environment variables for configuration
• Enable auto-start for critical servers
• Use descriptive tool prefixes to avoid conflicts

Remote Servers (Streamable HTTP and SSE)

• Use HTTPS endpoints when possible for security
• Set appropriate timeouts based on server response times
• Test connectivity before deploying to production
• Monitor server availability and health
• Include necessary authentication headers

Tool Prefixes

• Use short but descriptive prefixes (e.g., k8s, git, fs)
• Avoid generic prefixes like tools or server
• Be consistent across related servers

spec:
  toolPrefix: "custom"  # Tools will be prefixed as "x_custom_*"

Troubleshooting

Stdio Server Issues

Command not found:

# Check if the command is available
which npx
npm install -g @modelcontextprotocol/server-filesystem

# Verify the server definition
muster get mcpserver filesystem-tools

Permission errors:

# Check file permissions
ls -la /workspace
chmod +x /path/to/mcp-server

# Run with appropriate user
sudo -u mcpuser muster start mcpserver filesystem-tools

Remote Server Issues

Connection timeouts:

# Test connectivity
curl -v https://api.example.com/mcp

# Increase timeout
muster update mcpserver remote-api --timeout 120

Transport errors:

# Check server type and endpoint
muster get mcpserver remote-api

# For SSE servers, ensure endpoint supports Server-Sent Events
# For HTTP servers, ensure endpoint supports streaming HTTP

Authentication errors:

# Update headers for authentication
muster update mcpserver remote-api \
  --header "Authorization=Bearer new-token"

Upstream 5xx, a path not served yet, and reconnect backoff:

A remote server whose connection attempt fails for a transient reason (connection refused, DNS, timeout, HTTP 5xx, or a 4xx other than 401 to the initialize -- the 404 of a backend whose route is still held by the previous pod during a rollout, the 405 of a server that speaks only legacy SSE) is retried automatically. The wait between attempts starts at 30 s, doubles on every consecutive failure and is capped at 2 minutes; from the third failure on the server is reported as Failed (unreachable in core_service_status). The orchestrator checks every 30 s whether an attempt is due, so a server whose upstream has recovered is back within the cap plus one tick -- in Connected; in Auth Required for a server a person signs in to through muster; in Awaiting Session for a server served per session with the caller's own identity (forwardToken, tokenExchange), which is Connected while a session holds a live connection.

A 429 Too Many Requests or 503 Service Unavailable to the initialize is a transient refusal, not a transport mismatch: the log and status name the status (endpoint answered the initialize POST with HTTP 429 Too Many Requests; retry after 1s), never a legacy SSE server, and when the answer carried a Retry-After the next attempt is scheduled for exactly that delay instead of the doubling backoff. The status alone that means legacy SSE is the 405.

A tokenExchange server is also Failed, on the same schedule, while its client credentials Secret (clientCredentialsSecretRef) is missing or unreadable -- every caller's exchange would fail the same way -- and settles in Awaiting Session on its own once the Secret exists. An exchange that fails for every caller alike (the token endpoint does not answer, invalid_client, an unknown connector) puts the server in Failed with the exchange error as lastError and the class as the Ready condition's reason (TokenExchangeCredentials, TokenExchangeEndpoint, TokenExchangeConnector). A token endpoint that does not answer -- a timeout, a refused connection, a 5xx or a non-OAuth page from a gateway in front of it -- is probed again on the same backoff, with the exchange's client credentials and a subject token no issuer signed: the server stays Failed (never Unreachable) while the probe gets no answer, and returns to Awaiting Session on the first OAuth answer, with no caller involved. An OAuth error (invalid_client, an unknown connector) is not retried: the next successful exchange, a restart or a spec change clears it. A failure of one caller's own token (invalid_grant, an expired subject token) stays that session's and never changes the server's state.

# Why did the last attempt fail and when is the next one?
kubectl get mcpserver remote-api -o jsonpath='{.status.lastError}{"\n"}{.status.lastFailureHTTPStatus}{"\n"}{.status.nextRetryAfter}{"\n"}'

# The same from the events: HTTP status and scheduled retry per attempt
muster events --resource-type MCPServer --resource-name remote-api

status.lastFailureHTTPStatus is set when the endpoint answered -- 504 from a gateway or tunnel in front of a healthy server, 503 from the server itself, 404 from a backend whose route is not served yet -- and absent when nothing answered at all. Three environment variables on the muster process tune the schedule: MUSTER_MCPSERVER_INITIAL_BACKOFF (default 30s), MUSTER_MCPSERVER_MAX_BACKOFF (default 2m) and MUSTER_ORCHESTRATOR_RETRY_INTERVAL (default 30s), each a Go duration.

A connected server is also probed with an MCP ping every MUSTER_ORCHESTRATOR_HEALTH_CHECK_INTERVAL (default 30s, a Go duration), each probe bounded by the server's spec.timeout. Three failed probes in a row close the client and put the server on the schedule above with a reconnect due at once; metadata.consecutiveHealthCheckFailures in core_service_status shows the running count. Servers served per session (forwardToken, tokenExchange, OAuth) have no shared client and are not probed.

A probe heals the connection; it does not re-read the tools. A server learns its tools at registration and again when the server sends notifications/tools/list_changed -- which a backend redeployed behind its Service with another tool set never does, having never seen muster's session. Every MUSTER_AGGREGATOR_CAPABILITY_POLL_INTERVAL (default 5m, a Go duration) the aggregator therefore re-lists every connected server's tools -- and the resources and prompts of a server that declared them at its handshake, so a tools-only server costs one request -- and updates what changed, notifying connected clients as it does after a list_changed: a server with a shared client once, a server served per session through each session's own live connection into that session's list. The listings are spread over the interval rather than started at once: with a hundred connections, one every three seconds, in a fixed order, so a backend with a per-caller rate limit sees one listing per interval. Nothing is polled that is not connected, and a listing that fails leaves the cached list as it is. The log line for a change names its trigger, Capability refresh (poll): updated 3 tools for <server>, as against (notification).

The poll is the catch-all; a backend redeployed with another catalogue is usually re-listed before it runs. The new process does not know the MCP sessions the old one issued, so the next request on any connection to it is answered 404 and the client re-establishes its session with a fresh handshake (session recovery). That is muster's sign of a new process behind the address -- no watch on the backend's pods, so it works for a backend off the cluster too -- and every recovered session re-lists the server through the recovered connection: a shared client's within one health-probe interval, a session's on the session's next call. The server is marked rolled at the same time, and every other session's cached list from before the mark is re-listed, through that session's connection, before describe_tool, filter_tools or a tools/list serves it (bounded to 15 s, the cached list is served when the backend does not answer in time). Three more signs mark a server rolled where no session is lost: a listing that answers differently from the one cached after the last mark (a stateless backend has no session to forget), a call the backend refuses as invalid params (the caller followed a schema the backend no longer serves: the server is re-listed first, and when the called tool's schema turns out to have changed the refusal says so and points at describe_tool), and a connection a tool call opens for a session that holds a cached list from an earlier one. The log line names these triggers too: (recovery), (read), (refusal), (connect). What stays until the poll is a backend whose catalogue changed without a session being lost and that no session calls or reads meanwhile.

Advanced Configuration

Environment Variables for Stdio Servers

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: custom-tools
spec:
  type: stdio
  command: "python"
  args: ["-m", "my_mcp_server"]
  env:
    PYTHONPATH: "/usr/local/lib/python3.9/site-packages"
    API_KEY: "your-api-key"
    DEBUG: "true"
    LOG_LEVEL: "info"

Custom Headers for Remote Servers

apiVersion: muster.giantswarm.io/v1alpha1
kind: MCPServer
metadata:
  name: authenticated-api
spec:
  type: streamable-http
  url: "https://secure-api.example.com/mcp"
  headers:
    X-API-Version: "v2"
    Content-Type: "application/json"
  headersFrom:
    - name: Authorization
      valueFrom:
        type: Secret
        name: authenticated-api-token
        key: authorization
  timeout: 45

Monitoring and Health Checks

Check server status:

# List all servers with status
muster list mcpserver

# Get detailed server information
muster get mcpserver <server-name>

# Check if server is available
muster check mcpserver <server-name>

Inspect server state and logs:

# Server status and any startup error
muster get mcpserver <server-name> -o yaml
muster list mcpserver --all --verbose

# Aggregator logs (servers log through it) — there is no `muster logs`
muster serve --debug

Integration Examples

With Cursor/VS Code

Configure Cursor to use muster MCP servers:

{
  "mcpServers": {
    "muster": {
      "command": "muster",
      "args": ["standalone"]
    }
  }
}

With Other AI Assistants

Most MCP-compatible assistants can connect to muster's aggregator endpoint at http://localhost:8090/mcp.

Configuration Best Practices

1. Naming Conventions

• Use descriptive names: git-tools, k8s-cluster-prod
• Include environment in name for multi-env setups
• Avoid special characters and spaces

2. Configuration Management

• Store configurations in version control
• Use environment-specific overlays
• Document required environment variables

3. Monitoring

• Always enable health checks for production
• Set appropriate timeouts
• Monitor resource usage

4. Security

• Run with minimal required permissions
• Use read-only filesystems where possible
• Regularly update server binaries

Related Documentation

• Configuration Reference - Detailed configuration options
• HTTP endpoints - The endpoints the aggregator serves
• CRD Reference - Kubernetes CRD schema
• Architecture - How MCP servers fit into muster
• Server Configuration Schema
• Troubleshooting Guide