Event Reference and Troubleshooting Guide¶

This comprehensive guide covers all Kubernetes events generated by muster during CRD lifecycle operations, providing detailed explanations, troubleshooting steps, and operational best practices.

Overview¶

muster generates events for all major operations involving: - MCPServers: Service lifecycle, tool discovery, health monitoring - Workflows: Execution tracking, step monitoring, tool availability

These are the two resource types accepted by --resource-type (mcpserver, workflow).

Event emission is always on. In Kubernetes mode events are written as native Kubernetes Events (visible via kubectl describe and kubectl get events); in filesystem mode they are recorded in muster's on-disk event log. Either way they are queryable through the core_events tool and the muster events command.

Event Structure¶

All muster events include: - Timestamp: When the event occurred - Resource: The object the event relates to (MCPServer (or Workflow)) - Reason: Short, machine-readable reason code - Message: Human-readable description with context - Type: Event severity (Normal or Warning) - Source: Component that generated the event (muster)

MCPServer Events¶

MCPServers are external MCP (Model Context Protocol) servers that provide tools to muster. Events track their lifecycle, health, and tool availability.

CRD Management Events¶

MCPServerCreated¶

Type: Normal
Meaning: MCPServer CRD was successfully created in the cluster
Message Example: "MCPServer 'github-server' successfully created"
Triggered When: Running muster create mcpserver or applying YAML
Next Steps: Check if server starts automatically if configured

MCPServerUpdated¶

Type: Normal
Meaning: MCPServer configuration was successfully updated
Message Example: "MCPServer 'github-server' configuration updated"
Triggered When: Modifying MCPServer spec (command, args, environment)
Next Steps: Monitor for restart if changes affect running process

MCPServerDeleted¶

Type: Normal
Meaning: MCPServer CRD was successfully removed
Message Example: "MCPServer 'github-server' successfully deleted"
Triggered When: Running muster delete mcpserver or deleting YAML
Next Steps: Verify associated service instances are handled gracefully

Service Lifecycle Events¶

MCPServerStarting¶

Type: Normal
Meaning: MCPServer service is beginning startup process
Message Example: "MCPServer 'github-server' service beginning startup"
Triggered When: Service start requested or automatic startup
Next Steps: Watch for MCPServerStarted or MCPServerFailed within reasonable time

MCPServerStarted¶

Type: Normal
Meaning: MCPServer service started successfully and is available
Message Example: "MCPServer 'github-server' service started successfully"
Triggered When: Process starts and initial health checks pass
Next Steps: Tools should become available; check with muster list tools

MCPServerStopped¶

Type: Normal
Meaning: MCPServer service was stopped cleanly
Message Example: "MCPServer 'github-server' service stopped"
Triggered When: Running muster stop mcpserver or graceful shutdown
Next Steps: Tools from this server become unavailable

MCPServerRestarting¶

Type: Normal
Meaning: MCPServer service is being restarted (stop + start)
Message Example: "MCPServer 'github-server' service restarting"
Triggered When: Configuration changes require restart or manual restart
Next Steps: Brief tool unavailability expected during restart

MCPServerFailed¶

Type: Warning
Meaning: MCPServer operation failed (start, health check, or crash)
Message Example: "MCPServer github-server operation failed: server unreachable after 4 consecutive failures (endpoint answered HTTP 504, next retry in 2m0s at 2026-09-05T15:57:34Z): failed to initialize streamable-http MCP client: request failed with status 504: Gateway Timeout"
Triggered When: Process crashes, fails to start, or becomes unresponsive. For a remote server every failed connection attempt emits one event: connection failure N of 3 before unreachable (...) for the first two, server unreachable after N consecutive failures (...) from the third on. The parenthesis names the HTTP status the endpoint answered with (endpoint answered HTTP 504) or no HTTP response (connection refused, DNS failure, timeout), and when the next attempt is scheduled. The wait doubles from 30 s per failure and is capped at 2 minutes (MUSTER_MCPSERVER_MAX_BACKOFF).

Troubleshooting:

# Check server configuration
muster get mcpserver github-server --output yaml

# Verify command and arguments
# Check if port is available
# Review logs for startup errors

# Test manual startup (filesystem mode)
muster standalone

Tool Discovery Events¶

MCPServerToolsDiscovered¶

Type: Normal
Meaning: Tools were successfully discovered from MCPServer
Message Example: "MCPServer 'github-server' discovered 15 tools: github_create_issue, github_list_repos, ..."
Triggered When: Server starts and advertises capabilities
Next Steps: Tools are now available for Workflows

MCPServerToolsUnavailable¶

Type: Warning
Meaning: Tool discovery failed or previously available tools became unavailable
Message Example: "MCPServer 'github-server' tools unavailable: server connection lost"
Triggered When: Server crashes, network issues, or MCP protocol errors

Troubleshooting:

# Check server status
muster get mcpserver github-server

# Verify server is responsive
muster check mcpserver github-server

# Test connection manually
muster agent --repl
# > list tools  # Check if tools appear

Health and Recovery Events¶

MCPServerHealthCheckFailed¶

Type: Warning
Meaning: Health checks are consistently failing for MCPServer
Message Example: "MCPServer 'github-server' health check failed: timeout after 30s"
Triggered When: Health check endpoint non-responsive or returns errors

Troubleshooting:

# Check if server process is running
# Verify health check endpoint configuration
# Check server logs for errors
# Test health endpoint manually if exposed

MCPServerRecoveryStarted¶

Type: Normal
Meaning: Automatic recovery process began for failed MCPServer
Message Example: "MCPServer 'github-server' automatic recovery started"
Triggered When: Health checks fail consistently and recovery is enabled
Next Steps: Monitor for MCPServerRecoverySucceeded or MCPServerRecoveryFailed

MCPServerRecoverySucceeded¶

Type: Normal
Meaning: Automatic recovery successfully restored MCPServer
Message Example: "MCPServer 'github-server' automatic recovery succeeded"
Triggered When: Recovery process successfully restarts server
Next Steps: Monitor stability; investigate root cause of failure

MCPServerRecoveryFailed¶

Type: Warning
Meaning: Automatic recovery failed to restore MCPServer
Message Example: "MCPServer 'github-server' automatic recovery failed: max retries exceeded"
Triggered When: Recovery attempts exceed limits or continue failing

Troubleshooting:

# Manual intervention required
# Check underlying issues (ports, permissions, resources)
# Consider restarting manually
muster stop mcpserver github-server
muster start mcpserver github-server

Workflow Events¶

Workflows define sequences of tool executions. Events track configuration, execution, and step-level progress.

Configuration Management Events¶

WorkflowCreated¶

Type: Normal
Meaning: Workflow CRD was successfully created
Message Example: "Workflow 'deploy-app' successfully created"
Triggered When: Creating workflow via CLI or YAML
Next Steps: Check tool availability and test execution

WorkflowUpdated¶

Type: Normal
Meaning: Workflow configuration was updated
Message Example: "Workflow 'deploy-app' configuration updated"
Triggered When: Modifying workflow steps, tools, or parameters
Next Steps: Tool availability may change; test updated workflow

WorkflowDeleted¶

Type: Normal
Meaning: Workflow was successfully removed
Message Example: "Workflow 'deploy-app' successfully deleted"
Triggered When: Deleting workflow via CLI or YAML removal
Next Steps: Workflow tool is removed from aggregator

WorkflowValidationFailed¶

Type: Warning
Meaning: Workflow definition failed validation
Message Example: "Workflow 'deploy-app' validation failed: step 2 references unknown tool 'invalid_tool'"
Triggered When: Invalid workflow definition detected

Troubleshooting:

# Check workflow definition
muster get workflow deploy-app --output yaml

# Validate manually
muster check workflow deploy-app

# Common issues:
# - Invalid tool references
# - Malformed step definitions
# - Invalid parameter mappings

WorkflowValidationSucceeded¶

Type: Normal
Meaning: Workflow definition passed validation
Message Example: "Workflow 'deploy-app' validation successful"
Triggered When: Successful validation check
Next Steps: Workflow is ready for execution

Execution Lifecycle Events¶

WorkflowExecutionStarted¶

Type: Normal
Meaning: Workflow execution began
Message Example: "Workflow 'deploy-app' execution started with 5 steps (execution ID: abc123)"
Triggered When: Workflow executed via CLI or tool call
Next Steps: Monitor step completion events

WorkflowExecutionCompleted¶

Type: Normal
Meaning: Workflow execution completed successfully
Message Example: "Workflow 'deploy-app' execution completed successfully in 2m30s (execution ID: abc123)"
Triggered When: All steps complete successfully
Next Steps: Check execution results if needed

WorkflowExecutionFailed¶

Type: Warning
Meaning: Workflow execution failed
Message Example: "Workflow 'deploy-app' execution failed at step 3: tool 'docker_run' returned error (execution ID: abc123)"
Triggered When: Step fails and doesn't allow failure, or critical error

Troubleshooting:

# Check execution details
muster get workflow-execution abc123

# Review failed step
# Check tool availability
# Verify step parameters

# Test individual step if possible
muster agent --repl
# > call tool_name {"param": "value"}

WorkflowExecutionTracked¶

Type: Normal
Meaning: Execution state was persisted for tracking
Message Example: "Workflow 'deploy-app' execution state tracked (execution ID: abc123)"
Triggered When: Execution state is saved (start, steps, completion)
Next Steps: Execution can be queried for status and results

Step-Level Execution Events¶

WorkflowStepStarted¶

Type: Normal
Meaning: Individual workflow step began execution
Message Example: "Workflow 'deploy-app' step 'build-image' started using tool 'docker_build'"
Triggered When: Each step begins execution
Next Steps: Watch for step completion

WorkflowStepCompleted¶

Type: Normal
Meaning: Individual workflow step completed successfully
Message Example: "Workflow 'deploy-app' step 'build-image' completed successfully"
Triggered When: Step execution succeeds
Next Steps: Next step will begin (if any)

WorkflowStepFailed¶

Type: Warning (unless allowFailure=true)
Meaning: Individual workflow step failed
Message Example: "Workflow 'deploy-app' step 'deploy-service' failed: service already exists (allowFailure: true)"
Triggered When: Step execution fails

Troubleshooting:

# Check if failure was expected (allowFailure)
# Review step configuration
# Test tool manually
# Check step parameters and context

WorkflowStepSkipped¶

Type: Normal
Meaning: Step was skipped due to condition evaluation
Message Example: "Workflow 'deploy-app' step 'rollback' skipped: condition 'deployment_failed' evaluated to false"
Triggered When: Step condition evaluates to false
Next Steps: Normal behavior for conditional steps

WorkflowStepConditionEvaluated¶

Type: Normal
Meaning: Step condition was evaluated
Message Example: "Workflow 'deploy-app' step 'notify-slack' condition evaluated: true (continue execution)"
Triggered When: Conditional step evaluation occurs
Next Steps: Step will execute or be skipped based on result

Tool Availability Events¶

WorkflowUnavailable¶

Type: Warning
Meaning: Required tools became unavailable for workflow
Message Example: "Workflow 'deploy-app' unavailable: 2 required tools missing"
Triggered When: MCPServers providing required tools become unavailable

Troubleshooting:

# Check workflow tool requirements
muster check workflow deploy-app

# Check MCPServer status
muster list mcpserver

WorkflowToolRegistered¶

Type: Normal
Meaning: Workflow was registered as executable tool in aggregator
Message Example: "Workflow 'deploy-app' registered as action_deploy-app tool"
Triggered When: Workflow becomes available and is registered
Next Steps: Workflow can be called as action_deploy-app tool

WorkflowCapabilitiesRefreshed¶

Type: Normal
Meaning: Aggregator capabilities updated after workflow changes
Message Example: "Workflow capabilities refreshed: 15 workflow tools available"
Triggered When: Workflow registration/unregistration triggers capability update
Next Steps: Updated capabilities available to MCP clients

Event Querying and Analysis¶

Using muster events Command¶

# View all recent events
muster events

# Filter by resource type
muster events --resource-type mcpserver
muster events --resource-type workflow

# Filter by specific resource
muster events --resource-type mcpserver --resource-name github-server
muster events --resource-type workflow --resource-name deploy-app

# Filter by event type
muster events --type Warning
muster events --type Normal

# Time-based filtering
muster events --since 1h
muster events --since 2024-01-15T10:00:00Z --until 2024-01-15T18:00:00Z

# Combined filtering
muster events --resource-type mcpserver --type Warning --since 2h --limit 10

# Follow mode: print the matching events seen so far, then stream new ones as
# they occur via real server push (no client-side polling); press Ctrl+C to stop.
# The server sources the stream from a native Kubernetes watch (Kubernetes mode)
# or an fsnotify watch on the on-disk event log (filesystem mode).
muster events --follow
muster events --resource-type workflow --follow

# Follow honors the output format: --output json emits one JSON object per
# line (newline-delimited JSON), convenient for piping into jq.
muster events --follow --output json

In follow mode the resource/type filters (--resource-type, --resource-name, --namespace, --type) apply to both the initial snapshot and the streamed events. --limit bounds only the initial snapshot; the live stream is unbounded. --since / --until apply only to the initial snapshot — a watch only ever surfaces new events, so a time window does not constrain the stream.

Using kubectl (Kubernetes Mode)¶

# View all muster events
kubectl get events --field-selector source=muster

# Filter by object
kubectl get events --field-selector involvedObject.kind=MCPServer,involvedObject.name=github-server

# Filter by namespace
kubectl get events -n muster-system --field-selector source=muster

# Filter by reason
kubectl get events --field-selector reason=MCPServerFailed

# Watch events in real-time
kubectl get events --watch --field-selector source=muster

Event Analysis Patterns¶

Troubleshooting Workflow Issues¶

# 1. Check recent events for the workflow
muster events --resource-type workflow --resource-name my-workflow --since 1h

# 2. Check MCPServer events for required tools
muster events --resource-type mcpserver --type Warning --since 1h

Monitoring Workflow Executions¶

# 1. Check workflow execution events
muster events --resource-type workflow --resource-name deploy-app --since 30m

# 2. Filter for execution-specific events
muster events --resource-type workflow | grep "execution"

# 3. Look for step failures
muster events --resource-type workflow --type Warning

Health Monitoring¶

# 1. Check for health-related warnings
muster events --type Warning | grep -i health

# 2. Monitor recovery events
muster events | grep -i recovery

# 3. Check tool availability issues
muster events | grep -i "unavailable\|missing"

Troubleshooting Common Scenarios¶

Scenario 1: MCPServer Won't Start¶

Symptoms: - MCPServerFailed events - MCPServerToolsUnavailable events - Tools missing from aggregator

Investigation Steps:

# 1. Check recent MCPServer events
muster events --resource-type mcpserver --resource-name <server> --since 1h

# 2. Check server configuration
muster get mcpserver <server> --output yaml

# 3. Test server availability
muster check mcpserver <server>

# 4. Try manual start
muster start mcpserver <server>

Common Causes: - Port already in use - Invalid command or arguments - Missing dependencies - Permission issues - Network connectivity problems

Scenario 2: Workflow Execution Fails¶

Symptoms: - WorkflowExecutionFailed events - WorkflowStepFailed events - Incomplete workflow results

Investigation Steps:

# 1. Check workflow execution events
muster events --resource-type workflow --resource-name <workflow> --since 30m

# 2. Get execution details
muster get workflow-execution <execution-id>

# 3. Test failed tool manually
muster agent --repl
# > call <failed-tool> {...}

# 4. Check tool availability
muster check workflow <workflow>

Scenario 3: Tools Suddenly Unavailable¶

Symptoms: - Multiple ToolsUnavailable events - MCPServer tools become unavailable - Workflows fail with tool errors

Investigation Steps:

# 1. Check which MCPServers are affected
muster events --type Warning | grep -i "tools.*unavailable"

# 2. Check MCPServer status
muster list mcpserver

# 3. Check for connectivity issues
muster agent --repl
# > list tools

# 4. Recover affected servers. muster reconciles MCP servers from their
#    definitions and retries unreachable ones automatically. To force a
#    reconnect, re-apply the definition (kubectl apply / muster create) or
#    restart the aggregator. There is no `muster restart`.
muster get mcpserver <affected-server> -o yaml

Best Practices¶

Event Monitoring¶

Regular Health Checks

# Set up regular monitoring
muster events --type Warning --since 1h

Resource-Specific Monitoring

# Monitor critical resources
muster events --resource-type mcpserver --resource-name critical-server

Automated Alerting

# Example: Alert on warnings
if muster events --type Warning --since 5m | grep -q .; then
  echo "Warning events detected!"
fi

Event Analysis¶

Timeline Analysis: Use --since and --until to understand event sequences
Correlation: Compare events across related resources (Workflow + MCPServer)
Pattern Recognition: Look for recurring issues or patterns

Operational Guidelines¶

Warning Events: Always investigate Warning events promptly
Recovery Events: Monitor recovery success rates
Tool Availability: Track tool availability patterns for capacity planning
Execution Monitoring: Monitor workflow execution success rates

Event Retention and Cleanup¶

Kubernetes Mode¶

Events follow Kubernetes TTL policies (typically 1 hour)
Use external logging solutions for long-term retention
Consider event aggregation for monitoring

Filesystem Mode¶

Events persist in events.log file
Manual cleanup required
Suitable for development and debugging

Integration with Monitoring Tools¶

Prometheus Integration¶

Events can be scraped and converted to metrics for dashboards and alerting.

Log Aggregation¶

Events can be forwarded to log aggregation systems for analysis and retention.

Custom Dashboards¶

Event data can power operational dashboards showing system health and activity.

Conclusion¶

muster's event system provides comprehensive visibility into system operations. By understanding event types, their meanings, and troubleshooting approaches, operators can effectively monitor, diagnose, and maintain muster deployments.

Regular event monitoring, combined with proper alerting and response procedures, ensures reliable operation of muster-managed services and workflows.