Auto-restart#

DUMB includes an automatic restart system that monitors service health and restarts failed services to maintain system stability without manual intervention.

Overview#

The auto-restart system provides:

Health monitoring - Periodic health checks for each service
Automatic recovery - Restart services that become unhealthy
Exponential backoff - Increasing delays between restart attempts
Restart limits - Prevent infinite restart loops
Grace periods - Allow services time to initialize
Startup lifecycle gating - Do not restart services while the stack is still assembling
Verified recovery - Count success only after the restarted service is healthy

Auto-restart status Auto-restart indicators

How it works#

%%{ init: { "flowchart": { "curve": "basis" } } }%%
flowchart TD
    A([Service running])
    B{Health check}
    C{Threshold exceeded?}
    D{Restart limit reached?}
    E[Restart service]
    F([Stop retrying])
    G[Wait grace period]

    A ==> B
    B -- Healthy --> A
    B -- Unhealthy --> C
    C -- No --> B
    C -- Yes --> D
    D -- Not reached --> E
    D -- Reached --> F
    E ==> G
    G ==> B

Startup gate - Monitoring waits until DUMB startup is ready or degraded
Grace period - Each service receives its configured post-start grace
Health check - DUMB verifies the process, configured ports, and—where a safe local endpoint exists—the application itself
Unhealthy detection - Multiple consecutive failures trigger action
Restart attempt - Service is stopped and restarted
Recovery verification - Success is recorded only after health checks pass
Repeat - Continue monitoring after restart

Health states and application probes#

DUMB reports four application health states:

State	Meaning	Auto-restart behavior
Healthy	Process, ports, and application probe passed	Reset the consecutive-failure counter
Degraded	The application is reachable but reports a warning, or an optional probe is unsupported/protected	Keep the service running and show the reason
Starting	The application explicitly reports initialization or migration work	Keep waiting; do not count it as an Auto-restart failure
Unhealthy	The process/port failed or the application endpoint reports a real failure	Count toward `unhealthy_threshold`

Application probes are bounded, read-only, loopback requests cached briefly by the backend. Current probes include:

InfiniDysk backend /health
Servarr /ping for Sonarr, Radarr, Lidarr, Prowlarr, and Whisparr
Jellyfin /health, Emby application ping, and Plex /identity
Seerr status and Traefik Proxy Admin health
PostgreSQL pg_isready and pgAdmin ping
rclone's local RC version endpoint when RC is enabled

Services without a known safe application endpoint retain process and port checks. DUMB deliberately does not use content health, missing articles, indexer/provider warnings, or other external-integration checks as restart signals. Those conditions may require operator action, but restarting a healthy application usually does not fix them and can create a restart loop.

For InfiniDysk specifically, the maintained fork can keep its backend port open during a blocking database migration while /health returns 503 {"status":"migrating"}. DUMB reports this as Starting, so stack readiness waits for the real backend without repeatedly restarting a legitimate migration.

Configuration#

Auto-restart is configured globally in dumb.auto_restart:

"dumb": {
  "auto_restart": {
    "enabled": false,
    "restart_on_unhealthy": true,
    "healthcheck_interval": 30,
    "unhealthy_threshold": 3,
    "max_restarts": 3,
    "window_seconds": 300,
    "backoff_seconds": [5, 15, 45, 120],
    "grace_period_seconds": 30,
    "services": []
  }
}

Configuration options#

Option	Default	Description
`enabled`	`false`	Enable auto-restart globally
`restart_on_unhealthy`	`true`	Restart when health checks fail
`healthcheck_interval`	`30`	Seconds between health checks
`unhealthy_threshold`	`3`	Consecutive failures before restart
`max_restarts`	`3`	Maximum restarts within the window
`window_seconds`	`300`	Time window in seconds
`backoff_seconds`	`[5, 15, 45, 120]`	Backoff delays between restarts
`grace_period_seconds`	`30`	Seconds to wait after stack readiness or a later service launch before health checks
`services`	`[]`	Limit auto-restart to these process names

Backoff schedule#

To prevent rapid restart loops, DUMB selects delays from the configured backoff_seconds list:

Attempt	Delay
1	5 seconds
2	15 seconds
3	45 seconds
4+	120 seconds

This is an explicit list, not a calculated multiplier. You can replace it with any ordered list of non-negative delays. Attempts beyond the list use its last value.

Restart limits#

Services have a maximum number of restart attempts within a time window:

Default: 3 restart attempts in a rolling 300-second window
After reaching the limit, additional automatic attempts are suppressed while those attempts remain inside the window
Old attempts age out of the rolling window automatically
A manual start re-enables monitoring after an intentional stop, but does not erase the recorded rolling-window attempts

Restart limit reached

If a service keeps failing, investigate the root cause rather than increasing limits. Check logs for error messages.

Health checks#

For each explicitly selected service, DUMB verifies:

the tracked PID still exists and is not a zombie; and
each configured port, frontend_port, backend_port, or webdav_port (including corresponding values in env) accepts a TCP connection.

There is no standalone health_check object in the current config. Add a service to dumb.auto_restart.services; its entry may inherit all global values or override selected policy fields:

"services": [
  {
    "process_name": "Riven Backend",
    "enabled": true,
    "unhealthy_threshold": 4,
    "grace_period_seconds": 60
  }
]

An empty services list monitors no services even when the global enabled flag is true. Use exact process names from GET /process/processes or select services in the frontend panel.

Slow applications can use a longer service override without delaying checks for the rest of the stack. A 300-second override is a reasonable starting point for a Seerr instance that performs migrations or a first-run frontend build. The stack-wide readiness deadline is configured separately under dumb.startup; see Startup lifecycle.

Monitoring restart status#

Dashboard indicators#

The dashboard shows auto-restart status for each service:

Restart count - Number of restarts in current window
Last restart - Timestamp of most recent restart
Health status - Current healthy/unhealthy state

API endpoints#

Query restart status via the API:

# Get service status including restart info
curl 'http://localhost:3005/api/process/service-status?process_name=Riven%20Backend&include_health=true'

Response includes:

{
  "process_name": "Riven Backend",
  "status": "running",
  "healthy": true,
  "health_status": "healthy",
  "health_reason": null,
  "health_details": {
    "probe": "process_and_ports",
    "ports": [8080]
  },
  "restart": {
    "restart_attempts": 2,
    "restart_successes": 2,
    "restart_failures": 0,
    "recent_restart_attempts": 2,
    "pending": false,
    "next_restart_time": null,
    "disabled": false,
    "last_restart_time": 1736937000.0,
    "last_failure_reason": null,
    "last_exit_time": 1736936995.0,
    "last_exit_reason": "Port 127.0.0.1:8080 not responding",
    "unhealthy_count": 0,
    "unhealthy_threshold": 3
  }
}

WebSocket updates#

Real-time restart events via /ws/status:

{
  "type": "status",
  "processes": [
    {
      "process_name": "Riven Backend",
      "status": "running",
      "healthy": true,
      "restart": {
        "restart_attempts": 2,
        "restart_successes": 2,
        "restart_failures": 0,
        "recent_restart_attempts": 2,
        "pending": false,
        "next_restart_time": null,
        "disabled": false,
        "last_restart_time": 1736937000.0,
        "last_failure_reason": null,
        "last_exit_time": 1736936995.0,
        "last_exit_reason": "Port 127.0.0.1:8080 not responding",
        "unhealthy_count": 0,
        "unhealthy_threshold": 3
      }
    }
  ]
}

Disabling auto-restart#

Per-service#

Remove a service from the monitored list, or retain an explicit disabled entry:

"services": [
  {
    "process_name": "Riven Backend",
    "enabled": false
  }
]

Globally#

To disable auto-restart for all services, set dumb.auto_restart.enabled to false, or use the frontend panel. Per-service entries may remain stored for a later re-enable.

Best practices#

Appropriate thresholds#

Critical services (Plex, rclone): Lower threshold (2-3)
Background services (Zilean, NeutArr): Higher threshold (3-5)

Grace periods#

Fast-starting services: 10-15 seconds
Database-dependent services: 30-60 seconds
Services with startup tasks: 60-120 seconds

Monitoring#

Review restart counts regularly
Investigate services with frequent restarts
Check logs after restart events

Troubleshooting#

Service keeps restarting#

Check service logs for errors
Verify configuration is valid
Ensure dependencies are running
Check for port conflicts

Auto-restart not working#

Verify auto_restart.enabled is true
Confirm the exact process name is present and enabled in auto_restart.services
Check whether the restart limit was reached
Confirm the configured service port is accurate and reachable on container loopback

Restart delay too long#

Shorten the values in backoff_seconds
Review max_restarts and window_seconds before changing limits
Fix the underlying health failure instead of using a zero-delay retry loop

Dashboard - View restart status
Process Management API - API controls
WebSocket API - Real-time updates