Skip to main content

Troubleshooting Guide: Resolving 503 Service Unavailable Errors

Distributor troubleshooting for a 503 Service Unavailable error in Encompass: check maintenance windows, wait out traffic spikes, and what details to send Support. Requires v26.02+.

B
Written by Bailey Naas-Thornton

Minimum Version Required: 26.02

Pre-Conditions

  • Requires an active internet connection to test endpoint connectivity.

  • Requires administrative access to server infrastructure, load balancers, or application monitoring tools (e.g., AWS CloudWatch, Datadog, New Relic).

  • Requires visibility into the organization's deployment schedule and maintenance windows.

Steps

  • Check for Scheduled Maintenance: Verify if the target service is currently undergoing a planned restart, deployment, or routine maintenance. Check your internal communication channels or status page. If maintenance is ongoing, wait until the window closes and try the request again.

  • Monitor for Temporary Overload: Wait a few minutes and refresh the application. 503 errors are often temporary responses to brief traffic spikes.

  • Review Compute Instance Resources: Access your server monitoring dashboard to check for instance overload. Look specifically for CPU utilization nearing 100%, memory exhaustion (OOM errors), or exhausted database connection limits.

  • Verify Downstream Dependencies: Check the status of required downstream systems (e.g., databases, third-party APIs, or microservices). If a downstream service is experiencing an outage, it may be triggering an automated circuit breaker on your primary application, resulting in a 503 error.

  • Inspect Load Balancer Health Checks: Review your load balancer logs. Confirm whether application health checks are consistently failing. If they are failing, the load balancer may be removing instances from the available pool, leaving no servers available to process incoming requests.

  • Contact Support: If you have completed the above steps and the issue persists, please contact support. To ensure efficient troubleshooting, gather and provide the following specific information:

    • Exact error messages or codes encountered (including full HTTP response headers if available).

    • Screenshots or video recordings of the issue as it occurs in the user interface.

    • Log files (specifically load balancer logs, application error logs, and server access logs).

    • The exact steps taken by the customer that led to the issue.

    • The exact date, time, and timezone the issue was first observed, and details on any recurrence.

    • Any changes made to the system, network configurations, or code deployments immediately before the issue appeared.

Fields And Definitions

  • 503 Service Unavailable: An HTTP status code indicating that the server is currently unable to handle the request due to a temporary overloading or maintenance of the server.

  • Compute Instance Overload: A condition where a server runs out of critical processing resources (CPU, Memory, or Network Connections), rendering it incapable of accepting new tasks.

  • Circuit Breaker: An automated safety mechanism that halts requests to a downstream dependency that is known to be failing, preventing system-wide cascading failures.

  • Health Check: An automated process (often performed by a Load Balancer) that regularly pings a server to verify it is responsive and functioning correctly.

  • Load Balancer: A device or service that acts as a reverse proxy, distributing network or application traffic across a number of servers to increase capacity and reliability.

Best Practices

  • Implement Auto-Scaling: Configure auto-scaling rules based on CPU and memory utilization to dynamically add compute instances during high traffic periods, preventing resource exhaustion.

  • Optimize Health Checks: Ensure your load balancer health checks are robust and actually verify the application's ability to serve traffic, rather than just pinging the server's IP address.

  • Graceful Degradation: Design your application to handle downstream dependency outages gracefully (e.g., serving cached data) rather than dropping the connection with a 503 error.

  • Proactive Alerting: Set up proactive monitoring alerts for high CPU utilization, memory thresholds, and failed health checks so your team can intervene before a 503 outage occurs.

  • Communicate Maintenance: Always broadcast planned downtime to users well in advance and use custom, user-friendly maintenance pages instead of default browser 503 error screens.

Did this answer your question?