SeniorEng - Learn Real Software Engineering

Object Storage Down

Problem

Nimbus Object Storage in us-east-1 is returning 503 on all operations — PUT, GET, LIST, DELETE. Dozens of downstream services that depend on object storage are cascading.

Error rate

100% — all operations failing

Region affected

us-east-1 only

Incident started

09:37:00 UTC

Service Details

Cloud

Nimbus us-east-1

Service

Object Storage (S3-like)

Replication

3-way redundancy

Protocol

Nimbus Storage API v2

Scope

us-east-1 only

Downstream

50+ dependent services

Storage Health

Object storage operation status for us-east-1

All operations failing — 503 Service Unavailable

Every object storage operation in us-east-1 is returning 503. The storage layer itself is rejecting requests — not the API gateway, not the network.

OperationSuccess rateErrorp99 latency

PUT object0%503 Service UnavailableN/A

GET object0%503 Service UnavailableN/A

LIST bucket0%503 Service UnavailableN/A

DELETE object0%503 Service UnavailableN/A

HEAD object0%503 Service UnavailableN/A

COPY object0%503 Service UnavailableN/A

Sample error response:

HTTP/1.1 503 Service Unavailable

x-amzn-RequestId: 4a7e2b1c-f3d4-11e6-a8b9-c0143b4c3df4

Content-Type: application/xml

<Error>

<Code>ServiceUnavailable</Code>

<Message>Please reduce your request rate.</Message>

</Error>

Real incidents need a real screen.

Storage Health