๐Ÿ–ฅ๏ธ

Real incidents need a real screen.

Open senioreng.dev on your laptop for the full experience.

Liveยท00:00elapsed

Incident Workspace

Trading Dashboard

Environment

Production ยท NYSE

Cumulative P&L Loss $M

โˆ’$46M

avg ยท 1h

$218M$164M$109M$55M$0
โˆ’$1M
โˆ’$1M
โˆ’$1M
โˆ’$1M
โˆ’$1M
โˆ’$1M
โˆ’$1M
โˆ’$1M
โˆ’$55M
โˆ’$110M
โˆ’$165M
โˆ’$218M
09:0009:1509:3009:47

Order Volume k/min

8k

avg ยท 1h

25k19k13k6k0
1k
1k
1k
1k
1k
1k
1k
1k
23k
23k
23k
23k
09:0009:1509:3009:47

System Error Rate %

1%

avg ยท 1h

5%4%3%2%0%
1%
1%
1%
1%
1%
1%
1%
1%
1%
1%
1%
1%
09:0009:1509:3009:47

Open Long Exposure $M

$144M

avg ยท 1h

$700M$525M$350M$175M$0
$1M
$1M
$1M
$1M
$1M
$1M
$1M
$1M
$160M
$350M
$520M
$687M
09:0009:1509:3009:47

P&L History โ€” Today (August 1, 2012)

Time (ET)Cumulative P&LRate
09:30$0โ€”
09:31โˆ’$11Mโˆ’$11M/min
09:33โˆ’$35Mโˆ’$12M/min
09:35โˆ’$59Mโˆ’$12M/min
09:40โˆ’$119Mโˆ’$12M/min
09:45โˆ’$179Mโˆ’$12M/min
09:47โˆ’$218Mโˆ’$11M/min

What's Strange

Losses started exactly at market open (09:30 ET) and have been perfectly linear. No system errors anywhere. No exceptions. No memory pressure. The orders are executing successfully โ€” against the firm's own capital. Something has been buying at massive scale for 17 minutes.

P0 ยท Trading Incident

$440M in 45 Minutes

On-Call Alert โ€” 09:47 ET

P&L is down $218M since market open 17 minutes ago, falling at $11M/minute. No errors. No crashes. SMARS v2.4.1 deployed at 09:28 ET. All 8 trading servers report healthy.

ApexTrade is a market-making firm that transacts on NYSE. SMARS โ€” the Smart Market Access Routing System โ€” went live at market open today. Seventeen minutes in, P&L has collapsed. Losses are mounting at $11M/minute and there is no obvious cause. You are on-call.