🖥️

Real incidents need a real screen.

Open senioreng.dev on your laptop for the full experience.

Pull Request #2198

Add distributed Redis lock to prevent duplicate job processing

Implements a Redis-based distributed lock around job execution to ensure each job is processed exactly once, preventing the duplicate processing incident from recurring.

Ready to merge
processor/JobProcessor.java+18 additions
Viewed
1
public class JobProcessor {
2
    @Autowired private RedisLockService redisLockService;
3
    @Autowired private JobExecutor jobExecutor;
5
+    private static final int LOCK_TTL_SECONDS = 30;
7
    public void processJob(Job job) {
8
        // Job execution SLA: p50=8s, p95=45s, p99=118s, max=120s
9
+        String lockKey = "job_lock:" + job.getId();
10
+        boolean acquired = redisLockService.tryLock(lockKey, LOCK_TTL_SECONDS);
11
+        if (!acquired) {
12
+            log.info("Job already being processed, skipping", job.getId());
13
+            return;
14
+        }
15
        try {
16
            jobExecutor.execute(job);// can take up to 120 seconds
17
        } finally {
18
+            redisLockService.releaseLock(lockKey);
19
        }
20
    }
21
}
CL

chris_lead

Approved review

Lock acquisition and release look correct. SET NX EX is the right approach. The finally block ensures the lock is always released even if job execution throws.