OAuth token refresh blew up our staging env at 2pm on a Friday

We had a cron job that rotated refresh tokens for our internal dashboard API. Last Friday it hit a race condition where two workers grabbed the same token, both tried to refresh it, and the second one got a 401. My first thought was just retry it, but the retry logic was dumb and kept hammering the endpoint. Locked out the entire staging environment for our QA team for about 40 minutes. Had to manually revoke all sessions and write a quick mutex to serialize refreshes. Has anyone else hit this with their refresh token flow? I'm wondering if there's a standard way to handle concurrent refreshes without building my own lock.

1 comments

1 Comment

reesemiller1mo ago

The 40 minute lockout tracks with what I've seen too, refresh token races are brutal because the failure mode is usually worse than the original auth issue. We solved it by adding a short DB-backed lease on the token ID before any refresh call, so only one worker can even attempt it and the others just wait for the lease to expire. It's not a real mutex but it's way simpler than rolling your own lock service, and it handles the multi-instance case since it's based on the shared database.