Had a script fail silently for 3 days and lost $200 worth of data

Was scraping product prices for a side project but forgot to add error handling, so the script kept running empty queries while I thought everything was fine, and by the time I noticed on Saturday I had 72 hours of null values with no way to recover the original data, has anyone else dealt with a bug that didn't crash but just quietly ruined everything?

2 comments

2 Comments

mary7767h ago

Oh man, that is brutal. So did you at least figure out where exactly in the code the error handling was missing? I mean, was it something simple like not checking if the response was empty before writing the data, or was it a deeper logic issue with how you were looping through the pages? Idk, I always feel like the silent bugs are the scariest because they feel like they're sneaking up on you. Did you end up having to re-scrape everything from scratch or could you salvage any of it from backups?

knight.mason4h agoMost Upvoted

mary776 I feel your pain on that one. I had a similar thing happen a few years back with a project where I was scraping weather data. Turned out I wasnt properly handling a case where the API would return a 200 status code but with an empty body on certain days. It wasnt a loop issue or anything fancy, just a simple if response.content check that I skipped because I assumed a 200 meant data. I had to redo about three weeks of collection because the silent failures just piled up without anyone noticing. Your mileage may vary but I swear empty responses are the sneakiest thing in this line of work. Ended up salvaging maybe 60% from the raw logs, the rest was just gone like it never existed.