Alert I screwed up, really really screwed up!

[Neal]

Hello all,

My sincere apologies about today's downtime, it is all my fault! For those that know about Linux and the "sudo rm -rf" command if used incorrectly, it can wipe out a server, and yeah, that's what happened. A command of "sudo rm -rf /foo" was supposed to be run but I instead typed "sudo rm -rf / foo" (notice the space) and that command is a command to delete all files/folders and that little SPACE I added caused the entire server to get wiped!

So yeah, we're back up and running but the next issue is the backup system didn't run after the 17th of June so we're two days behind. Not the end of the world but it's not how it's supposed to work when I have systems in place that are supposed to backup daily, now it will be twice daily, and off site storage as well was already in place but also two days behind as they run together. Screw up #2!

For any new members that registered on or after the 17th and you can't login, please register again, we are starting back as of June 17th sadly.

I have a lot of things to test and dial in but we're up and running again.

Obviously this is the last thing I needed right now as my brain is fried handling my mother's estate. I'm glad for the moment at least we're back up and running. You always learn from situations like this and get better as a result. I found some weak areas that need attention and will address that immediately.

Once again, my sincere apologies. 100% my fault!

Edit: When something bad happens I always try to be better afterwards. During this reload we got upgraded to the latest OS version and also Claude walked me through the entire config for the latest optimizations so hopefully the site will be even faster than before and the backup system more reliable.

 

[Rollin Ollen]

Hello all,

My sincere apologies about today's downtime, it is all my fault! For those that know about Linux and the "sudo rm -rf" command if used incorrectly, it can wipe out a server, and yeah, that's what happened. A command of "sudo rm -rf /foo" was supposed to be run but I instead typed "sudo rm -rf / foo" (notice the space) and that command is a command to delete all files/folders and that little SPACE I added caused the entire server to get wiped!

So yeah, we're back up and running but the next issue is the backup system didn't run after the 17th of June so we're two days behind. Not the end of the world but it's not how it's supposed to work when I have systems in place that are supposed to backup daily, now it will be twice daily, and off site storage as well was already in place but also two days behind as they run together. Screw up #2!

For any new members that registered on or after the 17th and you can't login, please register again, we are starting back as of June 17th sadly.

I have a lot of things to test and dial in but we're up and running again.

Obviously this is the last thing I needed right now as my brain is fried handling my mother's estate. I'm glad for the moment at least we're back up and running. You always learn from situations like this and get better as a result. I found some weak areas that need attention and will address that immediately.

Once again, my sincere apologies. 100% my fault!

Edit: When something bad happens I always try to be better afterwards. During this reload we got upgraded to the latest OS version and also Claude walked me through the entire config for the latest optimizations so hopefully the site will be even faster than before and the backup system more reliable.

Relax! $hit happens.

 

[Kevin D Pem]

Been there done that! The results ain't pretty,but they are human. We are already over it,so put the bat down!

 

[Jim]

Good deal, Neal! There are actually some pretty positive sides to this!

• I woke up 3 days younger, so that's pretty good. (21 days in dog years!)
• All those posts that I regretted making, all gone now. So yeah, there's definitely an upside there.
• I was supposed to mow the lawn on Thursday, so now I won't have to do it until tomorrow.
• Time travel is real! And Neal is the first person in human history to pull it off using nothing but an admin panel, a backup file, and a brief moment of panic.
• I lost a bet with Trish yesterday, so now I know how to avoid that when it comes around in a couple of days.

Don't beat yourself up buddy. Most forum admins just maintain websites, you accidentally bent the fabric of space and time. That's a much more impressive accomplishment.

 

[rjshiflet49]

There are two types of programmers...those who HAVE deleted massive files accidentally, and those that WILL delete massive files accidentally.

At least you've got your first time behind you now. Any of us that deal in programming or website management certainly understand and forgive (and breathe a sigh of relief that it wasn't us this time).

Onward and upward.

 

[Underboss]

If you have been involved in a forum being down for a day, you may be entitled to compensation.

 

[Texas Clodhopper]

Another reminder of what a little "space" can accomplish.

 

[rvkev]

Relax NEIL!! You did not delete all the money in your mutual fund accounts and/or givu up posession to anything else. I hope everyone went outside to get some sunshine while getting an electronic break!!

---

Life is good and it is better to be seen than viewed!

 

[DennisT]

Those of us that work or worked in technology have either been there or will be.

Brought back a memory when I was doing consulting on a project accounting system - luckily their backups were darn near real time .

All good!

 

[Neal]

Thanks all. Got a good night sleep and did the final items this morning and all appears good and 100%. If anyone finds any issue, please let me know.

I'm primarily a Windows guy, been hosting Linux/Ubuntu for a few years mainly for sites like this. I knew of the sudo rm -rf threats, typically careful, I guess when properly exhausted and trying to do something simple, bam it hits. It only needs to hit once.

The interesting part is in prior server installs it was all about google and how to's for configurations. This time around I had Claude Desktop at my side (literally) with a paid Pro plan and let it walk me through ever step with recommended configs which was very interesting to see and learn from. AI is absolutely amazing, worth the $20 I paid yesterday as I knew I'd blow through the free plan I had in no time.

Resuming normal ops...we need pics, RV and camping pics. Post away, let's live the RV life through those of you lucky to be out doing it.

Thanks again for your kindness and amazing support!

 

[BGMAC]

No harm no foul!
Idea; on some of those commands that just a space could cause problems maybe create a script to run the commands

 

[GypsyR]

Ha! I only Needed to read the title and get as far as "sudo-rm" to think "Oh my God, you did NOT!" I feel for you. Nothing permanent, we'll sail through.

 

[Headhunter]

Good news is your back in full force. Relax now, been there and done that....only worse...LOL

 

[JBright11]

Nothing to stress on; recovery seems to have gone very well.
If you really think about it, perhaps some unintended benefit was achieved.

 

[team bradfield]

Glad you got it fixed