Alert I screwed up, really really screwed up!

[Neal]

Hello all,

My sincere apologies about today's downtime, it is all my fault! For those that know about Linux and the "sudo rm -rf" command if used incorrectly, it can wipe out a server, and yeah, that's what happened. A command of "sudo rm -rf /foo" was supposed to be run but I instead typed "sudo rm -rf / foo" (notice the space) and that command is a command to delete all files/folders and that little SPACE I added caused the entire server to get wiped!

So yeah, we're back up and running but the next issue is the backup system didn't run after the 17th of June so we're two days behind. Not the end of the world but it's not how it's supposed to work when I have systems in place that are supposed to backup daily, now it will be twice daily, and off site storage as well was already in place but also two days behind as they run together. Screw up #2!

For any new members that registered on or after the 17th and you can't login, please register again, we are starting back as of June 17th sadly.

I have a lot of things to test and dial in but we're up and running again.

Obviously this is the last thing I needed right now as my brain is fried handling my mother's estate. I'm glad for the moment at least we're back up and running. You always learn from situations like this and get better as a result. I found some weak areas that need attention and will address that immediately.

Once again, my sincere apologies. 100% my fault!

Edit: When something bad happens I always try to be better afterwards. During this reload we got upgraded to the latest OS version and also Claude walked me through the entire config for the latest optimizations so hopefully the site will be even faster than before and the backup system more reliable.

 

[Rollin Ollen]

Hello all,

My sincere apologies about today's downtime, it is all my fault! For those that know about Linux and the "sudo rm -rf" command if used incorrectly, it can wipe out a server, and yeah, that's what happened. A command of "sudo rm -rf /foo" was supposed to be run but I instead typed "sudo rm -rf / foo" (notice the space) and that command is a command to delete all files/folders and that little SPACE I added caused the entire server to get wiped!

So yeah, we're back up and running but the next issue is the backup system didn't run after the 17th of June so we're two days behind. Not the end of the world but it's not how it's supposed to work when I have systems in place that are supposed to backup daily, now it will be twice daily, and off site storage as well was already in place but also two days behind as they run together. Screw up #2!

For any new members that registered on or after the 17th and you can't login, please register again, we are starting back as of June 17th sadly.

I have a lot of things to test and dial in but we're up and running again.

Obviously this is the last thing I needed right now as my brain is fried handling my mother's estate. I'm glad for the moment at least we're back up and running. You always learn from situations like this and get better as a result. I found some weak areas that need attention and will address that immediately.

Once again, my sincere apologies. 100% my fault!

Edit: When something bad happens I always try to be better afterwards. During this reload we got upgraded to the latest OS version and also Claude walked me through the entire config for the latest optimizations so hopefully the site will be even faster than before and the backup system more reliable.

Relax! $hit happens.

 

[Kevin D Pem]

Been there done that! The results ain't pretty,but they are human. We are already over it,so put the bat down!