A Really Bad Day / IMPORTANT / Radical Transparency

A Really Bad Day / IMPORTANT / Radical Transparency

Rough Day

Today was a rough day for our co-founders...

Early in the projects life we setup a pair of NitroKey Storage devices for our founders to ensure proper security and safety of our core account credentials. These devices have been a great asset for our project and have kept our secrets safe.

They work so well we are currently in the process of resetting ALL of our accounts. Every account we have that's "shared" for things like hosting, dns, registrar, etc is currently being reset.

What?!?

Yep! We're resetting ALL of our accounts. All but a few are presently reset but there are still a few remaining.

Given the nature of this event we wanted to ensure we detailed what happened today.

Silver Lining

The silver lining is ALL of our services are still online and operations are presently unaffected.

A Series of Unfortunate Events

The Typo

When @jmf@ap.lollipopcloud.solutions tried to unlock their NitroKey they learned they hadn't remembered their password 100%.

Fun fact: the NitroKey Storage device will self-destruct and become un-usable after 3 failed password attempts.

Thanks to the security features of the NitroKey jmf was locked out of their NitroKey.

The Failed Firmware Update

Today @kemonine@social.holdmybeer.solutions updated the firmware on their NitroKey and the password database was reset as part of the upgrade process. This can happen and there are big warnings about such things happening on the NitroKey site. KemoNine was a dumbass and didn't double check with jmf ahead of the update.

Unfortunately the firmware update caused a reset of the NitroKey's password manager and KemoNine was locked out of the core Lollipop Cloud KeepassXC database in an instant. They had to reach out to jmf to get their access restored...

Fixes

Thankfully KemoNine hadn't lost their GPG key or access to the Lollipop Cloud infrastructure. Because of this we had some useful options for recovery. We could still recover if our founders were fully locked out of thier NitroKeys but thankfully that didn't happen.

Unfortunately this means KemoNine only had access to our infrastructure. Not our main password database.

Even better: KemoNine and jmf both have access to the main administrative e-mail addresses used by the project via their phones.

Once KemoNine verified server access, jmf and KemoNine worked together to begin the process of resetting all of our account passwords and 2FA setups. This process is nearly complete and we are waiting on some final support tickets to be complete.

Once the password and account reset process got under way KemoNine started working with jmf to get them back online with a Yubikey 4 jmf has on-hand. Based on today's events KemoNine will be deploying a Yubikey 4 with their GPG key and moving to a Yubikey 4 to match jmf's setup going forward.

Protip

If you're going to use security tokens, keep a couple spares around in case you need to shuffle suddenly.

bitwarden_rs and CryptPad

Interestingly we recently started looking at bitwarden_rs to help facilitate sharing of passwords between our core team. We were going to be deploying it as an internal service in the coming weeks.

Based on the events of today, we've begun the process of deploying it much sooner so we can better share and manage our account secrets between the co-founders.

This approach has the benefit of being password based (you can store random passwords on Yubikey 4 devices) and we're looking at U2F and Yubikey OTP support within bitwarden_rs as well.

We can even create a small encrypted storage device on our NAS backed by a Yubikey 4 NEO (the one we use to sign our builds) to help protect our data.

We also have CryptPad deployed which stores all data in an encrypted form. We're using it internally at the moment to manage our updated passwords (all randomly generated) to ensure they are fully encrypted and safe. Once bitwarden_rs is fully online and setup to our OpSec standards we will make the migration to bitwarden_rs from CryptPad.

Going Forward

Going forward we will continue to keep our secrets on secure storage, use hardware security tokens and keep our OpSec as good as possible.

This is exactly what we wanted to see in the event of our hardware tokens failing / locking us out / being reset. We didn't anticipate both "failing" at the same time but we're glad that our OpSec withstood failures like this. If we extend today's events to theft, a hacker wouldn't have been able to gain access via our tokens. We use strong passwords and the tokens self-destruct after 3 failed attempts. Knowing that our secrets were safe in the event of failures, resets or permanent locks makes us very happy with our early OpSec decisions.

We will be re-deploying KemoNine's NitroKey (after they confirm their Yubikey 4 works 100%) as a secure storage device kept in a very secure location as a 3rd hardware token. This will be to ensure that in the case of catastrophic failure we can recover faster and more efficiently than this time around.