Collaboration, Data, Passwords and Growth

Collaboration, Data, Passwords and Growth

Quick Recap

To quickly recap: on 2019/01/20 we lost ALL our passwords thanks to a bad firmware update and a forgotten password. These things happen and we're happy to report we've suffered no outages and have full control of all but one or two accounts. We've started the process for the last few and are waiting to hear back from our providers.


Early Stage

Early in our projects life we opted to deploy a pair of NitroKey Storage devices for holding our internal documentation, passwords, GPG keys and some other critical data that needs to be secure. We had maybe one server online and it was a shared hosting server.

We setup Syncthing so the owners of the NitroKeys could directly sync the data securely and avoid having to deploy some kind of self-hosted cloud storage and self-hosted password manager.

For the 6 months these were in service they worked surprisingly well. We didn't have to think terribly hard and we could share data pretty easily. We also deployed CryptPad early on and used it for collaborative sessions where our team members needed real-time, shared editing of documents.

Sync wasn't a burden and a lot of the collaboration was done when all team members were online so we didn't run into any real hiccups.

Life was good for our little project and we were making great progress on things. We don't regret our NitroKey decision and feel it's a great choice in the early stages of a project's life cycle. Especially so if you want to limit the amount of infrastructure that needs to be deployed and maintained.

Late Stage

As time marched forth, we grew. We've been collaborating more, generating more data and we've had a number of outside contributions. The Nitro Keys were holding but we knew they needed to replaced with more robust solutions. Then we lost it all on 2019/01/20. Our Nitro Keys "failed" thanks to a comedy of errors and we were forced to think about what was going to replace them.

As we've grown, expanded our infrastructure and our list of responsibilities increased, we really needed something more flexible than a USB stick device for our storage. Especially for our password database. We were fighting with the KeepassXC option as it was a pain to sync and we had a few access hiccups along the way. This wasn't an issue early-on but as we added more services, passwords, TOTP 2FA tokens and more... it became a bit unwieldy for us. Never mind what happens when all of the deployed NitroKeys fail and you get locked out of your password database...

We also had more than one person generating data that needed to be stored and organized in a place that was "central". Syncthing is great for individuals but doesn't scale well for teams after a point. Good for early team stuff, tricky once there is a lot of churn. Especially if async is desired for moving data between individuals (aka: two people don't have to be online at the same time to trade updates via sync).

After some thought, discussion and planning we settled on deploying NextCloud and bitwarden_rs to replace the Nitro Keys.

We now have the appropriate, private, secured infrastructure deployed such that bitwarden_rs (password manager) and NextCloud (files, calendar, contacts, more) can live on our private LAN and have the data stored on a lollipop with more secure storage than our lollipops hosting our public services.

Combined with CryptPad for collaborative editing of text, we have the trifecta of needs covered: passwords, data and collaborative editing.

As a later stage project these 3 services are a great way to achieve async collaboration, data sync and password access. It's also a more flexible setup for backups and we can more easily grow past 2 project principles.

Service Choices


NextCloud is the go to self-hosted cloud solution. It's well developed, has a TON of great features and is what you'd normally get from a "big" cloud provider like Google. Unlike Google we get to control our data, aren't data mined and we get to support open source. This was a very easy decision to make now that we have private Lollipops deployed for self-hosting our non-public data. Non-public data like our passwords, invoices and some other things.


bitwarden_rs is a rust implementation of the BitWarden API. It requires a LOT less hardware, can run on arm computers and sbcs (aka: Lollipops) and has a good group of developers. We've submitted some code changes that were accepted and the dev's treated us great. BitWarden is an OSS project too but the dependencies for self-hosting the main BitWarden service aren't compatible with the Lollipop. Because of this we went with bitwarden_rs as our preferred choice.


CryptPad is like Etherpad but with more features, more security and an active group of developers. We've been running it for a long time now and have submitted code changes to the upstream project. We've been treated wonderfully by their developers and they've been supportive of what we're trying to accomplish. We also like the security first approach and that the data is encrypted, even at rest. We use this very heavily for collaborative sessions (like editing this blog post) and it's a fundamental service we self-host and couldn't live with out.

Security Tokens

We are big supporters of hardware security tokens like the NitroKey Storage and Yubikey family. They help ensure that our GPG keys, ssh access and 2FA options are better secured and won't leak easily. Going forward we're moving our principles to Yubikey 4's for their GPG keys, ssh access and U2F setup (for NextCloud and bitwarden_rs in particular). These devices are a bit easier to work with and can be reset in the event of a failure. They are also more forgiving when it comes to forgotten PIN codes...

Going Forward

We've currently deployed private instances of bitwarden_rs, NextCloud and our principles have Yubikey 4's on-hand. We've already moved our new passwords into bitwarden_rs and moved our core data to NextCloud. BOTH are being backed up to secondary storage on our NAS daily to help insulate against disk failures.

Our Yubikey 4's are setup with the appropriate GPG keys, and U2F support. We're currently finalizing some of the SSH access items which will be addressed very soon. We still have SSH access, we just need to get one of the Yubikey's working with SSH prior to making the final swap.

KemoNine and jmf have been working hard to ensure that we don't suffer another failure like we did the other day. Our bitwarden_rs accounts all have access to the necessary Lollipop Cloud passwords. Our NextCloud accounts have access to all of our internal data. Our Yubikeys are guarding our individual GPG keys properly.

We also have a "master account" in bitwarden_rs and NextCloud that's tied to the main data being stored. If either or both of us get locked out of our accounts we can still recover by resetting the "master account" password and then resetting our individual accounts. This approach will help us mitigate the situation we found ourselves in the other day where both KemoNine and jmf were locked out of the core Lollipop Cloud passwords/data.

Lessons Learned

One final point for everyone.

We do NOT regret deploying the NitroKey Storage. As a small project it's a very good choice in hardware to help ensure private data is secure.

However, if you have the option of deploying NextCloud + bitwarden_rs + CryptPad... that's a much more robust choice that can grow and adapt more easily and faster to project needs.

We plan on updating our docs (see our Kanban Documentation Board for now) with information on deploying bitwarden_rs and the latest CryptPad releases. We've already updated our docs with how to deploy our NextCloud containers.

Hopefully those starting new projects in 2019 can leverage our self-hosting approach and just deploy an internal Raspberry Pi 3b or 3b+ running NextCloud, bitwarden_rs and CryptPad and skip a few steps. You can even deploy Gitea too if you don't need public infrastructure.