Last week was amazing. We launched two new Bitcoin integrations, BitSpark and Ziftr, and were overwhelmed with the positive response from our community. Thousands and thousands of people logged in and sent us (and the rest of the world) a strong message: people want Clef in more places and they want it there now. We’re excited to have more integrations launching in the next few weeks to fulfill that desire, so stay tuned.
Unfortunately, the week was not perfect. Our launch with Bitspark had some issues, so we wanted to take the time to explain to our community what exactly happened, what our response was, and how we’re going to do better moving forward. Before we dive into the details, we want to make one thing clear: the issues we faced were specific to the way the Bitspark integration was done and did not impact the security of Clef or any other integrated sites.
On Monday morning, we launched the BitSpark integration. To promote the launch, we sent an email to a segment of our users offering a 2500 bit (~75¢) reward to the first 1,000 users who logged in to BitSpark with Clef. This email, combined with some in-app experiences that drive users towards new sites that use Clef, sent a very high number of users to BitSpark within the first hour. Under the heavy load, the BitSpark website was intermittently down and our service was temporarily degraded. We quickly scaled up our infrastructure and things stabilized.
As our systems stabilized, we noticed a peculiarity with the Clef login to BitSpark: certain users who were logging in with Clef were being inadvertently shown user information that did not belong to them. We initially thought this was dummy information, but with the help of our users, discovered that it was actually information from a early BitSpark user. There was no ability to change this user’s information, withdraw balances, or do any financial damage, but the exposure of personal information was more than enough for us to be concerned.
When we realized that certain users were gaining access to the user information of another BitSpark user, we immediately shut down the Clef integration with BitSpark from our end. Any user who tried to log in with Clef (old or new) was greeted with a message saying the BitSpark integration was temporarily disabled and that we were investigating the issue. As soon as the integration was disabled, we began working with the BitSpark team to figure out what exactly went wrong and how best to proceed.
What was the issue?
After a thorough investigation, we identified the issue and began working with Bitspark to resolve it. The issue was specific to the way the Bitspark integration was done and did not impact the security of Clef or any other integrated sites.
When a user logged in to BitSpark with Clef, our APIs returned a response that correctly identified the authenticated user and securely provided their information. After that response was returned to Bitspark’s internal APIs, however, a logic error in their integration code caused certain users to be misidentified as an early BitSpark user and they were given access to that user’s information. The Bitspark team was quick to find the error, correct the logic, and deploy a fix. While we obviously would have preferred the integration issue to have never happened, their response was outstanding and we’re proud to have them as a partner.
What was our response?
For the last week, we’ve worked diligently with the team at BitSpark to resolve the issue and get things back up and running. Within 24 hours, we had Clef logins working perfectly; however, their full transition to doing remittances only has slowed down deployment of the fixed integration. Because Bitspark is based in Hong Kong and we are based in Oakland, things have taken a little longer than we’d like to ramp back up: while we hoped to have everything resolved in a few days, it has now stretched to a little more than a week. We’re still working with Bitspark to get Clef logins re-deployed and are hoping to have it done by early next week.
How are we going to do better in the future?
At Clef, one of our core values is to Be Better Today Than Yesterday. We’ve been working hard internally to to figure out how we can prevent an issue like this from happening in the future (and respond better if it does). To that end, we have two core improvements that we’ll be making in the coming weeks & months:
1. We are instituting an Integration Audit Program that will ensure Clef integrations are done correctly and prevent issues like this. With hundreds of integrations every day, we can’t do this for every website; however, all large site launches will be audited and if you’re interested in having our engineering team audit your integration, you can send us an email at email@example.com.
2. We did not do as good job as we should have communicating with our users what was going on as we worked to resolve this issue. In the future, we will give updates on critical events at a minimum of every 24 hours until we are able to write a full post-mortem.
Collecting the Bitcoin reward
When we promised 2500 bits to the first 1,000 people to log in to Bitspark with Clef, we were hoping to be able to distribute the rewards within the first 24 hours after logging in. Unfortunately, these issues complicated that distribution process and we’ve had to keep our users waiting. We are working with Bitspark right now to get the wallet addresses for the winning Bitspark accounts created with Clef and will be transferring the 2,500 bits in the next few days (at which point login with Clef will be redeployed). We’ll be sending a follow-up email when we’ve transferred the bits. We’re really sorry for this delay – we can and will do a better job communicating what’s going on in the future.
Clef is much more than just a product to replace passwords. We have 70,000+ websites that rely on our platform, and users all over the world who trust us every day to keep them safe. At every opportunity, we strive to build an organization and community that upholds our values and fulfills the promise of easy security for everyone. Experiences like this show us how we can improve and we promise to always be better.
Thank you for your support!