Just over a week ago, Bybit was hacked for nearly $1.5bn: Hackers steal $1.5bn from crypto exchange in ‘biggest digital heist ever’. By now, we understand the hack well enough to summarize updated best practices and lessons learned to reduce the likelihood of similar attacks.
But first, a big shout-out to Bybit CEO Ben Zhou and his team for handling this security incident like champs! It was very clear from the beginning and throughout that they were in control and following their incident response playbook. Ben Zhou’s Livestream, which he did about 2h into the incident, is both a good source of information and also a great example of good crisis communications.
Not sure if you can do the same? I recently wrote about this for you: You’ve been hacked, what now?
The Hack
Let me refer to SlowMost for the details, and below I will cover the highlights.
SlowMist: Hacker Techniques and Questions Behind Bybit’s Nearly $1.5 Billion Theft
Bybit’s $1.5 Billion Theft Unveiled: Safe{Wallet} Front-End Code Tampered
Bybit uses a (Gnosis) Safe multisig wallet as its Ethereum cold storage wallet, with several hardware wallets (such as Ledger Nano) as signing devices. Safe is used by thousands of users to store over $100bn crypto assets, and is generally considered quite secure.
The hackers modified Safe’s front-end website and were able to trick all of Bybit’s signers, including CEO Ben, into signing a malicious transaction that handed over control of the wallet to a smart contract owned by the attacker. In detail:
- On the Safe website, the transaction looked exactly as expected. However, under the hood, it used DELEGATECALL to hand over control of the wallet to another smart contract.
- This is because the hackers got into a Safe developer account, and uploaded a malicious code snippet to the website, which was running on AWS.
- Within minutes of signing, the hacker began moving all the Ethereum assets to hundreds of addresses that they controlled. They also removed the malicious code to cover their tracks.
In crypto, once assets are gone, there is no way to get them back without the cooperation of the new owner. In this case, there’s compelling evidence that the hacker is North Korea’s Lazarus hacking group, and they never return what they’ve stolen. Instead, they move the cryptoassets through several steps and addresses, some into bitcoin and some into stablecoins. Bybit has launched an effort to find, freeze and seize as many of the stolen assets as possible through the LazarusBounty program.
Preventing Safe multisig Hacks
There are two basic precautions that could’ve prevented the Bybit hack:
- Clearsigning: when approving crypto transactions, the full transaction details should always be verified on a secure and tamper-proof device. Approvers would have immediately realized that the transaction was malicious, and would not have approved it.
While Bybit used secure and tamper-resistant devices with the Ledger Nano, their screen is very small and not suitable for reviewing all the details of the transaction. Additionally, Safe Multisig is unable to extract and display only the relevant information, leaving approvers drowning in too detail on the small screen. There are workarounds, such as Safe Multisig Transaction Hashes, which are a must, until Safe and hardware wallet vendors improve the clearsigning situation.
To be clear: Bybit’s approvers did have an opportunity to see on their Ledger Nano screens that the transaction was malicious, but they would have had to notice that a single number was 1 (one) instead of 0 (zero). The Safe Hashes tool would have made this much easier to detect. - Supply-Chain Security: for high-security systems, all software components must be 100% fully controlled, including dependencies and internal libraries. Otherwise, attackers can modify any part of the running software and introduce malicious code.
Bybit trusted Safe to keep their service https://app.safe.global/ secure. And Safe, in turn, trusted its developers not to be hacked. Instead, Bybit could have run its own instance of the Safe web application and secured it using with its own resources. In addition, developer credentials and software release processes must ensure that hacked developer laptops cannot lead to changed code in production (change control).
In summary, approvers must always verify the relevant transaction details on the hardware wallet screen, this is known as clearsigning. And organizations must either operate all critical infrastructure in-house, or have very clear security-related SLAs with their service providers (typically ensured by SOC 2 or ISAE 3402 audit reports) to prevent supply-chain attacks.
Other Considerations
Some commentators have claimed that this couldn’t have happened with enterprise-grade cold storage custody (Zodia, Taurus, Fireblocks, and the like). Sure, these institutional custody vendors promise to have clearsigning, supply-chain security and change control in order, and are audited with SOC 2 or ISAE 3402 control reports. However, most if not all of them are closed-source black boxes, and clients need to trust that the processes will continue to work after the auditor leaves. Each organization must carefully evaluate the pros and cons of all these solutions.
Bybit provided immediate and credible assurances that any hacked assets would be fully covered by its treasury, and that no client assets would be affected. It is unclear if the cold wallet risk is kept to a manageable level by design, or if they were just lucky that no other wallets were drained at the same time. Either way; organizations should ensure that at no time can they be drained by more than what can be covered without impacting client funds, i.e. by insurance or their treasury. Measures to achieve this include segmentation of custody systems and limits on the amount of assets that can be in flight at any given time.
All in all, we as an industry have paid a high price for the lesson learned, but I sincerely hope that it will help to make clearsigning the standard a lot faster now, and that all Safe Multisig users will improve their security procedures to prevent further assets flowing to the North Korean regime.
