Public Cloud Security Breaches Documenting their mistakes so you don't make them.

FTX Bankruptcy

Date Created: 2023-04-16 Last Updated: 2024-04-13 Post Author:   Chris Farris
 AWS | MultiAccount | Secrets | 

Incident Details

Victimized Company:FTX & its creditors
Incident Dates:2022-11-11 to 2022-11-12
Disclosure Date:2022-11-11
Current Status:Most of the FTX funds have not been recovered. Forensic analysis of FTX is ongoing.

FTX, a crypto-currency exchange, found itself in bankruptcy. At the moment of the leadership transition, over $400 million in crypto-currency was transferred from FTX’s wallets. The FTX trustee management discovered many poor cloud practices during the unwinding process.

Incident

Details of the Incident

After a liquidity crisis, FTX is forced into Chapter 11 bankruptcy, and the United States Bankruptcy Court for the District of Delaware appoints new management. During the chaos of the crisis, new management, and the founders’ departure, an unknown party or parties transfer more than $400M of different cryptocurrencies out of FTX’s control. This is subsequently referred to in court filings as the “November 2022 Breach”. As stated by the new CEO in April:

The Debtors took over responsibility for a computing environment that had been compromised. A malicious actor had just drained approximately $432 million worth of crypto assets in hours; the FTX Group did not have the controls to detect the compromise, much less to stop it; and due to the FTX Group’s deficient controls to secure crypto assets, the Debtors faced the threat that billions of dollars of additional assets could be lost at any moment. (pg 42)

The new management, supported by a number of law firms, Kroll, Chainalysis, and Sygnia were tasked with finding the assets of FTX and 138 associated companies, identifying the creditors, and securing the FTX cloud environment.

Timeline

Date Event
November 2nd, 2022 Questions start to be raised about the balance sheet of one FTX-related company Alameda Research.
November 11th, 2022 FTX Officially files for Chapter 11 Bankruptcy, founder Sam Bankman-Fried resigned as CEO ; John Ray appointed new CEO by the United States Bankruptcy Court of the District of Delaware.
November 11th, 2022 Reports of unauthorized transactions surface in the crypto-currency community. Employees of FTX acknowledge a potential compromise and urge customers not to use the platform
November 12th, 2022 FTX’s new management acknowledges unauthorized transfers of crypto-currency and begins to move funds to off-line (cold) wallets.
November 17th, 2022 John Ray files an initial statement with the Bankruptcy Court stating, “Never in my career have I seen such a complete failure of corporate controls and such a complete absence of trustworthy financial information as occurred here.”
April 9th, 2023 FTX publishes “First Interim Report of John J. Ray III to The Independent Directors on Control Failures at the FTX Exchanges,” which outlines the bulk of the cloud security issues documented here.
August 25th, 2023 Kroll, the firm that was contracted to manage FTX Creditors, suffers breach due to SIM Swapping attack. Apparently scammers are using this to conduct phishing attacks against FTX creditors.
January 24th, 2024 Three individuals were indicted in the case of the missing $400M in crypto-currency
March 28th, 2024 FTX’s CEO, Sam Bankman-Fried, was sentenced to 25 years in prison resulting from his conviction of multiple counts of fraud

Attribution / Perpetrator

The identity of the perpetrator of the “November 2022 Breach” has not been identified. The DOJ has indicted three individuals for SIM Swapping in relation to a $400M crypto-currency theft that occurred around this time.

Sam Bankman-Fried has been convicted on multiple counts related to his management of the FTX companies. He was sentenced to 25 years in prison, and ordered to forfeit $11 billion. A number of other senior-level employees of the company, including Bankman-Fried’s co-founders, are cooperating with the Department of Justice.

Summary of coverage & other resources

Cloud Security lessons learned

The following Cloud Security lessons come primarily from John J. Ray III’s First Interim Report, and are cited by page number.

  1. MultiAccount Strategy - FTX primarily kept all private keys in the same AWS account with over a thousand compute workloads

    “the FTX exchanges and Alameda used a single, shared AWS account, meaning that a compromise of that AWS account would expose all three entities’ assets to misuse or theft.” (pg 37)

    “The FTX Group stored the private keys to its crypto assets in its cloud computing environment, which included over one thousand servers and related system architecture, services, and databases that it leased from Amazon Web Services.” (pg 24)

    “The FTX Group appears to have recognized the deficiency, because as of the petition date, FTX.US had begun a process of migrating to its own dedicated AWS account; because it did not complete that work, its assets remained within the shared account such that FTX.US lost approximately $139 million of its crypto assets during the November 2022 Breach.” (footnote 34 pg 33)

  2. Wallet Keys stored in Secrets Manager

    “private keys to billions of dollars in crypto assets were stored in AWS Secrets Manager … any of the many FTX Group employees who had access to AWS Secrets Manager or the password vault could access certain of the keys and unilaterally transfer the corresponding assets.” (pg 28)

    Footnote: “In the infrequent instances in which the FTX Group stored private keys in encrypted form, it stored the decryption key in AWS Secrets Manager and not in a protected form, such as HSM. As a result, the decryption keys could easily be retrieved by an unauthorized actor, thereby dramatically reducing the value of encryption” (pg 28)

  3. Secrets Mismanagement

    “the passwords for encrypting the private keys of wallet nodes were stored in plain text, committed to the code repository…, and reused across different wallet nodes such that if one were compromised, every other node with the same password could be compromised as well. Furthermore, wallet node servers were not securely segregated from connected servers such that anyone who compromised the FTX Group’s computing environment could potentially compromise its wallet nodes. (pg 29-30)

  4. MFA

    The FTX Group did not enforce the use of MFA in connection with two of its most critical corporate services—Google Workspace, its primary tool for email and document storage and collaboration, and 1Password, its password-management program. (pg 31)

  5. Lack of Monitoring or other security controls

    “the FTX Group did not have any mechanism to identify promptly if someone accessed the private keys of central exchange wallets holding hundreds of millions or billions of dollars in crypto assets, and it did not fully enable even the basic features offered by AWS to assist with cyber threat detection and response.” (pg 34)

    “the FTX Group did not learn of the November 2022 Breach until the Debtors’ restructuring advisor alerted employees after observing, via Twitter and other public sources, that suspicious transfers appeared to have occurred from FTX Group crypto wallets.” (pg 34)

    “The FTX Group similarly failed to institute any basic mechanism to be alerted to any “root” login to its AWS account” (pg 34)

    “For example, Amazon GuardDuty, an AWS feature that supports threat detection, was not enabled at all on FTX.com, and across the entities, VPC flow logs that can capture IP traffic information were only enabled to log the rejected traffic (and only in some networks)—they were not enabled to log the permitted traffic at all. The lack of these and other logs complicated the Debtors’ investigation of the November 2022 Breach.” (Footnote 36, pg 34)