From Near Loss to Victory: The $2M Blockchain Rescue
Our team recently encountered an emergency where $2M was at risk, and we immediately mobilized to secure the assets. This post-mortem analysis shares our journey, the challenges we faced, and the steps we took to safeguard funds from further exploitation.
Alert! Unusual Transactions
On December 8th, at 12:24 AM UTC, our monitoring system detected a malicious transaction on Polygon, where a hacker transferred out approximately $140k worth of USDC from the smart contract 0x829363736a5a9080e05549db6d1271f070a7e224.
Later that same day, at 2:53 PM UTC, the same contract triggered our alarms again, this time for a significant influx of assets into the vault.
The unusual signal caught our attention. The incoming sum was close to what had been stolen earlier. Could the hacker have just returned the stolen assets? We took a closer look at the smart contract. The first transaction seemed to be an attack related to the forwarded multicall—a novel vulnerability ThirdWeb had only recently disclosed. We have observed numerous similar attacks on-chain. The stolen funds had been swapped to ETH and bridged to Ethereum.
Code Red: The Race to Safeguard the Vault
The incoming USDC was not from the attacker. There were a few users continuing to deposit into the vault. It became clear that the DeFi project associated with the vault was still active; the owner and users seemed oblivious to the breach.
The spider-sense of a hacker warned us something bad might happen again. Around $124K USDC was still at high risk, and we suspected that other predators in the ‘dark forest’ might be on the prowl. We had to take immediate action to prevent any black hat hackers from seizing the funds!
However, what we knew about the vault was limited to only the source code of its smart contract and some addresses that had interacted with it. We didn’t know anything about the project behind the vault! Identifying and alerting the responsible team discreetly and promptly, without compromising the endangered assets, was a complex task. We were left with two options: fix the flaw or extract the funds before it was too late.
Failed Attempts to Mitigate the Bug
Based on our understanding of the root cause of the vulnerability, we knew there was a trusted forwarder in the vulnerable vault that could allow someone to counterfeit the msgSender() of internal transactions. For those interested, you can delve deeper into this type of bug.
An effective mitigation of this bug is to disable the trusted forwarder. The function setTrustedForwarder(address) was designed for this purpose. It requires the caller to be a specific governance address, but we could manipulate the caller identity!
Our plan was to leverage the bug to set the TrustedForwarder to a null address. We spent some time debugging based on this idea. yet, we missed a critical detail: the onlyGovernance() modifier didn’t rely on the compromised _msgSender(); instead, it validated using msg.sender directly.
Replaying the Exploit of the Original Hacker
Our initial misdirection cost us time, but fortunately, the funds remained untouched. Another hack could happen at any time! We decided to preemptively execute a white-hat hack to rescue the funds. This meant exploiting the very same bug to preempt any malicious attempts, an approach that's unconventional in the real world, but very common in the web3 world, where swift action is often crucial and self-reliance is key.
The most effective strategy is to replicate the original hacker’s transaction. Because the nonce of the signed inner transaction from the initial breach is not reusable, we have to replace the hacker’s address to our own and craft new signatures for the embedded transactions.
Bypassing the Ineffective Mitigation
We quickly tested and deployed our exploit code, however, it didn’t work on-chain! Had the funds already been moved? It was then we noticed this transaction:
Someone had paused the project! No further deposits or withdrawals could be made from the vault. It’s triggered from an address with the “Guardian” role. Great! The project owner was now clearly aware and responding to the emergency!
But did a pause mean secure? We didn't think pausing is a legit fix of the bug. Because the attacker can impersonate any user, including the “Guardian”, and simply unpause the project.
The potential workaround of the mitigation is pretty straightforward. The original hacker might have had a deeper understanding of the project's workings than us, but we had the advantage of speed. Rapidly, we deployed a refined exploit that included an unpause command, allowing us to resume operations and move all the at-risk funds to our secure vault. Finally relieved!
Onchain Guidance
We didn’t know who the owner of the vault was, but he must be desperate. So, we tried to deliver messages to him.
We published a message to ourselves, stating:
Don't worry, we are white-hat hackers. We will return the fund.
We sent a message to the Governance address, teaching the owner the correct patch of the bug:
Where should we return the funds to? First, you should disable the forwarder by 'setTrustedForwarder(0)'
Remarkably, within 10 minutes, the Governance executed a transaction to reconfigure the trusted forwarder to itself.
Finally, the bug is patched! We have time to delve further into the source code while waiting for responses from the governance.
Digging Beyond the Surface
Let’s check the huge inflow of $124k USDC first. Where was it from?
It’s processed by L2WormholeRouter and L2BridgeEscrow. Some components of this project are on another chain. The funds we transferred were just a portion of its TVL.
We got some clues from the wormhole router contract. By searching the keywords L1_FUND_TRANSFER_REPORT, we tracked down to a DeFi project Affine.
Gradually, we pieced together Affine's architecture:
Affine is a cross chain yield aggregator. It pools deposits on L2 (Polygon) and sends funds to L1 (Ethereum), maintaining an investment ratio of 47:3.
Since the manager on L2 has no direct knowledge of L1 status, a bot with the role “Harvester” is scheduled to synchronize the two layers via the wormhole network.
Upon receiving updated TVL data from L1, L2 decides whether to rebalance the portfolio by either withdrawing from or sending extra funds to L1.
After the initial hacker draining the vault on L2, the portfolio of Affine is heavily imbalanced. Consequently, the bot initiated a rebalance, prompting L1 to send funds back to L2. With over $2M on L1, roughly $120k - 6% of TVL - was transferred due to this automation.
Could we initiate a rebalance by ourselves? The rebalance is only triggered by a public function receiveTVL() in L2Vault, which requires the _msgSender() to be the L2WormholeRouter. Given the exploitability of _msgSender(), the hacker could have forced the L2 to rebalance and request the L1 to move all the 2M funds back to L2!
We were really surprised that our intervention had safeguarded not merely $110k but $2M!
Meanwhile, we realized what we and the hacker had done in the exploit. We had burned and withdrawn shares on behalf of a major Liquidity Provider (LP) in the pool. The exploit had not affected others. With this insight, it became evident that we needed to return the funds to the LP's address.
Closing
Despite the absence of an on-chain response, we had identified the project's owner. We took to Twitter, broadcasting our findings and tagging @AffineDefi in our post:
Upon their confirmation of the LP's address, we promptly returned the funds.
As a result, we were awarded with a $10k bounty for the white-hat hack.
With the funds now secure and the vulnerability patched, our work comes to a close. But our mission is not over. We stand ready to confront emerging challenges, ever committed to safeguarding the integrity of the blockchain ecosystem.
Timeline
2023-12-08 00:24 UTC - First attack detected.
2023-12-08 14:53 UTC - Alerted to a large influx of funds.
2023-12-08 15:30 UTC - Initiated an investigation into the unusual activity.
2023-12-08 16:40 UTC - Transferred funds to our secure vault for protection.
2023-12-08 17:09 UTC - Disclosed our status as white-hat hackers.
2023-12-08 17:20 UTC - Sent on-chain messages to the project owner.
2023-12-08 19:54 UTC - Posted a tweet requesting direct messaging with Affine DeFi.
2023-12-08 20:03 UTC - Affine DeFi reached out to us on Twitter.
2023-12-08 20:21 UTC - Detailed our actions in securing $110k USDC.
2023-12-08 22:24 UTC - Affine DeFi confirmed the plan for a public recovery.
2023-12-08 22:34 UTC - Returned the funds to the user.
2023-12-08 23:10 UTC - Received a bounty from Affine DeFi for our white-hat hacking efforts.