### 1.32.0 release incident **Summary** Between Wed 2023-03-22 09:10 UTC and Thu 2023-03-23 14:10 UTC, for 29 hours, the nearcore mainnet stable release (1.32.0) did not include two security patches previously released in 1.31.1. The patches address two vulnerabilities: one that occurs in block outcome root validation and the second one in total supply validation. These vulnerabilities, if exploited, could lead to state corruption and potentially loss of funds through the Rainbow Bridge and other undesirable outcomes. The incident was brought on by the release of nearcore version 1.32.0. It was caused by the fact that the patch addressing the vulnerabilities, released as 1.31.1 on Tue 2023-02-14, was not applied to the master branch as well. Thus, when the 1.32.0 branch was cut, the patch was not included. Mitigating the issue required making a subsequent nearcore release, 1.32.1, which included the security patches. The incident had no impact on mainnet. Throughout our observations and during the incident, it was noted that nodes without the security patches applied owned less than 33% of the total stake. This structure prevented anyone from exploiting the vulnerabilities. **Root cause description** On Sat 2023-02-11, Through a security alert, Pagoda was made aware of two vulnerabilities. One in block outcome root validation and another in total supply validation. A patch was developed the same day to address the vulnerabilities. The patch was silently distributed to validators who held more than 66% of the mainnet stake. On the same day, the validators applied the patch to their nodes. After securing the mainnet, the patch underwent additional testing via a dedicated branch named 0213_monday_thirteen. before being publicly released. With testing complete, the changes from 0213_monday_thirteen were applied on Tue 2023-02-14 to the current release branch at that point (1.31.0), creating a new 1.31.1 branch. This branch was used to make the public release of the security patches on Tue 2023-02-14. 5 weeks later, on Tue 2023-03-21, the next nearcore release branch was created, 1.32.0. This branch was based on the master branch at that point in time. Since the changes from 0213_monday_thirteen were only applied to the 1.31.1 branch, and not the master branch, the security patches were not included in 1.32.0. After release testing was completed, on Wed 2023-03-22 09:10 UTC, 1.32.0 was released, having a protocol voting date of Sun 2023-03-26 15:00:00 UTC. Validators began updating their nodes to this version, and mainnet started to have a combination of clients using 1.31.1 (patched) and 1.32.0 (unpatched). On Thu 2023-03-23 Pagoda received a report that the 1.32.0 release did not include the security patches from 1.31.1. This was verified to be accurate through a brief analysis Given the identified issues, a branch and release for 1.32.1 were prepared the same day. 1.32.1 was released on Thu 2023-03-23 14:10 UTC. The protocol voting date was kept unchanged compared to 1.32.0. Validators began the process of updating their nodes again, as such mainnet started to have a combination of clients using 1.31.1 (patched), 1.32.0 (unpatched) and 1.32.1 (patched). Upon receipt of the report, we started monitoring the percentage of mainnet stake owned by nodes that ran an unpatched version of nearcore (1.32.0). The percentage remained below 33% until all nodes upgraded to 1.32.1, effectively protecting the consensus from security exploits. **Summary of the corrective actions taken during the incident** Immediately after receiving indication that 1.32.0 did not include the security patches, our investigation immediately commenced (2023-03-23 13:34 UTC). The 1.32 release engineer quickly identified the missing patches and prepared the 1.32.1 release. The internal investigation started on Thu 2023-03-23 13:34 UTC and the 1.32.1 release was made on Thu 2023-03-23 14:10 UTC, in roughly 36 minutes. 1.32.1 was released with a CODE_RED_MAINNET tag to raise awareness and accelerate adoption. In parallel with the 1.32.1 release we also applied the security patches to the master branch and to the 1.33.0 release branch that was created prior to this incident. **These steps were taken to prevent the recurrence of similar issues in the future:** * Create unit tests for known security fixes that were addressed since mainnet was launched. Ensure all unit testing is accomplished as part of the release process. Priority high, ETA 2023-05-12 * Create a process that clearly captures how security patches are handled. Priority high, ETA 2023-05-12 Setup fuzz testing that targets the logic where the recent security vulnerabilities are found. Priority medium, ETA 2023-06-16 * Revise the current release runbook and add a step to guarantee that relevant cherry-picks from prior releases are incorporated into the present one. Priority medium, ETA 2023-06-16 * Create a dashboard showing the % of stake owned by validators running each version present in mainnet. Priority medium, ETA 2023-06-16