Incident Post Mortem: February 1st - 2nd, 2021 14/02/2021
On Monday, February 1st, the market saw a surge from XRP to reach $0.7182, surging by 81.2% in a matter of hours and dropping just as fast, and with it, an unprecedented amount of interest from traders across the globe.
First Instance of Degraded Performance
At 3PM Bahrain time, the tidal wave of interest for XRP across the Middle East was in full swing. Rain regrettably experienced critical downtime affecting both our website and mobile platforms caused by a large increase in traffic related to the price movement.
Figure 1. XRP Chart showing the increased volatility. [source: xrpcharts.ripple.com]
Figure 2. Exact time at which Rain was hit with increased traffic.
After round-the-clock efforts from our technical and operational teams, we were back providing a stable trading environment on our mobile and desktop website at 9:30PM with our mobile applications following after at 5AM, February 2nd.
Keeping our customers informed is of the utmost importance to Rain. During the downtime we provided regular updates via our website, social media and support channels to offer guidance as the situation unfolded.
We Will Improve
Offering unwavering availability to crypto-assets during volatile periods is paramount. We strive to meet and exceed our customers’ expectations at all times and offer our sincerest apologies that we were unable to provide the stability you required during this time. Rest assured we are doing all that we can to mitigate any future occurrences of such an event.
It is important to note that during this downtime, there was no impact on the confidentiality of customers’ information, their funds, nor on the integrity of their wallets.
Future Scalability Planning
Based on our analysis during and after the incident, our engineers are reviewing our technical architecture to increase our availability (to ensure we are fast and operational at all times) and increase our scalability (to make sure we can process a quick influx of requests).
Below are some of the improvements we have made since the incident:
- Enhanced efficiency and usage of our database connections; by improving our connection pooling logic, we increased the load our databases can handle by multiple orders of magnitude.
- Refactored internal logic of frequently hit endpoints (such as price information) to reduce response time by roughly 75%.
- Fine tuned caching configurations.
- Further decoupling of internal services.
The Ultimate Stress Test
On Monday, February 8th, following the announcement from Tesla it had bought $1.5 billion worth of Bitcoin in an SEC filing, Bitcoin prices increased by 21.94% to hit an all time high in excess of $47,000. With it, a new swell of excitement swept the market with traders eager to be part of the record breaking pump.
Social media quickly became awash with disgruntled users having issues logging into their crypto-brokerage of choice, and it was clear leading platforms were faced with their own stability issues. But not Rain.
Following our recent enhancements we are pleased to say during this recent price hike, whilst experiencing the same level of traffic and requests as the February 1st surge, our platform and service remained stable and operational at all times with zero downtime encountered.
We take the uptime and performance of our infrastructure very seriously and work hard to support the thousands of customers across the region that choose Rain.