The number of financial transactions on the Pismo payment processing platform soared 60% during Black Friday, compared to an average across the three other Fridays in November. We processed 5.6 million transactions on Black Friday, an all-time record for our platform!
Our monthly transaction value reached US$ 4.1 billion in November. Additionally, our platform monitoring system recorded more than one billion events – the simple operations that comprise a transaction. Once more, the scalability of our platform was hard tested in a real-world situation, and we are proud to say it worked flawlessly!
26th November was 2021’s official Black Friday, held directly after the American Thanksgiving holiday. In many countries, it’s the most important day of the year for retail commerce, with the highest sales volumes. And many retailers offered discounts throughout the whole month to attract shoppers.
Of course, our payments platform had to be well prepared for this considerable demand peak — and it was. The preparation started more than a month before Black Friday. And it required actions from several engineering squads at Pismo.
Here are a few preparation measures we took:
The clients’ vision
Before starting the preparations for Black Friday, we talked to our largest customers, which would likely increase their transaction volumes. They shared their estimations in a few scenarios — from a conservative forecast to an optimistic one. Then we used these numbers to start planning our Black Friday operations.
Of course, we wanted to ensure that our platform would be ready to cope in the most optimistic scenario, corresponding to an upsurge of 65% in the number of transactions. We also maintained direct communication with our clients’ technical teams — just in case we needed to sort out any technical issues.
We mapped which of our technical squads Black Friday would impact. This included teams related to payments, banking, sellers management, and transaction authorisations. Then, based on the forecasts received from our clients, we planned actions for each of these squads.
Many components of our platform scale automatically when demand rises. But, we also made manual adjustments to ensure we would have the best possible infrastructure configuration. We increased the number of Kubernetes pods for some applications. We also used Terraform, our infrastructure-as-service tool, to provide extra resources for databases.
We performed load tests on the main components of our platform to make sure they would stand the increased workload. Additionally, we used chaos engineering to simulate a variety of failures in infrastructure and specific components. Our goal was to ensure that our platform would operate normally in the unlikely event any of these failures occurred.
We also revised our contingency plans so that all the squads would know what to do if a critical tool failed. For example, if our system monitoring tool crashed, our engineers had a plan to use alternative means to gather data and keep track of how the platform was working.
Avoiding unpleasant surprises
To reduce the risk of any unforeseen glitches, we recommended that our squads avoid making significant updates on the platform during the Black Friday week. We didn’t decree a code freeze, meaning we could still do minor updates if needed.
We also rescheduled non-vital batch jobs to prevent them from running while the platform would be operating with an increased workload (You may call this a precautionary excess!).
Everything ran like clockwork
All our squads had engineers on duty during Black Friday, monitoring our platform continuously and ready to act if needed. But the operation went so smoothly that we decided to demobilise the team ahead of schedule on Saturday.
Once more, we proved that our platform is highly reliable and scalable. We look forward to the next Black Friday and to breaking even more records!