Adobe Commerce Cloud and Magento 2 benchmark performance summary. $1M disaster!
Many magento merchants and developers were scammed by the new Adobe Commerce Cloud performance white paper. Adobe released a new performance test of the “scaled” infrastructure to scum the merchants.
However, Adobe didn’t realize they showed how broken the Magento 2 platform is. They advertised how many requests Magento 2 and Adobe Commerce can handle; however, you need to see the cost of the resources you need to run this load. Magento 2 core performance was not improved, but an inadequate amount of resources were added. There is nothing about the improvement of the platform.
So what was advertised:
Performance highlights
- Orders — Processed 3,481 orders per minute while maintaining response times of less than 2 seconds for the 99th percentile (99% of the requests were serviced with a response time of less than 2 seconds).
- Page views — Handled over 2 million page views per hour while maintaining response times of less than 2 seconds for the 99th percentile.
- Effective SKUs — The customer profile included 242 million different price variations (eSKUs) for 250,000 products.
- GraphQL requests — System scaled to 10,500 GraphQL uncached requests per minute while maintaining response times of less than 2 seconds for the 99th percentile.
- Concurrent Admin users — System scaled to support 500 concurrent admin users while maintaining response times of less than 2 seconds for the 99th percentile.
Test environment
Performance benchmark results were obtained by testing against an Adobe Commerce 2.4.5 instance deployed in a Pro cloud environment with scaled architecture. The instance also had the Adobe Commerce B2B, Inventory Management, and Adobe Stock Integration modules installed, configured, and enabled.
Performance testing data for the test profile was generated using the Performance Toolkit.
Performance measurements are based on simulated day-to-day store activities for customers and business users. The values reflect a close-to-maximum throughput for each case but do not reflect unique business models, such as private sales or flash sales.
LUMA or Hyva Storefront
- 3000 concurrent users on the storefront
- Set to 30% CDN cache hit rate — previously, adobe tested performance with the 100% hit rate.
- Effective usage of the cache layer increases the number of page views number per hour.
What was advertised:
— 208,000 orders per hour — 58 orders per second
— 2m pageviews per hour — 30 % cached on CDN. 1.7m — 472 visitors per sec.
However, in a given second in that hour, visitors will arrive randomly and there will be fluctuations in the requests. Some seconds will be higher and some seconds will be lower, but with a constant rate averaged over a longer period, such as an hour.
I recommend always tuning your servers to at least x2 RPS, even if your load is only a fraction of that amount.
Estimate the number of visitors you expect for your site. It is important to test with the expected number of users and bots, as having a CPU-bound magento application that cannot scale will cause a bad user experience or even may take down your website. By testing, you can plan for the load and ensure that your site will function as expected.
Analyze the composition of the load. Not all pages place the same load on the server. For example, checkout pages and search pages use more resources than product detail pages. Typically, checkout traffic is only 2% or less. 60% of the modern e-commerce traffic is search.
Let's check how many resources for “scalability” magento 2 needs.
Infrastructure
For the performance benchmark, Adobe Commerce 2.4.5 was deployed on a scalable infrastructure with the following capacity.
Web node specifications
- vCPU 216 (72 x 3 nodes) — there, we can see no autoscaling. Scalable, it is just a separate Database and Web Node instance. It is just three AWS i3.metal instances with the raw price 5.952$ per hour in Frankfurt or $4344.96per month just for one server
Service node specifications
- vCPU 192 (64 x 3 nodes) m5.16xlarge AWS server with the cost $3.072 per hour or $2242.56 per month.
Nominal resources cost per second = $0.15 cents/sec
So we have 406 CPUs and 1.5 terabytes of RAM to handle 470 visitors on vanilla, not customized Magento 2. After customization, these numbers decrease at least twice if development is done well. However, in 90%, development is not good and just a bunch of extensions with a drop of performance drops 3–5 times.
So, a regular magento website requires 406 CPUs to handle 100 visitors or less per second and 20 requests per second. How much resources will cost?
3 * $4344.96 + $2242.56 * 3 = $$$45,832.32 per month or just $$$549,987.84 per year to handle a little bit over not cached 100 visitors per second with the server response time over 2s per page. Magento 2 is just a performance disaster that costs a lot of resources to run it. And it is just a raw AWS resources cost without Adobe interests and fees. Unfortunately, I can’t tell you a price for this infrastructure because Adobe keeps the information about Meganto in secret and sues everyone who tells the truth. However, it will be over $1,000,000 per year. Always verify the cost of the resources and decide if you wanna pay such money for such a bad Adobe Commerce performance.
Building our Magento SaaS platform, we created Serverless Terrafor Magneto infrastructure as a service that can handle 3000+ unchanged(FPC disabled) requests per second with a response time of 350ms vs 2000ms Adobe Cloud at a fraction of the cost (Approximately 1500$ per month). For sure, Magento 2 Core was highly optimized and customized by our Ukrainian team, which is fighting Russian invaders right now. Adobe is still doing business with Russia and Belarus, sponsoring the aggressors.
There is a video of the load test.
I don’t think the agencies can optimize their mage to this level. It is a unique skill to refactor magento. In most cases, agencies know how to sell but not how to write code.
Script of the Magento Less IaaS:
Dolls Kill Magento use case.
Even the successful Dolls Kill store was so excited by Magento's scalability and performance that they decided to go with Shopify.
Was Magento
Now Shopify
Shopify can handle more user requests not because of the resources but because of the optimized Ruby code.
The price of the resources scaling is important, especially for high-traffic websites. Any platform can handle the load; however, the price of infrastructure will differ dramatically. Yes, for small shops without traffic, magento works ok, but for the huge load, it will be an expensive disaster, and neither is it Hyva or Luma.
This article describes a systematic structure to consider the load on your infrastructure and its cost. Simply dividing the total load by the time period will lead to erroneous conclusions on the peak load on the servers. The correct way is to use various factors to arrive at the peak load. This process will ensure that the peak load is representative of the correct load and help you avoid unnecessary trouble during a production launch or a major advertising push.
Agencies are recommending Magento to their clients without realizing they need to know the architecture and capabilities of the system in advance. Also, Adobe has made quite an impressive miss advertising marketing campaign. I’ve noticed that some owners, managers, and eCommerce directors of online stores choose Magento as a platform with no preliminary performance examination of the system and do not realize the real architecture issues of Magento 2. It is not a platform where you can hire some Agencies, cheap overseas freelancers, or even Certified ones to make your store. Agencies can install 100–500 extensions and a new theme and kill your business. Even Adobe can’t fix issues and build a decent hosting platform. What they all can do is a Magento scam. If you think Varnish will fix all issues, you are one of them.
