Citizen Watch Adobe Magento Hyva theme vs. Salesforce Commerce Performance use cases.
Citizen Watch UK eCommerce challenges.
Citizen was entering the next phase of their growth with a strong need to attract new customers and were struggling to find an Adobe Commerce Partner that could deliver the eCommerce solutions that would allow them to do so.
Citexen issues before moving to Hyva.
- Website speed and performance
- Website usability
- Mobile Experience
Agencies decided Citezen that Hyva could resolve all performance issues. Let's check Citezen's website performance and not a useless Google page speed index.
Citezen hosted on the worst possible MAgento solution — Adobe Commerce Cloud, which can’t handle multiple requests without page speed degradation.
We will use the advanced Magento performance test tool on top of K6. It allows us to emulate real load on a website and impact Magento 2 site performance by users. It is not a silly test when pre-cached static HTML is cached by inserting a URL into Google page speed web form.
We made this tool publicly available and open-sourced:
Together with the MOZ tested the performance of over 1000+ Magento 2 websites returned in the search results for 2000 different search queries. In that study, we found a clear correlation between a faster time to the first byte (TTFB) and a higher search engine rank. We can consider good TTFB 200–600ms for uncached pages. Less is better.
https://edgemesh.com/blog/what-is-time-to-first-byte-ttfb
The TTFB metric captures how long your browser takes to receive the first byte of a response from a Magento 2 web server when you request a particular website page not gambled by Full Page Cache. If we are measuring performance with FPC, we are just fooling ourselves. Not all customers will have cached pages, and magento 2 has constant page flushings/invalidation. Murphy's law says you can't trust FPC (“Anything that can go wrong will go wrong.”). FPC is just a good bonus but not a silver bullet. Don’t be scammed by agencies that wanna sell you FPC performance as a website codebase performance. In the graph captured below from our research results, you can see websites with a faster TTFB, in general, ranked more highly than websites with a slower one.
By the way, Citizen is built on microservice architecture when Category/PLP(product listing page) and Search are offloaded to 3-D party none magento microservice Klevu. That was a really smart move. My recommendation to Klevu is to also microservices for Product/PDP(Product Display Page) pages. So magento will be just layout and checkout without broken product listings functionality.
Magento guys are like a monolith. However, clients are smarter and choose microservices.
So, we can test only product page performance because the category page is not magento at all.
The page will be random: https://www.citizenwatch.co.uk/aw1363-06w.html
Also, it's a really smart move for citizens not to use configurable products. All products are simple. Basically, configurable products in magento are performance broken.
To run the test, with 1 user we need run this command:
k6 run magento.js -e url=https://www.citizenwatch.co.uk/aw1363-06w.html -u 1 -i 10 --include-system-env-vars=falseResult:
INFO[0017] Tested URL: https://www.citizenwatch.co.uk/aw1363-06w.html source=console
data_received..................: 6.4 MB 382 kB/s
data_sent......................: 44 kB 2.6 kB/s
http_req_blocked...............: min=299ns med=660ns avg=25.71ms max=257.13ms p(95)=141.42ms
http_req_connecting............: min=0s med=0s avg=2.2ms max=22.07ms p(95)=12.14ms
http_req_duration..............: min=979.75ms med=1.65s avg=1.64s max=2.68s p(95)=2.3s
{ expected_response:true }...: min=979.75ms med=1.65s avg=1.64s max=2.68s p(95)=2.3s
http_req_failed................: 0.00% ✓ 0 ✗ 10
http_req_receiving.............: min=108.93ms med=138.53ms avg=146.04ms max=199.65ms p(95)=193.97ms
http_req_sending...............: min=91.1µs med=122.63µs avg=196.85µs max=534.1µs p(95)=527.01µs
http_req_tls_handshaking.......: min=0s med=0s avg=4.42ms max=44.29ms p(95)=24.36ms
http_req_waiting...............: min=827.71ms med=1.46s avg=1.49s max=2.55s p(95)=2.18s
http_reqs......................: 10 0.599077/s
iteration_duration.............: min=225.15µs med=1.59s avg=1.39s max=2.68s p(95)=2.32s
iterations.....................: 10 0.599077/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1The result TTFB for a single page generation is 2.18s. It's pretty bad for a simple page. Let's check 5 simultaneous user TTFB performances:
INFO[0016] Tested URL: https://www.citizenwatch.co.uk/aw1363-06w.html source=console
data_received..................: 32 MB 2.0 MB/s
data_sent......................: 219 kB 14 kB/s
http_req_blocked...............: min=268ns med=686ns avg=30.48ms max=315.06ms p(95)=302.76ms
http_req_connecting............: min=0s med=0s avg=2.46ms max=26.7ms p(95)=23.47ms
http_req_duration..............: min=982.36ms med=1.39s avg=1.5s max=3.28s p(95)=2.67s
{ expected_response:true }...: min=982.36ms med=1.39s avg=1.5s max=3.28s p(95)=2.67s
http_req_failed................: 0.00% ✓ 0 ✗ 50
http_req_receiving.............: min=113.32ms med=212.76ms avg=221.36ms max=436.78ms p(95)=390.63ms
http_req_sending...............: min=81.98µs med=119.01µs avg=142.13µs max=492.65µs p(95)=309.59µs
http_req_tls_handshaking.......: min=0s med=0s avg=6.47ms max=72.75ms p(95)=64.11ms
http_req_waiting...............: min=787.43ms med=1.15s avg=1.28s max=3.14s p(95)=2.48s
http_reqs......................: 50 3.106568/s
iteration_duration.............: min=218.9µs med=1.4s avg=1.47s max=3.28s p(95)=2.61s
iterations.....................: 50 3.106568/sResult 2.48 seconds per page
let's check 10 users:
INFO[0016] Tested URL: https://www.citizenwatch.co.uk/aw1363-06w.html source=console
data_received..................: 32 MB 2.0 MB/s
data_sent......................: 219 kB 14 kB/s
http_req_blocked...............: min=268ns med=686ns avg=30.48ms max=315.06ms p(95)=302.76ms
http_req_connecting............: min=0s med=0s avg=2.46ms max=26.7ms p(95)=23.47ms
http_req_duration..............: min=982.36ms med=1.39s avg=1.5s max=3.28s p(95)=2.67s
{ expected_response:true }...: min=982.36ms med=1.39s avg=1.5s max=3.28s p(95)=2.67s
http_req_failed................: 0.00% ✓ 0 ✗ 50
http_req_receiving.............: min=113.32ms med=212.76ms avg=221.36ms max=436.78ms p(95)=390.63ms
http_req_sending...............: min=81.98µs med=119.01µs avg=142.13µs max=492.65µs p(95)=309.59µs
http_req_tls_handshaking.......: min=0s med=0s avg=6.47ms max=72.75ms p(95)=64.11ms
http_req_waiting...............: min=787.43ms med=1.15s avg=1.28s max=3.14s p(95)=2.48s
http_reqs......................: 50 3.106568/s
iteration_duration.............: min=218.9µs med=1.4s avg=1.47s max=3.28s p(95)=2.61s
iterations.....................: 50 3.106568/sResult: 2.4 sec per request
Let's cech 20 users:
INFO[0016] Tested URL: https://www.citizenwatch.co.uk/aw1363-06w.html source=console
data_received..................: 64 MB 4.2 MB/s
data_sent......................: 443 kB 29 kB/s
http_req_blocked...............: min=292ns med=709ns avg=68.33ms max=357.45ms p(95)=355.07ms
http_req_connecting............: min=0s med=0s avg=9.91ms max=55.68ms p(95)=53.19ms
http_req_duration..............: min=1.14s med=2.67s avg=2.84s max=6.84s p(95)=5.38s
{ expected_response:true }...: min=1.14s med=2.67s avg=2.84s max=6.84s p(95)=5.38s
http_req_failed................: 0.00% ✓ 0 ✗ 100
http_req_receiving.............: min=139.71ms med=819.98ms avg=980.91ms max=2.06s p(95)=2.02s
http_req_sending...............: min=42.86µs med=107.78µs avg=141.88µs max=688.16µs p(95)=305.92µs
http_req_tls_handshaking.......: min=0s med=0s avg=16.38ms max=95.75ms p(95)=90.17ms
http_req_waiting...............: min=798.55ms med=1.45s avg=1.86s max=6.4s p(95)=4.69s
http_reqs......................: 100 6.511333/s
iteration_duration.............: min=216.01µs med=2.68s avg=2.86s max=7.2s p(95)=5.39s
iterations.....................: 100 6.511333/s
vus............................: 7 min=7 max=20
vus_max........................: 20 min=20 max=20Result: 4,69 seconds per a single request. Magento Citezen Hyva's implementation on Adobe Commerce Cloud can only slowly handle 20 user requests at a time max. Is it much? If you don’t have visitors, it is ok. However, if you have traffic is really low and Hyva doesn’t improve your performance, and will not save your ass except for faster JS downloading time.
Hyva witnesses will say it is still a good performance. All eCommerce is slow and sucks. How can we prove that it is only a Magento issue? Easy! Citezen US has a Demanware Salesforce Cloud website that lets you check the performance of the same page and product using another Demanware e-commerce platform.
Command to test :
INFO[0009] Tested URL: https://www.citizenwatch.com/us/en/product/AW1363-06W.html source=console
█ setup
█ teardown
data_received..................: 3.1 MB 359 kB/s
data_sent......................: 22 kB 2.6 kB/s
http_req_blocked...............: min=260ns med=651ns avg=11.19ms max=111.98ms p(95)=61.58ms
http_req_connecting............: min=0s med=0s avg=1.73ms max=17.35ms p(95)=9.54ms
http_req_duration..............: min=789.33ms med=821.08ms avg=853.63ms max=1.1s p(95)=1.01s
{ expected_response:true }...: min=789.33ms med=821.08ms avg=853.63ms max=1.1s p(95)=1.01s
http_req_failed................: 0.00% ✓ 0 ✗ 10
http_req_receiving.............: min=62.72ms med=80.45ms avg=90.65ms max=134.94ms p(95)=126.44ms
http_req_sending...............: min=98.87µs med=113.51µs avg=164.73µs max=507.79µs p(95)=369.03µs
http_req_tls_handshaking.......: min=0s med=0s avg=4.59ms max=45.91ms p(95)=25.25ms
http_req_waiting...............: min=687.01ms med=734.62ms avg=762.8ms max=968.65ms p(95)=903.1ms
http_reqs......................: 10 1.154569/s
iteration_duration.............: min=181.69µs med=801.91ms avg=721.62ms max=1.21s p(95)=1.04s
iterations.....................: 10 1.154569/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1Result: 900ms. It is just amazing. 2.18 ms Magento performance vs 900 ms Demanware performance!
Let’s check performance under the load:
INFO[0010] Tested URL: https://www.citizenwatch.com/us/en/product/AW1363-06W.html source=console
█ setup
█ teardown
data_received..................: 31 MB 3.4 MB/s
data_sent......................: 227 kB 24 kB/s
http_req_blocked...............: min=246ns med=652ns avg=31.34ms max=176.93ms p(95)=165.99ms
http_req_connecting............: min=0s med=0s avg=5.92ms max=37.06ms p(95)=32.62ms
http_req_duration..............: min=761.93ms med=1.83s avg=1.71s max=2.41s p(95)=2.39s
{ expected_response:true }...: min=761.93ms med=1.83s avg=1.71s max=2.41s p(95)=2.39s
http_req_failed................: 0.00% ✓ 0 ✗ 100
http_req_receiving.............: min=60.49ms med=1.12s avg=940.37ms max=1.45s p(95)=1.42s
http_req_sending...............: min=35.4µs med=112.83µs avg=134.42µs max=693.72µs p(95)=274.02µs
http_req_tls_handshaking.......: min=0s med=0s avg=14.69ms max=89.72ms p(95)=79.41ms
http_req_waiting...............: min=618.6ms med=751.27ms avg=775.4ms max=1.05s p(95)=976.11ms
http_reqs......................: 100 10.761041/s
iteration_duration.............: min=209.08µs med=1.82s avg=1.71s max=2.57s p(95)=2.55s
iterations.....................: 100 10.761041/sResult: 976.11ms. It is just amazing performance under the load, almost the same as the performace of the one user. It is just how well the e-commerce platform must work. 976ms Demanware vs 4690ms Magento TTFB.
I even checked the configured product performance it is the same good.
INFO[0009] Tested URL: https://www.citizenwatch.com/us/en/product/NB6030-59L.html?cgid=mens source=console
█ setup
█ teardown
data_received..................: 3.2 MB 363 kB/s
data_sent......................: 23 kB 2.6 kB/s
http_req_blocked...............: min=284ns med=558ns avg=11.67ms max=116.72ms p(95)=64.19ms
http_req_connecting............: min=0s med=0s avg=1.83ms max=18.39ms p(95)=10.11ms
http_req_duration..............: min=749.83ms med=861.09ms avg=873.75ms max=1.11s p(95)=1.05s
{ expected_response:true }...: min=749.83ms med=861.09ms avg=873.75ms max=1.11s p(95)=1.05s
http_req_failed................: 0.00% ✓ 0 ✗ 10
http_req_receiving.............: min=54.26ms med=68.02ms avg=76ms max=126.06ms p(95)=110.72ms
http_req_sending...............: min=77.83µs med=120.82µs avg=215.8µs max=1.12ms p(95)=692.29µs
http_req_tls_handshaking.......: min=0s med=0s avg=4.67ms max=46.7ms p(95)=25.68ms
http_req_waiting...............: min=677.38ms med=786.08ms avg=797.53ms max=987.47ms p(95)=943.06ms
http_reqs......................: 10 1.127463/s
iteration_duration.............: min=163.62µs med=830.4ms avg=738.96ms max=1.23s p(95)=1.09s How about the Google Page Speed Score of Hyva? Is it good as advertised?
Hyva Performance on Adobe Commerce Cloud
Still not advertised 100+.
Salesforce Cloud
Hyva has a slightly better score but it doesn’t mean faster. Pop-ups can ruin your silly score but not your TTFB performance. But again, this score doesn’t matter. It is not a Magento 2 but JS and Image download performance.
Speed insights is not about actual performance; people often misinterpret it as exactly this. Speed insights only tell you how heavy your website is compared to the blank page on the browser. It depends not on Magento core performance but more on user device Internet connectivity and device CPU performance. This micro-optimization doesn’t help you to make a website that works. However, bad TTFB will definitely affect your website at the time of your success.
Adoby Hyva Cloud vs. Salesforce Commerce Cloud Performance Pivot chart:
Happy fixing Magento 2 performance issues and bugs ;)
