Magento 2 AWS Cloud Terraform deployment and Varnish cache problem
If you deploy Magento on AWS, then you will be interested in this update which I received from Yehor Shytikov. He let me know of a number of updates to some open source projects :
and
that help set up Magento on AWS infrastructure with auto-scaling using AWS Cloud and Terraform. A new minimal branch helps you to deploy in a minimal infrastructure without redundant complexity like Varnish cache server.
NEW Magento Minimal Cloud Terraform infrastructure release. It supports the latest versions of the Terraform 1.0+ & Terragrunt 0.35+
This KISS infrastructure is designed for small, medium and extra-large merchants. It can handle almost any load. Also, it dramatically reduces management overhead and cost. Magento Fork makes Varnish the redundant solution for 98% of the merchants and is no longer the best practice.
Varnish requires additional load balancers, configuration, and EC2 server. Also, Varnish is not scalable because it keeps all the data in the shared memory it can’t be shared via network. Also, it has bad architecture — thread per request. 500 requests require 500 threads with a huge overhead. NodeJS works better for Magento cache than Varnish. When the cache hit ratio is very high (i.e. 100%). Varnish’s default configuration of thread_pool_max is too high. When there are too many worker threads, Varnish spends an inordinate amount of time in system call space.
After fixing Magento core performance issues Magento Redis can handle 1000+ cached requests per second without any redundant caches and is scalable. You are scaling the Cache layer together with the web layer. Amazing!
Varnish Cache is an HTTP server with an HTTP backend that can serve files. It has a threaded architecture, but no event loop. Because the written code can use blocking system calls, it’s easier to use than NodeJS or NGINX, where you have to deal with an event loop.
Varnish Cache has a weird way of logging to shared memory, not disk. It’s designed this way because logging 10,000 HTTP transactions per second to rotating hard drives is very expensive.
If 3,000 more users are being added to the waiting list every second, after three seconds, 9,000 users are waiting for the backend to deliver the content. Initially, Varnish would take that piece of content, talk to its 9,000 threads, and try to push it out all at once. We called this scenario a thundering herd, and it instantly killed the server.
Magento 2 FPC Varnish scalability issue
When your site gets swamped by a huge amount of visitors, tools like Varnish can improve scalability for static sites, but when user-specific content is needed, a hit to the backend webserver is still needed, causing scalability issues.
This Terraform release happened thanks to AWS Community Hero and Terraform Master: Anton Babenko! 90% of the infrastructure was built using his fantastic modules.
Amazon Web Services (AWS) Terraform offers a Magento 2 eCommerce cloud hosting (PaaS) solution to small and large businesses that want a flexible, secured, highly scalable, and low-cost solution for online sales and retailing.
AWS offers benefits such as being easy and simple to get started and get running. That will increase the developer productivity and enable automatic scaling, because everything, almost everything, is managed for you by AWS.
All you need to do is just provide your code and configure a little bit your environment, yet you have complete control, unlike high-cost and low-performance services such as Magento Commerce Cloud, which is owned by Adobe. In Magento Cloud, you have locked into the vendor’s bad architecture (single instance has all layers of the applications, no autoscaling, no PSI DSS and HIPAA compliance), but with AWS, if you need that granular, low-level control, you can just go and configure everything manually, everything is available to you under your AWS account.
