Autoscaling Magento Cloud Terraform Infrastructure as a code

Yegor Shytikov
8 min readMay 12, 2020

This post is based on the Magento 2 Cloud Terraform Autoscalable infrastructure as code for AWS Public Cloud git repository:

This Infrastructure is the result of years of experience scaling Magento 1 and 2 in the cloud. It comes with the best cloud development practices baked right in saving your business time and money.

Migrating Magento into the AWS cloud — comes with many advantages. You can operate workloads in new ways. When you only pay for what you use and add resources within minutes using auto-scaling.

Leverage your own AWS Account dramatically reduces your monthly spend instead of paying to an expensive managed hosting provider (PaaS, SaaS).

This Script can be used with any eCommerce/Web Platform WordPress, WooCommerce, Drupal, Shopware 6, Shopify APP, VueStorefront, Silyus, Oddo, ORO not just with Magento. It has the name Magento because it was designed for Magento at first. But I know projects they use it to run Enterparaci Java applications with auto-scaling. How to setup software into the cloud you can read below.

eCommerce Terraform deployer for Magento, WooCommerce, ShopWare, PrestaShop, Sylius automates deployments for key enterprise workloads on the AWS cloud. It launches, configures, and runs the AWS compute, network, storage, database, and other services required to deploy a specific workload on AWS, using best industry practices for security and availability.

This OpenSource project covers the implementation of Magento 2 Community/Enterprise Edition in a highly available architecture on the AWS cloud, using AWS services such as Amazon EC2, Amazon VPC, and Amazon RDS. Magento is an open-source content management system for e-commerce websites. This automated deployment builds a cluster that can run applications on any programming language (PHP, RUBY, JAVA, PYTHON) and CMS like Magento.

Enterprise Support/Installation/Development Package available.

Several Magento development Agencies select this custom cloud solution for their clients and they are willing to provide services/support for businesses based on this Open Source project. Nowadays this project has 10+ partners

More information: yegorshytikov@gmail.com

I also Have Ansible Magento Cloud provisioning implementation: https://github.com/Genaker/AWS_Magento2_Ansible

AWS Magento 2 Cloud Features:

  • True Horizontal Auto Scaling
  • Affordable(starting from ~300$ for us-west-2 region)
  • MySQL RDS scalable Managed by Amazon, multi az failover, vertical scaling without downtime
  • Compatible with RDS Aurora Cluster and Aurora Serverless
  • EFS — Fully managed elastic NFS for media and configuration storage
  • CloudFront CDN for static and media served from different origins S3 or Magento(EFS) as the second origin
  • Automatically back (point-in-time snapshot) up your code and databases for easy restoration.
  • 99.9 Uptime, multi az high availability
  • High security (Security groups, private infrastructure)
  • Network Address Translation (NAT) has Elastic(Static) IP and used for internet access for all EC2 instances.
  • Bastion host to provide Secure Shell (SSH) access to the Magento web servers.
  • Appropriate security groups for each instance or function to restrict access to only necessary protocols and ports.
  • Private Public Subnets — NAT gateway, Bastion server
  • All servers and Database are hosted in a private Network securely
  • System and Software Update Patches
  • DDoS Protection with AWS Shield
  • PCI compliant infrastructure
  • Redis cluster
  • Amazon Elasticsearch Service — Elasticsearch at scale with zero downtime with built-in Kibana
  • Different Application Scaling Groups (ASG)
  • Application Load Balancer(ALB) with SSL/TSL termination, SSL certificates management
  • ALB Path-Based Routing, Host-Based Routing, Lambda functions as targets, HTTP header/method-based routing, Query string parameter-based routing
  • Scaled Varnish ASG
  • Dedicated Admin/Cron ASG
  • You can easily add new autoscaling groups for your needs (Per WebSite/for Checkout requests/for API), just copy-paste code
  • Possibility to run the same infrastructure on Production/Staging/Dev environment, different projects
  • Automatic CI/CD (CodePipeline/CodeDeploy) deployments possible
  • AWS CodeDeploy In-place deployment, Blue/green deployment from Git or S3, Redeploy or RollBack
  • Deploying from a Development Account to a Production Account
  • Amazon Simple Email Service (Amazon SES) — cloud-based email sending service. Price $0.10 for 1K emails
  • Amazon CloudWatch — load all the metrics (CPU, RAM, Network) in your account for search, graphing, and alarms. Metric data is kept for 15 months.
  • CloudWatch alarms that watch a single CloudWatch metric or the result of a math expression based on CloudWatch metrics and send SMS(Text) Notifications or Emails
  • Simple and Step Scaling Policies — choose to scale metrics that trigger horizontal scaling.
  • Manual Scaling for Magento Auto Scaling Group (ASG)
  • AWS Command Line Interface (CLI) — tool to manage your AWS services. You can control multiple AWS services from the command line and automate them through scripts.
  • DynamoDb for logs, indexes, analytics
  • Lambda functions as targets for a load balancer
  • Elastic Container Registry (ECR) — fully-managed Docker container registry that makes it easy to store, manage, and deploy Docker container images!
  • You can use Amazon Elastic Container Service (ECS) instead of ASG with Service Auto Scaling to adjust running containers desired count automatically.
  • Awesome AWS documentation is Open Source and on GitHub
Magento Cloud infrastructure

Cloud Flat View

Magento Cloud Infrastructure

The infrastructure consists of multiple layers (autoscaling, alb, rds, security-group, vpc) where each layer is configured using one of Terraform AWS modules with arguments specified in terraform.tfvars in layer's directory.

Terraform uses this during the module installation step of terraform init to download the source code to a directory on local disk so that it can be used by other Terraform commands.

Public Terraform registry provides infrastructure modules for many infrastructure resources at registry.terraform.io.

Terragrunt is used to work with Terraform configurations which allows to orchestrate dependent layers, update arguments dynamically and keep configurations. Define Terraform code once, no matter how many environments you have. [DRY] (https://en.wikipedia.org/wiki/Don%27t_repeat_yourself).

Pre-requirements

Install HomeBrew on Linux

Paste at a terminal prompt:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

The installation script installs Homebrew to /home/linuxbrew/.linuxbrew using sudo if possible and in your home directory at ~/.linuxbrew otherwise. Homebrew does not use sudo after installation. Using /home/linuxbrew/.linuxbrew allows the use of more binary packages (bottles) than installing in your personal home directory.

test -d ~/.linuxbrew && eval $(~/.linuxbrew/bin/brew shellenv)
test -d /home/linuxbrew/.linuxbrew && eval $(/home/linuxbrew/.linuxbrew/bin/brew shellenv)
test -r ~/.bash_profile && echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.bash_profile
echo "eval \$($(brew --prefix)/bin/brew shellenv)" >>~/.profile

You’re done! Try installing a package:

brew install hello

If you’re using an older distribution of Linux, installing your first package will also install a recent version of glibc and gcc. Use brew doctor to troubleshoot common issues.

If you are using Mac you can install all dependencies using Homebrew:

$ brew install terraform terragrunt pre-commit

Manual install:

You can install Terragrunt manually by going to the Releases Page, downloading the binary for your OS, renaming it to terragrunt, and adding it to your PATH.

Install Terragrunt and Terraform Ubuntu Manual

sudo -s; ## run as a super user
export TERRAFORM_VERSION=0.12.24 \
&& export TERRAGRUNT_VERSION=0.23.2 \
&& mkdir -p /ci/terraform_${TERRAFORM_VERSION} \
&& wget -nv -O /ci/terraform_${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip \
&& unzip -o /ci/terraform_${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d /usr/local/bin/ \
&& mkdir -p /ci/terragrunt-${TERRAGRUNT_VERSION}/ \
&& wget -nv -O /ci/terragrunt-${TERRAGRUNT_VERSION}/terragrunt https://github.com/gruntwork-io/terragrunt/releases/download/v${TERRAGRUNT_VERSION}/terragrunt_linux_amd64 \
&& sudo chmod a+x /ci/terragrunt-${TERRAGRUNT_VERSION}/terragrunt \
&& cp /ci/terragrunt-${TERRAGRUNT_VERSION}/terragrunt /bin \
&& chmod a+x /bin/terragrunt \
&& rm -rf /ci \
&& exit

Test Terragrunt/Terraform installation(Optional):

terragrunt -v;
terraform -v

How to use it?

Step 0.

Terraform use the SSH protocol to clone the modules, configured SSH keys will be used automatically. Add SSH key to github account. (https://help.github.com/en/enterprise/2.15/user/articles/adding-a-new-ssh-key-to-your-github-account)

Git+SSH is used because it works for both public and private repositories.

Step 1.

Set credential. By default, access credentials to AWS account should be set using environment variables:

export AWS_DEFAULT_REGION=us-west-1 ## change it to your preferable AWS region
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...

Alternatively, you can edit common/main_providers.tf and use another authentication mechanism as described in AWS provider documentation.

Step 2.

Once all arguments are set, run this command to create infrastructure in all layers in a single region:

$ cd production
$ terragrunt apply-all

Alternatively, you can create infrastructure in a single layer (eg, autoscaling_3):

$ cd production/autoscaling_3
$ terragrunt apply

See official Terragrunt documentation for all available commands and features.

Demo video how it works

https://www.youtube.com/watch?v=kmnlrXSTQlM

Architecting your Magento platform to grow with your business can sometimes be a challenge. This video walks through the steps needed to take an out-of-the-box, single-node Magento implementation and turn it into a highly available, elastic, and robust deployment. This includes an end-to-end caching strategy that provides an efficient front-end cache (including populated shopping carts) using Varnish on Amazon EC2 as well as offloading the Magento caches to separate infrastructure such as Amazon ElastiCache. We also look at strategies to manage the Magento Media library outside of the application instances, including EFS shared storage solutions.

Destroy Terragrunt Magento Infrastructure

terraform destroy-all 

Infrastructure managed by Terraform will be destroyed. This will ask for confirmation before destroying it.

Magento Cloud

Production and staging environments

You can copy/paste the folder to create a new environment. Consider the following file structure, which defines three Magento environments (prod, project-3, stage) with the same infrastructure in each one (an app, a MySQL database, and a VPC):

└── magento
├── prod
│ ├── app
│ │ └── main.tf
│ ├── mysql
│ │ └── main.tf
│ └── vpc
│ └── main.tf
├── project-3
│ ├── app
│ │ └── main.tf
│ ├── mysql
│ │ └── main.tf
│ └── vpc
│ └── main.tf
└── stage
├── app
│ └── main.tf
├── mysql
│ └── main.tf
└── vpc
└── main.tf

The contents of each environment will be more or less identical, except perhaps for a few settings (e.g. the prod environment may run bigger or more servers). As the size of the infrastructure grows, having to maintain all of this duplicated code between environments becomes more error-prone. You can reduce the amount of copy-paste using Terraform modules, but even the code to instantiate a module and set up input variables, output variables, providers, and remote state can still create a lot of maintenance overhead.

Terragrunt allows you to keep your Magento backend configuration DRY (“Don’t Repeat Yourself”) by defining it once in a root location and inheriting that configuration in all child modules. Let’s say your Terraform code has the following folder layout:

stage
├── frontend-app
│ └── main.tf
└── mysql
└── main.tf

To use Terragrunt, add a single terragrunt.hcl file to the root of your repo, in the stage folder, and one terragrunt.hcl file in each module folder:

stage
├── terragrunt.hcl
├── frontend-app
│ ├── main.tf
│ └── terragrunt.hcl
└── mysql
├── main.tf
└── terragrunt.hcl

Now you can define your backend configuration just once in the root terragrunt.hcl file!

Video of how terraform AWS infrastructure as a code works :

Magento Multi-cloud deployments

Terraform provides Magento 2 Open Source Cloud infrastructure as a code approach to the provision and manage any cloud (AWS, GoogleCloud, Azure, Alibaba, or other types of services such as Kubernetes).

Terraform can manage popular service providers, such as AWS, GCP, Micosoft Azure, Alibaba Cloud, and VMware, as well as custom in-house and on-premises solutions.

--

--

Yegor Shytikov

True Stories about Magento 2. Melting down metal server infrastructure into cloud solutions.