Installing and optimizing Magento 2 on RedHat/AWS Linux 2

How To Install Magento to AWS Linux.

Why use Amazon Linux?

AWS Linux has an Extras mechanism in Amazon Linux 2 to enable the consumption of new versions of application software. Extras help alleviate the compromise between the stability of the OS and the freshness of available software. For example, now you can install newer versions of MariaDB on a stable operating system supported for five years. Examples of extras include Ansible 2.4.2, Memcached 1.5, PHP.7+, Nginx 1.12, Postgresql 9.6, MariaDB 10.2, Go 1.9, Redis 4.0, R 3.4, Rust 1.22.1

Open Source EC2 AWS Linux 2 / Centos 8 Magento 2 installation script:

Genaker/Magento-AWS-Linux-2-Installation

Available AWS Linux packages:

0 ansible2 available \
 [ =2.4.2 =2.4.6 =2.8 =stable ]
 2 httpd_modules available [ =1.0 =stable ]
 3 memcached1.5 available \
 [ =1.5.1 =1.5.16 =1.5.17 ]
 5 postgresql9.6 available [ =9.6.6 =9.6.8 ]
 6 postgresql10 available [ =10 ]
 8 redis4.0 available [ =4.0.5 =4.0.10 ]
 9 R3.4 available [ =3.4.3 =stable ]
 10 rust1 available \
 [ =1.22.1 =1.26.0 =1.26.1 =1.27.2 =1.31.0 =1.38.0 ]
 11 vim available [ =8.0 ]
 13 ruby2.4 available \
 [ =2.4.2 =2.4.4 =2.4.7 =stable ]
 _ 
 14 php7.2 available \
 [ =7.2.0 =7.2.4 =7.2.5 =7.2.8 =7.2.11 =7.2.13 =7.2.14
 =7.2.16 =7.2.17 =7.2.19 =7.2.21 =7.2.22 =7.2.23
 =7.2.24 =7.2.26 =stable ]
 _ lamp-mariadb10.2-php7.2 available \
 [ =10.2.10_7.2.0 =10.2.10_7.2.4 =10.2.10_7.2.5
 =10.2.10_7.2.8 =10.2.10_7.2.11 =10.2.10_7.2.13
 =10.2.10_7.2.14 =10.2.10_7.2.16 =10.2.10_7.2.17
 =10.2.10_7.2.19 =10.2.10_7.2.22 =10.2.10_7.2.23
 =10.2.10_7.2.24 =stable ]
 18 libreoffice available \
 [ =5.0.6.2_15 =5.3.6.1 =stable ]
 19 gimp available [ =2.8.22 ]
 20 docker=latest enabled \
 [ =17.12.1 =18.03.1 =18.06.1 =18.09.9 =stable ]
 21 mate-desktop1.x available [ =1.19.0 =1.20.0 ]
 22 GraphicsMagick1.3 available \
 [ =1.3.29 =1.3.32 =1.3.34 ]
 23 tomcat8.5 available \
 [ =8.5.31 =8.5.32 =8.5.38 =8.5.40 =8.5.42 =8.5.50
 =stable ]
 24 epel available [ =7.11 ]
 25 testing available [ =1.0 ]
 26 ecs available [ =stable ]
 27 corretto8 available \
 [ =1.8.0_192 =1.8.0_202 =1.8.0_212 =1.8.0_222 =1.8.0_232
 =1.8.0_242 ]
 28 firecracker available [ =0.11 =stable ]
 29 golang1.11 available \
 [ =1.11.3 =1.11.11 =1.11.13 =stable ]
 30 squid4 available [ =4 ]
 31 php7.3=latest enabled \
 [ =7.3.2 =7.3.3 =7.3.4 =7.3.6 =7.3.8 =7.3.9 =7.3.10
 =7.3.11 =7.3.13 =stable ]
 32 lustre2.10 available \
 [ =2.10.5 =2.10.8 =stable ]
 33 java-openjdk11 available [ =11 =stable ]
 34 lynis available [ =stable ]
 35 kernel-ng available [ =stable ]
 36 BCC available [ =0.x ]
 37 mono available [ =5.x ]
 38 nginx1=latest enabled [ =stable ]
 39 ruby2.6 available [ =2.6 =stable ]
 40 mock available [ =stable ]
 41 postgresql11 available [ =11 =stable ]
 42 php7.4 available [ =stable ]

Connect to our Instance

sudo ssh -i ~/.ssh/{pem.file} ec2-user@{ IP addres}

Install PHP 7.2, 7.3, 7.4, 7.5, 8 … 12 … using Amazon Linux Extras

## Instaling PHP
sudo yum remove php*
sudo amazon-linux-extras install -y php7.3
sudo yum install -y php php-common php-mysqlnd php-opcache php-xml php-mcrypt php-gd php-soap php-redis php-bcmath php-intl php-mbstring php-json php-iconv php-fpm php-zip
php -v
sudo service php-fpm restart

Installing Composer PHP

sudo yum install wget unzip
php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
sudo php composer-setup.php --install-dir=/usr/local/bin --filename=composer
composer -v

Install Nginx

sudo amazon-linux-extras install -y nginx1
nginx -v
sudo service nginx restart

Install MySQL CLI tool, not the server

sudo yum install mysql

MySQL is a simple SQL shell with input line editing capabilities. It supports interactive and noninteractive use. When used interactively, query results are presented in an ASCII-table format. When used noninteractively (for example, as a filter), the result is presented in tab-separated format. The output format can be changed using command options.

Instal Mysql Maria DB for Magento AWS Linux 2

Install Magento 2

# Installing Magento 2
composer create-project --repository=https://repo.magento.com/ magento/project-community-edition magento2CE
Warning from repo.magento.com: You haven't provided your Magento authentication keys. For instructions, visit 
https://devdocs.magento.com/guides/v2.3/install-gde/prereq/connect-auth.html
Authentication required (repo.magento.com):
      Username: 
      Password:
## Installing Git + Magento Sample Data Modules
sudo yum install git
git clone https://github.com/magento/magento2-sample-data.git
cd magento2-sample-data/
git status
git checkout 2.3
cd ..
cp -r magento2-sample-data/* magento2CE/

Routing Domain DNS to our server IP

Configure Nginx for Magento:

Examples from Ansible AWS Magento Cloud Deploy Script Repo:

https://github.com/Genaker/AWS_Magento2_Ansible/blob/master/templates/nginx/etc/sites-available/magento2.j2

sudo nano /etc/nginx/conf.d/magento2CE.conf
server {
       server_name {domain past here}; # or just _
listen 80;
set $MAGE_ROOT /var/www/htm/magento2CE/;
include  /var/www/htm/magento2CE/nginx.conf.sample;
}

Check if our Magento root path is /var/www/html/magento2CE/

Replace fascgi_proxy_pass fastcgi_backend with /var/run/php-fpm/www.sock

fastcgi_pass unix:/var/run/php5-fpm/DOMAINNAME.socket;

Or use this Amazing Repo to install Nginx :

wget https://raw.githubusercontent.com/magenx/Magento-nginx-config/master/m2_config_install.sh
sudo bash m2_config_install.sh
 — -> CREATING NGINX CONFIGURATION FILES NOW
 — -> Enter your domain name (without www.): _
 — -> Enter your web root path: /var/www/html/magento
 — -> Enter your web user usually www-data (nginx for Centos): apache

If you will use sockets;

sudo service nginx restart

Fix issues if you have

Now can open Magento Url and install it using a web interface or

You can also set up it using Magento CLI.

php bin/magento setup:install --backend-frontname="admin" --key="admin" --session-save="files" --db-host="localhost" --db-name="magento_2_db" --db-user="root" --db-password="root" --base-url="http://local.magento.com/" --base-url-secure="https://local.magento2.com/" --admin-user="admin" --admin-password="admin123" --admin-email="pramod.kharade@example.com" --admin-firstname="Pramod" --admin-lastname="Kharade"

First, we should create RDS: best size m5.large 140$ (North California location — San Jose):

*2.5 GHz Intel Xeon Platinum 8175

*2 vCPU

*8 GiB memory

*Up to 10 Gigabit networking performance

Step 1: Create an RDS DB Instance

After creation, RDS check security group port 3306Mysql should be open to your instance. You’ll need to add an instance IP address to the appropriate database security group via the AWS management console. Node IP address can be added to an existing group, or you can create a new one. Instruction: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithSecurityGroups.html#USER_WorkingWithSecurityGroups.Authorizing

When adding the Magento BI IP address, make sure you add a “/32” to the end of the address to indicate to Amazon that it is an exact IP address. Don’t worry; the AWS interface will make it clear that this is required.

Sorry I already have one. I also have a Redis ElacticCahe AWS instance, but it is better to use local Redis. Faster and cheaper. For MySQL, my recommendation is to use RDS because you have nice features like backup, instance resizing, performance monitoring.

create database magento2CE;

MySql has an error during the installation of Magento

Module ‘Magento_GoogleShoppingAds’:
Upgrading data…
[ERROR] PDOException: SQLSTATE[HY000]: General error: 1419 You do not have the SUPER privilege, and binary logging is enabled (you *might* want to use the less safe log_bin_trust_function_creators variable) in /var/www/HTML/magento2CE/vendor/Magento/framework/DB/Statement/Pdo/Mysql.php:91

Amazon RDS MySQL Functions

Fixing:

Better to do this before Magento installation:

Also, I have enabled Aurora Performance Shema for Performance insight monitoring.

The parameter group name is changed immediately, but the parameter group changes aren’t applied until you reboot the instance without failover. For more information, see Rebooting a DB Instance.

Reboot your instance.

OK, testing our instance.

Response time is 100ms cached:

Uncached: 360

This one, not an optimized configuration. Let’s configure PHP-opcache.

sudo service php-fpm restart

Testing again:

Result: 75ms

Not cached page

Result: 223ms

Not bad 143ms improvement

I saw a lot of issues when Magento PHP-opcache works without optimized values. Default PHP 7 op-cache values don’t work well with the Magento monster codebase.

This Magento 2 extension Monitor Ocache configuration for you:

https://github.com/Genaker/Magento2OPcacheGUI

OK, let add Redis:

sudo bin/magento setup:config:set — cache-backend=redis — cache-backend-redis-server=redis2.apgcjn.0001.usw1.cache.amazonaws.com — cache-backend-redis-db=0

sudo bin/magento setup:config:set — page-cache=redis — page-cache-redis-server=redis2.apgcjn.0001.usw1.cache.amazonaws.com — page-cache-redis-db=1

Now always after PHP files changes, we should restart PHP-FPM

Test:

Result:110ms

Result:315

Remote Redis slow down Magento almost twice:

Install Redis locally:

sudo amazon-Linux-extras install redis4.0

[ec2-user@ip-172–31–7–147 magento2CE]$ redis-cli -h redis2.apgcjn.0001.usw1.cache.amazonaws.com — latency
min: 0, max: 1, avg: 0.27 (317 samples)

remote redis

local redis

[ec2-user@ip-172–31–7–147 magento2CE]$ redis-cli -h localhost — latency
min: 0, max: 1, avg: 0.03 (401 samples)

Remote Redis is 10 times slower than local.

EC2 instance to connect to Redis using a TCP/IP connection or a Unix domain connection. The typical latency of a 1 Gbit/s network is about 200 us, while the latency with a Unix domain socket can be as low as 30 us. It depends on your network and system hardware. On top of the communication itself, the system adds some more latency (due to thread scheduling, CPU caches, NUMA placement, etc.…). System-induced latencies are significantly higher in a virtualized environment than on a physical machine.

If to store cache in files it will be the same fast if not faster as local Redis with a socket. Redis it is for scaling.

Lets Setup Redis as a socket

Go to: Sudo nano /etc/redis.conf

Uncomment:

# Unix socket.
#
# Specify the path for the Unix socket that will be used to listen for
# incoming connections. There is no default, so Redis will not listen
# on a unix socket when not specified.
#
unixsocket /tmp/redis.sock
unixsocketperm 775

sudo chmod -R 777 /tmp/redis.sock

sudo service Redis restart

Redis socket?

The same rule applies to MySQL. Local + Socket communication will be faster.

Magento has 60–100 Redis requests

on remote Redis, you will have latency 0.3, and you will have 30ms delay

on local TCP 127.0.0.1 latency is 0.03 you will have 3ms

with Redis on a socket, latency is 0.01. You will have a 1ms delay per page load.

When developers check a cached value in the loop, everything and per one page has 600 cache requests. Performance degradation is significant.

Also, you can improve Resids performance by switching to an instance with faster network throughput. But the price will be more x2–5 more. 0.3 it is for T3.small instance. M5 R5 instances have better network performance

If you need a swap

https://aws.amazon.com/premiumsupport/knowledge-center/ec2-memory-swap-file/

Magento with Auto Scaling

About Magento AWS Auto Scaling infrastructure, you can read in my next posts or check this GitHub repo:

https://github.com/Genaker/TerraformMagentoCloud

Magento 2 AWSCloud With Autoscaling