Magento Lambda EFS/NFS integration

AWS Lambda customers can now enable functions to access Amazon Elastic File System (Amazon EFS). Customers can easily share data across function invocations, read large reference data files, and write function output to a persistent and shared data store.

Image for post
Image for post

Magento merchants can now use AWS Lambda to build data-intensive applications, load larger libraries and models, process larger amounts of data in a highly distributed manner, and share data across functions, containers and instances. AWS Lambda will automatically mount the file system and provide a local path to read and write Magento Code and images (media folder) at low latency.

To use Magento Lambda with Amazon EFS, customers add an EFS Access Point ARN and the local mount path to their function configuration. AWS Lambda ensures function instances mount the EFS file system within the same Availability Zone.

AWS Lambda support for Amazon EFS is available in all Regions where Lambda and EFS are available.

CREATING AND ATTACHING AN EFS FILE SYSTEM TO THE MAGENTO LAMBDA

First, head over to the EFS console and click “Create file system“.

Next, configure the VPC and Availability Zones for the new file system. Yup, that’s right. You have to be inside a VPC in order to access an EFS file system. Lucky for us that the enhanced VPC networking has been rolled out to Lambda functions globally, so there is no more performance penalty for being inside a VPC! There is still a small chance of ENI and/or IP exhaustion, but these are far less likely.

Image for post
Image for post

In the next screen, there are some more configuration options. The most important configurations here are the throughput and performance mode. You can read more about these options in the official EFS Performance guide. In a nutshell, Bursting Throughput scales the throughput of the file system with its size — the bigger the file system, the higher its throughput. Whereas Provisioned Throughput gives you predictable throughput (in MiB/sec) but requires additional overhead for managing it, including a 24-hour cooldown period between throughput decreases.

One thing worth noting is that during the private beta we were told to stick with the General Purpose performance mode. This was because the Max I/O mode has higher latencies for file operations, which caused problems with the mount latency from Lambda. It’s possible that this has been resolved by now, but please consult the official documentation for more details.

Image for post
Image for post

Next, you can configure the file system policy. For more details, please check out the official documentation here.

Image for post
Image for post

To give Lambda functions access to the file system, we need to create EFS access points. These are application-specific entry points to an EFS file system. In the following example, we created an access point that allows root access to the /test directory in the file system. We can use access points to control what directories on a shared file system a function can access.

Image for post
Image for post

Next, review everything and hit “Create File System“.

Image for post
Image for post

Now that our new EFS file system is ready, head over to a Lambda function.

At the bottom of a Lambda function’s configuration page, you should see the option to “Add file system“.

Image for post
Image for post

Before you can attach the EFS file system, you first need to:

  • configure the function with access to the file system’s VPC
  • give the function IAM permissions for
  • ec2:CreateNetworkInterface
  • ec2:DescribeNetworkInterfaces
  • ec2:DeleteNetworkInterface
  • elasticfilesystem:ClientMount
  • elasticfilesystem:ClientRootAccess
  • elasticfilesystem:ClientWrite
  • elasticfilesystem:DescribeMountTargets

Once you have these, then click on “Add file system“.

And choose the file system and access point we had created earlier. Another important piece of configuration here is the Local mount path. This is where the /test directory (which we set up in the access point) would be mount to in the Magento Lambda execution environment. It’s also worth noting that the local mount path must start with /mnt/.

Image for post
Image for post

Once mounted, you will be able to read and write files on the EFS file system against the local mount path.

Known problems:

While I was testing the feature, I also noticed that it took an average of 8ms to read a 30bytes cache file, and 30ms to write the same file back to the mounted EFS file system. For a 13MB file, the read and write latency jumped up to an average of 170ms and 330ms respectively, which was often longer than the time it took to read & write the same file to S3! However, it’s worth noting that the S3 latency has a far greater variance and sometimes took a few seconds to complete.

What’s more, increasing the function’s memory setting didn’t yield any measurable differences. Nor did switching over to Provisioned Throughput mode and dialing up the throughput to 10 MiB/sec (which will incur a cost of $60/month).

This latency essentially rules out EFS for any latency-sensitive applications.

How to run Magento 2 in AWS Lambda read there:

Written by

Magento/APP Cloud Architect. Melting metal server infrastructure into cloud solutions.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store