Magento Lambda EFS/NFS integration
AWS Lambda customers can now enable functions to access Amazon Elastic File System (Amazon EFS). Customers can easily share data across function invocations, read large reference data files, and write function output to a persistent and shared data store.
Magento merchants can now use AWS Lambda to build data-intensive applications, load larger libraries and models, process larger amounts of data in a highly distributed manner, and share data across functions, containers and instances. AWS Lambda will automatically mount the file system and provide a local path to read and write Magento Code and images (media folder) at low latency.
To use Magento Lambda with Amazon EFS, customers add an EFS Access Point ARN and the local mount path to their function configuration. AWS Lambda ensures function instances mount the EFS file system within the same Availability Zone.
AWS Lambda support for Amazon EFS is available in all Regions where Lambda and EFS are available.
CREATING AND ATTACHING AN EFS FILE SYSTEM TO THE MAGENTO LAMBDA
First, head over to the EFS console and click “Create file system“.
Next, configure the VPC and Availability Zones for the new file system. Yup, that’s right. You have to be inside a VPC in order to access an EFS file system. Lucky for us that the enhanced VPC networking has been rolled out to Lambda functions globally, so there is no more performance penalty for being inside a VPC! There is still a small chance of ENI and/or IP exhaustion, but these are far less likely.
In the next screen, there are some more configuration options. The most important configurations here are the throughput and performance mode. You can read more about these options in the official EFS Performance guide. In a nutshell, Bursting Throughput scales the throughput of the file system with its size — the bigger the file system, the higher its throughput. Whereas Provisioned Throughput gives you predictable throughput (in MiB/sec) but requires additional overhead for managing it, including a 24-hour cooldown period between throughput decreases.
One thing worth noting is that during the private beta we were told to stick with the General Purpose performance mode. This was because the Max I/O mode has higher latencies for file operations, which caused problems with the mount latency from Lambda. It’s possible that this has been resolved by now, but please consult the official documentation for more details.
Next, you can configure the file system policy. For more details, please check out the official documentation here.
To give Lambda functions access to the file system, we need to create EFS access points. These are application-specific entry points to an EFS file system. In the following example, we created an access point that allows root access to the /test directory in the file system. We can use access points to control what directories on a shared file system a function can access.
Next, review everything and hit “Create File System“.
Now that our new EFS file system is ready, head over to a Lambda function.
At the bottom of a Lambda function’s configuration page, you should see the option to “Add file system“.
Before you can attach the EFS file system, you first need to:
- configure the function with access to the file system’s VPC
- give the function IAM permissions for
- ec2:CreateNetworkInterface
- ec2:DescribeNetworkInterfaces
- ec2:DeleteNetworkInterface
- elasticfilesystem:ClientMount
- elasticfilesystem:ClientRootAccess
- elasticfilesystem:ClientWrite
- elasticfilesystem:DescribeMountTargets
Once you have these, then click on “Add file system“.
And choose the file system and access point we had created earlier. Another important piece of configuration here is the Local mount path. This is where the /test directory (which we set up in the access point) would be mount to in the Magento Lambda execution environment. It’s also worth noting that the local mount path must start with /mnt/.
Once mounted, you will be able to read and write files on the EFS file system against the local mount path.
Known problems:
While I was testing the feature, I also noticed that it took an average of 8ms to read a 30bytes cache file, and 30ms to write the same file back to the mounted EFS file system. For a 13MB file, the read and write latency jumped up to an average of 170ms and 330ms respectively, which was often longer than the time it took to read & write the same file to S3! However, it’s worth noting that the S3 latency has a far greater variance and sometimes took a few seconds to complete.
What’s more, increasing the function’s memory setting didn’t yield any measurable differences. Nor did switching over to Provisioned Throughput mode and dialing up the throughput to 10 MiB/sec (which will incur a cost of $60/month).
This latency essentially rules out EFS for any latency-sensitive applications.
