As Alteryx developers, we are constantly trying to make our workflows more efficient. There are workflows that I will revisit after months of delivery in efforts to try and fine tune them so that they can run even faster. With the latest version of Alteryx Designer, the workflow processing engine will forever be changed for the better.
The Alteryx Multi-threaded Processing (AMP) is a new architecture of the Alteryx engine, which was introduced in version 20.2. It is a multi-threaded concept of processing your data in parallel packets rather than record-by-record and sequentially.
Alteryx Designer will be featured prominently because many of you probably have long-running workflows that you would like to run faster. Additionally, for Alteryx Server to further the positive effect on the execution time, the tuning really needs to start at the workflow level in Designer.
AMP was designed to go through large amounts of data quickly with the workflows generally running more rapidly. I liken the benefits to some of the benefits that Tableau reaped by creating .hyper data sources. That, too, was designed to process large amounts of data quicker.
Why the Processing Engine Change?
To truly answer this question, one must understand the differences between the two engines first.
There are three elements I would like to focus on: Data Processing, Hardware Performance, and Record Order. For the first element of data processing concerns how records in a data set are handled. Simply put, this is sequential process vs. parallel processing. The original engine handles data record-by-record whereas in the AMP Engine anything that can be done at the same time will be done at the same time.
Most of us use laptops for Alteryx Designer and we may have some memory and power associated with them. I have an eight-core laptop with 64 GB of memory. What may be considered fast on my laptop could really fly on a server if you have it. The larger the machine, the more resources allocated for that particular workflow.
Record Order – One caveat about using AMP is that you cannot expect the order that records go into a tool will be the same when they exit. I am specifically speaking about the Join tool. If you want to utilize AMP, please add a Sort tool immediately after a Join or Join Multiple to ensure the record order you want.
AMP Memory Usage
This section about AMP memory requirements is important, especially if you are going to be running Alteryx on machines with other memory-intensive programs. With the original engine, there needs to be at least 64 MB of memory to run a workflow. Anything less than 64 MB will throw an error. Additionally, the machine will use only one core even if you have four cores available to execute a workflow. Succinctly, the full power of the machine is not used to run a workflow.
However, with the AMP Engine there needs to be at least 400 MB per thread to run a workflow. Anything less than 400 MB will throw an error. If you have a four-core machine, the machine will use all four cores at 400 MB each to execute a workflow so 1.6 GB of memory will be used. Keep that in mind budgeting memory usage on your machine.
How do I activate AMP?
We are going to over the three ways to activate AMP: Alteryx Designer, Alteryx Server, and Alteryx Analytics Hub (AAH). AMP is fully supported by Alteryx for Designer and AAH, but not for Alteryx Server.
In the Workflow Configuration window, locate the Runtime tab. Toward the bottom of tab, there is a checkbox that states, “Use AMP Engine”.
At the Alteryx Server level, there is an area in the System Settings that activates the e2 or AMP Processing Engine. Locate the Engine checkbox near the bottom of the window and check the box as shown below.
Lastly, AMP is standard for AAH. There is no toggle checkbox for the user to activate. All workflows will be run using AMP even if it was not activated in Designer.
You are going to be faced with the question of whether to AMP or not to AMP with your Alteryx workflows. I will leave you with the following points that might help you with some of your decision point
- Run an existing workflow a few times in the original Engine, then run the workflow a few times with the AMP Engine
- Some workflows may run longer in AMP than before and may return more errors
- Macros may return more errors
- AMP performs better with typical prep/blend/analyze use cases within Alteryx
- Find/Replace, Join tools perform significantly better
- Original Engine highlights
- It may use only one core even if you have four cores available to execute a workflow
- The full power of the machine is not used to run a workflow
- If you have reads on the same file in a workflow, you will receive an error in AMP because there are concurrent threads
- Alteryx Server is not supported quite yet, Alteryx Designer and AAG are
By Chris Williams
Certified Alteryx Partner
and Tableau Desktop Specialist