When RADA moved all training online in 2020 as a result of the Coronavirus pandemic, adapting to the circumstances were at the forefront of everyone’s mind. With shows being closed overnight, and students unable to complete their degrees without working on RADA productions, our way of working had to change in a way we never thought it would.
Each department in RADA tailored online training to their needs, and the Lighting Department decided to continue with the scheduled productions ‘virtually’, rather than in person, using a combination of ETC EOS and Capture to pre-visualise lighting and automation.
The challenge with this is that suddenly the programmer and designer are no longer sat next to each other, in the same room, on the same Local Area Network (LAN). We were faced with establishing a stable network connection between the Production Lighting team, which now spanned 2 continents at distances of over 5000 miles.
Enter Virtual Private Networks (VPNs).
We found out early on that trunking sACN traffic over an off-the-shelf VPN is near impossible, for reasons such as: the VPN provider categorising broadcast, multicast and unknown unicast traffic as malicious behaviour so aggressively throttling the connection, or multicast traffic not being supported at all. To get around this, I had to find a solution that allowed me to virtualise a layer 2 connection, without the throttling of multicast traffic.
After some searching, I opted to use an AWS Linux instance based in London to host an open-source VPN, which acted as the server, and allowed me more control over the configuration of connections. Most importantly, I was able to relax the bandwidth cap usually placed on multicast network traffic.
Once I had set up port forwarding of ports 4500 and 500 (UDP) for IPSec traffic, port 1701 (UDP) for L2TP traffic, and 443 (TCP) for the software-based VPN over HTTPS, I successfully established a connection between two VPN clients on separate public IP addresses, and used sACNView to confirm that multicast traffic was not restricted. I tested over 200 fully populated universes sending sACN traffic over the VPN at around 50Hz.
The next step to make this project useable for RADA was to integrate ETC EOS into this setup. EOS and multiple NICs (especially virtual ones such as a VPN) were a little difficult to configure together. This took some trial and error. That being said, I found that EOS has no issue transmitting sACN across a stable VPN connection, but this was the easy bit. I wanted to be able to establish a Primary / Client connection.
To establish a stable Primary / Client connection, it took quite a lot of time. For hours I'd be met with the EOS 'red text' when attempting to connect to an available Master. I found that the solution was a combination of setting the right Network Interface Card (NIC) metric, gateway, and some tweaking of the VPN configuration server-side. It was a long process, but I got there. The only ongoing issue is that machines running MacOS cannot 'client' into a Primary because of various issues with MacOS requiring an internet connection to connect to the VPN, how MacOS handles its' equivalent of the Windows 'NIC metric', and how EOS on MacOS handles multiple NICs.
It was relatively easy to get Capture to see the sACN traffic on Windows machines, and MacOS again proved a little more difficult. The latency of the connection was <30ms end-to-end, and Capture was able to reproduce fades and effects smoothly, indicating a consistent and reliable connection.
Once I'd seen all of this running together, I presented the solution to members of the Lighting Department, and to make this accessible for the programmers and designers who were going to use this, I bundled the configuration and client into one installer, with a set of instructions and sent out a link to the department. Being a first year student at the time who believed he'd found a solution to the problem, I was confident, but nervous.
The great thing is that the VPN remained stable whilst the ‘virtual’ production period took place, and whilst now not in regular use, is easily spun back up should it need to be used again in the future. With the entire setup being TCP/IP based, systems we're used to using like Stamp, NDI, QLab and even shared storage pools could sit right on the VPN, bandwidth permitting.
Samuel Blakemore
London, UK
Copyright © 2023 Samuel Blakemore. All Rights Reserved.
ALPD Member #2941