Citrix NetScaler’s recently exposed security exploit, which allows attackers to bypass authentication and take full control of the load balancing infrastructure, is frightening. But it is hardly a standalone incident. Every product from every company may eventually succumb to a critical vulnerability. This raises some good questions around security. How does a vendor test the security of their product? If vulnerabilities are inevitable, what is the vendor response and customer strategy?
Validation of Security
Developers will always endeavor to create a secure product. Most companies, including Avi, have a robust suite of tests that each release must pass. These tests can be thorough, checking thousands of permutations of configurations and leveraging known exploit databases. However, these tests are somewhat limited, as they are the same tests every time and do not mimic the creative behavior of hackers.
Test automation is important, but vendors that expect this to be the foundation of their security have been drinking too much of their own Kool-Aid. Things get interesting when a product is put into the wilds of the Internet. At Avi, we have seen cases where a controller is installed in Amazon, and between the time the installation completes till the time an administrator first attempted to log in, the system has already been compromised. This was due to a standard “admin/admin” account and password scheme. Perhaps this isn’t a security vulnerability in the classic sense, but it highlights the drastic difference between lab testing and real world hacking. Avi no longer uses standard passwords for new deployments.
Equally important with test automation is external testing and validation. Avi undergoes regular pen(etration) testing, performed by third party white hat hacking companies. Pen tests mimic the creative and unexpected methods employed by hackers to find the holes a developer did not anticipate. The results of these pen tests, including any vulnerabilities and remediation, is available to Avi's customers. Learn more about Avi's pen tests.
Vulnerabilities Are Inevitable
If vulnerabilities are inevitable, then the focus should be on vendor and customer response. The vendor side is easy. Transparency and fast communication are key. Avi encourages all relevant customer contacts to sign up for security alerts and to follow appropriate best practices.
The customer responsibilities are a little harder. Implementation of best practices is good for prevention, but what if their load balancers are compromised?
NetScaler asked all customers to immediately upgrade to newer code versions which close their vulnerabilities. Easy to say, but upgrading ten pairs of hardware load balancers is a delicate and time-consuming task. Rather than a brutal weekend in a datacenter, you can push a button and go to lunch with Avi. The entire fabric is fully upgraded in minutes—no impact to production traffic— and has auto rollback should there be any issues.
NetScaler's recommendation to upgrade is good but disingenuous. Attackers may have had root access to the load balancers for years, so it's probable malicious code or accounts exist on the compromised load balancers. Rather than upgrade, all load balancers should be completely wiped and reinstalled from scratch. That's more than a long weekend for an admin. With Avi, automation is king. Infrastructure-as-code enables customers to completely delete and recreate the entire load balancing infrastructure. With tools such as Ansible, a customer can completely rebuild the entire fabric of 10 pairs of load balancers in 10 minutes by restoring a known good config. This is the heart of software-as-a-service model and the controller based architecture model.
Vulnerabilities will always exist, but a strong mitigation strategy is the best defense a customer can have. If you haven't already, watch one of Avi's technology webinars on upgrade and automated deployments, or ask to see a live demo. Then compare that to your current remediation strategy. Don't leave your load balancer security up to chance. Compare Avi Networks with Citrix NetScaler and F5 to discover the advantages of a dynamic and secure software load balancer.
