However, the contents of test environments (whether they're test frameworks, CSV files, or a rainbow of test data) can be very sensitive, and more importantly, your test systems could contain a decade or more of information related to the development of your systems, tools, and applications. In spite of the potential sensitivity of the data in these systems, the often ad hoc or ShadowOPS nature of their development frequently means that test systems do not undergo the same security scrutiny as other parts of your development environment or production systems. This means that the risk associated with them probably outstrips the regular investment that has been made to secure them.
From a security point of view, it’s time to right the balance ― to toughen up the soft underbelly of your test systems so they don’t undermine your test, dev, and production environments. Test systems constitute all of the resources, often under-served by operations and security teams, that are used by software testers (QA, SDET, SRE, customer service, product managers, etc.) to test and hone the products they want to release. These can include local virtualized servers (as used with tools like Docker, VMWare, or VirtualBox), CI systems like Jenkins and the test resources connected to them, demo or staging systems for manual testing or acceptance, and test databases for integration and load testing. Non-unit testing resources are the most susceptible since real resources are likely used in preference to in-memory, mock resources.
Properly built test systems are situated inside de facto isolated systems, so a system-under-test (SUT) can be controlled by the layer around it (i.e., a CI system like Jenkins or Travis CI, or language-specific test frameworks). Controlling this layer is important as is preventing access to SUTs and the systems and credentials. Building sandboxes is not quite enough, however; you need to build high walls around the sandboxes.
If you integrate tools to manage your test systems and share them with an operations team’s configuration management tools or deployment process, you can use the test systems to improve operations practices and tools while improving the repeatability and documentation of the development and the accountability of the test systems. In short, you can use the same authentication tools in test and production instead of providing anonymous access, which is an all-too-common practice.
If you improve the security of your test systems, you can also help your Development and QA teams by providing a safe, well-monitored environment in which you can test and deploy updated applications, test applications to recently released security patches, and improve the overall understanding of the way your projects provide access to the outside world. In other words, improved security improves the QA ability of your systems by providing another valuable perspective on the development and execution of your systems. Increasing the isolation of a system should always lead to increased security, knowledge, and testability of that system ― goals that will help to unite the QA/SRE (Site Reliability Engineer), Operations, and Security teams.
Given the importance of security, then, it is critical that you follow defined policies and procedures to ensure that your test systems are as secure as possible and don’t expose existing vulnerabilities or create new ones. With that in mind, here are three main areas where you can improve the security of your test systems, along with specific recommendations for each.
1. Set Up a PerimeterKeep test resources inside a VPN and consider isolating test VMs in a VPN that is separate from production or demo systems. Ideally, a VPN containing test-specific resources should have no access to or interaction with development or production systems or their credential systems.
Require authenticated access to all test environments. Do not allow any anonymous access. Anonymous or default access is only appropriate for well-developed local or local-virtualized testing against test data (i.e., non-real data).
Control the security groups of your test systems and be as explicit as possible. All inbound and outbound traffic should be over known ports. Do not simply allow inbound and outbound access for all ranges; for test systems, be specific about what ports are open and what is communicating over them. This improves your ability to monitor and test the system as well as the testing environment, thereby helping to achieve the goals of both the security and the testing teams.
Monitor all access to and communication with the test system’s networks, and set up sane alert rules to ensure that your test systems remain properly configured. Do not disable your security perimeter by ignoring logging access and networking information for test systems. This information enables you to learn about and define the known aspects of your systems, so do not turn them into unknowns. Be more specific about configuration in test systems, not less, especially when it comes to security.
2. Monitor Your Test Systems and Set Up AlertsAlert on platform actions such as those provided by AWS CloudTrail. Examples of activities to monitor include:
Backing up instances. EBS volume copying and snapshotting. Backing up test or development databases.Increase security alerting for login and communication events into test systems and networks. Assume that there should be no regular direct human access to automated test infrastructures. While no human should regularly be SCPing or FTPing anything to or from automated test systems, your test and QA teams should be considered responsible and trusted to access these systems but they should still be monitored for any manual interaction.
Increase security event logging in test systems, but be careful not to increase pages for test systems.
Monitor any human (i.e., non-automated) behaviors on test automation systems.
3. Improve Your Tests; Improve Your SecurityNever use default credentials. Default and root credentials are highly vulnerable targets, so disable them for any test system. Always run tests with known access credentials. This is especially true if you ever place production data on a test system to help debug major production issues.
Never use production data if you can help it. Instead, use generated, production-like data. To speed up this process, use database migration tools if your development teams are already using them.
Shared and hosted test resources are more exposed than production systems if they are placed under identical scrutiny because of their common direct access and manual configuration nature. Save any unauthenticated testing for your local test development, not for shipped tests for your CI pipeline. These systems can often contain valuable information in the form of old, forgotten credentials, and state actors are known to regularly breach systems and monitor actions and files on test systems to improve their ability to move laterally to connected systems.
Use secure methods to transfer sensitive configuration information to your test environments and prevent plain text secrets or passwords from being committed to source control or configuration management systems where this information can easily be leaked. Use systems such as Travis CI's encrypted environment variables or Jenkins’ credential store to dynamically inject variables into your build to reduce the possibility that secrets such as passwords are being checked into test code. This approach will also frequently improve the local development of tests and test infrastructure by making it easier to run tests in a variety of environments with diverse credentials, or even in local testing scenarios.
Encrypt or secure test database and system backups as you would any production backup.
Deprecate and remove any proof of concept test systems. There is a strong risk of credential leaking with any manually configured systems. If possible, move test systems into configuration management when graduating test systems out of the proof of concept, and ideally before a wide internal testing user base begins developing tests against a new test environment. Isolate all systems used to develop new test systems or patterns during development, and destroy them once more permanent testing resources are put in place (i.e., once tests have been integrated into a CI pipeline).
Always patch your test systems. These are just as vulnerable as any other system, and you should not develop or release code to unpatched servers. Testing software that needs to run in known unpatched environments should be tested on hardware-isolated networks ― which is not an ideal use case for the cloud.(Note: If you don’t think you can patch your production systems, stop reading this immediately. Then you can start learning how to patch and upgrade ALL your systems, platforms, and tools, and how make it easier to patch, test, and deploy these changes to your production systems.)At the very least you should begin using your test systems to help stage security patches to your production systems. (Test systems are strangely perfect for things like testing!)
Do not test with plaintext credentials. Instead, use one-time or temporary credentials during test runtime if possible. The following services work extremely well to provide temporary credentials:
