Companies often know they should be writing automated tests but “don’t have the time”. As a result, they may pay an outside agency to write their automated tests for them or assign it to a QA team. This situation is far from ideal: Developers should be writing the automated tests for their code. An outside agency or QA team can be used to write more functional, higher level tests as a safety net to that. I really believe this is the only correct way to do it. But, if you insist on having an outside agency or QA do it, here are some important dos and don’ts:
Do: Give them thorough training of the product and domain knowledge
The worst thing you can do is hire an outside agency and expect them to get started without even understanding the purpose of the product. They’ll have no idea whether it’s working in its current state or not. Without this understanding, they may be wasting time writing automated tests that check that the product doesn’t work.
You may think it’s relatively inexpensive and it’s a small price to pay, but they’ll also end up writing fragile tests that break all the time because they are testing the wrong things. It could end up like you hired someone to check the foundation of a house, but if that isn’t communicated, they may end up checking where the furniture is. If all that furniture is going to move and be replaced, the testers will waste a lot of time inspecting why their tests are failing and will probably need your team’s help to resolve the issues.
Without understanding the purpose of the product, they are likely to spend a lot of time testing parts of the code that don’t change and aren’t important. They should be focusing on testing the bread and butter of your product: The part of it that generates income.
Do give them an independent environment
It’s essential that they have their own environment they can test. Ideally, each tester would have their own database. This way, they can modify it in any way they need and check the results in their tests. If they are sharing an environment with others, the other people may modify the database in ways that break the agency’s tests. Even if this doesn’t ever happen, it will always be suspected as a reason for why a test has failed. Also, these other people may need to shut down the databases and services for their own purposes that interfere with the testers. This will add to the required communication and cause tests to fail or or be delayed.
Don’t just give them a piece of the database that’s there’s to test on. If they work on their section, then that’s the only section that will be tested. Give them their own entire database. By the way, when I say, “database” I don’t necessarily mean that literally. I mean whatever remote services your application depends on. It could be a Rest API, a messaging queue, etc.
Do give them access to the configuration
Obviously, this is a lot safer to do when they have their own environment because they can’t accidentally take down someone else’s. It can often make sense to do things like replace the persistence service with an in memory database. This can improve the speed of the tests (and therefore the feedback you get) by an order of magnitude. If you don’t give the agency/QA the ability to swap this out, then you are making them do their job with one hand tied behind their back.
If the outside agency/QA doesn’t have access to the code, the tests are almost guaranteed to take hours to run. This is because they won’t be able to mock out slow/dangerous parts of the code in their tests.
Do expect to have to communicate with them often
This is going to be a very common scenario:
Tester to Employee: “Hey, this test is failing all the sudden. Any idea why?”
Employee to Tester: “Yeah, that feature was removed/changed. You’re testing the wrong thing, now.”
This is one of the reasons I advocate against an outside agency/QA writing all the tests: The developers are always going to be one step ahead of the tests and the testers are going to be constantly playing catch-up.
If the developers are getting notified of the failing tests and it’s often a false positive because of this scenario, then developers are going to lose confidence in the notifications in the future. They’ll start ignoring the broken tests and just wait for someone to tell them that they’re “really broken this time”, not “fake broken”. Is this really an improvement over where you were before? But I digress…
This type of communication is going to have to happen constantly. Otherwise, nobody will know whether the results of their test are accurate or not.
Don’t stop maintaining the tests
As long as the product is being developed, the outside agency/QA can never be finished maintaining their tests. A new feature will be created that needs to be tested or an old feature will change and tests will have to be updated. If you start writing these tests you should be in it for the long haul. If you pay an agency $100,000 to write comprehensive tests and then you stop maintaining them, you just wasted $100,000. That’s why the worst mistake you could ever make is to consider the process “finished” and stop working on it.
Don’t let developers use this as a crutch
If developers know that others are writing automated tests for their code, then they may not be as disciplined about the code they write in the future. In other words, your developers may become less responsible when they know an agency has now become responsible for the quality of their work.
If you think about it, this whole situation is a little weird. A carpenter obviously would have someone look over their work, but he would not depend on that person to make sure he built something of quality. He would make a product of quality and be surprised if a 3rd party found and issue with it.
Conclusion
As I’ve said multiple times, you should not use an outside agency/QA as the only source of your automated tests, but I think it is great to use their automated tests as a safety net for bugs. Developers should be testing their own code. But, if you insist on testing your product this way, I hope these dos and don’ts will help you avoid some common pitfalls that others have made in the past.