How to seamlessly deploy on Fridays and still sleep tight at night?

In the world of start-ups, fast delivery and product iteration are crucial. However, this often proves to be problematic due to a combination of technological and human factors. Achieving a robust QA automation process is akin to running a marathon, not a sprint. After all, quality is built in, not bolted on.

Any change in production environments comes with a risk. A risk that has to be mitigated and managed in order for start-ups to thrive. So how, despite that risk, can you seamlessly deploy on Fridays and still be able to sleep tight at night?

You need to have confidence in your product.

And building that confidence involves many steps. One of them that I would like to focus on in this article is QA automation, specifically end-to-end tests.

QA automation relies on testing our software autonomously with the aid of external testing tools and technologies.

The benefits are many - the two main ones are being able to test your product at any point in time without the possibility of human error and saving time thanks to them running as an automated testing process.

While this sounds great, it also comes with many things that must be considered. QA automation is a whole process - not just writing automated test scripts and watching them run.

One of the obvious ones is test coverage. Tests are only useful if they check the proper places of the application. There are multiple cases to consider when planning which parts should be covered, for example:

There are no strict rules on how to approach this. You have to carefully consider your current situation, business goals, and your end-users, in order to be able to determine which should have the highest priority. Be pragmatic.

You now have some software tests in place. You have the most crucial parts of your application covered. Now it’s time to actually run them. This prompts a few more questions, namely: how, when, and where.

The “how” is probably the simplest to answer as it is usually dictated by your business’ needs. This mainly boils down to which browsers or platforms you should use. This is something that is best considered before the actual implementation of automated tests as it is reliant on the technologies that you choose.

“When” will mainly depend on the architecture of your application and the speed of your tests.

Ideally, you will want to run them on every commit for every pull request. If your project needs to be fully deployed onto an environment in order to be tested properly, a strategy of testing less often may have to be adopted.

The same applies if the process takes too long to run. Running an extensive suite of tests for every commit is sometimes simply not feasible. Triggering them manually on demand is also a good approach, as long as it is incorporated into your software development process properly. For example, running the tests before merging every pull request.

The ideal strategy on “where” to run them is an isolated CI/CD environment.

This ensures a perfect replica of the environment on each run and is very convenient due to being able to run in the background on remote resources. What gets often overlooked though, is running the tests locally. It is still a great way to get results earlier or be able to run them under specific circumstances.

It can also enable TDD (Test Driven Development), which can drastically speed up your implementation if done right.

It is also essential to consider the target environment of your tests. There are two main strategies that are usually considered: running the application locally next to your test runner or deploying the application first to a test environment and running the tests after.

This is usually the easier option, especially in cases of straightforward application architecture. There is no need for deployment, and the application communicates via a local network keeping the response latency and loading times to a minimum.

This is why this approach often results in faster test runs. It may not always be possible to use it though, especially if your application depends on third-party services or if the architecture is too complex to fully run locally in an isolated environment.

This may require a more complex setup but is overall a better representation of your production environment.

You should try to eliminate as many differences as possible between your test and production environments as this will more closely represent actual user interactions. It may also alleviate the need for additional smoke tests in actual deployments as just running tests is indirectly a smoke test by itself.

Regardless of the used environments, you should strive to replicate your production environment as closely as possible. A common good practice is to replicate your production application’s data (if possible) or try to make it as realistic as possible.

This step is often overlooked and neglected. No matter what solutions or technologies you use, the logs will always be somewhere.

What is crucial though, is that they contain valuable information and are visible in a convenient place and format for you to view at any time. Setting up proper test output is usually something that only has to be done once and can empower your workflow drastically.

Since the application is being verified automatically for you anyway - you should aim to have it provide as much information as possible that you would need to debug any potential test failures.

This can include:

Having this information provided in a convenient place (a pull request comment, a dedicated Slack, or even Jira) will prevent bad habits of ignoring test results and usually allow you to see precisely where the problem lies without the need for further debugging.

While the time required for running your tests is important, it’s usually not a deal-breaker. Make sure that it is within a reasonable timeframe, or if instant feedback is essential, ensure you have live access to test results (logs, artifacts).

If execution time becomes an issue, there are, of course, ways to improve it as well. One of them is test concurrency which makes the tests run simultaneously (scale vertically). While the concept is simple, there are a lot of nuances that need to be considered.

The most important one is that your tests need to be (mostly) isolated. They need to be able to run by themselves, preferably in any order. This requires them to have no dependencies on other tests both directly (one test producing results needed for other tests to run) and indirectly (test having side-effects that may cause other tests to fail, for example, data displayed in the application).

It’s best to plan with this approach in mind from the start as it usually leads to writing robust test scripts. They also need to have the proper setup and teardown logic in place which may not always be the most efficient approach (for example, having to log into the application before each test).

Another thing that needs to be considered is the tools and runtime environment of your tests. Not every framework supports concurrency the same way and some of them may make it easier (supported by jest since version 28) or harder for you to implement.

You will also most likely need to reconfigure your CI or change your workflows altogether. Tests will also require further organization as just running them randomly may deepen the inefficiencies - perhaps you do not need full test isolation, but a logical grouping of tests is enough.

There are a lot of things to consider during the implementation of test concurrency, which usually makes it a rather costly process that will also require constant maintenance. It is recommended to take care of the low-hanging fruits first. Maybe some tests have a lot of duplicated logic and it’s better if they are merged together. Maybe some long-running steps can be simplified (for example, by using the application’s API directly).

At the end of the day maybe it’s not the speed that you need but a little bit smarter test execution. It may not always make sense to run the full suite of tests on every single change. Maybe you only need to run the visual tests for that one simple CSS change. Maybe you only need to test one specific view which uses the unique component you changed.

Tests will not always pass. What is important to remember is to be able to properly understand the reason for their failures.

The most obvious one is simply that the feature being verified is not working as expected.

Another less frequent type of failure that is usually more demanding to fix is false negatives, aka flaky tests. These will report an error for functionalities that may be working perfectly.The cause is also not always obvious, it may range from your application’s code, the test logic itself, the test run environment, or even network conditions or time of test run. The time needed to debug these can differ wildly so every flaky test has to be considered by itself.

But what if it’s the other way around? What if a test is passing even though it is supposed to be failing? False positives can be even more dangerous as they can exist uncaught for long periods, giving you a false sense of security even though the underlying functionality may not be working.

One sure way to mitigate these is to use mutation testing, but other than simple unit tests, this can be extremely difficult to implement. False positives are usually a rare occurrence and can be easily prevented by good test organization and maintenance.

It is also very important to not get caught in the trap of having to polish every test to perfection. A couple of flaky tests may not make a difference at all in the grand scheme of things.

Sure, the red cross next to the pipeline name on your pull request is very intimidating and jarring, but is it actually worth fixing?

Time spent on fixing that one flaky test could be spent on the creation of multiple new, more useful ones. Of course, it’s also not good to get into the mentality of never maintaining older tests in favor of new ones.

You should also consider what functionality is being tested and how important it is. As long as you understand the reason for the lack of test stability and are still reviewing the results of other tests properly, not all tests have to be perfectly stable.

Be pragmatic, especially in start-ups where time and delivered value are of the essence.

In the end, stability is one of the most critical factors that affect whether your tests are bringing value to your product or not. Instability will quickly lead to unhealthy habits of ignoring the results altogether, even though among the flaky results are actual failures that need to be investigated.

To have confidence in your product, you need to have trust and control over the results of your testing process.

While a solid setup of QA automation can prevent most of the errors that can happen in your software development process, it should never be relied upon as the sole way of risk mitigation.

You should always be prepared with a plan B. Something will go wrong regardless of how many countermeasures are introduced due to the significant complexity of web applications and their environments.

Plan deployments in advance. If it’s a major one (especially one that includes database migrations), make sure you have a battle-tested rollback mechanism in place that you can employ yourself.

Something that can also go a long way in increasing your confidence is code review. Make sure you are not cutting corners - a lot of problems can be caught in the peer review process alone. The functionalities and business logic can most of the time be verified by looking at the code alone.

Make sure everyone in your development team reviews other people’s code as thoroughly as possible and questions any inconsistencies between the implementation and the specification.

There is a lot to consider when introducing QA automation into your project. This is why there are no certainties or strict rules that have to be followed in order to achieve full reliability. Every start-up is different and will require a specialized testing process that should also be adjusted as the project evolves.

If done right, automatic tests can be one of the greatest ways to give you confidence in your product. Confidence, which will become an enabler in many aspects of your start-up, not only Friday deployments.

Confidence needed? Scroll down and tell us more about your product's needs by filling out a submission. Next, you can expect an answer from our expert and free consultation call proposition to draw up a plan tailored to your business requirements. Give your product a new life. You deserve a stable solution that will help reach your business objectives.