00:27 Marcin Hello everyone. My name is Marcin Kokott, and welcome to Product Odyssey, a podcast series where we explore how to build digital products fundamentally better. Today, we have the pleasure of hosting Katja Obring, a distinguished trainer, coach, and advisor based in the UK. With over 18 years of extensive experience, Katja has held key roles such as QA tester, QA engineering lead consultant, and head of delivery. She is the visionary founder of Katja Coaching, where she partners with organizations to enhance their test leadership and strategy consulting. Katja excels at initiating transformative changes in testing processes.

01:01 Marcin In addition to her consulting work, she offers comprehensive online training programs and has recently launched her latest course on developing a testing strategy. For more details, you can always refer to the links provided in the show notes. Katja, it's a pleasure to meet you.

01:16 Katja Hi Marcin. It's a pleasure to be here.

01:18 Marcin We recently met at the CTO conference, and I was hugely impressed with your presentation. I wanted to invite you to discuss the testing topic. Let's set some context here. I would assume that we have been developing products for more than 25 or 30 years in an agile or any kind of modern manner. Yet, we still face situations where we are tirelessly bringing the product to the market, fighting for months to make everything happen.

01:57 Marcin Then on the launch date, excitement is in the air, but something goes wrong. We have defects, defects that put our reputation at stake. Nothing is working. We complain and go back to the drawing board to fix things. It sounds like a nightmare, but it's not just a story of yesterday; it still happens. Do you think we still struggle with having defects in products all the time?

02:30 Katja Yes, we still struggle with defects in products. There are two sides to this. Agile, while not a new concept, hasn't necessarily been well widely adopted. Many organizations, especially big ones, do something they call agile, but it's not truly agile. There's still a bit of the old waterfall approach.

03:02 Katja In many organizations, there are change approval boards, for example. No matter how good you are, there will always be defects. The question is how early you can catch them or, even better, prevent them. That's where we are these days.

03:25 Marcin Do you have any examples of failures you've spotted working as a consultant or any recent experiences around the worst-case scenarios that can happen?

03:38 Katja To be honest, the teams I work with focus on building quality from the beginning, so we generally find defects before the launch date. It's been a while since I've seen a seriously delayed launch. However, I did encounter one where the authentication system didn't work with the updates because they were only tested in the staging environment, which was configured differently from the live environment.

04:11 Katja This kind of problem still happens, especially with hosted solutions, not cloud solutions, where you can spin up an environment mimicking your live environment well. If you have a manually managed staging environment, discrepancies between staging and production can lead to all sorts of defects.

04:45 Katja Yes, we've seen that more often than we'd like to admit. Talking about the differences between staging and production environments, the improper implementation of agile approaches, and myths and anti-patterns people still hold, do you have examples of myths you see in the market?

05:34 Katja One of my favorite myths is that if you have great developers, they will not create any defects.

05:41 Marcin Oh, right. Yes.

05:42 Katja If you're just good enough, if you write good enough code, there will be no defects. That's really interesting. All the great developers I've worked with know that writing code inevitably creates defects. It's like breathing; you create carbon dioxide. It's just how it works. This myth is one of the biggest I still occasionally encounter.

06:15 Marcin That's probably connected to the fact that we tend to forget this is research and development. As with any industry, mistakes happen, and we need to find and fix defects early because they cost more later.

06:46 Marcin Exactly.

06:47 Katja If you have a simplistic application, your chances of writing relatively defect-free code are higher. But the more complex your application, the lower the chances because no one person can hold all the information about the whole system. This means things will go wrong. You need a mechanism to catch and handle issues quickly.

07:18 Katja Right. You need to have a mechanism in place to catch and handle issues quickly and effectively.

07:32 Marcin Yes, we need to debunk the myth that good developers write defect-free code. It's possible under certain circumstances, like a simple system or a slow development process, but complexity and speed matter.

08:03 Marcin This brings me to another myth: testers are just there to find defects and complain. Some people still see testers as unproductive.

08:31 Katja Unfortunately, this view is still prevalent. There is an old-fashioned school of testing focused on finding weaknesses and defects. Modern testing, however, focuses on building quality from the start, working with developers on automated tests, test cases, and clear requirements. Many production bugs result from misunderstandings between design and implementation, so early clarification is crucial.

09:48 Katja It's important to clarify requirements early to avoid building in defects.

10:10 Marcin Exactly. I'm trying to dig deeper to find fundamentally different approaches or hints for C-level executives to understand better and make better decisions. Based on research, most defects arise from envisioning how things will work and missing edge cases. You can't find all defects just by thinking long enough as a developer, especially in complex systems.

11:27 Katja Exactly. Building quality applications boils down to good communication within the team and among people with different roles. I'm not saying you need a dedicated tester on your team, but you need someone who thinks about quality and testing strategically.

12:07 Marcin Right. It's not about having a dedicated tester but rather someone skilled for the job and more efficient, thus lowering the project cost.

12:40 Marcin This brings up the human factor. Developers want to create and move fast, while testers focus on finding issues. Does this cause tension in your experience?

13:04 Katja Quality is everyone's job. One person should focus on quality, thinking about risks, proving implementations work, and avoiding negative effects elsewhere. Traditionally, there's tension between QA and developers, but modern testing focuses on a broader approach to quality.

14:16 Katja Modern testing focuses on a broader approach to quality.

14:30 Marcin Absolutely. Our audience includes C-level executives and startup owners. Assembling a team, should they hire a tester early on? What attributes should they look for in a good tester?

15:01 Marcin It depends, but bringing in a tester or someone who can cover the tester role early is a good idea. Build a quality culture from the start. For early-stage startups, hire someone with versatile skills, proficient in coding, and interested in the delivery process.

16:38 Katja It's important to bring in someone with versatile skills who can contribute to quality and the delivery process.

17:30 Katja A good tester thinks strategically, understands delivery processes, and can help the team hone their agile practices. They should be versatile.

18:01 Marcin Perfect. As a startup owner on a budget, should I hire another developer to speed up the time to market or a separate tester? Should I go for a manual tester or an automation tester?

18:34 Marcin Don't hire someone solely for testing. Look for a developer with an interest in testing or a testing background. Everyone on the team should think about testing, not just one person. Automate everything that makes sense, starting with unit tests and running them in the CI/CD pipeline. Manual testing should be exploratory, and done by product people and developers.

20:33 Katja Humans should interact with the product, but this should be a team effort, not a separate activity.

21:06 Katja Manual exploratory testing can be part of everyone's job, especially product people who use the application daily.

21:13 Marcin Right, it could and should be part of the job.

21:17 Katja Exactly. No need to separate these activities. Training in testing methods can be beneficial.

21:40 Marcin Quality isn't the responsibility of a single person. It's a team effort. For practical steps, how can startup owners ensure a quality mindset and culture?

22:20 Marcin Think about quality strategically. Have a test strategy, revisit it yearly, and maintain good development practices, like unit testing and a CI/CD pipeline. Also, have a disaster recovery strategy. Quick and effective recovery is part of quality.

23:39 Katja Yes, having a good disaster recovery strategy is crucial for maintaining quality.

24:02 Marcin Many startups prioritize defects without assessing impact or urgency. This can lead to prioritizing less impactful issues while missing critical ones.

24:39 Marcin Yes.

24:45 Marcin Regarding test strategy, should it focus on product risks or a mix of listing risks and having a mitigation plan?

25:11 Katja A mix. Start by listing all risks, then focus on the top three main risks. As your application and audience evolve, your risks and quality

26:25 Katja Don't focus too early on uptime metrics. Aiming for 99% uptime can be expensive and unnecessary for most businesses. Lowering it to 95% is easier and often sufficient.

26:39 Marcin Great point. Measuring quality isn't just about defect numbers or performance. It's about what matters to your business and customers.

27:10 Marcin Metrics should guide, not be the goal. I don't usually count defects in my teams. A zero-bug policy means fixing bugs immediately or within two weeks. If it's not important enough to fix quickly, it probably never will be.

27:51 Katja Having a short backlog is crucial. Close tickets that haven't been touched in three months. Focus on what adds value to your business now.

28:37 Katja Exactly.

28:38 Marcin A short backlog is essential. Avoid accumulating too many tickets and causing stress.

29:07 Katja Focus on what adds value to your business right now.

29:20 Marcin For measuring quality, what basic metrics should we look at?

29:45 Katja DORA metrics are a good start: deployment frequency, time to deployment, and how long it takes from idea to customer. Shorter times mean quicker feedback and better alignment with customer needs.

30:31 Katja This is probably the number one metric.

30:34 Marcin Right?

30:35 Katja Also, track how many releases cause production issues. This gives you an idea of overall quality.

31:23 Marcin Exactly. Counting defects alone isn't helpful. It's about how quickly you fix them and identify recurring issues.

31:46 Katja And ensure your regression tests run before release. Automated tests should catch issues early.

32:22 Katja Count only the bugs that make it to production. This gives a clearer picture of quality.

32:47 Marcin Yes, tracking bugs found and fixed before release isn't as useful.

32:55 Katja Exactly.

32:58 Marcin You're right. If automated tests catch bugs before release, there may not be a ticket. Developers should fix their code before it goes into the main branch.

33:19 Marcin You mentioned running regression tests every time the code is pushed. How long should these tests run?

33:49 Katja Ideally, under 10 minutes. If I start tests, I want them done by the time I return with a drink. If tests run too long, they should be shortened.

34:29 Katja Long test runs can be an anti-pattern. Fast feedback loops are essential. Overnight tests aren't fast feedback.

35:12 Katja If you have separate test teams, especially offshore, they may test through the UI, which is slow and prone to breaking. UI tests should focus on the UI, not everything else.

37:44 Katja UI tests are slow and hard to maintain. Separate UI and API tests for efficiency.

38:44 Marcin Testing should provide fast feedback on system performance. Large numbers of tests running overnight aren't effective.

39:11 Katja Some teams boast about thousands of tests, but many don't add value. Tests should provide valuable information about the system.

39:43 Marcin It's crucial to shift testing left, involving testers early in the process and ensuring team effort.

40:17 Marcin For C-level executives, focus on how testing helps deliver value to customers faster. Use DORA metrics and avoid making metrics the goal.

41:12 Katja Ask how testing helps deliver value quickly.

41:15 Marcin Yes, that's the key question.

41:18 Marcin Communicate with departments using DORA metrics for insights. Consider cost implications and avoid silos in testing.

42:27 Katja Testing should not be a separate quality gateway. All developers should be responsible for testing. Hire versatile people who can also write production code.

43:34 Katja Hire quality advocates to guide teams strategically, especially if you have multiple development teams.

44:53 Katja Quality advocates help ensure teams work harmoniously and avoid conflicts.

45:04 Marcin Testing should be integrated into the development process, not done sequentially or outsourced at the end.

45:36 Marcin The shift-left concept involves testing from the start. Continuous testing is key, starting in the design phase with techniques like story mapping.

47:04 Katja Continuous testing and story mapping are practical ways to shift testing left.

47:09 Marcin Yes, practical approaches like continuous testing and story mapping are essential. Quality is a team effort, not just the responsibility of one individual.

47:45 Katja Tools are not solutions but steps towards solutions. All problems in technology are ultimately people problems. Think about the people behind the tools for success.

49:40 Marcin This conversation has opened many doors for further discussion. Thank you, Katja. For those interested in learning more, we'll include links in the show notes. You can find Katja on LinkedIn and subscribe to her newsletter.

50:17 Katja Yes, find me on LinkedIn or subscribe to my newsletter at newsletter.katjacoaching.com. I also offer a free five-day email course on shifting left.

51:22 Marcin Thank you, Katja. It was an amazing discussion. Any final takeaways for our audience?

52:01 Katja Have a test strategy, build a quality culture from the start, and have at least one quality champion on your team. These are my main pieces of advice.

53:17 Marcin Thank you. If you see signs like nightly regression tests failing frequently, it's time to improve. Delete consistently failing tests and focus on valuable ones.

54:09 Katja Exactly. If nightly tests fail frequently, delete them and focus on what's important.

54:19 Marcin Thank you, Katja, and thank you all for watching. Don't forget to subscribe. Stay tuned and keep exploring. Thank you very much.

54:40 Katja Thank you. Bye-Bye.