00:36 Marcin: Hi, my name is Marcin Kokott and welcome in Product Odyssey, the series where we explore how to build products fundamentally better today with me is again, Maciek, CTO at Vazco, and Adrian, Tech Lead at Vazco. And today we'll be talking about developers productivity. Think that maybe doesn't sound controversial, but as soon as we'll land on discussions with the C levels or the startup owners, it's problematic to find out what is the metric, how to measure productivity, what is one single area that you can put on your table and see immediately is my team efficient or not?

01:14 Marcin: Is developer efficient or not? And let's start from basics – why we should actually measure and why there is so much need behind for measuring developers or team productivity. Any, any points to start with?

01:30 Maciej: I think the question why you want to measure developer productivity depends on the role you have in organization. So there are multiple roles that want to, wants an answer to this question, but the actual question is different depending on why you need to add this question. So for example, like CEO wants to know developer productivity for different reasons than CFO, right? And CTO wants wants to know the developer activity for completely different reasons than for example the HR departments wants to know.

02:02 Maciej: So I think the best idea here to start is to basically speed this question by role and answer it one by one.

02:08 Adrian: Yeah. Adding to that, you can even ask this question to, to lower levels of in your company. For example, even developers probably want some feedback or, or some metrics to know that they are performing well and, and so you can better evaluate whether someone deserves promotion or, or, or who is underperforming.

02:41 Adrian: So you can, so you can help those person like debug their productivity, right? So to find out what is going wrong,

02:51 Marcin: Right? Right. For promotions, for development and also from the perspective of the, the board, we can think about monitoring, am I on the right track with budget of time? I'm on the right track with the, with the product development and timelines <affirmative>. And so I want to just know if the organization is efficient. Any, any other specific examples or the use cases that, that you see are unique or you have spotted in, in your experience that are not from this BO box?

03:18 Maciej: I think one of additional reason that quite often is coming up in many organizations is we want to measure developer productivity because we want to measure everything and we measure other departments, right? So we measure hr, we measure sales, we know exactly detailed metrics on other like production kind of department. So we can make improvements. We, we can calculate whether this business line is actually profitable for us.

03:50 Maciej: So we have metrics everywhere. And at the same time there is this engineering department kind of like magical, we need them, they are doing a good job for us, but once we try to put some metrics on them, they tell, they tell us that, well, it's not possible. We cannot measure it. It's intangible. It's basically you cannot do it. Go,

04:15 Marcin: Yeah, yeah, yeah. Don't do that, don't do that. And we are searching for this one single metric, right? Ideally, yeah. So we are searching for a silver, a bullet and this one metric that tell me all who to fire, who to promote. Is it, is it on, on track or not, right? Yeah,

04:29 Maciej: Exactly. Other departments have you, you know, how many like deals are being closed, right? Mm. Or whether you're profitable or not. But in, in engineering departments, it's really hard to find this kind of one single metric. And for many non-technical or less technical CEOs or board, it is very attractive to them to also find this single metric for, for the engineering department.

04:53 Adrian: And, and this and this thought that, that you can measure every, every department comes probably from, from the past where the engineering that we know today consists like it, it, it was tried to be compared to the old things like when you are working in a factory, you are being measured, for example, how many cars did you produce? Right? <affirmative>. And, and we tried to reduce the complexity of, of the engineering department to, to this kind of like a single metrics, but like it, it's not desirable to be just a single metric, right?

05:38 Marcin: Yeah. And this is actually maybe not even complexity, right? But this, I agree with this, this, with this fact that we are trying to, to streamline that and conquer it to any other from the past. But this assumes, as you have mentioned, the production mode. So this, this kind of like a predictable approach where whereas else we know that product development is research and development with big focus on research. So we are in innovating, so it's not basically a sequential, right? So therefore you cannot apply the same metrics.

06:10 Marcin: Still we agree that you can measure that it's only the, the concept that it's for the different lenses, right? You need to look at it differently.

06:19 Adrian: <affirmative>. Yeah. I I would for sure say that you want to measure something or maybe you just want to measure something, but then you need to think about what you'll get from measuring that. Right? Right. Because going back to this example of the cars, like you, you can, you can assume that creating a car means that you can sell it. This means profit right In, in engineering that we know today, you kind of like, you don't, you don't produce many, sell many sellable co copies of the <affirmative> of the software.

07:05 Adrian: You create a software, right? And then you sell it based on various, so subscriptions or other ways of,

07:16 Marcin: So everything is unique, right? This is what you're trying to point out. It's not repeatable process. It's not that I'm creating 27 in this example, 27 cars or parts, right? I'm, I'm creating almost a different feature of the car or a different car every time, right?

07:31 Adrian: Kind of <affirmative> like I I would also say that this is more of a continuous process <affirmative> than like a finished one. Like creating a car means that it is done right from point A to B.

07:44 Marcin: Okay. And what you're suggesting is that in the software development, it's not so that you can define what is this and beginning and the end, right? That's what you're saying.

07:54 Adrian: Well, <laugh> in, in my career, even after you finish a product, there is the maintenance Sure.

08:05 Marcin: The lifecycle <affirmative>.

08:07 Adrian: Yeah. But,

08:09 Marcin: Okay, so it's, I think that what, what we are learning here is the, the concept of if we're measuring depends on, we haven't entered that yet, but it depends of what, what portion you want to measure of your work output or outcome. And what you're saying is that on the scale of the lifecycle of the product, we are not finishing a product. So it's not project management that I have a defined the project, it's finished, it's done, I'm going home. It's more or less co continuous evolution of that. And therefore it's hard to say that I'm product, if I finish this project and this kind of metric for developer number of completed projects, it's probably not the level that you want to handle, right?

08:48 Marcin: Yeah.

08:49 Adrian: What, what I'm trying to say here is that it just cannot be reduced to, to the simple thing that you can measure on <affirmative> <affirmative> to, to a one single thing. And that's basically it's like to, to sum up, right? Right.

09:04 Maciej: I think you mean actually no, like you need cost for a software, right? So once the, once you create the software, it is infinitely scalable, right? So okay, you can argue that there are hosting costs and and so on, but basically there are, those are like not, not important in the grand scheme of things. So basically there is like this diminishing cost of additional item, right? So this is completely different from like manufacturing where you can actually split the cost of producing something into like unit cost and overall production cost.

09:39 Marcin: <affirmative> before jumping into then the, the the what to measure or how, because this is already what, what we're attaching just wanted to close this on, on the level of justifying that the, we, we agree all that there is a need for measuring, right? It's not something that we, we want to say that there is no justification. So don't measure at all because there are wrong, the the need is there. We just need to agree what is a good way to measure it or a good way to evolve, right?

10:04 Maciej: I think the need is there and yeah, maybe I'll say something controversial, but I think that from a CTO standpoint, we should measure developer productivity. I'm not saying this is easy and I think we'll talk later on how to actually do it properly, but we should measure it otherwise people with those needs like CFO or CEO or board or, or other departments that actually have those needs, the needs would not go away just because we not want to <laugh> measure it. They'll find a way they will hire like McKinsey or consultancy, they'll basically go in, bring in their frameworks and they'll find some way to measure developer productivity.

10:43 Maciej: This may not be a right way to do it. This may be actually may produce a very, very bad result in the long time, in the long run, but still they will do it somehow. So to, to be proactive and to give them solutions for those questions, we should actually try to measure it before and in the process it'll also help us to be more, more productive.

11:08 Maciej: Yeah. Because there, there is many valid reasons to, to measure developer productivity, right? So in case of board or like CCEO or CFO, you want to make sure that if we put money on the project, this project is actually going to be, we're going to have some kind of return on it, right? <affirmative>, in case of HR department, we want to ensure that we promote the right people, that salary is fair, that we fire the right people, right?

11:39 Maciej: So let's say that we need to do a cut, right? Right. And how do we decide who we need to let go, right?

11:46 Marcin: Who to promote, right? And

11:47 Maciej: Work. If we cannot measure it efficiently, then yeah, then, then this very hard questions are becoming impossible, right? <affirmative>. And there are many, many valid reasons to actually measure developer productivity, but most often than not, the need of the questions that those group ask are actually, they're asking for a different thing. That they're, they're original, original needs are different that they're asking for, right? Because they want to create developers or, or different kind of metrics calculated to a, to a single digit to basically order some, some projects or people in line and then make a decision based on this most often than not.

12:34 Maciej: And I think the right approach here is to actually go one step deeper, ask why they want this measurement, why they want to have this list, what their actual goal is here, and then try to find out some solution to actually help them with their problems. And at the same time, do not harm the developers in the process.

12:55 Marcin: Right? Just to summarize those three points, because you have added one, you have added the urgency, right? So the, the fact that it's not only that the need is there, but if you don't address it as a development team, as an engineering lead, if you don't address it right now, address it right now, sooner or later somebody will find out either somebody who measured that or measured that correctly or not correctly. And therefore it's good that you find it on yourself sooner on. So it's urgency. And then those two points that, that you've mentioned now on, on conseque consequentially, the asking why and defining what is behind the question.

13:34 Marcin:** It's, it's actually almost moving back as to, to the product development and agile, right? If I have a user story functionality, I need to, it's good that I'm asking why the person need a fork.

13:43 Marcin:** Is it for eating or for killing somebody or anything else? So the the why behind is really important before you find out the metric. I remember the, the good metrics or frameworks for coming up with the, with the KPIs was gold question metric as you can rely on, which again is the same concept, right? You start with the first d the business goal, then the question, how to actually measure that. And then, then you only land on, on the metric, right? But that's, this is then we are opening the, the back with metrics.

14:13 Marcin:** So what are the good, or maybe even starting with the anti patterns, what are the worst metrics I can put on developers?

14:20 Adrian: Well, okay, so if you mentioned developers, then I would say that you, you you'll receive what you measure. So this will, this will make the company focused on hitting those metrics, on making sure that they are, they're high <affirmative> if you want them high or low, if you want them low. But this will basically shift your company in that direction. And this is, and you need to be very careful when you are deciding on the metrics that, that you that that you create.

15:00 Adrian:** Because especially developers will wanna maximize, they, they wanna maximize it and they wanna game it so they gain as much as possible from, from this, right? It's

15:16 Marcin: Not on the developers, right? It's every, every person if you measure me.

15:19 Adrian: Yeah, I'm just saying

15:21 Marcin: Identify it'll be happening there. Yeah, yeah.

15:22 Adrian: I'm just saying from, from my own perspective, right? <laugh>. Yeah. Okay. Yeah. So an example, a pretty common one. If you wanna measure the developer productivity based on the lines of code produced, then you'll receive a lot of lines of code. And especially right now in the era of, of AI and chat GPT and or communication co-pilots and co-pilot, like it can generate lots of codes that you certainly don't want to, don't want to maintain right in, in that form because <affirmative>, yeah, it, it is good at providing, at generating the, the code, but maintaining it, it's, it's not

16:11 Marcin:** Sure. So number of features, right? That would be the understanding, right? We are going to output,

16:17 Maciej: Yeah, you can measure number of features. You can for example, measure how many story points developers are comparable in a, in a sprint. So we got this velocity metric, right? And this is what many companies are actually doing and this can be done right? But it's also very easy to, to do it wrong <affirmative> and also received, but, but result, because there are going to be some kind of like inflation of it. There is also lots of opportunities to, to tweak the, to tweak the metrics and basically we don't want to have the feature out or this pull request out or the story points completed and so on.

16:58 Maciej:** This is not what we are actually measuring for, right? We, we mo at least most of the companies are not being paid for producing story points. The <laugh>, their goal is to actually put out on the market some kind of feature and we just use the story points to measure complexity of a, of a feature. But the goal is to have this feature on the market and to actually get some, make some impact on the market of this feature, right? So if you, if we are going to measure story points or for example, I dunno, pull requests with those features, right?

17:32 Maciej:** <affirmative>, then we are going to just have more pull requests because like you mentioned, developers tends to be like creative people and they're great on finding loopholes and optimizing things. So if there is possibility to game the metric, okay, so

17:47 Adrian: Just do it. I wanna, I wanna add actually one thing to, to this pull request metric because there can be use cases where this is actually useful because if you are having, if your developers are, have the tendency to create big pull requests and this makes a code review a pain point for other developers, then maybe if your goal is to, is to reduce the time for a review and, and make developers happy that they can quicker review code or get the code reviewed, right?

18:31 Adrian:** So then a number of pull requests might impact this, right? Because

18:36 Marcin: But that's not the productivity, right? It's the, the, the metric or the mean to create the right behavior, right? This is what you're saying. I

18:42 Adrian: Mean this might be a tool, right? Yeah, of course. But but again, like go, going back to what magic said is you need to ask yourself like why are you, why, why are you measuring this particular thing <affirmative>? And if you're go, if, if you are stating, if you're making a statement that creating a lots of pull requests will bring me money, then you're probably wrong, right?

19:12 Marcin: <affirmative>, okay, so don't measure, don't measure the what, what is the output in individual output? I get that point, right? So I

19:18 Maciej: Think you can measure because like, like you mentioned, this is a, this is a tool, right? And all, everything depends on how you're going to use this tool <affirmative>. So if you are using this metric of number of pull requests or lines per pull request and so on and so on, this is fine if for example, you as a CTO wants to like debug issues in your project, so why, why the pull request takes so long or other, other things. So this completely fine to do it in this regard. Once, once you put the money into the mix, so you start using these metrics for let's say like performance reviews by the HR department or any other reason that actually involves

20:00 Marcin: Some,

20:01 Maciej: Some effect that is very, that might be valuable for the developer, then this is where all additional guns will start to happen and they will basically try to optimize it. And this completely, completely makes the debugging the issue part of it. It just makes it impossible.

20:21 Marcin: It's, it's something that that's we've said two times or three times already, right? That's, and and I agree totally on that. That's, it's, there is no single metric for productivity. You can measure and you should measure a lot of things and you can measure it depends on what you want to achieve for a behavior. You can measure per request, you can measure number of lines of code if that's connected with some sort of right behavior change that you're after. And there is another point that you, you have mentioned, and this is the, the fact that you should measure lots of things as indicators, not as the one single metric that caused the, the decision, right?

20:55 Marcin: So it could be information for you, but <affirmative> it's depends on how you want to turn it into insight, which will be important. But let me challenge that a little bit from the metrics perspective. So lines of code, no story points. Note we have discussed that that's not ultimate metric. It should be what is the output. So a Adrian from today, your only goal is to increase revenue of our product by 25%. Go and tell me how productive you are.

21:25 Adrian: Okay? I would say that certainly it is not only me that that will ma make the money. And I want to emphasize here that we, or rather it is very hard to measure individual performance and it's, it's, it's easier and I believe it's, it's better for the company to measure the team's performance when it comes to going after the, the big goal, right?

22:00 Adrian: That that your goal is to make money to, to deliver things. And it is a lot easier to measure how many features a team delivered and how resilient they are. Like they are not broken on, on the production. And measuring that is, I believe a lot closer to the thing that you're looking for.

22:27 Marcin: Okay? Okay. So not individual metrics, but there is this point of, i, I can measure the team and then you'll say that it's not one team, but we have development teams and then there is a product management, there is a marketing. So I need to measure the, the whole department. But I'm trying to just maybe find out like I think behind the fact that if I'm measuring too low to on the, on the level of lines of code, it's bad and we need to know why. But if I will go too high and it'll be too far detached from myself, I don't know how I'm impacting that.

22:59 Marcin: Therefore it would be ultimate metric, right? If you can have a metrics and, and have a system that I know that if I'm doing this decision, it'll impact this metric and automatically it'll impact revenue. I would love to have a developers which are coming up at saying like, we need to do this feature, not this one because that's a higher revenue and we know it and that's how our hypothesis, right? But probably we're talking also about on what level I should have this, this question answer why or what is the impact of this metric? So the revenue probably is not the goal, right?

23:32 Marcin: Is there anything between, yeah,

23:33 Maciej: So this is a difficult question and we actually make it even hard currently because you ask about the bad ways of measuring it. We talk about like lines of code and never think, I think this is quite obvious. And also there is this big other part like, like the revenue, like the, the, the final impact. And if you are a for profit company, then yes, this should be the goal for everyone, you know, inside your company to actually improve the revenue or rather improve the shareholder happiness because it's not always the revenue, but there's a different topic for the different discussion.

24:08 Maciej:** But there are many stages between you are actually doing something as a developer and actually having this result as a, as a increased revenue. So one model that I particularly like is to categorize this into, into four phases. So to so to speak.

24:28 Maciej:** So there is, there is the effort. So all of the things that you actually are doing, you are trying to do the things that you can actually do depends on you, on you and on your soul actions based on those efforts, there is some output. So let's say that you do, you work for many hours, you write code, the output of it would be a code written, right? There is some kind of like some kind of deliverable that you can share with others, other people can also contribute to it work based on it.

25:04 Maciej:** And this is the output of, of your work. But many, many metrics actually focus on those first two steps. <affirmative>, because it's easier to, to measure, you can measure the effort, how long is someone working or the output, so how many code someone has written. But this is not, not the goal.

25:24 Maciej:** We're not writing for it. We, we are not writing the code for the sake of writing the code, right? It's most often it should be actually the opposite. So what we should measure is why we have written the code in the first place. So the outcome, so after the output we get the outcome. So what is the effect of the, the code that we have written? So in most cases there is some kind of feature or some kind of change in our app or some, some kind of behavior that will by this code induce on our users or our customers or any kind of change we want to make in the want to make in the, in the world.

26:06 Maciej:** And this is the, the outcome of, of our work <affirmative>. And this is not the end of the story because the outcome itself is not the the final step because what we actually are striving for is the impact. So the, so the last step, so the outcomes I hope should affect in the impacts that we are hoping for, right? <affirmative>. So this is the, the holy grail of of doing things basically. So we got the effort, output, outcome impact <affirmative> categories.

26:40 Maciej:** And as we go farther down the road, it's less and less individual com contribution. And more and more depends on basically the whole teams or even the whole organization.

26:56 Maciej:** And it's being harder and harder to measure, right? It depends on your metrics, it's,

27:02 Marcin: But what you're probably trying to say, just correct me if I'm wrong, it's not the, the fact that a high further, I, I will go in this graph, the, the better metric I will have, it's, it's about the fact that it's of course more meaningful metric down the road, but it need to be a system, right? <affirmative>, that's I need to, if I'm having one metric, I need to know what kind of metric and outcome it, it's impact and what kind of metric and impact it is actually connected, right? <affirmative>. So having this as a system to visualize the impact. So it's like, okay, or or if you go with the, with a good question metric, I need to have a connected system.

27:36 Marcin:** So I know this single metric is not for me to judge as a loan, but I need to judge it as a system, right? <affirmative>, so if this is not working, I'm looking what is the down the road, is that what what you're trying to, to say?

27:47 Maciej: Yeah, exactly. Yeah. So you should look it at as a, as a whole picture and basically all of those steps need some have needs to have some kind of check and balances to ensure that it makes sense overall, right? So written like spending lots of time on writing multiple code lines of code and delivering lots of feature, it's not meaningful on itself, right? Those features actually need to, needs to be, need to have the, the, the impact on the actual market and our users, right?

28:26 Marcin: Delivering training features doesn't mean anything. Yeah. Unless it's actually a return on investment or the impact. Yeah. But then if we'll go right now and, and other will go to the team and say like from now on we are measuring not the number of features that we deliver, we are measuring the impact. So the revenue itself, how you are measuring day to day that you're doing a good job or not, what's the measure of progress then? Are you able to derive it from this <affirmative>?

28:50 Adrian: So there are some methodologies for that that are maybe like a guidelines for us. What is a good outcome that that can lay down a good impact that, that have a good impact on your company? And like the first one that comes to mind is Dora, that comes from Google. And, and that one basically to put it simply, it measures for example how often you deliver things, how often you update your, your software, what is the time for that software to recover from errors on, on production environment and what, what is, how many features that you actually deliver to production are working the way you design it to be.

29:54 Adrian:** So what I mean here is are there any bug on production, right? <affirmative> and yeah, so these are like the main, main points of this, right?

30:04 Marcin: So what you're saying is that again, we, we've moved away from basically finding a good metric, one metric, but we are, we're talking about the system and what you're saying is that we should rely, and that's correct approach probably, right? That's we need to rely on, on the good practices that are on the market because they came from, from the experience RA is actually built framework from that. So taking experience, taking, building on the experience of the others, we can have those set of standard metrics to measure the individual and the team productivity in the modern systems.

30:36 Marcin:** They are mostly on DevOps, but basically this is the lead time. So I agree on that. There is again, probably we can mention the space extension of that. So we have, we can comment, connect that and, and have the system that that measures better. The latest McKinsey articles was actually pointing to that as, as we have discussed recently on this. But that's, that's giving you like the, the bucket of of or the CEO of metrics. What, what I can select any hints on should I take just everything?

31:06 Marcin:** And let's see what will come up and start to measure that because this will still no answer. Your, your my initial question, <affirmative>, how do you know of day by day that you're productive or not?

31:21 Marcin: And that's a hard question, right?

31:22 Adrian: And that's a hard question

31:24 Marcin: Because the, what I'm after is the, the fact that we, we are agreeing that the good metric will be probably not the lines of code, but what you do day to day is actually lines of code. The interesting thing is only to not look at it from the perspective of just this one separate metric.

31:44 Maciej: I actually disagree, okay? So like modern developer work, job they're actually doing is not writing code, right? So

31:52 Marcin: That's what I was after.

31:54 Maciej: Okay? So you, you, you got me if it life would be much simpler if all of the developers do every day is just to write code and then it would be much easier to hire new developers and probably by now all developers would be gone because AI could also write code and even not today, then in like next or next few years probably AI would be better at writing code than most of the developers.

32:25 Maciej: But writing code is just a small part of the developer workflow. The the actual goal of engineers is to engineering <laugh> things and to actually make the right things right to to, to evaluate, to, to find better solutions to a problem, to think about the problem and try some maybe other solutions that are probably better suited for solving the problem, right? So this is why the, all of the developers should be put in a loop with your business goals.

32:59 Maciej: Why? This is why putting the revenue or other high level impact metrics on their day to day minds is actually a good thing because this is how they can change their behavior. Maybe this feature does not make sense. Maybe they can so make, you can make, maybe they can drop it, maybe they can deprioritize other things, maybe they can, they can prioritize other things. So if we empower them to be proactive, if we put them into the loop, then they can actually, then they can make the impact, right?

33:31 Maciej: They can actually and force make the changes in the daily work that will actually have, have, that will actually have some impact on the outcome that you're looking for,

33:46 Marcin: Right? To too many often to too many. Often too, too often I, I see, and this from my experience, even from the past, I see this situation that we try to hide ourselves as I'm talking about the managers, and I'm talking about c-level, we try to to, to hide behind the, the metrics and make a decision from the perspective of the dashboard, right? And to just simplify it. And very often we find ourselves in the situation that I, I want to have this metric to just not go to the team and not ask and just make a decision based on that because it's too much for me. I cannot go there. Whereas if I'm even going to the team and I will sit down with them and I will start to ask question, are we effective or not?

34:22 Marcin: Immediately? I, I haven't seen the, the situation that the team will say, I dunno if I'm effective or not.

34:28 Marcin: I'm probably, you know, all the metrics are saying that it's good. No. Every single team will say, oh my god, do you want to have it alphabetically or by the impact we can change all this stuff. Which means that there is two things, right? There are two things. One is this is not the something that you can capture through the one metric, but it's obvious if you spend the time with a team and you can figure it out with a team, right? <affirmative>, what kind of improvement you can do, what kind of efficiency you can boost or if we have efficiency or not deliver experience, metrics and so on.

35:00 Marcin: But the other thing is the, the fact that this kind of single metric assumes that writing code is the only thing that you do and which is not the case, which is creative work and it's hard to capture speed of problem solving, right? So I have solved seven problems, class B and then 27 story points and two lines of code and produce five bugs. Am I productive? So there is the human factor of that probably. And then there is also the fact that measuring is not only measuring the right, the lines of code, this opens lots of doors, but I don't believe that if you ask anybody measure my productivity, they will point out those kind of things, right?

35:41 Marcin: And it's omitted and it guesses why

35:45 Maciej: Which part is omitted.

35:47 Marcin: I mean the, the fact that it's not the single metric, we, we still try to find out the, the server bullet for that, right? <affirmative>, I want to measure the developer productivity. Are they the productive? If I would come to a team, I can get that information right away, <affirmative>, if I'll bring to teams and and discuss how productive you are, what is your system? I can even say, and this is not the hard metric, but I can say which team is more productive or not, it's a feeling also <affirmative>, right? Yeah. And can I capture that as a metric?

36:15 Maciej: You can sometimes, but it's not, it's not easy to do, right? Because this is, this is basically a difference between like qualitative and quantitative metric, right? <affirmative>. So we want to have some kind of number at the end and we need this number because we probably want to sort d similar things in order, right? To make some decisions manage for Exel. Yes. This is why we win this number, right? And like you mentioned this, the actual pro, the actual product developer productivity is something that you cannot quantize into a single number.

36:48 Maciej: This is something that is obvious for someone working day on a day-to-day basis, right? So if you go and talk to some tech lead or other developers in the pro in the, in the project, they will most likely be aware that this guy is much more productive than this guy, right? This comes often is corrected with seniority and also with other factors.

37:10 Maciej: But people tend to feel this kind of thing once you go up and up and up, it's less visible and you need to be based on some kind of numbers. This is why you're asking to produce those numbers so you can act on them, right? Yeah. And it is really hard to to to, to quantize. And I think that the role of CTOs or or tech leads is to actually educate the people that are asking those questions, to educate them why this is a, a very difficult topic and try to rephrase the question how, what, why do you need this number?

37:50 Maciej:** How you going to use it? How maybe I can help you differently to solve your problem, right? Because maybe this is not the actual number you're going for. Maybe this, your need can be solved with something that I know that can be measured, but you do not because you don't have this hands-on experience, right? So the communication here is I think vital <affirmative> and we should measure, we should solve those needs of other groups in our organization, but we try to find out first why they want to do it and then try to find this best solution.

38:23 Maciej:** Maybe there is some kind of way to measure this feeling and, and quantize this <affirmative> into, into some kind of form we, yeah.

38:34 Adrian: Okay. And here I wanted to say that if you want to like going back to the question of, of measuring whether one developer is is productive or not, and the, the thing that magic mentioned is you, you need people to be hands-on with the team because it's, it's a people's perspective that know that this person is, is the most productive within the team.

39:07 Adrian: That's because it's cannot be grasped with just one little single metric because the, the world is just complex, right? <affirmative> and it's too many factors to, to put into, into place probably also it cannot be generalized for each company and probably it'll differ across companies <affirmative>.

39:33 Marcin: Okay. And even inside of the, so it'll be different between the teams and between the systems. And that's coming from the complexity from the different technology, right? It's it's a research part, not the

39:43 Adrian: Yeah, and and probably the goals that you set. Yeah. Right? That this is also a very important,

39:48 Marcin: That's again, thing, but isn't that then pushing us to understanding of then should I search for a metric or should I assume that every case is different and I need to make sure that I'm improving. So the still <affirmative> this concept of basically either system is, is improving or declining <affirmative>, there is no status quo. Always as an engineering leader, I should make sure that I'm improving with the team and continuously doing that rather than trying to find out the one metric that will tell me we are on the track or not. Right?

40:18 Adrian: <affirmative>, so I, I read, I read about this that I, some teams or rather team leaders or managers used to note what, what accomplishments they've achieved <affirmative> within the team. And probably if you have that you can probably also find out which persons stood by by this. And, and this is like something that can give you a good direction <affirmative> to, to, to kind of see what, what this person, what impact that person had.

41:03 Adrian: But I also wanted to mention one story that is not always like, sometimes you need to go very deep into the team to, to know that this person is productive or not because, and this is a story about the, the worst programmer I know. And there, there was this guy that produced no story points, each sprint and there was a decision to let that person go from, from the firm.

41:40 Adrian: But if you went down into the details, then it was that this person was actually helping all other developers on the team and making sure that team is, is performant, <affirmative> and without, and, and, and that guy on his own made, made the code base a lot more resilient because the, the solutions were, were well written and, and covered by tests.

42:14 Adrian: And basically the quality of, of the things of the features delivered was, was very high <affirmative>. And this is like you, you could sum it up with you, you should prefer quality over quantity, right?

42:30 Marcin: In in most of the cases. Yeah, in most of the cases. I could challenge that. Yeah. But that's, that actually opens another aspect which is also pretty interesting. And this is the fact that one thing is that writing a code is not always what you want to measure and what you want to focus on helping the team and removing obstacles or taking care of even through the prs. I would not say that this is like a go-to solution that you should always have a person

4oremove words in < > and < >, correct the transcript grammaticallyChatGPT

00:00 Marcin: We're searching for a sort of silver bullet and this one metric that tells me who to fire, who to promote. Is it on track or not? Right? Yeah.

00:07 Adrian: If you're making a statement that creating lots of pull requests will bring me money, then you're probably wrong.

00:14 Marcin: A good metric will probably not be the lines of code, but what you do day to day is actually lines of code.

00:21 Maciej: The CEO wants to know developer productivity for different reasons than the CFO and CTO.

00:36 Marcin: Hi, my name is Marcin Koko, and welcome to Product Oay, the series where we explore how to build products fundamentally better. Today with me are again, Maek, hello, CTO, Ed Vasco, and Adrian, hello Te Vasco. Today we'll be talking about developers' productivity. This might not sound controversial, but as soon as we land on discussions with the C-levels or the startup owners, it's problematic to find out what the metric is, how to measure productivity, what is one single area that you can put on your table and see immediately if my team is efficient or not.

01:14 Marcin: Is it developer efficiency or not? And let's start from maybe the basics of why we should actually measure and why there is so much need for measuring developers' or team productivity. Any points to start with?

01:30 Maciej: I think the question of why you want to measure developer productivity depends on the role you have in the organization. There are multiple roles that want an answer to this question, but the actual question is different depending on why you need to ask this question. For example, the CEO wants to know developer productivity for different reasons than the CFO, right? And the CTO wants to know the developer activity for completely different reasons than, for example, the HR department.

02:02 Maciej: So I think the best idea here to start is to basically split this question by role and answer it one by one.

02:08 Adrian: Yeah. Adding to that, you can even ask this question to lower levels in your company. For example, even developers probably want some feedback or some metrics to know that they are performing well, so you can better evaluate whether someone deserves a promotion or who is underperforming.

02:41 Adrian: So you can help those persons debug their productivity, right? To find out what is going wrong.

02:51 Marcin: Right. For promotions, for development, and also from the perspective of the board, we can think about monitoring if I am on the right track with the budget or time. Am I on the right track with the product development and timelines? And so I want to know if the organization is efficient. Any other specific examples or use cases that you see are unique or you have spotted in your experience that are not from this box?

03:18 Maciej: I think one additional reason that quite often comes up in many organizations is we want to measure developer productivity because we want to measure everything. We measure other departments, right? So we measure HR, we measure sales, we know exactly detailed metrics on other production departments. So we can make improvements. We can calculate whether this business line is actually profitable for us.

03:50 Maciej: So we have metrics everywhere. And at the same time, there is this engineering department that is kind of magical. We need them, they are doing a good job for us, but once we try to put some metrics on them, they tell us that it's not possible. We cannot measure it. It's intangible. It's basically you cannot do it.

04:15 Marcin: Yeah, yeah. Don't do that, don't do that. And we are searching for this one single metric, right? Ideally, yeah. So we are searching for a silver bullet and this one metric that tells me all: who to fire, who to promote. Is it on track or not? Right? Yeah.

04:29 Maciej: Exactly. Other departments have, you know, how many deals are being closed, right? Or whether you're profitable or not. But in engineering departments, it's really hard to find this kind of one single metric. And for many non-technical or less technical CEOs or boards, it is very attractive to them to also find this single metric for the engineering department.

04:53 Adrian: And this thought that you can measure every department probably comes from the past where the engineering that we know today was tried to be compared to old things like when you are working in a factory, you are being measured, for example, how many cars did you produce? And we tried to reduce the complexity of the engineering department to a single metric, but it's not desirable to be just a single metric, right?

05:38 Marcin: Yeah. And this is actually maybe not even complexity, right? But I agree with the fact that we are trying to streamline that and compare it to any other from the past. But this assumes, as you have mentioned, the production mode. So this kind of predictable approach, whereas we know that product development is research and development with a big focus on research. So we are innovating, so it's not basically sequential, right? So therefore you cannot apply the same metrics.

06:10 Marcin: Still, we agree that you can measure that, it's only the concept that it's for different lenses, right? You need to look at it differently.

06:19 Adrian: Yeah. I would for sure say that you want to measure something, or maybe you just want to measure something, but then you need to think about what you'll get from measuring that. Right. Because going back to this example of the cars, you can assume that creating a car means that you can sell it. This means profit. In engineering as we know it today, you don't produce many sellable copies of the software.

07:05 Adrian: You create software, right? And then you sell it based on various, so subscriptions or other ways of...

07:16 Marcin: So everything is unique, right? This is what you're trying to point out. It's not a repeatable process. It's not that I'm creating 27, in this example, 27 cars or parts, right? I'm creating almost a different feature of the car or a different car every time, right?

07:31 Adrian: Kind of. I would also say that this is more of a continuous process than a finished one. Creating a car means that it is done, right, from point A to B.

07:44 Marcin: Okay. And what you're suggesting is that in software development, it's not so that you can define what is the beginning and the end, right? That's what you're saying.

07:54 Adrian: Well, in my career, even after you finish a product, there is the maintenance.

08:05 Marcin: The lifecycle.

08:07 Adrian: Yeah. But...

08:09 Marcin: Okay, so I think that what we are learning here is the concept of if we're measuring, it depends on what portion you want to measure of your work output or outcome. And what you're saying is that on the scale of the lifecycle of the product, we are not finishing a product. So it's not project management that I have a defined project, it's finished, it's done, I'm going home. It's more or less a continuous evolution of that. And therefore it's hard to say that if I finish this project and this kind of metric for developer number of completed projects, it's probably not the level that you want to handle, right?

08:48 Marcin: Yeah.

08:49 Adrian: What I'm trying to say here is that it just cannot be reduced to a simple thing that you can measure into one single thing. And that's basically it, to sum up, right?

09:04 Maciej: I think you mean actually no, like you need cost for software, right? So once you create the software, it is infinitely scalable, right? So okay, you can argue that there are hosting costs and so on, but basically those are not important in the grand scheme of things. So basically there is this diminishing cost of an additional item, right? So this is completely different from manufacturing where you can actually split the cost of producing something into unit cost and overall production cost.

09:39 Marcin: Before jumping into what to measure or how, because this is already what we're attaching, I just wanted to close this on the level of justifying that we all agree there is a need for measuring, right? It's not something that we want to say that there is no justification. So don't measure at all because there are wrong metrics. The need is there. We just need to agree on what is a good way to measure it or a good way to evolve, right?

10:04 Maciej: I think the need is there and yeah, maybe I'll say something controversial, but I think that from a CTO standpoint, we should measure developer productivity. I'm not saying this is easy and I think we'll talk later on how to actually do it properly, but we should measure it. Otherwise people with those needs like CFO or CEO or board or other departments that actually have those needs, the needs would not go away just because we do not want to measure it. They'll find a way. They will hire like McKinsey or consultancy, they'll basically bring in their frameworks and they'll find some way to measure developer productivity.

10:43 Maciej: This may not be a right way to do it. This may actually produce a very, very bad result in the long run, but still, they will do it somehow. So to be proactive and to give them solutions for those questions, we should actually try to measure it before, and in the process, it'll also help us to be more productive.

11:08 Maciej: Yeah. Because there are many valid reasons to measure developer productivity, right? So in the case of the board or CEO or CFO, you want to make sure that if we put money on the project, this project is actually going to have some kind of return on it. In the case of the HR department, we want to ensure that we promote the right people, that salaries are fair, that we fire the right people, right?

11:39 Maciej: So let's say that we need to do a cut. And how do we decide who we need to let go, right?

11:46 Marcin: Who to promote, right?

11:47 Maciej: If we cannot measure it efficiently, then those very hard questions are becoming impossible. And there are many valid reasons to actually measure developer productivity, but more often than not, the questions that those groups ask are actually asking for a different thing. Their original needs are different than what they're asking for. Because they want to create developers or different kinds of metrics calculated to a single digit to basically order some projects or people in line and then make a decision based on this more often than not.

12:34 Maciej: And I think the right approach here is to actually go one step deeper, ask why they want this measurement, why they want to have this list, what their actual goal is here, and then try to find out some solution to actually help them with their problems. And at the same time, do not harm the developers in the process.

12:55 Marcin: Right. Just to summarize those three points, because you have added one, you have added the urgency, right? So the fact that it's not only that the need is there, but if you don't address it as a development team, as an engineering lead, if you don't address it right now, sooner or later somebody will find out either somebody who measures that correctly or not correctly. And therefore it's good that you find it on your own sooner. So it's urgency. And then those two points that you've mentioned now, on consequentially, the asking why and defining what is behind the question.

13:34 Marcin: It's actually almost moving back to the product development and agile, right? If I have a user story functionality, it's good that I'm asking why the person needs a fork.

13:43 Marcin: Is it for eating or for something else? So the why behind is really important before you find out the metric. I remember the good metrics or frameworks for coming up with KPIs was "goal question metric" as you can rely on, which again is the same concept, right? You start with the first business goal, then the question, how to actually measure that. And then, then you only land on the metric, right? But that's when we are opening the back with metrics.

14:13 Marcin: So what are the good, or maybe even starting with the anti-patterns, what are the worst metrics I can put on developers?

14:20 Adrian: Well, okay, so if you mentioned developers, then I would say that you'll receive what you measure. So this will make the company focused on hitting those metrics, on making sure that they are high if you want them high or low if you want them low. But this will basically shift your company in that direction. And you need to be very careful when you are deciding on the metrics that you create.

15:00 Adrian: Because especially developers will want to maximize, they want to maximize it and they want to game it so they gain as much as possible from this, right?

15:16 Marcin: Not only the developers, right? It's every person if you measure me.

15:19 Adrian: Yeah, I'm just saying from my own perspective, right? <laugh>. Yeah. Okay. Yeah. So an example, a pretty common one. If you want to measure developer productivity based on the lines of code produced, then you'll receive a lot of lines of code. And especially right now in the era of AI and Chat GPT and co-pilots, it can generate lots of code that you certainly don't want to maintain in that form because it is good at generating the code, but maintaining it is not.

16:11 Marcin: Sure. So number of features, right? That would be the understanding, right? We are going to output.

16:17 Maciej: Yeah, you can measure the number of features. You can, for example, measure how many story points developers are comparable to in a sprint. So we got this velocity metric, right? And this is what many companies are actually doing and this can be done right? But it's also very easy to do it wrong and also receive bad results because there are going to be some kind of inflation of it. There is also lots of opportunities to tweak the metrics and basically we don't want to have the feature out or this pull request out or the story points completed and so on.

16:58 Maciej: This is not what we are actually measuring for, right? At least most of the companies are not being paid for producing story points. Their goal is to actually put out on the market some kind of feature and we just use the story points to measure the complexity of a feature. But the goal is to have this feature on the market and to actually make some impact on the market of this feature, right? So if we are going to measure story points or, for example, pull requests with those features, then we are going to just have more pull requests because like you mentioned, developers tend to be creative people and they're great at finding loopholes and optimizing things. So if there is a possibility to game the metric,

17:47 Adrian: Just do it. I want to add actually one thing to this pull request metric because there can be use cases where this is actually useful because if your developers have the tendency to create big pull requests and this makes a code review a pain point for other developers, then maybe if your goal is to reduce the time for a review and make developers happy that they can quicker review code or get the code reviewed,

18:31 Adrian: Then a number of pull requests might impact this, right? Because...

18:36 Marcin: But that's not productivity, right? It's the metric or the mean to create the right behavior, right? This is what you're saying.

18:42 Adrian: This might be a tool, right? Yeah, of course. But again, like going back to what Maciej said, you need to ask yourself why you are measuring this particular thing. And if you are making a statement that creating lots of pull requests will bring me money, then you're probably wrong, right?

19:12 Marcin: Okay, so don't measure the output in individual output. I get that point, right?

19:18 Maciej: I think you can measure because, like you mentioned, this is a tool, right? And everything depends on how you're going to use this tool. So if you are using this metric of number of pull requests or lines per pull request and so on and so on, this is fine if, for example, you as a CTO want to debug issues in your project, so why the pull request takes so long or other things. So this is completely fine to do in this regard. Once you put the money into the mix, so you start using these metrics for, let's say, performance reviews by the HR department or any other reason that actually involves

20:00 Marcin: Some,

20:01 Maciej: Some effect that might be valuable for the developer, then this is where all additional problems will start to happen and they will basically try to optimize it. And this completely makes the debugging the issue part of it impossible.

20:21 Marcin: It's something that we've said two or three times already, and I agree totally on that. There is no single metric for productivity. You can measure and you should measure a lot of things, and it depends on what you want to achieve for a behavior. You can measure pull requests, you can measure the number of lines of code if that's connected with some sort of right behavior change that you're after. And there is another point that you have mentioned, and this is the fact that you should measure lots of things as indicators, not as the one single metric that causes the decision, right?

20:55 Marcin: So it could be information for you, but it depends on how you want to turn it into insight, which will be important. But let me challenge that a little bit from the metrics perspective. So lines of code, no. Story points, no. We have discussed that that's not an ultimate metric. It should be what is the output. So, Adrian, from today, your only goal is to increase the revenue of our product by 25%. Go and tell me how productive you are.

21:25 Adrian: Okay. I would say that certainly it is not only me that will make the money. And I want to emphasize here that it is very hard to measure individual performance and it's easier, and I believe it's better for the company to measure the team's performance when it comes to going after the big goal, right?

22:00 Adrian: Your goal is to make money, to deliver things. And it is a lot easier to measure how many features a team delivered and how resilient they are, like they are not broken on production. And measuring that is, I believe, a lot closer to the thing that you're looking for.

22:27 Marcin: Okay, so not individual metrics, but there is this point of, I can measure the team, and then you'll say that it's not one team, but we have development teams, and then there is product management, there is marketing. So I need to measure the whole department. But I'm trying to just maybe find out, like, I think behind the fact that if I'm measuring too low, on the level of lines of code, it's bad and we need to know why. But if I go too high and it'll be too far detached from myself, I don't know how I'm impacting that.

22:59 Marcin: Therefore it would be the ultimate metric, right? If you can have a metric and have a system that I know that if I'm doing this decision, it'll impact this metric and automatically it'll impact revenue. I would love to have developers who are coming up and saying, we need to do this feature, not this one, because that's higher revenue and we know it and that's our hypothesis, right? But probably we're talking also about on what level I should have this question answer why or what is the impact of this metric? So revenue probably is not the goal, right?

23:32 Marcin: Is there anything between?

23:33 Maciej: So this is a difficult question and we actually make it even harder currently because you ask about the bad ways of measuring it. We talked about lines of code and other things, I think this is quite obvious. And also there is this big other part, like the revenue, like the final impact. And if you are a for-profit company, then yes, this should be the goal for everyone inside your company to actually improve the revenue or rather improve the shareholder happiness because it's not always the revenue, but there's a different topic for a different discussion.

24:08 Maciej: But there are many stages between you actually doing something as a developer and actually having this result as increased revenue. So one model that I particularly like is to categorize this into four phases.

24:28 Maciej: There is the effort, all the things that you are actually doing, the things that depend on you and your actions. Based on those efforts, there is some output. Let's say that you work for many hours, you write code, the output of it would be code written. There is some kind of deliverable that you can share with others, other people can also contribute to it and work based on it.

25:04 Maciej: And this is the output of your work. But many metrics actually focus on those first two steps because it's easier to measure, you can measure the effort, how long someone is working, or the output, so how much code someone has written. But this is not the goal.

25:24 Maciej: We're not writing the code for the sake of writing code, right? It's most often it should be actually the opposite. So what we should measure is why we have written the code in the first place. So the outcome, so after the output we get the outcome. What is the effect of the code that we have written? In most cases, there is some kind of feature or some kind of change in our app or some behavior that this code induces on our users or our customers or any kind of change we want to make in the world.

26:06 Maciej: And this is the outcome of our work. And this is not the end of the story because the outcome itself is not the final step because what we are actually striving for is the impact. So the last step, so the outcomes I hope should result in the impacts that we are hoping for. So this is the holy grail of doing things basically. So we have the effort, output, outcome, impact categories.

26:40 Maciej: And as we go further down the road, it's less and less individual contribution. And more and more depends on basically the whole team or even the whole organization.

26:56 Maciej: And it's harder and harder to measure, right?

27:02 Marcin: But what you're probably trying to say, just correct me if I'm wrong, it's not the fact that the further I go in this graph, the better metric I will have. It's about the fact that it's of course more meaningful metric down the road, but it needs to be a system. I need to, if I'm having one metric, I need to know what kind of metric and outcome it impacts and what kind of metric and impact it is actually connected. So having this as a system to visualize the impact. So it's like, okay, or if you go with the "goal question metric," I need to have a connected system.

27:36 Marcin: So I know this single metric is not for me to judge alone, but I need to judge it as a system. So if this is not working, I'm looking at what is down the road. Is that what you're trying to say?

27:47 Maciej: Yeah, exactly. So you should look at it as a whole picture and basically all of those steps need to have some kind of checks and balances to ensure that it makes sense overall. So spending lots of time writing multiple lines of code and delivering lots of features is not meaningful on its own. Those features actually need to have an impact on the actual market and our users.

28:26 Marcin: Delivering training features doesn't mean anything unless it's actually a return on investment or the impact. But then if we'll go right now and say to the team, from now on we are measuring not the number of features that we deliver, we are measuring the impact. So the revenue itself. How are you measuring day to day that you're doing a good job or not? What's the measure of progress then? Are you able to derive it from this?

28:50 Adrian: So there are some methodologies for that that are maybe like guidelines for us. What is a good outcome that can lay down a good impact on your company? And the first one that comes to mind is Dora, that comes from Google. To put it simply, it measures, for example, how often you deliver things, how often you update your software, what is the time for that software to recover from errors in a production environment, and how many features that you actually deliver to production are working the way you designed them to.

29:54 Adrian: So what I mean here is, are there any bugs in production? And yeah, so these are the main points of this.

30:04 Marcin: So what you're saying is that again, we've moved away from basically finding a good metric, one metric, but we are talking about the system. And what you're saying is that we should rely, and that's probably the correct approach, we need to rely on the good practices that are on the market because they came from the experience. Dora is actually a built framework from that. So taking experience, building on the experience of others, we can have those set of standard metrics to measure individual and team productivity in the modern systems.

30:36 Marcin: They are mostly on DevOps, but basically this is the lead time. So I agree on that. There is again, probably we can mention the space extension of that. So we can connect that and have the system that measures better. The latest McKinsey article was actually pointing to that as we have discussed recently. But that's giving you the bucket or the CEO of metrics. What can I select? Any hints on should I take just everything?

31:06 Marcin: And let's see what will come up and start to measure that because this will still not answer my initial question, how do you know day by day that you're productive or not?

31:21 Marcin: And that's a hard question, right?

31:22 Adrian: And that's a hard question.

31:24 Marcin: Because what I'm after is the fact that we are agreeing that a good metric will probably not be the lines of code, but what you do day to day is actually lines of code. The interesting thing is only to not look at it from the perspective of just this one separate metric.

31:44 Maciej: I actually disagree. Modern developer work is not writing code.

31:52 Marcin: That's what I was after.

31:54 Maciej: Life would be much simpler if all developers do every day is just write code and then it would be much easier to hire new developers and probably by now all developers would be gone because AI could also write code and even if not today, then in the next few years AI would probably be better at writing code than most developers.

32:25 Maciej: But writing code is just a small part of the developer workflow. The actual goal of engineers is to engineer things and to actually make the right things right, to evaluate, to find better solutions to a problem, to think about the problem and try some other solutions that are probably better suited for solving the problem. This is why all developers should be put in a loop with your business goals.

32:59 Maciej: This is why putting the revenue or other high-level impact metrics on their day-to-day minds is actually a good thing because this is how they can change their behavior. Maybe this feature does not make sense. Maybe they can drop it, maybe they can deprioritize other things, maybe they can prioritize other things. So if we empower them to be proactive, if we put them into the loop, then they can actually make the impact, right?

33:31 Maciej: They can make the changes in their daily work that will have some impact on the outcome that you're looking for.

33:46 Marcin: Too often I see, and this is from my experience, even from the past, that we try to hide ourselves. I'm talking about the managers and the C-level, we try to hide behind the metrics and make a decision from the perspective of the dashboard, right? And to just simplify it. And very often we find ourselves in the situation where I want to have this metric to just not go to the team and not ask and just make a decision based on that because it's too much for me. Whereas if I'm even going to the team and I will sit down with them and I will start to ask questions, are we effective or not?

34:22 Marcin: I haven't seen a situation where the team will say, I don't know if I'm effective or not.

34:28 Marcin: I'm probably, you know, all the metrics are saying that it's good. No. Every single team will say, oh my god, do you want to have it alphabetically or by the impact? We can change all this stuff. Which means that there are two things. One is this is not something that you can capture through one metric, but it's obvious if you spend the time with a team and you can figure it out with a team, what kind of improvement you can do, what kind of efficiency you can boost or if we have efficiency or not, deliver experience, metrics and so on.

35:00 Marcin: But the other thing is the fact that this kind of single metric assumes that writing code is the only thing that you do and which is not the case, which is creative work and it's hard to capture the speed of problem-solving. So I have solved seven problems, class B and then 27 story points and two lines of code and produced five bugs. Am I productive? So there is the human factor of that probably. And then there is also the fact that measuring is not only measuring the lines of code. This opens lots of doors, but I don't believe that if you ask anybody to measure my productivity, they will point out those kinds of things, right?

35:41 Marcin: And it's omitted and I guess that's why.

35:45 Maciej: Which part is omitted?

35:47 Marcin: I mean the fact that it's not the single metric, we still try to find the silver bullet for that. I want to measure the developer productivity. Are they productive? If I would come to a team, I can get that information right away. If I bring the teams and discuss how productive you are, what is your system? I can even say, and this is not a hard metric, but I can say which team is more productive or not, it's a feeling also, right? Can I capture that as a metric?

36:15 Maciej: You can sometimes, but it's not easy to do, right? Because this is basically a difference between qualitative and quantitative metrics. So we want to have some kind of number at the end and we need this number because we probably want to sort similar things in order to make some decisions.

36:48 Maciej: The actual product developer productivity is something that you cannot quantize into a single number.

36:56 Maciej: This is something that is obvious for someone working on a day-to-day basis. So if you go and talk to some tech lead or other developers in the project, they will most likely be aware that this guy is much more productive than this guy. This often correlates with seniority and also with other factors.

37:10 Maciej: But people tend to feel this kind of thing. Once you go up and up, it's less visible and you need to be based on some kind of numbers. This is why you're asking to produce those numbers so you can act on them. And it is really hard to quantize. And I think that the role of CTOs or tech leads is to actually educate the people that are asking those questions, to educate them why this is a very difficult topic and try to rephrase the question: why do you need this number?

37:50 Maciej: How are you going to use it? How maybe I can help you differently to solve your problem? Because maybe this is not the actual number you're going for. Maybe this need can be solved with something that I know can be measured, but you do not because you don't have this hands-on experience. So the communication here is vital. And we should measure, we should solve those needs of other groups in our organization, but we should first try to find out why they want to do it and then try to find the best solution.

38:23 Maciej: Maybe there is some way to measure this feeling and quantize it into some kind of form we can use.

38:34 Adrian: Okay. And here I wanted to say that if you want to go back to the question of measuring whether one developer is productive or not, and the thing that Maciej mentioned, you need people to be hands-on with the team because it's a people's perspective that knows this person is the most productive within the team.

39:07 Adrian: That's because it cannot be grasped with just one little single metric because the world is just complex and there are too many factors to put into place. Probably it cannot be generalized for each company and it'll differ across companies.

39:33 Marcin: Okay. And even inside, it'll be different between the teams and between the systems. And that's coming from the complexity of the different technology. It's a research part, not the...

39:43 Adrian: Yeah, and probably the goals that you set. Right? That this is also very important.

39:48 Marcin: That's again the thing. But isn't that then pushing us to understanding that should I search for a metric or should I assume that every case is different and I need to make sure that I'm improving? So this concept of either the system is improving or declining, there is no status quo. Always as an engineering leader, I should make sure that I'm improving with the team and continuously doing that rather than trying to find the one metric that will tell me we are on track or not.

40:18 Adrian: I read about this that some team leaders or managers used to note what accomplishments they've achieved within the team. And probably if you have that you can also find out which persons stood by this. And this is something that can give you a good direction to see what impact that person had.

41:03 Adrian: But I also wanted to mention one story that sometimes you need to go very deep into the team to know if this person is productive or not because, and this is a story about the worst programmer I know. There was this guy that produced no story points each sprint and there was a decision to let that person go from the firm.

41:40 Adrian: But if you went down into the details, it was that this person was actually helping all other developers on the team and making sure that the team is performant. And that guy on his own made the codebase a lot more resilient because the solutions were well-written and covered by tests.

42:14 Adrian: And basically the quality of the features delivered was very high. And you could sum it up with you should prefer quality over quantity, right?

42:30 Marcin: In most of the cases, yeah. In most of the cases. I could challenge that. But that actually opens another aspect which is also pretty interesting. And this is the fact that one thing is that writing code is not always what you want to measure and what you want to focus on. Helping the team and removing obstacles or taking care of the PRs. I would not say that this is a go-to solution that you should always have a person who is not writing code but watching out for everybody else, there is some balance there, but it's a great anecdote and a point to take from that. But there is one more aspect there.

43:03 Marcin: And we tend to, through simplification, always fall into the trap. The productivity of a person, even if we have one metric, will be the productivity of the day.

43:13 Marcin: It could be different today, it could be different tomorrow, not because I'm better tomorrow, but for example, tomorrow I have fewer obstacles in the way or it was better weather. So let's just remember it's a human factor, right? It's not that I'm performing the same every time, it's not execution of a procedure. So there are multiple factors we need to consider. Nobody wants to just go to work and perform as poorly as possible.

43:45 Marcin: At least nobody that I know. But that's something to take into consideration, right? You cannot take it without the context itself. Therefore it's like if I would ask you what is your example of the most productive day, you would not mention 27 story points, right? You would probably paint some picture of how that looks.

44:11 Adrian: Like a single day.

44:12 Marcin: Yeah. Let's go into that, right? Like an example of your most productive day.

44:16 Adrian: I think I see where you're going. So my perfect productive day is a day where I'm not interrupted, and I could actually even close with that thought, but there are lots of things, right? And it's very rare that I don't have any meetings during that day. But you can optimize for this non-interruptions and a deep focus state of mind, right?

44:57 Adrian: So this especially works best when you need to produce the code, right? But again, this is just a part of your work. Another thing is that when goals are straightforward, when I know ahead the goals and they are pretty straightforward, I don't have any more questions, then I'm also very productive because I don't have to wait for someone else to answer my questions, for example.

45:38 Adrian: Right. When I'm also not blocked by other persons. And this is also again the time spent, for example, waiting for code review that you're waiting on. So lots of factors that...

45:53 Marcin: I don't want to land on the statement that you cannot measure that and just measure the number of meetings, and if there are more than one or two interruptions, then you don't have productivity. But definitely the thing that we need to pull from that is that if you have a metric, if you want to measure and you measure the developers, make sure that you take the context also. If the environment is productive, if they have interruptions, if they have what they need, and only after that take into consideration what is the productivity.

46:26 Marcin: And then putting that into probably trying to move it back to something as a takeaway or something more tangible. I can say that there is no one single metric. Build a system, measure it correctly, measure the environment. But let me state a problem from the C-level or the startup owner. I want to start a project. I have two offers from two different companies and maybe I have worked with them, maybe I have examples of them.

46:57 Marcin: Let's assume that's how they worked. I need to compare them because the price is almost the same, but I need to say which one is better. Is there anything that you would rely on to compare them?

47:11 Maciej: Right? But there are good metrics. We spend lots of time talking about bad metrics, but there actually are good metrics. I would like to follow up on the topic of the perfect day, right? If you ask an engineer what is your perfect productive day, they'll describe that you need to go into the zone and this makes you productive. And this is the most efficient, but if you look from the perspective of a product owner or product manager, it's actually the days where the work is moving fastest forward towards your goals, right?

47:54 Marcin: Towards the plan.

47:56 Maciej: So this is why if you can, you should rather measure team productivity rather than individual productivity first. It's easier to measure. Then you balance out all of those things that we discussed a moment ago. So you can have a bad day, someone else can have a bad day. Also, it balances out the people that are actually very useful for the product. The worst programmer case that you mentioned, right? This person actually helps the product move forward by a lot.

48:28 Maciej: Even though it's not contributing code directly, it doesn't mean that taking this kind of person off the project makes the velocity of the whole team go down, not up, right? So this person is obviously beneficial for the team, even if it's not contributing directly. This is why we should rather, if we can, measure the team's performance because

48:55 Marcin: Okay, team performance and any examples? You have mentioned velocity. Again, it's the...

48:59 Maciej: The complexity points depend on the case. This is why I also like Dora metrics. The DevOps research and assessment metrics, because they are quite easy and applicable to any project on any team, in any industry because they're quite independent of the specific things for different areas.

49:25 Marcin: But that means that you're relying on the lead time of delivering functionality.

49:30 Maciej: Yes. And this is a good start. Basically, this is a good base to start on. This should not be your final metrics because they usually are not enough on their own. But this gives you a good start to, for example, compare between teams. So if you have two sub-product teams, you can use Dora to easily compare them and most often you'll have a pretty good measurement between those teams. So if someone gets a better lead time, their deployment frequency is much higher, they get less failure rate then,

50:10 Marcin: But what about the cases where you have the,

50:11 Maciej: Obviously, they have to be better even if they're doing a worse job,

50:16 Marcin: Wait a minute, are they better or have a better environment? I mean, I can have a team that is working with the payment component and the other team which is working on registration, a simple component with faster lead times with the functionality. I can understand that the metrics will tell me that this is a better team. I would not jump to that kind of conclusion. It needs to be connected with the complexity of work, right?

50:44 Maciej: Not exactly. One metric won't tell you which team is better. It'll tell you which team is more efficient and you want the teams to be efficient

50:55 Marcin: In the current conditions.

50:55 Maciej: Yes. Of course, everything needs to be looked into in context. So if there is some, maybe the less frequent deployment times for the payments team are okay because the platform needs to be more stable, and so on. But still comparing more frequent deployments to less frequent, the more frequent is always better. This is something that you can try to optimize. So we can compare, those metrics are agnostic enough.

51:26 Maciej: So we can compare even different kinds of product teams. And the Dora organization standing behind it also has some kind of report so we can compare

51:41 Marcin: On to the benchmarks on the market.

51:43 Maciej: Yes. Benchmarks in your industry, so we can basically see whether you are ahead of the pack or lagging behind. So this is a good start. If you are capable enough to create space metrics, so contrary to Dora, space doesn't provide you out-of-the-box metrics that you can use. This is more like a thought experiment to force you into actually creating those metrics. They give some examples. Some of them are not that great because I believe there is also an example of lines of code that we've already debunked that this is not a good metric as a

52:19 Marcin: Standalone, no.

52:20 Maciej: Yes. So if you are capable enough of creating more specific metrics, then great, you should do it. But this is not an easy feat. We actually built a tool sometime ago just for this reason to actually help create space metrics for the teams because it's not an easy task. We use AI for this. This was a hackathon project. Later on, a few months later, we got CTO Compass, which is our tool, which basically starts from the idea.

52:55 Maciej: So right now this tool is not only producing you space metrics, you can still ask it to do so, but it's more like a general advisor. But it starts from exactly this need. So the initial need was, it's hard to create those metrics, and I need someone to basically discuss with to create those metrics. I can hire some consultancy, or you can use this self-service tool.

53:22 Marcin: Yes. It's all the things that we have mentioned, right? You need to start from the question, from the goal, and then build a system around that. It's not only to start to measure story points, therefore these kinds of AI consultancy solutions or the example of CTO Compass can help you there. And this is the missing piece of the puzzle that we're discussing, would be probably the answer to how to approach the measuring of productivity. Just one thing to build on top of that, because I think it's actually something that is not mentioned in Dora and space, at least not directly, which is obvious.

53:56 Marcin: And this is the fact that if you're measuring a metric and comparing it to a benchmark, this is a point in time with my teams, I always prefer to see the trends and see if we're improving or not, rather than the point in time because it could be different due to different factors. Sometimes I would prefer a team that is improving their speed of development dramatically and they are not the best in the benchmark than a team that is delivering on a stable or declining and the lead time is going down.

54:27 Marcin: In the benchmark, they'll be better, but the trend that they are on, that's the difference. So look at the trends also. That would be my point.

54:37 Maciej: Yes, it's even more important because the best benchmark is actually your team from the past. So to make sure that you are actually moving in the right direction. And Dora is great for this because it's so agnostic that you cannot go wrong if you improve those metrics. So if you improve those metrics, even if you're still behind based on the overall benchmarks for the industry, you are still doing a good job. As a CTO, I know the changes that I introduced into the teams, the workflows are actually beneficial because you're moving in the right direction. If the metrics go down, then I know that something bad is happening and I need to react accordingly.

55:23 Adrian: Yes. And about that, like the debugging of the teams. So if something is going wrong and you just notice this trend that everything is going down with your metrics, then there is a suggestion to use surveys and survey your developers, your people at the company, like what is going on?

55:54 Adrian: And here is a tricky thing, you need to carefully build those surveys. And there's no one gold survey to gather all the data that you need. But you can start with something like how are you feeling with the developer experience? You can focus on that.

56:19 Marcin: That's amazing. I'm smiling because actually we are experimenting in one team about this concept, which is probably not connected to the question of how to measure productivity. But during the retrospective, the team suggested that we are closing lots of action points on the retrospective, which seems productive, right? We are going somewhere, but the perception is, okay, we are closing 27 action items, but are we actually improving? Are we feeling that it's okay? And there was a great request to can we actually start to measure developer experience and can we start to measure even the mood? So you see the trend.

56:53 Adrian: Yes, exactly.

56:56 Marcin: Am I improving in that?

56:57 Adrian: And this is also that we are people, right? And people tend to have their mood impact their productivity, right? And they can, of course, burn out and then they will not be productive at all. This is the worst case, of course. But also things that annoy them, like they need to wait for the tests to finish for a long time. And this disrupts their work and also makes them angry, right?

57:32 Adrian: Because not everybody is happy that they have free time because the tests are running, right? So I can watch some...

57:42 Marcin: Unproductive work then.

57:45 Adrian: But in reality, this is just very disturbing.

57:52 Marcin: Yes. And measuring the potential and how much you can use that, right? Because for sure you have the developers who are better, more senior and then more junior ones. But even if you have an ideal situation, you have two experts and two same level experts. One can be more productive today and as we have discussed, the other will be slow only because of the distraction and maybe the team concept, even what kind of team I'm working in. So we need to look at it from a holistic perspective.

58:23 Maciej: And there's this other thing that actually personal individual performance is not correlated at all with team performance. So you can have a team of experts and stars that are great on their own, but the actual team performance could be really bad. And also the other way around, you can have a not that experienced team, but they're actually delivering great value. This is why if your final goal is to actually deliver some products, and you usually need to have some kind of team to assemble a team to make it, you should rather measure the team's productivity rather than individual productivity.

59:01 Maciej: Even look at sports, even the best stars combined in a single team would not guarantee you that you have a great team. So you can have a striker from Barcelona, a goalkeeper from Juventus, put them in a single national team.

59:19 Marcin: Now we're talking about Euro.

59:22 Maciej: Yes. And it won't guarantee you to produce the best team. So you cannot correlate one from another. This is why you should also, depending on the question, why you are measuring it, measure it completely differently. So if you're measuring your team's performance, you're comparing vendors, you're making sure whether you need to continue this business line, you should rather look at team performance, their outcome, their impact. If you have a different kind of question, if you have a question of who you should promote or performance review kind of stuff, then you should rather look at individual performance.

60:00 Maciej: And there are different ways, different metrics.

60:03 Marcin: And again, it'll be not one metric, it'll be the set of things and the indicators. So we need to remember quantitative and qualitative. We need to remember human factors and the context.

60:13 Maciej: And there's also the Goodhart's law, which states that once a measure becomes a target, it stops being a good measure. So once, for example, performance reviews, if you explicitly make promotions based on some of those metrics, those metrics will be gamed and

60:36 Marcin: Yeah,

60:36 Maciej: We've discussed they won't be a good metric anymore.

60:39 Marcin: It could be intentional, right? If I want to have the right behavior, it's okay with that, but let's remember.

60:43 Maciej: But it's really hard to do it. You need to have checks and balances to ensure that it's not possible to game the metrics for personal gains because people tend to do it. A good practice is to, once you introduce some kind of metrics, is to try to red team those metrics. So basically try to hack them, try to go into the shoes of the people you are trying to measure and see whether they can game the system because they will do it.

61:15 Maciej: This is partly guaranteed.

61:17 Marcin: I like that. I like also the concept that if you have a metric, even one single metric, like time to PR or time in the PR review, or lead time of the feature, if you see that this is declining, your hypothesis is that it's actually going bad and you need to react. If you analyze it, you will look at the whole value stream of the whole team and you analyze it, how that fits into the whole picture. Two hours in the code review, is it bad if we are delivering 27 times a day and have continuous integration and so on, or if we're releasing once a month and this is basically the situation with the banking cases for example.

61:57 Marcin: So having that put in the context will always make a difference.

62:01 Adrian: Yes. It'll make the difference also in that sense that if you are focusing on the deployment rate, then you'll probably have a bigger bill because your continuous integration systems will run much more.

62:18 Marcin: Telling my behavior.

62:20 Maciej: This is why there always needs to be some counter metric to ensure that this does not go out of hand.

62:25 Marcin: Exactly. And you have indicators and you have the leading metrics and then the indicators.

62:30 Maciej: Even if you put revenue as the only goal, it's also not a good metric because people would tend to optimize the process for short-term returns. And at the same time, for example, customer satisfaction may go down and in the long run, the company can even go down just because people want to have those bonuses based on the increased revenue. So this is why there always needs to be some countermeasure, some checks and balances to make sure that we are not hurting some other important metrics because we are aiming for a different one.

63:10 Adrian: Yes. Sometimes it can hurt. Like if you are targeting individual contribution to gain money, then you might lose some big fishes. Because it usually is the team effort to catch the big ones. And a single person cannot handle it solely.

63:40 Marcin: Yes. You need to look at it as a team. Even the best individuals, as mentioned, one hero will not change all the time. The team is the game here. You need to have a team contribution and I would rather have the best team, which is on average in individuals, than one hero and a non-performing team due to that fact.

64:08 Adrian: You certainly don't want everybody for themselves, right?

64:12 Marcin: Exactly. And therefore I don't want to measure them like that. So this would be the thing.

64:19 Marcin: I wanted to open more points here because we have started to talk about the area of personal development and how to create behaviors that the team will gain performance. We have opened the topics of financials and maybe the topic of how to select the right team with a cost-efficient approach. We can talk about that in the next episode as this episode is extensive enough. So just to wrap it up right now, we've talked a lot about the metrics, how to measure, what to measure, and how to look at it from your perspective. Three takeaways from the C executives.

65:02 Marcin: What would be your points from the discussion today? Who to start?

65:08 Maciej: I think I can start. So basically the first thing is to understand why you want to measure the metrics. The need for measuring developer productivity can come from many different people, from many different roles, many different departments, and try to first understand why they actually want to have this metric in the first place. In the case of the CFO, they are curious why the engineering department costs so much.

65:45 Maciej: Maybe a different approach would be better than, for example, the HR department when they need to give some promotions or do a performance review. If you are a CTO and want to debug problems in your teams because you are aware that the performance is not in a place where it could be, you want to debug it, then you'll probably use different measurements.

66:12 Maciej: If you're a developer, you want to make a self-assessment. This is also a completely different reason. So first try to understand why there is this need for measuring and then try to ask some follow-up questions. Why do you really need to measure it? And then ask once again why you really need to measure it. Yes. So this is the only way to actually be able to put the good metrics down the road. Once you get the actual reason, then you can try to find the metrics that you can try to calculate.

66:47 Maciej: The main framework that you need to have in mind is the effort to output to outcome to impact. And on the first two steps, basically the effort and output, you can measure it just for debugging purposes.

67:02 Maciej: So you should not make those measurements goals, but rather metrics that you use to ensure that your project is in good health. Once you start rewarding people based on those metrics, you will be gamed and you should not try to do it. You should put goals on the outcomes and impact where if people try to game the system, this is actually a good thing because you wanted them to actually do it.

67:32 Maciej: You want them to be proactive to play to the same goal. You should put your actual measurable goals that you want to reward people for over there. It is important to always red team your metrics, try to hack the system yourself, make sure that there are checks and balances in place so that all of the metrics are healthy and you cannot by optimizing one metric hurt different metrics.

68:06 Maciej: Right.

68:07 Adrian: Yes. And from my perspective as a team leader or a tech lead, I would focus on the teams and not individual performances when it comes to measuring the overall impact. Of course, you can still use the measures of individuals to debug what is going wrong within the team, but this is only on that level and this is because first of all, measuring team performance is a lot easier and will give you better results.

68:47 Marcin: Okay. Amazing insights. From my side, I can say probably three things that resembled the most in my head. One is the measurement of productivity is not the work exactly the lines of code that a developer is doing, but it's actually far more the things, the problem-solving team effort, trying to find out how and what to build, not only building it. It's not bricklaying and building a house. The team effort also is important and from this context, the second thing is think about not template solutions.

69:21 Marcin: So there is no silver bullet as we are discussing. Probably there is a metric that is useful, maybe a set of metrics, but you need to find out what works for you. And then using goal question metric, using different approaches to find out what kind of set is good for you.

69:36 Marcin: And asking those questions is really important. And the third thing is actually combined. So one is go and see and a metric will not always show you the whole picture. So it's really worth going down to the teams and actually spending time with them and figuring out if they are productive or not. If this is your goal and see for trends in the whole picture. Am I improving? Am I declining? And this is what is important. And probably many more topics that I would summarize here and write a pretty long article from if we are just trying to summarize the points.

70:13 Marcin: But that was super interesting. Hopefully we'll continue on this one. Thank you very much for joining us and listening. Hopefully that was insightful for you and you'll take something from this discussion. You can find us on YouTube, Apple Platform, and Spotify. Subscribe and send us comments or feedback if you find it fitting. Looking forward to seeing you in the next episode. Thank you. Take care.