Technical debt isn’t technical
Posted: December 5, 2015 Filed under: Programming 11 CommentsTL;DR
Technical debt is not primarily caused by clumsy programming, it is a third-order effect of poor communication. Technical debt is a symptom of an underlying lack of appropriate abstractions, which in turn stems from insufficient modelling of the problem domain. This means that necessary communication has not taken place: discussions and decisions to resolve ambiguity and make informed trade-offs have been swept under the rug. Technical debt is the reification of this lack of resolution in code.
The technical debt meme
For a while now, I’ve been wanting to write about technical debt. As we all know, technical debt is a very successful meme in software development – it needs no introduction as a concept. Like any good virus, it has self-replicated and spread throughout the software development world, even reaching into the minds of project leaders and stakeholders. This is good, since the notion of technical debt brings attention to the fact that the internal quality of software matters – that there are aspects of software that are invisible to anyone but the programmers, but still have very visible effects – in the form of prolonged quality problems, missed deadlines, development grinding to a halt and so forth. For this reason, we should tip our hats to Ward Cunningham for coming up with the term. It gives us terminology that allows us to communicate better with non-technical stakeholders in software projects.
Why technical debt is a misnomer
That’s not what I want to write about however. What I want to say is that technical debt is also a deeply problematic notion, because it speaks little of the causes of technical debt, or how to fix them.
The usual story is that technical debt stems from project deadlines. If the code is inadequate, sloppy or otherwise “bad”, it has probably been written in a hurry because the project leader said so. This indicates that time is the cause of our problems, and also conveniently places the responsibility of the mess on someone else than the developers.
This is certainly true in some cases; we have all written code like that, and for those exact reasons. I just don’t think it’s the whole story, or even a major part of it. It seems to me entirely inadequate to explain the majority of technical debt that I’ve seen on software projects. The so-called technical problems go much deeper than mere sloppiness of implementation, and reveal fundamental problems in the process of understanding of the business domain and how that understanding is captured and represented in software. In particular, it is very common to see weak abstractions that fail to represent the richness of the domain. The code tends to be overgrown with conditional and flags, which indicates a weak model that has handled evolution and change very poorly – by ad hoc spouting of extra branches and the booleans needed to navigate them as appropriate for different use cases. Complexity grows like ever new epicycles on the inadequate model – easily recognizable as things in your code that cannot be given meaningful names because they have no meaningful counterpart in the problem domain. The end result is a horrific steampunk contraption of accidental complexity.
This makes the code extraordinarily difficult to reason about. Hence, it would seem that the so-called technical debt really stems from modelling debt; the code lacks the higher-level concepts of a rich domain model that would make it possible to express the use cases more directly.
The currency of technical debt is knowledge. — @sarahmei
In DDD terms, modelling debt indicates that insufficient knowledge crunching has taken place. Knowledge crunching involves learning about the problem domain and capturing that knowledge in a suitable domain model. This is a communication-driven process that involves identifying and resolving ambiguity in the problem domain, and expressing the domain as clearly as possible. Most of all, it is a chaotic and messy process that involves people and discussion. Insufficient knowledge crunching in turn points towards the ultimate cause of technical debt: poor communication.
Communication is the principal portion of the “technical debt.” Messy code is just the ever-increasing interest. — @nycplayer
Why technical debt is misrepresented
So if technical debt isn’t really technical – or at least not ultimately caused by technical issues – why do we keep referring to it as technical debt? Unfortunately, it seems to me that developers have a tendency to look for technical solutions to soft problems.
Technical tasks are alluring because, unlike modelling and communication, they have no psychological dimensions, and tend not to lead to conflict. I don’t want to add to the stereotype of the programmer as particularly socially inept; suffice it to say that most people will prefer to avoid conflict if possible. Technical work is a series of puzzles to be solved. Modelling work uncovers human issues, differences of opinion, different focus, different hopes for the application, even personal conflicts. Figuring out what the application should do exposes all of these issues, and it is painful! This is why everyone is quoting Conway’s Law these days.
And so, even as widespread as the meme technical debt is, it seems to be poorly understood, even by developers – perhaps even particularly by developers! Indeed, the Wikipedia page for technical debt – no doubt authored by developers – currently lists 11 causes of technical debt, but lack of understanding of the problem domain is not one of them! “Lack of knowledge” sounds promising, until you read the explanation: “when the developer simply doesn’t know how to write elegant code”(!)
Again we see the focus on the technical aspects, as if technical debt were caused by clumsy, unskilled programmers with nagging, incompetent project leaders – and hence as if it were fixable by some virtuous programmer – a master craftsman, no less! – using generic, context-free principles like SOLID, dependency injection and patterns. It is not! Code hygiene is certainly a virtue, but it is no substitute for modelling, just like frantically washing your hands is not sufficient for successful surgery. Getting lost in code hygiene discussions is like arguing about the optimal kinds of soap and water temperature while the patient is dying on the operating table.
And yet it is indirectly true: a developer who doesn’t know the importance of understanding the problem domain, of proper modelling, will certainly fail to write elegant code. Elegance of the implementation can only stem from an elegant model that reflects a deep understanding of the problem addressed.
Down payment
This has deep ramifications, in particular in how we address technical debt. Refactoring is another successful meme in software development, and we often use it to describe the process of making down payments on technical debt. But if technical debt isn’t just clumsy code, if instead it is clumsy code caused by unresolved ambiguity in the problem domain, then it is poorly addressed by rearranging code. We need to start in the other end, with a better understanding of the problem we are trying to solve, and with modelling concepts permeating the code instead of branches and booleans. This is what Eric Evans calls “refactoring towards deeper insight”. Unless we have a model to drive our efforts, there is no reason to believe that we will be able to do much better than before. Refactoring without an improved domain model is just hubris.
A rewrite will end up with the same problems as the original unless you close the understanding gap. — @sarahmei
To conclude
That’s what I wanted to say about technical debt. It’s not very technical at all. It’s about code that gets bad because humans fail to communicate when trying to solve problems in some business domain using software. It usually is.
ample code 
“Unfortunately, it seems to me that developers have a tendency to look for technical solutions to soft problems.”
I agree, and I think it’s to large extent the issue of how we understand programming and how we learn it. We learn all about technology, etc. but unless somebody has a personal interest or wants to do something else (e.g. become a team lead, manager, UX specialist, …), there’s no clear way how to work on all those “other” (i.e. non-technical) skills. Also they don’t seem to be a good investment, because there’re not required for getting a better job or promotion. It’s a shame because those skills can (and should) be taught and learned.
A great write-up of a problem that many DDD practitioners see, and one that I will happily refer to.
I think the claim that technical debt isn’t really technical is quite insightful. However, if it’s because of insufficient domain understanding or abstraction as you say, it would still be a technical problem. Where your original insight becomes both more accurate and more useful, IMO, is in observing that technical debt is usually a process problem. In my experience, it’s most often the result of too many features, too little time, developers not being held to an appropriate standard, reviewers/architects hamstrung in their efforts to block sub-par patches, or some combination of these factors. All of the “knowledge crunching” and so on in the world won’t fix those kinds of problems.
Jeff:
Thanks for your comment!
I agree that there are process problems that go beyond mere lack of modelling effort. In my opinion, modelling work is particularly likely to suffer when the process is broken. It is true that knowledge crunching will not solve those problems – you need to fix the process so that there is room to do any knowledge crunching at all! It may well be that these process problems are even more severe than the modelling problems I’m talking about. I’d argue that inadequate modelling is *also* a process problem though, since you need to establish understanding that explicit modelling is necessary in order to do quality work (at least in a non-trivial domain, which is basically all of them, at least in my experience).
I still think it’s useful to distinguish between modelling debt and technical debt though, because I think we should stive to make causes and consequences explicit. Some technical debt *is* purely technical and can be fixed with better understanding of languages and tools for instance. My claim is that quite a lot of it is caused by inadequate modelling (which may in turn be due to a toxic process, as you say) and as such is not fixable by technical measures alone – i.e. reimplementing, applying patterns, doing IoC, going asynchronous or switching to micro services will not help at all. That’s where the knowledge crunching comes into play.
Weronika:
Interesting perspective! It is probably the case that a lot of us became programmers because we have certain skills that are well tuned to solving technical tasks (strong analytical skills, for instance). It is not obvious that we are equally good at navigating soft issues, differences of opinion, negotiation and so forth – all of which are very important when collaborating with others. (I don’t want to say we’re particularly bad at it, just not particularly good.) As you say, some developers might not even be motivated to cultivate those skills. But I think it’s absolutely necessary in order to become a good developer of software, not just a great implementer. And it should be possible to leverage our analytical skills to do good modelling work – after all it’s problem solving just like programming is.
Trond:
Thanks, that’s much appreciated!
Great article, hadn’t considered it this way before but, along with DDD, BDD looks more and more like a way of heading of the cause of technical debt you talk about here. For me the key pillars of BDD are communication, modelling and understanding.
From my experience BDD also seems to talk more to including the other stakeholders outside developers, I don’t think I’ve ever heard a non-developer talk about DDD. This has piqued my interest of having DDD in the toolbox though, so I’ll be reading about DDD over Christmas!
Neil:
Thanks, and thanks for commenting! I’d say the communication, modelling and understanding are the core concepts in DDD as well – at least that’s my focus. I’ve spoken to non-developers, in particular UX people, who also see and feel the need for a shared model and language. They too experience how software projects suffer without it. It’s very problematic when non-developers and developers start to view the software in different, non-compatible ways. Non-developers start thinking that the developers are difficult because they protest against changes or features at seemingly arbitrary times. Developers start thinking that they are the only ones that truly understand the software and that they are the ones who must pay the cost of change. And so on and so forth. It’s all very unhealthy. At my company we’re trying to use the domain model and a shared (“ubiquitous”) language as tools to prevent this from happening.
Good article. Far too much code is written that shouldn’t be simply because it’s usually more attractive to developers to find technical solutions to problems.
However, often, especially in large, complex systems, technical debt really is experienced technically, as models that are imposed from the outside, and which the development team has no authority or power to change: organization-permeating data and code that it may not be economic to throw away, and that only a determined effort led from the top of the organization will be able to change.
Creativity can still be applied to minimize the horror within particular projects, but one will still have to deal with it at the interfaces. In this case, it’s not a question of being unwilling to communicate, but simply of being realistic: attacking the root cause of the problem is simply beyond the authority and powers of the team. Going further tends to be career-limiting, or at least job-limiting; the sort of risk that many people won’t (or can’t reasonably, considering other responsibilities) take.
The same often applies at a higher level, in the form of technical standards that don’t adequately model the domain.
sc3d: My apologies for not responding to this comment earlier. I think you’re spot on. There are many forces in software development that influence the modelling effort, as well as the implementation effort. Organizational issues are definitely one of them.
As you may know, the so-called “strategic” DDD patterns discuss issues like these, and suggest various strategies that might be applicable depending on context. [See http://www.infoq.com/presentations/strategic-design-evans for a talk by Eric Evans on this subject.] For me at least, the strategic part of DDD is the most important one.
I think it’s always useful to try to draw a “context map” to make explicit the interaction and relationships between models (and the people/parts of the organization that owns them), and the strategies suitable for mediating between models. If you’re lucky, you can create an “Anti-Corruption Layer” which handles the external influence at the border of your model, allowing your own model to remain optimized for your needs. But if you face great external pressure and have little leverage, you may be forced to use more of a “Conformist” strategy, where you simply accept that your model will use suboptimal concepts imposed from the outside. Of course, if you have multiple external influences like that, your job will be much harder – and it will definitely have very real impact on the technical level as well. Modelling problems are always reflected in the code.
Interesting perspective. However, it does not seem to be completely true that the reason behind using the “technical” term, is because of developer’s tendency to look for technical solutions to soft problems. According to me, it’s called “technical” because most developers (technical guys), while having to develop a feature, would know about certain lack of information and thus, may not be able to properly model the system.
We can take a simplest example of a billing system, that needs to be developed from scratch just in day’s time (Business pressure). During the negotiations, the guys at business side (mostly non technical) might settle down on something as simple as just being able to calculate the total cost. They certainly know that there is lack of information but may not have a clue of how it might result in issues that might affect the development cycle in future. In this case, only the technical folks (developers) would know while implementing that the system, that their code may not easily support additional features, like “Discounts” and thus, would record it as something they (technical) need to address in future (debt).
Yes, I agree that the cause of technical debt may not be entirely technical. But the reason for calling it “technical debt”, in my perspective, is because, the technical folks would definitely know the issues in their currently selected design and would note it from their end for future reference.