Data Management’s Technical Debt Crisis
One of my favorite pet phrases is, “Pay me now, or pay me later; either way, you’re gonna pay me.” This is my way of pointing out that the decision being made at that moment could have long-term effects on the product being developed. I want people to see that their sub-optimal decisions can come back to bite them in the buttocks, even if they’ve made them for all the right reasons. What I’m referring to here is what is popularly called “Technical Debt”. This is mostly seen as an application development thing, but only because it’s really just the app dev guys that seem to talk about it much.
Technical Debt
So, what precisely is technical debt? I’ll let Mr. Martin Fowler explain it to you. (@martinfowler on Twitter)
Technical Debt is a wonderful metaphor developed by Ward Cunningham to help us think about this problem. In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.
The metaphor also explains why it may be sensible to do the quick and dirty approach. Just as a business incurs some debt to take advantage of a market opportunity developers may incur technical debt to hit an important deadline. The all too common problem is that development organizations let their debt get out of control and spend most of their future development effort paying crippling interest payments.
How Does This Apply to Data Management?
The debt metaphor is used to simplify the notion that we create liabilities for ourselves when we make sub-optimal decisions. We know and understand that there is a right way to do something, but we choose, for whatever reason, to take a path that is something less than that right way. For example, I can choose to do a half-assed job on the lawn on Saturday morning in order to catch the start of the game. With this, I’ve made an intentional decision to create a debt that I know will have to be paid off at some point. As long as I’ve thought it through though, and feel that its being done for a good reason, in theory, its permissible to accrue this debt. In this example, not only does my lawn look like crap within a few short days, but I’ve also created more work for myself the next time I have to mow the lawn. Additionally, the interest payments that I will be making on this decision will likely be made daily and come in the form of the abuse I will catch from my wife for how the lawn looks.
Given that I’ve shown that this is a general, all-purpose metaphor, how can it not apply to data management?
Where is There Technical Debt in Data?
So, where do we find technical debt in the data management world? Just listen to the conversations around you. From cubicle huddles and coffee machine conversations to the morning scrums, people are making decisions about how to move forward with the tasks on the project. Some of those decisions will be about cutting a corner here or there.
Specifically, data integration is an excellent place to find technical debt, and its interest payment usually bites you with poor performance. Same thing goes for data modeling and physical database design. Can I have technical debt in MDM? Yup. Data services? You bet. Wherever you can cut corners, you will find technical debt.
Ironically, data quality is one of the first things that comes to mind when I start talking about this with my peers, but I don’t see this as technical debt. Rather, poor data quality is itself an interest payment against one or more debts created elsewhere. Things such as the app dev choosing not to implement validation rules within a data input application, or the data entry folks choosing to enter junk data into a field to get around a poorly designed validation rule. (Honestly, data quality issues should probably be more appropriately described as compound interest given that the further downstream the data gets, the worse the problems become.)
Nervous Yet?
Does any of this make you nervous? It should. When you consider that a good-sized majority of our IT budgets are spent on operations and maintenance (status quo type stuff), and not on creating anything new, you should be very nervous. Having lopsided budgets like this prevents your team from being able to properly support the needs of the business in a timely fashion. And of course, if their needs aren’t being met by your team, they’ll likely go somewhere else to try to get it done, or try to do it themselves.
If you don’t start actively managing your technical debt situation, you will eventually be forced into technical bankruptcy. I assume at this point that you can see how that’s gonna play out.
Pay me now, or pay me later; either way, you’re gonna pay me.
The technical debt topic typically creates a good bit of chatter. What do you have to say about it? Leave a comment.
—————————————————————
If you found value in this article, please consider sharing it with your peers using one or more of the social networking buttons below. To receive updates when new articles are posted on this site, please consider subscribing to the RSS feed or email using the appropriate buttons in the top-right corner of this page. As always, I appreciate your continued support and encouragement.
