Software Development Fragmentation

About 15 years ago, I got my first software engineering job, and the way things were done back then—was different.

I was working in tourism/hospitality, and we were building a B2B product targeted at hotel owners and travel agencies. The product was big and had vast feature set. But in addition to the main functionality of the product, we were also building an accounting and BI (business intelligence, a fancy word for analytics), submodules. We were also building tools that would export our accounting data to standardized industry tools like SAP, but nevertheless this practice of building in-house accounting or analytics is practically non-existent in modern software development.

If you Google for “how to code an accounting system”, you will land on a bunch of Reddit or HackerNews posts telling you “Don’t”. I’m not sure whether this advice stems from the incompetence of the current generation of developers, who are afraid to build anything more complex than saving a few JSONs in a database; or it’s a genuine concern to avoid wasting time on trivial tasks that can be outsourced to a third party SaaS. Sure, accounting systems are complex, and there are best practices to follow like “don’t use floats for money” and “please don’t use floats for money”, but nevertheless they are not impossible to build.

Despite this fact, today, the consensus seems to be to outsource any mundane task like accounting or analytics to a third party service that is designed for these particular tasks. Hence, such approach creates a fragmentation in software. Your product is no longer standalone, and in order to be used properly it needs to be tied to these third party SaaS products. This, often, leads to absurd scenarios where you collect analytics in tool A, but this tool is so general—in order to appeal to the masses it has to be—that you end up building complex data pipelines to move data from said tool, to yet another tool designed for visualization. In reality, you have all the data you need, you just need to augment it with some additional analytics collection and build a proper visualization that makes sense for you.

You might say that it’s a waste of developers’ time and company money to develop yet another analytics tool where there are a dozen existing solutions that cost a cup of coffee and a croissant a month. But this reminds me of the fallacies described in “Mythical Man Month”. Sure, you save time on development, but you did not account the time needed for training and mastering said tools. And let’s be honest: there is no such thing as due-diligence. You can’t possibly determine whether the tool is right for you by reading their landing page or talking with their sales rep. The only way to effectively evaluate the tool is by using it. And by the time you realize that it’s not exactly what you need, you are already knees deep in it, and the sunk cost fallacy starts to kick in, so you overengineer a solution to build a data pipeline to a data warehouse as if you are a warehouseman working in a sorting center at Amazon. But hey, at least you didn’t have to reinvent the wheel™ and waste company money and resources. Well done!

All this could have been avoided if you built a custom tool tailored for your needs. In the end, you have all the data, why spread it across multiple third party tools in order to later build a data pipeline that unifies it all together? But no, we need to fragment. We must not need to reinvent the wheel. And yes, sometimes it makes total sense. But I don’t know what is it with the modern software development practices, but we seem to rely too much on third party solutions, as if the role of a software engineer was reduced down to a procurement employee who stitches together a solution based on dozen third party services.

But you need two for a tango, and as the inefficiencies grow on the development side, so do SaaS companies tend to meet the demand (or maybe they induce it?) to create dull software offerings. I have yet to see a development company that settled on one particular tool, whether it’s for knowledge sharing, project management, or analytics. It’s always a combination of multitude of tools because “while Jira works good for us, we actually use Monday for our Gantt progress”; “well, yes, we collect analytics in Google Analytics, but then there is a process that moves data to snowflake to visualize it”, etc. SaaS models make sense from financial perspective, but they are far from being the most optimal technical solution, as they focus on wider distribution thus offering generalized solutions.

I’d argue that on-premise/self-hosted tools with access to source code behind a license that forbids redistribution, or dedicated development team, would be a better solution overall. A company would host the needed tool on their infrastructure—thus removing the need for the tool provider to comply with dozens of regulations (“oh, but we need you to have an EU datacenter because GOVERNMENT”)—while at the same time retaining the ability to modify the tool to suit their specific needs. In the end, you have a data store, you just need to enhance this data in order to make sense of it for you. Why would you move analytics data storage to a third party data broker requiring them to comply with whatever regulations you need to comply with, while accepting the risk of data mismanagement, downtime, or being a hostage of their pricing plan—when all this can be achieving by hosting your own, whether purchased or developed in-house, tool?

I know why. Because the fragmentation happens not only on the software level, but also on the infrastructure level. Modern software development is so convoluted, that nobody really knows how the deploy process works or why the fuck it takes 45 minutes for your code to reach production. Asking for a custom deploy solution, outside the established practices by the company, will result in a firm denial from DevOps as their custom terraform wrapper doesn’t know how to deploy Joe Shmoe’s analytics tools.

So we are left at the mercy of our Lord and Savior the REST API, which allows us to fragment our software further and further, while benefiting no one. Neither you, whose data is now being spread across N services and requires complex data pipelines, nor the SaaS developers, who try to appeal to the masses by building as abstract as possible—benefit from it. Really, an analytics tools that shows you how many people are online on your website for $20/m, or “CONTACT US FOR ENTERPRISE” pricing is the best we can do? Should I remind you that phpBB had this functionality built-in for decades?