Have you ever wondered why developing software represents a new challenge with each new project again and again?
You have been programming for a long time and starting to see some amount of effort is always there with the next project. Or you have been managing projects for quite a while and beginning to wonder why there are always delays and unpredicted difficulties. Or maybe you’re just a curious user who keeps wondering about why some projects rise with ease and shine their energy around while others are painfully crawling their way up the hill, set back by countless bugs and compatibility issues until finally giving up and rolling down the hill out of the common view. What is so difficult in software development really? The software industry has been there for decades. A vast variety of projects have been done to cover just about any problem domain. New programming languages come to replace the old ones and bring new features that increase productivity of developers and replace days and months of work with a single function call. New modern methodologies have replaced the old ways and polished the process down to the last button click. Lots of books on project management have been written, read, commented on, improved and accepted as the working guidelines. Lots of case studies about project success and failures have been presented, studied and even taught in universities. Every single aspect of creating a software product has been addressed with the exact guidelines on how to proceed.
The reality is, all those guidelines and recommendations just don’t work for software projects.
They simply don’t.
What’s wrong with the software industry? Why any other industry has managed to come up with some guidelines which when followed lead to guaranteed results with the defined quality in a predictable timeframe?
Consider automotive industry. If you wish to build a car, you know how to proceed. You decide on its working characteristics and you start on. You need to take into consideration the material properties, what metal of what thickness you will use. The wheels sizes, their width, the grip. The aerodynamic properties of the hull. The maneuverability as the function of its mass, size, wheels location and mass distribution. You know what to pay attention to in order to achieve the needed result. You can target the specified total price of a unit and you can give a good estimate on how long the engineering process will take. After all, you’ve built a car so many times it’s just a routine to do it one more time.
Or you decide to build a house. You get a spec on what capacity it has to offer. You hand it over to an architect who designs a project in the specified timeframe. Then you get the design approved and start on. You somehow manage to plan the whole operation from preparing a piece of ground for construction works up to completing interior design. You can even give estimation to local folks when they will have this ugly construction place disappear from their eyes to be replaced with a new modern building ready to receive new residents. It works. You have the experience, you apply it again and you get the expected result right on schedule. Fine!
It even works with non-engineering activities. You plan to come up with some product to the market. You conduct certain research into the consumer needs, interests and values. You study comparable products, analyze their properties, their popularity, their perception by various consumer groups, their history. You gather this data, process it and apply it to the design of your new product. It works most of the time. You aim at specific customer groups, adoption rate, sales, maybe you even think of a specific niche in a sociocultural environment. You calculate it all and you likely get there.
But it doesn’t work for software products. No matter how you try. The outcome just seems to be… random. Yes, a random result with no direct correlation to the process, money invested, time spent and effort put into it.
How is that possible, you keep asking yourself no doubt all the time?
Is it us who’re doing something wrong? Do we follow a wrong methodology? Aren’t we investing enough in quality control? Did we pick up a wrong programming language or a framework to do the job? Or is it our programmers who are not qualified enough to developer a product on time and money? Do we maybe need to hire more smart guys with a Ph.D. degree to drive use into the bright future?
Now we’re getting somewhere. But before starting to look for an object or a person to blame, please close your eyes and think about the nature of your work.
What to think about it, you say. It’s just another industry branch, like construction, car manufacturing, doing market research, breeding animals or developing chemicals. We’re developing software. We have our processes, our standards, our project management and our specific quality control. That’s all there is to it.
Nope, it’s not, I’ll answer. In fact, what you described is of little importance or even relevance to the understanding of what is so special about software development. What is the reason of the unpredictability of the final results? Why do we regularly miss our deadlines? Why some projects fail and others flourish? Why does market step on your professionally developed project without noticing it and embraces with their heart some little program an amateur wrote over the weekend?
Something is very different about the very nature of software development.
Think of other industries. There is usually a known process to follow. Most problems and issues are studied and precisely described. There are guidelines about the usage of components or building blocks. Their properties, weak and strong sizes are studied and analyzed. There is knowledge about how to put them together, how they work in combination, what side effects there are to expect. You can estimate the required amount of components, their price, their availability, how many workers you need, how long it will take until the final product comes from the assembly line. Even if you get something slightly wrong, the experience adds to the common knowledge, so that the next time you do a similar project you won’t make the same mistake again. In fact, working in many industries is like building a toy with some sort of a LEGO constructor. You just assemble blocks together and it works. Blocks can be physical components, tools, human and financial resources and other assets. Even if you begin something experimental, after a few times you gather enough experience and observations to document your work. It is now possible to repeat your success by following your instructions and guidelines.
In software development, it is not as easy as putting the constructor blocks together. I know, you believed otherwise. Well, you were wrong. In fact, there isn’t even a constructor available.
Deep down to the machine level, it’s all about manipulating data. From data come data structures, manipulation procedures make algorithms, these little pieces build higher-level components, which in their turn make up conceptual models until it reaches that level where human brain can embrace the whole and assemble the big constructor blocks into something meaningful.
But before that happens, you have actually to build a constructor. And it can be as difficult as the life.
Remember, those are just bytes behind the scenes. And like electrons and protons which come to make atoms which build up molecules, you can turn those bytes in just about anything. Square electrons, triangular protons, atoms which stay together with only members of the same family. Molecules, which only react with another molecules if they provide the necessary credentials. Molecules, which can fly. Or molecules with some intelligence to change their properties and behavior to mimic other materials when appropriate. Remember the Replicators from the Star Gate series? Some built simple moving machines. Others evolved to human-form constructions. When it was needed, they came together to form a space ship.
That’s exactly where the challenge of software development comes from. You have to build your basic elements first before you start turning the chaos into a beautiful world. And you should know what properties to give them so that you can use them everywhere to create more complex objects. You will need to create seas which do not flood continents. Then you add mountains which stick there, not take off when the wind blows and do not erupt to destroy vegetables and animal life. Now, you’ll have to add plants which will feed animals and keep the ground firm. The insects which will pollinate the flowers. The birds which will feed on fruits and eat insects while leaving them enough to do their job. Finally you add homo sapiens who will evolve into civilizations without making wars or building hadron colliders to create black holes to finally engulf the entire planet (joking). And that is provided your basic elements are suitable for all those tasks and you don’t have to rebuild them again.
You’re probably laughing already at the analogy. You shouldn’t really. Building a big software project is like creating a microworld. There are basic elements (data types), continents (modules), plants and animals (data structures), the forces of nature (data transformations, pipelines, workflows) and the laws of physics (protocols, data formats). And the complexity sometimes is so high it will scare you if you try to embrace the whole in your mind.
Happily, programmers have found the ways to deal with that complexity. You decompose the whole into parts, then each part in subparts until it reaches the level you can start on. You create smaller components then assembly them into bigger ones, then those into modules, until the modules make up the originally conceived whole.
Being capable of performing this kind of work requires certain skills. It calls for mind abilities to decompose the whole into components. It requires one to simultaneously think about multiple pieces in order to design basic components which will suit in several places. One also needs to think on different abstraction levels simultaneously in order to see the big picture, predict possible problems, bottlenecks and the directions a particular structure could evolve into. One needs to design an architecture that not only suits the current specs but is resilient enough to accommodate possible changes that are likely to happen. When they do, the general structure should hold and not cause disruptions in operations, damaging data or shutting down other systems. The issue has to be contained by design, until the developer makes the necessary adjustments to prepare the system for updated reality. And this is not easy, believe me. It requires a special organization of the mind to design software systems. It requires a lot of experience in order to know what and when to expect and prepare for the unexpected. It also requires the strong will to do your best.
If you’re curious, I’d advice you to read a fascinating story by Robert Sheckley, it’s called "Budget Planet". It’s about building a planet by construction workers then shipping the work to the customer. You will laugh at those dialogues of the customer complaining about the bugs and extending the budget, while a suddenly appearing end user will talk about the actual living experience on the freshly built planet. A wonderful reading!
But back to developers and design. One reason why a big software project results in an unstable and chaotically built construction likely built by some medieval tribes, is that because it is built by… well, members of those species. Meaning, by an inexperienced or unqualified individual or a team. Seriously, not everyone is ready to architect a software project. Not everyone is even capable of it. Having a degree in Computer Science does not guarantee quality result. A Ph.D. degree is neither a required nor a sufficient condition to be able to perform well. Because knowledge may lack experience. The same is true for a long field record. Because practice may lack knowledge and understanding. Even knowledge and experience may not be enough if there is no aptitude. Given that many companies hire programmers primarily based on their degree, official track record and how nice they look in a suit with a tie, it becomes clear that many teams are assembled of… well, often just casual people.
Even when knowledge, experience and abilities are all present, there may simply be no will or interest to build a quality product. This is sadly true for many projects out there. They’re steered by management unwilling to let the capable people do their best. Quality, speed, price. Pick up any two. As you know, the quality is usually left out. Because speed and price are the only tangible qualities usually understood by non-technical managers or uneducated customers. Quality belongs to the geek language. Something that cannot be specified, measured, estimated, something which is only mentioned by programmers who cannot make it to the business case. How do you explain quality to a customer who’s been seeing offers on freelancing sites like "Will build a Facebook clone for $50"? Right… You were saying something about the optimized architecture, improved query performance and some abstract code metrics which matter for later code maintenance? Skip it.
Another big risk factor in software projects is changing requirements. Beginning with the initial lack of those, of course. A customer wants to have software that does something around a problem area. Fine, a good team will talk to the customer and get a picture of what he needs. They’ll follow an agile process, will continuously talk to the customer and will finally come up with the good result that will please the customer. The problems come with those small changes or feature requests that the customer generously supplies later in the game. Sometimes these requests are just a minor nuisance to the team. But some of them really get in the way of the successful project completion. They sometimes require some good reengineering of the project architecture which is no ten-minutes-patch although it may seem this way on the customer side.
In no other "engineering" industry can a customer add new requirements which will requiring the partial reengineering of the work or even rebuilding it from the scratch. Imagine you have completed the construction of a nice big building with an underground parking, reception, elevators and a restaurant on the roof. The next day the customer would call and ask you to make some more room in the parking for his new private aircraft he just bought. Oh, didn’t I tell you this before? I was going to have my aircraft and I was planning to park it in the underground garage. How rebuild the complete building? You’re kidding right? How can you be so incompetent? Just move up a building, insert some more bricks and the garage will get higher, right? You can’t do that, I see. I obviously hired the wrong team. I’m going to switch the contractor, thank you. Or he’ll tell you he wants so supply the restaurant with some logistic company so he would need the trucks to enter the reception and go into the elevators to the upper floor where they will be unloaded. That’s ridiculous. In no other branch can you make such unrealistic requests. But with software projects, such requirements come all the time, because customers believe with software, everything is possible.
The sad thing, they are quite right most of the time. You can really adjust a virtual garage to accommodate a virtual aircraft. Hell, you can even make the garage fly along with the plane (whatever that would mean for virtual objects). It’s just about abstract concepts to manipulate the data. Data structures and algorithms can be adapted to meet many needs.
Even if this is true, this does not mean doing so would be easy. Though a virtual building, the efforts of its rebuilding can be pretty real and huge in terms of people’s work, time and money. As you build bigger on smaller, you can’t really change those big blocks without adapting the small. And if you do so, you may have to rebuild the other big blocks which depend on those modified smaller components. Because if you don’t, the rest of the software may stop working. Often, to avoid this trouble, a small patch is applied to some intermediary component to solve an immediate need. It can work for now. But then another request comes in and you apply one more fix. After some time of progressing with those patches, you get something like a virtual house infested with virtual termites. Everything is starting to decay. Unpredictable bugs, fixes which extend their influence to many other modules. Usually, a bad influence. You start to patch those, and something else breaks. After a while, it feels like a house is falling apart. Now the concept of abstract quality is starting to take shape. Pity, it’s too late.
Now, if you’ve been in the customer role and reading my writing… Please think the next time about those feature requests you make to the development team. When they warn you about possible problems, they may likely have a genuine case. It’s not that they’re lazy or just reluctant to do some more work. When you initiate a new project, bring all your ideas and considerations to the team. Don’t keep surprises for the later stage of the project. It may be impossible to make changes so the surprise will be your surprise. When you decide on a contractor, don’t necessarily choose the one who can do it all for only $50 and ship the product by the end of the week. The quality matters. And as far as the deadlines go, understand the complex nature of the software creation and be prepared to extend those deadlines as needed.
Because you’ve just ordered a small world to be built specially for you.