Large IT systems frequently fail, at a huge cost to the U.S. and world economy. The reason for these failures has to do with complexity. The more complex a system is, the more likely it is to fail. It is difficult to figure out the requirements for complex systems. It is hard to design complex systems. And it is hard to implement complex systems. At every step in the system life cycle, errors accumulate at an ever increasing rate.

The solution to this problem is to break large complex systems up into small simple systems that can be worked on independently. But in breaking up the large complex systems, great care must be done to minimize the overall system complexity. There are many ways to break up a system and the vast majority of these make the problem worse, not better.

If we are going to compare approaches to reducing the complexity, it is useful to have a way of measuring complexity. The problem of measuring complexity is orthogonal to the problem of reducing complexity, in the same way that measuring heat is orthogonal to the problem of cooling, but our ability to measure an attribute gives us more confidence in our ability to reduce that attribute.

To measure complexity, I consider two aspects of a system. The first is the amount of functionality. The second is the number of other systems with which this system needs to coordinate. In both cases, complexity increases exponentially. This indicates that an effective “breaking up” of a large system needs to take into account both the size of the resulting smaller systems (I call this functional complexity) and the amount of coordination that must be done between them (I call this coordination complexity.) Mathematically, I describe the process of “breaking up” a system as partitioning that system.

The problem with trying to simultaneously reduce both functional and coordination complexity is that these two pull against each other. As you partition a large system into smaller and smaller systems, you reduce the functional complexity but increase the coordination complexity. As you consolidate smaller systems to reduce the coordination complexity, you increase the functional complexity. So the challenge in reducing overall complexity is to find the perfect balance between these two forms of complexity.

What makes this challenge even more difficult is that one must determine how to split up the project before one knows what functionality is needed in the system or what coordination will be required.

Existing approaches to this problem use decompositional design, in which a large system is split up into smaller systems, and those smaller systems are then split into yet smaller systems. But decompositional design is a relatively random process. The outcome is highly dependent on who is running the analysis and what information they happen to have. The chance that decompositional design will find the simplest possible way of partitioning a system is highly unlikely.

This paper suggests another way of partitioning a system called synergistic partitioning. Synergistic partitioning is based on the mathematics of sets, equivalence relations, and partitions. This approach has several advantages over traditional decompositional design:

• It produces the simplest possible partition for a given system, in other words, it produces the best possible solution.
• It is directed, meaning that it always comes out with the same solution.
• It can determine the optimal partitioning of a system with very little information about the functionality of that system and no information about its coordination dependencies, so it can be used very early in the system life cycle.

To someone who doesn’t understand the mathematics of synergistic partitioning, the process can seem like magic. How can you figure out how many baskets you will need to sort your eggs if you don’t know how many eggs you have, which eggs will end up in which baskets, or how the eggs are related to each other? But synergistic partitioning is not magic. It is mathematics. This paper describes the mathematics of synergistic partitioning.

The concept of synergistic partitioning is delivered with a methodology known as Simple Iterative Partitions (SIP.) It is not necessary to understand the mathematics of synergistic partitioning to use SIP. But if you are one of those who feel most comfortable with a practice when you understand the theory behind the practice, this paper is for you.

If you are not interested in the theory, then just focus on the results: reduced complexity, reduced costs, reduced failure rates, better deliverables, higher return on investment. If this seems like magic, remember what Arthur C. Clarke said: “Any sufficiently advanced technology is indistinguishable from magic.” The difference between magic and science is understanding.
 

ObjectWatch was founded in 1994 by Roger Sessions, who now serves as its CTO. He is the originator of The Software Fortress Model for Enterprise/Service-Oriented Architectures and is widely recognized for his decades of experience in enterprise architectures, distributed systems and Service-Oriented Architectures. To learn more about how to contract with Mr. Sessions and put his extensive experience to work for your company as you create your next-generation enterprise solutions check out our consulting services.