Programmer Musings

The Modular Monolith

Continuing the line of thought from last time (Sharp Tools vs. Frameworks), another issue I see in quite a few frameworks and some systems is a code anti-pattern I'll call The Modular Monolith.

We all know that modularity is a good thing to have in a system. Modular code, in general, reduces coupling between components, allows easier reuse, and simplifies understanding. When done correctly, each module can be analyzed, tested, and understood independently of most of the rest of the system. In object oriented programming, the smallest module we work with is the class. Usually a group of classes work together as a subsystem (or package). In other paradigms, the smallest module might be a library.

In any case, we are all pretty much familiar with the benefits and concepts of modularity. At the present time, it would probably be hard to find anyone that does not accept that modularity is a good design principle.

The Monolith

In some systems or frameworks, you may run into a problem using a single class. When you try to include the class, you find dependencies on other classes. In some cases, this is perfectly reasonable. If the class you are including requires some low-level utility classes to do its work, that's understandable. But sometimes, you find that the class depends on other classes at the same level of complexity. If some of those classes depend on other classes that depend on other classes, you can eventually reach the point of needing the entire framework (or system) to use any part of it.

In this case, we no longer have modular code, we have a monolith. The oxymoron modular monolith refers to the fact that the code is modular in the sense that there are modules. The problem is that the modules are so tightly coupled together that they might as well be a single monolithic stone. No piece can be used without bringing the whole structure along for the ride.

Causes

One mechanism that can cause this problem is the over-use of Singletons. The Singleton, by its nature, can cause hidden coupling between the classes that use the Singleton and the classes the Singleton uses. This is one reason why the test-infected are usually against the use of the Singleton pattern. A system with multiple Singletons can result in connections that are almost impossible to unravel.

Another cause of this increased coupling is low-level classes that depend on high-level classes. This violation of layers is almost guaranteed to generate blobs of classes that must always be used as a unit. Many systems cause this problem through trying to connect error-reporting to low-level code. As the error reporting becomes more advanced, it brings in subsystems unrelated to the purpose of the utility class.

The One True System

Frameworks aren't the only place you can see this anti-pattern. I've also run into this problem with systems written by people who have only worked in one system. The idea that code could be used outside the system never occurs to them. You can often recognize this situation with people who load up the multi-megabyte widget processing system as part of a piece of code to count the lines in a text file. Since they have always worked on this system, they treat it as the whole programming universe. All code must exist in the system.

The bad news is that these systems seem to be built on the modular monolith and the modular monolith further reinforces the attitude that everything must be done as part of the one system. This positive feedback makes stopping the behavior in either case almost impossible.

Sharp Tools vs. Frameworks

Maybe it's a reflection of when I started programming, but I've always had problems with frameworks. I prefer having a sharp set of tools to having a do everything framework. For me, the framework breaks down if I need to do something that does not match exactly what the framework designers wanted to do.

A good friend of mine, Rick Hoselton, once described this to me as the baloney slicer problem. He pointed out that some tools were absolutely spectacular at one job (say, slicing baloney), but he preferred a really sharp knife. With a sharp knife, you can do a lot of things once you know how to use it safely. You can even slice baloney. The knife is more dangerous, but it's more useful in general.

While I agree that a sharp knife is more generally useful than the baloney slicer, there are times (like when I need to slice 100 pounds of baloney) when the baloney slicer is the right tool for the job. My biggest problem with frameworks and some libraries is the combination of the framework and the Golden Hammer syndrome. This tends to cause the designers to adapt the framework outside the area it was designed for. Usually, it starts with a little change and moves on to some really monstrous results.

To stretch Rick's analogy a little further, I need to slice some ham. I am pointed to a feature of the framework that is kind of a combination cheese grater/wood chipper/jackhammer. (This is obviously the right piece of the framework because it cuts something, duh!) Now what comes out of this device is not really a slice, but that okay because we have some mostly edible processed food stuff that can be used to glue the pieces back into a slice-like mass. Obviously, this is much better than the knife, because it is part of the framework and it can reduce 100 hams into something resembling slices in a few minutes.

Obviously, I'm exaggerating to get the point across. However, I am surprised how often I run into this problem when using frameworks or extensive libraries. Sometimes, we're told that the framework is how we do things around here. Or, maybe we just don't want to re-invent the wheel. I suspect part of the issue is that the people who built the framework are just trying to show that their work is useful.

Unfortunately, they have lost track of the simple fact that a tool optimized for solving one problem will not be as effective for solving others. This is not a condemnation of the tool, its just a fact of life. As you optimize a tool, you automatically limit its scope. After all, optimal tools have a narrow focus, that's what makes them efficient. When you try to use an optimized tool for something it wasn't designed for, you lose all its advantages.

While frameworks have their place, they are not the solution to every problem. Don't forget that small, sharp tools are very important as well.

Flexibility and Rigidity in Software

One of the dimensions in which you can vary a design is in terms of the amount of flexibility it supports. Trade-offs along this dimension show up in many design decisions for various kinds of software. In some cases, software is made less flexible to improve predictability. In other cases, the adaptability of the software is paramount, so the design tends more toward flexibility. I believe that most programmers are comfortable with making these kinds of design trade-offs.

Programming Languages

There is one area of programming where this spectrum of flexibility vs. rigidity becomes more of a religious issue. When it comes to programming languages, programmers seem to divide them into extremes of order and chaos. Fans of statically typed languages look at dynamical languages as chaotic, undisciplined messes. Dynamic programming language advocates look at statically-typed languages as inflexible and overly verbose. (This actually predates the dynamic/compiled debate, I remember when us C programmers called Pascal a bondage and discipline language because of its rigidity.)

It is not quite this cut-and-dried, Java is considered more rigid or orderly than C++, even though both are considered statically typed languages. Among dynamic languages, Ruby is considered more flexible than Python. Different people will view different features as being too flexible or overly rigidity where others will find those same features to be perfectly reasonable.

Flexibility Means Adaptability

Even if you agree with me so far, I expect to tick off a lot of people with my next argument. I believe that very flexible languages are better than more structured languages when it comes to adapting to new programming paradigms and techniques.

Recently, Bruce Eckel wondered about Java: Evolutionary Dead End? Paraphrased, his argument is basically that new features being added to Java appear awkward and ugly because the changes have not set well with the design of Java.

Ten years ago, many of the proponents of the Java programming language pushed the clean, orthogonal design of the language. This mostly clean design has not handled change very well, possibly because there is not enough flexibility to handle modifications. Any changes to the language look bolted on. These features often don't look or work quite right because the original design had no room to accommodate this kind of change.

The C programming language, on the other hand, was very simple with sometimes multiple ways to do the same thing. The syntax was a bit more flexible with escape hatches in the system to allow programmers to subvert the system to get low-level work done. This slightly messy design was relatively easily modified to support first object oriented programming and later generic programming. Although many people complain about the C++ programming language, adding functional programming features with a little template magic was relatively easy compared to what the Java community is currently going through.

Even More Flexibility

Some other languages have handled massive changes with even more adaptability. (If I haven't offended you yet, get ready.<grin/>)

The Perl programming language added object oriented features with some relatively small syntax changes when moving from Perl 4 to Perl 5. Although many people would suggest that Perl syntax is the ultimate in chaos, this flexibility allowed powerful OO functionality without complete replacement of the language. Moreover, Perl's adaptability has allowed Perl programmers to explore different approaches to class design and try out strange features like the ability to modify a class that was created by someone else. (Yes, Perl already had that feature. Ruby did not invent it.) We have had people experimenting with different OO syntaxes and features over the last ten years and some of the results are going into Perl 6 as the only standard way to build classes.

At the same time, Larry added a few more features that made closures quite simple in Perl. Real lexical variable and code references were the main changes. Almost ten years later, Java is having trouble with this concept because the everything is an object approach and the closure/anonymous function approach have a relatively large impedance mismatch. This is not to say that Java is wrong, just that the design doesn't accommodate this kind of change very well.

Ruby and Python also used flexibility in their language design to explore new features. However, neither of these languages has quite the unbridled flexibility of Perl. Although many people consider that a bad thing, they miss a very important point. This was an explicit design feature of Perl.

Turn it to 11

But, there are languages that are more flexible even than Perl. I spent a fair portion of my career working in the Forth programming language. Two of the most interesting things about Forth is it's lack of syntax and its ability to run arbitrary code at compile time. This allows a good Forth programmer to tailor the syntax of the language to the problem at hand. It also allows things that would be practically impossible in another language.

For example, Rick Hoselton and I added a relatively powerful and efficient exception handling mechanism into the Forth we were using at the time in a couple of days. At another point in time, some of us implemented a prototype-based object model in less than a week. We also added RAII-type resource management at another time. Because the language was so malleable, ordinary programmers could make these design changes without completely redesign the language.

The downside of this flexibility is also obvious. A bad (or overly clever) programmer can generate code that no one will ever be able to figure out. However, good programmers can extend the language in useful ways, exploring language features and changes in paradigms without needing large standards organizations or completely new languages.

Back to Mainstream

If I haven't lost you already, I should reassure you that I am not advocating that every language should be as free and adaptable as Forth or Perl.

But, I am suggesting that we look at our languages a bit differently. As we've seen with Java, tool support and standardized libraries can be much easier with more rigid languages. These languages also support the ability to do very effective work with groups having a range of levels of experience. (The leader of one project I worked on decided we would work in Java, not because of any technical superiority of the Java programming language, but because you can hire Java programmers straight out of school.) Many types of applications are very easy to build in Java because of the strong frameworks supporting those kinds of applications.

Many Python proponents argue that their approach of only one way to do it makes it easier for new people to learn the language. There are fewer nooks and crannies to get lost in. But Python is more flexible than Java. A few years ago, Guido made some backward-incompatible changes to enhance the object model based on lessons learned as people pushed the edges of that part of the design. Although it took a major change to the language to really improve the object model, it was people pushing the edges of what could be done that helped design the change.

Perl advocates point to the flexibility of the language and lack of rigid rules as some of the features that help us get work done. The TMTOWTDI slogan is more than a platitude. It is a recognition that the adaptability of the language gives us the ability to do both amazing and horrible things in our code.

What people should try to realize is that these different levels of flexibility are not good or bad, they are different. Usually, languages steal from one another when they are first created, the more flexible languages keep stealing ideas and adapting to new situations. Maybe the more orderly languages could benefit from exploring new techniques in a more flexible language before committing to another one true solution to the problem.