Programmer Musings

Don't Lose Sight of the Basics

I've recently been reminded of something I used to tell entry level programmers repeatedly: Review the basics.

And really I mean all of the basics:

• data structures
• basic algorithms
• basics of algorithm analysis
• language syntax
• core CS concepts
• testing strategies

In many programming positions, you can get away with coasting on these things for a long while. But, as you let your grasp of the basics deteriorate, you lose flexibility. You begin to use similar solutions to different problems you encounter. If you don't reverse the trend, you can become that guy that applies the same library/data structure/programming paradigm to every problem, no matter how badly it fits.

Some would ask why you should be familiar with lots of basic tools, when you can do your job with just a few. After all, if your employer is happy with your solution, who cares if it is the best? This is ultimately quite short-sighted. My argument has two parts.

The Practical Argument

From a practical standpoint, being familiar with more data structures and algorithms means that you are a more valuable employee. Even if your employer does not know what these tools are, who is more likely to get the raise: the guy whose caching algorithm fell over as the number of things to cache increased by a factor of 10 (because it was based on a linear array search) or the gal whose algorithm handled the change because it was based on a tree or a hash? Do you think the person who reduced the memory footprint on a critical server by changing data structures is going to be the first to lose his job?

Statistically, most of us will change jobs at some point. Do you think you will impress a potential employer with your conviction that no problem should need any structure other than a linked list? How about if you know everything there ever was to know about hash tables, but you can't sort an array? How about if your solution to every problem involves a SQL query?

The Craft Argument

Many of us became programmers from a love of solving problems and learning. At that point, everything was new and exciting to learn. Eventually, programming can become just a job. You fix bugs, write some new code, collect a pay check and go home. But, that does not have to be the end of it. We are all craftsmen to some extent.

To keep up with any craft, you need to maintain your tools. In programming, those tools are:

• your editor,
• your language,
• libraries, and
• your brain.

Part of maintaining your programmer brain is challenging it and honing your skills. When programming was new to you, you probably spent time implementing really simple programs to see how they worked. This exercise added basic tools to your mental toolbox. If you don't use those tools, they get rusty and can fail when you need them.

In times past, when a craftsman worked with his hands, the practice of maintaining and upgrading his tools was critical to being able to do the craft. You can bet that a woodworker kept all of his chisels sharp, clean, and well protected; including the ones that he would only use once in a great while. When he needed that tool, out it comes to solve the problem it was meant for.

We need to do the same with our mental tools.

Conclusion

Over the next few posts, I'm going to review a subset of the basics. This is partly an exercise to force me to clean my mental tools and partly to serve as an springboard for you to have an idea about concepts you might want to brush up on. I don't expect to cover everything you should know, but these posts should serve as a good start.

Programmer Beliefs

The practice of developing software depends on a large body of arcane knowledge and skills. Almost all of this knowledge is based on logical principles and built up to the point that we can do amazing work. Because programming has it's roots in applied mathematics, there is an impression that programming is also a logical, rational process.

The Rational Ones

Many stereotypical programmer types seem to consider themselves to be rational people. In fact, we seem to believe that we are more rational than the average person. Our work requires a level of detail that most people can't deal with (or so we believe). Computers just do logical manipulation of bits, and our programs are (fundamentally) just logical and arithmetic manipulation of bits and numbers. Therefore, our work is definitely logical and rational.

There's just one problem with this view. Some of what we do is pure personal preference. That is not surprising, since we are still human. For that reason, we still make many decisions based on what we like or what we are used to, rather than for purely rational reasons.

Rationalizing the Preferences

However, we tend to believe that (at least in software development) we are making decisions for logical, rational reasons, not because of mere preferences. That sounds too much like choosing something because it's fashionable. Or worse, picking something for no reason. As rational software developers, we would never do that (assume appropriate amount of sarcasm in that statement).

If you want to see programmers completely lose their cool, start a discussion about the best editor, programming language, or indenting or brace-placement style. There's a reason why people refer to these as religious wars. We've all seen programmers get into major arguments over these issues, and they all have careful and sometimes very detailed arguments about why their choice is the right one. More importantly, since each of them believes they have made the choice for rational, objective reasons, the logical conclusion is that anyone who doesn't agree must be wrong. And the rational thing to do is convince the other person of the flaws in their belief and force them to bow to the better objective argument.

The Subjective Preferences

There's just one problem. These choices really are subjective. Each of us made the choices we did for very irrational reasons:

• It was the first style or language we learned.
• Someone we admired used that editor.
• The idiosyncrasies of that tool fits our brain better than the alternatives.
• We want to fit in with the community that uses that tool.

This point was driven home to me years ago, when another programmer and I were discussing brace placement in C. We were trying to develop a standard for our group and the two of us came down solidly in opposite camps. As we were arguing, I suddenly realized that he did not see the code the same way I do. I don't mean this in any deep philosophical sense. I mean it literally. The placement of the braces affects the way I pattern match while scanning code. But, when he saw code, the braces were less important than the indenting.

I actually stopped arguing for a while when I realized that. I finally was able to explain what I saw when looking at the code and we ended up deciding to use the style that was most common in the code, rather than changing to either of our positions.

Don't Assume

When one of these issues comes up, remember that they really are matters of preference and convention. No one approach is provably, rationally, better than the others. It's more important to realize that the other person is making the choice for reasons that make sense to them, and you won't be able to convince them otherwise. The important point is for us to compromise enough to be able to work together, and solve real problems.

Which Programming Languages are Write-Only?

A comment on the Security Now Podcast got me thinking about the concept of write-only languages. Many times over my career, I've heard people refer to a language as write only. The implication being that no one can read the code once its written. This phrase does not refer to some magical IP technology that prevents people from stealing code you've written. It reflects the belief that the language in question is so badly designed, uses such obscure syntax, or is so powerful that any program written in the language becomes almost impenetrable to anyone trying to read the code.

I would like to humbly suggest that this concept is (mostly) nonsense. The cause of the biggest complaints seems to be unfamiliarity with the language in question. Just like most of us would find ancient Egyptian or Middle English hard to read, it can be difficult to understand a programming language if you don't know the basic syntax. Even once you learn the syntax, there is a huge difference between understanding code written in the simplest form of the language and an idiomatic program written in the language.

Perl

I have programmed in Perl for a long, long time, well over a decade in fact. I have been told right to my face by fans of other languages that it is flatly impossible to write readable or maintainable code in Perl. It always comes as a shock to me, since I have written and maintained small programs and large systems in Perl for years. It would probably come as a bigger shock to the teams that worked on the Human Genome Project that used quite a bit of Perl (and were apparently able to read it). Not to mention quite a few companies that develop and run Perl code all of the time (Booking.com, the BBC, TicketMaster, and cPanel all come to mind.)

When pressed, I often find that the person making this claim has never used Perl, or possibly only wrote a handful of quick scripts. This is hardly enough experience to make a reasonable judgment.

C++

I also have quite a bit of professional experience working in C and C++. I have been told by some that these languages are also write-only. Yet, there are millions of lines of code in each of these languages running everything from embedded systems up to large research projects, financial institutions, and video processing. Obviously someone finds this code at least somewhat readable or it would not be maintained.

Forth

The comment from Steve Gibson that kicked this off involved the Forth programming language. Now, Steve is an expert at computer security and (by all accounts) an excellent programmer. But, my experience maintaining and developing a very large Forth code-base over a period of eight years, is quite different than his opinion. Forth is very different, if you are used to most modern languages. But, that does not make it write only.

I particularly find this comment amusing from someone who writes Windows software in assembler. I believe that it is possible to write readable assembler, despite not having seen many examples. But, I'm willing to bet there are many out there who would declare assembler to be the ultimate write-only language.

Others

When I was pursuing my degree in CS, I had the opportunity to learn and program a bit in Lisp. While the syntax is not my favorite, I found well-written Lisp to be mostly readable. I've had a similar experience with the small amount of Haskell I've done. The same goes for SmallTalk and so on.

Language or Program?

While I'm not willing to suggest that a particular language is write-only, I can agree that some programs are effectively write-only. But that trait does not appear to be limited to a particular language. In every language I've worked in or even just read, I've seen examples of readable code and examples of garbage I hope to never have to maintain.

In many cases, the really bad examples come from people unfamiliar with the language or just inexperienced. The more readable code normally came from more seasoned programmers. The readable code also tended to become more readable as I became familiar with the idioms of the language or even the team that wrote the code.

Viewpoint

Although it is probably not universally true, most of the people I have heard scream write only language have experience with only one or a few languages. Much like we Americans who only know one language and consider all of those foreign languages too hard, many of these one-language programmers are still focused on the One True Syntax and have not moved on to the concepts behind the syntax. (Forgive the stereotype, but it's a good analogy.)

In some cases, the opinion was not a case of single-language blinders, but just plain unfamiliarity.

Moving Beyond the Insults

Not everyone is going to like every programming language. That's a given. Some programming languages don't fit the way you think. But, claiming a language is write-only seems ridiculous. Most of the languages I don't like, I've programmed in long enough to be able to read them, even if I would prefer not to.

I also made an explicit decision after a discussion with another programmer years ago to have no opinion on any language I haven't used. Even if other people have really bad things to say about it. I don't know what it's like, so I have no opinion. I find this makes things easier if the job ever requires me to do a little maintenance on a language I don't know.

I've also found in the past that the languages that worked the most differently from the way I think were the most valuable to learn. Those languages were the ones that taught me the most fundamental aspects of programming, because I got to see familiar programming concepts in a completely new context.

Update: I realized that I should have made clear, that Gibson had made clear that his difficulty with reading Forth was due to unfamiliarity. He does have experience with other languages.

The Literals of Functional Programming

I was listening to an older episode of Software Engineering Radio where they interviewed Martin Odersky on Scala (Episode 62). In the interview, Odersky made a comment about closures being the literals of functional programming. This statement struck me as surprising. The more I thought about it, the more interesting and subtle the concept became.

In functional programming, functions are first-class values. (OOP advocates often say first-class objects, but this concept is more fundamental than that.) If a function is a value, like an integer is a value, a closure (or anonymous function) is the equivalent of a literal (like 2). This idea has been wandering around in my head for the last few days. The more I think about it, the more interesting it appears.

To get the most that you can out of an analogy, you need to apply what you know from one side of the analogy to the other to see if it gives any insight. Let's start with what we know about literals:

Magic Literals

We all know the problems with magic literals. Leaving them scattered around the code is a guaranteed way to make unmaintainable code. If the same magic literal is used in multiple spots, the making changes is harder than it needs to be. We normally solve this problem by replacing the literal with a defined constant. If that constant is well-named, this improves the readability of the code. (If the constant is not well-named, it can actually make things worse.)

There are a few cases where we may decide to use a literal despite this fact.

The initial index for a loop is almost always 0 (for C, C++, Perl, Java, etc.). There's really no need to name this literal, because it's function is obvious and it will not change (for a given language). Just as importantly, it's hard to come up with a name that means more to us that the value of the constant itself (please don't name it ZERO).

If you are working with date manipulation code, the literals 7 and 12 are obvious enough that we might not choose to make the constants DAYS_PER_WEEK or MONTHS_PER_YEAR. This decision is not quite as obvious, but it would be relatively easy to justify not making a constant. On the other hand, most people would probably prefer MINUTES_PER_DAY to 1440 and SECONDS_PER_DAY to 86400. This shows that if the use of the literal is obvious enough, we may not need to name it. If there's any question about the purpose of the constant, we give it a name.

Another possibility would involve implementing an algorithm from a book that contained a large number of single use magic literals. You would probably keep the literals and reference the original algorithm in a comment rather than add constants that weren't in the original algorithm. In this case, being faithful to the original algorithm trumps concerns over magic literals.

Function Literals

The first thing that is worth considering about anonymous functions from this analogy is the possibility that giving the function a name might be wise. If the function is used in many places, giving it a name is probably good for maintenance. (Just like making a constant out of a numeric literal simplifies maintenance.) Likewise, code containing a large number of anonymous functions is likely to be hard to maintain.

Also like the magic literal issue, there may be cases where the anonymous function is obvious enough that no name is needed. I'm going to propose some examples using Perl for the syntax. This is mostly because Perl has a fairly reasonable syntax for anonymous functions. It also has three operators for manipulating lists that make use of anonymous functions: map, grep, and sort, Let's use map. (The map operator generates a new list by applying an anonymous function to all of the elements of another list.)


@newlist = map { $_*2 } @oldlist;


Once you know that $_ is the argument to the subroutine, it is obvious that this anonymous function doubles its argument. In this context, @newlist is a list where all of the elements are the doubles of the elements of @oldlist. A similarly obvious example (with better named variables) would be:


@elapsed_secs = map { $_*60 } @elapsed_minutes;


This anonymous function converts minutes to seconds by multiplying by 60. Giving the function a name would not really clarify things any. What about


@newlist = map { $_*($_*17 + 33) + 42 } @oldlist;


This example is the equivalent of a magic literal in the code. I can't imagine anyone looking at this and deciding that it is perfectly clear. Much like code containing the magic literal 17, this function should be named. Otherwise, the code is hard to read and maintain.

Filtering with Predicates

Another place that anonymous functions are useful is when filtering a list using grep. Like the map cases above, we can see that simple tests make good function literals.


@positive = grep { $_ > 0 } @unfiltered;


This anonymous function obviously selects positive numbers. Just like the next one selects numbers below a cutoff.


@selected = grep { $_ < $cutoff } @unfiltered.


However, just like the map, we can have examples that would benefit from naming the function.


@filtered = grep { $_ > 3 && $_ < 13 && $_%2 == 0 } @unfiltered;


It's going to be practically impossible to understand this without some help, so turning this anonymous function into a named function is probably worthwhile.

Further Consideration

Like numeric literals, functional literals can make code harder to understand. However, giving all closures names as a knee-jerk reaction is not a valid answer. Consider naming anonymous functions when it would make the code clearer without breaking it. I'm still not sure whether anonymous functions are as bad for maintenance as numeric literals. More thought and experimentation is needed.

So far, I don't feel like I've made it through very many of the implications of anonymous functions being the functional programming equivalent of literals. I'll probably write more later, as more implications come to me.

"Shortcomings" of the Web

I was reading The JavaScript Phrasebook on Safari and ran across one of my favorite pet peeves when people write about Web technologies.

Cookies are not a specific browser technology, but a mechanism in the client/server model to overcome one major shortcoming in the HTTP protocol. HTTP is stateless, which means that the protocol does not have a memory.

There are many people writing about web-based technologies that really appear not to understand the design of the web. Statelessness is not a shortcoming, it is a feature. Contrary to what these people apparently think, the designers of HTTP did not forget to include state in the protocol. Most client/server protocols before that point either kept state as part of the protocol or relied on a persistent connection for the session (which is just state at the TCP level). The default approach for HTTP would have been to do the same. The designers of HTTP explicitly chose not to keep state in the protocol.

One of the reasons that the web exploded was the fact that the HTTP protocol is stateless. The average web server does not need to retain connection information on every browser request that it receives. This made developing early web applications and specialized servers much easier. No need for storage to track browser sessions. No need to keep open connections or determine when to expire old sessions. No need for complexity that has nothing to do with serving individual pages to a browser.

Part of the genius of the minimal approach taken by HTTP is the fact that stateful applications can be built on top of it through mechanisms like cookies. (There were more approaches at the beginning. Cookies eventually won out.) The great thing is that systems that don't need stateful connections don't pay for them in complexity and bandwidth. For systems that need the extra state, the complexity and extra bandwidth is part of doing business.

HTTP is a relatively minimal protocol with mime-headers as an extension mechanism. This design decision has proven itself as being an extremely robust and stable protocol. The original version was implement sometime around 1990. RFC1945 described version 1.0 in 1996. Since that time, we have had one new version 1.1. This protocol has been stable for almost 20 years and underlies every request over the web.

Despite the success of this design, some people still find the need to point out shortcomings in a design they don't understand.

Because of all of this I remain amused when someone, who obviously doesn't understand the concept of a minimal design, refers to that design as falling short.

Review of The Productive Programmer

The Productive Programmer
Neal Ford
O'Reilly, 2008

When I found out about this book, I knew I had to read it. I have been a programmer for a long time and I'm always on the lookout for tips about how to become more productive. I expected a list of tricks that would make individual development tasks easier. This book does have some of those, although not as many as I expected.

The book is divided into two major parts. The first section, Mechanics, contains a number of useful tricks and tips organized in four major categories: Acceleration, Focus, Automation, and Canonicality. This organization is the first useful feature of the book.

Instead of just listing random tips, Ford spends a bit of time on what makes us productive and groups his tips accordingly. This approach not only gives specific tips, but also gives you a framework into which you can put your own tips. By reviewing the tips in a particular section and thinking about the tasks you do every day, you open up the possibility of spotting ways to make yourself more productive.

Although the book has a strong developer focus, many of the tips would be useful for any computer user that wants to be more productive. Much of Ford's advice flies in the face of the GUI/user friendly path taken by modern operating systems and programs. He makes a strong argument that a command line is more effective in the hands of someone who has learned to make use of it.

The second section, Practice, looks at higher-level practices that can make you more productive. In this section, Ford touches on Agile practices (TDD and YAGNI), good design, philosophy, meta-programming, and tools, among other things. In each case, he focuses on each practice in terms of productivity rather than methodology or dogma. This section will not teach you how to develop in an Agile fashion, but it does show how some practices make you more productive.

This section suggests some answers to the high-level questions we all struggle with:

* How do you make certain the code you write is the best it can be and solves the problem you need to solve?
* How do you avoid writing code that does not need to be written?
* How do you get the most out of your tools.

Many programmers will be familiar with many of the tips or, at least, the ideas behind them. Unless you have decades of experience on multiple kinds of systems, you will probably still find a few new items. At the very least, it may remind you of tricks you had forgotten.

I would definitely recommend this book to any programmer or computer power user. If you've only worked on Windows or always develop with a powerful IDE, prepare to have your beliefs challenged. If you have experience with the command line, this book still provides a lot of benefits in tools to improve the more GUI-based OSes, if you need to work there.

Perl != Regular Expressions

Yesterday, I wrote an essay on a comment made in Jeff Atwood's Coding Horrors blog about regular expressions. While Atwood spends quite a bit of time on the two problems joke and talks passionately about regexes as a tool, he does take a swipe at Perl that seemed somewhat uncalled for.

I have seen Perl bashed by lots of people over the years. Some have never written (or read) any Perl, but still feel qualified to bash the language. I've also been told that the only reason that I could possibly like Perl is if I had not written or maintained anything serious in Perl. When I point to large applications running in a 24/7 data centers that I worked on for years, they normally change the subject. Pointing to the number of financial institutions, research groups, and large corporations that depend on Perl to function is also illuminating.

Regular Expressions

I often see regular expression bashing and Perl bashing tied together. There seems to be this weird meme running around that regexes are the only tool in the Perl toolbox. Both Atwood and Zawinski (see Atwood's post that started this line of thought) seem to take the viewpoint that Perl is nothing but regexes or that Perl somehow forces you to to do everything through regexes.

Anyone who has worked with Perl for very long has seen that the language has very strong support for regex processing. This makes sense because the original goal of the language was text processing and regexes are made for text processing. For many, Perl was their first introduction to industrial strength regexes. Maybe it's not too surprising that they decided to overuse that powerful little tool.

Perl is also a general purpose language that supports OOP as well as a procedural style. It also supports list processing and functional programming. There are modules on CPAN for controlling hardware, accessing databases, biology, and astronomy. There are also natural language processing modules, XML parsers, and web frameworks.

Perl is a powerful, flexible language that some of us use to get actual work done.

Passionate About Perl

As much as some people hate Perl and find it necessary to build up their language of choice by bashing Perl, some of us find the language to be a natural tool for many jobs. I have worked professionally in half a dozen general purpose languages over my career. But for solving a problem quickly, I normally turn to Perl. And I don't just mean for quick and dirty scripts. If I need to solve something in a short period of time and be sure it will work for years, I also often use Perl.

Bjarne Stroustrup once said:

There are only two kinds of programming languages: those people always bitch about and those nobody uses.

I would say that both C++ and Perl definitely fall into the first category.

People often pick on other languages. We've all heard the complaints (that are at best half true). Almost everyone picks on Basic for being a bit of a kiddie language. Cobol is the old-style business language that rots the brain. C++ is too baroque. Python is obsessed with indentation. Lisp has too many parentheses. Staticly typed languages are too obsessive and dynamic languages are either too slow or too loose.

Although the fans of each language might insult other languages in a friendly sort of way, mention Perl and the vitriol begins to fly. I know quite a few people who really like Perl and I know at least as many who truly hate it. As Kathy Sierra pointed out a few years ago If some people don't HATE your product, it's mediocre..

Based on the level of hate that Perl seems to inspire, it's safe to say that the language is definitely not mediocre.

Regular Expressions as a Problem

Jeff Atwood had an interesting post on regular expressions a few days ago. Most of the article talks passionately about the usefulness of regular expressions, strategies for writing readable regular expressions, and tools to help write and debug regular expressions.

Along the way, Jeff points out a comment that usually pops up when the topic of regular expressions appears:

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

The post points to some interesting further research on the quote and then gets down to the meat of his regular expression advocacy. But not before taking a swipe at Perl with:

Should you try to solve every problem you encounter with a regular expression? Well, no. Then you'd be writing Perl, and I'm not sure you need those kind of headaches.

I know I shouldn't be, but I'm continually surprised by how many people really seem to dislike Perl.

The Two Problems

First off, I have to say that I do find the two problems quote to be amusing and somewhat clever (at least, the first time I heard it). According to Jamie Zawinski, who originated this version of the quote, he got it from another version before the used sed instead of regular expressions. Honestly, you could probably insert the name of any technology that you wanted to bash in that spot and get an equivalent quip.

Part of why the quote is funny is a real kernel of truth. Any time someone learns a new technology, they attempt to apply it to every problem that comes along. Doing so almost always results in an additional problem. (Using the wrong tool for the job.)

I would, however, like to to counter it with a question from a good friend and mentor, Rick Hoselton:

What do you call a programmer with only one problem?

Unemployed.

Review of Beautiful Code

Beautiful Code
Edited by Andy Oram & Greg Wilson
O'Reilly, 2007

Although I was really looking forward to reading Beautiful Code when I first heard it was coming out, I found myself a bit disappointed by the reality of the book.

Each chapter is written by a different master coders, giving their views on what makes code beautiful. Some of the essays did a very good job of explaining things that I agree are critical to beautiful code: readability, succinctness, clarity, etc. Other essays discussed qualities that I would not have called beautiful. Some of these included qualities from some of the ugliest code I have ever worked on. The fact that I agreed strongly with some chapters and disagreed just as strongly with others made the book hard to read at first.

Eventually, I realized that this was just another case of beauty is in the eye of the beholder. The qualities that each author praised simplified their lives in the context of the problems that they were solving. In that context, the code was beautiful. In another context, those same qualities would not be as good.

Although I disagreed with some of the authors, I think the overall message was a good one. Some authors praised relatively universal qualities. Others praised particular architectures as making a huge difference in the code. But, all of the authors provided some insight into what different professional programmers see when they look at code. The differences in viewpoints are probably as important as the individual essays.

Overall, I would recommend this book for strong intermediate to senior software developers. I think a junior programmer might not have enough experience to be able to recognize the difference between generally good qualities and qualities that are good in a particular context. Even for the intermediate to strong programmers, I would warn that some of the chapters are definitely harder going than others.

I really don't think this book is for everyone. If you are more interested in learning libraries in your language of choice for solving today's problems, this book won't be of any use. If you have work on enough projects, in enough languages, to really understand that the one, true way to develop software is a myth, then the various viewpoints in this book will help you think about what code beautiful to you.

Abstraction as Compression

Long ago, I was trying to convince a friend of mine that Object Oriented programming was not all just snake oil when he asked me a fundamental question.

What's the difference between an object and a thingie?

In some ways, this question has guided my understanding of objects ever since. Fundamentally, what makes one collection of member (instance) data and member functions (methods) an object and another nothing more than a collection of data and code? What is the fundamental nature of an object?

In one sense, the answer can be summed up with my favorite quote from Ruminations on C++:

use classes to represent concepts

In a broader sense, objects are all about abstraction. Most of programming, and OO programming in particular, is an exercise in abstraction. We want to separate what you need to know to perform some action from the details you don't need to know. Abstraction is the name we give for selectively hiding or ignoring the details we don't care about so we can focus on what really matters. Abstraction is what allows us to work with files instead of magnetic domains arranged in tracks on spinning platters on a hard drive.

Any time you give a simple name to a complex collection of behaviors, you have created an abstraction. But, not all abstractions are created equal. A collection of random pieces of data and methods in a FooLib class is not a particularly good abstraction. Yes, it collects together information under a single name. Unfortunately, the simplest translation of that name is the source code. In order to understand any piece of the functionality, you need to go look at how it's implemented.

Good Abstractions

A simple, good abstraction is a stack class. There is an independent concept in software of a stack. You don't need to understand the actual implementation details and internal data. All you need is to know about the push and pop methods. A few other methods might be added for looking at the top of the stack without removing an item and for determining the number of items in the stack. However, calling the class Stack brings along a bunch of expected behavior without need of explanation.

One of the greatest benefits of the whole design patterns movement was good names and definitions that can be used as high-level abstractions. You don't need to know about the implementation to know that an Iterator allows traversal of a container, or that a Factory creates other objects. In fact, by giving a complicated concept a simple name, we have performed a kind of compression.

Information Compression

When I call an object an adapter, you immediately know that its purpose is to convert the interface of a class into a match different interface. You also know something about expected costs of this delegation and that the adapter itself doesn't need to provide any major functionality of its own. You also know that it is likely that the adapted class either cannot be changed, or that changing it would affect too many other systems. It is also likely that we are using this older class in a new interface.

But, I don't need to say all of that, I just say the class is an adapter. That is a fair amount of compression, reducing a whole paragraph into one word.

A good abstraction provides compression of a lot of information into a single concept. Part of the compression involves the amount of work or added information needed to decompress the information. As a friend of mine once pointed out: ISBN is a really strong compression algorithm. Any book can be compressed into a 10-character string; but decompression is a bummer.

Decompressing a good abstraction to gain understanding requires some amount of additional information. If this information is general (like design patterns), you can reuse the explanation many times, reducing the cost of the decompression for each use of that pattern. If the only explanation for what the class does is the source of the class itself, there is not much abstraction. This is more like the ISBN example. To understand what ISBN: 0-596-51004-7 expands to, you need to get and read the book (Beautiful Code).

Good Abstractions

One way to recognize a good abstraction is to examine the level of compression (including the amount of information needed to decompress). If the only way to understand the abstraction is to read the source (and re-read the source, ...), odds are the abstraction is not very good.

If understanding a particular class requires a bunch of extra information that happens to be part of the business domain, we may still have a good abstraction. In that case, the extra information may be able to be amortized across several other classes.

Abstraction as information compression may be a useful concept for determining if any of your classes are actually thingies.

The Modular Monolith

Continuing the line of thought from last time (Sharp Tools vs. Frameworks), another issue I see in quite a few frameworks and some systems is a code anti-pattern I'll call The Modular Monolith.

We all know that modularity is a good thing to have in a system. Modular code, in general, reduces coupling between components, allows easier reuse, and simplifies understanding. When done correctly, each module can be analyzed, tested, and understood independently of most of the rest of the system. In object oriented programming, the smallest module we work with is the class. Usually a group of classes work together as a subsystem (or package). In other paradigms, the smallest module might be a library.

In any case, we are all pretty much familiar with the benefits and concepts of modularity. At the present time, it would probably be hard to find anyone that does not accept that modularity is a good design principle.

The Monolith

In some systems or frameworks, you may run into a problem using a single class. When you try to include the class, you find dependencies on other classes. In some cases, this is perfectly reasonable. If the class you are including requires some low-level utility classes to do its work, that's understandable. But sometimes, you find that the class depends on other classes at the same level of complexity. If some of those classes depend on other classes that depend on other classes, you can eventually reach the point of needing the entire framework (or system) to use any part of it.

In this case, we no longer have modular code, we have a monolith. The oxymoron modular monolith refers to the fact that the code is modular in the sense that there are modules. The problem is that the modules are so tightly coupled together that they might as well be a single monolithic stone. No piece can be used without bringing the whole structure along for the ride.

Causes

One mechanism that can cause this problem is the over-use of Singletons. The Singleton, by its nature, can cause hidden coupling between the classes that use the Singleton and the classes the Singleton uses. This is one reason why the test-infected are usually against the use of the Singleton pattern. A system with multiple Singletons can result in connections that are almost impossible to unravel.

Another cause of this increased coupling is low-level classes that depend on high-level classes. This violation of layers is almost guaranteed to generate blobs of classes that must always be used as a unit. Many systems cause this problem through trying to connect error-reporting to low-level code. As the error reporting becomes more advanced, it brings in subsystems unrelated to the purpose of the utility class.

The One True System

Frameworks aren't the only place you can see this anti-pattern. I've also run into this problem with systems written by people who have only worked in one system. The idea that code could be used outside the system never occurs to them. You can often recognize this situation with people who load up the multi-megabyte widget processing system as part of a piece of code to count the lines in a text file. Since they have always worked on this system, they treat it as the whole programming universe. All code must exist in the system.

The bad news is that these systems seem to be built on the modular monolith and the modular monolith further reinforces the attitude that everything must be done as part of the one system. This positive feedback makes stopping the behavior in either case almost impossible.

Sharp Tools vs. Frameworks

Maybe it's a reflection of when I started programming, but I've always had problems with frameworks. I prefer having a sharp set of tools to having a do everything framework. For me, the framework breaks down if I need to do something that does not match exactly what the framework designers wanted to do.

A good friend of mine, Rick Hoselton, once described this to me as the baloney slicer problem. He pointed out that some tools were absolutely spectacular at one job (say, slicing baloney), but he preferred a really sharp knife. With a sharp knife, you can do a lot of things once you know how to use it safely. You can even slice baloney. The knife is more dangerous, but it's more useful in general.

While I agree that a sharp knife is more generally useful than the baloney slicer, there are times (like when I need to slice 100 pounds of baloney) when the baloney slicer is the right tool for the job. My biggest problem with frameworks and some libraries is the combination of the framework and the Golden Hammer syndrome. This tends to cause the designers to adapt the framework outside the area it was designed for. Usually, it starts with a little change and moves on to some really monstrous results.

To stretch Rick's analogy a little further, I need to slice some ham. I am pointed to a feature of the framework that is kind of a combination cheese grater/wood chipper/jackhammer. (This is obviously the right piece of the framework because it cuts something, duh!) Now what comes out of this device is not really a slice, but that okay because we have some mostly edible processed food stuff that can be used to glue the pieces back into a slice-like mass. Obviously, this is much better than the knife, because it is part of the framework and it can reduce 100 hams into something resembling slices in a few minutes.

Obviously, I'm exaggerating to get the point across. However, I am surprised how often I run into this problem when using frameworks or extensive libraries. Sometimes, we're told that the framework is how we do things around here. Or, maybe we just don't want to re-invent the wheel. I suspect part of the issue is that the people who built the framework are just trying to show that their work is useful.

Unfortunately, they have lost track of the simple fact that a tool optimized for solving one problem will not be as effective for solving others. This is not a condemnation of the tool, its just a fact of life. As you optimize a tool, you automatically limit its scope. After all, optimal tools have a narrow focus, that's what makes them efficient. When you try to use an optimized tool for something it wasn't designed for, you lose all its advantages.

While frameworks have their place, they are not the solution to every problem. Don't forget that small, sharp tools are very important as well.

Flexibility and Rigidity in Software

One of the dimensions in which you can vary a design is in terms of the amount of flexibility it supports. Trade-offs along this dimension show up in many design decisions for various kinds of software. In some cases, software is made less flexible to improve predictability. In other cases, the adaptability of the software is paramount, so the design tends more toward flexibility. I believe that most programmers are comfortable with making these kinds of design trade-offs.

Programming Languages

There is one area of programming where this spectrum of flexibility vs. rigidity becomes more of a religious issue. When it comes to programming languages, programmers seem to divide them into extremes of order and chaos. Fans of statically typed languages look at dynamical languages as chaotic, undisciplined messes. Dynamic programming language advocates look at statically-typed languages as inflexible and overly verbose. (This actually predates the dynamic/compiled debate, I remember when us C programmers called Pascal a bondage and discipline language because of its rigidity.)

It is not quite this cut-and-dried, Java is considered more rigid or orderly than C++, even though both are considered statically typed languages. Among dynamic languages, Ruby is considered more flexible than Python. Different people will view different features as being too flexible or overly rigidity where others will find those same features to be perfectly reasonable.

Flexibility Means Adaptability

Even if you agree with me so far, I expect to tick off a lot of people with my next argument. I believe that very flexible languages are better than more structured languages when it comes to adapting to new programming paradigms and techniques.

Recently, Bruce Eckel wondered about Java: Evolutionary Dead End? Paraphrased, his argument is basically that new features being added to Java appear awkward and ugly because the changes have not set well with the design of Java.

Ten years ago, many of the proponents of Java pushed the clean, orthogonal design of the language. This mostly clean design has not handled change very well, possibly because there is not enough flexibility to handle modifications. Any changes to the language look bolted on. These features often don't look or work quite right because the original design had no room to accommodate this kind of change.

The C language, on the other hand, was very simple with sometimes multiple ways to do the same thing. The syntax was a bit more flexible with escape hatches in the system to allow programmers to subvert the system to get low-level work done. This slightly messy design was relatively easily modified to support first object oriented programming and later generic programming. Although many people complain about C++, adding functional programming features with a little template magic was relatively easy compared to what the Java community is currently going through.

Even More Flexibility

Some other languages have handled massive changes with even more adaptability. (If I haven't offended you yet, get ready.<grin/>)

Perl added object oriented features with some relatively small syntax changes when moving from Perl 4 to Perl 5. Although many people would suggest that Perl syntax is the ultimate in chaos, this flexibility allowed powerful OO functionality without complete replacement of the language. Moreover, Perl's adaptability has allowed Perl programmers to explore different approaches to class design and try out strange features like the ability to modify a class that was created by someone else. (Yes, Perl already had that feature. Ruby did not invent it.) We have had people experimenting with different OO syntaxes and features over the last ten years and some of the results are going into Perl 6 as the only standard way to build classes.

At the same time, Larry added a few more features that made closures quite simple in Perl. Real lexical variable and code references were the main changes. Almost ten years later, Java is having trouble with this concept because the everything is an object approach and the closure/anonymous function approach have a relatively large impedance mismatch. This is not to say that Java is wrong, just that the design doesn't accommodate this kind of change very well.

Ruby and Python also used flexibility in their language design to explore new features. However, neither of these languages has quite the unbridled flexibility of Perl. Although many people consider that a bad thing, they miss a very important point. This was an explicit design feature of Perl.

Turn it to 11

But, there are languages that are more flexible even than Perl. I spent a fair portion of my career working in the Forth language. Two of the most interesting things about Forth is it's lack of syntax and its ability to run arbitrary code at compile time. This allows a good Forth programmer to tailor the syntax of the language to the problem at hand. It also allows things that would be practically impossible in another language.

For example, Rick Hoselton and I added a relatively powerful and efficient exception handling mechanism into the Forth we were using at the time in a couple of days. At another point in time, some of us implemented a prototype-based object model in less than a week. We also added RAII-type resource management at another time. Because the language was so malleable, ordinary programmers could make these design changes without completely redesign the language.

The downside of this flexibility is also obvious. A bad (or overly clever) programmer can generate code that no one will ever be able to figure out. However, good programmers can extend the language in useful ways, exploring language features and changes in paradigms without needing large standards organizations or completely new languages.

Back to Mainstream

If I haven't lost you already, I should reassure you that I am not advocating that every language should be as free and adaptable as Forth or Perl.

But, I am suggesting that we look at our languages a bit differently. As we've seen with Java, tool support and standardized libraries can be much easier with more rigid languages. These languages also support the ability to do very effective work with groups having a range of levels of experience. (The leader of one project I worked on decided we would work in Java, not because of any technical superiority of the Java language, but because you can hire Java programmers straight out of school.) Many types of applications are very easy to build in Java because of the strong frameworks supporting those kinds of applications.

Many Python proponents argue that their approach of only one way to do it makes it easier for new people to learn the language. There are fewer nooks and crannies to get lost in. But Python is more flexible than Java. A few years ago, Guido made some backward-incompatible changes to enhance the object model based on lessons learned as people pushed the edges of that part of the design. Although it took a major change to the language to really improve the object model, it was people pushing the edges of what could be done that helped design the change.

Perl advocates point to the flexibility of the language and lack of rigid rules as some of the features that help us get work done. The TMTOWTDI slogan is more than a platitude. It is a recognition that the adaptability of the language gives us the ability to do both amazing and horrible things in our code.

What people should try to realize is that these different levels of flexibility are not good or bad, they are different. Usually, languages steal from one another when they are first created, the more flexible languages keep stealing ideas and adapting to new situations. Maybe the more orderly languages could benefit from exploring new techniques in a more flexible language before committing to another one true solution to the problem.

A Different Look at Years of Experience

I've been spending a fair amount of my time the past few years helping hire programmers. I have also spent time training both entry-level and senior programmers. So, Jeff Atwood's recent blog entry The Years of Experience Myth struck me as particularly interesting.

While I agree that the number of years of experience a developer has is mostly uncorrelated with their programming ability, I'm not sure I would go as far as Jeff did in suggesting that anything more than a year of experience is not worth considering. I think years of experience do count for something, we just can't use that measure in hiring decisions. As a very wise mentor of mine used to be fond of saying

There's a big difference between five years of experience and one year of experience five times.

There is no way to tell from a resume whether or not a person is in the first category or the second. Some people manage to get a year's experience under their belt and stop learning. They can continue to work on the same kinds of projects, applying the same kinds of techniques, without ever learning anything new. On the other hand, some people never stop learning. These are the ones that other programmers go to when they have a sticky problem that is holding up a project, or to help on that impossible bug.

Understanding vs. Mastery

One point where I specifically diverge from Jeff's post is in the suggestion that once a programmer has 6 months to a year of experience in a technology, they either get it or they don't. A specific example that does not match this is object oriented programming. Although it has become the dominant programming paradigm, it does take some work to really understand it. But, that's a topic for another day.

In every language or technology I've used in almost two decades of programming, there was a difference between learning to use the basic tools and actual mastery. Mastery takes time. Unfortunately, we cannot tell how well someone has mastered a topic by the number of years they have spent doing it.

Hiring

This leads back to Jeff's post. I agree that there is a problem with hiring practices that depend on the number of years that you have spent working on a particular technology. In the worst case, I have seen job postings that are looking for exactly the same experience as the person they are replacing. (You've seen these. They have 4 or 5 n years experience using technology Y entries. Some of the technologies are only used by two companies on the planet. etc.)

I'm not willing to write off years of experience. But, I do tend to use it in a different way. If I see someone whose resume states 5 years of experience in C++ or Java and after a phone screen, it appears that her understanding is closer to 1 year, I am quite skeptical about whether or not they will be able to learn. If someone claims 2 years of experience and has about that level of understanding from the phone screen, he is probably a good candidate. A few times, I have run across someone with a year of experience on their resume, but with a real understanding of the topic. We want this person, she is a motivated learner.

Job Postings

I did, however, like Jeff's point that a job posting that focused on a laundry list of years of experience in particular technologies was an indicator of a position I might not want to work. That should be taken with a small grain of salt, since the people doing the work may not be the same as the ones writing the job posting.

All in all, it was a thought-provoking post (at least to me).

The Show Me Response

This week, there was an interesting post on Jeff Atwood's Coding Horror blog. This essay was on Shuffling. Shuffling is an application of random numbers, which is a particular interest of mine. I will write more on that subject in an upcoming essay.

One of the comments caught my eye because it echoed a sentiment I have fought several times in my career. The commenter referred to a particular simple solution to the problem and said

Random removal from a mutable array is the same approach I used last time, and the same approach I'll use next time, unless someone shows me why it's wrong.

This answer really bothered me. It wasn't particularly about the solution the person chose. I also would prefer not to abuse a particular person. After all, depending on the domain, it might be a reasonable answer. My problem is with the assumption that it is someone else's job to prove that the simple (naive) solution is not the best one.

I often run into this problem with code written by domain experts. The idea that the naive solution they created in 30 seconds of thought is probably better than the one created by a computer science expert after a great deal of research is baffling to me. The fact that the documented solution is easy to implement and supplied by many libraries makes this comment even more amazing. Just because the computer scientist doesn't know about your domain is no reason to discount their knowledge of the computer science domain.

As a professional programmer, I feel that improving my knowledge of algorithms is part of the craft of writing code. When looking at a new problem, I often try to check my books and online references for better solutions, especially if the problem seems particularly general. After all, many of the general problems in computer science have been studied for 30 or 40 years. Many of the people doing that research are extremely bright. They were also probably pretty knowledgeable about the area of their own research. It is relatively safe to assume that they knew more about the subject than I do.

Granted, our field often involves trade-offs. Sometimes the well-studied general algorithm is not the best solution to the particular problem you are working on. But, without understanding the general algorithm you can't know that. I have often heard comments along the line of our problem is much more complicated than this toy research problem, so we'll use our solution. (As an interesting footnote, almost every place I've ever worked in about two decades of software development was convinced that they are solving harder problems than everyone else.)

If the person making the comment has carefully considered the algorithm in question and can point to specific issues with the general algorithm, that is one thing. We all make design trade-offs all the time. However, if you don't bother to look at the general algorithm and assume that your approach is good enough, you are missing an opportunity to learn and you are potentially generating worse software in the bargain.

I have repeatedly worked on code where someone is implemented a horrible naive algorithm without bothering to look at the implications. For example, a recent performance problem was caused by an O(n2) algorithm that was easily converted into an O(n) algorithm with a small amount of thought. Unfortunately, the original programmer could not be bothered to look for a better approach. After all, in the small test sets he had checked the naive solution was fine.

I don't think it is unreasonable to expect a professional programmer to improve his or her craft by studying classical solutions to known problems. I also feel that, as professionals, we should welcome the opportunity to improve our understanding and our tools when a problem we are studying turns out to have a standard, well-researched solution. Understanding the solution improves our ability to analyze algorithms in the future. Seeing how the experts build and analyze algorithms will help you improve your ability to do the same.

Deciding that it is someone else's job to prove that the standard solution is better than your quick thought is handing the job of improving your skills to someone else. As a professional programmer, I realize that knowledge is what allows me to do my job. Seeking knowledge and understanding is, therefore, the most important thing I can do to improve my skills as a programmer.

Although I have been called arrogant many times, even I am not so arrogant as to believe that I am smarter and know more than all of the people working in software over the last 40 years.

The IP Goose, revisited.

A couple of years ago, I wrote in The IP Goose about some effects that too strict an IP policy can have on developers. In the intervening two years, I've had some other insights that I believe are important.

First of all, I stand by my original essay. Companies should be able to protect the software they have hired people to write. They should have some protection against an individual taking the information they have learned to take business away from them. A carefully crafted IP agreement can do that.

In the previous essay, I also described how practice is needed to improve skills and increase knowledge. I still believe that development outside of work is needed to practice, learn new things, and keep unused skills from fading. Unfortunately, I've also come to believe that the effects of a draconian IP policy can be more damaging than I first thought.

Habit

One important part of practice is consistency. As long as you continue to practice anything, you tend to improve or, at least, maintain your skills. Obviously, if you stop practicing, the skills and knowledge begin to degrade. More importantly, you also start to lose the habit of practice. The longer you are not practicing, the more effort is needed to get back into the habit of practicing and the easier it is to backslide. This tends to make regaining the skill you are practicing even harder.

I'm sure many of you have seen this effect in different fields: martial arts, sports, music, cooking, etc.

Well, the time off from practice caused by a draconian IP agreement will have the same effect on your programming as an enforced absence from skiing or judo would have on your performance in those areas. Subtly, priorities have shifted. There's always something else that needs to be done. You haven't had time to practice for x months, what's another couple of days? It takes quite a while to overcome that inertia, and that whole time your skills continue to degrade.

Long Term Effects

I think that an overly strict IP agreement not only has a detrimental effect on our skills while you work under it, but also a long-term degradation in your ability to practice.

I worked for a while under a really strict IP agreement. Following the letter of the agreement meant that any software I touched would belong to the company. Working on Open Source projects was obviously out of the question. If I wanted to work on a project on my own, I had to contact the appropriate person in the company's legal department and obtain permission, in advance. If necessary, I had to get sign-off from my supervisor that the project was in no way related to my work with the company. Needless to say, quickie little projects to try out a new technology or technique were no longer worth the effort.

It has taken quite some time to regain the habit of working on projects on my own time. I was able to get back to the little stuff pretty quickly, but bigger project require a lot more effort to start and work on than they used to. Fortunately, I still enjoy writing software, so I have a strong incentive to keep working at it.

Objects, Reuse, and Complexity

Over the last few years, I have spent a lot of time talking to lots of different people about object-oriented programming (OOP). I have spent a fair portion of the last three years interviewing and screening people over the phone for development positions. This activity has caused me to spend some time re-evaluating what I know about objects.

Reuse?

Some of the people I talk to say that OOP is about reuse. They say the main thing we get out of objects is reuse. We can reuse the data and functionality from a class by deriving from it. We get more reusable code by packaging it up in classes. Unfortunately, reuse is not confined to OO. Back when I was doing structured programming, we captured reusable code in functions. We used libraries and modules for larger granularity reuse. Obviously, OO is not the only way to reuse code. So, it's kind of hard to claim that the main reason for OO is code reuse.

OO Concepts

So, if reuse is not the purpose for objects, what is? One of the maintenance benefits of OO has to do with the concept of encapsulation. If a class is defined reasonably, the only way to access its data is through its member functions (methods). The practical result of this is the amount of code that can change a piece of data is limited. In a large system, this drastically reduces the amount of code that must be examined to troubleshoot a data problem.

OO provides good support for abstraction. It is possible to make an abstract interface without OO, but it requires more discipline on the part of the programmer and the clients of the library. With classes, it is easier to specify an abstraction and encourage clients of your class to use it.

Complexity

These two concepts are different aspects of the same issue: controlling complexity. By reducing the number of methods that can touch a given piece of data, you are reducing the communication paths in your code. This reduces complexity by introducing constraints in the way data is accessed and modified. Abstraction also reduces complexity by encouraging the client programmer to focus on the concept of the class rather than on its implementation.

In some cases, the complexity isn't actually removed, but only quarantined inside the class. This helps keep complexity inside the class from leaking out, and complexity from the surrounding system from leaking into the class. In many cases, reducing the way complex things interact is the best tool we have for managing complexity in our systems.

By providing interfaces that hide implementation details behind some form of abstraction, we reduce the complexity that the programmer needs to be aware of at any given point in time.

Increasing Complexity

This also highlights one of the points where OO has failed in its promises. Without careful thought about your designs, an OO system can add to the complexity in a system. Several kinds of OO design decisions can add complexity to a system:

• deep hierarchies
• badly thought-out interfaces
• objects that are just collections of random data and methods
• large quantities of unrelated classes

Another dark side of objects is that hiding the complexity of the system allows us to develop even more complex software because we don't have to deal with all of the complexity all of the time. This has caused many people to ignore the complexity cost of their designs.

Conclusions

People who really understand that OO is about reducing complexity avoid adding one more method to a class to make it more reusable. Classes with only one responsibility abstract the implementation details while providing a lower complexity interface to the functionality. Combine two classes inappropriately and the complexity increases again, because you need to know more to use the two halves of this split-personality class.

Keeping the importance of reducing complexity in mind should help in the creation of cohesive classes. Using classes to provide coherent interfaces that hide complex implementation details should help to reduce the complexity in our software.

Form vs. Style

Ever since O'Reilly published the new book Beautiful Code, there seem to be references everywhere to what makes code beautiful. While a few people have suggested that beauty should not influence code (see the comments section for the suggestion that there is no beautiful code), I believe that most people have seen code that they thought was particularly well done, and might even be considered beautiful.

I suspect that most people focus on the wrong thing when then think of beautiful code. I would like to separate what most people consider beautiful into two parts: form and style. By form, I mean the fundamental design or structure of the code.

• Does it solve the problem at hand?
• Is it understandable?
• Is it as simple as possible, but no simpler?
• etc.

On the other hand, style refers more to issues that may be important, but are less fundamental.

• Does it follow the coding guidelines of the group?
• Is it written in the appropriate language?
• Is it a clever or cool solution?
• etc.

Many people focus too much on style at the expense of form. I think that is partly because style is easier to codify than form. Almost anyone can enforce a brace style or naming convention, but making or recognizing a simple solution is much harder. Programmers are also attracted to complexity and are often seduced by clever solutions.

The real downside is that people often try to fix a bad form by piling on more style. This never generates a beautiful solution, but it might obscure the extent of the bad form. Like the fins on a 1950's Cadillac, it looks cool now, but it won't look as good later.

Ugly Code

I don't usually write beautiful code. I do focus hard on the form, but I am rarely completely happy with my solutions. Despite this, I feel I have some advantage in this area. Early in my career, I was exposed to quite a bit of atrocious code. This code depended heavily on global variables, misused language idioms, and preserved old code mistakes as running comments in the code. It was hard to read and difficult to maintain.

This code formed the basis of my desire to improve, I never want to leave behind code like that. Over the years that followed, I tried to hone my skills at recognizing what I called good code.

Beautiful Code

At one point, I was working with a promising young programmer on a redesign of an important part of the system we were maintaining. We had spent time discussing the requirements and had mostly worked out the general shape of the solution we would use. At that point, I needed to leave for the day and we agreed to work on it some more the next day. I came in early the next morning and took a look at the code. She had laid out the beginnings of the code without filling in all of the details. Each major piece of functionality had its own well-named function. Higher level functionality was built in terms of the lower level pieces. Every function had a clear name that explained the function's responsibility. The general gist of the design was already in place with placeholders for anything that we had not had a chance to work out.

This was by far the most beautiful code I had ever seen. I actually dragged another senior programmer over to see this code. The style was not yet perfect, There were many levels of stuff that could have been added on top of this code. But, the form of the solution would have been obvious to anyone seeing the code. As we worked over the next few weeks, we did make some comprises that marred the initial beauty of the code and we layered on some style to make the result a bit prettier. In the end, the original beauty remained mostly intact and visible.

Evaluating Code

I've used that code as my yardstick ever since. I may not write beautiful code every time, but I have good examples of beautiful and ugly code in mind that serve as my guides. As I write code, I try to evaluate my code against those examples.

The Flaw of Everything is an Object

Several newer languages (as in less than 10 years old) have been designed around the idea that everything is an object. Since I started programming professionally almost two decades ago, I was programming before the object craze became mainstream. I have done the OO thing for over a decade and am not as impressed as I once was.

Although not everyone will agree with me, I suspect that OO is not always the best choice. In fact, in some applications, a simple, straight-forward procedural approach is better.

Many OO advocates will point out that the procedural approach doesn't scale. They tend to forget that a large fraction of the code written in the world doesn't need to deal with terabytes or petabytes of data. Much code isn't called thousands of times a second. Even more code is not mission critical. And some applications run on systems without the resources for the overhead of OO.

Learning OO

I also find that the object approach appears to be harder for people to really master. Many people claim to have learned OO programming in a few weeks, but it seems to take at least two years for people to really get it. Before that point, new OO programmers end up creating collections of data and methods with no cohesion or multiple classes with intimate knowledge of each other's inner workings. It usually even takes a few months before they stop inheriting everything just because they can. I won't even discuss the objects that serve only as a holder of a handful procedural programming methods that show no signs of abstraction.

On the other hand, I've seen non-programmers pick up a little bit of procedural programming relatively quickly. Just enough to automate some portion of their computer usage and save a little time. This is some of the appeal of many of the dynamic languages. It's easier to whip out a little code to get a job done. Would I recommend this kind of programming for life- or mission-critical applications? No, but that doesn't make it useless.

Not The One True Way

I suspect that to objects have become the latest Golden Hammer in our field. It has also been around long enough that people have worked in the OO paradigm long enough that they have begun to believe that there is nothing else.

As such, people tend to forget that the is overhead involved with using OO. I'm not just talking about memory and CPU overhead. While it is true those forms of overhead exist, both are getting cheaper fast enough that they are not the major problem (unless you are doing embedded systems). The more important overhead is conceptual. Everyone starts with the obvious (and incorrect premise) that programming objects are just like objects in the real world. This view of objects is easy to understand, but very limited. Amusingly enough, in the real world it is obvious that everything is not an object. Energy, thoughts, emotion, and probability are fundamentally different than a table or cup. Forcing them into the same framework would not make much sense.

Concepts

In most of the OO programming I've have done or seen only a tiny fraction of the classes have anything to do with real physical objects. I think the best breakthrough for me was the comment concepts are classes in the book Ruminations on C++ by Koenig and Moo. This finally solidified for me the non-real world things that need to classes. I had done this many times without really having a reason. Wrapping your mind around these concepts and learning to work with them seems to take months to years for every programmer I've ever worked with, taught, or mentored. Along the way, they normally reach several Aha! moments that move them toward real ability with OO.

This does not sound like the best approach for people who are just beginning to learn to program or for people who only need to program as a sideline to get their real work done.

Even in the most fanatical of OO programming languages, we eventually reach the level of individual statements. These are not objects, and making them objects would increase the difficulty in understanding. However, most people overlook that.

Language Feature?

I have come to believe that the everything is an object meme has nothing to do with ease of learning or use of the language. I suspect that the approach only simplifies things for the language designer. After all, as a user of the language, how does having an integer be an object really make your coding or design easier. In order to make the code usable, these integer objects either need special syntactic support in order for us to write mathematical expressions somewhat naturally, or the language needs support for operators built in to the classes. Both complicate the language for the (marginal) benefit of being able to say that all of my numbers are objects, just like everything else.

As a rule, I find this assertion to be a quirk of a given language rather than a particular bonus; kind of like Lisp's parenthesis, C++'s templates, or Perl's punctuation variables. They are part of the language and are only good or bad in as much as they help or hinder my ability to learn the language and read and write useful code.

Diets, Analogies, and Agile Programming

I enjoy reading the Worse Than Failure site as an antidote for programmer arrogance. While a junior programmer might look down on the people who made the reported mistakes, those of us who have been in the business for a while can recognize many of the mistakes. They still make a good laugh.

A few days back, the article The Great Pyramid of Agile surprised me with a bad analogy. I recommend that you read the article first, so that I don't accidentally misrepresent Alex's position. The article compares new programming methodologies to fad diets. Unfortunately, an analogy does not always show what you intend it to show. In this case, I feel the analogy is a clean miss.

Alex refers to the Agile methodologies as the new fads and declares that they all generally fail to deliver. He then goes on to prove that Agile doesn't work based on two arguments:

• a bad Agile analogy based on building pyramids
• a comment by evangelists for Agile methodologies

I agree with Alex that the pyramid analogy is ridiculous. But, a bad analogy does not mean that the thing being described is ridiculous. Unfortunately, when most of the anti-Agile people use building-based anologies they fail just as badly. Writing software really isn't much like building bridges or skyscrapers any more than it is like building pyramids.

His other argument is based on the fact that when an Agile project fails, the evangelists often say that it failed because there were not enough good people on the project. Whether true or not, this argument serves no purpose in proving that Agile is a good idea. In fact, when a traditional project fails, the programmers are normally blamed for not following the process. This argument also does not prove that the traditional process is a good idea.

A Tale of Two Diets

So let's go back to Alex's analogy with a twist. Say we have two people trying to lose weight: Bob and Carl. Bob starts the new French Fry fad diet. Carl starts a new program involving reducing his calorie intake, walking every morning, and going to the gym three times a week. Both Bob and Carl start off well and after a week they are enthusiastic and telling all of their friends. After a month, they are losing weight and going strong. After six months, the enthusiasm is wearing off and they are starting to backslide. After a year, they are both back to their old ways. In both cases, the evangelists for the method, say that the problem is that they didn't stick with the program, not that the program didn't work.

Based on most of the research, the reduced calorie and increased exercise approach will really have long term benefits, if you can maintain the discipline to stay with the program. Unfortunately, from this example, you cannot tell which is the real thing and which is the fad.

Back to Software

I see Agile programming practices in much the same way. Shortly after the Extreme Programming books started coming out, I started researching the process with a skeptical eye. I tried a pilot program working with XP and was impressed with the benefits we saw. I also noticed how hard it was to keep up with the practices, even when we could see tangible benefits.

Over time, the industry has seen many variations of Agile methodologies. What many people on both sides of the argument seem to miss is a side effect of the Agile methodologies. When practiced carefully, many of the Agile practices improve an individual's software development skills. This is the whole people over process thing.

There is a large body of evidence that suggests that the number one indicator of the success of a development process is the quality of the people working on it. If the Agile practices can really improve the quality of our developers, they may actually be on to something. The problem is developing software is like just about anything else, improving your ability requires discipline and hard work. Not everyone is willing to put in that kind of effort, no matter what we call the approach.

I think most people will agree that we have shown repeatedly in our industry that a new great, whizbang process will not turn a group of people who can barely type into a productive, professional development team. Despite this fact, many companies (and managers) seem to feel that if they just sprinkle on the right methodology, the people they have writing code will become a world-class development group. This is somewhat akin to suggesting that giving me the right checklist of steps to apply would make me into an Olympic athlete.

To be honest, I don't know if the Agile practices will have long term benefits. I have found some of practices to help improve my personal programming skills, at least in the short term. I have not seen enough benefit from some of the other practices to be able to keep doing them. That may be a failing of the practice or my own discipline. I find that I like the idea of improving my personal skills as a way of improving development on my projects. As a pretty good programmer with a fair number of years of experience, I would like to believe that improving my skills would be more effective than applying some magic methodology to a random group of people off the street.

Programming for Everyone

Can just anyone learn to program or is that skill limited to a few of us?

This question or some variant has been on my mind for most of my career. Early in my career, I thought programming required inherent talent, so only some people were capable of learning to program. As I gained more experience, I began to believe that programming was a skill that most people could master with sufficient incentive and desire. During the dot-com boom, I saw a large number of people get into programming because the salaries were good. After dealing with code written by some of these, I realized that (monetary) incentive and desire weren't enough. Some of these people couldn't program to save their lives, despite the fact that they were paid to do just that.

A related question is the inherent difficulty of programming. According to many people, programming is easy. After all, if you go to the bookstore, you will see books like Teach Yourself Programming with Java in 24 Hours or Teach Yourself Ruby in 21 Days. Some people look at the titles of books like these and ask themselves "How hard can it be?" I've seen many managers that seem to be of the opinion that they can hire any kid off the street and teach them to program.

On the other hand, many of us know that programming is harder than some would like to believe. (Why do we resist the idea that programming might be hard?). I've seen entry-level programmers struggle with basic concepts. I know from personal experience that most of the people I talk to have no grasp of what I do.

So here we have two different ways to look at the same issue:

• Is programming really hard or can anyone do it?
• Does programming require some inherent skill or can just anyone do it?

It seems that every time I come back to this issue, I have a different view based on my current experience. Recently, I stumbled across some research that suggests there may be an answer to the second question above. A summary of the research (Coding Horror: Separating Programming Sheep from Non-Programming Goats) gives you a taste of the issue. Papers from the actual researcher fills in the details. It seems that many people find certain basic programming concepts impossible to master.

Although that is a strong argument, more research is needed to settle the issue once and for all. But does that really mean that a large portion of the population is incapable of any level of programming?

An Analogy

To answer that, I'm going to do something dangerous and argue from analogy. When I wrote Programming and Writing I pointed out that there were interesting parallels between programming and writing. So, I'm going to use most people's understanding of writing to make a case for most people being able to program, after a fashion.

Most people that have been through a public or private education in a developed country learn the basics of reading and writing. This education doesn't work for everyone and there is quite a bit of debate about why. For the purposes of this analogy, I'm going to ignore those people. Even after school, most of us continue to use our reading skills for reading signs, documents, credits on movies, and such. But, what about writing? How many people could write a critical essay, or a short story, or a novel after they finish school? I would dare say that most of us could not. Some might harbor a great American novel somewhere in their skulls, but few ever really begin writing it.

I see most programming as much like the problem of writing that essay or novel. Both require knowledge, some inherent talent, and a degree of skill. And, like writing that essay or novel, not everyone has the necessary skill and talent to write a significant program. But, that's not the end of the story. Most of us still use writing to a small extent even though we'll never be a Shakespeare, or even a romance novelist. We write a quick note to tell a family member where we are going to be tonight. We write grocery lists. We do Christmas cards. We write letters or email to friends and family. This is still writing.

Is there an equivalent in programming? I believe there is. Unlike many computer users, I am a big fan of the command line. Point and click are fine for some things, but give me a command line and I'm much happier. The reason is a little discovery I made around two decades ago, working in DOS. I wrote my first batch file. I stuck 2 or 3 commands in a text file and was able to run them as one command. Now, I had been writing code for several years before that point. Even then, this wasn't a new concept. Heck, even I had seen it several times before. But at that point, I recognized a different kind of programming than I had been doing in C, Fortran, or Basic. This was making the computer work for me instead of the other way around.

Later, when I first worked on a Unix system, I found an entire culture and system built around this concept. But, that still didn't take away the basic cool part. I could build a little text file that would tell the computer to run multiple commands at once instead of having to type them myself. This is something I don't get from a GUI interface. I can't compose multiple commands together very easily.

Back to the question at hand. This little batch file or shell script approach is not real programming, any more than writing a grocery or todo list is real writing. But, I suspect that most everyone could learn to do this level of programming with a little help. With a little more help, I'm sure most people could do a bit more. They might never be able to write a full accounting system, or a word processor, or even a web application. But, given a little training they could probably teach the computer a few tricks.

The Right Tool for the Job

I ran across an unusual article a couple of months ago, Ethical Software by Alex Bunardzic » Right Tool for the Job is a Myth. My first thought was that the author must be out of his mind. After some reflection, I have a somewhat different view. Now I just think he is mistaken. I would definitely suggest reading his article and the associated comments so that you understand his viewpoint. I wouldn't want to misquote anyone.

The author maintains that the phrase the right tool for the job is a meaningless statement because in the software world there are not as many choices as in the real world. Honestly, I don't know what to say about that. Most of my programming career has been spent narrowing an infinite number of possible solutions down to a few that might possibly work. Since I can't get even a basic grip on his general statement, let's look at some of the specific examples.

TCP/IP

For example, when implementing a software system that is going to be distributed across a network of computers, it is pretty darn impossible to choose the right tool for the job — because there is only one tool to use, and that tool is TCP/IP.

This statement can be easily refuted by pointing out that UDP is still in wide use on the Internet. Several distributed systems depend on this alternative to TCP, including DNS, VoIP, and TFTP. Obviously, there are multiple choices for the right tool in the area of distributing systems. Moreover, if you are working with small devices that do not support a TCP stack, you might end up using something completely different like the protocols from the ZigBee specification.

Each of these communications protocols has different strengths and weaknesses. Part of what makes programming interesting is the ability to look at these various choices and decide which advantages you want and which trade-offs you can live with. To a small extent, Bunardzic is partially right. More and more devices are getting TCP stacks, so it is becoming a viable option more often than it once was. But, that does not mean that it is the only choice.

Relational Databases

What choices of tools do we have at our disposal if we wish to implement a relational database? No choices. There is only one tool — SQL (Structured Query Language).

The comments pick this one apart pretty well. The main argument is that the question is narrowly defined to specify basically one answer. If we change the question to one that a developer is more likely to encounter: How do we store the data we need for our application?, the answer is less clear. If you only have a handful of key/value pairs, a property file or INI file is probably a better choice than a relational database. If you need to store gigabytes of of floating point information, raw binary files in a directory may be the best solution. For other problems, a CSV file might be exactly what you need.

In most of these situations, SQL would not be an option, because the data is not relational.

Business Logic

The last example solves the Business Logic problem with Object Oriented Programming. Although OOP is the current default programming paradigm, that does not mean that a rules-based expert system might not be a better solution in some domains. I've seen a number of environments where a pipeline of simple procedural pieces did as good and clean a job as any OO design. In many cases, these alternatives will not be best, but that does not mean that they do not exist.

Missing the Point

The worst problem with this article is that it misses the point of most of the right tool for the job comments I've seen over the years. At least in my experience, this suggestion was normally used when someone was really misusing technologies. Here are some examples of decisions that might generate this kind of comment:

• Using an RDBMS to store a set of a dozen key/value pairs for configuration.
• Using a CSV file to store information on a thousand employees.
• Using a SOAP message to send a yes or no response.
• Using a shared directory on a file server to coordinate a dozen machines in a distributed network.
• Using FORTRAN 77 for an application doing extensive text parsing.
• Attempting to write device drivers in Java.
• Using almost any programming language to pick 5 random names out of a list of 100, one time only.

In almost any application, there is the possibility that someone will choose a suboptimal technology. In many of those cases, it is just a matter of ignorance of the alternatives. Education and experience are the only solutions.

I'd like to close this with a quote from Larry Wall that captures the essence of this problem.

Doing linear scans over an associative array is like trying to club someone to death with a loaded Uzi.

In Defence of Simplicity

Recently, Joel Spolsky wrote about Don Norman's article Simplicity Is Highly Overrated. Joel used this as a springboard to another talk of how simplicity doesn't matter in Joel on Software: Simplicity. He referenced and expanded on his views from Strategy Letter IV: Bloatware and the 80/20 Myth. As I've said before, I often find Joel's writing to be interesting and insightful, but sometimes I can't figure out what he is talking about. This is another of those times. You should read the articles referenced above before continuing, I don't want to misrepresent anyone's arguments,

I don't own a successful software business like Joel and I'm not a design guru like Norman, so my opinion may not mean as much as theirs. But, I have to disagree with the conclusions Joel draws in this articles.The first is that bloat isn't a problem. Although I do agree that you can waste time trying to slim down software unnecessarily, I don't think it is reasonable then to declare that reducing bloat is always a bad idea.

Any work on software involves tradeoffs. You need to determine what is the best use of the company's resources. Sometimes spending time optimizing software is not the best use of time. However, as the size of the code increases, there are other effects to consider:

* more code means more possibilities for error
* more code to read when trying to find and diagnose bugs
* more code means more work to train new people
* more chance for accidentally duplicated code

All of these items apply whether the extra code is due to more features or badly written code. If the extra code is due to features that your users want, the disadvantages are outweighed by the benefits (usually). However, if the code is due to features are truly not needed by your users or simply bad coding practices, there is no benefit to offset the disadvantages.

The developers aren't the only ones to see disadvantages in the extra code. The effects that the user sees are different, but no less annoying:

* program takes up more disk space
* program takes longer to load
* program takes longer to shut down (sometimes)
* program takes more memory

These effects can range from inconsequential, through annoying, to unusable. Different issues will be worse for different users or different usages of the program.

What is Bloat?

Joel seems to take the position that the extra code is always the result of a conscious decision to add code for functionality beneficial to the user. If that were the case, deciding that the extra features aren't bloat is probably reasonable. Unfortunately, not all extra code is the result of such careful consideration.

A lot of the bloat I've seen has been caused by inexperienced developers coding features inefficiently. I'm not talking about reducing the code size by using extremely cryptic twisted code. I'm talking about the case where someone is storing data in a list and then writing linear search code a dozen times to find items in that list. Or, maybe the case of the inexperienced programmer that writes his own sort algorithm when the library provides one already.

There is also the case where a developer adds a feature for free because it was easy or because he felt like it. If no one wants or needs this feature, it is bloat. As the code gets larger, there's also the phenomenon of partial implementations of features that get left in the code so I can finish it later. These things all add bloat to the code without adding any features for the user. As such, I believe that Joel or anyone would have to admit that this kind of bloat is bad.

Simplicity and User Choice

Now, I'd like to turn to Norman's argument. Based on some of his earlier work, I'm assuming that his article is not saying that simplicity is bad. Instead, what I think he means is that the consumer does not always appreciate simplicity. Often, this does appear to be the case. However, one of my favorite Bruce Schneier quotes is

The user's going to pick dancing pigs over security every time.

People sometimes choose more features even when they don't need them, and won't use them. People also chose sweets instead of healthy foods and sitting on the couch watching mindless television to exercise. This does not mean that sweets and watching television are better than the alternatives, it just means that people do not necessarily make the best choices. This is meant more as an observation than as a judgement. I suspect that Norman's article is the same kind of observation.

Simple Successes

My final point involves actual examples of simplicity that people do choose. The first is everyone's poster child for simplicity, the iPod. I don't believe that the iPod shows that removing features makes a successful product. I'm pretty sure that Apple did not start out with a jukebox-size product and start removing things to make a jukebox lite. Instead, they appear to have carefully considered what features people really want and use in a portable music player. Anything that was not essential, they removed. They also focused on making it attractive both physically and with careful marketing. The result is a success. Note that this was the result of real engineering tradeoffs, not just slashing features.

The second example would probably annoy Joel, given his fondness for using Mozilla as a bad example. After years of work, the Mozilla Suite was a clean, powerful implementation of a full-featured Internet client. The browser was coming along well and the people who used it, liked it. Then came Firefox (as Phoenix). This browser was the result only including the parts of the Mozilla client that was absolutely essential to browsing the web. The result was much simpler than the full Mozilla Suite, and people liked it. The development team then began to add functionality that enhanced the browser, without branching out into the other parts of the Internet client space. In the process, Firefox has gone from no market share in November 2004 to around 12% in late 2006. In Europe, the adoption rate appears to be higher. Given that most computers do not come with Firefox installed, anyone who uses it has to install it for themselves. The people that are using Firefox have explicitly chosen to do so, unlike the larger percentage of people using IE.

Once again, this is not the result of randomly removing features that most people don't use, Instead, Firefox seems to be the result of a strong focus on what is actually needed in a browser to improve the user experience. If a feature does not meet these criteria, it appears to be left out.

Conclusion

In my view, simplicity is a desirable feature of a product and bloat is an undesirable (mis)feature. However, simplicity is not the only criterion by which a product should be judged. The development of any product involves tradeoffs, making the wrong tradeoffs usually causes the product to fail. In general, competing with a feature-rich product with a stripped-down product usually won't succeed. Under the right circumstances, simplicity, elegance, good design, and marketing can complement a deficit of features.

Programming for Other Programmers

One of the odd things about programming is that you are writing for two different audiences. One is the dumbest device ever invented (it does exactly what you tell it to do) and the other is a (hopefully) intelligent programmer who will need to read this code later. We spend a fair amount of time figuring out how to make the computer understand what we write. I'm not so sure that we spend as much time on the other part. In fact, the only advice I have ever seen for writing for the next programmer is the admonishment not to write advanced code because later programmers won't understand it.

I think it is time to challenge that advice. I'm not going to advise that we don't take the other programmer into account when writing code. (After all, you may be that programmer someday.) I challenge the fact that we should write code to the lowest common denominator. Code should be written to a level appropriate to the job at hand. For example, third party library code is often written using more advanced idioms. This often results in more power per line of code. Since it will not be modified by novice programmers, there is no need to make it overly simple. Code that is expected to be used and maintained by entry-level programmers should be written using much simpler idioms.

If we don't allow more advanced idioms in a code base, the code never progresses beyond what would be written be an entry-level programmer. This will make retaining programmers who want to improve their skills harder. No one wants to spend their entire career maintaining entry-level code. If you are forced only to use entry-level idioms, that also makes it harder to keep up with advances in the the state of the art. After all, what's the point of learning cool new approaches, if you will never get a chance to use them.

Just about every advanced technique I've ever seen has been avoided by people because it is too hard (objects, exceptions, templates, operator overloading, STL, table-driven code, patterns, etc.). When the technique is new, that is probably a valid comment. But, as the community develops experience with the technique, we discover both its weak points and strong points. When the strengths outweigh the weaknesses, we should really begin developing our own experience with a technique.

A well-written piece of advanced code can also convey more information in a small space. Some really good examples of this are the algorithms from the C++ standard library. Which would you rather maintain in code:

    if(vec.end() != std::find( vec.begin(), vec.end(), 42 ))
    {
        std::out << "Found" << std::endl;
    }

    for(std::vector::iterator it = vec.begin();
         it != vec.end();
         ++it)
    {
        if(42 == *it)
        {
            std::out << "Found" << std::endl;
            break;
        }
    }

    for(int i = 0;i < vec.size();++i)
    {
        if(42 == vec[i])
        {
            std::out << "Found" << std::endl;
            break;
        }
    }

Although the last is easiest to understand for someone who has never seen the STL algorithms, You still need to read quite a bit of code to figure out what it is doing. The condition is buried inside the loop. There are also several places where a coding error is possible. Accessing the element is slower (because we have to index and dereference each time). The second drops the speed penalty for indexing, but is even more complex to read. Once you get used to the first approach, it is smaller, faster, and harder to get wrong.

Should we avoid the first approach because an entry-level programmer might not understand it?

Some people will think this is a silly example. After all, the first is officially supported by the C++ standard and STL algorithms are no longer considered advanced by most C++ programmers. But I still remember when this was considered beyond the cutting edge. Following a mandate to not use this approach would result in doubling the size of this code and requiring more reading for everyone who maintains the code from now on.

Although I am advocating the use of advanced techniques, I am not suggesting that we use them indiscriminately. If the argument that entry-level programmers won't understand it isn't enough to prevent the use of a technique, the argument that it is new or advanced isn't enough to make you use it. Every new technique requires some learning time. A technique or idiom has to prove that it gives enough benefit to overcome this learning cost. The important point that many people seem to forget is that the more of these techniques you learn, the easier it becomes to learn the next one.

Operator Overloading and Naming

One of the things Java programmers seem to really like to hate in C++ is the concept of operator overloading. I have heard this from many Java programmers and seen it written by others. According to many, operator overloading makes C++ impossible to understand because you can't tell what a piece of code is doing just by looking at it.

When looking for this statement on-line, the first example I found was Joel Spolsky's article, Making Wrong Code Look Wrong - Joel on Software under the section A General Rule. I'll pick on Joel for two reasons here. The first is that the comment has very little to do with the topic of an extremely good article. Second, that I usually agree with Joel on the big things, but often disagree on smaller issues.

Joel uses the following example to prove his point:
i = j * 5;

He states, with complete conviction, that if you see that same snippet of code in C++, you don’t know anything. Nothing.

I believe that the operator overloading issue is a red herring. To prove this, I will transform the code a little.

assign( i, multiply( j, 5 ) );

This would be perfectly legal code in C++, Java, or a number of other common general purpose languages. I could easily define this code to do exactly the same as the original code when i and j are integers.

Now, when you see this snippet of code, what do you know? Even without operator overloading, you don't really know anything. If the programmer used good naming conventions, however, you could infer that j is multiplied by 5 where the multiplication is something appropriate for an object of j's type. You could further infer that the result of this operation is assigned to i using whatever conversions are appropriate for this assignment.

Furthermore, if we used a more object oriented syntax:

i.assign( j.multiply( 5 ) );

Although we now know that both i and j are object types, we still don't know any more than we did from the previous two examples. But, we still don't know what those methods do. For example, multiply could make a call to a relational database with the supplied parameter to give a string that i.assign() uses to make an HTTP request to a web server to update its current state.

If you found that this was the case, you would probably abuse the original programmer about using extremely bad naming. You probably wouldn't ban the use of the method names multiply and assign.

The issue, as I see it, is not that operator overloading is a problem. It's that some programmers name methods, variables, and/or classes badly. The fix is not to ban certain names, but to educate those programmers.

In my experience, most of the bad uses of operator overloading happen when programmers are new to the idea and they think it will be clever or cool to use an operator when a method name works better. Education and experience usually fixes that. (Others may have had different experiences, but that is what I saw.)

Maybe we should stop beating the operator overloading dead horse, and spend more time educating junior programmers about the importance of good names and the Principle of Least Surprise. In fact, the advice cited in Joel's article above is a good start for this kind of education.

The Importance of the Correct Notation

Several of the past few essays have focused on the notation aspects of programming languages. The major benefit of a notation is making what you want to say concise without losing it's power in the process. To use an example from an earlier essay, supporting the use of normal algebraic notation for working with matrices allows the programmer to work at a high level, without spending too much time on the details of the operations. Unfortunately, this conciseness of expression is both an advantage and a disadvantage of any particular notation. If the reader is not familiar with the notation, they can easily be confused by what is happening. As I said earlier, a powerful notation is useful for the expert, but harder for the novice.

Another interesting issue with notations is that the best notations are limited in scope. Trying to use a notation outside its scope leads to more confusion. A good bad example for me is the use of + by some languages to concatenate two strings. This notation seems relatively intuitive for most programmers and works rather well for many languages. Unfortunately, in some scripting languages, this notation causes a problem. I first ran into this in ECMAscript. When adding a number to a string that contains a number, what should be the outcome?

var str = "10";
alert( str + 0 );

Should the output be "100" or "10"? Since the + notation is used in two overlapping contexts, you can see a real cause for confusion. This problem would have been solved by using a different notation for concatenation. Then, this notation would interpret the string as the number 10 and add 0 to it.

Obviously, using the same notation for adding two integers and two whole numbers simplifies the use of the language. But, often using a different notation for different concepts makes the language clearer. By the same token, having different sets of notations for operations that are different is very handy. If the operation isn't in some way the same the notation should look different.

In the early '90s, C++ was becoming popular in the PC world. Borland C++ was a popular compiler in this market. Borland produced a set of container classes that used many features of C++ that were considered advanced at the time. Unfortunately, like most of us, the compiler writers did not have much experience with these features and they made some really bad notational mistakes. One of them was to use operator overloading to add items to their containers.

This turned out to be confusing because not everyone agreed what was meant by adding an item to the container. Did it add at the front, the back, or somewhere else? When using + to add to the container, the operator modified it's left hand argument. This is not how + works for integers. In a later version of the compiler, Borland did a very brave thing by completely changing the interface based on these experiences. They realized that they were using the wrong notation, and broke a significant amount of code by changing the interface to one that was more effective.

This is a good example of a bad notation making a feature harder to use than necessary. (I know some people will suggest that this proves operator overloading is a bad thing. I'll come back to that subject in another essay.) By changing the notation to be more precise, Borland made the next version of their containers easier to use correctly.

The important point here is that notations are like interfaces. Good ones make saying what you want easier and more correct. Bad ones are easy to misuse. More importantly, in both interfaces and notations, what is good or bad is determined partially by the context in which it will be used. In the + example above, the problem with using + for string concatenation was that ECMAscript also converts strings to numbers and back automatically. This generates an ambiguity in the notation. This ambiguity is caused by the context of the operation. In the containers example, replacing + with insert and append (I don't remember the exact method names, but they were something like this), allowed more precision at the cost of somewhat more verbosity.

The important point is to use the right notation for the job.

Notation vs. Paradigm

Recently, I've been taken with the idea of programming languages as notation. When most people look at a programming language, they see syntax and possibly an underlying paradigm. I see the same, I just find the idea that each language also provides a unique notation for expressing ideas quite appealing.

Lately, I've been thinking about how notation relates to programming paradigms. In order to explore this topic, I need to begin with some definitions. Most programmers are familiar with one or two programming paradigms: the object paradigm, the functional paradigm, the generic programming paradigm, and so on. But, if asked, most could not explain what a paradigm is. The third definition of paradigm on Dictionary.com seems to fit our needs the best. This way of viewing reality definition definitely fits the way programmers apply programming paradigms.

The term notation is not used as much in our industry, so I would suggest the second set of definitions from Dictionary.com as being closest to what I mean: a technical system of symbols used to represent special things.

The key difference between these two is that a paradigm defines how you think about problems and solutions, and a notation defines how you write about problems and solutions. This leads to the obvious (to a programmer) question, are these two concepts completely independent. It's probably safe to say that the answer is no. (The dot notation used by many languages to make method calls is not very useful in a non-object oriented language.)

The next obvious question is how separate are they. Most of us a familiar with at least a couple of different object-oriented languages, since that is pretty much the dominant programming paradigm right now. Many of the differences between these languages are not about paradigm, but about notation. We often argue about:

• "." vs. "->"
• pointers vs. references
• null vs. 0
• garbage collection vs. RAII
• messages vs. methods
• interfaces vs. abstract base classes
• inheritance-based polymorphism vs. duck typing
• static typing vs. dynamic typing

The interesting thing is that in most cases, replacing one of these with the other would not change your ability to program with the language. None of them have anything to do with the paradigm. (Although I've heard people in some groups claim that a language can't be object oriented without: garbage collection, exceptions, message passing, or strong typing.) In reality, you can do object design and programming in assembly language if you choose. The problem is that the notation of assembly language is not well suited to the paradigm.

The key to a good notation is that it simplifies writing about your problems and solutions. As such, it may be worth suggesting that different notations may be useful for different problem domains and kinds of solutions. I also think that some people are more comfortable with some kinds of notation than others. Although we try not to admit it, often the choice of notation is more a point of personal preference (or habit) than of technical merit.

This is not to say that all notations are equal, some notations are more powerful or clearer than others. Even in the procedural paradigm, C is usually considered clearer than assembly. A really good example of the differences in notation is the implementation of a 2D matrix. In a language that supports matrices natively, you can add two matrices like this:

C = A + B

and multiply two matrices like this:

D = A * B

In some OO languages, you can build a matrix class and (with operator overloading) get an equivalent notation.

C = A + B
D = A * B

If the language doesn't support operator overloading, you might end up with a notation that looks like this:

C = A.add( B )
D = A.multiply( B )

Without some kind of support for user-defined types, the notation gets increasingly messy.

Now, if you do a lot of work with 2D matrices, the first notation is going to be important to you. It requires less typing, it is clear, and it's mostly what we all learned in school. If you haven't had need of 2D matrices since your last math class, the difference in notation will not be important.

This leads back to an important point. Notation determines how you write your solutions. It is possible to write object-oriented code in C. Unfortunately, the notation available in C does not support OO comfortably. The development of C++ (and Objective C and Java) made OO more popular by supplying the notation needed to support this paradigm.

The competing concepts of expressiveness and clarity are part of what defines a notation. Any time you are confronted with a new notation, the clarity point is driven home. A new notation is more difficult for you to use than one you know. On the other hand, as you become more comfortable with a notation, the expressiveness concept becomes more important. A good notation must be clear enough that new people can learn to use it, yet expressive enough to solve real problems.

Different languages make these tradeoffs in different ways. Just as importantly, different people have different tolerances for where they will accept the tradeoff. Some people prefer expressiveness in a language. They take the view that they will only be novices for a short while compared to the time they use the notation. These programmers are willing to trade a steeper learning curve (less clarity) for the power a more expressive notation gives. At the other extreme are people who prefer clarity over expressiveness. These people assume that most people will not be able to handle a very expressive notation. To this group, a simpler, clearer notation is more important than expressiveness because more people can understand it more readily.

Notice that this concept is independent of paradigm. Each notation in each paradigm makes these (and other) tradeoffs differently, based on the experience and biases of that language's creators. So, the next time you find yourself in a fight over which language is more object-oriented, you might want to consider if you are arguing paradigm or notation. It may not stop the holy war, but this viewpoint might at least confuse your adversaries.(<grin/>).

Libraries vs. DSLs

Many years ago, I was working on application software that we configured and shipped to nearly a hundred clients. Although we did the first installation on a machine that we provided, the customer had to install updates a few times a year. (This was before most people knew about the Internet.)

A few of us tried to convince the president of the company that we could make good use of a professional installation program. But, he could not see the point. He pointed out that we were already working in C, and C could do everything that the installer could do. So he concluded we could just write our own. He could not see the use of a specialized language for installation, what we would now call a Domain Specific Language (DSL).

Amusingly enough, we were already using a DSL to build all of the software. This guy was a big fan of make. We used a reasonably complex build system based around make to generate over twenty different variants of the program.

His question is actually a valid one, although at the time I didn't have the terminology to make my point. Everything I can do with a DSL, I can do in a general purpose language. In fact, I can package up the functionality for the DSL in libraries or classes and even make calling it almost as easy as using a DSL. So, why would a DSL be a advantage? I believe the answer is notation. The DSL provides no increase in functionality. But what it does do is provide a better notation for working in the domain.

Let's take make as an example. Even though almost everyone complains about the tab thing in Makefiles and the reliance on the shell, make is still in wide use. Anything you can do with make can be done with any general purpose language. It would even be possible to build libraries for common functionality needed by build tools. Yet, we still use make or other tools similar to them. Why? Despite its flaws, the make DSL is very effective for specifying dependencies and tasks needed to build one set of resources from another set of resources.

SQL is another good example of a DSL, I remember accessing databases before SQL was wide-spread. Every database API was slightly different. They all provided the same basic functionality, but there were differences. You also needed to write quite a bit of common code in every place you used a particular database's API. Then SQL came along, and things changed. No, it was not a complete language, but it did allow you to talk to the database at a higher level. In the beginning, SQL libraries were not any faster (and were often slower) than the native APIs. Nevertheless, the notation was more convenient for the kinds of operations that you need to perform on a relational database.

In many domains, a library is enough to allow you to get your work done. When the domain is understood well enough that we can really talk about it at a high level, it becomes possible to make a DSL. There seems to be a large difference in usability when a DSL is designed correctly. Even if a DSL is not perfect (like make), it may be enough better than the alternatives that still use it...even if we still reserve the right to complain.

Resource Management Revisited

A couple of years ago, I wrote a set of three essays on Resource Management: The Semantics of Garbage Collection, More Thoughts on Resource Recovery, and Other Resource Recovery Approaches.

In summary, my argument in these articles was that the standard mark-and-sweep garbage collection falls down for two major reasons, it doesn't deal with any resources other than memory and it fails to account for object lifetime issues.

Recently, a comment showed up on the Artima forums that shows that I'm not the only one to wonder about these issues: Java Community News - When GC is Not Enough: Resource Management in Java. This comment was inspired by the article Java theory and practice: Good housekeeping practices. Both the article and the comment discuss handling resources other than memory and how important it is to be careful in your resource management.

I find amusing the advice to be careful with memory management in a context where garbage collection is used to remove the need for memory management. Aren't these two issues essentially the same thing? I can't count the number of times I've been told that garbage collection is required because ordinary programmers can't be trusted to manage memory correctly. (An argument that I don't particularly agree with despite examples proving the point.) If the argument holds for memory, why doesn't it hold for file handles and database connections? I would argue that it doesn't really hold for memory either.

I don't believe that garbage collection solves the memory management problem any more than it solves the resource management problem. I think that programmers should be accountable for managing both memory and resources. While we don't want to do this manually, giving up the ability and responsibility completely is not the answer. A better solution would be to have tools that support resource management without giving away the ability to manage them.

One of the most useful idioms in C++ is Resource Acquisition is Initialization (RAII). The key concept is tying the ownership of each resource to an object and using the object lifetime to determine when the resource is in use. Unfortunately, the normal mark-and-sweep garbage collector implementation discards the concept of defined object lifetime, so we don't have that ability in a language depending on that form of GC. In fact, Goetz refers to that issue when he talks about the problems with finalizers in the Good Housekeeping article cited above.

Unfortunately, the only solution in a language without defined object lifetimes is to explicitly manage the resources everywhere they are used. Both Goetz and one of the comments in the forum point out that using a try - finally allows you to force this cleanup to happen correctly. Unfortunately, what neither of them mention is that this violates the DRY principle (Don't Repeat Yourself). Now, everywhere you use a particular resource you will need to copy this identical piece of code. This repetition is almost guaranteed to be a source of bugs.

While I agree that memory management in a language without garbage collection is harder than it needs to be, I still maintain that classic garbage collection is not the solution. Unfortunately, I suspect that the advocates of mark-and-sweep will either continue to ignore the issue or try to force some unnatural connection between other resources and memory. This way they can apply their favorite hammer to the job.

Domain Specific Languages, a Renewed Interest

I've seen quite a bit of interest in Domain Specific Languages (DSLs) on the Internet lately. Some good examples include Martin Fowler's exploration of the subject:

* MF Bliki: DomainSpecificLanguage
* Language Workbenches: The Killer-App for Domain Specific Languages?

He does point out that this is not a new idea. He uses Unix as an example of a system that uses a large number of DSLs. The subject has gotten enough interest to reach the point where people are discussing when it is a good approach to apply (Artima Developer Spotlight Forum - How and When to Develop Domain-Specific Languages?). Others are beginning to apply the term DSL to extend their area of interest (Agile Development with Domain Specific Languages).

So, from this we can guess that interest in DSLs is on the rise. As Fowler pointed out, Unix has been a nexus for the creation of DSLs, including:

* make
* regular expressions
* awk
* sed
* yacc
* lex
* dot

and many more.

Recently, I have seen the suggestion that extending or modifying a general purpose language is a powerful way to build a useful DSL. To some extent, this is also well-trodden ground. The classic example of this approach was implemented using preprocessors to provide facilities not present in the original language, both Ratfor (for structured programming in FORTRAN) and cfront (for object oriented programming in C) used this approach.

A recent article, Creating DSLs with Ruby, discusses how the features of the Ruby language make it well suited to building DSLs without a separate preprocessing step. The Ruby language apparently supports features that supports creating simple DSL syntax that is still legal Ruby code.

This is a very powerful technique that is not very easy to do with most languages. Amusingly enough, this technique is also not very new. In fact, there was a general purpose programming language that was designed around the concept of writing a domain language: Forth. If I remember correctly, Charles Moore once described programming in Forth as writing a vocabulary for the problem domain, defining all of the words necessary to describe the answer, and then writing down the answer.

The Forth language is different than most programming languages you might have encountered because it has almost no syntax. What looks like syntax is actually a powerful technique for simple parsing, combined with the ability to execute code at compile time. This allows for extending the capabilities of the language in a very powerful way. One interesting effect of this feature is that many good Forth programmers naturally gravitate toward the DSL approach when solving problems above a certain level of complexity. We firmly believe that some problems are best served by a language, not an arcane set of configuration options.

Forth does give us an important insight into the problems with DSLs, as well. There is is a well-known joke among Forth programmers:

If you've seen one Forth...you've seen one Forth.

Unlike more traditional programming, Forth programs are built by extending the language. A new programmer trying to learn a Forth system needs to learn the new dialect including the extensions used in this environment. This is not technically much different than learning all of the libraries and utility classes used in a more traditional system, but there is a conceptual difference. In Forth (or a DSL-based system), there is no syntactic difference between the extensions and the base language. The language itself can be extended without an obvious cue in the code to say when you have changed languages. This means that a new programmer may not recognize a new piece to learn as readily as when seeing an obvious library call.

This becomes a very important tradeoff: Which is more important, ease of learning for new programmers or power for advanced users? A well-designed DSL gives the advanced user a succinct notation to use to express hie or her requirements concisely and precisely. This is the appeal of the DSL. The downside is that this represents a new set of knowledge for each programmer relating to troubleshooting and debugging. It also requires more care in the design to develop a consistent and usable notation.

As usual, the tradeoff is the key. We need to be able to decide if the benefits outweigh the disadvantages and build the system accordingly. I wish there was a magic formula that could be applied to tell how or when a DSL would improve a system. Unfortunately, I have not seen a sign of such a formula yet.

Programming and Writing

Over the years, I have come to the amusing realization that many people feel like they could program if they just tried. You know the type. He (or she) has used computers for a while. He picked up Computer Programming for Total Morons or Object Oriented Programming in C++ and Java in an Hour and a Half and now expects to be able to write code like a pro. Those of us who have been programming for a while, brace ourselves for the inevitable opinions and declarations about building large systems based on experience with single-page programs.

I have also realized that there is another field that is very familiar with this problem, writing. Similar to the way that many people seem to think that they can program just because they can enter code in an editor, many people also seem to think they can write just because they can put words on a page. Ask anyone who really writes for a living and they can tell you how little the ability to write words on a page has to do with actually writing. Despite this disparity, anecdotal evidence suggests that many people still expect to write a great novel someday. (Although, I suspect that percentage is smaller than it was a few decades ago.)

A couple of years ago, I compared writing and programming in The Recurring Myth of Programmer-less Programming. I made a very superficial connection between these two disciplines in order to make a very different point. Since then, this connection has been simmering in the back of my head and I have come to realize that they have more in common than I originally thought.

One thing these two disciplines have in common is communication. Now, I realize this is not a particularly earth-shattering revelation, but bear with me a bit. It often comes as a surprise to most people that communication is hard. If you are just talking with your friends, family, and co-workers, you probably manage to be understood most of the time. When misunderstandings do arise, most people seem to assume that someone wasn't paying attention or something.

The first time you try to communicate with someone from a different culture, you begin to see how hard communication can be. Even if you can speak a common language, misunderstandings abound. If you are communicating through a low-bandwidth channel like email, communication becomes even harder. Most people seem to assume that these kinds of communication problems are the other guy's fault. Really communicating doesn't allow for that particular cop-out. Cicero is supposed to have said

The aim of writing is not simply to be understood, but to make it impossible to be misunderstood.

This is much harder than most people realize. When you are speaking, you can watch your audience and clear up misunderstandings as they arise (at least in theory). When you are writing, you don't really know who your audience is. There is no feedback, so misunderstandings linger.

In programming, this is equally hard because your writing has two very different audiences: the computer and other programmers. The difficulty comes from the fact that the computer needs everything described in mind-numbing detail. It cannot make the leap from one concept to another. On the other hand, people (even programmers) are not very good at reading information at that level of detail. This difficulty of working at two very different levels at once is one of the things that separates an actual programmer from a coder who dumps code into an IDE or editor.

Thinking and communicating at these two very different levels is extremely difficult. Over the years, we have developed better programming languages to reduce the amount of detail we need to explicitly write for the computer. Unfortunately, this has not made the complexity go away. Instead, the programmer has to be aware of the details of each statement and the context in which it is made. While this is somewhat easier for humans than the alternative, it does not mean that the details have gone away. It just means that you don't have to think about all of them constantly.

A good programmer has to be able to think at multiple levels of detail at the same time. These levels include:

• user-level details of the solution, such as the desire to select items by name
• high-level details of the program, such as the need to find items quickly by name
• medium-level details that the program needs, such as data structures and algorithms appropriate to the job at hand
• low-level details that the machine requires, such as limits on data types, memory, and speed of the processor.

Not all of these details matter all of the time. But a good programmer has to be aware of them and know when particular details can be ignored. The programmer also has to be aware of the constraints and requirements of the problem he (or she) is trying to solve.

A good programmer is aware of multiple solutions to many different problems and is capable of making various trade-offs in choosing between them. Whereas a coder has probably found a solution that has worked in the past and tries to apply it to every problem.

Fortunately, some coders eventually gain enough experience to become actual programmers. Unfortunately, the percentage of coders that become programmers is probably close to the percentage of people that actually do write their great novel. Unfortunately, unlike potential writers, these coders are actually producing code that someone will need to maintain and repair.

Now, sooner or later someone is going to point out that here I am writing, when I have no business claiming to be a writer. Actually, I don't claim to be a writer. I think my best credentials for writing are that I know I'm pretty bad at it. For that reason, I work pretty hard at each article I write, trying very hard to get my point across without leaving too much probability of being misunderstood. As such, this article has been one of the hardest for me to write.

Thoughts on Measuring Proficiency

In On Proficiency, I talked a bit about what characteristics make up the concept we call proficiency. In trying to hire programmers, a company often tries to measure proficiency to find the best candidate for the job. Over a year ago, I began thinking about how to measure proficiency, and found it to be harder than I thought.

Unfortunately, proficiency is not a simple variable that can be measured. A particular person is likely to rate differently on each of the four aspects of proficiency. Some will have great theoretical knowledge, but no skills. Others will have skills and aptitude to spare, but lack the theoretical knowledge to be efficient.

Most of these aspects are a spectrum. A person may have a strong aptitude or none at all, someone else may be in between. A person may have a little interest, a lot of interest, or would rather have a root canal than even think about it. These aspects are more or less outside a person's control and are easy to recognize if you see the person working in the field. Someone with no aptitude will not be able to make the simplest things work. Someone with interest and a great deal of aptitude will be able to adapt to new requirements or approaches fairly quickly.

The other two aspects are also not simple variables. A person's knowledge of a field may be broad and shallow, deep and narrow, or somewhere in between. The skills a person brings to the job may involve heavy experience with a few tools and techniques, light experience with a large number of tools and techniques, or some combination. Although, these aspects are the easiest to measure, they are also very easy to misinterpret.

Many employers look for the deep and narrow knowledge, asking for people that have spent the last 10 years doing essentially the same work that the employer needs. On the other hand, someone with a broader knowledge of the field can bring insights and new ways of approaching problems. If the person's knowledge is too broad, they probably have not had time to develop any depth in a particular area. However, someone who has knowledge of several fields has probably shown the ability to adapt and learn as needed to solve problems. This is an important characteristic. It is possibly more important than extremely deep knowledge of the topic of the day.

A person's skill set is relatively easy to measure. Can the candidate do task X with technology Y? Has the person ever used technique Z? However, the absolute answers to these questions are less important than many people assume. If a person has never used a particular skill, but is experienced with a related skill, the lack may not be a problem. A practical example would be does a programmer know a particular language. If the answer is yes, then you know he/she can use the language. If the answer is no and they've only used one language, then you know they won't be immediately useful. If, however, the programmer has experience working in several languages, the actual yes or no answer is less important. He/she will probably be able to pick up the syntax of the language and work almost as effectively as the person who answered yes. More importantly, if the language of the day sinks beneath the waves and a new language is required, this programmer will be able to adapt.

What makes measuring these even harder is that the different aspects interact in interesting ways. We all know the programmer that has an amazing amount of theoretical knowledge, seems to know every aspect of the tools he/she uses, and is extremely interested in the topic, but always seem to solve the same problems, over and over again. In this case, several attributes combined in a way that is detrimental to the programmer involved.

Many of us have also seen the bright entry-level guy/gal that has interest and aptitude galore, who spends much of his time reinventing known solutions to known problems because he lacks the theoretical knowledge that would prevent this waste of time.

Sometimes we've also seen the one that doesn't know the language you are working with, hasn't used any of your tools, but after a month is the person everyone goes to for help with design or style or even code.

After over a year of pounding on this idea, I still don't feel that I have any insights into how to really measure proficiency in our field, except work with a person long enough to recognize some of his/her strengths and weaknesses. I've seen many people hired that I thought would be a good fit and find them lacking in one or more of the attributes we need. I've also seen some that looked like a complete waste of time, that turned out to be just what we need.

If anyone has found any wonderful insight into this issue, I'd really love to hear it.

On Proficiency

In order to become proficient at any activity, a person needs four things:

• aptitude
• interest
• knowledge
• skills

To some extent, you should be able to judge a person's proficiency by measuring each of the four attributes.

The first of these is the only one that a person has no control over. If you don't have the aptitude (or talent or knack) for an activity, no amount of the other attributes will make you truly proficient. For example, very few programmers have the physical aptitude to play professional sports. No amount of interest or training will make one of us into a linebacker for the NFL. In most cases this isn't a problem, because most people that have no aptitude for an activity either have no interest in it or lose interest in it after repeated failure. This is not a bad thing because different people have different aptitudes.

Interest is the quality that usually causes an aptitude to be noticed. If you have a true talent for music, but no interest, you probably will not become proficient in that area. Unlike aptitude, interest can change over time. For instance, a person can be exposed to an activity and find that it comes easily and that person's interest increases. On the other hand, an activity that has always held your interest may turn out to require more work than you plan to supply and your interest can wane. Aptitude and interest together seem to be a requirement of becoming proficient in an activity.

Most activities require some abstract knowledge to become truly proficient. Different activities require differing quality and quantity of knowledge. A person can acquire this knowledge through books, classes, and reasoning about experience. Obviously, different activities handle the acquisition of knowledge differently. Unfortunately, abstract knowledge is not always directly applicable to an activity.

The last piece of the puzzle is a set of skills that are used as part of the activity. Skills can only be developed by doing the activity. Different activities rely on differing amounts of skills. Unfortunately, many people pick up skills in an activity by rote and tend not to recognize the acquisition. Skills are acquired either alone or though the help of a mentor or senior person in that field.

Some of these attributes are innate, like aptitude. Some can possibly be taught to anyone, like knowledge or skills. However, unless all of these attributes are present, someone will never really become proficient in an activity. In order to analyze the productivity of software developers, we need to identify and analyze these attributes. Although aptitude and interest are hard to analyze, we can identify which knowledge and skills might lead to more productive software developers.

Part of the disparity we currently see when measuring productivity in programmers is caused by ignoring one or more of these attributes. When attempting to measure productivity, we look at years of experience or education to recognize equivalent candidates to compare. Unfortunately, abstract knowledge and some skills are easy to identify, but they do not make up the whole of proficiency.

Kinds of Problems

Of all of the lessons I have learned doing software development, one of the most important was to recognize what kind of problem I'm trying to solve. This sounds pretty trivial, but I'm not talking about the categorization you are probably thinking of. As software professionals, we tend to look at all problems as solvable. We can partition the problems we observe into multiple logical categories. This problem seems to be mostly a database problem. That problem is going to be mostly user interface. This other problem is going to be mostly about performance. Some problems cross multiple categories.

The hardest problems we have to solve as software professionals don't fall into these categories. Over time, I have begun partitioning problems into one of two categories before doing anything else. They are

• technical problems
• business problems

These categories are not hard and fast. Most of the problems we work on cover both kinds of issues. But, it is important to realize that some problems fall more into one category than into the other. Unfortunately, by our natures we are drawn to technical problems. Sometimes that pull is so strong that we forget about the other kind of problem. We assume that every problem can be solved given the right technical approach. Unfortunately, that isn't the case.

Several times in my career, I have supplied a good technical solution to a problem only to be shot down by someone in a non-technical role. In many cases, I was forced to implement a significantly inferior solution and then watch it fail in most (if not all) of the ways I had predicted. In most cases, I had to apply nasty hacks and band-aids to keep this bad solution running despite all of my warnings about how much this solution would cost in the long run. I have also seen many other programmers suffer a similar fate.

Eventually a couple of really good managers managed to get me to understand that sometimes the problem is not technical, it's a business problem. No amount of technological know-how can solve a business problem. Sometimes, the technically worse solution is more correct for the purposes of the business. It took years of fighting these kinds of issues before I finally came to understand this idea. Sometimes, the best you can do is to let the powers that be know what the ramifications of a technical decision are and then accept that they may overrule you for reasons that may never make any sense (to you).

Some of the best managers that have put me in this position went the extra step to let me know that they understood the implications of what I was saying. Sometimes this made things easier because I might be allowed to fix things later. Sometimes the issues were purely political and no amount of work or insight could solve the problem.

Eventually, I came to understand that no amount of tech can really solve a business problem. Throwing myself into the work to prove my solution is best or working long hours to bypass the issues are counter-productive. Now, when I recognize one of these kinds of business problems, I supply all of the information that I can to the appropriate individual. Then I do what I'm told and try to do the best I can within the constraints placed on me. Most importantly, I try not to be bothered by the illogical solutions I'm required to implement. For all I know, the whole project will be scrapped for political reasons and I'll never have to maintain the disaster I've been ordered to create.

It's not much consolation, but it's better than making myself sick over it.

Domain Knowledge and Programming

Over the years I've been a programmer, there seem to be two distinct areas of knowledge for any project. The first relates to programming. The second relates to the domain of the project. Over the almost two decades I've been programming professionally, I have worked in many domains. Interestingly, I have never started working on a project as a domain expert.

At different times, people have told me that they were looking for a programmer with extensive experience in X (whatever their domain is). I have even seen people go so far as to suggest that they would not hire a programmer who did not have a degree in the domain of interest. This seems completely backwards to me.

When hiring a construction company to build a bank, do we only consider companies that employ bankers? When finding a mechanic to work on an ambulance, would you only talk to those with EMS training? So why would you require a programmer to have a degree in biology to program for medical research?

Over my career, I have come to the conclusion that we programmers should always develop a some domain knowledge as part of a project. But, we never need to go too deep. The real need is to understand enough of the domain to be able to translate the real knowledge of the domain expert into a form that the computer can execute. Our real skills come in this translation, not in the knowledge of the domain. More importantly, what domain knowledge we have should match that of the customer. This is only guaranteed if we learn from the customer in the process of doing the job.

As an example, let's say that I went to work for an engineering firm. I have an electrical engineering degree, so this seems like a natural fit. However, my knowledge of the electrical engineering field is now twenty years out of date, and fuzzy to boot. Would that knowledge be a help or a hindrance to any work I might do.

Having some knowledge of a particular domain might be an advantage. Having no preconceptions about the domain might be a bigger advantage. In many of the programming positions I have held, I managed to do some good by approaching the problem differently than the domain experts. The key is to extract the knowledge from the domain experts in a form we can execute. If I accomplish that goal, I consider the project a success.

My field is software development. As such, I need an extensive knowledge of tools, methodologies, and techniques for the development of software. I also need the kinds of skills that are needed to develop software. One of those skills is the ability to stuff information into my head and retain it long enough to make it make sense to a computer. Most of the best programmers I know regularly do the same.

The reason for this rant is a problem I have run into several times in the last few years. I have been talking with someone about a job or a piece of code they want written and they decide either that I can not do the work or that I should be paid less because I am not an expert in their field. I have heard similar comments from friends. One fellow programmer told me how an employer in a medical research institution reduced the amount they were willing to pay him because he did not have a degree in biology. Now, I am pretty certain they had plenty of biologists on staff. But, they were hiring for a programming position, and a biology degree does not make you a better programmer.

I'm not sure what it is about our craft that makes people to think that there is nothing to it. Many seem to believe that they could pick it up if they just expended a little effort. But, of course, their field is something that you need loads of training to be able to do. Not like programming.

Most of the better programmers I've known seem to think about it differently. They assume that most people's area of expertise requires work, knowledge, and years of practice to be good. They also know that it takes similar work, knowledge, and years of practice to be good at what we do. That's why most of the code that I've seen written by domain experts is such a mess. They see code as a minor sideline away from the real work. We see code as the artifact we produce to help other people get on with the work they are interested in.

Participation vs. Hacking

In The Architecture of Participation vesus(sic) Hacking, Carlos Perez argues against points he sees in the essay Great Hackers by Paul Graham. Having read the two essays, I find Perez's comments enlightening, but maybe not in the way he intended. I found things in both essays that I agree with, and things in both that I disagree with. There are some that I feel deserve comment.

Perez begins by focusing on a comment made by Graham that of all the great programmers he can think of, only one programs in Java. Perez, and others, take immediate offense at this comment and move to respond.

However, they miss an important point of the comment. Graham specifies of the great programmers he can think of. Later in the essay, Graham describes how hard it is to recognize a great programmer unless you've worked with him or her. I could make a similar comment that none of the great programmers I know program voluntarily in Lisp or Python. This does not mean that great programmers don't write in those languages, I just don't know any.

Furthermore, I think I'd go even farther to point out that Java currently attracts a large number of programmers of varying levels of skill for one reason only; someone will pay them to work in Java. As a side effect, you are more likely to find average or below average programmers working in Java than great programmers simply because there are a lot more to sort through. That does not detract from the great programmers you will find.

Having blasted Graham for an attack on his favorite language, Perez goes on to attack other "Hacker" languages. He writes that

Nothing has ever been built in Lisp, Python or any other "Hacker" language that comes even close to the innovation in the Eclipse platform. I challenge anyone to point one out.

Many people in the comments section of his essay point out many counterexamples. There's no need for me to repeat those here.

I do find it interesting that Perez starts his whole essay based on one off-handed slap at Java and in the process makes an equivalent unsupported swipe at Perl (the write-only comment). I still find this ongoing misconception to be annoying. I've programmed professionally in Fortran, C, C++, Perl, Forth, and Java. I haven't seen a single one of these languages that did not allow write-only code. (In fact, I've seen some spectacular examples in Java.) I've also seen beautifully written code in all of these languages. The readability of any given piece of code is more a reflection on the particular programmer and the problem at hand than it is on the language.

Perez also finds confusion by what he calls Graham's definition of the word "Hacker". I find this amusing. Graham is using the definition of hacker that was accepted before the media appropriated it for "people who write viruses and other bad things". (See Jargon 4.2, node: hacker) I remember this definition of hacker in a version of the Jargon file back in the late 80's.

Perez's final mistake is in what he perceives as Paul Graham's fatal flaw. He points out that sooner or later, every programmer enters maintenance mode and that no hacker would remain at that point. Perez defends this idea by quoting Graham's comment that hackers avoid problems that are not interesting. He proceeds to use the legacy code argument to show why you shouldn't rely on hackers or hacker languages.

Unfortunately, Graham didn't say anything about great programmers and maintenance mode or legacy code. So this argument has nothing to do with Graham's essay. Moreover, I have seen many cases where the beginning of the project was better described by Graham's death of a thousand cuts comment than the maintenance of any legacy system I've worked on. More importantly, GNU software and the entire open source movement is driven by programmers (many of them great programmers) maintaining software for years.

All in all, I think Perez would have made his points better if he had not taken the Java comment quite so personally.

Programmers and Pattern Matching

Contrary to expectations, many programmers do not solve problems through logic. This should not be surprising, as the same is true of humans in general. But many programmers I've known whole-heartedly disagree with that comment.

I've noticed that many really good programmers are extremely good at pattern recognition and solve many problems through a creative use of that skill. The human brain actually has amazing capabilities for pattern recognition. Two really good examples of this skill are language and facial recognition. When you think about it, much of language consists of recognizing patterns of sounds at a level below conscious thought and reinterpreting that into a higher level meaning. The ability to pick up subtle clues from a person's expression or recognize a person after many years (and the physical changes that come with that time) is truly amazing when you stop to think about it.

Most really good programmers are fascinated by patterns. They often play with patterns in ways that non-programmers might find bizarre. Many programmers really get caught up in puns, word-play, debate, poetry, and other forms of linguistic patterns. Many are fascinated by puzzles and mazes. Some spend lots of time with art. In all of these cases, the programmers seem to be exercising their pattern skills more than most.

It turns out that pattern recognition can be a really efficient method of solving problems. Either deductive or inductive reasoning requires walking through all of the relevant data of the problem one step at a time and connectinve them all together. Pattern matching allows you to focus on the shape of the problem to recognize a problem you've seen before. If you don't agree that the pattern recognition is easier, compare recognizing the face of a friend to the effort of describing that friend in enough detail for someone to draw them.

This ability allows a programmer to solve complex problems by breaking it into simpler problems that are recognized and therefore already (or simply) solved. This approach leaves the programmer free to concentrate on the new, interesting problems, instead of wasting brain power on the stuff that he/she recognizes. Over time, this develops into a very large mental list of patterns and solutions that can be applied at a moment's notice.

Some programmers allow this habit to get them stuck in a rut. They always apply the same solution even if better techniques exist. The better programmers are constantly learning so that they have a set of tools to apply to most problems. This still allows the efficiency of the pattern recognition approach with the power of a (near) custom solution if necessary.

Other Resource Recovery Approaches

In The Semantics of Garbage Collection, I explained why I don't like the term garbage collection. I also introduced the term resource recovery, and suggested that this change in naming could generate a useful change in viewpoint.

In the next article, More Thoughts on Resource Recovery, I traced some of the history and side effects of the main current approach to garbage collection. I also pointed out where I think this approach makes a wrong turn.

Now, I'd like to turn to alternative approaches. Some that exist, and some that don't (yet).

The simplest and easiest to understand form of memory management is stack-based, local variables. The life-time of the variables matches the scope in which they are declared. Memory is automatically cleaned up when execution leaves the scope. If we could limit our use of memory to the current scope and any functions called from that scope, the memory problem would vanish. Unfortunately, this approach is too restrictive for most programming projects. At the very least, the lifetime of data in an object must persist beyond the scope of the constructor.

C++ has a very clean system for dealing with memory or any other resource. By defining both a constructor and a destructor, you can guarantee that memory is allocated when an object is created and freed when the object is destroyed. In the same way, you can allocate or acquire any resource at construction time and release or free it at destruction time. C++ programmers refer to this as the resource acquisition is initialization (RAII) idiom. Using this idiom consistently with only stack-based or member-data objects will also prevent resource leaks.

Unfortunately, this is still not the whole story. Sometimes, there is the need to share the same object between two places in the code or objects. There is also sometimes the need to have objects that live outside the particular scope in which they are created. Although, there are still special cases that can be handled simply, there is no way to always be able to recognize every way in which the programmer might want to deal with memory. For this reason, many computer scientists gave up and went the garbage collection route.

As I said in my last entry, the simplest case of automatic memory management uses reference counting. In this approach, every object carries with it a count of the number of references to it that currently exist. Many systems have relied on this relatively simple form of resource recovery. It has been re-implemented for use in many objects over time. There are a few subtleties to the implementation, but in general this approach works surprisingly well.

Reference counting has two problems, one minor and one major. The minor problem is the extra overhead of dealing with the count. For a small object, the count could be a significant increase in the size of the object. Dealing with the reference counts adds a small amount of effort when a new reference is made or an old one goes away. However, the real killer is circular references. A references B, that references A. Even if the last external reference to A or B goes away, these two items still have non-zero reference counts.

This issue is the result of a fundamental flaw in the reference counting approach. Reference counting mixes up life-time with ownership. These two concepts are strongly related, but they are not the same. In the C++ world, this problem has been explored in the Boost C++ Libraries. In this library, different programmers have designed a set of smart pointers with different semantics. You might wonder what kinds of semantics a pointer could have.

In many cases, the lifetime of an object is only really tied to one of it's references. That reference can be considered the owner of the object. The only time that reference counting makes any sense is when more than one reference must be the owner. So, let's look at ownership a little more closely. How do we define the ownership of an object? If an object is only referenced once, that reference is the owner. This includes objects on the stack. The only owner is the "reference" on the stack. Temporaries created by the compiler also only have one owner, the temporary reference that goes away at the next sequence point. Member or instance data usually only has one owner, the object it resides in. As much as we try to ignore them, globals and class data also only have one owner.

All of those kinds of references make up the easiest case. When the owner reaches the end of its life so does every object that it owns. The C++ destructor concept handles this very well for member data. Most languages handle it well for local objects. The C++ compiler handles this for temporaries. This is the easy case that everyone pretty much understands without thinking. It's also relatively easy to prove correctness for this case. Unfortunately, the whole world does not fit in this category.

Only slightly more complicated is assignment of ownership. This is when the old owner gives up it's ownership when making a new reference. The Standard C++ std::auto_ptr embodies this concept. This is often seen when a method or function returns an object. The old owner, the called method, gives ownership of the object to the calling method. Then the old reference ceases to exist. This sometimes also happens when putting items into a collection. The final case is when an object reference is copied to a new reference and the original reference is pointed to a new object. (Think manipulation of a linked list.)

The next level of complexity involves temporary shared access. This is the kind of sharing that happens when a temporary reference is made to an object but the original reference remains the owner. The most obvious example of this is passing an object to a function or method by reference. The function has a temporary reference to the object, but the object is owned by the calling code.

The interesting thing about all of these is that the compiler actually could determine the owner of the object fairly reliably in each case. By replacing the normal references with specialized references that either do or don't take control as needed, the compiler could convert each of these cases into something very close to the simple single owner, stack-based approach described earlier.

In these cases, we could get timely resource recovery and useful object lifetime semantics without resorting to a full-blown garbage collection system.

Unfortunately, that's not the whole story. The rest of the story will need to wait for later.

More Thoughts on Resource Recovery

In The Semantics of Garbage Collection, I explained why I don't like the term garbage collection. I also introduced the term resource recovery, and suggested that this change in naming could generate a useful change in viewpoint.

Many programmers have been indoctrinated with the belief that garbage collection is the solution to all memory problems, and that any language that does not use garbage collection is destined to be full of memory leaks. I disagree. But to really understand why I think otherwise will require a little history.

In the Beginning

Well, at least in the beginning of my programming career was Fortran. In the first dialect of Fortran I ever used, all memory was global memory. Even local variables in a subroutine were allocated as global memory; they just weren't visible outside the subroutine. There was no allocation of memory. Therefore, there were no memory leaks. There was also no recursion, because calling a subroutine from itself would wipe out the current values of its "local" variables.

These conditions no longer apply to Fortran, but they were the state of Fortran when I started. (Or at least the state of Fortran as I understood it then.)

The Stack and the Heap

Later in my career, I began to work in more modern languages. The first of these was C. In C, there were two types of memory besides the static memory I knew from Fortran. There was the stack and the heap. (I had actually seen these before, but I'll ignore that detail for now.)

The stack had the wonderful property that it would grow as you needed it and automatically be reclaimed at the end of the function (or block). It also allowed multiple copies of a function to have its own local variables, giving the possibility of recursion. Most of the time, stack memory is easy to use and is recovered automatically. As long as you didn't try to put a huge amount of stuff on the stack or recurse too deeply, everything worked out fine.

The heap had another wonderful property. I could allocate as much of it as I wanted (within reason) and give it back when I was finished. This allowed me to work with different amount of data without guessing the biggest size I could deal with ahead of time. However, this feature came with a dark side: the memory leak.

Objects

Then I discovered Object Oriented Programming. One of the most wonderful features of objects for me was the whole concept of the object lifetime. I could have an object that allocated memory as part of its construction and that would free that memory at destruction time. No more memory leaks! I could make allocated memory act like the stack memory that I had always liked. Well, of course things did not really work like that. People can allocate objects as well and automate the whole process of leaking memory.

Perl and Memory Management

At about the same time I found C++ and OOP, I also discovered Perl. Now Perl did not have objects at the time, but it did do automatic handling of memory in many cases. Over time, Perl developed real local variables (called my variables) and objects. Perl supported a simple garbage collection system called reference counting. Unlike the more popular mark and sweep garbage collection, reference counting has two really important virtues: it's simple to understand and it preserves object lifetime.

In a reference counting scheme, every object keeps count of the number of references pointing to it. As a reference goes away, the count of the references is decremented. When the count reaches 0, the object is really destroyed. This happens at a well-defined time, when the last reference is removed. This method is simple to understand and works very well in general. In fact, to my knowledge, reference counting has only one flaw: circular references. If object A references object B and object B references object A, and no other references point to either of them, we have a memory leak. These two objects are not referenced anywhere, but they cannot be reclaimed.

Reference counting works particularly well when objects are only referenced once or twice. This covers the simple object creation case and many uses of temporary objects created by the compiler. These objects are created and destroyed exactly the way you would expect.

Mark and Sweep

The great flaw of reference counting is one of the reasons that the mark and sweep approach was developed. Since mark and sweep starts from known references and works through all memory to find any that is referenced, circular references are not a problem. Unfortunately, mark and sweep suffers from a few problems of its own.

• The garbage collector can not be allowed to delete or move memory that is part of the stack.
• The time taken to garbage collect is a function of the amount of currently allocated memory, not the amount of memory to reclaim.
• The garbage collector does not usually respect object lifetimes.

The first point is why Java does not allow objects to be created locally on the stack. Otherwise, objects on the stack would need to be cleaned up differently than objects on the heap. With mark and sweep, this would require recognizing what parts of the objects are in which locations and treating them accordingly. The easiest solution was to define the problem away and only allow heap-allocated objects.

In order to perform the mark portion of the algorithm, we will need to touch each reference to every object to verify what is still in use. In order to perform the sweep portion of the algorithm, all referenced objects need to be moved so that the old memory can be reclaimed. More advanced versions of the system partition objects in different ways (new and old objects, etc.) to reduce this overhead to some extent.

For me, the worst effect of mark and sweep is the loss of object lifetime. An object's lifetime should run from the point where it is created to the last place it can be referenced or when it is explicitly killed. With the mark and sweep system, there is a period of time when the object is left in a kind of limbo. It cannot be referenced, but it hasn't been cleaned up. It's like a ghost of an object.

The Wrong Turn

Although I understand the purpose of garbage collection, it seems to me that we've taken a wrong turn. There was a time when we had stack-based data that was cleaned up automatically and was easy to deal with, and we had heap-based data that was more powerful, but could be misused. Now we've added garbage collection to reduce the chance of misusing the heap-based data, but to make the garbage collection simpler, we've thrown out stack-based objects and defined object lifetimes.

In the quest to stop memory leaks, we've made every object creation a memory leak and provided an automatic system to clean up the leaks we've made. Moreover, the loss of a defined time for the object to die has resulted in the loss of the destructor concept. We cannot be guaranteed that the object's cleanup/destructor/finalizer method will ever be called (even in normal operation).

This is why I have been thinking that maybe a new approach is needed. Next time, I plan to explore a possible new approach.

The Semantics of Garbage Collection

I have a confession to make. I've never really been comfortable with garbage collection.

Now, before you dismiss me as a Luddite who has obviously never worked with an advanced programming language, let me explain.

I'm familiar with many different forms of garbage collection. I have worked in languages with and without garbage collection. I've worked with Java, Lisp, and Perl; each with its own concept and implementation of garbage collection. I have seen well-done garbage collection free programmers from the drudgery of dealing with their own memory management.

But, despite all of that, I don't really like garbage collection. Strangely enough, when I finally examined why I didn't like garbage collection, I realized that my problem was more semantic than anything else. I don't like the name and what it implies. I don't think of used memory as garbage to be collected. I think of it as a resource to be recovered. The funny thing is that this shift in viewpoint results in a whole new set of assumptions about how resource recovery should work.

Timeliness of Recovery

The first consequence of this change in terminology is a change in timeliness.

Garbage is something we don't want to think about. Taking out the garbage is something we tend to put off until we have to deal with it. (The trash can is full, it's starting to smell, others in the house are complaining, etc.) This philosophy shows itself in the current crop of garbage collection systems. Most of the garbage collectors I've seen follow some variation of the mark and sweep algorithm. In this algorithm, a background task walks through memory marking all referenced objects. Then, it sweeps out the unused objects, freeing up their memory.^*

Unfortunately, running this algorithm uses CPU cycles that we obviously want to make use of elsewhere in our code. So most systems put it on a low priority thread; one that only gets called when there is nothing else to do (or when we begin running low on memory). Since this thread is just picking up our garbage, it doesn't need to be high priority. As long as the garbage is picked up before it gets in our way, we don't really care.

If you think of memory as a resource, things shift subtly. Resources are not things I'm ignoring, they are something I want. In fact, if it is a scarce (but renewable) resource, I want to make sure it is recycled or recovered as soon as possible. That way it will be available the next time I need it. It is no longer an issue of trying to put off this nasty chore off as long as possible. Now I want to try to handle it as soon as possible, without adversely impacting my other work.

Other Resources

Another advantage of thinking of memory as a resource, is that this puts it in the same category as other resources that we need to deal with. Most operating systems have limits on the numbers of certain resources. If a program tries to use too many without returning them to the system, it risks failure. Some kinds of scarce resources include:

• File handles
• Database handles
• TCP/IP sockets
• Process slots
• Address space
• Mutexes and semaphores
• Window handles
• Communications bandwidth

But, for some reason, we tend to treat memory as fundamentally different than these other resources. We tend to try to close/release each of the items in the list above as soon as we can. Because each is scarce and we know we will need it later. But, the current wisdom in garbage collection is to recover memory only when we have no choice or nothing better to do.

I find this approach disagrees with me on a gut-level.

*Okay, it usually involves copying the live objects around and doing some other maintenance work. But the simple description is enough for the purposes of the discussion.

The Fallacy of the One, True Way of Programming

What is it about programmers that cause them to latch onto one approach and try to apply it to every problem? Maybe it's a characteristic of people in general. Each new paradigm, methodology, or even language seems to grab hold of many programmers and make them forget anything they ever knew. They seem to develop an overwhelming need to re-implement and replace all that came before the new tool.

This isn't always a bad idea. Done deliberately, this technique can help you evaluate a new methodology or language by comparing the solution of a known problem with a previous solution of that problem. Unfortunately, many of us seem to assume that the new way is better and apply the new approach despite any evidence that it does not solve the problem more effectively.

Over time, I've come to the conclusion that there is no One, True Way of Programming. This is not my own insight. I have read this and heard this from numerous people with more experience. Earlier in my career, I did not believe them. I didn't disagree, I just didn't believe. Through experience with multiple paradigms, languages, and methodologies, I've finally come to understand what those senior people were trying to say.

Multiple paradigms are different tools in my bag of tricks. Each language is another tool that I can apply to a problem. Different methodologies give me different approaches to solving a problem. But, in the end, no single set of these tools can help me solve all problems. Each combination has strengths and weaknesses. One of the most important skills we can develop is the ability to recognise the right tool for the job.

Many years ago, I saw this point made in a book. (One day I'll remember which book.) The author pointed out that very different techniques and processes are needed when you are building a skyscraper and when you are building a doghouse. The author pointed out that the techniques that would be used to architect a skyscraper are not cost-effective when building a doghouse. I think I would now go a little further and suggest that the techniques are not appropriate, and that cost is one easy measure of why they are not appropriate.

Flow and Miller's Magic Number

Recently, I stumbled across Mental State Called Flow and was reminded of an idea I had about flow and Miller's Magic Number. When in the state of flow, I've sometimes had the experience that the problem I was working on was much easier to understand than it seemed to be when I was not in flow. In observing and talking to other programmers, I've come to the conclusion that this is not an unusual experience.

Several years ago, I worked with a senior programmer who always seemed to be in flow (or hacking, depending on who you asked). Unfortunately, the code he wrote was often difficult to understand. After a few years, I made a breakthrough. I could usually understand his code while in a state of flow. While in flow, not only I could understand his code, I could also refactor it into a form that I (and others) could understand under normal circumstances. Other programmers in the group had similar experiences.

One day, everything seemed to come together in a blinding flash. During flow, Miller's Magic Number is larger. Normally this number appears to be around 7 chunks of information that you can deal with at one time. But during flow, the number of chunks seemed to be much higher. I have no idea how large the number is, but problems that seem to be at the edge of my ability normally are quite easy to deal with while in flow.

This means that flow can have a downside. If you are not careful with how you write the code, this larger number of information chunks will always be required in order to understand the code. You have probably seen this phenomenon in code that was written as part of a long hacking run. It usually works, sometimes even works very well. But you can't quite make sense of why it works like it does. This may be one of the reasons that hacking code has gotten such a bad name. Code like this is nearly impossible to maintain.

However, you can use this enhanced facility to understand the problem and still strive for simple, elegant code. The result is code that solves a difficult problem in an obvious way. Much of the time, this solution is only obvious in hindsight. You required more information while working on the problem than you normally could handle in order to see the simple and obvious solution.

In the Jargon File, the term hack mode is used to describe this form of flow that particularly focuses on The Problem. I believe the ability to use the benefits of the increased magic number, without making your code require this concentration to understand is one of the more important skills to learn in programming.

Update on Competence

In Programmer Musings: More Thoughts on Mastering Programming I referenced the four levels of competence without being able to remember the original source. According to Four Levels Of Competence, the source is the "Kirkpatrick Model" by Donald L. Kirkpatrick.

Idiomatic Programming

Like any natural language, most programming languages support idioms. According to the American Heritage Dictionary of the English Language, an idiom is

A speech form or an expression of a given language that is peculiar to itself grammatically or cannot be understood from the individual meanings of its elements, as in keep tabs on.

Programming idioms can, of course, be understood functionally from its individual elements. The interesting thing about most programming idioms is what they say to the programmers who will read the code later. A programming idiom can reveal many things about the author of the code. To some extent, the idioms you choose are a function of your understanding of the language, what you may have been taught in school, and the other languages you have programmed in.

I made a comment on this subject earlier in Textual Analysis and Idiomatic Perl on Perl Monks. In that article, I pointed out that idioms are cultural, just like those in natural languages. Also like in natural languages, if you only have experience in one culture and language, the idioms of another appear unnatural or wrong. When learning a language, you learn the syntax and obvious semantics. You have to live with the language for a long while to learn the idioms.

The result of this is that the idioms of a language are foreign to new speakers. As they continue to use the language, simple idioms become part of their vocabulary. At first, they will misuse or over-use these idioms. Over time, they develop an appreciation for the subtleties of the expression. Eventually, those idioms are only used when necessary. With more exposure, more idioms become assimilated into your vocabulary. Soon, you learn to speak/program the language relatively fluently. Also, like natural languages, you may always retain a bit of an accent, based on your earlier languages.

This explanation seems to explain the course I've seen many programmers, as well as myself, take with a new programming language.

1. You learn the basics.
2. You begin to recognize and fight against the idioms.
3. You slowly recognize the sense of simple idioms and try to use them.
4. You over-use the idioms you know.
5. You develop a reasonable understanding of those idioms and begin using them correctly.
6. You learn new idioms and go back to 4.
7. Eventually, you program with the language instead of fighting it.

The simpler the language, the quicker you get to the end of this cycle. But, a simpler language is not as expressive. So, you find yourself doing more work than should be necessary. A more expressive language takes longer to learn, but it's easier to pass on subtle shades of meaning to the maintenance programmer (possibly yourself).

This is not to say that you must use advanced idioms to be effective in a language. Sometimes, a simple expression is all that is warranted. But, complex ideas and complex problems often have subtle solutions. Subtle shades of expression can help the reader of your code recognize when they need to pay attention. In the example from my Perl Monks note, I talk about variations in intent expressed by different conditional expressions in Perl.

Idiom can also serve as a warning. An unusual or advanced idiom should tell the reader that he is entering dangerous territory. This is a message that it's time to pay close attention. A friend of mine is fond of referring to advanced code with the phrase Here there be dragons like on old maps. A properly placed idiom can do that more effectively than a comment. More importantly, if a good programmer runs into an new or unusual idiom, he should know to tread carefully and pay attention.

I think that idiomatic programming is a lot like becoming fluent in a natural language. You may be able to communicate with someone from Mexico using your high school Spanish. But going beyond simple, straight-forward communication will probably take a lot of effort. In some cases, that's enough. You can be mostly right when asking where is a restroom or how to find a taxi. But if you needed to argue a business or legal case or explain a medical condition, I'm sure you would want every possible nuance of the language at your disposal.

Programming is one of the most complex things that humans do. You have to be able to take complex concepts and translate them in such a way that they communicate to two very different audiences: an incredibly literal device and an intelligent person who may need to make subtle changes later. Explaining the instructions correctly to the computer is hard. Communicating your intent to the other programmer is even worse. If that programmer misunderstands your intent, he might ruin all of your carefully thought-out work. Well-chosen idioms can help in communicating that intent.

Refactoring, Factoring, and Algebra

I saw the article Refactoring reminiscent of high school algebra on Artima yesterday and it made me remember a new connection of my own.

In the article, David Goodger is working on a refactoring problem that is turning out to be worse than expected. After a little reflection, he manages to solve the problem. But the code had to get messier before it got better. In the end, he makes a connection between refactoring code and multiplying polynomials in high school algebra.

I had made a similar connection several years ago when teaching programming to entry-level programmers. We talked a lot about well factored code and how important it was to keep your code factored. (This was before the term refactoring became mainstream.) I had several students ask how to recognize when the code was well factored.

Now I truly hate to use the answer "experience" when someone asks me a question like this, so I spent some time talking with other senior programmers and thinking about it and came to a realization similar to Goodger's. (Re)Factoring code is like factoring composite numbers. You stop factoring composite numbers when you have nothing but prime factors left. You stop (re)factoring code when the code has reached the point of being minimal. Each piece does one thing and does that well. Your program is then the composite of these fundamental pieces. This is also a lot like factoring polynomials.

Now, I think I would go a little further to say that code that embodies one concept or action or code that has a simple or obvious name is factored enough. To try to factor it any further would make it more complicated.

I think another really important part of Goodger's article was the point that sometimes the code has to be made uglier before you can finish refactoring it.

A Report on UML Fever

ACM Queue often has very insightful articles, and this one is no exception. The article ACM Queue - Death by UML Fever - Are you (or your developers) sick? covers a major problem in software development in a somewhat humorous fashion. I've seen several variations of this problem, but I never thought of it as an illness.

For those who are appalled at the article and who consider the author to be a heretic, you might to step over to another article in the same issue. In ACM Queue - The Fever is Real -, Grady Booch comments on the UML Fever, both the article and the affliction.

Another good piece of commentary on the subject is java.net: Death by UML-more [April 23, 2004].

The most important thing to take away from all of these articles is that UML is a tool (or a set of tools), not a life style, not a religion, and certainly not a solution to all problems everywhere. I'll be glad when more people use the tool was it was intended, instead of using in yet another round of holy wars.

More Thoughts on Mastering Programming

Reading Dave Thomas's blog on Code Kata pointed to PragDave: MoreKata. I also stumbled across How To Practice on Chad Fowler's blog.

Both of these entries discuss exercises that we can use to master the art of programming. Perhaps surprisingly, they use metaphors and concepts from martial arts and Zen. Those of us who have been programming for a while can easily see the parallels.

I'd like to add a description of attaining mastery of a subject that I saw once a long time ago. (I can't find the reference right now, or I would give credit.) The description listed four phases of understanding.

1. Unconscious incompetence
2. Conscious incompetence
3. Conscious competence
4. Unconscious competence

Programming Practice

I'm not quite sure what lead me to this blog entry, but I find PragDave: Code Kata by Dave Thomas to be a very interesting idea. I had never thought of describing coding practice as similar to kata.

I did stumble into a similar idea once many years ago. I had just finished reading Software Tools by Kernighan and Plauger, when I came to the conclusion that implementing some of the classic Unix text tools from scratch would be a good way to improve my programming. None of the programming problems themselves were particularly hard, but I got the idea that building a clean implementation of each tools would help me focus on practical aspects of my programming.

Over the next few months, I built a number of the classic tools. In each case, I worked hard to generate a clean, minimal, and complete implementation of each tool. I went over each piece of code several times to ensure that there was nothing unnecessary or unclear in the code. When I finally declared this experiment finished, I really felt that my coding and design skills had improved dramatically.

I haven't thought about that exercise in a number of years. Now that PragDave has reminded me of it, I think I may need to pick a similar project to see if I can help keep those mental muscles in shape.

The Recurring Myth of Programmer-less Programming

Why is it that some people regularly suggest that someday soon we won't need programmers? I've recently been seeing this as part of the IT outsourcing debate. But, this isn't the first time I've seen it. The first time I saw this meme was shortly after I started programming professionally 17 years ago. I talked to some other people at the time and they said it was old at that time. In all of this time, I have yet to see it happen.

Why does this meme keep coming back? Why are some people so drawn to the idea? And, just as importantly, why is this a myth?

The most recent version of this meme has compared programmers to typists. After all, a decade ago no one would have believed that there would have been no need for typists. But now everyone types their own documents and there is no need for someone who's title is typist. So, with an interesting jump in logic, someday soon we won't need programmers either.

I'm sure that most people developing software would consider the comparison of programming to typing completely invalid. But, how do you explain that to someone who wants to fire the experienced developers to replace them with cheap college students or workers in third world countries? After all, if programming is equivalent to typing, shouldn't we just replace expensive programmers with cheap typists and make more money?

I think this a great example of a mistaken analogy. It would be more accurate to compare typists to scribes and programmers to writers.

There was a time in history when scribes were the only people trained in writing. But much of what the average scribe did was copy or transcribe the works of others. Some scribes were scholars, and many were quite skilled. But, most scribes did not do original writing. As more people learned to write, the need for scribes went away. Much like the use of word processors reduced the need for typists.

However, just because people learned to use pen and ink to write down everyday information, the need for writers did not go away. In fact, writers were more in demand as people became more literate. There is more to writing than the skill of using a pen and ink.

Likewise, there is more to programming than typing on a computer. I've noticed that many people who don't program or who have only a cursory understanding of programming assume that there's not much to it. After all, you figure out what you want to do, you type it in to the computer and the computer does it. How hard can that be?

Much like writing, there is more to programming than meets the eye. I've heard it said that for a good writer, getting an idea is only the beginning. There are hours of research and rewrites and real work that goes into almost everything worth reading. Much like a good piece of writing, a good program may look easy once it's finished, but not everyone is able to research and design and rework to get that finished product.

The two hardest parts of programming are similar to the hardest parts of writing. In both, you have to be able to really think clearly and understand to a degree that most people don't understand. Secondly, in writing and programming, you have to be able to communicate those thoughts.

Unfortunately, in programming you have two very different audiences. The first is the dumbest machine ever invented. Computers do exactly what you tell them to, no matter how dumb the instructions are. No person or animal would ever follow instructions that blindly. The second audience is other programmers (or yourself some time in the future). This audience has to be able to understand what you wrote well enough to adapt it at a later time.

Because this is much harder than almost any non-programmer can fathom, there is this myth that we can replace programmers with something mechanical or with non-programmers. This is much like saying that we'll replace the people that write the company's press releases or contracts with trained monkeys. After all, how hard can it be to crank out this stuff. The answer is (in both cases) some of the writing/programming can be done practically automatically. But the stuff that really matters, the stuff that makes the difference between success and failure needs someone skilled and talented to write it.

Miller's Magic Number

George A. Miller's paper The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information discussed some limits of the human brain with respect to information processing. In particular, his research had found that people are unable to keep up with more than 5-9 different chunks of information at a time. This is actually why phone numbers in the United States are seven digits long, or more accurately, why they used to be an exchange and four digits. (The exchange was eventually replaced by three digits.)

I know a lot of you are thinking that information cannot be true. After all, you know you can keep more things in mind at one time. Right? According to Miller, the key is the word chunks. These chunks can be different sizes. A concept that carries a lot of information is still a single chunk. This is why is is harder to remember 10 randomly chosen numbers or words than it is to remember the words to a song.

A large portion of the history of programming has been devoted to making our chunks more meaningful. Higher level languages allow us to work without keeping trying to remember what each register is holding and how many bytes to we need for that jump instruction. Each succeeding level allows us to keep more in our heads by making the chunks bigger.

But that only works as long as the concepts map well to single chunks. If you don't have a name for a chunk of information or a concept, it takes up more of your memory. One of the more useful effects of Design Patterns was not new OO techniques. It was the new names. Suddenly, you could refer to the Singleton Pattern instead of this class that there's only one of in the whole system but is available to everyone, sort of like global data but not quite.

This same concept applies to user interface design. Grouping related items on the screen and putting the most commonly used items where they are immediately accessible are two ways to reduce the amount of your mind tied up by keeping up with what goes where.

The concept of chunks and Miller's magic number applies in many places in programming. Here's a few to ponder:

• Good variable names make it easier to remember what they are used for.
• Global variables make code more difficult to understand, because you use up chunks thinking about those variables.
• Good method names replace all of the implementation details with one chunk.
• Flags are bad for the same reason as global variables.
• A good programming metaphor helps you develop because more concepts are tied into one chunk.
• Programming paradigms help you solve programming problems by giving you names for concepts you need to work with.
• A programming language is well suited for a problem when you can express your solution in fewer chunks in that language. (Some might say you can express it more naturally in that language.)
• Good data structures help to design by reducing a number of related variables into a single coherent chunk.
• Good classes embody one chunk of information. Bad classes either express multiple chunks or none.

The Paradigm Paradox

I've already written some of my ideas about paradigms, but the most important point still remains. I maintain that the real benefit of a paradigm is not the way of thinking that it defines, but the contrast between this way of thinking and others. Almost every programmer I've ever met that only knows one paradigm seems to be hampered by an inability to look a problems from more than one viewpoint. I believe that much of the benefit that is derived from changing to the newest paradigm comes from the change. This would also help to explain why the decision to train people only in this newer paradigm does not automatically create better programmers.

Most of the entry-level programmers that I've worked with have been about the same level of effectiveness no matter what paradigm they were trained in. This has been true of structured programmers, modular programmers, and object oriented programmers. This effect also applies to the languages they were trained in. I haven't seen any improvement in new Java-trained programmers over the older C-trained programmers or the Pascal-trained programmers before them.

Each generation of programmers seems to make a lot of similar kinds of mistakes. Although, the syntax of a new language may prevent you from making certain particular mistakes, there are certain classes of mistakes that remain. All of these entry-level programmers seem to start with a misunderstanding of complexity and an inability to look at a problem in an unusual way. They immediately begin to apply the paradigm without first understanding the problem. They also tend to solve problems in older code by adding more code, even when that is usually not the best solution.

No new paradigm that I've seen has solved these fundamental problems. I think that the effort to learn and work with more than one paradigm (or language) forces you to realize some of the differences between a good solution and an artifact of the paradigm (or language). For example, one of the most basic attributes of a good design is simplicity. This has been true in each of the programming paradigms that I have used. It is also true in mathematics, logic, and science.

Given two designs that meet all of the requirements, the simpler one is usually the best. One of the most important aspects of the OO paradigm is its ability to deal with complexity. A side effect of using OO is that many people add more complexity, because they can handle it. They are much more likely to build a framework of supporting classes, even for problems that don't require them. In this way the OO paradigm can lead an inexperienced programmer to build a worse design through the addition of unnecessary complexity.

If you have experience in more than one paradigm, you are more likely to recognize this unnecessary complexity than if you've just done OO programming. Why? Because if you could solve the problem more simply without the object approach, you would see that solution as well. The contrast between these two possible solutions would cause you to look for ways to get the benefits of OO but the simplicity of the other solution. The result is often a simpler design.

On the other hand, some problems actually only have complex solutions. When comparing two or more approaches in this case, the complexity is more apparent because you can see the difficulty with using the non-OO approach. Nevertheless, the second viewpoint can provide insight into other difficulties that might be masked by the OO design. In this case, the second viewpoint does not replace the OO design, it enhances it.

So, to reiterate, the most important feature of a new programming paradigm is the contrast between that paradigm and the ones that preceeded it.

My other blog entries on programming paradigms:

• Paradigms limit possible solutions
• Paradigms Lost
• Paradigms Found

Bad Names and Good Bad Names

And just to prove that some ideas appear in many places all at once, Andy Lester explores naming again in O'Reilly Network: The world's two worst variable names [Mar. 07, 2004]. Apparently, Lester is reviewing the second edition of Code Complete, one of my favorite books on programming, and says there is "an entire chapter devoted to good variable naming practices."

The comments on the Lester's weblog are a spirited little discussion on variable naming. But I have my own thoughts, of course.

A large portion of programming is actually thinking. We spend time thinking about a problem and convert our thoughts and understanding into code. As such, names are critical to clear thought and code. One comment on Andy Lester's weblog suggested that the time spent coming up with good names was better spent writing code. This is an attitude I have heard many times in the past and have worked hard to remove when teaching.

Good names help when reading code. More importantly, good names help when thinking about the code, both during maintenance and during development. Humans are only capable of keeping a small number of distinct concepts in mind at once (7 +/- 2 according to George Miller), a good name can help by abstracting away a large amount of unnecessary detail. This allows you to focus on the important part of the code.

I try hard to always use good names for variables, functions, classes, programs, and any other artifact of my programming. I have learned over time that good names make my programming flow more easily. When I run into a method or variable that I can't name, I know that I don't truly understand that part. Many times I will give it a placeholder name until I understand it well enough that a name presents itself. At that time I try to always go back and rename the item to show my new understanding. To me, good naming is part of good design.

In contrast, if you use good names in general, there are places where bad names can be good ones. For example, i, j, and k are lousy names for variables. But, as loop variables, they become useful because we are used to them. Since the early days of FORTRAN these were the canonical loop control variables. Most people would recognize that without thinking about it.

One of my favorite bad variables is dummy. I only use it in one circumstance, I need a placeholder in an argument list that will receive a value that I will not use. Anybody looking at this code should understand the meaning pretty quickly. For example, in the C++ code below, dummy is used to receive a delimiter that I don't care about.


config >> row >> dummy >> column;


I also have one really bad program/script name that I use regularly. The name is so bad that it carries with it an important meta-meaning. I often write one-shot, throw-away programs or scripts and name each of them doit. These scripts are for jobs that are easy enough to rewrite that I would spend less time rewriting it than I would spend trying to remember what I named it last time. I often write the doit.pl Perl script for work that is a little too complicated for me to do as a one-liner, but not complicated enough to really build a real program for.

The meta-meaning of the doit program is interesting. If I find a doit script or program in any directory, I delete it. It was designed as a throw-away and I know that it is not really worth keeping. Every now and then one of my doit scripts evolves into something useful while I am working through a problem. At that point, I give it a real name and it becomes part of my toolkit, at least for that project.

The subject of names in programming is more important than many people realize. Some really bright people have written on the subject. Steve McConnell had some good advice in Code Complete. Andrew Hunt and David Thomas explain some of the reasons that bad names cause harm in The Pragmatic Programmer. Andy Lester had another weblog entry on names a short while back, O'Reilly Network: On the importance of names [Feb. 15, 2004]. In the comments to that entry, he pointed to Simon Cozen's article Themes, Dreams and Crazy Schemes: On the naming of things from a couple of years ago.

Review of The Logic of Failure

The Logic of Failure
Dietrich Döner
Perseus Books, 1996

This is a spectacular book on why people make mistakes. This book gives wonderful insight into design and troubleshooting that you won't get from any traditional book on programming.

A large portion of the book describes how very intelligent people make what seem like blindingly stupid mistakes. It shows how people stop thinking logically when there is a delay in feedback between cause and effect. It also shows how people react when the situation seems to be getting worse despite their efforts. You have probably observed some of these behaviors in people when things really begin to go wrong.

Amusingly enough, a friend of mine pointed out that this book would have been much more successful except for the mistake of focusing the title on failure. With that focus, obviously some of people avoided the book because the don't want to contemplate failure.

Although not a normal book on programming, I would recommend this book to any programmer who needs to do troubleshooting or development under the gun. It may not solve your problems, but it may help you understand when people including you start reacting to a stressful situation.

Paradigms Found

In my weblog entry Programmer Musings: Paradigms Lost from about five days ago, I talked about some of the negative effects of programming paradigms. That entry could give the impression that I don't believe in new paradigms. Nothing could be further from the truth. However, I think the main benefit of new programming paradigms is often overlooked.

Each programming paradigm that comes along gives us new tools to use in our quest to create great solutions to hard problems. As I said before, these new tools don't necessarily invalidate the last paradigm or its tools. So, what does a new paradigm give us?

Conventional wisdom says that each new paradigm has given us better tools and techniques for dealing with complexity. The object oriented paradigm allows us to deal with more complexity than the modular paradigm which came before it. These new tools and techniques come with a cost; a relatively high learning curve. It also came with some complexity of its own. This is not to say that the new complexity and learning curve aren't worth it, but it is important to acknowledge that cost. This cost is especially high on projects where the old paradigm was enough. A very good example of this cost is the standard Hello World program.

In the modular style of C, this would be

#include <stdio.h>
int main(void)
 {
  printf( "Hello World\n" );
  return 0;
 }

In the object oriented style of Java, this would be

class Hello
{
  public static void main( String [] args )
   {
     System.out.println( "Hello World" );
   }
}

In the object oriented style, we need to know about a class, static methods, public access, the signature of the main method, a String array, and the proper method of the out object in the System package to call for printing to the screen. In the modular style, we need to know the signature of the main function, how to include the I/O library, and the name of the library function to print a string.

Granted, in a larger, more complex program this overhead in understanding would not be quite so overwhelming. In a simpler or less complex program, this extra overhead may not be worthwhile.

You might ask "why not just learn one paradigm?" After all, if the latest paradigm handles complexity best, why not just use that one. Part of the answer is because of this understanding overhead. I've seen many cases where people did not automate a process they were going to manually execute dozens of times because it was not worth writing a program to do it. When you make this decision, the computer is no longer working for you. We're programmers. The computer is supposed to do the grunt work, not us.

Okay, why not just use the more advanced paradigm even on small quick-and-dirty programs. In a small program people often find that they must spend more time supporting the paradigm than solving the problem. This is why shell scripting is still used. Why write a program to automate a half dozen commands you type to get your work done when a short shell script or batch file can solve it for you? Agreed, it doesn't support the latest paradigms, but it does get the computer working for you, instead of the other way around.

However, we still haven't touched what I feel is the most important thing about new paradigms. New paradigms give you a different way to think about solving problems. This is not meant to imply a better way, just a different one. If you have two (or more) ways to look at a problem, you have a better chance of coming up with a great solution.

By having two or more ways to view a problem, you increase the number of approaches you can use to tackle the problem. Maybe you don't need a large object framework to handle this problem, maybe a straight-forward procedural filter implementation will do. In another problem, you might have too much required flexibility to deal with in a procedural program maybe the Strategy Pattern with appropriate objects is a better approach. Then again, possibly a little generic programming with the STL is exactly what you need.

The unfortunate problem with programming is that the complexity never goes away. Different paradigms just manage the complexity in different ways. Some paradigms handle some kinds of complexity better than others, but they do it by changing the form of the complexity, not by making it disappear.

The most important thing about knowing multiple paradigms is that it allows you to decide how to manage the complexity in any given problem. By giving yourself more options and more ways of looking at the problem, you increase the chances of finding a good match between solution and problem no matter what problem you are trying to solve. That, in my opinion, is the most important advantage of a new paradigm.

Paradigms Lost

An earlier weblog entry, Programmer Musings: Paradigms limit possible solutions, spent some time on what paradigms do for programming.

Now, I'd like to consider a slightly different take on programming paradigms. Why do programming paradigms seem to take on the force of religion for so many in the software field?

Although I haven't been a professional programmer as long as some, I have been in the business long enough to weather several paradigm shifts. Now before you run screaming away from that annoyingly over-used phrase, I really mean it in the literal sense: shifting from one dominant paradigm to another.

When I started structured programming was king. That was later refined with modular programming. I remember a brief stint in literate programming that never really seemed to catch on. The current big thing, object oriented programming, has even spun off sub-paradigms like aspect oriented programming. Let's also not forget generic programming that hit the mainstream with the C++ Standard Template Library and Design Patterns introduced by the Gang of Four book. Along the way, I even dabbled in functional programming.

Each of these, in it's time, helped programmers to build better code. Each paradigm has strong points and weaknesses. What I don't understand is why the people adopting each new paradigm find it necessary to throw away good ideas because they were associated with an older paradigm. I am continually amazed that someone would look at a perfectly reasonable program in the structured programming style and dismiss it as not object oriented. So what. Does it do the job? Is it still plugging away after being ported to three different operating systems and who knows how many environments? If the answer to these questions is "yes", who cares how it was designed?

A couple of weeks ago, I saw the weblog entry O'Reilly Network: Overpatterned and Overdesigned [Jan. 28, 2004], where chromatic blasts the overuse of patterns. I was genuinely amused and heartened by his example and the comments that followed. One of the points of his entry was that design patterns can be overused. I am willing to go a little further, any paradigm can be overused.

The biggest problem I see with programming paradigms is a misunderstanding of what they really mean. A paradigm is a way of thinking about problems and solutions. Using a paradigm should not be a life altering event. In my experience, no paradigm is inherently better than all others. However, it seems that as each new paradigm is discovered, a large fraction of the people using it feel the necessity to ignore all of the lessons of previous paradigms, as if they somehow no longer apply.

The really funny thing is that, over time, we see the same lessons come back, often with different names. In the old structured/modular programming days, you could improve a program by reducing the coupling between unrelated code and increasing the cohesiveness of your modules. Of course, no one does structured programming now. Instead we would refactor our classes to increase independence between our classes possibly through the use of interfaces or abstract classes. We would also work to provide minimal interfaces for our classes. We want to make sure that those classes only do one thing. Sounds familiar, doesn't it.

Hopefully, one day, we will learn to treat a new paradigm as just another tool in our kit. Remember carpenters did not throw out their hammers and nails when screws and screwdrivers were invented. Each serves a slightly different purpose and provides useful options. I see programming paradigms in a similar light.

The Forgotten OO Principle

When talking about Object Oriented Programming, there are several principles that are normally associated with the paradigm: polymorphism, inheritance, encapsulation, etc.

I feel that people tend to forget the first, most important principle of OO: object lifetime. One of the first things that struck me when I was learning OO programming in C++ over a decade ago, was something very simple. Constructors build objects and destructors clean them up. This seems obvious, but like many obvious concepts, it has subtleties that make it worth studying.

In an class with well-done constructors, you can rely on something very important. If the object is constructed it is valid. This means that you generally don't have to do a lot of grunt work to make sure the object is set up properly before you start using it. If you've only worked with well-done objects, this point may not be obvious. Those of us who programmed before OO got popular remember the redundant validation code that needed to go in a lot of places to make certain that our data structures were set up properly.

Since that time, I have seen many systems where the programmers forgot this basic guarantee. Every time this guarantee is violated in the class, all of the client programmers who use this class have a lot more work on their hands.

I'm talking about the kind of class where you must call an initialise method or a series of set methods on the object immediately after construction, otherwise you aren't guaranteed useful or reliable results. Among other things, these kinds of objects are very hard for new programmers to understand. After all, what is actually required to be set up before the object is valid? There's almost no way to tell, short of reading all of the source of the class and many of the places where it is used.

What tends to happen in these cases is the new client programmer copies code from somewhere else that works and tweaks it to do what he/she needs it to do. This form of voodoo programming is one of the things that OO was supposed to protect us from. Where this really begins to hurt is when a change must be made to the class to add some form of initialisation, how are you going to fix all of the client code written with it. Granted, modern IDEs can make some of this a little easier, but the point is that I, as the client of the class, will need to change the usage of the object possibly many times if the class implementation changes.

That being said, it is still possible to do some forms of lazy initialisation that save time at construction time. But, the guarantee must still apply for a good class. After construction, the object must be valid and usable. If it's not, you don't have an object, you have a mass of data and behavior.

The other end of the object's lifetime is handled by a destructor. When an object reaches the end of it's life, the destructor is called undoing any work done by the constructor. In the case of objects that hold resources, the destructor returns those resources to the system. Usually, the resource is memory. But, sometimes there are other resources, such as files, database handles, semaphores, mutexes, etc.

If the object is not properly destroyed, then the object may not be accessible, but it doesn't really die. Instead, it becomes kind of an undead object. It haunts the memory and resource space of the process until recovered by the death of the whole process. I know, it's a little corny. But, I kind of like the imagery.

This concept also explains one of the problems I have with some forms of garbage collection. Garbage collection tends to assume that the only thing associated with an object is memory. And, as long as the memory is returned before you need it again, it doesn't really matter when the object dies. This means that we will have many of these undead objects in the system at any time. They are not really alive, but not yet fully dead. In some cases, you are not even guaranteed that the destructor, or finalizer will be called. As a result, the client programmer has to do all of the end of object clean up explicitly. This once again encourages voodoo programming as we have to copy the shutdown code from usage to usage throughout the system.

So keep in mind the importance of the lifetime of your objects. This is a fundamental feature of object oriented programming that simplifies the use of your classes, and increases their usefulness.

What is a programming idiom?

In a natural language, an idiom is a phrase that conveys more information than the individual words combined. In order to understand an idiom, you must have knowledge of the culture of the person using the idiom, as well as a good grasp of the language.

Programming idioms are similar. They involve uses of particular aspects of the language in ways that convey extra information to the reader of the code. Sometimes idioms are used to expand the abilities of a language. Other times, idioms are used to restrict what you can do with a construct. In all cases, however, the idiom relies on an understanding of the culture from which it is formed.

Paradigms limit possible solutions

Different paradigms are basically different ways of thinking about or looking at problem solving. Programming paradigms usually also include ways of organizing code or design.

The main purpose of any paradigm is to reduce the number of possible solutions to a problem from infinity to a small enough number that you have a chance of picking one. Most proponents of a given paradigm argue strongly that their paradigm of choice removes more possible invalid solutions without seriously impacting valid solutions.

In actual fact, every paradigm eliminates both valid and invalid solutions to any given problem. This is not necessarily bad. However, it does mean that by choosing a particular paradigm, you are closing yourself off from protentially useful solutions or ways of approaching a problem.