Programmer Musings

Refactoring, Factoring, and Algebra

I saw the article Refactoring reminiscent of high school algebra on Artima yesterday and it made me remember a new connection of my own.

In the article, David Goodger is working on a refactoring problem that is turning out to be worse than expected. After a little reflection, he manages to solve the problem. But the code had to get messier before it got better. In the end, he makes a connection between refactoring code and multiplying polynomials in high school algebra.

I had made a similar connection several years ago when teaching programming to entry-level programmers. We talked a lot about well factored code and how important it was to keep your code factored. (This was before the term refactoring became mainstream.) I had several students ask how to recognize when the code was well factored.

Now I truly hate to use the answer "experience" when someone asks me a question like this, so I spent some time talking with other senior programmers and thinking about it and came to a realization similar to Goodger's. (Re)Factoring code is like factoring composite numbers. You stop factoring composite numbers when you have nothing but prime factors left. You stop (re)factoring code when the code has reached the point of being minimal. Each piece does one thing and does that well. Your program is then the composite of these fundamental pieces. This is also a lot like factoring polynomials.

Now, I think I would go a little further to say that code that embodies one concept or action or code that has a simple or obvious name is factored enough. To try to factor it any further would make it more complicated.

I think another really important part of Goodger's article was the point that sometimes the code has to be made uglier before you can finish refactoring it.

More Programming Practice

I just recently read Valueless Software, written by Chad Fowler wrote about a year ago. In this article, he describes an approach to practicing music by ignoring the result and focusing on the process. He contends that software people might benefit from a similar approach.

I have actually seen this approach used before in C and in Perl. For years, I read the results of the Obfuscated C and Obfuscated Perl contests with amusement and bemusement. I never attempted them, but I found interesting information in trying to understand them.

In Perl, I have also seen the practices of one-liners, japhs (Just Another Perl Hacker) and Perl poetry as being ways of playing with the language. Of these, I've only spent much time on one-liners. Recently, I've also seen a new game called Perl Golf, the object of which is to reduce the amount of Perl required to produce exactly the same result.

Over the years, I have heard many people describe each of these practices as horrible. They say these practices teach bad programming and encourage the writing of unreadable code. Interestingly, my experience has shown that many people who were extremely good at these practices also tend to write very clear and elegant code. Sometimes they did use more advanced idioms, but only when the situation warranted it.

Once or twice, I have heard people explain that these contests and games help you get the bad stuff out of your system so you can focus on writing clean code where it matters. But this article from Fowler makes me think about it a little differently. Maybe working on one-liners and obfuscated code contests are just another way of practicing our craft. The output doesn't matter. In fact, you know for certain this output will never be used anywhere else. What matters is the understanding of the language you gain from pushing it further than you thought it could go.

Just as importantly, Fowler's observation also covers the other side of this practice. A novice musician needs to practice much differently than an accomplished musician. The novice needs more structure, and needs to practice doing it right. The master musician understands the basics, but he or she still practices to improve. The more advanced practice focuses on refining things that the novice might not even recognize.

Developing software is similar. A novice programmer learns the wrong lesson from trying to build a one-liner to solve a programming problem. The novice needs to learn how to write clear code that solves the problem at hand. The more accomplished programmer should have learned those lessons already. But, to really understand his/her tools, the accomplished programmer needs to push and practice with the edge cases to really grok how the tool works and how to think about problems and solutions.

A Report on UML Fever

ACM Queue often has very insightful articles, and this one is no exception. The article ACM Queue - Death by UML Fever - Are you (or your developers) sick? covers a major problem in software development in a somewhat humorous fashion. I've seen several variations of this problem, but I never thought of it as an illness.

For those who are appalled at the article and who consider the author to be a heretic, you might to step over to another article in the same issue. In ACM Queue - The Fever is Real -, Grady Booch comments on the UML Fever, both the article and the affliction.

Another good piece of commentary on the subject is java.net: Death by UML-more [April 23, 2004].

The most important thing to take away from all of these articles is that UML is a tool (or a set of tools), not a life style, not a religion, and certainly not a solution to all problems everywhere. I'll be glad when more people use the tool was it was intended, instead of using in yet another round of holy wars.

More Thoughts on Mastering Programming

Reading Dave Thomas's blog on Code Kata pointed to PragDave: MoreKata. I also stumbled across How To Practice on Chad Fowler's blog.

Both of these entries discuss exercises that we can use to master the art of programming. Perhaps surprisingly, they use metaphors and concepts from martial arts and Zen. Those of us who have been programming for a while can easily see the parallels.

I'd like to add a description of attaining mastery of a subject that I saw once a long time ago. (I can't find the reference right now, or I would give credit.) The description listed four phases of understanding.

1. Unconscious incompetence
2. Conscious incompetence
3. Conscious competence
4. Unconscious competence

Programming Practice

I'm not quite sure what lead me to this blog entry, but I find PragDave: Code Kata by Dave Thomas to be a very interesting idea. I had never thought of describing coding practice as similar to kata.

I did stumble into a similar idea once many years ago. I had just finished reading Software Tools by Kernighan and Plauger, when I came to the conclusion that implementing some of the classic Unix text tools from scratch would be a good way to improve my programming. None of the programming problems themselves were particularly hard, but I got the idea that building a clean implementation of each tools would help me focus on practical aspects of my programming.

Over the next few months, I built a number of the classic tools. In each case, I worked hard to generate a clean, minimal, and complete implementation of each tool. I went over each piece of code several times to ensure that there was nothing unnecessary or unclear in the code. When I finally declared this experiment finished, I really felt that my coding and design skills had improved dramatically.

I haven't thought about that exercise in a number of years. Now that PragDave has reminded me of it, I think I may need to pick a similar project to see if I can help keep those mental muscles in shape.

The Recurring Myth of Programmer-less Programming

Why is it that some people regularly suggest that someday soon we won't need programmers? I've recently been seeing this as part of the IT outsourcing debate. But, this isn't the first time I've seen it. The first time I saw this meme was shortly after I started programming professionally 17 years ago. I talked to some other people at the time and they said it was old at that time. In all of this time, I have yet to see it happen.

Why does this meme keep coming back? Why are some people so drawn to the idea? And, just as importantly, why is this a myth?

The most recent version of this meme has compared programmers to typists. After all, a decade ago no one would have believed that there would have been no need for typists. But now everyone types their own documents and there is no need for someone who's title is typist. So, with an interesting jump in logic, someday soon we won't need programmers either.

I'm sure that most people developing software would consider the comparison of programming to typing completely invalid. But, how do you explain that to someone who wants to fire the experienced developers to replace them with cheap college students or workers in third world countries? After all, if programming is equivalent to typing, shouldn't we just replace expensive programmers with cheap typists and make more money?

I think this a great example of a mistaken analogy. It would be more accurate to compare typists to scribes and programmers to writers.

There was a time in history when scribes were the only people trained in writing. But much of what the average scribe did was copy or transcribe the works of others. Some scribes were scholars, and many were quite skilled. But, most scribes did not do original writing. As more people learned to write, the need for scribes went away. Much like the use of word processors reduced the need for typists.

However, just because people learned to use pen and ink to write down everyday information, the need for writers did not go away. In fact, writers were more in demand as people became more literate. There is more to writing than the skill of using a pen and ink.

Likewise, there is more to programming than typing on a computer. I've noticed that many people who don't program or who have only a cursory understanding of programming assume that there's not much to it. After all, you figure out what you want to do, you type it in to the computer and the computer does it. How hard can that be?

Much like writing, there is more to programming than meets the eye. I've heard it said that for a good writer, getting an idea is only the beginning. There are hours of research and rewrites and real work that goes into almost everything worth reading. Much like a good piece of writing, a good program may look easy once it's finished, but not everyone is able to research and design and rework to get that finished product.

The two hardest parts of programming are similar to the hardest parts of writing. In both, you have to be able to really think clearly and understand to a degree that most people don't understand. Secondly, in writing and programming, you have to be able to communicate those thoughts.

Unfortunately, in programming you have two very different audiences. The first is the dumbest machine ever invented. Computers do exactly what you tell them to, no matter how dumb the instructions are. No person or animal would ever follow instructions that blindly. The second audience is other programmers (or yourself some time in the future). This audience has to be able to understand what you wrote well enough to adapt it at a later time.

Because this is much harder than almost any non-programmer can fathom, there is this myth that we can replace programmers with something mechanical or with non-programmers. This is much like saying that we'll replace the people that write the company's press releases or contracts with trained monkeys. After all, how hard can it be to crank out this stuff. The answer is (in both cases) some of the writing/programming can be done practically automatically. But the stuff that really matters, the stuff that makes the difference between success and failure needs someone skilled and talented to write it.

Review of Programming Pearls

Programming Pearls, Second Edition
Jon Bentley
ACM Press, Addison-Wesley, 2000

One of my favorite programming books of all time now has a second edition. I've got to say it's as good as the original. Many years ago, I stumbled across Jon Bentley's Programming Pearls. It is a great book on programming. It did not focus on the methodology or programming language of the day. Instead Bentley gave solid, fundamental information about programming. He explained that solving programming problems is about thinking and gave lots of problems to help you think about what you do. After reading Programming Pearls, I found More Programming Pearls, and read that one as well. These books really highlight the difference between a programmer and someone who just slaps code in place to get a paycheck.

The new edition carries on in that tradition. Most of the columns are updated. A few examples have been changed to make sense for today's computing. Most of the code examples are now in C++. But the core of the original information and the thrust of the book remain the same. Bentley's writing style is very entertaining and clear, and his insights are fundamental to the work we do. The examples and problems in the book provide a great opportunity to sharpen your skills.

Every programmer should read this book.

Miller's Magic Number

George A. Miller's paper The Magical Number Seven, Plus or Minus Two: Some Limits on Our Capacity for Processing Information discussed some limits of the human brain with respect to information processing. In particular, his research had found that people are unable to keep up with more than 5-9 different chunks of information at a time. This is actually why phone numbers in the United States are seven digits long, or more accurately, why they used to be an exchange and four digits. (The exchange was eventually replaced by three digits.)

I know a lot of you are thinking that information cannot be true. After all, you know you can keep more things in mind at one time. Right? According to Miller, the key is the word chunks. These chunks can be different sizes. A concept that carries a lot of information is still a single chunk. This is why is is harder to remember 10 randomly chosen numbers or words than it is to remember the words to a song.

A large portion of the history of programming has been devoted to making our chunks more meaningful. Higher level languages allow us to work without keeping trying to remember what each register is holding and how many bytes to we need for that jump instruction. Each succeeding level allows us to keep more in our heads by making the chunks bigger.

But that only works as long as the concepts map well to single chunks. If you don't have a name for a chunk of information or a concept, it takes up more of your memory. One of the more useful effects of Design Patterns was not new OO techniques. It was the new names. Suddenly, you could refer to the Singleton Pattern instead of this class that there's only one of in the whole system but is available to everyone, sort of like global data but not quite.

This same concept applies to user interface design. Grouping related items on the screen and putting the most commonly used items where they are immediately accessible are two ways to reduce the amount of your mind tied up by keeping up with what goes where.

The concept of chunks and Miller's magic number applies in many places in programming. Here's a few to ponder:

• Good variable names make it easier to remember what they are used for.
• Global variables make code more difficult to understand, because you use up chunks thinking about those variables.
• Good method names replace all of the implementation details with one chunk.
• Flags are bad for the same reason as global variables.
• A good programming metaphor helps you develop because more concepts are tied into one chunk.
• Programming paradigms help you solve programming problems by giving you names for concepts you need to work with.
• A programming language is well suited for a problem when you can express your solution in fewer chunks in that language. (Some might say you can express it more naturally in that language.)
• Good data structures help to design by reducing a number of related variables into a single coherent chunk.
• Good classes embody one chunk of information. Bad classes either express multiple chunks or none.

Review of Peopleware

Peopleware
Tom DeMarco and Timothy Lister
Dorset House Publishing Co., 1987

If you are a manager trying to get better work out of your programming staff, you need to read this book. If you are a project lead trying to be determine how to make your team more effective, you need to read this book. If you are a programmer trying to decide if there are any good managers out there, you need to read this book.

Peopleware is the best book I've ever read on the subject of managing programmers. The advice may actually apply to managing other kinds of workers, I'm not qualified to say. DeMarco and Lister really hit the nail on the head when dealing with programmers though.

All of the best managers I've ever had followed at least some of the advice in this book. Some just naturally managed that way, some changed their style after reading this book. On of my best managers once explained the relationship between the manager and team by saying Your job is to make me look good. My job is to give you everything you need to do your job. DeMarco and Lister are much more explicit about their advice, but it basically boils down to that.

As a manager or project lead, you should be focused on removing obstacles and getting your people what they need to do their jobs. This can mean information, or access to people who can give them what they need, or training. What many of the best managers I've had find is that good people will do almost anything if they've got what they need. As a side effect, the manager looks good, because his people are always productive.

This book is also describes the jelled team. Anybody who has ever worked with a team in this state will recognize it. This is the state where the team truly is more than the sum of it's parts. Almost everyone I've know who has been part of a jelled team would be willing to work on almost any project to be part of such a team again.

This book cannot be too highly recommended.