Programmer Musings

Thoughts on Measuring Proficiency

In On Proficiency, I talked a bit about what characteristics make up the concept we call proficiency. In trying to hire programmers, a company often tries to measure proficiency to find the best candidate for the job. Over a year ago, I began thinking about how to measure proficiency, and found it to be harder than I thought.

Unfortunately, proficiency is not a simple variable that can be measured. A particular person is likely to rate differently on each of the four aspects of proficiency. Some will have great theoretical knowledge, but no skills. Others will have skills and aptitude to spare, but lack the theoretical knowledge to be efficient.

Most of these aspects are a spectrum. A person may have a strong aptitude or none at all, someone else may be in between. A person may have a little interest, a lot of interest, or would rather have a root canal than even think about it. These aspects are more or less outside a person's control and are easy to recognize if you see the person working in the field. Someone with no aptitude will not be able to make the simplest things work. Someone with interest and a great deal of aptitude will be able to adapt to new requirements or approaches fairly quickly.

The other two aspects are also not simple variables. A person's knowledge of a field may be broad and shallow, deep and narrow, or somewhere in between. The skills a person brings to the job may involve heavy experience with a few tools and techniques, light experience with a large number of tools and techniques, or some combination. Although, these aspects are the easiest to measure, they are also very easy to misinterpret.

Many employers look for the deep and narrow knowledge, asking for people that have spent the last 10 years doing essentially the same work that the employer needs. On the other hand, someone with a broader knowledge of the field can bring insights and new ways of approaching problems. If the person's knowledge is too broad, they probably have not had time to develop any depth in a particular area. However, someone who has knowledge of several fields has probably shown the ability to adapt and learn as needed to solve problems. This is an important characteristic. It is possibly more important than extremely deep knowledge of the topic of the day.

A person's skill set is relatively easy to measure. Can the candidate do task X with technology Y? Has the person ever used technique Z? However, the absolute answers to these questions are less important than many people assume. If a person has never used a particular skill, but is experienced with a related skill, the lack may not be a problem. A practical example would be does a programmer know a particular language. If the answer is yes, then you know he/she can use the language. If the answer is no and they've only used one language, then you know they won't be immediately useful. If, however, the programmer has experience working in several languages, the actual yes or no answer is less important. He/she will probably be able to pick up the syntax of the language and work almost as effectively as the person who answered yes. More importantly, if the language of the day sinks beneath the waves and a new language is required, this programmer will be able to adapt.

What makes measuring these even harder is that the different aspects interact in interesting ways. We all know the programmer that has an amazing amount of theoretical knowledge, seems to know every aspect of the tools he/she uses, and is extremely interested in the topic, but always seem to solve the same problems, over and over again. In this case, several attributes combined in a way that is detrimental to the programmer involved.

Many of us have also seen the bright entry-level guy/gal that has interest and aptitude galore, who spends much of his time reinventing known solutions to known problems because he lacks the theoretical knowledge that would prevent this waste of time.

Sometimes we've also seen the one that doesn't know the language you are working with, hasn't used any of your tools, but after a month is the person everyone goes to for help with design or style or even code.

After over a year of pounding on this idea, I still don't feel that I have any insights into how to really measure proficiency in our field, except work with a person long enough to recognize some of his/her strengths and weaknesses. I've seen many people hired that I thought would be a good fit and find them lacking in one or more of the attributes we need. I've also seen some that looked like a complete waste of time, that turned out to be just what we need.

If anyone has found any wonderful insight into this issue, I'd really love to hear it.

Review of Hardening Linux

Hardening Linux
James Turnbull
Apress, 2005.

I've read several books over the years focused on increasing the security of a Linux system. I think Hardening Linux may be the best example I've seen of a practical book on the subject. Turnbull walks through a number of potential security vulnerabilities and gives specific advice on locking them down.

The book includes the expected grounding in basic security principles like denying access by default and strong passwords. Even though Turnbull spends most of the book on securing individual vulnerabilities, the book is not just a checklist of things to change. He also explains the reasoning behind the changes and some of the side effects that you should be aware of. This explanation is probably the most important feature of the book. If you need to set up a bastion host, secure a mail server connected to the Internet, or just want to know something about the dangers of providing services to the Internet, this book gives you a strong grounding in the issues that determine the security of a server.

As a good example, I have read several articles and books that tell you how to configure a firewall. But, I don't think I have ever seen a better description of why you make some changes and not others. I particularly liked Turnbull's discussion of the issues with configuring a firewall remotely. Anyone who has accidentally locked themselves out of the system they are configuring would appreciate his advice.

I especially liked the the chapter on securing FTP. Turnbull firmly states that FTP is extremely hard to secure. He covers the various reasons why it is hard to secure. He states firmly that in the interest of security, you should not run an FTP server. Then, he goes step-by-step to show how to provide as secure an FTP server as possible if you really have to have one. This stands in stark contrast to an academic approach of ignoring systems that can't be made secure. Turnbull does a great job of making that point, and then bowing to the necessity that some people will need the service anyway.

Although I really enjoyed the book, it did have a couple of flaws. One is inherent in the topic: this is a really dense book. If you don't do serious system administration on a daily basis, you may find this book to be slow going. I sure did. However, this material is well covered despite being dense. Working through the material is worth the effort. The other problem has to do with editing. I found a relatively large number of minor typos, grammar errors, and mistakes in examples. Although it was still possible to understand what the author was saying, I feel that these errors made the task of working through the book a little more difficult.

Overall, I would definitely recommend this book to anyone who needs to secure a Linux server for any reason. If you are interested in what needs to be done to secure a mail server, DNS server, FTP server or any other public server, this is definitely a book you need to read.

Conversion to Subversion: Tags Revisited

In Conversion to Subversion: Tags, I explained how I cleaned up the tags portion of the CVS dump file in order to generate a subversion repository in the format I wanted. Lars Mentrup emailed to say that I had not been clear enough on a few points. After re-reading what I wrote, I have to agree.

In the original article, I referred to working on the initial copy from trunk. This was obviously missing a subtle point. All of my changes started with the dumpfile that had been filtered to contain the project and tag entries. Given this filtered file, I was talking about the initial copy of the tags. This is the equivalent of

  cp 'trunk' 'tags/FIRST_RELEASE'

Now, I really did not want the entire tree copied to the tags directory, and I wanted the tags directory structured differently. So, I changed the equivalent of the above command to the equivalent of

  cp 'project1/trunk' 'project1/tags/FIRST_RELEASE'

Now the dumpfile has a large number of extraneous delete commands that are the equivalent of

   rm tags/FIRST_RELEASE/project2
   rm tags/FIRST_RELEASE/project3
   rm tags/FIRST_RELEASE/project4
   ...

Since I changed the initial copy, these are no longer needed. So, I just delete them from the dumpfile.

The result of all of these changes is a single dumpfile with the 'project1' project information and the 'FIRST_RELEASE' tag information. I can use svnadmin load to load the project and its tags into the repository.

Hopefully, this will clarify what I meant. Thanks again to Lars for taking the time to let me know where I was unclear.