Programmer Musings

Thoughts on Measuring Proficiency

In On Proficiency, I talked a bit about what characteristics make up the concept we call proficiency. In trying to hire programmers, a company often tries to measure proficiency to find the best candidate for the job. Over a year ago, I began thinking about how to measure proficiency, and found it to be harder than I thought.

Unfortunately, proficiency is not a simple variable that can be measured. A particular person is likely to rate differently on each of the four aspects of proficiency. Some will have great theoretical knowledge, but no skills. Others will have skills and aptitude to spare, but lack the theoretical knowledge to be efficient.

Most of these aspects are a spectrum. A person may have a strong aptitude or none at all, someone else may be in between. A person may have a little interest, a lot of interest, or would rather have a root canal than even think about it. These aspects are more or less outside a person's control and are easy to recognize if you see the person working in the field. Someone with no aptitude will not be able to make the simplest things work. Someone with interest and a great deal of aptitude will be able to adapt to new requirements or approaches fairly quickly.

The other two aspects are also not simple variables. A person's knowledge of a field may be broad and shallow, deep and narrow, or somewhere in between. The skills a person brings to the job may involve heavy experience with a few tools and techniques, light experience with a large number of tools and techniques, or some combination. Although, these aspects are the easiest to measure, they are also very easy to misinterpret.

Many employers look for the deep and narrow knowledge, asking for people that have spent the last 10 years doing essentially the same work that the employer needs. On the other hand, someone with a broader knowledge of the field can bring insights and new ways of approaching problems. If the person's knowledge is too broad, they probably have not had time to develop any depth in a particular area. However, someone who has knowledge of several fields has probably shown the ability to adapt and learn as needed to solve problems. This is an important characteristic. It is possibly more important than extremely deep knowledge of the topic of the day.

A person's skill set is relatively easy to measure. Can the candidate do task X with technology Y? Has the person ever used technique Z? However, the absolute answers to these questions are less important than many people assume. If a person has never used a particular skill, but is experienced with a related skill, the lack may not be a problem. A practical example would be does a programmer know a particular language. If the answer is yes, then you know he/she can use the language. If the answer is no and they've only used one language, then you know they won't be immediately useful. If, however, the programmer has experience working in several languages, the actual yes or no answer is less important. He/she will probably be able to pick up the syntax of the language and work almost as effectively as the person who answered yes. More importantly, if the language of the day sinks beneath the waves and a new language is required, this programmer will be able to adapt.

What makes measuring these even harder is that the different aspects interact in interesting ways. We all know the programmer that has an amazing amount of theoretical knowledge, seems to know every aspect of the tools he/she uses, and is extremely interested in the topic, but always seem to solve the same problems, over and over again. In this case, several attributes combined in a way that is detrimental to the programmer involved.

Many of us have also seen the bright entry-level guy/gal that has interest and aptitude galore, who spends much of his time reinventing known solutions to known problems because he lacks the theoretical knowledge that would prevent this waste of time.

Sometimes we've also seen the one that doesn't know the language you are working with, hasn't used any of your tools, but after a month is the person everyone goes to for help with design or style or even code.

After over a year of pounding on this idea, I still don't feel that I have any insights into how to really measure proficiency in our field, except work with a person long enough to recognize some of his/her strengths and weaknesses. I've seen many people hired that I thought would be a good fit and find them lacking in one or more of the attributes we need. I've also seen some that looked like a complete waste of time, that turned out to be just what we need.

If anyone has found any wonderful insight into this issue, I'd really love to hear it.