The Evidence of Things Seen and Unseen

Our first home was unfinished. (It was the only way we could afford one at the time.) We finished the second floor ourselves, and we learned how to build it along the way, so it wasn’t of the highest quality. The house itself wasn’t framed perfectly plumb or square to start with, but houses never are, and we had to square up the interior walls. We did pretty well, though a couple of times I toenailed walls out of position and had to bang them back into place, usually by adding one or two more sixteen-penny nails to get a majority-rule location. Given our limited funds, we used a few pieces of warped lumber rather than buy more, and there were plenty of rose marks where we overhammered, but those imperfections were covered by sheetrock. The sheetrock gapped slightly in a few places, and I misplaced or overdrove a few drywall screws, but I aways added more as necessary, and anyway all that fumbling was covered by a skim coat of plaster. There were a few ripples in the skim coat, but vigorous sanding smoothed most of them down and the remaining flaws we covered with the finish carpentry. (By then I was using a nailset.) And where there were small imperfections in the baseboards, I painted, or we wallpapered, so as to hide them. We sold the house when our family grew, and the new owner today is unaware of where I had to bang a wall true or shim a two-by-four or plaster over an empty screw hole unless he guts the upstairs. (Don’t worry, everything was done to code.) Each layer of work repaired, or hid, the imperfections of the previous steps.

I worked at Digital Equipment Corporation during its halcyon days, when it was the second largest publisher in New England. (IBM was first.) Digital submitted about a third of the entries for our local STC publications competition and won about a third of the awards. Our corporate style was solid and our work deserved the recognition. We also had a full team producing each book: managers, writers, editors, illustrators, production specialists. There was no way for a judge to distinguish between, say, an update that was produced by a good writer needing minimal editorial support and a new document that was produced by a less-than-competent writer who required extensive rewriting, heavy editing, and last-minute assistance. The final results might be equally good—and the competition judged results, not effort—but the effort and workmanship that went into different documents varied significantly. In a sense, an information product that a user finds effective can be “good” in that it was created with minimal effort and fuss, or “bad” in that it was created with much extra effort. That’s the basis of the distinction between product quality and process quality.

Whether you’re building a house or creating an information product, poor workmanship may be invisible to the end user, but it makes building and maintenance slower and less efficient. At one company, while revising a manual, I wondered why adding a few words to certain paragraphs was ruining the formatting. I found non-breaking spaces throughout the source files, and eventually realized they were in material that the previous writer had copied and pasted directly from Lotus Notes email messages. Aside from adding zero value, the writer had created a maintenance problem. At another company, a writer helping out on a large project contributed a dozen screenshots but named the files completely differently from the convention established for the first 300. It was possible to find those rogue files in the art directory, but it took much longer than it should have. A similar problem can occur when a team member doesn’t follow the filename conventions for a document assembled from 500 XML topics stored in a CMS system.

A file can be correctly named but its contents can be incorrectly formatted or tagged. For instance, if headings are mistagged using bold (a presentation format) instead of title (an information type), they will look the same when rendered but can’t be reused, and probably won’t correctly render, in another medium. At his presentations, the always entertaining Neil Perlin sometimes buries his face in his hands to convey the horror of one of his contracts, trying to convert into online help thousands of pages of Word source files, when he discovered that every heading and format was a restyled Normal paragraph. I’ve seen the same problem with files converted to XML format. I can understand how it happens in automated conversion, but it’s lousy if a new writer does it to existing material!

Having judged in publications competitions, I recognize superficial similarities to certification evaluation. In both cases, submissions are evaluated by multiple trained evaluators, working double blind against established criteria. But our certification evaluation has a significant advantage over publications competitions. We ask not just for work samples, but also for for work artifacts and commentaries. We go behind the finished product and examine the workmanship that went into the framing and plastering. We don’t go to the source-code level, but we definitely probe beneath the surface.

For example, we ask for samples of work; but we also ask what the applicant’s role in producing it was, what was planned, why the applicant made the design decisions, and why there are deviations from that design (if there are any). This kind of commentary-based assessment is common in the field of education, and there is called “true assessment.” By not just showing work but also discussing it, the applicant must demonstrate not just competency but conscious competency: this is what I did, and this is why I did it.

In fact, I’m enthusiastic about our criteria for certification, because they allow an applicant to demonstrate competence even in the absence of work samples that demonstrate the competency. Let me explain how. If Digital won any publications awards back in the day because judges liked our use of Chinese Red to indicate user input, it was a mistake, because color is not supposed to sway judging. Why Chinese Red? I don’t know; it was a corporate decision made before I started there. Today blue is much more fashionable. But why? If you ask writers, illustrators, or document designers today why they use blue highlights in their work, you will get a range of answers (one of which is my current answer):

• The last writer used blue, and I’m just doing an update.
• I was told to use blue.
• The template uses blue.
• Blue is our corporate logo color.
• I don’t know why I use blue.
• I think blue is pretty.
• I think blue is restful.

To be clear, we don’t ask about highlight color as part of our certification. But given our method of evaluation, if we did, even an applicant whose work samples are all in black and white could discuss color design decisions. You can show us what you do, yes; but you can also show us what you can do.

Our evaluation methodology levels the playing field between applicants who work in large organizations and those who work in small groups, or who work alone. We are able to get beneath the surface and see the workmanship.