Post History

Source: https://writers.stackexchange.com/a/33594
License name: CC BY-SA 3.0
License URL: https://creativecommons.org/licenses/by-sa/3.0/

Source: https://writers.stackexchange.com/a/33594
License name: CC BY-SA 3.0
License URL: https://creativecommons.org/licenses/by-sa/3.0/

It is extremely difficult to measure the performance of a technical document because it is hard to gather the data and hard to interpret the data when you have it.

Let's start with the aim of technical communication. The aim is to make the user of a product productive by enabling them to use the product confidently and correctly. The logical measure of performance, therefore, is user's mean time to productivity.

The problem is, measuring user's mean time to productivity is very difficult. Virtually impossible in many cases. You simply cannot be there to observe them at work, nor can you instrument them or their work or the docs to gather the relevant data.

The Web does let us measure how often a document is read and how long a reader spends on it. The problem is, neither of these is an indication of document performance.

- A technical document gets read when the problem it describes occurs. This has nothing to do with the quality of the document and everything to do with the quality of the product it describes.

- The amount of time that the reader spends reading the document is no measure of its quality, since a good document could give the reader the information they need quickly, while a bad one might force the reader to read to the end and still not tell them what they need to know. 

Finally, there is the issue of the relative value of a document. If the client's business loses a million dollars a minute when the server goes down, then the topic on how to restore the server after a crash is the most valuable topic in your doc set. But if your product is reliable, it will also be one of the least read topics in your doc set. Other commonly read topics may be worth only a few bucks in revenue each time they are read. They will score a lot higher in your metrics, but they deliver far less value in reality.

The best you can really do in many cases is to measure how well your docs ahere to known-good principles of design and rhetoric. It is a very imprecise measure and there will always be debates about which design principles and rhetorical practice best fit the current circumstances. (This is why answers on this board can never be provable in the way answers on SO are provable.)

A number of people have suggested performance measurements over the years but they are all either too expensive or too indirect to be certain. Better than nothing, perhaps, but certainly not definitive, and potentially quite misleading. (The problem with all indirect measurement is that it tempts you to optimize for the metric rather than for actual performance.)

Original score: 10