It may be difficult for software engineers to appreciate that a literary index is a work of art, and, done right, most usually a labour of love.
In software, indexing is a mechanical process, subject to automation. In literature, indexing is a subjective art, requiring enormous cognitive effort and juxtapositions of intellectual scale to achieve a useful, informative result.
When is an index more useful than full text search? I derive a lot of pleasure when I can open a huge text on my laptop and instantly find exactly what I'm working for by searching for one or two words.
An index is a curation of content. If done well it saves you the time that you would otherwise spend trying to craft exactly the right search expression to find what you're looking for from a full-text search.
Indexes are conceptual and semantic, they're curated.
Full-text search is typographical, it finds string literals (if you're lucky, regexes). But common terms or strings which appear frequently as substrings of larger words will trip you up (again: regexes can handle word-boundary search).
But trying to find a concept, such as the C, B, D, S or R programming languages, the X Window System, the band The The, or Alex Jones, journalist, vs. Alex Jones, batshit crazy conspiracy fraudster, won't be well servee you by FTS. An index will reveal those.
The distinction has some similarities to how Web search these days tends to expand search terms entered to attempt to interpret meaning rather than literal text. At Web scale, that's sometimes useful, often not. Within a single book where context is far more likely to be constrained, and the search space a few billion times smaller, the approach is much more useful.
Another thing an index offers is the ability to turn to it and get a conceptual overview of a book independent of the table of contents. At times when there are combatting ideas or authorities, it's helpful to be able to open a book, flip to the back, and see if it mentions some concept at all, as well as to see, at a glance, if it's a very brief mention (a single page), an extended discussion (several pages), or repeated occurrences throughout the work (numerous sections and references). (Mortimer Adler's How to Read a Book specifically recommends doing this as part of an initial assessment of a work.)
An index will also break down complex topics by listing out subdivisions, say, "policy", with breakdowns of military, foreign, domestic, energy, trade, crime, justice, civil rights, etc., etc.
I'm also a fan of FTS, and it can be exceptionally useful. But it is not a total replacement for well-constructed, thoughtful indices.
One of my favorite side stories in "The Name of the Wind" by Patrick Rothfuss is the University library, where they reveal a history of hard fought, vicious battles between factions of librarians, over distant catalog and indexing systems. It drives home how complex and vast a task it must have been to organize large numbers of books before computers.
Rereading this right this very moment and that part is totally awesome. Also, all the money calculations in that book showing how hard it is to be a poor student.
But I am grinding my teeth because there is no book 3! When! If ever? Patrick, come on, man, you did such a good job on these two, finish it!
"Long ago I had dinner with six French copy editors. Authors, they told me, work by simple principles: never check a fact or a date, never verify a reference and never spell a proper name the same way twice."
I was about to quote Laurence Sterne, "They order, said I, this matter better in France." But then I read "long ago", and I wouldn't bet that copy editors have hung on any better in France than in the United States.
(I did not consult an index for this comment, but I did go to gutenberg.org to verify, and it turned out, correct the quotation.)
I do use indexes to books--I am no scholar--chiefly when I remember that the author said such-and-such about so-and-so, and haven't the time to read through all the many pages to track the reference down.
13 comments
[ 3.4 ms ] story [ 75.4 ms ] threadIn software, indexing is a mechanical process, subject to automation. In literature, indexing is a subjective art, requiring enormous cognitive effort and juxtapositions of intellectual scale to achieve a useful, informative result.
1. Cross-references to other index entries.
2. Listings for synonyms of the thing you're looking for.
3. Sub-entries under the primary thing, like:
Full-text search is typographical, it finds string literals (if you're lucky, regexes). But common terms or strings which appear frequently as substrings of larger words will trip you up (again: regexes can handle word-boundary search).
But trying to find a concept, such as the C, B, D, S or R programming languages, the X Window System, the band The The, or Alex Jones, journalist, vs. Alex Jones, batshit crazy conspiracy fraudster, won't be well servee you by FTS. An index will reveal those.
The distinction has some similarities to how Web search these days tends to expand search terms entered to attempt to interpret meaning rather than literal text. At Web scale, that's sometimes useful, often not. Within a single book where context is far more likely to be constrained, and the search space a few billion times smaller, the approach is much more useful.
Another thing an index offers is the ability to turn to it and get a conceptual overview of a book independent of the table of contents. At times when there are combatting ideas or authorities, it's helpful to be able to open a book, flip to the back, and see if it mentions some concept at all, as well as to see, at a glance, if it's a very brief mention (a single page), an extended discussion (several pages), or repeated occurrences throughout the work (numerous sections and references). (Mortimer Adler's How to Read a Book specifically recommends doing this as part of an initial assessment of a work.)
An index will also break down complex topics by listing out subdivisions, say, "policy", with breakdowns of military, foreign, domestic, energy, trade, crime, justice, civil rights, etc., etc.
I'm also a fan of FTS, and it can be exceptionally useful. But it is not a total replacement for well-constructed, thoughtful indices.
But I am grinding my teeth because there is no book 3! When! If ever? Patrick, come on, man, you did such a good job on these two, finish it!
https://news.ycombinator.com/item?id=28597724
Also reviewed in History Today 21 days ago:
https://news.ycombinator.com/item?id=28430765
I was about to quote Laurence Sterne, "They order, said I, this matter better in France." But then I read "long ago", and I wouldn't bet that copy editors have hung on any better in France than in the United States.
(I did not consult an index for this comment, but I did go to gutenberg.org to verify, and it turned out, correct the quotation.)
I do use indexes to books--I am no scholar--chiefly when I remember that the author said such-and-such about so-and-so, and haven't the time to read through all the many pages to track the reference down.