I know from a similar German project (http://www.deutschestextarchiv.de) and they have two independent non-German speakers transcribe the digital facsimiles to ensure that the transcriptions are as accurate as possible.
In fact, those mistakes look more like accurate transcriptions of Early Modern manuscripts - with their looser spelling rules and often idiosyncratic use of letters.
It's kind of interesting that they look like the same errors as those generated by OCR.
The difficulty of deciphering the text makes this huge task even more impressive!
It's precisely those idiosyncrasies of early modern orthography which make it difficult to use an off-the-shelf OCR package, which is presumably why these are hand-transcribed instead.
Perhaps there is a specialist antiquarian OCR package which can deal with long s, interchangeable u and v, non-standardised spelling, etc, but I have yet to come across one.
Have you looked at The Early Modern OCR project? My understanding is that they're working on exactly that as well as simply better tools for reviewing & retraining on a large scale:
Not exactly released. And not exactly to public. As the site says "Please note that it is very rare for us to set up free trial access for individuals. Individual pricing is not available and all trial requests will be considered on a case-by-case basis."
That first site is remarkably poorly designed for actually finding the information, a common theme I find for websites created by librarians. There should just be a box at the top listing the download links!
Web developers working for libraries: Far too often I visit a site and and am confronted by acres of text explaining what the project is, who is involved, how to enter data in forms and all sorts of hand holding, BUT NOT THE ACTUAL DATA! Usually I find there's some link hidden in the least visible part of the page, like the lower right hand side, that actually lets me get started! Does Facebook have paragraphs of text with a welcome message from Mark, explanations of what Facebook is, how you use it, or does it just let you dive in and get started?
edit: And despite all that text it doesn't explain what phase I and II are.
I should say this is an amazing bit of work and it's really important that it's being released public domain, and a good sign of the direction libraries are taking. It's just a little frustrating that the final delivery step is so obfuscated.
I would like to point your attention to this:
You don't have permission to access /cgi/t/text/text-idx on this server.
Apache/1.3.39 Server at eebo.odl.ox.ac.uk Port 80
One of the problems searching in these books is that there is no standardized spelling in early modern literature.
There is a project called DREaM, at McGill to standardize for "distance reading" (macro analysis).[1] It uses a program called VARD (a text preprocessor trained to correct spelling).[2]
Strangely, this application is licensed with the creative commons. I think this means that it is closed source. Does anyone know of any open source alternatives?
It cannot handle such an immense amount of data,[3]
There are lots of different CC licenses. None of them are closed-source, although there are some which don't allow modification (No-Derivs / ND). Others disallow copying for commercial use (NonCommercial / NC), and a couple allow almost anything (Attribution / BY, No Rights Reserved / CC0). https://creativecommons.org/choose/
> There are lots of different CC licenses. None of them are closed-source, although there are a few which don't allow modification.
Licenses that don't allow modifications are closed-source. At least, they are inconsistent with the Open Source Definition (specifically, with criteria #3: "The license must allow modifications and derived works, and must allow them to be distributed under the same terms as the license of the original software.")
I think that the distribution of VARD is allowed (if non-commerical), but editing the source code isn't. I think this because I cannot find the source for VARD.
That's why I think it's strange it is licensed CC.
21 comments
[ 3.0 ms ] story [ 39.6 ms ] thread!!! This is not silicon valley. I wonder how they ensure accuracy.
Link to the books http://ota.ox.ac.uk/tcp/
> TO THE RIGHT VVORSHIPFVLL MAISTER RObert Clarke,
The mistakes look like typical OCR errors.
[1]: http://tei.it.ox.ac.uk/tcp/Texts-HTML/free/A01/A01716.html
It's kind of interesting that they look like the same errors as those generated by OCR.
The difficulty of deciphering the text makes this huge task even more impressive!
Perhaps there is a specialist antiquarian OCR package which can deal with long s, interchangeable u and v, non-standardised spelling, etc, but I have yet to come across one.
http://emop.tamu.edu/
Files are hosted by github and box. Will Internet Archive be included?
For marketing this accomplishment, a few choice examples of long-inaccessible text may attract new readers.
That first site is remarkably poorly designed for actually finding the information, a common theme I find for websites created by librarians. There should just be a box at the top listing the download links!
Web developers working for libraries: Far too often I visit a site and and am confronted by acres of text explaining what the project is, who is involved, how to enter data in forms and all sorts of hand holding, BUT NOT THE ACTUAL DATA! Usually I find there's some link hidden in the least visible part of the page, like the lower right hand side, that actually lets me get started! Does Facebook have paragraphs of text with a welcome message from Mark, explanations of what Facebook is, how you use it, or does it just let you dive in and get started?
edit: And despite all that text it doesn't explain what phase I and II are.
I should say this is an amazing bit of work and it's really important that it's being released public domain, and a good sign of the direction libraries are taking. It's just a little frustrating that the final delivery step is so obfuscated.
There is a project called DREaM, at McGill to standardize for "distance reading" (macro analysis).[1] It uses a program called VARD (a text preprocessor trained to correct spelling).[2]
Strangely, this application is licensed with the creative commons. I think this means that it is closed source. Does anyone know of any open source alternatives?
It cannot handle such an immense amount of data,[3]
[1] http://earlymodernconversions.com/introducing-dream/
[2] http://ucrel.lancs.ac.uk/vard/about/
[3] http://www.matthewmilner.name/2014/11/18/VARD-and-EEBO-TCP/
Licenses that don't allow modifications are closed-source. At least, they are inconsistent with the Open Source Definition (specifically, with criteria #3: "The license must allow modifications and derived works, and must allow them to be distributed under the same terms as the license of the original software.")
http://opensource.org/osd-annotated
That's why I think it's strange it is licensed CC.