Ask HN: How can I automatically scan and catalog a mountain of books?
He buys bulk lots from deceased estates and bookstores that are closing down. Entire shipping containers are being gifted to him and showing up at his barn. The barn is full and he is now storing in shipping containers outside.
There is great quality books among this quagmire but it takes hours of searching to find them. I figured HN might be able to point me to a solution where I could quickly photograph the front cover and have a script/google images compare the image to online info to index the title and author and then perhaps list them online...
I dunno, it just seems like such a treasure trove of books that he will sell for practically nothing because he loves books and hopes that they will find their way to people who want them - the barrier is allowing customers to find what they are looking for.
Thoughts?
141 comments
[ 2.9 ms ] story [ 192 ms ] threadDefinitely not a "scan/beep/scan/beep" kind of thing. More like "scan . . . uh, scan . . . scan, damn you, SCAN I say! (beep) Okay . . . now the first problem is that 'The Sands of Mars' which I am holding is definitely not 'Great Montana Flapjack Recipes' on B&N, let's try the library of Congress . . . . nope, not 'Annals of 1959 Steelmaking', so (tap tappity-tap...)"
What was causing the mismatches? Bar codes that did not contain the ISBN? Non-unique numbers?
https://www.cs.bgu.ac.il/~ben-shahar/Teaching/Computational-...
Won't be as accurate as barcode scanning, but will be definitely less time consuming.
Annoyingly, I was trying to scan multiple books in charity shops, when one of my favourites started putting their own stickers directly over the barcode (when it existed).
The API is severely rate-limited (1rps), non-standard oath and badly documented, but you should be able to get some xml out of it and parse that however you'd like.
Goodreads cannot recognize anything, but since it works on either book cover or barcode it will work on lots.
If a book has an ISBN, often Zotero will manage to find it using the magic lookup button. Just enter the ISBN (DOI's work too!) and it will usually find the book you meant. That covers about 90% of books with an ISBN.
The rest would have to be entered manually.
Zotero is not a full-blown inventory manager, but it may suit your needs.
If they really do need to be cataloged, then the next thing is to forget all about trying to inventory the entire thing. Instead, you're going to partition the collection into "easy to catalog" and "hard to catalog": pick a section of the barn and make this the organized area. Get a barcode scanner (https://www.newegg.com/Barcode-Scanner/SubCategory/ID-583) and throw together a quick API client that'll take an ISBN and display a title, author, edition, and picture. If it comes up correct, great: book goes into the cataloged section. If it doesn't, it goes somewhere else. Make it really simple, so that a single keystroke can accept that book into inventory.
Grocery stores have to regularly inventory everything on the shelves. I worked for an outfit once that wanted to do it all in-house, so we bought the commercial Telxon handheld wireless devices and I set about figuring out their software. Turned out that they just wanted to speak basic telnet to a server at a pre-configured IP address, so I put together a sloppy little telnet server interface and staff were able to count the entire store right on the devices in a few hours. That's way more complicated than what you'll need to do, so, y'know, your thing is doable. You'll have the added benefit of free online book databases and better hardware and easier-to-hack-together software.
Also might not be a bad idea to talk to your local librarian. They're book nerds too and he or she might have an actual library science degree. This would be right up their alley.
For insurance purposes, probably, yeah, especially if there's anything rare, antique, valuable, etc.
https://dearauthor.com/ebooks/dear-jane-ebooks/dear-jane-can...
I second this. Also, maybe check with university libraries or university MLS/MLIS programs (Masters of Library Science). This is not a new problem, and they would be aware of existing tools/methodology. Also, maybe you could get a grad student/intern to help.
I've seen a few of these, and the basic minimum difference between "pulp waiting to happen" and "bookshop" is basic shelving. Different shelves by category: fiction vs non-fiction and their subdivisions. Within the shelves, alphabetise. Now it's possible for browsers to actually find things. When you put them on Amazon this will also help find them for shipping.
This process will also help you find the stacks of duplicates. You'll have a crate of 50 Shades and Twilight and Stephen King. The Stephen King will eventually resell; the others won't.
This page from the excellent Barter Books on their acceptance policy may be of some help: https://www.barterbooks.co.uk/html/About%20Us/Incoming%20Boo...
Shelving in instead central to many specialty shops!
Twenty unsorted banana boxes: every book just becomes a blur (as in, “Oh great, another Danielle Steele novel, and another bible, and another Excel 97 for dummies, …”).
Most people do not enjoy that more then once or twice.
In a bookstore, this is a virtue. But we appear to be dealing with a book barn. Perhaps the patrons of book barn have not wandered in by accident while searching for a bookstore :)
For OP: I think you might be better off photographing the ISBN and then using a service or script to do a lookup and associate that with a cover and title. Various editions might not have their covers recorded in a database, and titles will give many dupes, but an ISBN will uniquely identify a book.
Additionally, OCR'ing stylized text is problematic. I wouldn't expect easy reading of covers, particularly of used books.
The answer to this is important in order to properly size the effort. If the goal is to impove the business efficiency of the store, then it should be seen from ROI stand-point. Even the fellow customers/rummaggers could be engaged with a right incentive and tools. Otherwise, the next estate books container shipment will negate the gains of ordering.
Basically, is OP ready to overhaul the operations or is just willing to do something nice just for now?
Ideally, a book cover and info page should be scanned/photo'ed on first touch either by receiver or shopper and sticker coded somehow as processed, then left wherever. In case any jewel-worthy titles uncovered from OCR, the book could be located (stickerwise, date log, crate, whatever) and brought to prominence and priced as appropriate.
Unleashing imagination, as an incentive and QC strategy - some sort of automated OCR and lookup could locate the pointed book on amz or elsewhere and thus reviews and going price vs rummager's deal.
Either way, I'd see this more of a business question, rather than a technical one. Donating a technical solution is fun, but without changing the operations process is not going to be sustainable.
I worked on a similar project, with thousands of books to organize. We did just that, because it seemed common sense. We had a couple of issues:
1. We found that sorting books by category took quite a lot of time. For every book we had to spend a couple of seconds deciding in which category it should go. By the time we started wishing we had put them all together, sorted only by alphabet, it was already late.
2. Shelving books alphabetically is a problem when they keep coming and going. How do you know how much space to leave on the shelf for a specific letter? Maybe now you have 10 books by Stephen King, but the next container will have 50 more. If you don't leave enough space you'll have to shift all the books after that letter.
This is what card files were originally invented for— number all the shelves, put your new book wherever there’s room, and put a card with the title and shelf number in a sorted drawer. When you get rid of the book, get rid of the card.
That being said, if you do want to do image comparison for covers, books without covers usually have a copyright page with most of the info on it. Use that to determine what a book is when the other method fails. Throwing together some cheap bookshelves with plywood and 2x4's will greatly help with the finding part, but while scanning use some big bins to do a rough sort.
And I can't stress enough you HAVE to throw out books. It's clear that there's a space issue and if he's willing to get them for free but has a hard time getting rid of them, that's hoarder behavior, not just eccentricity.
http://tellico-project.org/
I worked at a small startup in the early 2000s that somehow got massive contract to digitalise a Middle Eastern Oil & Gas company's very very extensive documentation library. We had an e-learning product where you could use a scanner to digitalise a printed book into online documentation.
Demos of scanning a book or two was really impressive. So surely scanning more than a million books/manuals/charts will be just as easy. Not quite.
Think we calculated it would take years as the bottleneck is the manual unbinding and re-binding before and after scanning. Scale that to a million and it was not the 2 months project initially forecasted. Buying more scanners and hiring more local staff scaled that part horizontally and improved the speed but still a long project.
However the client "forgot" to pay us for a few months, the bank and our accountants forgot to check and we went bankrupt soon before we really got started. Though at least I got a trip to the Middle East for a few weeks.
Otherwise, as stated elsewhere in this thread, Zotero can usually find books with very little information:ISBN or title. It might be worth trying to set up an OCR with it.
In any case, if you go to the length of taking a picture for each book, you might as well save them and make the dataset public, for OCR training purposes (and a second pass). There is also the mechanical Turk option if you go this way.
And as someone stated already, you should plan the physical layout in advance.
Scanning is only worthwhile for books which are not already available in electronic form ... somewhere.
If you bought from Amazon, there's sometimes an option to get the ebook cheaply. Archive.org has many books. There are also e-books at public libraries, so it may be enough to keep a list/photos/calibre of all your titles and discard rarely-accessed books.
http://eresources.loc.gov
though this is to scan whole books in case there are rare ones..
It also shouldn't be hard to up the speed by taking pictures of a stack of books - if you take an image of a stack of books and crop it book by book, training models to recognise books shouldn't be that hard but you could also use a CV solution (firebase, amazon, azure again) and then from the books it found in the stack ping the API for each one. This could probably be the fastest way if you can take a panorama and have it search from that.
Anyways, if you do it - try to get the price, ISBN and editions from the results.