Being the thoughts and writings of one Gustaf Erikson; father, homeowner, technologist.

This category contains posts about books and reading

Thursday, 2006-01-12

The Complete New Yorker on your hard drive

Mr. Jalopy mentions that you can store the contents of the Complete New Yorker on your hard drive. (I got this link from BoingBoing.)

However, the mention is a bit sketchy.

What you have to do is update the table Issues in the SQLite database used by the application. A value of 9 in the field DiskID means hard drive. The rest of the IDs refer to the different disks.

I didn’t notice this, but the very first issue is on the hard drive, which gave anyone with some nous a clue about how this stuff works. If you look in the “Issues” folder of the intallation folder you’ll see a file called 1925_02_21.djvu. This is the first issue.

So you have to copy all the *.djvu files from the separate DVDs (located in the “Issues” folder there) to the “Issues” folder on the harddrive, then update the database.

Warning: I take no responsibility for any stuff that may happen to your installation if you follow these instructions. This said, I believe that if you mess up, you can reinstall the app.

You have to install a SQLite client. I use Cygwin and downloaded the source, compiled and installed. There are precompiled Windows binaries, that I suppose can be used from the Windows command prompt. Mr. Jalopy mentions a graphical application on the internet.

The file you use is ny-sqlite-3.db. Making a backup of this file is a good idea.

$ /usr/local/bin/sqlite3.exe ny-sqlite-3.db

There are two ways to approach this.

Brute force

Simply copy every single file from every DVD to your harddrive. This is a good approach if you have the space (around 50 GB). When you’ve done that, simply run

sqlite> update Issues set DiskID = 9 where DiskID <> 9;

Now all your issues are available from the harddrive.

Disk by Disk

If you don’t have the space, like me, maybe two or three disks will be enough. You’ll have to find which issues are on which disk and update the Issues table accordingly.

Here’s how to find which years have a specific DiskID:

sqlite> select  min(Year), max(Year), DiskID from Issues
        group by DiskID order by min(Year);

Below are the contents of my database.

  • Hard drive: 9
  • DVD 1 (1998—2024): 8
  • DVD 2 (1984—1997): 4
  • DVD 3 (1974—1983): 7
  • DVD 4 (1965—1973): 2
  • DVD 5 (1957—1964): 6
  • DVD 6 (1948—1956): 1
  • DVD 7 (1937—1947): 5
  • DVD 8 (1925—1936): 3

To move DVD 3 to the harddrive, copy all the files, then run

sqlite> update Issues set DiskID = 9 where DiskID = 7;

More SQLite tips

To see the contents of the database:

sqlite> .tables

To see the structure of a specific table:

sqlite> .schema <table>

Update 2006-01-13: I got a mention on BoingBoing. Also, Nick posted an update on how to do this on a Mac, including moving the files to another location! Wow, wish I had symlinks on this OS…

Update 2006-03-02: I got a mention on Securityfocus. The whole article is great, it really points out the absurdity of DRM:

If the goal is just to frustrate users, then why use DRM at all, since you must realize that un-DRM’d copies of your materials are going to circulate? And even if Joe can’t break the DRM, he’ll eventually figure out how to use a P2P network, or ask his nerd friend for help, and then you’ve got another unauthorized copy and an upset and now more knowledgeable former customer. What publisher wants that?

Thanks to all the commenters with their tips and tricks on getting this to work on various hardware and software platforms.

Updated on Thursday, 2006-03-02. Posted at 19:30, in the books category. Comments [20]
Submit this story to: » del.icio.us » digg » reddit. Search for it on technorati.

Comments

Richard Blumberg wrote at 2006-01-12 20:20:

Any ideas about how to copy to an external USB or FireWire Drive? Would that be Drive A?

Gustaf wrote at 2006-01-12 21:42:

As far as I know, there are only IDs for the DVDs (obviously from 1 through 8, and maybe 10 and up). Looking at the structure, I doubt that there's provision for an external drive.

What you can do however is install the application itself on the external disk. Then put all the *.djvu files in the "Issues" folder of that installation folder.

Good luck.

Nick wrote at 2006-01-12 22:10:

Here is how to do it on a Mac (at least, this is what I did), including putting the files on a separate drive from the Application.

1) Installed sqlite3 using Fink.
2) Copied all the "Issues" files from the DVD's to my spare hard drive.
3) Ran the sqlite commands as described above - note that the ny-sqlite-3.db file is located in /Library/Application Support/The New Yorker, as is the Issues folder.
4) Copied the contents of /Library/Application Support/The New Yorker/Issues to the Issues folder on my spare drive.
4) Made a soft link from /Library/Application Support/The New Yorker/Issues to my copy of Issues (note that an alias in the finder did not work - I had to create the soft link on the commandline).
5) Voila! Speedy New Yorker with no DVD swapping.

Larry Racies wrote at 2006-01-16 15:31:

Anyone have any idea how to do this on a machine running Suse Linux?

Gustaf wrote at 2006-01-16 15:34:

I doubt that this works, as the reader application seems to be specific for Windows and Mac.

Maybe you can run it under emulation?

Ogden Nash wrote at 2006-01-17 16:36:

Knowing nothing about databases, this was still very easy for me (WinXP), thanks to the scrivener. (I had the space and so wanted to copy all 50 GB to my C: drive.)

Some add'l tips:

For the database change, I used a free download called Visual SQLite

http://www.download.com/Visual-SQLite/3000-10258_4-10407146.html?tag=lst-0-2

in which you do two operations: browse to the NYer Viewer folder on your HD and open ny-sqlite-3.db, and then in the "execute" field, type in the string

update Issues set DiskID = 9 where DiskID <> 9;

Now you can close the SQL viewer and delete it, you're done.

The other tip is re: copying all the *.djvu files over to the Issues folder. Copying that many files with Windows is irritating: It's slow (30 min per DVD on my box, if all goes well), any failed copy aborts the whole operation, there's no log to see what happened, and in case of failure, there's no way to recopy the files without sitting there and leaning on the N key to say no to overwrite requests.

Instead, download the free Karen's Replicator, which solves all the above problems, plus, if you happen to have two DVD-reading drives, allows you to set up unattended copying on two drives at a time by just setting up two jobs.

The Replicator also seemed to be able to copy files that Windows copy choked on, so that's nice.

http://www.karenware.com/powertools/ptreplicator.asp

Gustaf wrote at 2006-01-17 21:45:

Thanks for those tips! I had problems copying the files too, resorting to Cygwin's bash prompt in the end. The Replicator sounds like a good app to use for this kind of stuff.

jonesey wrote at 2006-01-20 18:54:

So has anyone tried installing the application on a USB or firewire drive yet? You've indicated it will likely work, but I'd sure appreciate a report of success before investing in a drive for that purpose. If so, I will finally feel good about installing my Christmas present for the first time on my space-limited laptop without worrying about disk swapping!

John wrote at 2006-01-28 05:23:

Jonesey, it works. I installed it on an 80-gb Lacie drive and it works beautifully. There's some lag, say 1-3 sec, in opening a new page in a new issue but it's nothing to be bothered about.

john wrote at 2006-01-28 05:56:

Yep, I have it installed on an external Lacie drive. Works well.

jonesey wrote at 2006-01-31 01:00:

Thanks John! I'm assuming the lag is only for landing in a new issue, not a penalty when going from page to page in the same issue. I've been thinking about getting one of those Lacie portable USB-powered drives. How do you like it? Would the New Yorker fit on a 60 Gb drive or is that cutting it too close? If I have 5-6 Gb left over I'm going to install Totally Mad (1952-1998 Mad Magazines and Specials) on the same drive if they're "compatible"?

David Drucker wrote at 2006-01-31 20:38:

One more question about installing to an external HDD: Does the NYer Viewer application have to be installed on that disk as well, in order for the drive numbers to match? I've already got the app on the internal HDD, and wonder whether the database can deal with the issues being on the external.

Someone wrote at 2006-02-04 20:47:

Jonesey, The Lacies are good. I'm extremely happy with hem. This is my second buy, the first one is still going strong after 4 years. Yes, the lag is only for a new issue. I suggest you go for 80gb cos NYer takes 56gb so far and every year they'll add another 6gb. And, yes, you may install Toally Mad, on that, too. BTW, how's this MAD product, scan/quality wise?

David, Yes, you'll have to install it on the external drive. your current install will not recognise the issues on external HDD. To put it simply, all the issues have tobe in the subfolder titled "issues" which is in the new yorker folder that's created when you install the program. Solution: uninstall the program and reinstall on ext. drive.

Jonesey wrote at 2006-02-09 07:08:

I just installed the Totally Mad set and I'm a little disappointed by the quality. I can't compare to New Yorker scans because I have yet installed it. With Mad, there are very noticeable artifacts, and text can be slightly hard to read in small sizes. There are some halftoning/moires issues of course.

But considering the Mad coverage and cost, I'm still happy. Now my challenge is to try to install them to a hard drive to avoid disk swapping, especially noticeable when using their search function. I was able to create image files and mount disk 1 and 2 with Daemon Tools on two virtual drives - it works, I can read issues from both disks. However Daemon is limited to 4 virtual drives and Mad comes on 7. I tried several other techniques, modifying the Mad's CNM.INI file that seems to have settings affecting media storage, but no joy. Any help out there would be appreciated.

Chowdhry wrote at 2006-02-10 18:20:

Has anyone tried to do the same thing with the National Geographic?

John wrote at 2006-02-12 06:43:

Try microsoft's free ISo mounting tool (do a search on their site). It should support as many virtual drives as there are unused drive letters on your system.

Jonesy wrote at 2006-02-13 06:15:

The Microsoft tool works like a charm, I was able to load all 7 CD images from the Totally Mad collection.

The tool is called Virtual CD Control Panel and is about 1/2 way down on the following FAQ page, under MSDN Subscriber Downloads, "What are ISO image files and how do I use them?"

http://msdn.microsoft.com/subscriptions/faq/

or download here: http://download.microsoft.com/download/7/b/6/7b6abd84-7841-4978-96f5-bd58df02efa2/winxpvirtualcdcontrolpanel_21.exe

Unfortunately it won't work in Windows 98SE which I had hoped. The 7 ISO images all fit comfortably on a single DVD.

Jonesey wrote at 2006-02-15 03:03:

Now a problem on New Yorker Complete topic regarding the missing issues. Those who follow this topic know that several issues are not on the DVDs and must be downloaded through the program via Internet. These issues are reported to be:

Sept. 6, 1947; Feb. 20, 1989; April 3, 1989; July 3, 1989; Aug. 14, 1989; Aug. 28, 1989; Oct. 9, 1989; Oct. 16, 1989; and Nov. 27, 1989

Anyway, some will download but several will not. It gets part way through the process then halts and the New Yorker program gives me an error indicating among other things it could be a server error. I have an e-mail in to their support, but wondering if anyone else has run into this.

Jonesey wrote at 2006-02-15 15:49:

I was finally able to download, but here's a public FTP where you can download the missing issues directly.

ftp://69.27.254.40

At the prompt, login:

Username: tcny

Password: guest

There's also update files (probably for Mac and PC) and a ny-sqlite-3.db file in case your original becomes damaged I suppose.

moi wrote at 2006-02-26 05:07:

I loaded the whole thing (NY) on my hard drive and marked the directory as sharable with a share name of 'new yorker'. I could NOT run the .exe on the share from another PC on the network, however, discovered that there is a directory entry [HKEY_LOCAL_MACHINE\SOFTWARE\New Yorker Viewer] "\\myotherpc\new yorker" that, when I installed the software 'locally' seem to make the locally installed software see the database and issues directory on the share.. and work. no guarantees... your mileage may vary...

Submit a comment

Please enter comments as plain text only; HTML is not supported. Submitting an URL is optional.

Comments are moderated and may not appear immediately.

Comments are closed for this story.