I have been cooperating on and off with the good people from Dot.Tunes, based in Australia. I have coded bits of the application, notably the iTunes Library XML file import engine. Back when they integrated my engine, D-T saw huge improvements while importing the XML database, especially with large files. If memory serves, the previous import engine stalled on some files. With the first two or three version of my code, large files took some time to import, but at least D-T didn’t fail. When I mean large, I mean real large. Like 50,000 record-large. 35MB+ of XML to digest and convert to sqlite. Gulp.
50K records took, give or take, 20 minutes. Note the use of past tense. Took. Now, it takes 3 minutes. Yessir. On the same machine. But to achieve that I had to break out the big guns. The real big ‘uns. I had been playing for a while with the idea of writing some iTunes/mp3/etc related code in Erlang and possibly Yaws. So one of the things I started writing is a new engine for the iTunes Library XML format, using message passing, concurrency and gazillions of lightweight threads.
The result is astounding. 2,000 records take 2 seconds on my MacBook, and/or a late-model PowerMac G5. 50,000 records take three minutes on that same PowerMac G5 – I don’t have yet the file to make the tests, and I am quite curious to see how the MacBook will fare. To be fair, there’s extra overhead then to inject the .sql file into an sqlite database, but it’s minimal. The only problem is that we’ll need to ship CEAN, a stand-alone version of Erlang. That adds +/-12MB to the package, but I think it’s well worth it.
We’re still in the testing stages, and Jeff over at D-T will certainly make announcements when we’re ready, but I am already quite excited by the progress we’ve made….