Also, I'm pretty sure I've figured out how to import not only vBulletin 3.x attachments, but any attachments added in 2.x as well. This will be important for boards that have been around since 2003 or earlier (like mine). (ps: what a mess)
As a note to other import makers (especially ones importing smaller boards), I'm running into timeouts quite a bit because of the size of my board. I can't do single-query imports for things like private messages because their format is so different; I need to process each one with PHP. Doing this for 68,800 private messages while trying to thread them into conversations is a problem.
If you're doing PHP processing during an import, be sure to chunk it up to make sure that larger boards can use your importer too. There's a BIG difference between importing a 200-user, 5K-discussion board and a 20K-user, 100K-discussion board!
I've taken to creating "batches". I set a limit to how many it will process at a time, store the ID# it left off with in the config file, and then keep repeating the step (incrementing batch #s so the user knows it's still working), each time starting where it left off, until I run out of records to process. Then it goes to the next step.
@Todd That was an excellent tip on the multiple inserts, by the way. I ran into that big time migrating Wordpress. I'll make sure to build up my queries before sending them; thanks.
Avatars have been imported as Pictures, and private messages have been imported into Conversations.
A) vBulletin doesn't thread private messages (well, not until 3.8 or so anyway). The importer threads them with title/contributor matching instead. That creates some significant overhead so it imports in batches of 200 (and keeps the user updated where it is in the process).
[kvetch] vBulletin creates invalid serialized arrays of who private messages are To (by including members' names which have been passed through htmlentities - semicolons, yay!) and instead of telling me that unserialize was failing, the error backtrace told me a foreach loop was failing AND pointed me at the wrong loop. It took over 7 hours of failed imports to untangle this. >:-| Hate vBulletin. [/kvetch]
What's left: threads/posts and attachments. Then I need to write the password converter so I can actually log into the mess and make sure it works in Vanilla (I'm just eyeballing the database to confirm it's working for now).
@Mark I structured this parallel to the Vanilla importer (i.e. I created a /apps/garden/controllers/importvb.php file and accompanying Views). When it's ready, do you want me to drop those files into my branch and send a pull request, or do you want me to put 'em somewhere else?
@lincoln - that's a good question. I wonder if we should make both of them plugins - so others can use them as guidelines for their custom importer plugins? What do you think?
Sorry to jump in on this conversation but I think importers should be plugins. Adding one or two to the core may set a precedent going forward and it could never end - and for someone like me who is building a brand new website with nothing to import from, I dont really want my installation to have importers that will never get used, just my two cents though...
@Mark@benno What about an importer app? That'd allow us to keep them all in one spot to A) eventually meld them together at our own pace and easily delete it all if you're not importing or done importing.
@Lincoln Sounds like a plan to isolate it. However, depending on the update/upgrade plan for Vanilla 2, when someone downloads the latest tarball/zip and extracts it into their working directory, the "importer" would effectively be re-installed?
@benno Sure, but A) I don't think having import files around is any more of a big deal than having the install files and I've deleted the install files out of every download of vBulletin for the last 6 years so it's not that big of a deal to me I'd value a good, simple first-run experience over potentially adding a step to upgrades if you're worried about having the extra files there.
Fair enough! Maybe I'm biased as I'm building from nothing with a virgin install....
However, on the flip side, most other packages are shipped with an importer of some description.
I know they are not going to do any harm in that folder, providing they are written to be "admin" access only. It'd be a shame to bulk up the Garden application with lots of importers where no more than one would be used at the outset, so if it was to happen I'd vote either plugins / seperate importer application.
On a totally unrelated note - I may however, @Lincoln, come asking for "community building" tips as you seem to have a fairly big setup going! ;-)
My hope re: the importers would be that, as they accumulate, we could merge them into a single comprehensive import process that would minimize how expansive the code would get. I'm pretty sure that's on @Mark's long-term plans too.
Comments
As a note to other import makers (especially ones importing smaller boards), I'm running into timeouts quite a bit because of the size of my board. I can't do single-query imports for things like private messages because their format is so different; I need to process each one with PHP. Doing this for 68,800 private messages while trying to thread them into conversations is a problem.
If you're doing PHP processing during an import, be sure to chunk it up to make sure that larger boards can use your importer too. There's a BIG difference between importing a 200-user, 5K-discussion board and a 20K-user, 100K-discussion board!
I've taken to creating "batches". I set a limit to how many it will process at a time, store the ID# it left off with in the config file, and then keep repeating the step (incrementing batch #s so the user knows it's still working), each time starting where it left off, until I run out of records to process. Then it goes to the next step.
Back now.
A) vBulletin doesn't thread private messages (well, not until 3.8 or so anyway). The importer threads them with title/contributor matching instead. That creates some significant overhead so it imports in batches of 200 (and keeps the user updated where it is in the process).
What's left: threads/posts and attachments. Then I need to write the password converter so I can actually log into the mess and make sure it works in Vanilla (I'm just eyeballing the database to confirm it's working for now).
However, on the flip side, most other packages are shipped with an importer of some description.
I know they are not going to do any harm in that folder, providing they are written to be "admin" access only. It'd be a shame to bulk up the Garden application with lots of importers where no more than one would be used at the outset, so if it was to happen I'd vote either plugins / seperate importer application.
On a totally unrelated note - I may however, @Lincoln, come asking for "community building" tips as you seem to have a fairly big setup going! ;-)
My hope re: the importers would be that, as they accumulate, we could merge them into a single comprehensive import process that would minimize how expansive the code would get. I'm pretty sure that's on @Mark's long-term plans too.
//edit: And I figured out how to make a new repo for the password importer plugin: http://github.com/lincolnwebs/VbulletinPassword