Please upgrade here. These earlier versions are no longer being updated and have security issues.
HackerOne users: Testing against this community violates our program's Terms of Service and will result in your bounty being denied.

Building a vBulletin 3.x to Vanilla 2 importer

245

Comments

  • LincLinc Detroit Admin
    edited October 2009
    Also, I'm pretty sure I've figured out how to import not only vBulletin 3.x attachments, but any attachments added in 2.x as well. This will be important for boards that have been around since 2003 or earlier (like mine). (ps: what a mess)
  • ToddTodd Chief Product Officer Vanilla Staff
    @Lincoln this is seriously awesome stuff. I think the user meta table would also make a good general purpose plugin.
  • Very interested in this. Thanks! (And good luck!)
  • LincLinc Detroit Admin
    Thanks @Todd :)

    As a note to other import makers (especially ones importing smaller boards), I'm running into timeouts quite a bit because of the size of my board. I can't do single-query imports for things like private messages because their format is so different; I need to process each one with PHP. Doing this for 68,800 private messages while trying to thread them into conversations is a problem.

    If you're doing PHP processing during an import, be sure to chunk it up to make sure that larger boards can use your importer too. There's a BIG difference between importing a 200-user, 5K-discussion board and a 20K-user, 100K-discussion board!

    I've taken to creating "batches". I set a limit to how many it will process at a time, store the ID# it left off with in the config file, and then keep repeating the step (incrementing batch #s so the user knows it's still working), each time starting where it left off, until I run out of records to process. Then it goes to the next step.
  • ToddTodd Chief Product Officer Vanilla Staff
    Yeah, that's why the Vanilla 1 import chunks things. Another note is if you put a series of inserts into one insert statement then it is waaay faster.
  • LincLinc Detroit Admin
    I haven't abandoned this; I've had to reorder my priorities twice since my last post. I'm probably 2 weeks away from getting back to this. Mah bad :(
  • LincLinc Detroit Admin
    edited November 2009
    Or 4 weeks. Close enough? (I was redesigning Icrontic.com and moving it from 3 Wordpress blogs to Wordpress MU).

    Back now.
  • LincLinc Detroit Admin
    @Todd That was an excellent tip on the multiple inserts, by the way. I ran into that big time migrating Wordpress. I'll make sure to build up my queries before sending them; thanks.
  • LincLinc Detroit Admin
    edited December 2009
    Avatars have been imported as Pictures, and private messages have been imported into Conversations.

    A) vBulletin doesn't thread private messages (well, not until 3.8 or so anyway). The importer threads them with title/contributor matching instead. That creates some significant overhead so it imports in batches of 200 (and keeps the user updated where it is in the process).

    B) [kvetch] vBulletin creates invalid serialized arrays of who private messages are To (by including members' names which have been passed through htmlentities - semicolons, yay!) and instead of telling me that unserialize was failing, the error backtrace told me a foreach loop was failing AND pointed me at the wrong loop. It took over 7 hours of failed imports to untangle this. >:-| Hate vBulletin. [/kvetch]

    What's left: threads/posts and attachments. Then I need to write the password converter so I can actually log into the mess and make sure it works in Vanilla (I'm just eyeballing the database to confirm it's working for now).
  • Keep it up thohugh nie work
  • LincLinc Detroit Admin
    @Mark I structured this parallel to the Vanilla importer (i.e. I created a /apps/garden/controllers/importvb.php file and accompanying Views). When it's ready, do you want me to drop those files into my branch and send a pull request, or do you want me to put 'em somewhere else?
  • MarkMark Vanilla Staff
    @lincoln - that's a good question. I wonder if we should make both of them plugins - so others can use them as guidelines for their custom importer plugins? What do you think?
  • Sorry to jump in on this conversation but I think importers should be plugins. Adding one or two to the core may set a precedent going forward and it could never end - and for someone like me who is building a brand new website with nothing to import from, I dont really want my installation to have importers that will never get used, just my two cents though...
  • LincLinc Detroit Admin
    edited December 2009
    @Mark @benno What about an importer app? That'd allow us to keep them all in one spot to A) eventually meld them together at our own pace and B) easily delete it all if you're not importing or done importing.
  • @Lincoln Sounds like a plan to isolate it. However, depending on the update/upgrade plan for Vanilla 2, when someone downloads the latest tarball/zip and extracts it into their working directory, the "importer" would effectively be re-installed?
  • LincLinc Detroit Admin
    edited December 2009
    @benno Sure, but A) I don't think having import files around is any more of a big deal than having the install files and B) I've deleted the install files out of every download of vBulletin for the last 6 years so it's not that big of a deal to me :) I'd value a good, simple first-run experience over potentially adding a step to upgrades if you're worried about having the extra files there.
  • Fair enough! Maybe I'm biased as I'm building from nothing with a virgin install.... :)

    However, on the flip side, most other packages are shipped with an importer of some description.

    I know they are not going to do any harm in that folder, providing they are written to be "admin" access only. It'd be a shame to bulk up the Garden application with lots of importers where no more than one would be used at the outset, so if it was to happen I'd vote either plugins / seperate importer application.

    On a totally unrelated note - I may however, @Lincoln, come asking for "community building" tips as you seem to have a fairly big setup going! ;-)
  • LincLinc Detroit Admin
    edited December 2009
    @benno Any time :)

    My hope re: the importers would be that, as they accumulate, we could merge them into a single comprehensive import process that would minimize how expansive the code would get. I'm pretty sure that's on @Mark's long-term plans too.
  • LincLinc Detroit Admin
    edited December 2009
    Sweet, sweet victory. I logged into Vanilla tonight with my freshly-imported vBulletin account, old password and all.

    //edit: And I figured out how to make a new repo for the password importer plugin: http://github.com/lincolnwebs/VbulletinPassword
Sign In or Register to comment.