Please upgrade here. These earlier versions are no longer being updated and have security issues.
HackerOne users: Testing against this community violates our program's Terms of Service and will result in your bounty being denied.

Importing Custom Data

I want to import some discussion threads from forums on a custom website. The old site is here: http://oldforums.contraptionmaker.com/

What would be the best way to get this site's data exported into a format that Vanilla can import? The site is not built on any of the platforms that your Porter tool supports so I'm looking for the best way to get the data imported.

I've looked at the Vanilla Porter on Github (https://github.com/lincolnwebs/VanillaPorter) and I see that you can add a class file to create a new export type. Can a class be created to import our custom forum data? Is this something you could do for us?

Let me know what you think the best way is to handle it.

Thanks,
Eddie

Tagged:

Comments

  • hgtonighthgtonight ∞ · New Moderator

    Welcome to the community!

    The first thing I would do is have a duplicate db setup locally to test your exports against. Creating as export class is generally as simple as mapping the fields from your system to Vanilla's. There is some dummy data floating around (you can find one such collection here: ) that you can compare the output of your export class to.

    We would be happy to get you started if you can share your forum's db schema.

    Search first

    Check out the Documentation! We are always looking for new content and pull requests.

    Click on insightful, awesome, and funny reactions to thank community volunteers for their valuable posts.

  • Hi There,

    Ok I have setup Mongo DB on my local box and have the database setup. I've downloaded the Porter project from Github. Attached is the database (The files that start with "contraption").

    The dummy collection you sent didn't come through. Thanks a bunch for your help.

    Eddie

    cmdb.zip 503.5K
  • hgtonighthgtonight ∞ · New Moderator

    That appears to be a binary file of some sort. Could you just share the db schema you are using?

    A plaintext export would be most helpful.

    Search first

    Check out the Documentation! We are always looking for new content and pull requests.

    Click on insightful, awesome, and funny reactions to thank community volunteers for their valuable posts.

    UnderDog
  • R_JR_J Ex-Fanboy Munich Admin

    As you are speaking of MongoDB, I guess you do not have a "normal" db scheme. Porting that will be a lot harder since you have to match your data into any kind of table structure before you can even think of importing that into Vanilla...

    UnderDog
  • hgtonighthgtonight ∞ · New Moderator

    While I just have a passing knowledge of MongoDB, collections are almost the same things as tables, right?

    Search first

    Check out the Documentation! We are always looking for new content and pull requests.

    Click on insightful, awesome, and funny reactions to thank community volunteers for their valuable posts.

    UnderDog
  • Yeah this is my first time working with MongoDB. That file is a mongodb database. It looks like Mongo has an export tool that I could use to export it to CSV, would that help?

    UnderDog
  • R_JR_J Ex-Fanboy Munich Admin

    Yes, that would be a good point to start. I expect you will get several csv files. I'm an Excel wizard. If you need help with transforming your files you can pass me some samples. But I'm not sure if I can do anything before monday...

    hgtonightUnderDog
  • Which script is that old forum? I don't recognize it. As my fellow mates are saying : You need to figure out the DataBase structure of that old forum.
    You have the opportunity to make your own class and then import script, however, it's only once, so if you figure out the DataBase structure, you can better go over your options. A few are easier to make your own import script.

    With the Import script you can manipulate the data better before you import it into the MySQL database

    peregrine
  • The old forums are a custom application. I wasn't involved with making it, but my client says I can look at the source if need be. I will go ahead and export the mongo database to csv and start from there. I'll post the files here once I have them. Thanks guys!

  • Ok I've exported the CSV's. One thing that I don't get is that these old forums had users, but I see no data for users in the exported data. What are my next steps?

    bin.zip 115.8K
  • x00x00 MVP
    edited March 2014

    I like mongo.

    Like they say development is broken you need three different models: one for objects, one for database, and one for caching.

    Mongo = one model for everything.

    @eddieo‌

    Yes you are missing the user collection

    I will PM you.

    grep is your friend.

  • businessdadbusinessdad Stealth contributor MVP

    @x00 said:
    Mongo = one model for everything.

    True. It's very flexible, but it will haunt you forever if you are not tidy in designing the data models. I heard developers saying that they like Mongo because it allows them to skip the "boring planning and model design phase" and just dive into coding. Good luck fixing the inconsistencies... :neutral_face:

  • hgtonighthgtonight ∞ · New Moderator

    Assuming this is a full dump of your discussion structure, this looks like a pretty simple mapping. The biggest deal will be converting your hexadecimal ids to decimal ids (and dealing with any truncating issues).

    Search first

    Check out the Documentation! We are always looking for new content and pull requests.

    Click on insightful, awesome, and funny reactions to thank community volunteers for their valuable posts.

  • If people don't architect, they will struggle.

    On the other hand, if you ever need to fix inconsistencies, it is not harder then relational databases, necessarily.

    The biggest stumbling block for sql folk is not thinking in terms of relational databases. But rather loosely coupled collections, and understanding the performance considerations are totally different, so therefore you don't worry about the same things.

    I also like some feature of mongo which are provide something different from normal traditional database use, like capped collections and tailable cursors. Very useful for PubSub situations.

    I think mongo is a often a better solution then say redis, becuase it is stores, but also it is fast retrieved from memory.

    grep is your friend.

    peregrine
  • @hgtonight said:
    Assuming this is a full dump of your discussion structure, this looks like a pretty simple mapping. The biggest deal will be converting your hexadecimal ids to decimal ids (and dealing with any truncating issues).

    mongo doesn't do incremental ids, as mongo has a totally different storage philosophy to sql. Object ids are not actually hexadecimal increments,

    this is not a parallel conversion either.

    grep is your friend.

  • x00x00 MVP
    edited March 2014

    I should say SLQ flavoured relational databases to be correct. SQL is not a an engine or an optimizer, just a query language.

    grep is your friend.

  • hgtonighthgtonight ∞ · New Moderator

    The ids don't have to be incremental. A Discussion (in Vanilla) is just defined as a collection of comments sorted by a certain criteria (generally insert date).

    If you really want to re-assign the IDs so they are incremental, denormalize the db, sort by insert dates and replace the IDs with low incremental numbers. Then normalize the data out into Vanilla tables.

    This is an unnecessary step, imo, but to each their own.

    Search first

    Check out the Documentation! We are always looking for new content and pull requests.

    Click on insightful, awesome, and funny reactions to thank community volunteers for their valuable posts.

  • I think you are missing the point, it is mongo that is not relational, you have to relationally map during the conversion, so the relationship work in sql.

    I as I said it is not a parallel conversion between "tables" for one this app works differently than vanilla it object/data model is different.

    grep is your friend.

  • hgtonighthgtonight ∞ · New Moderator

    I am missing the point.

    Who cares what mongo does if the data is still there?

    Search first

    Check out the Documentation! We are always looking for new content and pull requests.

    Click on insightful, awesome, and funny reactions to thank community volunteers for their valuable posts.

  • the data is there, there isn't a simple parallel map, I have nothing further add.

    grep is your friend.

Sign In or Register to comment.