Stuck in import process

mikemetis

I want to transfer a phpBB board to Vanilla and was able to get an export file thanks to Vanilla Porter 2.3 (https://open.vanillaforums.com/addon/porter-core)

Then I started the import at the Vanilla forum:

Source phpBB 3.*
Vanilla Export 2.3
HashMethod phpBB

But I am stuck at point 10 "Update Columns" (5/11) -> Column 'CountComments' cannot be null

I browsed through the export file but could not find any NULL value in the corresponding data. As all discussions and users have been transferred, I have two groups of questions:

1.) How critical is it for a successful import that points 10 to 13 (Update Counts, Custom Finalization, Add Activity, Verify Import) are properly executed? What is missing if they are not executed?

2.) How can I find the reason for being stuck? What does 5 out of 11 mean? Which 11 items is it referring to?

Any input is highly welcome!

PS: Please also see the screenshot attached showing at which point the import process is stuck.

R_J

Updating counts can be done later, therefore it wouldn't be a problem. But certainly all the other steps should be processed.

In order to be able to proceed, I would advise to change one Vanilla file, do the import and restore that file.

Edit /applications/vanilla/settings/structure.php and search for ->column('CountComments', 'int', '0') and make it ->column('CountComments', 'int'). You should find such a line 3 times in that file.

mikemetis

Hi @R_J , thank you for taking care. I did as you told me and made the changes at all three positions:

But it still stops at point 10. Update Counts: 5 of 11.

It either has to do with the data I want to import or with an uncaught exception in the code. Do you know which script is doing the import? Can I make a change there? Or should I change the data file? If so, which values should I change?

R_J

In /applications/dashboard/controllers/class.importcontroller.php you find

    public function go($transientKey = '') {

...

        if ($imp->CurrentStep >= 1) {
            if ($this->Form->authenticatedPostBack()) {
                $imp->fromPost($this->Form->formValues());
            }
            try {
                $result = $imp->runStep($imp->CurrentStep);
            } catch (Exception $ex) {
                $result = false;
                $this->Form->addError($ex);
                $this->setJson('Error', true);
            }

            if ($result === true) {
                $imp->CurrentStep++;
            } elseif ($result === 'COMPLETE') {
                $this->setJson('Complete', true);
            }

It basically count steps and those steps are defined in /applications/dashboard/models/class.importmodel.php

    public function runStep($step = 1) {

...


        $method = $steps[$step];
        $result = call_user_func([$this, $method]);
    }

...

    public function updateCounts() {
        // This option could take a while so set the timeout.
        increaseMaxExecutionTime(60 * 10);

        // Define the necessary SQL.
        $sqls = [];

        if (!$this->importExists('Discussion', 'LastCommentID')) {
            $sqls['Discussion.LastCommentID'] = $this->getCountSQL('max', 'Discussion', 'Comment');
        }

        if (!$this->importExists('Discussion', 'DateLastComment')) {
            $sqls['Discussion.DateLastComment'] = "update :_Discussion d
         left join :_Comment c
            on d.LastCommentID = c.CommentID
         set d.DateLastComment = coalesce(c.DateInserted, d.DateInserted)";
        }

... (and so on)

Do you feel save "debugging" something? I would add decho(__LINE__); after each if...line. Then you should be able to see where execution stopped.

mikemetis

Hi @R_J , with your information I was able to spot where the "error" is happening.

It is in line 2060 of /applications/dashboard/models/class.importmodel.php when the script executes the SQL statement $this->query($sql);

The statement is

update :_Category p
set p.CountComments = (
select sum(c.CountComments)
from :_Discussion c
where p.CategoryID = c.CategoryID)

Unfortunately, I do not know what to do now, as I am not able to see the actual data. I guess p stands for parent and c for child and that something does not match here. Is there a way to see the actual data (i.e., the IDs used) of the query which leads to the error?

R_J

Without further explanation, so that you can proceed (I will add that explanation afterwards): visit yourforum.com/utility/structure, follow the instructions on the screen and try again to import

R_J

I forgot about the import process. The structure file that I asked you to change is used for the Vanilla installation process. So changing it without doing anything else is useless. By visiting /utility/structure you force a reload of that structure file and after that your table GDN_Category will allow NULL in column CountComments.

And based on the above this is what is happening: The column CountComments of the table GDN_Category should be updated. It holds the information how many comments the discussions of the given category alltogether has. Therefor it updates the field with the sum of the field CountComments of the table GDN_Discussion for each category.
"Take the sum of GDN_Discussion.CountComments per Category and update GDN_Category.CountComments for all categories"

I guess the problem is that you either have empty categories or that the field CountComments in the table GDN_Discussion already shows NULL values.

But whatever it is: when you change the database structure so that it accepts NULL values for this columns and the problem shouldn't show up any more.
Instead of changing the structure file you could edit the databse directly, if you know how to do it. I would advise to make "0" the default value for GDN_Category.CountComments and GDN_Discussion.CountComments and allow NULL values.
After the import process you should do three things:
1. recreate the old structure file
2. visit yourforum/utility/structure again (to restore the database structure)
3. visit yourforum.com/dba/counts which is another helper which amongst other counts, also looks at the Category.CountComments column

mikemetis

Executing these two MySQL statements made the import work:

ALTER TABLE GDN_Category MODIFY CountComments int NULL DEFAULT 0;
ALTER TABLE GDN_Discussion MODIFY CountComments int NULL DEFAULT 0;

I did this because the other approach did not work at first because I realised only after the MySQL changes that I have to use:

yourforum/index.php?p=/utility/structure instead of yourforum/utility/structure for my installation.

But both /utility/structure and /dba/counts are helpful features!

Thank you very much @R_J !!!