Forum showing unread discussion when all discussions are read

MrCaspan

I have been having this issue since the day one when running 3.3 and now I am running 2021.012 and the issue is still persistent. Please note I am using the 'Unread Category Photo' plugin and the home page of my forum is set to Category view.

Yesterday I marked the entire forum as read and as expected everything was marked as read all, categories were showing that there were no unread discussions. This morning a new member joined and I read their message but when I went back to the home page the site still showed the Introduction category had unread discussions. I thought maybe it was pulling the image from cache so I did a hard reload of the site to ensure it was pulling new content if available and it was still showing as unread discussion in this category. When I open the category there are NO unread discussions.

This issue has plagued me since day one and I was hoping that a different issue we figured out about MySQL not returning INT values form the database was the issue but sadly the issue persists. Any ideas where to start looking to see why this is happening?

MrCaspan

Here is a video of the issue happening

https://drive.google.com/file/d/1LbCQb0z5k314WJ7yCsbWYNtyTtsq5-13/view?usp=sharing

Ill try to upload again showing what happens when someone else posts something on the site

MrCaspan

And here is an example of what's happening after seeing me marking the entire forum as read

https://drive.google.com/file/d/1gzyPxt-TawsZhx0YCSyFWo4ps6P6R1aS/view?usp=sharing

R_J

Not sure if this can easily be fixed, but let's see what we can do.

In my test forum, I have an empty category and that shows as "Unread" although there are no discussions in there. Well, if you call it unread or read is equally true or wrong, but it contains no unread discussions, therefore I'd prefer if it wouldn't be marked as unread.

The CategoryModel returns a category with a field called "Read" which determines the CSS class. So I want an empty category to be Read = true. I had a look at the code in CategoryModel which calculates that field:

      // Calculate the read field.
      if ($category['DisplayAs'] == self::DISPLAY_HEADING) {
          $categories[$iD]['Read'] = false;
      } elseif ($dateMarkedRead) {
          if ($lastDateInserted = ($category['LastDateInserted'] ?? false)) {
              $categories[$iD]['Read'] = Gdn_Format::toTimestamp($dateMarkedRead) >= Gdn_Format::toTimestamp($lastDateInserted);
          } else {
              $categories[$iD]['Read'] = true;
          }
      } else {
          $categories[$iD]['Read'] = false;
      }

What we can see from this snippet is the following:

a) If your category is a heading, it will always get the CSS Unread class, no matter if there are unread discussions in any of the sub categories!

b) The DateMarkedRead and the DateLastInserted columns are compared and CSS class is set based on that comparison. DateMarkedRead is fetched from the UserCategory table with a fallback to the Category tables column value.

For my empty category issue the solution is easy: I simply run the following query against my database to fill every empty DateMarkedRead field: UPDATE `GDN_Category` SET `DateMarkedRead` = '2021-01-01' WHERE `DateMarkedRead` IS NULL; Now my empty category doesn't get that Unread CSS class any more. Yes, a user would be able to solve that problem for himself, but why confuse _every_ user when the solution is so simple?

But that doesn't help you much. In fact I'd need your support to get more information and I'd ask you to insert one line of code into the CategoryModel so that we get some data to look at.

Could you insert the line decho($categories); right before the closing parenthesis of /applications/vanilla/moels/class.categorymodel.php method joinUserData? Should be line 2185 and it would look like that:

       // Add permissions.
       foreach ($iDs as $cID) {
           $category = &$categories[$cID];
           self::instance()->calculateUser($category);
       }
decho($categories);
   }

That decho function will show it's output only to the admin user and to nobody else. Your page will look awful to you after you have inserted that line, but only to you. You will see the php array with the category information. Could send me that via PM so that I can take a look at it? Just from looking at the code I'm somewhat lost.

R_J

By the way: there is some sort of caching implemented for the categories and I suspect that the LastDateInserted value in the php array in the debug output doesn't match the real database value.

So theoretically you can already do the next step without waiting for my reply. I assume you are facing that issue and get the debug output. The information I would be looking at would be the fields

1. LastDateInserted and
2. DateMarkedRead

in the debug output and in the tables Category and UserCategory (where UserID is the your admins user ID)

MrCaspan

Okay so here is the data. The site thinks that the category "Introductions" has unread messages right now. If I do a decho as you suggest and look at the array for Introductions it looks like this

[80] => Array
        (
            [CategoryID] => 80
            [ParentCategoryID] => -1
            [TreeLeft] => 4
            [TreeRight] => 5
            [Depth] => 1
            [CountCategories] => safePrint{0}
            [CountDiscussions] => 930
            [CountAllDiscussions] => 911
            [CountComments] => 4431
            [CountAllComments] => 4374
            [LastCategoryID] => 80
            [DateMarkedRead] => 2021-08-19 13:58:30
            [AllowDiscussions] => 1
            [Archived] => safePrint{0}
            [CanDelete] => 1
            [Name] => Introductions
            [UrlCode] => introductions
            [Description] => After you have confirmed your email you will need to introduce yourself to the forum by posting small post about yourself, does not have to be anything long. Once you do this and your post has been approved you will have access to the rest of the forum!
            [Sort] => 4
            [CssClass] => Category-introductions
            [Photo] => category-read.jpg
            [BannerImage] => safePrint{null}
            [PermissionCategoryID] => 80
            [PointsCategoryID] => safePrint{0}
            [HideAllDiscussions] => safePrint{0}
            [DisplayAs] => Discussions
            [InsertUserID] => safePrint{0}
            [UpdateUserID] => 18905
            [DateInserted] => 0001-01-01 00:00:00
            [DateUpdated] => 2020-04-05 19:47:54
            [LastCommentID] => 420346
            [LastDiscussionID] => 35245
            [LastDateInserted] => 2021-08-19 18:59:05
            [AllowedDiscussionTypes] => safePrint{null}
            [DefaultDiscussionType] => safePrint{empty string}
            [Featured] => safePrint{0}
            [SortFeatured] => safePrint{0}
            [UnreadPhoto] => category-unread.jpg
            [AllowFileUploads] => 1
            [Url] => https://pokerforum.ca/categories/introductions
            [PhotoUrl] => https://pokerforum.ca/uploads/category-read.jpg
            [LastTitle] => Hi everyone
            [LastUserID] => 6432
            [LastDiscussionUserID] => 25345
            [LastUrl] => //pokerforum.ca/discussion/35245/hi-everyone#latest
            [DateLastComment] => 2021-08-19 18:59:05
            [Unfollow] => safePrint{0}
            [Following] => safePrint{true}
            [Followed] => safePrint{false}
            [Read] => safePrint{false}
            [UserCalculated] => safePrint{true}
            [PermsDiscussionsView] => safePrint{true}
            [PermsDiscussionsAdd] => safePrint{true}
            [PermsDiscussionsEdit] => safePrint{true}
            [PermsCommentsAdd] => safePrint{true}
        )

So the important stuff I think are these 3 values in the array based on the code you mentioned above

[DateMarkedRead]   => 2021-08-19 13:58:30
[LastDateInserted] => 2021-08-19 18:59:05
[Read]             => safePrint{false}

In this case this means according to the code this category is NOT read, problem is there is not any unread discussion in this category and if I use the LastDateInserted and look to see what comment that was, I did read that comment but it seems like by reading that comment it did not update the DateMarkedRead to the current Date Time value. Ill keep playing but i have to see what I think is happening is actually happening

Do you know where the code is that set the DateMarkedRead value to the current date time when viewing a discussion or comment?

Ill report back

MrCaspan

@R_J I am looking at the post that the forum says was last posted to the Introductions category. It is a comment that I have read and can confirm that. If I go to that comment and stay on the page and keep refreshing my DateMarkedRead for the introduction category stays the same. This DateMarkedRead value seems like a value that is manually set. Does this make sense? It's almost like the forum forces you to have to manually mark a category as read to change this value instead of looking to see if there is any unread discussions or comments in this category? Does that sound right?

MrCaspan

I guess it would be very database intensive to go to every single discussion in a category and iterate over it to see if any of those discussions are read for this user. Not to mention if you load the home page you would have to iterate over every single discussion every single page load just to find out if read or not

R_J

Is

DateMarkedRead]   => 2021-08-19 13:58:30

what's in your UserCategory table for your UserID and the CategoryID?

MrCaspan

Yep Category ID 80 is the Introductions Category

R_J

Too tired to look closer at this today, sorry. But inspecting my test forum I saw that it set UserCategory.DateMarkedRead together with UserDiscussion.DateLastViewed of the corresponding discussion

From what you are describing, it sounds as if the UserDiscussion table is updated, while the UserCategory is not. And so the problem is not in the Read/Unread functionality but it comes up after the UserDiscussion is updated.

MrCaspan

No problem RJ I'm gone for the weekend anyways so I'll do some testing next week hopefully we can get to the bottom of this

R_J

Oh wait, maybe all is well but you only have some discussions that are not read. There is an unlinked method that you can call (it doesn't scale well, therefore its deprecated, buts till functional): /discussions/unread

R_J

Electric toothbrushes are a great invention. I had time to look at the code and I found the reason, but no solution.

This should make the problem recreatable:

User writes a comment in discussion A

You do not read the discussion, but use "Mark Read" (which creates or updates a row in UserCategory)

User writes a comment in discussion B

You read that, which creates or updates an entry in UserDiscussion

Now here comes a snippet from the DiscussionModel:

   /**
    * Mark categories that this discussion was in as read.
    *
    * This method was extracted from `DiscussionModel::setWatch()`.
    *
    * @param object $discussion
    */
   protected function markCategoryReadFuzzy($discussion): void {
       /**
        * Fuzzy way of trying to automatically mark a category read again
        * if the user reads all the comments on the first few pages.
        */

       // If this discussion is in a category that has been marked read,
       // check if reading this thread causes it to be completely read again.
       $categoryID = $discussion->CategoryID;
       if (!$categoryID) {
           return;
       }
       $category = CategoryModel::categories($categoryID);
       if (!$category) {
           return;
       }
       $wheres = ['CategoryID' => $categoryID];
       $dateMarkedRead = $category['DateMarkedRead'];
       if ($dateMarkedRead) {
           $wheres['DateLastComment>'] = $dateMarkedRead;
       }
       // Fuzzy way of looking back about 2 pages into the past.
       $lookBackCount = Gdn::config('Vanilla.Discussions.PerPage', 50) * 2;

       // Find all discussions with content from after DateMarkedRead.
       $discussionModel = new DiscussionModel();
       $discussions = $discussionModel->get(0, $lookBackCount + 1, $wheres);
       unset($discussionModel);

       // Abort if we get back as many as we asked for, meaning a
       // lot has happened.
       if ($discussions->numRows() > $lookBackCount) {
           return;
       }

       // Loop over these discussions and exit if there are any unread discussions
       while ($discussion = $discussions->nextRow(DATASET_TYPE_ARRAY)) {
           if (!$discussion['Read']) {
               return;
           }
       }

       // Mark this category read if all the new content is read.
       $categoryModel = new CategoryModel();
       $categoryModel->saveUserTree($categoryID, ['DateMarkedRead' => DateTimeFormatter::getCurrentDateTime()]);
       unset($categoryModel);
   }

Vanilla loops through all discussions which are newer than DateMarkedRead. If there is no unread discussion, UserCategory.DateMarkedRead would be updated and everything would be fine. But That first DateMarkedRead is not UserCategory.DateMarkedRead, but Category.DateMarkedRead and that's a logical error.

A quick gaze gave me no hope that there would be an elegant fix for this

MrCaspan

So should I post this as an issue to GitHub?

MrCaspan

I have pit it in GitHub as it looks like a bug/issue

https://github.com/vanilla/vanilla/issues/11091