Feed Discussions: how to reliably trigger a fetch of feeds?

DivineDominion

I don't get predictable discussion creations with this plugin. Sometimes it works, sometimes it seemingly doesn't react for hours.

To test, I added a feed that updates every minute: https://lorem-rss.herokuapp.com/feed

The 'checkfeeds' URL endpoint and the plugin page all say that things have updated successfully:

$ curl https://.../plugin/feeddiscussions/checkfeeds

{
    "AutoImport": true,
    "https://lorem-rss": {
        "herokuapp": {
            "com/feed": {
                "Historical": false,
                "Refresh": "1m",
                "Category": "8",
                "Added": "2021-10-14 18:14:38",
                "LastImport": "2021-10-14 19:08:09",
                "URL": "https://lorem-rss.herokuapp.com/feed",
                "LastPublishDate": "2021-10-14T19:08:00+00:00"
            }
        }
    },

I also tried `curl -I` and found I sometimes get a 201 Created HTTP response, but nothing seems to be created.

However, if I look into category 8 (it's an internal moderator category a.t.m. as a staging ground from where mods move feed-based discussions to the appropriate categories) -- there's nothing from 19:08 (GMT) like the response indicates; there hasn't been anything for about 30 minutes now.

Before that, I had trouble getting the plugin to import feeds, too, but all of a sudden it imported 10 minutes worth of feeds at once. When I noticed and deleted these discussions from the mod forum, 4 new ones appeared. Since then -- silence.

To test another trigger, I posted a new discussion into the mod forum, and suddenly nearly a dozen new feed-based discussions appeared. I deleted these, too, waited, refreshed the browser, still nothing new. A minute later, 2 new posts. 3 minutes later, nothing; post new discussion, feed posts have been created.

--------------

Bottom line: I don't understand what triggers the update. I understand that the feeddiscussions.js isn't imported on every page. But shouldn't the manual CURL request to the checkfeeds endpoint work as well?

Is curl not working because of the Content-Security-Policy: frame-ancestors 'self' mydomain.com otherdomain.net being set? When I insert a link like:

<a href="/plugin/feeddiscussions/checkfeeds">refresh</a>

... using the browser's dev tools, then the referesh works as well.

I'd look forward to run this on a cron job, but for that, it'd have to do something reliably, first. It's as if the new discussions are "on hold" until someone posts something manually. Should the .js file be imported in other places to trigger more frequent updates? Shouldn't a `curl`-able version be available, too?

DivineDominion

It appears I can copy all of the Firefox header info and make a curl request including the cookie.

curl -H "Host: my-website.com" \
-H "Accept: */*" \
-H "Accept-Language: en-US,en;q=0.5" \
-H "Accept-Encoding: gzip, deflate, br" \
-H "X-Requested-With: XMLHttpRequest" \
-H "DNT: 1" \
-H "Connection: keep-alive" \
-H "Referer: https://my-website.com/discussion/498/test2/p1?new=1" \
-H "Cookie: ......." \
-H "Sec-Fetch-Dest: empty" \
-H "Sec-Fetch-Mode: cors" \
-H "Sec-Fetch-Site: same-origin" \
-H "Sec-GPC: 1" \
https://my-website.com/plugin/feeddiscussions/checkfeeds

Without the cookie, no dice. So it also appears that either me being actually logged in for this session is important, or that the cookie incidentally helps in making the request pass the origin test. Not sure.

Kaspar

This might help

https://open.vanillaforums.com/discussion/comment/254364/#Comment_254364

Kaspar

Last change was "set cookies false"

https://github.com/vanilla/addons/blob/master/plugins/FeedDiscussions/class.feeddiscussions.plugin.php

https://github.com/vanilla/addons/commit/ab139bfb8d592541d47350437a3b303158ceb331#diff-46b48f03fff764ea9c9d8f66c5429934e9294ea2566319784d27b953cf298ced

DivineDominion

Thanks for the links!

1. The current version of the plugin has neither a "check manually" button nor a way to edit feeds, so the tips sadly don't apply :(
2. From what I understand, the "set cookies to false" change applies to the fetching of RSS feed contents. Could be a measure to anonymize the request. I also don't believe anything goes wrong there, because errors thrown at that point should be forwarded when I do a curl request, too; at least the HTTP status code '201 Created' wouldn't be possible if an error is thrown and execution interrupted.

The flow of things seems to be this: discussionController_beforeDiscussionRender_handler inserts the JS file only if checkFeeds(false) (i.e. no auto import) returns true, i.e. there have been changes to the RSS feeds that are subscribed. (See code) So feeds are checked once to determine if there have been changes, and then the JS is added with the intent of making the import, so they are checked again, and this time imported, potentially with the intent to make this async thanks to JS instead of blocking the original request?

But the JS just calls the checkfeeds endpoint:

jQuery(document).ready(function () {
  $.get(gdn.url('/plugin/feeddiscussions/checkfeeds'));
});

I do understand that this results in a GET request that I can emulate with curl _if I include the cookie data_. What I don't understand, though, is how this is even a factor.

The /plugin/feeddiscussions/checkfeeds endpoint's code is this:

    public function controller_CheckFeeds($sender) {
        $sender->deliveryMethod(DELIVERY_METHOD_JSON);
        $sender->deliveryType(DELIVERY_TYPE_DATA);
        $this->checkFeeds();
        $sender->render();
    }

It changes the response header to JSON, checks the feeds again (implicit $autoImport=true, the default parameter value), and renders the JSON output in the end. The setData changes are reflected in the JSON output, and the JSON looks fine to me. (See first post) Everything's in there.

• Is anything in pollFeed not working unless the current session/cookie is set? I don't see that in the method. But it might be an internal detail?
• Is the Publish event never fired and the loop aborts early, before the discussion data is inserted? That sounds plausible, but what are the reasons for this? After all, $this->EventArguments['Publish'] = true is unconditionally set early on and not changed to 'false' as far as I can tell. (The if statements sound like practically useless conditions to me)

I'm not sure how to debug this except for using $sender->setData(...) and then look at the JSON.