Please upgrade here. These earlier versions are no longer being updated and have security issues.
HackerOne users: Testing against this community violates our program's Terms of Service and will result in your bounty being denied.
email digest
mikerds
New
Hi, I am wondering if there is a plugin that allows me to email new posts weekly to all users? It would be very helpful to have something like this. It is a good way to keep people on the forum. I saw it with our nextdoor social network and got addicted to reading their forum after they started sending out email digest with new posts.
Thanks.
Mike
0
Comments
This doesn't exist, and I'll explain why:
What this all means is that the only reasonable way most servers can generate emails from PHP is, at the appointed hour at which an email is supposed to go out:
On a site with a hundred members to email, you'd never notice this. On a site with millions of members, this would present as a very serious bug. We already have this issue with the "Advanced Notifications" feature, which is why we restrict it to Moderators/Admins by default and counsel everyone to not let their entire user base opt into it. It's a very serious limitation for which we have first-hand witnessed very serious repercussions. This isn't a theoretical problem, it's one that's caused us much grief.
vBulletin and a number of legacy platforms have this feature, exactly by providing the non-scaling bad experience I described above.
At Vanilla, we have a strong objection to adding features that do not scale to millions of users in a reasonable way for open source. We don't want to provide a feature like this and then start seeing discussions about how slow the product can be etc. when folks don't understand the trades that have been made.
I'm open to discussion about the problem if someone has other suggestions, but please understand we've had a long hard think about this for years and still don't have a good solution besides the message queue.
I've had a request for such a plugin recently and I have denied it because it is beyond my capabilities. I see two options, both of which are not suited for being Vanilla core extensions from my perspective:
1. A task queue which is worked through by cron jobs
That would require a dedicated server or at least a hosting provider which allows setting up cron jobs. Furthermore there must be some mailer on the server installed and setting up a mail server which wouldn't be seen as a spammer from other mail servers is far from bein trivial.
I would consider that a more theoretical option...
2. External mail provider
There are several providers which allow very sophisticated mail actions: my favorit is SparkPost which allows creating recipient lists which is a requirement since you would have to take category permissions as well as people who have opted out from such a "newsletter" into account.
And they offer templates, which would allow more personalized info mails.
The reason why I think that something like that wouldn't be suited for Vanilla core is that there is no standard mail API. You couldn't provide a digest without a professional mail service. If you tie digest and mail service together, you force forum admins to use one dedicated service althought there might be reasons for them to user other services. Creating a digest plugin based on one special mail service provider is a waste of time, in my opinion.
It would be great if Vanilla would provide a "MailService" class like it provides an Authenticator class or an OAuth2 class and comes up with "sample" implementations.
It would be paradise if such a Gdn_Mailer class would exist, providing standard views for most common tasks like CRUD for recipient lists and templates, autmoatically syncing user information and everything which could be seen as a most common feature of such services.
Seeing that with an example implementation of a
class SparkPostPlugin extends Gdn_Mailerimplementation would enable plugin authors to start resource heavy mail actions.It would be even more convenient if that would be part of the core class.email.php so that all mails would be routed through an external service if available.
But the workload for something like that is enormous! You need to gather common features, get at least a rough understanding of how some of those APIs are built and you would need to get an understanding which requirements developers would have concerning such a class.
I would assume that Vanilla Inc. uses their own mail servers so I wouldn't see any motivation for writing such a standard class and an example implementation. If something like that doesn't come from the OS community, mail heavy tasks will stay niche solutions developed by paid pros for individual forums.
@Linc: Just brainstorming: MailChimp has an API I think.
It might be possible to keep the MailChimp mailing lists synchronized with the Vanilla ones who opt in, and update the campaign body with the API, and just let MailChimp automatically mail every week or so.
You only have to make requests to one webpage a few times, instead of having to send thousands of emails yourself.
Fair warning: I've barely used MailChimp so it might not be able to do what I'm proposing.
Unrelated: It seems by the way that the Debug parameter is set on this forum. Is that intended?
EDIT: Ninja'd by R_J who explains why it's a bad idea :P
I should've been clearer about the "sending" part: I mean the action of compiling the email contents per user and triggering the sending per user, not the specific outgoing email part. Sendmail's work doesn't really effect PHP since it is indeed its own process, and we already have built-in support for third-party services like SendGrid (which we use for cloud).
We have a full HTML email builder in 2.4, I believe. We're not really interested in Vanilla being a full-featured mailing service; that is a separate class of product. We just want to be able to dispatch nice emails, which we can already do.
This is what a message queue essentially is: a cron-based task queue offloaded to a separate server.
Oh, I think I misunderstood the question. As far as I understood the question, mikerds wants an email like "Hey, look at the most popular [X] discussions on the forum!" send, so about the same email to all users right?
Or does he want an unique email to all unique users?
If it was a general email, you'd only have to add a customer to the mailchimp list every time someone opts in on your forum (quick), remove them if they opt out (quick), update the email template that the weekly mail uses once with the general email (quick), and then just let mailchimp do its thing and let it mail that email to all users (slow, but we don't care since it's not happening on our forum).
If he wants an unique mail for every unique user every week, then moving to an external mailing service indeed doesn't help for speed :P
You're introducing complexities by outsourcing your preferences management. What happens when those user lists updates get out of sync from failed requests? How do you update unsubscribes from MailChimp into Vanilla? What action triggers updating the email build? Is the schedule stored on the MailChimp side or the Vanilla side? If it's MailChimp doing the timing, what if the content set/update request fails? I bet there's a lot more here to think about, that's just what I came up with in 2 minutes.
You could build a "MailChimp Email Digest" addon for sure, but that's quite different than it coming thru the forum's main email system, and there will be a good amount of overhead from doing it that way.
But SendGrid has "campaigns", which I think would be able to take away most of that problem.
1. Who should receive mails?
Starting with an Opt-In: when the user ticks a checkbox in his profile and saves that, an API call is made which uploads mail and user name.
Starting with an Opt-Out: more difficult, because it would need either a) an export for all relevant user info as csv so that admins can upload it to SendGrid or b) upload all user inf via API (maybe facing a timeout for large user bases)
When a user changes his mail, the mail address on SendGrid must be changed: one potential API call on profile save.
Users can be grouped in lists, users who have unsubscribed can be excluded (native feature from SendGrid).
2.1. Content of the mail: templates
Campaigns can use placeholders. So the body must be created only once (in an ideal world)
2.2. Content of the mail: permissions
User lists should be defined by Vanilla roles (which would also be nice for "manually" created campaigns) because you would have to care about permissions. If you send a digest like:
There have been 37 new discussions in the past week.
Our top five, recommended for reading:
1. Title in Category, written by Author
2. Title in Category, written by Author
3. Title in Category, written by Author
4. Title in Category, written by Author
5. Title in Category, written by Author
you would need to make sure that the recipients have read access to those 37 new discussions.
I would say that it comes down to assembling as much mails as you have roles. It would take some time to create that digest mail, depending on how much work you put into finding out the most interesting discussions. If you simply sort them by CountComments, it shouldn't take too much time.
I don't think you can start more than one campaign with an API call so that you would have to run maybe a dozen calls against the SendGrid API.
I wouldn't be happy if I was the user who would have to wait for the reply of twelve API calls to any server, though, but at least I think that this is much less effort than creating one mail-task per user.
But if you would provide more detailed information like "Hi R_J, you have 3 new messages and 13 unread discussions" there would be no way out: you would at least have to send a short JSON string per user.
Okay, that might be a show stopper for a "simple" mailer service solution...
... and you have repeated to the above (before I said it) with your previous post, Linc...
Syncing user mails shouldn't be a problem: those APIs have return codes and a failed return code would require an additional action. That would come back to an internal queue system, I guess.
Unsubscribing (as far as I understood it) can be achieved by using webhooks.
But yes, timing the outgoing mail will be a huge problem.
From a product planning perspective, it's difficult for me to justify allocating R&D time to more tightly coupling Vanilla with a third-party service provider (SendGrid) than to invest in a generic cloud solution (message queue) that we can use for myriad scenarios.
Even from an open source prospective: Would you rather we give you a "here, do this super complicated / possibly environment & skillset restrictive -but free! - setup" (message queue) or "here, pay for this third party service to get this feature you want easier" (SendGrid)?
It feels like the point of open source is the former, so that's the kind of thing we're going to most likely build support for into core without providing a turnkey solution. That's the route we've taken with plenty of other things in the past (memcached support, Sphinx support) and I think it's the path we plan to continue taking.
Well, it just has been what you've asked for: a suggestion. I also think that relying on paid services (even if there are free plans) isn't a natural fit to open source software.
But speaking of mass mailings, those services at least have to be considered.
Myriad scenarios? Could that be understood as you are planning to implement a general task queue?
Yes. This is how we plan to handle time-specific (or power-intense) tasks in general.
@linc wrote:
Understood. My question is whether the experienced delay in page load created by inserting an expensive process in the middle affects every user on the forum or just the one unlucky user whose page load triggered the expensive process. Clearly, if it affects everyone then a cron process will have the same "global" effect.
Also, it seems that the key issue is your first item:
If the expensive task is broken down to many less expensive subtasks (think of it as breaking a process to several sequential steps) then the response time degradation is spread over a larger amount of time. With that in mind, I think that it is possible to architect an overall mechanism that supports such subtasks (clearly it is still the responsibility of the developer to break their tasks to subtasks, and it is still their responsibility to control the size of each subtask).
Depends on further qualifying what "expensive" means. If it's mostly time-intensive, it probably will only effect the unlucky user. However, if it caused enough of the processor to get eaten up (less likely these days) or blocked MySQL transactions because they get stacked up (still very plausible) you could see site-wide effects.
Sure. Like, maybe only 20 emails can get sent at once. But now you're adding quite a bit of complexity for a gatekeeping system, and you'd need to also architect it for lower-traffic sites that may only get a few users a day. The appropriate "chunk" for a 20-user site perhaps looks different than a chunk of work for a 2-million user site. I'm not sure the challenge of figuring that out is something we should attempt since it seems a risk for failure.
Conversely, a message queue is a standard thing used in our industry, and lets us spin off tasks basically as-is.
True, but it changes the matter from the single-threaded PHP issue to server resources, which I think is good.
I think that the multi-step approach I mentioned is more general than the message queue. True, this thread discusses bulk emails, but I have in mind a more general facility - one that addresses different "batch-type" processes, bulk email being just one example. Other examples: Importing from external blogs/sites, feed readers or other background processing. All face the same issue, all could benefit from some solution.
This does not negate the preference to use cron - the cron process still needs to invoke a Vanilla url. That url could still affect the entire site if the work isn't broken down into chunks...
I think this could be entirely implemented with a plugin. I'm not volunteering now (too busy), but I'm saying it could be done independent of the core Vanilla development.
A scenario like providing a data export for a user with tens of thousands of posts - or the entire site - makes it difficult to batch.
I know it's possible to do this in a plugin, because I built timed functionality into my bot addon over a year and a half ago.
That doesn't make it a good idea to build tablestakes features like an email digest on top of it.
I don't know, If the ten thousand are broken down to 1000 at a time and invoked by cron - maybe it's not so bad...
If you review the number of detours and tradeoffs taken in the course of the conversation to get to this point, the resolution being "maybe it's not so bad" is... really bad.