Questions about optimal mail settings and experiences with high amounts of mails
R_J
Admin
Vanilla uses PHPMailer to send mails internally. You can add the credentials of a SMTP server to use. As far as I can tell, there are three possibilities to send mails:
- Over the SMTP server which has been configured
- No SMTP server, sendmail or similar set up on hosting server
- No SMTP server, no sendmail (some PHPMailer magic?)
What would be the most effective? Are there any SMTP mail servers which are in any way better than a server that you can setup by yourself? What could be the pros an cons of using sendmail?
The ActivityModel seems to handle each mail separately. This is could be seen as a benefit since you can take influence by hooking into events. But if I had a discussion which has been bookmarked by a relevant number of users and that discussion gets a comment, all users will be notified. To my understanding, that will involve sending as much single mails, as there are users who have subscribed to that discussion.
What would be the impact of, let's say 500 users who have bookmarked a discussion? Or of 1k or 10k? @Linc: are there any actions Vanilla Inc. performs to ensure that sending mails is no bottleneck at any time?
Are there any experiences directly tied to Vanilla and sending a lot of mails? It would be nice to even get a feedback like "10k users and no problems here" - maybe you, @phreak, can say something like that...
Comments
R_J and & have been discussing this issue offline, which prompted his post.
I'm preparing to stand up a new forum... well not so new. I own & manage a LinkedIn professional-technical group with ~12k members. Over the past few years, the Groups platform has devolved to the point that it no longer serves our needs. In particular, LinkedIn eliminated the ability to subscribe to individual discussion threads, i.e., no more email notifications when comments are added to topics of interest. Not surprisingly, activity in my group fell off a cliff after that. So it's important that our new forum platform be able to handle the volume of mail we expect to generate.
Outgoing email is pointed to my sponsor's SMTP mail server. At this point, I'm not concerned with their mail server's resources. They can expand the capacity based email loads. But I need to make sure that the forum software itself isn't affected by the number of members who bookmark a given discussion thread.
This gets at the way Vanilla hands off outgoing mail to the SMTP server, and whether this happens in parallel with browser request processing, with one not affecting the other.
What got me concerned about this is reading comments on the EMailSubscribe page about how posting takes longer when there's a large number of subscribers, i.e., the time it takes for focus to return to the browser after clicking the Post button. Hopefully someone out there can reassure me that Vanilla's built-in notification processing doesn't suffer this same issue.
You would ideally use a transaction mail service like sparkmail / mandrill
note ideally you would be using their proper api rather than directly through an email protocol
here is one plugin
https://open.vanillaforums.com/addon/mandrillmailer-plugin
personally I prefer sparkmail since mandrill now requires you to have a mailchimp account.
grep is your friend.
it would be best if you could somehow optimise notifications through use of templates and key/value replacements to reduce the amount of request, but might require quite a bit of work to get this working.
grep is your friend.
in other would you would send the user data and the select service templates and not every single email. But thsi quite an advanced technique.
grep is your friend.
@x00 great to see you around, since I guess you have a lot of experience even with forums with higher traffic: have you ever heard of proplems because of the notification mails? ginahoys concern has been that if there is a discussion which has been bookmarked by 10k members receives a comment, it will come to timeouts and lags because of the notifications which are sent.
Looking at the CommentModel I see that there is save() and save2() and that ActivityModel->saveQueue() is the last command. My assumption is that everything would run smoothly and if mailing fails for whatever reason, there should be no problem for the user who is writing the comment. And the mails are still "pending" so that they simply will be sent later on.
Simply from the fact that no one complained about that before makes me optimistic, but I have no experience (or even an idea) at all as to how many mails are getting "too many" for a smooth processing.
even on large userbases it is rare that a discussion is bookmarked by 10k users, so behaviour counts for a lot.
grep is your friend.
I have experienced some really big forums with no issues with this, but behaviour counts for a lot, it depends how they use it.
A big problem on some forums is run on discussions, these are counter productive.
There is weird thing that happens sometimes where people try to create forums or categories within a single discussion, this defeats the purpose of the forum architecture.
e.g. the "meetup thread", when a meet up category would be more suitable. Then the create the "london meet-up thread" and this runs on. You want to promote specificity an relevance.
There is very little benefit to everyone getting a notifications becuase someone has necroposted some run on thread that used to be a sticky.
grep is your friend.
Some forum software has some limited batch sending such as phpBB. However out the box they want to be self contained. So still request driven.
Regarding ActivityModel I have been able to use it to create deferred process which then add notification (no email) to sending daemon. This has minimal effect on requests.
You can similarly create a defered process for the mailer, which would use a mail sending deamon.
grep is your friend.
If you have 500+ users bookmark a single discussion (which I've never seen happen, and we host some of the largest forums on the Internet) that could indeed be a problem. The emails are currently sent synchronously with the page request, meaning the person making the post that generated the email would be stuck waiting for them all to send before the page request finished.
In my experience, the only way that mailing becomes a true bottleneck is if you give "Advanced Notifications" permission to everyone, and they are able to receive emails for EVERY discussion or comment in one or more categories.
We're moving to sending email asynchronously on cloud, but I don't think that's a reasonable thing to expect most open source users to do because it involves setting up separate infrastructure (a messaging queue).
That said, I would be shocked if @ginahoy runs into any problems with a 12,000 user forum. I'd outsource the email to SendGrid or some such so the actual processing happens off-server, and you'll be fine.
We run forums with millions of users like this and do not see bottlenecks.
Many thanks for that info!
Thanks for the feedback.
With LinkedIn's Groups platform (before they eliminated the 'follow' flag and email notifications), I had no clue how many people were following a given discussion thread. I'm sure most threads in my new forum will end up with dozens rather than hundreds of subscribers. My concern would be with the occasional outlier that jambs up the board. Sounds like I just need to wait to see if this actually becomes a problem before spending time on a maybe.
With Vanilla, I don't see a way for Admins to tell how many members have bookmarked a discussion. I suppose I could refer to the database if I'm curious about a particular thread @ the GDN_UserDiscussion table(?)
I do have one question about notifications. Although I don't plan to give Advanced Notification permissions to users, I would like to give users the ability to receive notifications for new Discussions as they are posted. They would then have to bookmark the Discussion if they want to receive comment notifications. Is there a way to do that?