Well, since I posted this, I've disabled whispers and implemented an improved User Messages extension "made" by one of my forum members.
That did solve some perfomance issues but, with half a million posts, things were getting slow again...
Checked what queries were slowing things down and solved it by replacing /library/vanilla/Vanilla.Class.CommentManager.php with this: http://pastebin.com/Ku8yTs4w
So, my sugestion:
-> disable and delete all whispers in the database
-> replace file above
It isn't beautiful - I'm no coder or somehow instructed, it's a DIY solution - but it works.
It's fast again.