Is there a way to put custom forms in a popup?
Take a look at e.g. the discussion move feature. You have to style an anchor with some special classes and then simply point it to a view that displays a form. Use the Form class isAuthenticatedPostback to make that secure
Vanilla comes with a powerful JS thingie that allows you to simply add a classname to your link
<a class="Popup" href="linkToMyVanillaPage">My new pop up</a>
I use Vanilla forum pages to house a lot of custom things. And Vanilla will grab the page with this url parameter ?DeliveryType=VIEW that means all else is excluded that is not the content specific.
The potential is limitless.
Scroll down this page for inspiration
And you can use the popup JS directly if the form is in your template on the present page
jQuery.popup( '', $('#myFormDiv').html());