Okay Mark, as a simple exercise I thought I'd port SmartyPants as another extension for Vanilla. And it was fairly painless. However, because I don't understand your system well enough I need a couple of pointers.
I built it based on Markdown, but as I expected, when I installed and enabled it, it behaved in the same manner as the other editing "modes" -- in other words, you can only toggle between Markdown, HTML, SmartyPants, and so forth.
SmartyPants should really be enabled simply as a boolean filter, and it should filter the post only after any of the above extensions make their pass.