I really like the layout and color setup (aside from the green) how its darker around the catergory names
Is this a custom one?
This is a paying customer of Vanilla and I would guess they also got their theme by them. At least that is no theme from the download section
Yes, it is a custom theme which is proprietary. You can tell by looking at the page source and seeing what theme css is loaded.
I'm no expert, but it looks to me like you could get a similar layout based on Bootstrap.
Vanilla is easily customised by either editing or adding CSS rules to a custom theme.