I tried going to the category file in my themes folder and commenting out all code that included the word "count", but all that did was remove my categories. Is there any way to do this?
If I understand what you want, the easiest way would be to do it via css, using display: none on the relevant elements. Use a web inspector to find what the element names are.
You can use the plugin CSSEdit to make css changes on the fly.
As a rule, don't edit core files; it just opens you up to trouble when Vanilla updates.