I was the same way. Metal came to me much later.
Pantera is a Texas band and I think their rhythm comes from that Texas blues thing, like what ZZ Top has. I've yet to really pin down what that is, but it's definitely a feel thing, or a subtle rhythmic thing.


