When I first started taking philosophy in University, metaphysics appeared to be a very promising pursuit. It seemed to me to be the gateway to magic; it held the possibility of alternate universes ruled by different orders. If materialism denied the existence of magic, well then, just do away with it--it was just one metaphysical premise amongst others, and the best part of metaphysics was that you could spin conjectures to your heart's content, and never be called on to prove any of them, except by a clever twist of logic and a big maybe (and who could prove me wrong?) It was, after all, meta-physics, supposedly the basis of physics itself, except that, being non-empirical, you didn't have to bother with all of that annoying evidence. All the worlds of fantasy and science-fiction beckoned--and somehow, I could justify them. But as I learned more about it, metaphysics lost its lustre.
Metaphysics isn't about magic or the supernatural--at least, not necessarily. It is a discussion of the ultimate nature of reality. It is, however, beyond the bounds of the empirical or strictly logical. This means that mathematics is not a branch of metaphysics, unless you assert that a particular mathematical model describes ultimate reality independent of empirical verification. If the model corresponds to reality and is demonstrated empirically to do so, then you're doing physics. This is why some physicists have become dismissive of string theory. So much of it lies outside of the bounds of empirical testing that by claiming that it represents physical reality, theoreticians may in fact be practicing metaphysics. But it is not the mathematics itself that make this metaphysics, only the claim that they represent reality without empirical justification. You can construct an infinite variety of mathematical worlds by varying your basic axioms, but most of those worlds will have no bearing on reality as we know it. Exploring these worlds mathematically is just mathematics. Asserting that these worlds actually underpin physical reality, without any empirical evidence to back this claim, is metaphysics.
What distinguishes metaphysics from physics, or branches of logic or mathematics, is that it makes truth claims without any means of verification or disproof, either through induction or deduction. This is the key point that many people seem to miss--if you can actually prove the claim, you are not practicing metaphysics, but science. And conjectural musings about alterate universes without truth assertions are fiction, usually science-fiction or fantasy.
To say that everything is metaphysics isn't really saying much, because it amounts to saying that everything is based upon the ultimate nature of reality. But that does not mean that all theory is metaphysical, because metaphysics consists entirely of conjectural assertions. If all theories are metaphysical, then all theories, regardless of correspondence with observable fact, are equal. For any metaphysical claim, there are contrary claims which stand on equal footing. Since metaphysical claims cannot be proven or disproven, lumping science into metaphysics places it on equal footing with alternate metaphysical systems which have no empirical basis. Even the emphasis on empiricism becomes negotiable, and this is where magic can be reintroduced. If you don't have to prove it, you can believe anything you want. This is the launch point for systems of New Age magic, and the same position has now been adopted for the defense of religion. This is why religious believers want to argue metaphysics--as soon as you join in the discussion on this ground, you've lost. You have unwittingly agreed to the premise that empirical evidence has no bearing on truth, and once you've done that, they can claim anything they want, and there's nothing you can do about it.
So while not all metaphysics leads to magical beliefs, all magical beliefs begin with metaphysical assertions. And since metaphysical assertions are, by their very nature, beyond proof or disproof, metaphysical arguments can never be resolved. Metaphysics is thus the final refuge of philosophical scoundrels, whose intentions are to force you to acknowledge your ignorance in a domain where there is nothing but ignorance, and use this admission of ignorance to sneak their argument past you. The only way to win an argument on metaphysics is to refuse to engage in it, which is precisely what scientists do. This is not dodging the point, but recognizing that there is no point.
This idea, by the way, has its equivalent in analytic philosophy, in some forms of mysticism, and in apophatic theology. The thrust of these traditions is that you cannot talk about the unknowable because you are talking nonsense, and worse, you may convince yourself that the nonsense you are arguing about is true, leading you away from the real truth.