Why does my cat hiss at my newborn?

My son is 6 weeks old. My Tomcat is 4 years old & stays with my Dad. Ive only brought my Son around the cat a couple times & every time I do, he hisses?? I have a female cat that is perfectly fine around the baby. What is the deal with my male cat? Im afraid he is going to attack!