Are poisonous houseplants dangerous to health?

Certain houseplants are known to be poisonous. Are they poisonous only if ingested or touched? or are they deadly even just having them around?