Interestingly, in the era of this expression, “hard man” often had a somewhat negative connotation like “calloused,” which I gather is generally not the meaning intended by those who use it today.