'I'm just really shocked that his perception is that I have given nothing and done nothing to facilitate his relationship with his son...' It's not 'his perception', Amy. It's his way of trying to get his way and only that. You have gone above and beyond to fascilitate their relationship. If he's told you he's a shitty father, that's where all this BS is coming from..........GUILT! It's bad enough that he has all this guilt and he can't take responsibility for it, so he's trying to pass it on to you.
Don't let him, stick to your guns, and if he happens to turn tail, then so be it. You cannot MAKE him change, only he can do that. Yes, unfortunately your DS will suffer, but all you can do is try to soften the blow somewhat. But stand strong and KNOW you've done your child well!