mistyme, I know that right now it seems like a competition, especially when you're involved in the adversarial family court system. But please consider that when your child loses you and your ex both loses. So there really is no winner in situations like this.
I know this is going to sound harsh, but your ex had a child from a previous relationship and you felt that he wasn't a good father to that child. Yet, you chose to make a baby with him anyway. Now you have a child who could quite probably grow up fatherless. The odds are against children who grow up fatherless. They are less likely to graduate school, less likely to go to college, more likely to have drug or alcohol problems, more likely to have children while in their teens, more likely to be abused or become abusers... the list goes on. So, these are the odds that you chose for your child.
Your road is not going to be easy. You're going to have to find a way to allow your ex into the child's life enough so that the child can grow up feeling like he is loved by, and important to, both of his parents. That may mean that you have to let go of rigid standards and allow your ex to see the child when he is willing and able, and to not make the child think that something is wrong if he's not. That may mean keeping an open line of communication with someone that you dislike. That may mean having to see your ex, or allowing your ex to visit with the child when you're not there (when it is appropriate for his age). But, whatever you choose to do, if your child grows up feeling like his father doesn't want him, or doesn't care about him, it will do immeasurable damage.
From what it sounds like, he has no real desire to exercise his visitation at this point. Since that's the case, you're in no danger by setting up a comprehensive parenting plan
so that, when he does become ready, it's in place and your child can have the benefit of a relationship with his father. There are some great parenting plans on this site that you can use as a template.
Remember, one of the worst things you can do to a child is deprive them of a relationship with one, or both, of their parents. They need to feel loved and cared for by both parents. If your ex loves and cares for him, even if he isn't living the life you think he should be living, that's all your son really needs. To make it so difficult for your ex to be a part of your childs life that he chooses to disappear is to set your child up for a lifetime of hurt and problems.