If you are getting child support, you have the custody. As a father he has visitation rights which he might fight for if you don't allow him to see your child. I don't think at this point you should go to court and waste time and money.. but do keep documentation if anything comes up later