His lack of responsibility and ongoing involvement with his son doesn't give him the right to wreak havoc during a holiday. At the very least, he should be required to give 15 days notice and then at your discretion based on the child's schedule. How old is the child? Once he's in school and has school programs, or church programs, sporting events, friends, etc. he's not going to be agreeable to Dad interrupting his plans all the time, especially if he's not there on a daily or at least weekly basis.
I know I have told him I need at least 10 day notice for a holiday, that I cannot know last minute. I am willing to bend over backwards to make it possible for him to see his son, but he doesn't want to I guess. My son is 5, he is in kindergarten. I talk good about his dad and everything too, I never say a bad thing about his bdad, but when the few times he does go with his dad he shows anger towards his dad, because he isn't around at all, also sometimes my son doesn't remember him. I think the whole situation just stinks.