His lack of responsibility and ongoing involvement with his son doesn't give him the right to wreak havoc during a holiday. At the very least, he should be required to give 15 days notice and then at your discretion based on the child's schedule. How old is the child? Once he's in school and has school programs, or church programs, sporting events, friends, etc. he's not going to be agreeable to Dad interrupting his plans all the time, especially if he's not there on a daily or at least weekly basis.