Do the temp custody papers give you any visitation days/times? If you do, send him a text or email that you will be picking the kids are for the next visit according to the current court papers. Then try and pick them up. You can also try calling or sending his lawyer a letter asking when you can see the kids until the next court date.
Those papers are still legal BUT you can use the time you were home in defense at your next court hearing. You will need to for a hearing to give you joint custody
and specific days/times for each of you to have the kids. If you do not have any specific dates/times in last set of papers, then you are at his will until you have an order in place.