Look at it from his side, maybe he's doing you a favor by being direct. Some families just have a really dry, quiet vibe and any kind of edgy humor totally bombs. Could be your timing is off and you're telling a risky joke right when everyone's trying to pass the gravy.
Honestly, you have to read the room. I've been there, making a joke about the turkey being dry right as my aunt is serving it. The silence is brutal. Sometimes the best move is to just match the energy in the room for the first hour. Listen more, ask simple questions about the food or their drive over. Save the edgy stuff for when you see someone else crack a smile first. It's about fitting in, not standing out, at least at the start.