I sure do not think you stepped on anyone's toes for initially asking. I do not know what happened exactly between you and your opponent's family after they said no. Ideally maybe your coach should have handled it at that point maybe even from the start. I guess there is not much you can do about it if they say no (I mean in this case they have that right to wait for the later match). You could have scratched the final match if you needed to leave (not the best option for your son). I've seen people do that though. To me it definitely is understandable to have tried to save your family 3.5 hours on a weekend day.