My nephew started at 4, almost 5. He is in his 3rd year of 6/U. He actually will wrestle some at 8/U do to wanting to wrestle up to a little better competition.
I personally don't think 4 is too young. Just don't expect him to have the attention span the other kids do. 4 year olds just don't have it. They start many other sports at 4, why not wrestling???