No idea on the activeness of the fighting leader, but to decline a challenge allows the gym to be forcefully taken away from you. If someone thinks they can beat you for the gym, it's only fair to let them. Besides, you have the uber and they don't, so theoretically, they shouldn't win unless there is that much a skill gap, which means they deserve it more than you.
Hope that helps :3