I think ugly is a strong word to how you're saying he's unattractive. Personally, I'm more put off by the age gap because you can really notice the difference in the actor/actress' looks. Compared to all the other glamour actors and veteran ones, yes, his visuals aren't the most youthful but just like Jo-Jung suk, he has individual charms. He still is a charming man for his age and character.
I hope you're not put off by this drama because of this! :)
Just be more careful with the word choice of ugly. Unattractive, to some/ many, yes but it does come off as strong and superficial!