In extended Higgs models, a discrete symmetry is needed in the quark sector to avoid tree-level flavor-changing neutral currents. However, this is not necessarily the case in the lepton sector. We consider a model in which one Higgs couples to quarks and three others couple to the electron, muon, and tau, respectively. This four-doublet model is presented with the full scalar potential and the gauge and Yukawa couplings. The constraints from boundedness, perturbativity and oblique parameters are incorporated as well as constraints from meson-antimeson mixing, radiative B-decays, and the diphoton Higgs decay rate. We also consider bounds from searches for heavy neutral and charged scalars at the LHC. Since the Standard Model Higgs couplings match predictions very well, we focus on the alignment limit of the model. It is shown that for a wide range of parameters, the lightest additional scalar, pseudoscalar and charged scalar can have substantial decays into electrons and muons (in contrast to the usual leptonic decays into taus). An interesting signature in the neutral sector would be the production, through vector boson fusion, of a pair of scalars, each of which decays into an electron or muon pair.