I used a diaphragm briefly and had to leave it in 12 hours after intercourse. Are you sure the four hours is correct? I realize things could have changed since I last used it, though

But the 12 hours was to give the spermicide time to kill all the sperm.
Anyway, no, semen should NOT be getting past the rim of the diaphragm if it has been fitted correctly. That defeats the purpose of the device! When you go to take it out, there should be a tremendous suction that you have to break. If you're able to pull it out easily without a "pop," then I would say it is the wrong size, positioned incorrectly, or being dislodged during intercourse.
Cervical secretion inside the diaphragm is normal, since the diaphragm is fitted around the cervix and the secretions are collecting in it.