Enter Gilly, who's peppering Sam with random facts High Septon Maynard recorded -- the High Septon apparently was fond of writing down every time he took a shit. But among the steps and bowel movements, Gilly asks Sam to define "annulment" which Sam does without pausing to consider what Gilly is saying. She goes on to reveal that Rhaegar got his marriage annulled in Dorne, and was remarried in a secret ceremony. The Tower of Joy is on the northern edge of the Red Mountains of Dorne. See where this is going?
Without mentioning them by name, Gilly has discovered that Rhaegar's first wife, Elia Martell (RIP Martells) was officially cast aside in favor of Lyanna. Had Ned and Robert only known that Rhaegar had married the woman he kidnapped, they probably would've been cool with it and wouldn't have had to stage a rebellion to overthrow the Mad King, right?