All mammals lose the tolerance to lactose as they age. Except for a handful of mutants.

Maybe you’re descended from one of them. Note that cheese was a way to store milk with very little lactose so it was edible by all.