I don’t think the argument about color perception is about the actual color sensing organs (eyes). we can track where that differs i.e. colorblindness

I think it has to do with the way the brain perceives the input from the eyes. so I may see the world inverted from how you see it and I think that would remain true if we somehow swapped eyes successfully.