Sound is moving air and carries energy. With the right frequency and intensity it could definitely physically fuck you up even if your ears are covered.

Wikipedia:

Studies have found that exposure to high intensity ultrasound at frequencies from 700 kHz to 3.6 MHz can cause lung and intestinal damage in mice