When you are exposed to loud music or noise, it is your hair cells which are damaged. Hearing loss occurs because loud sounds are really just large pressure waves (like when you stand next to a subwoofer and can "feel" the bass). These large pressure waves bend the stereocilia too far, sometimes to the point where they are damaged. This kills the hair cell. Since cochlear hair cells can not grow back, this manifests as a permanent hearing loss.