Aversiveness of sounds and its underlying physiological mechanisms in mammals are poorly understood. In this study we tested the influence of psychophysical parameters, motivation and learning processes on the aversiveness of anthropogenic underwater noise in phocid seals (Halichoerus grypus and Phoca vitulina). We compared behavioural responses of seals to playbacks of sounds based on a model of sensory unpleasantness for humans, sounds from acoustic deterrent devices and sounds with assumed neutral properties in different contexts of food motivation. In a captive experiment with food presentation, seals habituated quickly to all sound types presented at normalised received levels of 146 dB re. 1 μPa (r.m.s., root mean square). However, the fast habituation of avoidance behaviour was also accompanied by a weak sensitisation process affecting dive times and place preference in the pool. Experiments in the wild testing animals without food presentation revealed differential responses of seals to different sound types. We observed avoidance behaviour at received levels of 135–144 dB re. 1 μPa (sensation levels of 59–79 dB). In this experiment, sounds maximised for ‘roughness’ perceived as unpleasant by humans also caused the strongest avoidance responses in seals, suggesting that sensory pleasantness may be the result of auditory processing that is not restricted to humans. Our results highlight the importance of considering the effects of acoustic parameters other than the received level as well as animal motivation and previous experience when assessing the impacts of anthropogenic noise on animals.