Acute effects of anthropogenic sounds on marine mammals, such as from military sonars, energy development, and offshore construction, have received considerable international attention from scientists, regulators, and industry. Moreover, there has been increasing recognition and concern about the potential chronic effects of human activities (e.g., shipping). It has been demonstrated that increases in human activity and background noise can alter habitats of marine animals and potentially mask communications for species that rely on sound to mate, feed, avoid predators, and navigate. Without exception, regulatory agencies required to assess and manage the effects of noise on marine mammals have addressed only the acute effects of noise on hearing and behavior. Furthermore, they have relied on a single exposure metric to assess acute effects: the absolute sound level received by the animal. There is compelling evidence that factors other than received sound level, including the activity state of animals exposed to different sounds, the nature and novelty of a sound, and spatial relations between sound source and receiving animals (i.e., the exposure context) strongly affect the probability of a behavioral response. A more comprehensive assessment method is needed that accounts for the fact that multiple contextual factors can affect how animals respond to both acute and chronic noise. We propose a three-part approach. The first includes measurement and evaluation of context-based behavioral responses of marine mammals exposed to various sounds. The second includes new assessment metrics that emphasize relative sound levels (i.e., ratio of signal to background noise and level above hearing threshold). The third considers the effects of chronic and acute noise exposure. All three aspects of sound exposure (context, relative sound level, and chronic noise) mediate behavioral response, and we suggest they be integrated into ecosystem-level management and the spatial planning of human offshore activities.