Dissertation Index
Author: Louden, Sarah J Title: Cross-Sensory Perception in Music and Visual Media: A Neurocognitive Approach to Multimedia Counterpoint Institution: University at Buffalo SUNY Begun: March 2013 Completed: April 2018 Abstract: Despite differences in language and methodology, the fields of music theory, multimedia studies, and cognitive neuroscience are exploring many of the same questions. This dissertation seeks to define and describe multimedia counterpoint using parallels between musical counterpoint and multisensory scene analysis. Research in crossmodal and intramodal association, crossmodal modulation, and multisensory integration is used to define the concept of a multimedia voice, to describe contrapuntal relationships between voices, and to explore the perceptual effects of those relationships in a work. The dissertation presents the Crossmodal and Intramodal Association (CIA) model to map relationships between music and visual parameters in three dimensions: type, intensity, and quality. The model is applied in two analyses: the \"color crescendo†from Schoenberg’s opera Die glückliche Hand and a comparison of two scenes from Pink Floyd’s film The Wall. Keywords: multimedia, counterpoint, perception, crossmodal, analysis, cognition, neuroscience, psychology, Schoenberg, Pink Floyd TOC: LIST OF FIGURES.................................................................................................................... x LIST OF TABLES......................................................................................................................... xiii LIST OF AUDIOVISUAL WEB EXAMPLES...............................................................................xiv ABSTRACT ...........................................................................................................................xvi CHAPTER 1: COMMON GOALS IN ART & COGNITIVE NEUROSCIENCE ................................. 1 §1.1 Historical Precedence and Compositional Approach................................................................. 3 §1.2 Relating Topic to Multimedia Analysis and Science.................................................................11 §1.2.1 Comparison: Crossmodal and Intramodal Association....................................................... 11 §1.2.2 Interaction: Effects of Modal Congruency, Incongruency, and Modulation ...................... 16 §1.2.3 Grouping: Multisensory Integration and Perceptual Grouping ......................................... 20 §1.3 Bridging the Gap and Unraveling Artistic Intuition ................................................................22 CHAPTER2:SIMILARITYANDINTERACTION:MODALASSOCIATIONSANDMODULATION 27 §2.1 Two Classic Examples of Crossmodal Modulation and What They Show Us ...........................30 §2.1.1 The McGurk Effect ............................................................................................................ 30 §2.1.2 Sound-Induced Flash Illusion ............................................................................................. 31 §2.1.3 What is Happening? Filling in the Gaps ............................................................................ 34 §2.2 The Acquisition and Universality of Modal Associations .........................................................37 §2.2.1 Crossmodal Association and Intuition ................................................................................ 38 §2.2.2 Types of Crossmodal Associations and How We Acquire Them.......................................41 §2.3 Routes for Exploring Modal Associations ................................................................................43 §2.3.1 Crossmodal Association and Language .............................................................................. 44 §2.3.2 Crossmodal Association and Synesthesia ............................................................................ 53 §2.3.3 Laboratory Studies in Cognitive Psychology and Neuroscience ......................................... 66 §2.4 The Perceptual Effects of Congruent and Incongruent Relationships.......................................73 §2.4.1 Congruency: Amplification & Enhancement ..................................................................... 74 §2.4.2 Incongruency: Modulation & Counterpoint.......................................................................86 §2.5 Summary ................................................................................................................................89 CHAPTER 3: THE CROSSMODAL AND INTRAMODAL ASSOCIATION (CIA) MODEL ............. 93 §3.1 Johannes Itten’s Color Sphere .................................................................................................95 §3.1.1 Definitions: Hue, Value, and Saturation ............................................................................ 95 §3.1.2 Reimagining Itten’s Model ................................................................................................ 98 §3.2 The I Axis: Associations Based on Intensity ..........................................................................100 §3.3 I-Axis Counterpoint in Schoenberg’s “Color Crescendo†.......................................................102 §3.4 The T Axis: Associations Based on Type ...............................................................................121 §3.4.1 Difficulties in Measurement ............................................................................................. 122 §3.4.2 Using Crossmodal Associations with Odor and Taste for T-Axis Definition .................. 124 §3.4.3 Shape and Hue.................................................................................................................. 129 §3.4.4 Instrumentation and Hue ................................................................................................. 131 §3.4.5 Suggestions for Future Research Regarding T-Axis Associations .................................... 142 §3.5.4 T-Axis Counterpoint in the “Color Crescendo†..................................................................... 144 §3.6 Q Axis: Associations Based on Quality .................................................................................148 §3.7 Q-Axis Counterpoint in the “Color Crescendo†........................................................................ 155 §3.8 Conclusions on the “Color Crescendo†..................................................................................157 §3.8.1 Perceptual Effects of Contrapuntal Relationships ........................................................... 159 §3.8.2 Other Analytical Considerations....................................................................................... 161 §3.9 Conclusions ..........................................................................................................................163 CHAPTER 4: VOICING & INTERACTION: GROUPING AND CROSSMODAL MODULATION... 165 §4.1 How Does Grouping Influence Perception?...........................................................................168 §4.1.1 The Influence of Grouping Cues on Perceptual Organization ......................................... 169 §4.1.2 The Influence of Grouping on Crossmodal Modulation .................................................. 172 §4.1.3 Resulting Scenarios Based on Grouping Strength ............................................................ 174 §4.2 How Does Grouping Help Define Multimedia Voice Leading? .............................................175 §4.3 Grouping Principles and Their Effects ..................................................................................179 §4.3.1 Gestalt Grouping Principles ............................................................................................. 180 §4.3.2 Audiovisual Spatial and Temporal Co-Occurrence .......................................................... 186 §4.3.3 Crossmodal Association or Grouping by Similarity .......................................................... 206 §4.3.4 Grouping Based on Causality ........................................................................................... 211 §4.3.5 Grouping Based on Cognitive or Semantic Correspondence............................................ 213 §4.3.6 Learned Associations & Prior Knowledge ........................................................................ 214 §4.3.7 Grouping Summary .......................................................................................................... 216 §4.4 Comparing Audiovisual Grouping in Two Clips from The Wall .............................................217 §4.4.1 Counterpoint and Formal Structure in “Goodbye Cruel Worldâ€...................................... 218 §4.4.2 The Opening Scene: “The Little Boy that Santa Claus Forgot†....................................... 233 §4.5 Conclusions ..........................................................................................................................235 CHAPTER 5: FINAL THOUGHTS ......................................................................................... 237 APPENDICES: ..................................................................................................................... 241 2-1: Audio-Visual Crossmodal Associations .................................................................................241 2-2: Audio-Visual Intramodal Associations ..................................................................................243 2-3: Bibliography of Crossmodal Correspondences .......................................................................245 3-1: Annotated “Color Crescendo†Excerpt from Die glückliche Hand ..........................................251 3-2: Griscom’s Color-Instrument Association Tables Reorganized by Instrument Family..............256 3-3: T-Axis Associations and Regions...........................................................................................257 3-4: Comparison of Axis Associations in Schoenberg’s “Color Crescendoâ€.....................................258 4-1: Table of Audio-Visual Modulations.......................................................................................259 BIBLIOGRAPHY .................................................................................................................. 261 Contact: sarah.j.louden@hofstra.edu |