Language and Mental Imagery

Wade Munroe, University of Nebraska, Lincoln

There is much to appreciate in Bence Nanay’s monograph. In particular, Nanay’s analysis of mental imagery is especially insightful. The expression, “mental imagery,” is a technical term introduced in experimental psychology’s nascency. As Nanay rightly argues, our use of, “mental imagery,” qua term of art, should aim to maximize theoretical usefulness in cognitive explanation. Although it’s still common to treat mental imagery as a necessarily conscious phenomenon, Nanay establishes that mental imagery ought to be analyzed without recourse to conscious experience. Instead, Nanay defines mental imagery as perceptual representation that is not directly triggered by sensory input.

         Although I have quibbles with various aspects of Nanay’s definition (and, in particular, his use of the expression, “not directly triggered,” and his distinction between triggering and modification), for the sake of space, I focus my comments on Nanay’s claims about the relationship between language and mental imagery in chapter 19. Nanay boldly asserts that, “language processing itself essentially involves mental imagery,” (p. 148) a claim I will call ‘LEIM’ for short. Although never explicitly stated, we can gather what Nanay means by “linguistic processing” from the evidence he cites in favor of LEIM,

1. Evidence that linguistic labels facilitate object recognition, visual object detection, and visual search (e.g., Gary Lupyan’s Label-Feedback Hypothesis; Edmiston & Lupyan, 2015; Lupyan, 2012; Lupyan, Mirman, Hamilton, & Thompson-Schill, 2012; Lupyan & Swingley, 2012).

• Evidence that concrete words are easier to recall than abstract words (Walker & Hulme, 1999).

• Evidence for the use of mental and motor imagery (in Nanay’s sense of the terms) in discourse comprehension in which a comprehender constructs a discourse representation over several sentences/utterances (Zwaan, 2016).

Thus, by “language processing,” Nanay doesn’t mean the operations of the language faculty in the narrow sense (Hauser, Chomsky, & Fitch, 2002) nor the operations of the language faculty in the broad sense in, say, the selection of morphophonological elements, the bundling of phonemic segments into syllables, etc. LEIM seems to be restricted in scope to discourse comprehension and semantic processing, e.g., single word comprehension, sentence comprehension, semantic association, pre-linguistic message construction that subsequently initiates linguistic processing proper (or form encoding) in speech planning, etc.

So, how does LEIM fare when restricted to discourse or semantic processing? It’s telling that the authors Nanay cites explicitly distance themselves from LEIM. For instance, Nanay cites an article by Michelle Liu (2022) on the role of mental imagery in polysemy processing in which Liu writes,

It is worth noting that the simulation view [i.e., grounded accounts that take conceptual processing to frequently involve the simulation or ‘neural reuse’ (Anderson, 2010) of modality-specific systems—systems that subserve perception and action] need not reject the claim that language processing involves abstract symbol manipulation. Nor must it commit to the claim that simulation is necessary or sufficient for all language processing. The key idea is that, in many situations, simulation is part of understanding language. (ibid, p. 177; emphasis mine)

Similarly, Nany cites an article by Fabrizio Calzavarini (2019) on the pictorial view of meaning in which Calzavarini writes,

On the one hand, some neuroimaging and behavioural/patient data support the hypothesis that visual (imagery) cortex, particularly in the left posterior/middle fusiform gyrus, plays an active and perhaps critical role in semantic competence, as far as concrete, high IMG words are concerned [where a high IMG word, in Calzavarini’s sense, is a word that “appears, introspectively, to be quickly and spontaneously associated with a mental image”]. On the other hand…I argued that more precise anatomical evidence is needed in order to conclusively demonstrate that preservation of visual (imagery) cortex is a necessary, essential condition for the understanding of high IMG words…(ibid., pp. 53-54; emphasis mine)

Several meta-analyses find that traditional language areas are the most likely to exhibit increased activation with low IMG/abstract words (e.g., Binder, Desai, Graves, & Conant, 2009; Xiaosha Wang et al., 2018). Thus, as implicitly suggested by Calzavarini, the strongest case for LEIM can be made in the domain of high IMG/concrete words (or the processing of corresponding concepts).[1] However, even when we restrict focus to the comprehension of high IMG/concrete words, LEIM is implausibly strong. Let’s take color expressions, which are high IMG/concrete expressions par excellence. There is considerable similarity in the color knowledge of sighted and congenitally blind persons, e.g., knowledge of the color of various objects (Xiaoying Wang, Men, Gao, Caramazza, & Bi, 2020), as well as in their abilities to reason about, e.g., the structure of color space (e.g., red and orange are closer than red and blue; Saysani, Corballis, & Corballis, 2018) or why something might possess a particular color (e.g., polar bears have black skin to better absorb heat in the Artic climate; Kim, Aheimer, Manrara, & Bedny, 2020). Additionally, a variety of neuroimaging work shows that, while sighted persons engage both posterior visual areas and anterior language-related areas during tasks that tap color knowledge, congenitally blind persons engage only the latter (Bi, 2021).

We don’t merely use language as a covert cue to activate information realized in brain areas outside of those that subserve the language faculty—areas where real semantic or conceptual processing occurs. Instead, language is itself a proper part of semantic and conceptual processing. Linguistic distributional statistics—roughly, linguistic-entity-to-linguistic-context co-occurrences, e.g., lexical-item-to-lexical-item, lexical-item-to-discourse-contex, etc., co-occurrences—serves as a rich source of semantic information (e.g., information about word meanings) but also information about the extralinguistic world encoded in the utterances of language users. Color knowledge of the congenitally (or early) blind is best represented with distributional semantic models that abstract and generalize over linguistic distributional statistics to represent word meanings as vectors in a semantic space (Günther, Rinaldi, & Marelli, 2019). Congenitally (or early) blind persons extract information about color from language and reason with the information using the same faculties involved in parsing and producing utterances about color.

I am happy to accept that, in many situations, simulation is part of semantic processing. However, if there is some domain of semantic processing that essentially relies on mental imagery, I’ve yet to see much evidence for it. Pace Nanay, language isn’t the ‘icing on the cake’ (p. 146) of simulation or neural reuse of sensorimotor areas in semantic processing. Language is part of the batter.

Works Cited

Anderson, M. L. (2010). Neural reuse: A fundamental organizational principle of the brain. Behavioral and Brain Sciences, 33(4), 245-266.

Banks, B., Borghi, A. M., Fargier, R., Fini, C., Jonauskaite, D., Mazzuca, C., . . . Woodin, G. (2023). Consensus paper: Current perspectives on abstract concepts and future research directions. Journal of Cognition, 6(1), 62.

Bi, Y. (2021). Dual coding of knowledge in the human brain. Trends in Cognitive Sciences, 25(10), 883-895.

Binder, J. R., Desai, R. H., Graves, W. W., & Conant, L. L. (2009). Where is the semantic system? A critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral Cortex, 19(12), 2767-2796.

Calzavarini, F. (2019). The empirical status of the pictorial view of meaning. Journal of Consciousness Studies, 26(11-12), 33-59.

Edmiston, P., & Lupyan, G. (2015). What makes words special? Words as unmotivated cues. Cognition, 143, 93-100.

Günther, F., Rinaldi, L., & Marelli, M. (2019). Vector-space models of semantic representation from a cognitive perspective: A discussion of common misconceptions. Perspectives on Psychological Science, 14(6), 1006-1033.

Hauser, M. D., Chomsky, N., & Fitch, W. T. (2002). The faculty of language: what is it, who has it, and how did it evolve? science, 298(5598), 1569-1579.

Kim, J. S., Aheimer, B., Manrara, V. M., & Bedny, M. (2020). Shared understanding of color among congenitally blind and sighted adults.

Liu, M. (2022). Mental imagery and polysemy processing. Journal of Consciousness Studies.

Lupyan, G. (2012). Linguistically modulated perception and cognition: The label-feedback hypothesis. Frontiers in psychology, 3, 54.

Lupyan, G., Mirman, D., Hamilton, R., & Thompson-Schill, S. L. (2012). Categorization is modulated by transcranial direct current stimulation over left prefrontal cortex. Cognition, 124(1), 36-49.

Lupyan, G., & Swingley, D. (2012). Self-directed speech affects visual search performance. Quarterly journal of experimental psychology, 65(6), 1068-1085.

Saysani, A., Corballis, M. C., & Corballis, P. M. (2018). Colour envisioned: Concepts of colour in the blind and sighted. Visual cognition, 26(5), 382-392.

Walker, I., & Hulme, C. (1999). Concrete words are easier to recall than abstract words: Evidence for a semantic contribution to short-term serial recall. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25(5), 1256.

Wang, X., Men, W., Gao, J., Caramazza, A., & Bi, Y. (2020). Two forms of knowledge representations in the human brain. Neuron, 107(2), 383-393. e385.

Wang, X., Wu, W., Ling, Z., Xu, Y., Fang, Y., Wang, X., . . . Bi, Y. (2018). Organizational principles of abstract words in the human brain. Cerebral Cortex, 28(12), 4305-4318.

Zwaan, R. A. (2016). Situation models, mental simulations, and abstract concepts in discourse comprehension. Psychonomic bulletin & review, 23, 1028-1034.

---

[1] Arguably, expressions don’t lie on a one-dimensional, concrete-abstract spectrum. The notions of ‘concreteness’ and ‘abstractness’ are multidimensional (Banks et al., 2023). However, it’s beyond the scope of this post to discuss the issue further.