Brains Blog Symposium on Concepts at the Interface
Author’s Reply to Commentaries
Nicholas Shea
Institute of Philosophy, School of Advanced Study, University of London
nicholas.shea@sas.ac.uk
Reply to Sarah Fisher
Turning to Sarah Fisher’s helpful commentary, she raises a challenging question about concepts in LLMs. I was very pleased to have a chance to discuss this with her in the CLEA online symposium about the book a couple of months ago. Sarah points to several attributes of human concept-involving thinking that are not found in LLMs. They lack many of the special purpose systems in which humans perform simulations. They do not seem to have sensorimotor representations or a cognitive map of space, for example, and it is not clear whether they have evaluative representations, much less affective responses.
This surely does mean that current AI systems cannot go in for the same kind of concept-driven simulation as humans do (according to my account in the book). However, some of these capacities may be added to future AI systems. There is also the question of what it takes, for example, for an episode of computational processing to count as a sensorimotor simulation. If an LLM learns the qualitative similarity space of the colours (Søgaard 2023), does it then have a special-purpose representation of the colour space in which it could run simulations? If so, it may be that even current LLMs have a special purpose system with the capacity for sensory representation. These are all tough questions, both empirically and theoretically. Mechanistic interpretability of trained DNNs is in its infancy. Theoretical refinement will doubtless have to continue in parallel with unfolding new findings about the models.
Sarah’s discussion presses the question of which attributes of conceptual thinking are essential to the nature of concepts and which are not. Many of the features I point to could be systematic and important aspects of the way concept-involving thinking works in humans without being necessary to concept possession. Does a lack of the capacity for ‘plug-and-play’ with special purpose resources disqualify an AI system from possessing concepts?
My inclination here is to take ‘concept’ to be a theoretical term and to allow that there are many different ways it can be defined. I present an account in which concept-driven thinking draws on rich and heterogeneous stores of information, takes place in working memory, calls on the existence of a global playground, and makes use of executive functions like inhibition, goal-sensitive direction of attention, and metacognitive monitoring and control. We could define a notion of concept which required all this to be in place to count as possessing concepts. Alternatively, we could define a much narrower notion, perhaps just focusing on the capacity for general-purpose combination. That would allow a much wider range of systems to count as concept-possessors, extending to young children, computational systems and perhaps some non-human animals. But then to possess concepts in that sense would not confer the capacity for the powerful forms of productive inference that I have described, except in those systems that have some or all of the further attributes I have just listed.
It might turn out eventually that there is a clear best way to define ‘concept’, perhaps because there is a sharp natural kind in the vicinity. Or it could be that actual cognitive systems, natural and artificial, display more or less of these attributes in a graded fashion. Either way, one way to proceed is to be clear about which collection of attributes we are pointing to and to say what they explain. That is what I tried to do when building up a picture of thinking in the human case. Rather than arguing for a preferred way of defining the concept concept, I tried to lay out a set of explanatorily relevant psychological features and to show what they explain.
That is also why I started, not by defining ‘concept’, but by pointing to instances: the representations which form freely-recombinable elements of the thoughts we entertain in deliberation. Proceeding by ostension means that even these features need not be necessary properties of concepts. It could turn out that the representations involved in deliberation are not freely-recombinable. The flexibility of cognition could perhaps be achieved by some clever system of inferences between representations displaying only special-purpose compositionality. Then we would say that we were wrong about one of the properties of the phenomenon we were pointing to. We could then adjust our theory of concepts accordingly, without changing the subject.
The upshot is that we do not have to hold that plug-and-playability is a requirement for genuine concept possession in order to argue that a deep difference between human cognition and (current) LLM-based chatbots is that the computer systems do not go in for the hybrid form of cognition that I describe in the book. If I am right, this is what allows humans to partially avoid, and partly solve, the notorious frame problem. It also makes human thinking an especially powerful way of working out what is the case and planning what to do.
LLM-based systems can certainly be supplemented with pictorial or video input. When interfacing with a robotic controller they might count as making use of sensorimotor knowledge. Language-derived tokens and the state space embeddings learnt from natural language may provide a crucial architectural and developmental resource [Buckner forthc]. On the other hand, it could turn out that fitting different special purpose representations together in an interconnected overall representation of a situation – a global playground – is critical. That too could be just around the computational corner. Or it could be that this element of human cognition is far beyond the ambit of current or near-future computational systems.
References
Buckner, C. (forthc). The talking of the bot with itself: Language models for inner speech. In H. Cappelen & R. Sterken (Eds.), Communication with AI: Philosophical Perspectives. OUP.
Søgaard, A. (2023). Grounding the Vector Space of an Octopus: Word Meaning from Raw Text. Minds and Machines, 33(1), 33-54. https://doi.org/10.1007/s11023-023-09622-4