About 20 years ago I read about an experiment, where they played a game involving a ball with some young kids that had just started to learn to speak. They did a test to evaluate their vocabulary and played the game. Then they came back a year later. Again they evaluated their vocabulary, and asked if they recall the game.
The interesting part was that the kids that had not learned the word ball when they first played, but since learned it, would not use that word when recalling the rules of the game. Based on this experiment, they suggested that there was a close link between the available vocabulary and formation of memory.
I told my dad about this. He had his major in Norwegian (our native language) and had taught Norwegian to immigrants from Pakistan and similar in the 70s, and had a keen interest in languages.
He found it very interesting, as he had read studies showing that immigrants who lost their native language also lost knowledge they learned before they emigrated. Things they had learned at school or university for example.
So both sets of studies seemed to point to the same cause, that the memories formed were closely linked to the language you know at the time of formation.
As such I'm not terribly surprised that the same seems to be happening with visual memories, as this article suggest. Then again, totally not my field.
As an ESL who grew up in Slovenia, did most of his work in English, and now lives in USA with full immersion[1] – the memory language barrier is very noticeable.
Talking to my mom and sister about things that happened in Slovenia is easy and fluent. Talking about things that happened in USA is … hard. I have to translate, I can’t find words, the concepts make no sense, everything is awkward.
Make me talk about software engineering in Slovenian and I sound like a bumbling fool. It just doesn’t work. Every 3rd word is English. Entire sentences sometimes.
[1] I speak Slovenian maybe a couple times per month nowadays
This is likely because language uptake is most successful when "unrooted" to a pre-existing language framework. You're utilizing distinct memory/knowledge graphs.
You sound like a bumbling fool because you're having to forge novel connections between the two recall graphs.
It's why I look on awe on people like my high school Spanish teacher who fluently spoke 11 languages. When she got upset, you never knew which was going to come out!
I think that's fairly normal. When you learn a concept in a language it's difficult to convey it in a foreign language if you don't have the vocabulary, even if you're a native speaker. That happens to me all the time.
> He found it very interesting, as he had read studies showing that immigrants who lost their native language also lost knowledge they learned before they emigrated.
That example would be a negative aspect of it, but forgetting things in itself can be positive. For example, I wonder if this might also work for trauma and rebuilding a new life after migrating to a different country.
“People think that epilepsy is divine simply because they don’t have any idea what causes epilepsy. But I believe that someday we will understand what causes epilepsy, and at that moment, we will cease to believe that it’s divine. And so it is with everything in the universe” — Hippocrates
Honestly, this is kinda scary. Some of the concepts that are stored in close proximities feel like they tell... stories. Possibly very individual ones.
For example, this is the "tag cloud" for voxel [26,35,42] right ("Good, very reliable"):
This brain map was constructed from fMRI scans of participants listening to fictional podcasts called "The Moth". The fact these words are all grouped together suggests a high level of `arousal` which may be related to an emotive part of the story's narrative. You may very well have deduced part of the story.
"child" does seem like an outlier when you consider individual words but it's the overall context that's important.
Your second example highlights many words that related to time.
On some level it makes intuitive sense that people listening to the moth would relate the stories they heard to other stories, either that they had experienced directly or indirectly, fictional or non-fictional. It does seem really personal and almost invasive regardless!
I can only assume/hope that people consent to bring imaged like this.
No, that's your fantasy and I wonder what dark thoughts you have that you made this leap. For example you inserted "death" (and for that matter "child") out of the blue when it wasn't even there! Out of thre 3 words you speculate about, only one is in the list above, two are purely your own mind's product.
The first cluster seems to do with homemaking, accommodation-finding, living arrangements and stresses involving this.
The second one is about family and processes/conflicts/stress around it.
It is really interesting to see how these are tangled together!
One simple experiment you can do is, think of a cup then a image pops up. That image will likely be the image of recently used cup or something you cherished deeply. Now after a while if you read "a cup" you see same associations! Of course I removed a lot of details, but it's really easy to verify the idea presented here.
And something like CLIP[0] seems to be a good direction because it combines both visual and language aspect!
I have aphantasia[0], when I think of a cup, no image pops up. I can think about the properties of a cup: it holds a drink, it's a physical object, and if I specifically think about what color or shape a cup is, I can describe it; but I never visualise it.
I wonder how my brain works in this context. I also have wondered if this contributed to me being fairly good at programming and systems thinking; but dismal at art and anything creative.
Can you feel the cup? Can you imagine holding it in your hands, caressing the rim, scooping the inside, and feeling the pressure on your hands? Can you imagine your hands getting wet or even the pain of a thousand shards of glass as the cup suddenly shatters? Does it make a sound? Can you smell the tea? Can you taste it?
Not the same person, but have a similar condition, so I’ll take a go at it. Sample size of one here.
> Can you feel the cup?
No. There really isn’t a specific cup.
> Can you imagine holding it in your hands, caressing the rim, scooping the inside, and feeling the pressure on your hands?
Yes, I can imagine these actions, but it’s more about remembering what it might be like to feel that sensation again and is somewhat disconnected from the cup. I end up wanting to remember a different, real cup, when I might have done those actions. Like how I can remember when I last cut off circulation in my hands carrying a heavy plastic bag, say. I know how that feels, but it’s not something I’m particularly good at imagining visually - with a mental image.
> Can you imagine your hands getting wet or even the pain of a thousand shards of glass as the cup suddenly shatters?
Not quite, and no, I can’t really imagine the shattering. I mean, I can remember a time when a cup fell and shattered and sort of associate that, but just “visualizing” the pain or shattering seems impossible. I can play concept association and “imagine” the cup exploding outwards like a grenade, but it’s not visual or concrete, it’s more conceptual and I lose any connection to pain or feeling or hands and fingers. The more I try to make up without associating with a memory, the more abstract (verbal, less real) the “imagining” seems.
> Does it make a sound?
I can imagine sounds, and get songs stuck in my head, so I have some idea of how mental visualization should work for me, but it doesn’t visually work that way. It’s interesting that to me, just now, imagining the sound of a cup shattering produces an associated concept mentally than the reverse … I can’t easily go from a shattered or falling cup concept to the sound, but I can relatively easily do the reverse: I can go almost instantly from imagining the sound of a shattered cup to thinking of a shattered or falling cup and what that might look like or how the shards might feel.
> Can you smell the tea?
Again, thinking of smells first brings back sounds and memories more strongly than visuals would for me, and when I was trying to think of an image, it was abstract - there were no sounds or smells in particular and it wasn’t something I could interact with. More of a model of a cup. Even thinking of the interactions or how something would feel, I now realize, is thinking about feelings and memories of feelings - that visualizing my other senses such as touch, smell or hearing is much easier for me than to visualize based on appearances.
> Can you taste it?
Not from the mental imagining of the cup, no. If I remember the sounds of when I was at a tea house, or if I pull up a photo of a cup of tea on my phone, then I can imagine the taste. But it feels disconnected to try and purely jump from a “visual” cup to thinking, it tastes like tea. Thinking of how certain foods taste does bring up memories and can make my mouth water imagining or remembering how certain foods tasted, though. Thinking of a taste can bring back the smells, the feeling of heat rising off a dish or hot plate, the sounds of sizzling, and a memory of what it would look like. But there still isn’t really a “picture” and if I wasn’t looking at a photo on my phone, all I could tell you about my visual imaginings would be words or concepts. I can tell that something is missing in my imagining because of how different it is to go from non-visual senses to other non-visual senses. But visuals … don’t have the same mental impact unless I can see them in front of me, e.g. on a computer screen. For others, remembering a layout might trigger memories of details, for me it is the opposite - I remember the layout of a flowchart by remembering which details are connected and then from there I can describe the diagram’s layout in my head.
It’s about the same as anyone else, I think, though I remember lyrics more strongly if I remember the concepts or melody or rhythm that I can associate with the lyrics. If I have a particularly strong feeling about a song or lyric or if I repeat it and sing along, I can easily remember it for some time after. It’s not like my memory works better or worse than others (as far as I know), it’s more like I can’t as easily store and recall a visual memory without converting it first to concepts or details and storing those instead. The idea that I can see someone’s face clear as day in my mind or see my life flash before my eyes seems incredibly foreign, almost like a superpower.
The problem seems to be visual memory, not auditory or verbal. And it’s not that I can’t understand visual concepts better - e.g. a YouTube video or diagram can have a greater impact on my understanding than a verbal speech or audiobook. But it’s about remembering concepts and details - there’s no “fuzzy visual” for me to latch on to. Either I remember a detail or concept clear enough, almost as if it were described in words, or I don’t.
If anything, I find it weird that I can easily recognize visuals or logos that are similar to what I’ve seen before. E.g. to say “that looks like the Spotify logo,” while needing to actually look up the Spotify logo to “see it”. I suspect I end up “pattern matching” the concepts and details I’ve seen visually - e.g. round, green and black, sans-serif semibold type, rectangular, relative thickness/margins, and so on… and match on that. Like anyone, my memory can be faulty or incomplete - I’ve less reason to remember something I haven’t seen or experienced as often.
I do wonder what a scientific study might uncover or have uncovered about this condition. All I can say is of anyone else I know who might have it, most don’t think about it as much as I have. It’s uncomfortable to think that you’re deficient in some way for being unable to do what “everyone else” can do, especially when it doesn’t seem to affect your day-to-day life in a particularly visible way.
I suspect it is why I liked computer software so much, though. I could move and manipulate visual things so easily that otherwise seemed impossible. I could imagine that just like glasses help correct vision deficiencies, a future AR headset could easily help correct a lack of visual memories through automated capture and some form of recall querying. Visual mental manipulation would probably be harder to mimic though, and more like today’s software, I imagine, in which you have to “issue instructions” a bit more. If I had to say an obvious advantage to this condition, it’s that any visual brainstorming I do can be easily shared as I do it, since I’d be using paper or digital means anyway.
By comparison, if I have a song idea, it’s nearly impossible to reproduce it exactly. I’ve never had that issue visually - either the visual in front of me matches the idea I had or it doesn’t, there’s no perfect visual idea in my head to try and match up to or fall short from. In that way, it is perhaps a bit easier for me to “live in the moment,” and to share ideas I have in a visual format, not that I’ve given it much thought before. I think it leads me to have less creative visuals though, as their basis is almost always a concept of some kind, or merging of concepts.
I've read a number of times about aphantasia, but I'm still unsure about things. For instance, if I'm asked to think of a cup in my minds eye, I would say yes, I can. Therefore I don't have aphantasia. But it is not at all like looking at a real cup or a picture of a cup. It is far more liminal and fleeting.
One bit of evidence of this is when I tried LSD a few times in the past, I don't get CEV (closed eye visuals), while some people report rich, colorful scenery. All the same, I do get visual impressions without actually seeing anything. For instance, while listening to a particular song it generated nothing but blackness in my minds eye, yet all the same I had the impression of looking at glossy flower made of hard glass. I was getting the interpretive conclusion of vision without any of the other information flow that comes along with actually seeing the object.
When I visualize things in my mind, it is kind of like that too -- I can describe what I'm seeing but I don't really see it.
I don't know how much LSD you did or what your body weight was or anything else that might be chemically relevant. But I would be deeply surprised by an individual being immune to the closed eye hallucinations.
When you look at a lightbulb for a moment and then close your eyes, do you get the after image? The "closed eye hallucinations" are just that normal after image chopped up in frequency space. LSD exposes the fact that a lot of what the brain does can be understood in terms of analog signal processing. The most reliable way to get a high quality viewing of the crystal patterns is with TV static. Baring that, any randomized constant texture such as a carpet will suffice. The reason is because random noise evenly fills up the entire spectrum in frequency space, so whatever frequencies are cut, de-phased, or otherwise affectef are guaranteed to be present in an image of pure noise. Another reliable glitch is that all lines below a certain thickness will tend to wobble or drift in and out of visibility. Again, construct a line as a really thin gaussian wave packet and see what happens when the high frequency terms pick up phase drift.
LSD hallucinations have nothing to do with aphantasia.
I got vivid, rich, and colorful closed-eye imagery from psilocybin (20-20mg) and no open-eye visuals from it. LSD was the reverse -- obvious open eye visuals, no closed eye visuals (up to 200ug).
LSD was brought up because of the specific experience of having visual impressions of things without actually seeing anything. My sober "minds eye" is like that too -- I can describe what I've been asked to conjure, but I don't really see it.
> That image will likely be the image of recently used cup or something you cherished deeply.
I ended up imagining a comically ornate, little, blue, fine-china teacup. It's not one I've used recently. In fact, I can't remember ever using such a thing! But the image arose effortlessly so I must have seen it at some point, no? Maybe in a videogame or at a friend's house?
I don't know; the same thing happens when I attempt this exercise with the words "tree" or "car" or "pencil". It's almost like I'm imagining my own, personal platonic ideals.
On the other hand, when I think of a duck, it's a mallard rather than a call duck, and I definitely see more mallard than call ducks in my day-to-day life.
It makes sense. If I'm asked to visualize a cup, I probably first think of an emptiness that can be filled, then I enclose that, while I'm enclosing it I remember that it needs a hole to be filled or drunk from, and an orientation so I can put that hole on the top and its opposite on the bottom so I can sit it on a table - it has become a cylinder, now I remember a particular, very cylindrical glass that I own that has a Norman Rockwell painting screened onto it, then I realize that I've thought the word "glass," so I get rid of the painting and make it plastic. Then I make it blue, probably because that's the first color I go to when somebody asks me to name a color.
If I come across the word "cup" for a little while, I'll just remember that time when I tried to visualize a cup, and think of a blue cylindrical cup.
Likewise. I think, “can hold liquids, handheld, might be hot or cold,” probably a white ceramic comes to mind, maybe something I might have seen at IKEA. No handle, no decorations, a cup isn’t defined by those and they don’t show up in my imagining of one. Smaller in diameter yet about as high or maybe a bit taller than a bowl. Bigger than a thimble, roughly the size of my hand, or easy to palm. Probably in a kitchen or on a dining table. It’s all rather conceptual for me.
What’s weird for me is if I do the reverse, and instead of trying to jump from visually imagining a cup to associate other senses, if I instead think of the feeling of a warm cup in my hands, I can more easily imagine what it smelled and tasted like, and even a particular memory comes to mind too. But if I try to do the reverse, such as visualizing that same memory as a picture in my head, I can’t recall the heat or other details, I end up thinking of either concepts (words) or it’s like a photograph and flat, not real life. For me, associations from “imagery” are harder than other kinds of associations and don’t have much impact at all. Strangely, I can associate positionally, though. E.g. that something is on a third shelf or inside a closet, and have that trigger memories. But I can’t do so by remembering what the shelf or closet looked like.
" memory isn’t a facsimile of past perceptions that gets replayed. Instead, it is more like a reconstruction of the original experience, based on its semantic content."
I thought memories as reconstruction not recall model was an already pretty commonly accepted view.
One of the examples were people with seemingly perfect memories of their lives. On closer inspection it turned these perfect memories were actual constant replays/refreshes.
EDIT: One example I remember reading about was that Nixon aide who claimed perfect recall but actually he was was conflating multiple dates into one.
So it is with us with our childhood memories, we might be conflating multiple occurrences into one when we reconstruct our stories.
The ability to generalize patterns is the link between current brain-computer-interface (BCI) technology that must be recalibrated daily and solutions that might generalize for a single brain across time or even multiple brains (of which no two are alike).
Clearly, we respond to our fellow human beings not by processing the activity of their 80 billion neurons, but by recognizing latent patterns in their expressions and the vibrations of the air around them (sound/speech).
It looks like these models are the beginning of a path where software ascends that learning curve.
The other comments in this thread talking about aphantasia has caused me to realise/think I have that condition (and have had interesting chats with my family about it today).
Now in context of your comment I'm wondering if people with aphantasia wouldn't perhaps do better at (current) BCI than those who have very image-intensive ways of thinking?
Or perhaps it's the other way around - perhaps someone with aphantasia would do worse at BCI, especially if the current tech is based on interpreting images or something.
To take a single-sentence stab at summary: It seems the brain stores semantic descriptions of what we perceive, and memories are reconstructions from those descriptions rather than retrieval of the actual perception.
Wow! When you read these things, and combine that with the advances in AI, it really seems that understanding what happens in the brain is within reach.
Less worried about AGI and all that, more worried that taking away all the 'magic' of how minds work does it take away some of the magic of how the minds work, i.e. it could be quite depressing for many to think of ourselves as 'mechanistic' automatons. Yes that has been discussed for years, but as that becomes fact and with blueprints ... I dunno, I think I quite liked the world how it was before that!
Also the more we can model the minds workings, the more some sod is going to exploit that for commercial benefits i.e. advertising, propaganda etc.
> I dunno, I think I quite liked the world how it was before that!
I agree. I don't know why but when I read this article and the responses here stating how things can be related and can be measured, I suddenly felt this sigh? disappointment? for 'demystifying the mind'.
Yes exactly. It won't take long till someone exploits it which would be more robust than the prediction models we have right now and that's what I don't want a single dime of.
I know we're not there yet, and perhaps we won't ever be, or we'll be there in 10 years, but we don't have to map every single thing in the brain in order to exploit the foundational behavior.
Not just for years. For centuries now. Wasn’t that people’s main concern about Darwin, after all? “Once it is proven that we have descended from apes, where does that leave us?“
Not sure why you are getting downvoted. I would say that AGI is even more dangerous than the Atom bomb. It will still get invented though, we just cannot help ourselves. And to be honest, I would be first in line to invent the thing if I could.
Anecdata: I was born in Russia but came to the US at the age of 11 years old. I think in English exclusively.
One tidbit I read is that people who think in a foreign language like me are in some decision processes more-rational [0]. I think I don't get hung up on emotionally-stirring words, or get offended by vulgarities, as they feel more abstract to me than visceral. May be the same mechanism.
I can think in Russian, but with a 5th-grader-like vocabulary I'm missing out on ways of articulating some things. It also takes effort. And now that I'm trying it out as an experiment - it feels like I have a concept/idea, and then I'm simply trying to narrate it.
I speak Russian to parents and my brother, but it often feels like I have a concept I want to articulate and have to use a bunch of non-SAT words to vaguely gesture at what I want to say; or, I simply don't know a word like "easel" in Russian and feel like I'm playing the game "taboo" :P
I think naturally I gravitated out of thinking in Russian since it's too much effort for no payoff. One thing I miss about Russian are all the prefixes you can add to a verb, like "drive over", "drive into", etc are accomplished with short prefixes: "pere-" or "v-". It doesn't sound like much, but when you can apply the dozen-ish prefixes to the word "fuck" you realize the potential English is missing ;)
I do wonder if neuroscience may validate Chomsky's hypothesis, that langauge evolved and serves primarily as a tool for thought and not as a tool for communication, in his lifetime.
It seems intuitive that perception and memory are one and the same. How we perceive and how we recall is intrinsically bound to the neuron superstructure. Changes in one affects the other. The mind really is an interconnected prediction machine. Fascinating stuff!
The interesting part was that the kids that had not learned the word ball when they first played, but since learned it, would not use that word when recalling the rules of the game. Based on this experiment, they suggested that there was a close link between the available vocabulary and formation of memory.
I told my dad about this. He had his major in Norwegian (our native language) and had taught Norwegian to immigrants from Pakistan and similar in the 70s, and had a keen interest in languages.
He found it very interesting, as he had read studies showing that immigrants who lost their native language also lost knowledge they learned before they emigrated. Things they had learned at school or university for example.
So both sets of studies seemed to point to the same cause, that the memories formed were closely linked to the language you know at the time of formation.
As such I'm not terribly surprised that the same seems to be happening with visual memories, as this article suggest. Then again, totally not my field.