The Future of Remembering, Article I: Foundations of Forgetting

This is the first in a four-part series from the Personal AI Team. The Personal AI team is enabling individuals like you to create a Personal AI that always remembers, so you never forget. Just how the team is making that happen will be explored through four distinct lenses: Scientific, Technological, Design, and Entrepreneurial. The first article is from Marc Ettlinger, Computational Linguist at Google and Personal.ai Scientist.

On Forgetting

Humans forget. It’s part of what makes us human and striving to remember has shaped a tremendous amount about how we live.

Think about that for a moment. Think of all the things in our lives that we’ve created or that we do because of our imperfect memory: The origin of written language was to help us remember financial transactions. We carry little notebooks to remember our ideas. The invention of photography has helped us to remember visual scenes. Statues, pyramids, and buildings have been built to remember rulers and legacies past. Maps help us remember how to navigate places we’ve been to before. Innumerable religious and secular holidays serve to commemorate, to remember, events in history. There’s also research to suggest that there’s a purpose to forgetting: Forgetting helps us reason and understand by forcing us to generalize and abstract.

While forgetting has led to incredible things, both profound and mundane, it also causes us no end of frustration and even catastrophe. Memory loss can even lead to us forgetting who we are and has otherwise resulted in untold tragic loss. This is why humanity has worked so hard to combat it. Many examples of this listed above are what can be termed cognitive technologies: things we’ve developed to augment our brain’s function. Writing is a cognitive technology that augments our ability to remember and also our ability to communicate. Coffee is a fun little cognitive technology that enhances our concentration and alertness and has been credited with launching the Renaissance in Europe. Other cognitive technologies to help with memory include number systems, dictation machines, the computer, and even giving something a name has been shown to improve people’s memory for the concepts they represent.

So, what’s next? What are the next sets of cognitive technologies on the horizon? We can look to the artists, dreamers, and story tellers and juxtapose that with a solid understanding of what technology can currently support to chart a path for the next critical innovations in memory augmentation.

The Next Cognitive Technologies

For inspiration, consider the following bits of science fiction:

• Black Mirror: “The Entire History of You” — Grain technology records people’s audiovisual senses, allowing them, and others, to re-watch their memories.
• Black Mirror: “Be Right Back” — Technology now allows people to communicate with artificial intelligence built from loved ones’ online communications and social media profiles. Eventually, there is an embodied version, a robot.
• Neuromancer — Among many other harbingers of the future, this seminal cyberpunk novel explores a future where people’s consciousness can be connected and uploaded to the matrix, a precursor to the internet.
• HAL 9000 in 2001, Samantha in Her, KITT in Knight Rider, Max Headroom, WOPR from WarGames, etc — Computers that are interacted with conversationally and have many human-like traits, like personalities and desires.
• Altered Carbon — Your entire memory is stored in your Stack chip that integrates with your mind seamlessly allowing people to transfer from one body to another.

Distilled, we can highlight a number of key features of what the next generation of memory augmentation could look like:

1. Ubiquitous Multi-Modal Capture of Experiences — Capture all senses we experience and perhaps even our thoughts and emotions
2. Meaningful Abstraction and Reasoning — Don’t just capture the raw data, but understand the data in a way that allows us to query it, find it, interact with it and perhaps even learn from it.
3. Ambient Presence — Accessible any time, any place
4. Natural Human-Computer Interaction — The closer it is like interacting with a human, the better. Indeed, the pinnacle of this may be a digital copy of ourselves with all our memories.

In other words: A representation of you and your memories that you can interact with like a human being.

And so where are we, in terms of technology, in achieving each of these?

Ubiquitous Multi-Modal Capture: We can currently capture anything that can be sensed — particularly sounds and images, but not thoughts and emotion. The limitations here are mainly bandwidth and storage.

Meaningful Abstraction and Reasoning: Some of the technology is quite advanced: Object recognition can do an excellent job of parsing a scene for objects, Automatic Speech Recognition can get within 5% error rate of understanding the words in an audio stream. Classifiers have reached pretty high rates of doing things like identifying relevant entities, classifying text in any number of useful ways and extracting meaningful summarizations. Where it certainly falls short is genuine AGI and reasoning.

Ambient Presence: With cloud computing, you can access unlimited compute with a phone and a cellular data plan.

Natural Human-Computer Interaction: While we don’t quite have Her or HAL, technologies like Alexa and Siri have gotten pretty far in creating a way to communicate naturally with computers. No longer do you need to know how to code, or even click, to do something like find out the weather or the Red Sox score. You can just ask.

Personal.ai’s Approach

These four characteristics are the essence of what we’re working to build at Personal.ai. Our approach is incremental, building out functionality to support specific use cases relating to memory retention and recall. The specific problems we are seeking to address are:

Short Term to Long Term Memory:

The human memory system is complex, with many subsystems proposed to explain what we remember and forget at different time scales and in different ways. A key phenomenon is rapid memory decay of most of the things we experience juxtaposed with very long term memory for other things. The ability to make long term memories out of short term memories has been argued to be a key to intelligence and reasoning.

Personal.ai’s solution: One core use case for Personal.ai is to record your memories and make them readily accessible to browse and recall. So, you can ask it what the action items were from this morning’s meeting, the name of your friend’s spouse you met the week before, or the Berkeley restaurant recommendation you got last month. Personal.ai creates a digital long term memory — called a memory stack — that is structured and lasts forever.

Lost Train of Thought:

There is a very interesting phenomenon where we lose our train of thought, often because of an interruption. This is because there is only one memory subsystem available for focusing and keeping track of what you are working on at any given moment. Once something else enters that subsystem via external interruption, or even other intrusive thoughts, that original stream of thought is gone, and must be retrieved from longer-term storage. Depending on what you’ve forgotten, that might be pretty difficult.

Personal.ai’s solution: Personal.ai records and displays what you’re saying in real time, with no need for offline processing. Personal.ai also uses generative transformers to create predictions of what you might say next. Both of these things can help prime you when you forget what you were saying.

Predictive Response:

Not a memory challenge, but the final core use case is Personal.ai being an AI version of you that can respond to prompts. Similar to autocomplete, Personal.ai can take as a seed a question or a statement and respond based on your memories and your previous responses.

Personal.ai’s solution: This functionality comes later — you can imagine the input necessary to mirror you — but eventually Personal.ai can automatically respond to tweets, answer questions from people in your organization, or pre-write emails and slack messages for you to approve.

Summary

In the long term, AI will have transformative effects on our lives. It can be difficult to know how, however, and figure out what advancements will have the clearest impact. In the short term, though, we see a path where specific AI technologies around classification, speech recognition, text prediction, and personalized machine learning can immediately address some of the shortcomings of human memory. Personal.ai will help remember the things we want to remember, keep track of what we want to say, and even figure out what it is we want to say. In this way, Personal.ai is another in the line of cognitive technologies that we have been inventing for generations to do these very same things.

In the next post we’ll dive more deeply into the technology we will use to solve some of these problems. Then in a later science post, we’ll go into more detail about how these various memories systems operate in humans and how Personal.ai will integrate with and augment them.