Why is there so much A.G.I. talk? And what can we do about it?

You can tell all kinds of stories about why A.G.I. talk exists but it is really built into the DNA of the computational work that goes into building AI. It's best to dial down the polemics on A.G.I.

[This post draws on another post I wrote a long time ago, about another AI controversy.]

My colleague Ben Recht has a characteristically acerbic post in which he asks: why is there so much talk about the so-called “artificial general intelligence” (A.G.I.) these days? Unlike some other writers who treat the issue with some deference and as a philosophical concept worth deconstructing, Recht tries to demystify it by leaping straight from the philosophical sacred (A.G.I. is about the limits and possibilities of computation and technology) to the organizational profane (A.G.I. is a bogus concept, a boondoggle, inflicted on us by a gaggle of Silicon Valley insiders who would like us to give them permission to build more technology by having us think that there’s more at stake than their advancement and profits).¹ For Recht, talking about A.G.I. means being in thrall to a religious cult, so “[the] AGI god shares the values of the Silicon Valley founder, funder, and engineer.”

Obviously, A.G.I. is a story we tell about technology so far be it for me to argue that this story is wrong. Stories can’t be wrong or right; they have many different characters and events that can be interpreted in multiple ways. So if Recht wants to emphasize that the A.G.I. story is a high-intellectual gloss on something that is profoundly grubby—profits—there is nothing inherently wrong with emphasizing it.

But I think there is a reason that the A.G.I. concept keeps emerging over and over in the history of the 75-year old research enterprise called AI (the AI enterprise is older than Silicon Valley). And that has to do with the practices of AI engineers and researchers.

What are these practices? In what I have found to be one of the best descriptions of what it means to do computational research, Phil Agre, who worked both as an AI researcher and a social scientist, points out that AI researchers rarely care about ideas by themselves. Rather, an idea is only important if it can be built into a technical mechanism, i.e. if it can be formalized either in mathematics or in machinery. Agre calls this the “work ethic”:

Computer people believe only what they can build, and this policy imposes a strong intellectual conservatism on the field. Intellectual trends might run in all directions at any speed, but computationalists mistrust anything unless they can nail down all four corners of it; they would, by and large, rather get it precise and wrong than vague and right. They often disagree about how much precision is required, and what kind of precision, but they require ideas that can be assimilated to computational demonstrations that actually get built. This is sometimes called the work ethic: it has to work (p13, my emphasis).

What exactly does the “work ethic” entail? AI researchers end up performing a delicate two-step between the “social” and the “technical” domains; this is built into the DNA of AI research but it can also explain its fraught politics.

Agre argues that the work of AI researchers can be described as a series of moves done together, a process that he calls “formalization”: taking a metaphor, often in an intentionalist vocabulary, (e.g. “thinking,” “planning”, “problem-solving,” chatting, playing a game), attaching some mathematics and machinery to it, and then being able to narrate the working of that machinery in intentional vocabulary.

This process of formalization has a slightly schizophrenic character: the mechanism is precise in its mathematical form and imprecise in its lay form; but being able to move fluidly between the precise and the imprecise is the key to its power.²

But it is hard to build working programs; and to build working programs requires a compromise in the “idea” that the working program is supposed to demonstrate. Here is Agre again:

To get anything nailed down in enough detail to run on a computer requires considerable effort; in particular, it requires that one make all manner of arbitrary commitments on issues that may be tangential to the current focus of theoretical interest. It is no wonder, then, that AI work can seem outrageous to people whose training has instilled different priorities—for example, conceptual coherence, ethnographic adequacy, political relevance, mathematical depth, or experimental support. And indeed it is often totally mysterious to outsiders what canons of progress and good research do govern such a seemingly disheveled enterprise. The answer is that good computational research is an evolving conversation with its own practical reality; a new result gets the pulse of this practical reality by suggesting the outlines of a computational explanation of some aspect of human life. The computationalist’s sense of bumping up against reality itself—of being compelled to some unexpected outcome by the facts of physical readability as they manifest themselves in the lab late at night—is deeply impressive to those who have gotten hold of it. Other details—conceptual, empirical, political, and so forth—can wait. That, at least, is how it feels. [p13, my emphasis].

This is why the debate over A.G.I. keeps happening over and over in the history of AI. As Recht describes it in his piece, A.G.I. has often been either already there or just around the corner at various different times in the last 75 years; I’ll add to his examples:

“Our system played checkers and chess and improved” - AGI

“Our system proved theorems in a manner that was better than Bertrand Russell” - AGI

“Our system looked at a block world through a camera and understood what to move to change it” - AGI

“Our system was able to give an output of a patient diagnosis when given an input of a whole list of symptoms” - AGI

Recht’s examples:

“Our system beat Atari” - AGI

“Our system beat Lee Sedol at Go” - AGI

“Our system beat some people at DOTA” - AGI

“Our system solved a Rubik's Cube” - AGI

“Our system is the best chatbot ever” - AGI

This is a direct consequence of the “work ethic”; AI researchers, by default, often use an intentional vocabulary to narrate the workings of their programs to play checkers, prove theorems, or decipher a Rubik’s cube; outsiders, just as often, are often able to poke holes in what these programs do. This is also why AI researchers have routinely complained that once they build their programs to do some activity, that activity is routinely removed from the list of “intelligent” activities.

That’s why, unlike Jasmine Sun, I am not worried that “Some company will declare that it reached AGI first, maybe an upstart trying to make a splash or raise a round, maybe after acing a slate of benchmarks.” (To be fair, Sun is not that worried either as her next lines demonstrate.³) That’s because the nature of computational work is such that there will always be room for argument about whether the system is really intelligent. These can be, without a doubt, tiresome arguments, but they will keep happening and the horizon of what A.G.I. is will keep changing.

---

All of that said, I am not a fan of the A.G.I. discourse. As I wrote in a previous post, the impact of AI will be decided a lot more by the kinds of AI products we create, rather than some underlying feature of the technology. Think of it this way: you might be a good cook, but that doesn’t mean your restaurant is successful. All AI models have to be part of good products to make any kind of impact on the world.

The "A.G.I." category hinders rather than helps in thinking about technological change and human displacement.

But I also think that critics of A.G.I. are just as much to blame for the fact that the A.G.I. argument is so unproductive and so tiresome as are the A.G.I.-pilled people who think it is just around the corner.

It doesn’t seem fair to me to ask AI researchers to change a practice—their “work ethic”—that forms the basis of their research. But it would be helpful if they could rein in their impulses a bit. As the AI researcher Drew McDermott put it in his marvelously titled “Artificial Intelligence Meets Natural Stupidity” article written in the 1970s, some of the feuds over early AI really could have been avoided if the AI researchers had used more technical names for their systems rather than “wishful mnemonics.”

Many instructive examples of wishful mnemonics by AI researchers come to mind once you see the point. Remember GPS? (Ernest and Newell 1969). By now, “GPS” is a colorless term denoting a particularly stupid program to solve puzzles. But it originally meant “General Problem Solver,” which caused everybody a lot of needless excitement and distraction. It should have been called LFGNS–“Local-Feature-Guided Network Searcher.”

But critics of the A.G.I. concept also seem to me to stretch their argument beyond what its scope should be. Recht, for instance, dismisses much of the current work on computational models by saying

[A.G.I.] adherents ran reinforcement learning on their companies. They threw billions of dollars' worth of spaghetti at the wall until something stuck.

But is that really fair? For much of the AI researcher community operating in the 1960s to the 1990s, the achievements in computer vision and text processing would be miraculous, beyond their wildest expectations.

That said, I prefer arguments that tend to minimize the achievements of our current AI models (even if in an unfair way) to the maximalist arguments in which AI programs companies are the greatest dangers to humanity itself.

For instance, in their much-cited paper, Timnit Gebru and Émile P. Torres argue that those in pursuit of A.G.I. are essentially eugenicists. This argument seems to me to both flatten the goals, aspirations, and practices of all the people who have been engaged in building AI over the last 75 years.⁴

But it is also, perversely, a boost for the concept of A.G.I. After all, if the A.G.I. builders are eugenicists in technologists’ clothing, and therefore the single largest danger to the world, then everything then has to be re-oriented around NOT getting to A.G.I. at all costs. Lee Vinsel has called this mode of critique as “criti-hype” and the criti-hype of A.G.I. is just as bad as the hype around it.

1. This is also mostly Max Read’s interpretation but delivered in somewhat more abstract language.

2. As the historian Hunter Heyck has argued in his stellar biography of AI pioneer and polymathic genius Herbert Simon, the concept of the “program” allowed Simon and many of his colleagues to try to solve one of the biggest problems in the social sciences: how to tie together the sciences of choice and control.

3. She continues:

We’ll all argue on Twitter over whether it counts, and the argument will be fiercer if the model is internal-only and/or not open-weights. Regulators will take a second look. Enterprise software will be sold. All the while, the outside world will look basically the same as the day before.

4. It also seems to flatten what “eugenics” means; it is no longer a movement that agitated successfully for certain government policies around fertility and sterilization; it is literally a set of very diffusely defined beliefs. In Gebru and Torres’ definition, you become a eugenicist by the very act of trying to build an intelligent computational model.