The fol­low­ing is a draft of the intro­duc­tion from the book.

What is Genetic Programming?

I’ve noticed that when you look up “genetic pro­gram­ming” at Google and read the top hits, it often sounds as though the writer imag­ines you already know what he means by the phrase. After twenty years, here’s what I think: Nei­ther you nor they know what they mean by the phrase.

But then I’m not even sure I know.

I use the phrase, of course. “Genetic Pro­gram­ming.” “GP.” And I act as though I know what I mean. It’s what I do.

Let’s try some more research. It seems like maybe you have an Inter­net where you are, and your copy of Wikipedia isn’t bro­ken. Go see what they say about genetic pro­gram­ming there.

Come back when you’re done. I’ll be here.

❧

OK, so as I read it — at least as of this writ­ing, and Wikipedia being what it is — “Genetic Pro­gram­ming” is some kind of computer-sciencey thing that does arti­fi­cial intel­li­gence with genes that con­nect ‘+’ signs and stuff in lit­tle trees. If you read closely, there’s some­thing about com­puter pro­grams that write them­selves auto­mat­i­cally. Plus there’s a lot of dif­fer­ent alter­na­tive approaches to it… what­ever it is. And based on the word­ing and the edit his­tory of the Wikipedia page some ways of doing it are clearly bet­ter than oth­ers… at some­thing… even I don’t quite under­stand what.

Also there’s muta­tion and crossover.

Yeah, that sounds tech­ni­cal enough, right? Can we agree to move ahead with that?

Ah, yes.… I thought not. Let me look around for a bit.

❧

How about this? Here is a very good book I rec­om­mend to all my stu­dents: The Field Guide to Genetic Pro­gram­ming by Ric­cardo Poli, William B. Lang­don and Nic McPhee. It’s avail­able elec­tron­i­cally! You can read it now.

❧

No? Not quite done?

All right, let’s bring out the big guns. How about Sean Luke’s Essen­tials of Meta­heuris­tics. I vouch for it whole­heart­edly: it’s full of inspir­ing machine learn­ing things, all explained sim­ply. And also avail­able elec­tron­i­cally. Read that!

❧

Before we go any far­ther, let me tell you how this is going to end:

The stuff we call “genetic pro­gram­ming” is an inco­her­ent suite of tech­ni­cal habits — design pat­terns, mod­els, idioms — most often used to accel­er­ate human inno­va­tion.

All that; not that

The sen­ti­ment isn’t new. It just doesn’t get repeated often enough.

It’s a cliché when the author of a a tech­ni­cal work starts off by say­ing he’s a “bit of a heretic”, imply­ing that what he’s about to impart will prob­a­bly get the reader in trou­ble if repeated in the wrong com­pany.

For one thing it helps pro­mote a sense that the for­mal dis­ci­pline as “dynamic” and “lively”. You know, with beardy codgers and plucky upstarts con­ven­ing in lux­u­ri­ous Vic­to­rian audi­to­ria to threaten one another with walking-sticks before rac­ing to the Pole to show those fools what a real dinosaur looks like.

Also a nasty back-handed recruit­ing trick, if you ask me. I’ve been to way too many meet­ings, and they would have all been much bet­ter if we’d had walking-sticks, let alone dinosaurs.

The prover­bial “bit of heresy” can also be help­ful when the author is feel­ing self-conscious about play­ing fast and loose with details, or wants to puff up his own author­ity, or might even be fail­ing to give credit to col­leagues who deserve it. I write these words on the anniver­sary of one par­tic­u­larly noto­ri­ous exam­ple of the lat­ter, so don’t think it doesn’t hap­pen: being an “out­sider” sug­gests to the inno­cent reader that you might have thought all this stuff up on your own.

Telegraphed “heresy” can also be ped­a­gog­i­cally use­ful. If only they keep read­ing, the read­ers might be let in on a juicy bit of gos­sip about you know… that whole Leib­niz – New­ton thing, or… have you heard about how Alexan­der the Great really com­pared as a ruler to his dad? Keeps them from falling asleep or skip­ping to the answers in the back of the book.

But then — and you can’t tell me you didn’t see this com­ing: some­times it’s true.

So this is my hereti­cal ver­sion of What Genetic Pro­gram­ming Actu­ally Is:

I have no damned idea.

It’s all over the place. No, seri­ously — you have no notion what a bur­den it can be, try­ing to write one one of these intro­duc­tory overviews.

First we would have to review some his­tory. I’d point out that seven or eight (or a dozen) inde­pen­dent thinkers invented Genetic Pro­gram­ming through the last fifty years. They each called their vari­a­tion some dif­fer­ent thing¹, and the details of imple­men­ta­tion were all dif­fer­ent, and some of the vari­a­tions are little-known while oth­ers are huge stars. None is everything.

Then to be fair I would have to say not only what all those inven­tors did back then, but also sum up all the impor­tant things the ten thou­sand sub­se­quent peo­ple work­ing with GP did in their papers and books and arti­cles and con­fer­ence posters on the sub­ject. Plus there’s all the domain-specific appli­ca­tion work. Plus the com­mer­cial and pro­pri­etary meth­ods, each one vying for authen­tic­ity and authority.

But that’s just a raw fact-dump. So next I’d need to cover the trends and cul­tural norms, themes and motifs, note­wor­thy genealo­gies and regionally-distinct Schools of Thought.

And then I’d need to fix some of your mis­con­cep­tions because “Genetic Pro­gram­ming” may be the most mis­lead­ing tech­ni­cal name in the whole world. I’d point out that it’s not genetic algo­rithms even though it sounds the same. It’s not really any­thing like math­e­mat­i­cal pro­gram­ming or con­straint pro­gram­ming. It’s not philo­soph­i­cally any­thing like bio­log­i­cal evo­lu­tion, even if you squint. It’s not quite the same as machine learn­ing (or it is, depend­ing on who you ask), not least because say­ing so pisses off the Sta­tis­ti­cians (who know bet­ter). It’s not just evolv­ing LISP trees, it’s evolv­ing all kinds of struc­tures and plans and algo­rithms and ideas and art. It’s not just sym­bolic regres­sion. It’s not a lot of things, apparently.

So what is it?

No, really

What­ever it isn’t, I can say that Genetic Pro­gram­ming is the cumu­la­tive work of a huge num­ber of very smart peo­ple. Thou­sands of researchers and prac­ti­tion­ers around the world. They have almost all been pas­sion­ate vision­ar­ies, and have all done amaz­ing things to… well, to achieve what­ever Genetic Pro­gram­ming turns out to be for in their diverse indi­vid­ual cases.

I am reminded that the soci­ol­o­gist Andrew Abbott pub­lished a very inter­est­ing and read­able book in 1988, which has helped me quite a bit to under­stand what GP actu­ally is. Abbott’s book is called The Sys­tem of Pro­fes­sions: An Essay on the Divi­sion of Expert Labor.

What? Why shouldn’t I define it with a soci­ol­ogy book? How is it you have paid so lit­tle atten­tion to the rant thus far?!

Any­way, in Sys­tem of Pro­fes­sions, Abbott describes the dynam­ics of pro­fes­sion­al­iza­tion. That is, how tech­ni­cally astute peo­ple with over­lap­ping tech­ni­cal roles come to self-identify and pro­mote their shared inter­ests by cre­at­ing (and even­tu­ally polic­ing) a pro­fes­sion. In Abbott’s model, pre-professional “fields” arise when­ever diverse peo­ple find them­selves explor­ing and exploit­ing par­tic­u­lar new oppor­tu­ni­ties — espe­cially new tech­ni­cal inventions.

His story of the stages of pro­fes­sion­al­iza­tion includes the devel­op­ment of regional and social com­mu­ni­ties of shared inter­est, then com­mu­ni­ties of prac­tice… then at some point they name them­selves. Then the boundary-setting starts, and the self-definition, and the author­i­ta­tive self-regulated train­ing and cre­den­tial­ing sys­tems, and finally — as a pat­tern, not a rule — we find them build­ing legal infra­struc­ture, rang­ing from Asso­ci­a­tions to Unions to state-licensed reg­u­la­tory bod­ies.²

No, this isn’t a digres­sion. You asked. Well, OK, I asked rhetor­i­cally for you: What is Genetic Programming?

And I answer, not at all rhetor­i­cally: Genetic Pro­gram­ming is a “field” emerg­ing from the inter­ests of diverse peo­ple, who find them­selves explor­ing and exploit­ing a par­tic­u­lar new oppor­tu­nity. It is their shared prac­tices and norms, their habits and their goals.

I could define GP as “the search for for­mal algo­rith­mic struc­tures by using meta­heuris­tics inspired by bio­log­i­cal evo­lu­tion”, but it can­not merely be that. Because (as you’ll learn first-hand) you don’t have to use evolution-like things to search.

I could try to uniquely iden­tify GP as “meta­heuris­tic opti­miza­tion of struc­tures, as opposed to tra­di­tional para­met­ric search or analytically-derived opti­miza­tion algo­rithms”. But (as you’ll learn first-hand) we some­times use those other things too. GP can’t just be evolv­ing pro­grams, because some peo­ple evolve anten­nas and bridges and molecules.

GP can’t just be for data min­ing, because some peo­ple evolve com­pletely abstract proofs. It isn’t about the tools or techniques.

It is, in fact and not just metaphor­i­cally, a com­mu­nity of self-identified peo­ple who share a way of try­ing to solve problems.

Ask­ing what GP “is” at this point in its pro­fes­sional his­tory is like ask­ing what “pro­gram­ming” is: Pro­gram­mers use com­put­ers to solve prob­lems for peo­ple. They don’t do it in any par­tic­u­lar way, except that most of them type on a key­board.

But “typ­ing” is not pro­gram­ming. Just as “evolv­ing code” is not GP.

Look at pro­fes­sional com­put­ing. You can eas­ily see pro­fes­sional bound­aries between the many peo­ple who write pro­grams. There are Soft­ware Engi­neers, and Com­puter Sci­en­tists, Pro­gram­mers and Ana­lysts. And of course there are those who pre­fer the label Soft­ware Devel­op­ers, so they can self-differentiate them­selves as the ones who actu­ally know how to col­lab­o­rate and make pro­grams that peo­ple can actu­ally to use to do stuff.³

I’m quite seri­ous: “Genetic Pro­gram­ming” lives some­where a bit ear­lier in the same pro­fes­sion­al­iza­tion story. As Rick Riolo has said many times: “It’s an art try­ing to become a craft.”

If you ask them, most will say they are doing auto­mated search for abstract struc­tures that solve prob­lems. But the details vary wildly, and every real or the­o­ret­i­cal prob­lem is still a spe­cial case.

So for the time being Genetic Pro­gram­ming is what peo­ple do, who self-identify as “using Genetic Programming.”

Tozier, that isn’t really very helpful

Yeah, trust me: I am totally on your side.⁴

But I have writ­ten this book, and you are read­ing it. Rather than think­ing you and I are both crazy, explain it this way:

If we play our cards right, we can our­selves define Genetic Programming.

I don’t mean to imply “GP is what you think it is”. I mean the field is so young and mal­leable, that you can learn to do amaz­ing things with­out ever being told you’re doing it wrong.

In these last twenty years I’ve seen for­tunes made, dis­ease treat­ments invented, patentable inven­tions piled thou­sand deep, philo­soph­i­cal and the­o­ret­i­cal prob­lems set­tled, space probes launched, robots that learn to walk in their dreams.…

Peo­ple can use GP to cre­ate things they could oth­er­wise only imag­ine. Here’s my lit­tle True Heresy, stated another way: Those peo­ple are not using GP to “auto­mat­i­cally invent” things. It isn’t a magic inven­tion machine.

It’s an accelerator.

I’ve hung out with a num­ber of these folks, through the years. They’re not smug geniuses… as a rule. Rather, they walk around in a sort of daze, telling one another how sur­prised they were by what they were shown when they started using GP.

A human being invents when she uses GP to con­sider a mil­lion out­ra­geous struc­tures and lay­outs no sane design engi­neer could incre­men­tally develop. The “inven­tion” hap­pens when she — a standard-issue human being—notices that some of those mil­lion designs is interesting.

That’s the same thing a tra­di­tional design engi­neer does, but faster. The effort is in a dif­fer­ent place.

A human being explains some­thing the world when he uses GP to con­sider a thou­sand novel mod­els of data, in less time than a tra­di­tional sta­tis­ti­cian can eval­u­ate two. The “expla­na­tion” hap­pens when he — a standard-issue human being—notices that some of the best mod­els invoke rela­tion­ships between vari­ables that nobody else had never mentioned.

That’s the same thing a tra­di­tional sta­tis­ti­cian work­ing with a domain expert does to explain the world, but faster. The effort is in a dif­fer­ent place.

And so on: an artist explores a thou­sand com­po­si­tions; a bio­med­ical researcher exam­ines a dozen or a hun­dred genomes and a mil­lion gene expres­sion pro­files; a trader mon­i­tors a mil­lion port­fo­lio man­age­ment rules.

The same thing they would nor­mally do. But more. The effort is in a dif­fer­ent place: on the think­ing.

Genetic Pro­gram­ming doesn’t auto­mate think­ing or cre­ativ­ity or any of those things. It helps peo­ple notice things.

GP is a prosthesis

Think about writ­ing — you know, with a pen, on paper. Writ­ing isn’t “auto­mated mem­ory”. Or think about pro­gram­ming com­put­ers. It isn’t “auto­mated arithmetic”.

Writ­ing and pro­gram­ming extend your mind. Writ­ing is a pros­the­sis in the sense that it offloads mem­o­ries to a long-term exter­nal stor­age medium. Pro­gram­ming is a pros­the­sis in the sense that it cal­cu­lates stuff really really fast.

But nei­ther one is “auto­matic”. Harry Pot­ter notwith­stand­ing, there are no self-writing pens, and no self-programming computers.

See what I did right there? There are no self-programming com­put­ers. That includes Genetic Pro­gram­ming, regard­less of what you may have heard from the nerds down the street.

I can’t tell you how many peo­ple I’ve seen come to GP, hav­ing read the hype about auto­mated inven­tion and stuff. Like a per­son who wants to write bet­ter, so she gets a really pow­er­ful pen. The per­son who wants to learn to pro­gram games, so he gets a really pow­er­ful com­puter.

How do you learn to write? How do you learn to pro­gram? Same with GP. Through guided prac­tice. We think a bit, we try some­thing, we learn if we’re lucky, and maybe we solve some problems.

And if we’re very good problem-solvers, we can use GP to help our­selves become use­fully surprised.