The fol­low­ing is an extract from the immi­nent Prag­matic GP book. I’m post­ing it because it’s the first place I’ve man­aged to tie together some­thing that’s been bug­ging me for years, about the fail­ures so many Very Smart pro­gram­mers and com­puter sci­en­tists have expe­ri­enced, over and over in some cases, when try­ing to build use­ful, inter­est­ing, robust GP sys­tems that solve com­plex prob­lems. I sus­pected that part of the prob­lem was in their pro­gram­ming style, but I real­ize here that col­lab­o­ra­tion habits are really the key to suc­cess in GP. More on that later; now I’m going to go pub­lish the rest of this project.

This Cargo-bot project will be the first of a half-dozen we will work through together, so it seems impor­tant to say a bit about what I expect.

As I said, my inten­tion for this effort is that you learn by doing, that is by build­ing your own GP sys­tems, bottom-up, more or less from scratch. But it’s the nature of com­puter pro­gram­ming that it’s sur­pris­ingly dif­fi­cult to com­mu­ni­cate, except by exam­ple. There are some abstruse math­e­mat­i­cal ways, and some hand-wavey (but over-constrained) indi­rec­tions like that awful pseudocode crap, but really in the end I think the only way for you to purely learn by doing would be to write code with me — in person.

More­over, there are a lot of inter­est­ing and impor­tant sub­jec­tive aspects to suc­cess­fully build­ing a GP project, that don’t come across in code as such. Why is that there? What does this do? How did you make that jump? Indeed, the lack of those sub­jec­tive aspects in the other writ­ing on GP are what’s sparked me sit­ting here typ­ing today: they’re cru­cial; they’re the Lan­guage of the Project.

So at least for this first project, I’ve decided to write the code myself. There is still no oblig­a­tion that you use the same pro­gram­ming lan­guage (Ruby) that I will — in fact, I hope you won’t. I hope you’ll re-write the code in some other lan­guage I’ve never heard of, and post it on GitHub, and share it with the other folks out there as more exam­ples of how this can be done. But what­ever you do, it had bet­ter still have all the same accep­tance tests I’ll be describ­ing myself, and demon­strate (auto­mat­i­cally!) that all those fea­tures are present in your code, and the tests pass.

Beyond the fact that it will sur­face some of the impor­tant sub­jec­tive aspects of the project that are nor­mally elided in descrip­tions of GP, there’s a sec­ond (and more impor­tant) rea­son I’m going through this process of writ­ing the code in plain sight.

Peo­ple don’t like being sur­prised. Remem­ber what I’ve said a few times already: GP (when it works) accel­er­ates the process of dis­cov­ery, which is to say you being sur­prised by some unex­pected result or pat­tern.

I’ve found that many well-trained folks, young and old, who have learned and inar­guably mas­tered com­puter sci­ence and pro­gram­ming to the point where they can squint real hard at a prob­lem and smash-key their way into a text edi­tor and in a few hours pro­duce some exotic self-contained crys­talline beauty of an algo­rithm… well, those peo­ple often react poorly to being sur­prised. They are so used to being right, to intu­it­ing what they’re going to do, and then doing just that; they ham­mer out the code they envi­sion (though maybe hit­ting the back­space key a lot) with­out ever hav­ing to speak to another per­son or do much more than rtfm now and then.

In my expe­ri­ence, those peo­ple have the most trou­ble build­ing and using a GP sys­tem. This is noth­ing to do with their cod­ing abil­i­ties, or their under­stand­ing of sys­tem design or syn­tax or parsers or any of that com­put­ery stuff. It has a lot to do with them see­ing the thing they build — the GP soft­ware process — make the same sort of blue-sky, intu­itive, often sur­pris­ing design deci­sions that they make. Any well-made GP sys­tem will quickly spit out “design deci­sions” that may or may not make sense (it’s hard to tell some­times), but which in any case are noth­ing like what the pro­gram­mer expected. And in my expe­ri­ence Very Smart pro­gram­mer folks are not often equipped to deal with that.

This is a prob­lem of com­mu­ni­ca­tion, not code-writing. I think it may be asso­ci­ated with a bun­dle of habits Very Smart pro­gram­mers develop in our cul­ture, which tend to keep them from hav­ing to dis­cuss what and why they are writ­ing all the indi­vid­ual lit­tle incre­men­tal bits of code they write. They’re often so smart, they need not have any help visu­al­iz­ing an entire start-to-finish solu­tion in their heads, or do any explicit intro­spec­tion, or spell out why they have made one deci­sion ver­sus another. But if any­body with some dif­fer­ent vision of an equally effec­tive start-to-finish solu­tion starts chang­ing their code­base, or changes the rules or goals in mid-project, or argues with them about their basic toolkit… well, it doesn’t go well. More often than not they freak out, as a mat­ter of my expe­ri­ence. Some­times there is slamming.

This is not a crit­i­cism, but rather a diag­no­sis: Very Smart pro­gram­mers are, in my expe­ri­ence, eas­ily thrown off track by hav­ing unex­pected changes made to the project they’re work­ing on. Or by cop­ing with unex­pected or emer­gent behav­ior that arises dur­ing a project. Or by hav­ing their fun­da­men­tal assump­tions about a domain ques­tioned. Or by hav­ing to inte­grate or adapt to an alter­na­tive design approach.

Any well-built GP sys­tem, work­ing on any inter­est­ing project, will most def­i­nitely do all those things.

I am not a Very Smart pro­gram­mer. I’ve had to stum­ble my way through all the cod­ing I’ve ever done, and I have very poor visualizing-everything-start-to-finish-in-my-head skills. So in the last decade or so, pro­gram­ming in my not-Very-Smart way, I’ve been obliged to speak with other peo­ple a lot, and explain what it is I’m doing in each iter­a­tive deci­sion I make, and ask them what they meant by what they were doing, and all those other things Very Smart pro­gram­mers are able to do auto­mat­i­cally with­out con­sult­ing other people.

Along the way, I’ve dis­cov­ered that there is a set of tools for that. Com­mu­ni­ca­tion and col­lab­o­ra­tion tools. And I’ve found that those same tools, or at least very sim­i­lar ones, help to deal with this weird “col­lab­o­ra­tion” thing one needs to do with a GP sys­tem. It’s not col­lab­o­ra­tion, and it’s not com­mu­ni­ca­tion; but it is some­thing a lot like the mutual men­tal mod­el­ing we do when we speak to one another (which, you may be amused to dis­cover, is some­times referred to as Prag­mat­ics).

So as I go along, in my not-Very-Smart way, I am going to be build­ing in some of those safe­guards I’ve learned, and using tools that were orig­i­nally designed to fos­ter col­lab­o­ra­tion between human beings work­ing together on code. You, since you may be Very Smart, might not have encoun­tered these before; in my expe­ri­ence many pro­gram­mers who work in the Uni­ver­sity or other uncol­lab­o­ra­tive set­tings may have missed them. Worse, you may have encoun­tered them (for exam­ple as prac­tices that form part of the method­ol­ogy referred to as Agile by some folks), and dis­missed them as being unim­por­tant to you in your work-life.

They form a cru­cial buffer­ing inter­face, one that will help you avoid being sur­prised into fail­ure. Do not ignore them.