The Barbell scissors

In this issue:

• Just the right program?
• Sex finally understood!
• The Barbell scissors.

Just the right program?

In looking at the basis for modern machine learning and AI, I note how closely the needed mathematical background matches the program of the entrance exams at French engineering schools (Centrale, Mines, Polytechnique, Normale Sup' and others).

Not only in math but in applications of math through physics. This observation reminds me of how pleasantly surprised I was, when coming as a student to Stanford CS from Polytechnique (together with my friend Claude Baudoin, who will confirm or contradict), to see how much we already knew of the mathematical basis required by many courses.

It is not just that so many of the techniques originated with the people whose names we had read in the tall engraved panel dominating the main auditorium of our school (from Monge to Cauchy, Laplace, Legendre, Poisson, Hermite, Liouville, Fourier, Lagrange, Ampère, Poincaré, Hadamard,  etc., down to the very person teaching us, Laurent Schwartz, who was explaining how to use the Dirac delta without feeling ashamed when watching ourselves in the mirror the next morning). It is also that the mix was the exact one I need now. I blame myself bitterly for not working hard enough back then. So many opportunities lost. I have had to re-learn some of the stuff, sometimes missing the concision and elegance of our original course notes.

Most of the knowledge in question is taught not at the schools themselves (which admit students only at the equivalent of an upper graduate division in the international system) but before, in the special “preparatory classes” which train students for their competitive — and highly selective — entrance examinations.

The focus in on mathematics and (with a ratio that depends on the school) classical physics, leading to learning very much the kinds of techniques that today's AI needs, from linear algebra to analysis and optimization. Plus a few exotic topics; I always liked topology, for example, and was not taken aback when discovering denotational semantics. There is also a dose of humanities, mostly writing and foreign languages.

It cannot be an accident that when one looks at the CVs of the AI movers and shakers in Silicon Valley and at Google, Open AI etc. (not just Mistral!) one finds so many graduates of these schools. Centrale alumni seem to be particularly visible these days.

One of the characteristics of the teaching, in the schools themselves but even more in the harsh regime of the preparatory classes, is its focus on drilling. You do exercise after exercise. This practice has an immediately utilitarian purpose: the more exercises you do the higher the chance that at the competitive exam you will get a question through which you have already gone. But it has the very strong practical consequence that concepts are not just taught abstractly but repeatedly practiced in their application.

Another way in which the system differs from programs in (for example) the US is that in the preparatory classes it leaves little room for individual choice of topics. Once you get into the engineering schools themselves the picture changes; it becomes more à la carte. Until then, however, since everyone in a given program must pass the same competitive exam, everyone more or less takes the same courses. Very different from the individual cocktail that students — at least starting in the second year — define for themselves in many institutions of other countries. While one may regret the blow to individual choice, the system has the advantage that a school admitting students can pretty much assume what they know (and don't). To use a term that I have never heard mentioned in France, it very much  embodies the Shu Ha Ri principle: first you do what you are told, no questions permitted (I am exaggerating a little); once you pass muster, you earn the right to ask; finally you earn the right to talk back. Not at all in line with fashionable pedagogical theories of the day, but not so bad either if your aim is to build the TGV and Ariane, or for that matter Mistral and AMI Labs.

The system has for decades been the recipient of regular and harsh criticism. One of its problems comes from the very fact of its difference: it is difficult to explain to the rest of the world, particularly in the European “Bologna” one-size-fits-all stranglehold. That part is not too bad; the schools have found way to position themselves in a reasonably understandable way. The more stinging criticism focuses on the elitist nature of the system and its reliance on well-honed uniform drilling, where the only goal is to get the best possible rank in the competitive examination. Still, it has worked pretty well for a couple hundred years (it date backs to the Revolution and Napoleon).

Too well perhaps. I do not think it is part of its explicit design goals to train solid engineers (for free or, in the case of the most elite school, actually paying all students a small salary), including  an ever-growing share of recent immigrants who just hop on the French springboard on the way, all for the greater advancement of San Francisco. If that is a consolation, at least it benefits someone.

I finally understood sex

(<Warning> I posted a variant of the following comment on a private group where someone took it as an attempt at deep philosophy. IT IS NOT. It is a nerd's feeble attempt at something that not all nerds have heard about, called “light humor”. Consider that it is prepended with a smiley.</Warning>)

Neural networks use compositions of individual units, each a linear transformation composed with a “nonlinearity” such as ReLU or sigmoid. We need the nonlinearities because composing just linear transformations, however many of them, can only yield one humongous linear transformation, which cannot discriminate between complex things.

In the history of 20-th century mathematics there is a fascinating figure, Stanislaw Ulam, a close friend of von Neumann and the inventor of many concepts as well as a key contributor to the Los Alamos war effort. He wrote a highly entertaining and instructive memoir, Adventures of a Mathematician. I see that it was made into a film, which I definitely will watch, but in the meantime I found a YouTube video of Ulam himself: a talk he gave in 1980 about von Neumann and other topics. In spite of the video's bad technical quality it is worth watching. Starting around 55:10, talking about von Neumann's and Wiener's self-reproducing automata, Ulam says:

What remains to be done is not so much, it seems to me, the study of developing reproduction itself, but differentiation, that is to say evolution, which depends on sex, or non-linear phenomena, because that's what it amounts to.

At last, sex explained! In the boring linearity of everyday life, sex is the little sigmoid, tanh, StopMax and such (you can add “Heaviside” but that is your responsibility). Do we still need after that to read about Anna Karenina, Madame Bovary and Lady Chatterley?

The risk of the Barbell scissors

Many people are concerned today with the future of the job market for software developers. I will come back at some point with elements of an analysis, but on a more fleeting note it may be useful to mention what seems to be a general tendency at the moment.

Using an analogy that I first heard in a talk by A. Maza, we can call it the Barbell scissors. In finance, “Barbell” denotes an investment strategy which chooses both very safe and very risky holdings, ignoring those in-between. This newsletter is about software engineering, not investment, so the term just serves as a metaphor for what some IT managers having embraced vibe-coding as a process appear to be practicing systematically: hire either greenhorns, fresh out of school, or very senior developers. Retain no one in-between.

While the rationale is clear it raises many questions, into which I will not go now. I think, however, that it is useful to draw attention to this component of today's reality, which may or may not persist, but definitely presents a serious career risk for the many competent professionals who risk finding themselves stuck between the blades of the Barbell scissors.

Cover photo: A rose