Red Plenty by Francis Spufford

When I first visited Moscow, in what now seems a far-distant era, a giant red neon sign beamed Lenin's famous phrase "Socialism plus electrification equals communism" across the Moskva river. The slogan encapsulated several central aspects of communist thought. First, optimism for the future. Second, that science and technology were both by definition progressive forces but were impeded by capitalism; only under communism could they enable the building of a society of abundance for the many, not just the few.

The roots of this respect for science lie deep in Marxist thought, not least because Marxism was seen as itself a science. When Friedrich Engels's fragmentary text, The Dialectics of Nature, dating from 1883, was rediscovered in the 1930s, it was regarded as demonstrating that nature itself conformed to the philosophical principles that he and Marx had formulated in their decades of collaboration. Then in 1948 the American mathematician Norbert Wiener published his path-breaking book (and invented the word itself) Cybernetics, a way of thinking about how self-regulating systems interact dynamically with their environment, both changing it and being changed by it. Cybernetics ("circular causation") was a way of understanding how systems could show apparent goal-directed behaviour without consciousness. Soviet philosophers seized on the concept, elevating it to the status of a "fourth law of dialectics".

Following Stalin's death and the slow thaw, initiated by Khrushchev, that lasted from the mid-1950s to the mid-60s, Soviet planners, economists, physicists and mathematicians flourished. They persuaded the Soviet leadership that, using cybernetic principles and the newly developed computers, the centralised, planned Soviet economy could at last be made efficient. By 1980, Khrushchev claimed, the Soviet Union would overtake America; communism would have defeated capitalism. For a while, in the aftermath of Sputnik and Gagarin's space flight, it looked as if he might be just be right.

So what went wrong? It is this question that Francis Spufford explores in Red Plenty. Not as history, nor yet exactly as a novel, but in a series of loosely linked chapters, each a vignette in which fictional characters rub shoulders with real ones. The cast list at the front distinguishes real from invented, although two of the latter, as Spufford makes clear, are "stand-ins" for the real economist Abel Aganbegyan and the molecular geneticist Raissa Berg (the latter still alive at last count and now resident in the US). The key real figures are the mathematical prodigy and later Nobelist Leonid Kantorovich, whose initial calculations on how to improve plywood production blossomed into a comprehensive plan for the economy, and Sergei Lebedev, designer of the first generations of Soviet computers.

Spufford has long had a somewhat eclectic interest in the interactions of science, technology and society, as evidenced by Backroom Boys, his tales of post-1945 British "boffins", and he has certainly done his homework here. It isn't every work of historical faction that is backed up by 70 pages of footnotes, references and sources. He speaks no Russian and has built heavily on the work of English-language historians of the Soviet Union such as Sheila Fitzpatrick, but his learning sits lightly. Admittedly, his fictional characters are two-dimensional types without inner life, appearing often only for a single chapter. They are chosen to illustrate daily details of Soviet life, from the primitive conditions of collective farms and the career moves of young, upwardly mobile party apparatchiks to the semi-criminal underworld of the fixers who helped to circumvent the idiosyncratic inefficiencies of central planning, and the brutally authoritarian medicalised childbirth of 1960s Moscow.

It is through their voices that the impossibilities of Khrushchev's dreams are revealed. The planners were, above all, rationalists, committed to a cognitive view of the world that allows no space for human frailties, greed, corruption, or mere inadequacy. The fictional, but all too realistic, Maksim Maksimovich Mokhov is deputy director for chemical and rubber goods at Gosplan, the USSR's central planning agency. He is faced with a problem at a viscose plant. It shouldn't have happened; the plant is fairly new and the process simple – new machinery, simple inputs of chemicals and woodpulp, and "Trees into sweaters! Brute matter uplifted to serve human purposes". But one machine is unaccountably wrecked. The plant managers request a new machine – but that requires revising the targets for the factory that produces the machine. And that requires replanning the inputs into the machine tool factory. Everything connects in a nightmare combinatorial explosion. Down the line, only the semi-criminal fixer, so characteristic of Soviet life, can sort out the problem.

Human error and human corruption are inevitable features of being human and not a machine, and need to be built into planning assumptions. What cybernetics should have taught the mathematicians and planners, tucked away in the relative comfort of the Siberian science city of Akademgorodok, and the Moscow offices of Gosplan, is that systems work best when self-organised from below, not centrally planned from above in a command economy. Although this is now widely recognised, it has been an expensive learning process in terms of both political and individual human tragedy.

Self-organisation is a fundamental feature of living systems, and was indeed well understood by Soviet biologists of the 1930s (and some in the west), before Stalin's dogmatic destruction of scientific creativity – and of the creators themselves. Spufford is weaker on his biology than on technology. He also pays far too little attention to what fatally weakened the Soviet economy – the escalating arms race with the US. It wasn't just that there was no trickle-down from military innovation into the desperately inadequate production of consumer goods. Even civil science was denied the computers, centrifuges and relative freedom from rigid planning constraints that so privileged the military.

But could it have worked? Can we ever once more believe that we, the people, could create a just, equal and abundant society? Red Plenty ends with the question that must carry all our hopes and fears, as Khrushchev, the deposed pensioner, broods: "Years pass. The Soviet Union falls. The dance of commodities resumes. And the wind in the trees of Akademgorodok says: can it be otherwise? Can it be, can it be, can it ever be otherwise?"

Steven Rose's The 21st Century Brain is published by Vintage.