The Soul of A New Machine

There are many, many project management books that purport to reveal the ultimate system for surmounting the myriad challenges to releasing a product on-time, in-budget, and with all the promised features. It's a popular and useful genre, even if much of the material is just reshuffled and rebranded old bromides, but sometimes the most helpful and memorable way to offer project management advice is to just pick a single case study and dive in deep to explore the group dynamics that result - or don't - in a successful product. I'd previously read Mountains Beyond Mountains, Kidder's excellent profile of Dr. Paul Farmer, and this much earlier work, which won him a Pulitzer, is just as detailed, thoughtful, and revealing. It's about the show more race from 1978 to 1980 by one team of computer engineers at Data General to develop and release a 32-bit "minicomputer" (one of the many charmingly antiquated terms that will give those who know their industry history a smile) called the Eclipse in competition with another, more-prestigious team that's been given a more glamorous project in a shiny new office, with the fate of the company looming in the background. Heroes and villains are the keys to great drama, and so as the narrative follows the protagonists, who are working on "Eagle", a 32-bit extension of the existing 16-bit line of computer hardware instead of the brand-new computer of their dreams that they imagine their counterparts are gleefully assembling, their struggles to design, build, test, debug, and actually finish a computer without more hacks, kludges, and shortcuts than are absolutely unavoidable in such a short time take on a mythic glow that anyone working on a big project in the tech industry under a tight deadline will immediately recognize, despite the passage of nearly 40 years.

If I had to pick a single part of the book that best-represents why the book would make a worthy addition to a computer engineering syllabus, it would be the chapter "The Case of the Missing NAND Gate". It's an almost self-contained episode towards the end of the book, where, late in the development cycle, several engineers are attempting to debug an erratic logic failure, which occurs just often enough to be indicative of a real problem but not so often as to be easily reproducible. Kidder relays the team's efforts to determine if this diagnostic failure is at root a software or a hardware issue, with an amusing layer of "antagonistic camaraderie" on top of their troubleshooting, as all of them had a hand in designing the machine and each wants to solve the problem but none wants to have the root cause bear their fingerprints. This was back in the era when computer design involved the frequent use of oscilloscopes and it was often a genuine question if chips on a board weren't properly spaced for optimal signal timing, so fans of vintage computing will really enjoy as Kidder walks the reader through the finer points of system caches, assembly microcode, page faults, and logic gates while various engineers, working in shifts, propose and reject theories to explain the anomaly. It's a genuine puzzle, and Kidder does a great job explaining just what the problem is and why it's so difficult to diagnose and eventually solve, translating the arcane technical details of the fault with the various components of the system architecture until it's not just lucid but even enthralling. Here's his rendition of one potential explanation from one engineer named Guyer:

"The diagnostic program originally puts the target instruction at address 21765, and then, sometime later on, it moves the target instruction to 21766. But the IP never gets word of the change, though the System Cache does. Now, sometime after the target instruction is switched from mailbox 21765 to 21766, the program directs Gollum to execute the instruction at 21766. The IP receives this command and looks through its cache. It says to itself, in effect, 'Mailbox 21766? I've got that address and there's an instruction in it. Let's run it.' But in the I-cache, the target instruction is still at 21765, and mailbox number 21766 contains an error message. In short, the I-cache contains an outdated piece of memory. Why didn't it get updated along with the other parts of the memory system? Maybe, Guyer writes, the System Cache is to blame. The System Cache is supposed to know exactly what is in the I-cache. If an instruction or data gets moved to a new address, the System Cache is supposed to tell the IP to throw away the outdated mailbox and get the new one, the one with the target instruction in it. Somewhere back in the program, Guyer figures, the System Cache lost track of what was in the I-cache. It forgot that the IP had the target instruction in mailbox 21765, and so, when the change was made in the location of the target instruction, it never told the IP to get rid of the old, outdated mailbox. Guyer likes this hypothesis. He records it with mounting enthusiasm; and describing it later, he repossesses the feeling, speaking rapidly, gesturing with both hands. Then he stops, puts his hands on the table, and says, 'Of course, it was completely wrong.'"

The book is also notable for broader reasons. Massachusetts was a much larger center of the technology industry in the 1970s and 80s than it is today, and the "Route 128" cluster competed directly with Silicon Valley for talent and prestige. However, the Eclipse team's main antagonists were not in California but in North Carolina, giving the modern reader a glimpse of the "flight to the Sunbelt" in embryo that has helped the Research Triangle, among other places, at Massachusetts' relative expense. Data General was founded by former employees of Digital Equipment Corporation; I've read articles arguing that Massachusetts' relatively strict enforcement of noncompete agreements was a major force that drove tech firms to less strict jurisdictions, but that doesn't seem to have been as large an issue here as the typical lure of lower taxes. However, prospective MBAs should scrutinize closely the decision by corporate management to have two different teams working on overlapping products, as ultimately the highly-regarded North Carolina team working on the prestigious brand-new 32-bit machine (dubbed "the Fountainhead Project", with hilarious irony) was upstaged by the "Eagle" team, whose less-ambitious 32-bit extension of the 16-bit Eclipse became a huge moneymaker for the company. Now, hindsight is 20/20, and it's obviously impossible to consistently tell ex ante if internal competition, which is often positive, will in the end have wasted resources. After all, the Eagle team did produce an extremely successful product, although we don't know how much was spent on the Fountainhead team. But lack of clear focus is always risky, and corporate politics can have damaging downstream effects on teams of even very smart people.

But any look into the subtleties of nerd psychology has to account for the fact that the drive to create cool technology is often far more powerful than any corporate folly, even and perhaps especially if that involves extremely long hours of hard work. Occasionally the concept of "mushroom management" is invoked, which turns out to mean "put 'em in the dark, feed 'em shit, and watch 'em grow", and one paradoxical upside of not being the top brass' favorite project is that, with protective leadership, that can actually mean more opportunity to produce. There's an interesting detail in the life story of Tom West, the top manager for the Eagle project: "He went to Amherst College, in western Massachusetts, where he studied the natural sciences. He did so without academic distinction, and it happened that Amherst was just then embracing a new Calvinist fad called the underachiever program: young men whose brains seemed much better than their grades were expelled for a year, so that they might improve their characters. At Amherst, certainly, and possibly in the entire nation, West became the first officially branded underachiever. It was something he'd always remember." This story takes place after the end of the naive cyberhippie movement of the Whole Earth Catalog/"All watched over by machines of loving grace" era, so that technoromance had been firmly replaced by a more modern engineering sensibility, but there's still poignancy of the ceremony at the end of the project, where the team members come to grips with how much of themselves they've put into what would be released as the Data General Eclipse MV/8000, elevates what could have been just an unusually lengthy product diary into an account of creation that justly deserved its Pulitzer. One of the engineers had a typical complaint:

"What a way to design a computer! 'There's no grand design,' thinks Rosen. 'People are just reaching out in the dark, touching hands.' Rosen is having some problems with his own piece of the design. He knows he can solve them, if he's just given the time. But the managers keep saying, 'There's no time.' Okay, Sure. It's a rush job. But this is ridiculous. No one seems to be in control; nothing's ever explained. Foul up, however, and the managers come at you from all sides. 'The whole management structure,' said Rosen. 'Anyone in Harvard Business School would have barfed.'"

Maybe, but the reason why his project shipped and his rival's didn't wasn't because he had superior consultants from Harvard. As Kidder recounts from attending a trade conference: "It seemed to me that computers have been used in ways that are salutary, in ways that are dangerous, banal and cruel, and in ways that seem harmless if a little silly. But what fun making them can be!" show less

This was interesting for its own merits, a snapshot back into a specific time (and a specific project) of the computer industry.

But I found it *really* interesting for the description of personalities, work-life balance issues, issues of burn-out, feelings (and description) of people becoming 'battle-hardened' veterans, constant in-fighting... it captures beautifully many of the issues that (in large part, though not solely) lead me to leave (aka, run screaming away from) the software/tech industry 30 years after the events described here. It was in fact shocking to read those descriptions and to realize just how 'cultural' the tech culture is (for better or for worse.)

Hold on to your hats, kids! We're taking a trip back to the late 70s, where there were more than 2 or 3 types of computer to choose between, but they cost half a million dollars and were the size of refrigerators. This book relates the development of a new computer at Data General, a highly successful manufacturer of the time, though forgotten today.

This is really one of those plus ça change, plus c'est la même chose things. While it is so much of its era - maybe the bronze age of the computer industry - so many things have barely changed. Seeing which things are identical and which are unrecognizable is one of the fascinating things about this extremely interesting book.

Having worked in the broad field of computing for 15 years, the show more processes of developing and delivering a technical project were at the same time familiar and alien to me. The speccing, creating, iterating, integrating, debugging are described very convincingly and were very recognizable. But the way those things were achieved - the equipment used, the amount of documentation, the way issues were tracked - was all very different to practices today (and for good reasons at the time). I would have enjoyed a bit more technical detail, but then I imagine that I'm a bit more technically proficient than the target audience.

The characters of the engineers, their goals, and the counter-intuitively dysfunctional ways of getting the most out of them is remarkably similar to what one might encounter in the field of computing today (to be honest, I'm only familiar with development on the software side, but I assume the same broadly hold true on the hardware side). This project tended to use much younger engineers than standard in the industry, and motivated them by giving them a high level of responsibility for their particular areas - which meant they felt that they had to work ridiculous hours. This was good for those involved, but also meant that the creation of the machine was much cheaper than it would have been with more experienced people working regular hours (that said, it also came with a higher risk of failure).

Additionally, the office politics of a large tech company are well depicted, and should be instantly recognisable to anyone who's ever worked in such an organisation.

The story of the creation of the machine has some strong parallels with Revolution in the Valley - about the develpoment of the first Mac (remarkably only about 5 years later). In both, the computer was thought of (at most) secondary importance within the companies building them. And both had a team of highly-motivated, young engineers driving them forward, with ridiculous workloads, and thriving on their "outsider" status. Data General's computer was not revolutionary like Apple's, but that is not really what this book is about. It is a fascinating insight into the inner workings of the computer industry - a field which affects all our lives and yet is somewhat opaque. It is remarkable both as a historical account, and also for how relevant it is today. show less

I'm sure there are some people out there who find his writing too soft-hearted or lacking in technical detail, but for my money, Tracy Kidder just might be the best investigative journalist out there. In "The Soul of a New Machine" he traces the development of a computer -- code-named "Eagle" -- developed by the Data General company of Westborough, Massachusetts. As other reviewers have mentioned, Kidder's subject is not the machine itself but the people behind it -- driven, gifted computer designers working in what was then a fairly new and rapidly growing field. Kidder's ability to describe his subjects is unparallelled -- his sketches of the engineers he observed are both memorable and perceptive, and he's got a wonderful facility show more for divining what motivates each member of Data General's team. His writing's most remarkable quality, however, is the feeling of intimacy it transmits to his readers. The tech-heads Kidder covered poured every ounce of their energy into the Eagle, and "The Soul of a New Machine" does more than effectively record their hopes, ambitions, frustrations, and fears -- it also makes them seem comprehensible and emotionally accessible. One gets the feeling that the author didn't just report on his subjects' project, he suffered through it with them. This immediacy makes a real difference: it allows Kidder to take what might have been a dry, technical subject and wrings real drama from it. "The Soul of a New Machine" is so emotionally involving it reads something like a cliff-hanger. During the book's last few chapters, I earnestly hoped that those engineers would get their computer out the door and see it find a place in the wider world.

Some other reviewers have also described the book as a bit "dated," and well, yes, it is. After all, many of Kidder's subjects used pencil and paper to work out their computing designs. Still, it's an interesting and valuable historical document from an era when computers were a niche market restricted mostly to specialists rather than an everyday presence in the lives of billions of people. Kidder's patient and deliberate, taking readers of the late seventies, who were probably much less tech-savvy than today's average book-buyer, through the very basics of what was then modern computing. This is the real foundation of the computer age -- ones and zeros, basic chip design, and machine code -- and even in a world where applications seem to get much more attention than hardware, it was useful to read up on this stuff. Of course, now that we can fit the computing power of dozens of Eagles in our front pockets, it's hard not to think of Data General's engineers as cybertronic cavemen, but reading the relatively primitive methods they used to design a state-of-the-art computer in just a few short months makes their achievement seem all the more impressive. Recommended to computer enthusiasts and non-ethusiasts alike. show less

40 years later, this book is still the definitive narrative of what life at a tech company is life. From the focused vision of engineers, to the political battles of upper management, to the relationship with marketing, to the creepy guys the sole woman on the team had to put up with ... it's all still the same. It's been on my to-read list for a while, wish I'd gotten to it sooner.

It's also fascinating to look up the career trajectories of the engineers involved (many of them are still working, usually as VPs/CEOs).

An underrated part of this book is just how good the technical writing is. It goes from "all computer data is 1s and 0s represented by different voltages across a circuit" to accurately describing a thorny bugs with show more registers, gate logic, the team is struggling with quickly and clearly. If the book was just this stuff, it would be a towering achievement of writing.

As an aside, if you've watched Halt and Catch Fire you'll be shocked at how much season 1 is lifted from this book: The overall plot of a rogue executive starting a computer project; the strategy to hire recent grads who won't know how daunting the tasks they are given are; the team getting obsessed with the game Adventure (the episode where Bos gets stuck in the maze is directly lifted from the book); at one point, Cameron even bemoans the machine's soul. I hope Tracy Kidder got royalties. show less

About 6 years ago, a sort of scandal rocked the gaming industry related to a blog post by a woman known as "EASpouse". The blog post criticized EA's labor practices at the time, which required employees to work massive amounts of unpaid overtime, as they were salaried employees. By massive, I mean about 12-16 hour days, 6 days a week, regularly. This was a big deal among gamers, because very few of us had ever had the opportunity to peek behind the curtain like this. It was likely that most of us viewed game development with a variation of the way that Roald Dahl as a child imagined the inside of the Cadbury Chocolate Factory near the boarding school he attended (which later led to Charlie & the Chocolate Factory).

The Soul of a New show more Machine, by Tracy Kidder shows that such working conditions are nothing new. The book follows the development process of Data General's micro-computer (sort of like a rack mounted server, except it's the size of the whole unit, but essentially only being one of the server nodes), that would be a successor to their Eclipse line of microcomputers, code named the Eagle, and later released as the MV/8000. The book goes into both the personal and technical aspects of the development process, profiling the various men (and a few women) involved in the project, and giving a description of the technical aspects of the process for the layman.

While the technical bits (pardon the pun), are enjoyable, the book's strength, and where it spends most of its time, is in profiles of the people. The book paints a bleak picture of the inner workings of Data General. The working conditions at Data General, particularly on this project, are brutal. Much as with EA Spouse, employees are salaried, with no overtime pay, and work 12-16 hour days, 6 days a week. As the project goes on, project leads and younger employees are worn down. Often, employees at Data General observe that the company brings in a lot of new fresh recruits, and few stay at the company after they turn 30. Many of these new recruits drop out for various reasons, and often employees discuss the company's sweat-shop like working conditions. As the project moves into the heat of summer, the air conditioning breaks, turning their windowless basement office into a sweltering oven, which they can't even leave the door open for, for security reasons. Only after the employees strike do they fix the air conditioning.

By the end of the book, several of the project leads, themselves burned out, leave the company, and while some of the employees on the Eagle team stay on, many more have left.

Tracy Kidder got an impressive amount of access at Data General when he wrote this book, and while he's honest and truthful about what happened there, Data General, at least to my 21st century mind, comes out of this book smelling like shit. I base this solely on what Data General does, and I know this because Kidder doesn't whitewash - he thankfully calls it right down the middle.

While the book is never accusatory, it makes clear that Data General is a predatory employer. It preys on young, semi-idealistic college Engineering graduates, who don't have a lot of job experience and are looking more for interesting problems to solve, interesting work to do, than a big paycheck. They promise them interesting problems, and briefly, very briefly, warn them that there will be long hours and possibly a limited social life, that this job will become their life. To meet the deadlines required of them they will have to give up friends, family, and the outside world, living only the job, for months or years at a time. Plus, because they're salaried, despite all the hours they get that would be overtime, they're only making their standard pay grade.

It chews up 22-24 year old kids, and spits them out at 30, burnouts who had great potential, but were consumed by their jobs. They don't say if many of these former employees stay in the industry, and some certainly do - Ray Ozzie, creator of Lotus Notes and current Chief Software Architect at Microsoft is a Data General veteran. However, those who leave the industry with a sour taste in their mouth will probably leave worse off then they would be if they worked somewhere else. Had they been actually paid overtime, they could have possibly built a nest egg that could have allowed them to retire early, or to at least take their time looking for work elsewhere.

While some poor decisions related to processor architecture helped to kill Data General right before the dawn of the 21st century, it is my suspicion that the boom in Silicon Valley may have inspired a brain drain. Nicer weather, a less oppressive corporate culture. For people who wanted more money, there was the change to come in on the ground floor of companies which had the potential to be worth millions and get significant stock options. For those who preferred challenge, they could face whole new challenges when designing new systems and new architectures at the new companies in the Valley.

In summary, the book is a high resolution snapshot of the early days of the computer industry, before the internet started to permeate our lives in subtle ways - computerized tax processing, credit cards, ATM machines, and so on, leading up to the more overt ways it would later find its way in - Bulletin Board Services, E-Mail, and finally, proper web pages. People interested in the history of the computer industry will certainly find this fascinating. People who don't care about the history of computing can still find something in the profiles of the people in this project, and how the project's process slowly wears them all down. show less

I was drawn to this book in 2008 because of my two decades of experience with Data General (DG) computers, both the 16-bit Nova class and the 32-bit MV class of minicomputers. My experience included system administration, script writing, programming in timeshare BASIC, setting up smart PCs as workstations, and running a help desk. I was too busy working with DG systems in the 1980s to realize that our MV was the product of a history-making process.

In the mid 1970s, when Digital Equipment Corporation announced the VAX series, their first 32-bit minicomputers, DG responded with a crash course called the "Eagle Project." This project is the subject of Tracy Kidder's Pulitzer prize-winning book, The Soul of a New Machine (1981). Kidder's show more book single-handedly made Data General's MV line of minicomputers the best documented computer project in history.

Kidder's book reads more like a fast-paced novel -- with somewhat less sex and violence -- than like a pedantic history book.

My Favorite Chapter: My curiosity was piqued when I came to Chapter 6, "Midnight Programmer." Being one myself, programmers were people I could identify with. But as I turned the pages, the story line gradually began to lose its appeal as I read about "Microkids" who worked on "Microteams" at their crowded desks in "Micropits" at DG. After all, my work as a programmer was done as a loner, not as a team member. To be productive, I needed quiet and solitude. Then I read this sentence that instantly drew me back into the narrative: "Much of the engineering of computers takes place in silence, while engineers pace in hallways or sit alone and gaze at blank pages." "Yes," I said to myself, "that's the way I write new software -- like an engineer at DG designs new hardware."

My Favorite Quote: I almost came unhinged when I read, "Not Everything Worth Doing Is Worth Doing Well." This was a favorite saying of Tom West, the chief designer in charge of the Eagle Project, the principal character in Kidder's narrative. In saying this, West meant, if you can do a quick-and-dirty job, and it works, do it. As a manager, West was pushing to get things done on time and on cost. As a programmer for whom errors are intolerable, this is a piece of managerial advice that I could never, ever internalize or respect. This saying may have worked for The Microkids at DG, but not for programmers like me.

We stayed with our DG MV at my institution for about a decade. Our operating system was AOS/VS, the most commonly used DG software product. It included CLI (Command Line Interpreter) allowing for complex scripting, DUMP/LOAD, and other custom components. Our MV ran our tailor-made software on demand with very little down time.

Although we were happy with the reliability of our Data General systems at my institution, the time came when a major change in software vendors necessitated a corresponding change in hardware. We replaced our 32-bit Data General hardware with 64-bit Alpha hardware from Digital Equipment Corporation.

Looking back on the experience two decades later, I must say in retrospect that the DG MV series was too little too late. While Data General was investing its last dollar into a dying minicomputer market, the personal computer was rapidly on the rise. For example, the 32-bit MV/8000 went out of DG's door in 1980. Barely a year later, in 1981, the 16-bit PC/XT went out of IBM's door. It was just a matter of time until PCs overtook the minicomputer market with the arrival of 32-bit, then 64-bit, then dual-core, then quad-core PCs.

This book is still a good read more than twenty-five years after it was written.

Trivia: Tom West, the protagonist in our story, was Tracy Kidder's college roommate. show less