In the early 90's Apple was desperate for a new operating system. Past failures such as Pink and Copland left the company in need of a quick fix. With a lot of money in the bank, they started looking outside the company for a solution. The field was narrowed to NeXT and Be, companies started by ex-Apple employees. In December of 1996 a choice was made, to quote Gil Armelio, "We picked Plan A instead of Plan Be.", marking the return of Steve Jobs to Apple. Few can argue that Apple made the right decision, Apple's success under the leadership of Steve Jobs is legendary. However, the NeXTstep operating system Apple acquired from NeXT needed a lot of work before a transition could be made to satisfy Apple's goal of shipping a modern OS with their hardware. During this transition period, Apple had to decide what pieces of software would make up the new operating system. There would need to be a kernel, a compatibility layer to run old Mac OS software, and a new application framework to replace the Mac Toolbox.
Apple's newly acquired OS, NeXTstep was built on the Mach 3.0 kernel, the same microkernel used by GNU/Hurd. Though by then the Mach kernel was largely abandoned, another kernel, Linux, was gaining support and seeing rapid development. The project at Carnegie Mellon to develop Mach had ended in 1994, two years before Apple acquired NeXT. As early as 1991 papers were published documenting performance issues with the Mach kernel. At the time there was much debate over kernel design, 1992 marked the now famous flame war between Linus Torvalds and Andy Tanenbaum over monolithic vs. microkernel design. Even though Apple got Mach from NeXTstep, it still remained to be seen what kernel Apple would use for their new OS. The higher-level pieces of NeXTstep, such as the Objective-C frameworks, had already been ported to other operating systems in the form of OPENSTEP. Apple's future was not tied to the Mach kernel and other options were being considered.
Apple was no stranger to Linux, the company had been working with both the Mach and Linux kernels as early as October 1995 when they collaborated with Open Group (then Open Software Foundation) to create MkLinux. MkLinux was a project to get Linux running on top of the Mach kernel as a server process. In 1996 Apple gave away copies of MkLinux at their World Wide Developers Conference and MacWorld Boston. 1996 also marked the first complete port of the Linux kernel to PowerPC. A lot of work was being done on Linux for PowerPC, both inside Apple and by the community. Apple continued work on MkLinux until 1998 when development was transitioned to the community. It is unclear what Apple's strategy was regarding Linux then, but by 1997 Apple had demonstrated Rhapsody, the NeXTstep-based OS that would eventually become Mac OS X. In 1998 Steve Jobs announced Mac OS X Server, which shipped in 1999. By this time Linux had gained wide spread industry support as a server OS and was poised to move onto the desktop.
Despite known performance problems, prior failures like GNU Hurd, and the Mach project ending at Carnegie Mellon, Apple had based their future operating system on Mach instead of Linux. The new kernel, XNU, was released under an Open Source license as part of Darwin. XNU is a hybrid kernel combining Mach, FreeBSD, and a C++ driver API called IOKit. Unlike Linux, Darwin has seen very little community support mainly because of Apple's refusal to provide enough information and support. With the release of Mac OS X for Intel, Apple has essentially closed development of Darwin by not releasing the x86 version. This leaves only Apple's internal team of kernel engineers to improve XNU, making it nearly impossible for Apple to compete with Linux or Microsoft in kernel development. Like Mach 3.0, the XNU kernel has known performance problems which are slowly being addressed, but these are problems that have already been solved in other kernels. Apple can still scrap XNU and adopt the Linux kernel.
The mid-90s presented possibly the best opportunity for Apple to adopt the Linux kernel. The recent Intel switch could be counted as another missed opportunity, but a switch is still possible and it makes more sense for the long term than continuing development of their own kernel. The Linux kernel has thousands of developers and support from the industries biggest names: IBM, Sun, Novell, HP, Google, Intel, and Red Hat. The Linux kernel has also been ported to more architectures than any other kernel, which gives Apple a good foundation for embedded devices like iPods. By adding Mach-like IPC and porting IOKit to Linux, they could smooth the transition for the relatively small number of developers using XNU kernel APIs. With Mac OS X 10.4 (Tiger), Apple has only just begun providing stable kernel interfaces, so these developers are accustomed to rewriting code every major release. Apple can continue their strategy of vendor lock-in by shipping closed source drivers using modules. A downside is that the Linux kernel has to remain licensed under the GPL so Apple loses control of licensing. Steve Jobs hates giving away control. And to continue benefiting from the Linux development, Apple would be at the mercy of Linus. When he decides to change major sub-systems in the kernel, Apple would need to update their patches. Still considerably less effort than maintaining their own kernel though. Apple would also need to become more of a team player in the Open Source world, but that is a small price to pay for instantly gaining a team of thousands of developers. Apple's users could benefit from having a true enterprise server-class kernel, hundreds of drivers, and cool new features like FUSE or Xen.
2006/02/26 - Erik points out that while development of Mach at Carnegie Mellon had ended, Apple acquired one of the leaders of Mach development from NeXT, Avie Tevanian. This led me deeper into Apple's decision to use the Mach kernel. According to John Gruber, because of Avie, Apple didn't even consider other kernels, even their own NuKernel, the kernel for Copland. Other factors seem to have contributed to the decision though. I've been told that Jeff Robin was the head of the NuKernel team and he left Apple to work on SoundJam shortly after the Copland project was scrapped.
I'd also like to remind everyone that this paper was written for fun and education, as some seem to be taking it way too seriously. Personally I doubt Apple will switch kernels and they had some good reasons for choosing Mach, but it is fun to imagine how things could have been different.
cipher brings up a good point, I don't give enough reasons for Apple to make the switch. I gloss over what I consider the biggest reason: the enterprise market. Apple has shown interest in the market with Xserve and Xserve RAID, but for the enterprise to take Apple seriously, they're going to need a lot more support. Linux has that support and performance now. Businesses spend considerably more money than consumers, so this is a market that has huge revenue potential for Apple, especially now that they've got a decent partner (Intel) for producing high-end machines in volume. Then you've got the trickle down effect: people use a Mac at work and they're more likely to buy a Mac for their home.
Jason Toffaletti has been using Macs since 1987 and programming for them since 1995. In 1999 he helped write Macster, which later became Napster for the Mac. After years of dabbling in Linux, he fully switched in 2004 and now uses Ubuntu Linux at home and work.
Jason would like to thank Larry Osborn, Jonathan Baudanza, and Allan Hsu for proof reading and helpful editorial commentary.