Published on The O'Reilly Network (http://www.oreillynet.com/)
 http://www.oreillynet.com/pub/a/bsd/2003/07/17/openbsd_core_team.html
 See this if you're having trouble printing code examples

The Essence of OpenBSD

by Cameron Laird and George Peter Staplin
07/17/2003

A half-dozen OpenBSD principals participated in emailed interviews on their work. Here's what they had to say:

O'Reilly Network: How did the team form?

Different members came at different times, so a couple of us will answer.

deraadt [Theo De Raadt]: Well, the history of when I started OpenBSD might be well-known by most. Early on, the first team members were people who were unhappy with NetBSD. In particular, quite a few Swedish people joined ... about a year later a security focus started in the project, as some people from a Calgary company called Secure Networks started helping, and then ... after that I have kind of lost track, since it has been almost eight years ....

dhartmei [Daniel Hartmeier]: I didn't know about OpenBSD back when it was started (1996, if I'm not mistaken); I learned about the split from NetBSD later. I joined the team in June 2001, when my work on the new packet filter was imported into the source tree. First contacts with other developers were based on email and ICB [Internet Citizen's Band; this email explains more of its role in OpenBSD's history], and the rules and rites are passed informally from the veterans to the newcomers. Once a year we meet at a "hackathon," which is how I met most people for the first time in real life, to discuss new ideas and make plans for the next year. When people have shown the required skills in the past and are interested, they get invited to these events and recruited. There are no formal positions filled; basically anyone with skill and time is welcome to work in the areas he or she is interested in. There is a strict peer review process where the more experienced developers review any changes to the source tree before they get committed, and provide feedback. As one or several developers spend more time in a particular source part, others will consult them when working on those parts, like maintainers.

jason [Jason Downs]: Can't speak to the time before I was brought in (Jan. of 1998), but for myself, I tried installing NetBSD and OpenBSD on an ancient machine at the time (a Sun 4/330), and neither one worked. I hacked around the problem by installing a previous release and then read through the change logs to figure out what happened, and submitted a bug report (with patch) and got mail from some Theo de Raadt dude asking if I wanted an account. My life hasn't been the same since. Generally, if you prove yourself to be good at something needed in the project, you'll eventually get asked. There's a sort of mentoring process where a new developer is brought in and given limited rein until he/she is "proven."

miod [Miod Vallat]: I cannot answer for the 1995-1999 years, since I was not "officially involved" in OpenBSD back then. I myself was invited to join the OpenBSD team in early 2000, after I had been working independently on my own, and have had good communications with several developers. One can say that there are, roughly said, two ways to enter the team: either by providing something new and valuable to the OpenBSD project, or providing valuable and reliable help on various tasks for some time until an existing team member gets bored of having to finish the work for you. In both cases, this is a recognition of your merits and the quality of your work. New developers regularly show up, and can be long-time users that want to contribute in return, as well as newcomers with good skills ... there is no preselection criterion.

drahn [Dale Rahn]: Back in 1995, it formed by Theo (one of the founders of NetBSD) breaking off from NetBSD and forming his own development group.

I (Dale) had worked with Theo on NetBSD because earlier that year Theo, Chuck Cranor, and myself all had developed NetBSD ports for the Motorola MVME147 (embedded computer). I had never been closely associated with NetBSD, but I had used their software on i386 and Amiga for some time. Before releasing my MVME147 port I had never had much communication with the developers. BTW, I had been along for the ride with *BSD since the beginning; I purchased my first PC machine to run 386BSD 0.0 in 1992 before graduating from Purdue.

Theo had seen my work on my MVME147 port and knew that I had access to hardware and documentation since I worked at Motorola. He was working on porting (then OpenBSD) to the MVME162 board and wanted it to run on the MVME167 (very similar boards which were faster than the MVME147), but a new serial driver needed to be written for the MVME167. Theo convinced me to write the driver for the hardware and when I had it mostly working, he invited me to be a developer.

Team additions normally occur when someone new shows up and appears to have skill or potential to develop the necessary skills. When they seem to be submitting decent changes, and seem responsible, they are invited to be developers. Their changes to the tree are monitored closely, to prevent errors from being introduced into the tree and to direct the developer's abilities and improve the quality of future changes.

ORN: What do you mostly work on?

dhartmei: Primarily, I'm working on the packet filter, both the kernel and the related userland parts. As a consequence, I'm familiar with the TCP/IP stack and networking code, and can help with problems in there. I have done some work in other parts and try to help with tasks that require help from many developers, like the ongoing replacement of unsafe string-handling functions.

jason: Drivers. Floating-point emulation. Drivers. The sparc64 port. Drivers, and a tiny bit of userland stuff (usually to go with kernel stuff, like the bridge, that I've written).

miod: I am mostly working in the kernel area, although I contribute userland changes from time to time. My main area of work is the support of old, dog-slow, obsolete, and generally not-worth-a-dime-anymore architectures, such as the various Motorola 680x0-based systems, as well the VAX systems, for example. However, I have been working on every architecture supported by OpenBSD.

Supporting obsolete architectures help ensure the overall portability and correctness of the operating system code, and in fact, this work often turns into janitor work: a design peculiarity of a specific architecture or a run-time strange behavior forces me to check how our code behaves on all of the supported architectures, and make the behavior consistent.

drahn: UGH. What areas have I not worked on (scratches head)? Currently my main tasks are port maintainer for Mac PPC (Apple PowerPC hardware) and security improvements.

Most of my recent work has involved the W^X (Writable xor eXecutable) security enhancements to OpenBSD. The goal of this effort is to make it much more difficult, if not impossible, to allow a hacker to inject code into a machine and then execute it. Adding this security feature to i386 has involved many changes in the compilation environment and the kernel to allow us to bend some of the typical assumptions into a form that allows W^X protection.

Other pending projects include Advanced Power Management (APM) sleep support for Apple laptops, W^X support on PowerPC, developing audio drivers for newer Apple machines, improved PowerPC VM/system calls, PowerPC SMP?

ORN: Do you compare your work to that of others, and if so, how does it compare?

drahn: While it might be interesting to compare my work with other developers' work, it often is quite difficult to compare. The areas I work in tend to be areas where few other developers work; sometimes finding someone to review my code is hard enough.

miod: Developers usually have their own areas of expertise, and know the others' areas. If an area is shared with other developers, or two areas share many concepts or code, developers have to communicate with each other and share their work. It's not an obligation, but it's the only way to do reliable work. And there are always new ideas to borrow from your peer's work.

ORN: It seems that with most systems driver developers make up most of the kernel team. Is this true with OpenBSD? Who writes most of the drivers? Do you usually use other BSD drivers when possible? Do you sometimes use Linux drivers as an example?

drahn: This brings up the frequent problem with open source development: getting documentation for hardware. Most of the time, if a *BSD driver is available, it is much, much better to port that driver or at least start with that driver and make improvements where possible, hopefully feeding the changes back to the original authors.

Sometimes when hardware documentation is not available, the only source of information is from Linux device drivers. GPL prevents us from taking their driver and releasing it as a restriction-free driver. It is sometimes possible to read the GPL driver, reverse-engineer what the hardware does, and then write a driver based on that understanding.

jason: Systems driver developers are a big part of the group, but not the biggest. The largest group is probably the folks who handle third-party software "ports." Most of the drivers written from scratch are written by a very few people within the project. The majority of drivers come from the other BSDs, but this is not always a trivial undertaking. For instance, FreeBSD has a completely different disk subsystem, CAM, which makes porting SCSI drivers from there difficult. They also have a very different API for DMA handling. Linux drivers, besides the GPL copyright issue (which I will not address), are difficult to follow because of the use of "magic constants": using hex digits instead of symbolic names. In some cases, they can be used as supplements for vendor-provided documentation, but because of different kernel semantics and magic constants, it's difficult to use them more directly.

miod: This is probably true. A lot of developers are involved only in kernel code, or mostly in kernel code (like myself). A large part is also focusing on third-party applications (port collection), and then a few developers are working mostly on userland. However, there are no fixed boundaries; people work on what they like to work on.

As for driver support, a significant set of drivers is shared between the main three BSD flavors, regardless of where they originated; this is, for example, the case for a lot of network card drivers, the USB subsystem, and crypto hardware. I usually do not look at the Linux drivers, but the hardware I write drivers for is generally either not supported by any free operating system, or only by other BSD flavors.

ORN: As OpenBSD has grown, have you lost any vital developers, and why?

drahn: I was lost to OpenBSD for over a year, several years back. I have seen two main causes for loss of developers: One, lack of "free" time on the part of the developer, where they no longer have enough time in their life to continue working on OpenBSD. Two, personality conflicts.

ORN: What books were most helpful to learn how to do your particular specialty?

jason: The Design and Implementation of the 4.4BSD Operating System is a must, but primarily I have spent my time (and still do!) reading datasheets for various chips and then figuring out how they relate to existing drivers. I've learned a few things about datasheets:

  1. Prerelease glossies are useless (they don't even make good litter liner).
  2. Preliminary datasheets (before the chip is released) are problematic, since inevitably, something in the design changes and they forget to update the datasheet.
  3. Bad datasheets are almost as bad as no datasheets.

ORN: Sun has been defiant about providing UltraSparc III information, so that you could make a proper port to Sun's newest processor. The Linux developers that have added support for this processor signed non-disclosure agreements (NDAs). De Raadt has said that he will not do this. Has this tarnished your image of Sun, and has there been any progress towards adding a port for this processor?

jason: What really irks me is that the Linux guys did nothing to help the community at large by signing the NDAs. It's nice that a very small group of Linux folks can fix bugs in the bowels of their kernel, but not having docs generally available hurts the community. The Linux folks had the power to do this right a while ago.

My image of Sun is that of many other chip vendors who think that keeping the documentation closed somehow helps their business model. So they are not tarnishing my image of them at all, only living down to it, unfortunately.

miod: I am sure disappointed by this situation, especially since Sun has not been known for double-speak like this in the past. Like probably most of the UNIX-contaminated people out there, I have been happily using Sun hardware for years, and I am still satisfied with this hardware, but the company seems to be getting more and more unfriendly (to say the least) and I don't know if I will continue using Sun hardware over the next few years, if only because I do not trust the hardware manufacturer as I used to five or 10 years ago.

ORN: What systems other than OpenBSD do you use, and why?

jason: I have one Windows machine ... It runs Quicken and Quickbooks exclusively (mainly because the alternatives do not fit my needs).

miod: Being a Unix hardware collector, I use a lot of non-OpenBSD systems, such as AIX, IRIX, and Tru64 UNIX; also, NetBSD on some platforms which are not supported (currently) by OpenBSD. However, in my day-to-day use, the only operating system I use besides OpenBSD is IRIX (SGI's version of UNIX) because SGI workstations have amazing graphics capabilities, and IRIX is the only operating system that fully supports this hardware. And I really enjoy hardware OpenGL screen savers. I also used to run OS/2 happily for 10 years ('90-'99), but it feels like it was in a previous life!

drahn: Well, since a lot of the work I do is on Apple Macintoshes, I occasionally boot Mac OS 9/X on some of the machines. Sometimes this is to play games, sometimes this is to run software that is not available on OpenBSD. While I have had at least one PC-based machine since 1992, I have never had Windows installed on any of my machines.

ORN: If OpenBSD didn't exist, what would you most likely be using/doing?

jason: Probably working for a stuffy government contractor or something. Not having nearly as much fun. =)

miod: Well, OpenBSD evolved from NetBSD, and I had been a satisfied NetBSD user before I switched to OpenBSD in late 1997. So, if OpenBSD did not exist, I would probably be using NetBSD instead.

ORN: OpenBSD has used CVS practically since the OpenBSD project started. Has CVS scaled well for what you do, or have you been looking into other solutions?

drahn: CVS is a fair tool. However, it has its own set of problems. Some of the OpenBSD developers were hopeful when we first saw OpenCM. However, that system appears to have its own set of problems. Currently, it is not more desirable than CVS.

miod: Despite a few minor annoyances, I am pretty satisfied with CVS. I know that a few developers have been considering other software, especially the promising OpenCM, but I have not been researching this myself. CVS matches my needs so far.

ORN: It was noted in a message by De Raadt that i386 and the PowerPC are not capable of doing per-page execute permission. Are there any plans to have a workaround that might be slower, but provide something similar to this?

miod: As far as I know, there are already workarounds (in 3.3 for PPC, post-3.3 for x86), which are a best effort to compensate for the lack of real X bit in these processors.

drahn: OpenBSD has an implementation of W^X protection for i386 in -current. The current version has been implemented with no noticeable slowdown. Some issues have been raised that may require some adjustment to the current strategy.

We have plans for slightly modified version of this strategy for PowerPC; however, coding for that has not started.

ORN: Two AMD Hammer (x86-64) boxes have been donated to the project so that OpenBSD will support this architecture. What are your thoughts about this processor? Is it better from a security standpoint than x86 (non-executable pages of memory, etc.)?

drahn: The Hammer implementation is much nicer in that it has a per-page execute bit; this allows W^X support with no special tricks. Other than this and a >32-bit address space, the machine is basically just a standard PC. OK, I have to admit that at the current time the fastest processors available are PCs (x86 based).

ORN: Have you considered writing system programs in a safer language than C?

miod: I don't know much about the "new" languages of the last few years (such as Ruby or C#). But among all the languages I know, I feel C is the best-suited for systems programming, if only because it provides very few barriers, thus allowing you to do exactly what you want to do. The drawback is that you have to be very careful in what you do, and sincerely, I'd rather have to be twice as more cautious than trusting a compiler and its runtime library to do this work for me. A friend of mine likes to describe OpenBSD as "craftsmen work," and I agree with him here. Craftsmen pay attention to all the details, and only C will let you check all the details.

drahn: The problem with writing code in most languages other than C is that what they really do is limit what the developer is able to do, or wrap the actions with additional code. This often comes at a cost of code speed or obscures the code.

ORN: Do you have a need for more developers, and if so, what do you need help with?

jason: Drivers. I could really use more good hands in the network driver area.

drahn: More developers would be good; however, managing new developers can actually use up more time than writing code ourselves. This is done with the hope they stay around until they are actually useful.

miod: I think every software project needs more developers -- the more, the better. In the OpenBSD case, a few filesystem, memory subsystem, and non-TCP/IP network gurus would be really appreciated, although every area of the whole system is worth working on.

ORN: What 10 improvements would you like to make to OpenBSD in the future?

drahn:

Cameron Laird is the vice president of Phaseit, Inc. and frequently writes for the O'Reilly Network and other publications.

George Peter Staplin is a student in Utah whose own programming focuses mainly on computer graphics. He works mostly with open-source variants of Unix, including OpenBSD.


Return to the BSD DevCenter.


oreillynet.com Copyright © 2003 O'Reilly & Associates, Inc.