Project History

From Open Watcom

Revision as of 21:44, 25 May 2012; view current revision
←Older revision | Newer revision→
Jump to: navigation, search

Open Watcom's history is quite long; it is likely one of the older software products still around after several decades. In summer 1965, a team of undergraduate students at the University of Waterloo in Canada developed a FORTRAN compiler (called WATFOR) that ran on the University's IBM 7040 systems. The compiler was soon ported to IBM 360 and later to the famous DEC PDP-11.

In early 1980s a brand new version of the compiler was created that supported the FORTRAN 77 language. It ran on two platforms, the IBM 370 and the emerging IBM PC. The PC version of WATFOR-77 was finished in 1985 and in the same year support for Japanese was added. In 1986, WATFOR-77 was ported to the QNX operating system.

The early compilers were written in a portable language called WSL or Waterloo Systems Language. In late 1980s the developers rewrote the existing code in C and from then on all new development was done on C, with some C++ thrown in since early 1990s.

In parallel to the FORTRAN compilers Watcom developed optimizing C compilers. When the first PC version (Watcom C 6.0) was introduced in 1988, it immediately attracted attention by producing faster code than other compilers available at that time. The Watcom code generator supported both the C language and FORTRAN and was portable across multiple platforms. Generation of tight code, availability on multiple platforms (DOS, Windows, OS/2, and Windows NT in one package) and the ability to cross-compile made Watcom C and C++ compilers quite popular in mid-1990s. Around 1993-1996, nearly all DOS games were developed with Watcom C, including famous titles such as DOOM, Descent or Duke Nukem 3D. For a non-exhaustive (albeit long) list of software developed with the Watcom compilers, see List Of Watcom Created Software.

Contents

Open Sourcing

The last commercial release of Watcom C/C++ and F77, version 11.0, was somewhat unfortunate (to the extent that objects which are not only inanimate but also intangible can be said to be unfortunate). It was a significant release in terms of new features and internal changes. Support for 64-bit integers was added to the C and C++ compilers and runtime libraries, namespaces and RTTI were introduced to the C++ compiler. The disassembler was rewritten and the linker and librarian were modified to use a library abstracting object file formats.

All those were good things. The problem was that version 11.0 was released at a time when Sybase was backing out of the development tools business. There were simply not enough resources to adequately test and fix the 11.0 release. There was for instance an infamous bug that caused the linker to be nearly unusable for creating OS/2 executables, and incremental linking (used by the IDE) had a nasty tendency to crash. As a result, many developers preferred to stick with version 10.6, which was very solid.

Sybase decided to pull the plug on the product because they realized that Windows was their primary market and that they could not compete against Microsoft -- no one could. Many customers were understandably unhappy about this decision because they chose the Watcom tools for their strong cross-platform support, excellent extended DOS support, very effective debugger, or extremely flexible #pragma aux capability, and there simply was no replacement (there still isn't).

Fortunately, Sybase agreed to open source the compilers. This was possible -- among other things -- because the guys at WATCOM International were true DIYers and created their own complete development system with almost zero dependencies on any third party tools or source code.

11.0c Release

SciTech received a CD with the version 11.0c source code (not released, the last Sybase release was 11.0b) in 2000. A second CD contained the source text for all documentation, which used to be built on a separate machine. The first phase of the project was not open source; a team of non-Sybase developers was to work under a NDA, essentially as unpaid contractors, and release the 11.0c "patch".

The source code was not really a mess but not exactly clean either. As tends to happen with closed source projects, the build was rather finicky and depended on specific versions of many tools that were checked into the source tree. Although it wasn't supposed to, it made many assumptions about directory structure. The build actually used version 10.6 compilers, except for the C++ runtime which is tied to a specific C++ compiler version and needs to be built with a special C++ compiler build.

SciTech spent a lot of time getting the beast to build at all. It is not easy to find one's way around 2 million lines of unfamiliar code with very little help. The build system was rather haphazard and inconsistent; some parts were nice and clean, others were hardcoded and ugly. It was obvious that the projects were set up by many different developers who each had different ideas about what a makefile should look like. The fact that a full build took well over two hours on a fairly fast machine did not help either. And once the code was building, SciTech had to go in and fix some of the most glaring bugs.

In late 2001 SciTech released a beta of the 11.0c "patch". It was essentially version 11.0 with all third party components removed -- Microsoft Win32 SDK, IBM OS/2 Toolkit components, MFC, the BlueSky Visual Programmer. The 11.0c patch was intended as an upgrade for existing 11.0 owners and could not work on its own, although it was possible to get it to work with some tweaking. In 2002, the final 11.0c patch was released.

Open Watcom 1.0

The open source release presented new challenges. Sybase had to go over every single file in the source tree (tens of thousands of files) and make sure that only source code owned by Sybase was released. The largest (unfilled) hole created in the process was removal of the QSSL owned parts of the QNX runtime library.

Once SciTech had the files that were cleared for release, the biggest problem was how to build them. The 11.0c build used a few commercial tools (several versions of MASM, OPTASM, PharLap's 386|ASM, and so on) that could no longer be used. All x86 assembly source that was not already using wasm had to be converted to it.

At the same time SciTech also had to switch the C/C++ compilers and use version 11.0c instead of 10.6. This was a significant effort. Not so much because version 11.0c was buggy -- by that time all major bugs were ironed out -- but because it used different defaults. Lots of code in the source tree assumed tight structure packing, which was the default up to version 10.6. In 11.0 the default was 8-byte alignment (2-byte for 16-bit compilers) and many components stopped working. SciTech had to debug each problem, find the offending structures, and declare them properly -- or at least jam in the -zp1 switch.

The final issue was what to do about the loss of Win32 SDK and OS/2 Toolkit. Without them, the compiler could not be used for Windows and OS/2 development. Using the Microsoft Platform SDK and IBM OS/2 Toolkit was possible, but neither was freely available. SciTech ended up using the w32api project from the MinGW folks for Win32 support, and rolled their own new headers and libs for OS/2. It was a lot of work, and both the w32api and os2api projects are continually updated, but in hindsight it was a good decision. We can now tweak the headers and libs as we need to, and more importantly, we can freely distribute them -- meaning that one can eg. cross compile to Windows or OS/2 from Linux.

Life After 1.0

Once version 1.0 was released (in 2003), our goal was to ensure that Open Watcom could build itself. We have succeeded in this task and for building on Windows or OS/2, the only prerequisite (besides source code) is the previous Open Watcom release. For building on Linux, GCC 3.x will do the job of bootstrapping the build. This lack of dependencies may be unique for a project of this size, and it makes it remarkably easy to set up a build environment on a clean system.

A fair amount of work has been done on the compilers and libraries -- many C99 language features and library functions have been implemented, the C++ compiler has been much fixed and improved, especially in the area of template support. Review the list of release changes for information about what has been added.

An ongoing and significant effort, though "invisible", is the addition of many regression tests of the compilers and libraries. Sybase used commercial test suites that could not be open sourced, and some components were simply not adequately tested (a few still aren't). Testing a compiler that supports so many target platforms takes some effort and automation is a must. But it is, of course, a requirement for releasing a quality product, which is what we aim to do.

Life After 1.6

After the release of Open Watcom version 1.6, and as a result of changing business plans, SciTech no longer directly supported the development of Open Watcom. Instead support in the form of financing server resources switched to Perforce, the company responsible for the version control system used by the Open Watcom project. In addition there was some reorganization of the Open Watcom project's administrative structure. However, work continues and these changes have not disrupted Open Watcom's long term progress.

Version History

Watcom C 6.0

  • DOS host and target only
  • Included a debugger and full set of runtime libraries
  • Generated better code than other compilers at the time

Watcom C 7.0/386

  • First 32-bit version, DOS host and target only
  • Supported PharLap DOS extender
  • Did not come with linker or debugger

Watcom C 8.0/386

  • Added linker, librarian, debugger, and profiler
  • Added graphics library

Watcom C 8.5/386

  • Shipped with DOS/4GW
  • Windows 3.0 supported (Win386 extender)
  • Unicode support
  • OS/2 hosted executables added

Watcom C 9.0/386

  • OS/2 2.0 host and target support
  • 486 optimizations
  • Based pointer support

Watcom C 9.01/386

  • Windows 3.1 support

Watcom C 9.5/386

  • C++ compiler added
  • Pentium optimizations
  • Windows NT host and target support

Watcom C/C++ 10.0

  • 16-bit and 32-bit tools merged into single package
  • Graphical IDE for Windows and OS/2
  • Redesigned debugger
  • C++ class browser added
  • Precompiled header support
  • Windows resource editors added
  • MFC included

Watcom C/C++ 10.5

  • Windows 95 and NT 3.5 support
  • Native C++ exception handling on OS/2 and Win32
  • TCP/IP remote debugging

Watcom C/C++ 10.6

  • Structured exception handling in C
  • Improved compatibility with Microsoft compilers

Watcom C/C++ 11.0

  • Namespace, RTTI, and new style cast support in C++ compiler
  • 64-bit integer support
  • Incremental linking support
  • COFF and ELF object file support in linker and librarian
  • Microsoft clone tools added
  • Multi-byte character support in libraries
  • DLL based tools for better IDE integration

Timeline

A brief timeline of Watcom C and C++ compiler releases follows:

1984 
Waterloo C for S/370 released
1985 
Work on current code generator codebase started
1988 
Watcom C 6.0 released
1989 
Watcom C 7.0 and C 7.0/386 released
1990 
Watcom C 8.0 and C 8.0/386 released
1991 
Watcom C 8.5 and C 8.5/386 released
1992 
Watcom C 9.0 and C 9.0/386 released
1993 
Watcom C/C++ 9.5 and C/C++ 9.5/386 released
1994 
Watcom C/C++ 10.0 released
1995 
Watcom C/C++ 10.5 released
1996 
Watcom C/C++ 10.6 released
1997 
Watcom C/C++ 11.0 released
1998 
Watcom C/C++ 11.0B released
1999 
Sybase issues end-of-life notice for Watcom C/C++ 11.0
2000 
Sybase announces open sourcing of Watcom tools
2001 
Watcom C/C++ 11.0c Beta (September 27) released
2002 
Watcom C/C++ 11.0c (December 21) released
2003 
Open Watcom 1.0 (January 28) and 1.1 (August 12) released
2004 
Open Watcom 1.2 (January 7) and 1.3 (August 3) released
2005 
Open Watcom 1.4 (December 14) released
2006 
Open Watcom 1.5 (April 26) and 1.6 (December 15) released
2007 
Open Watcom 1.7 (August 18) released
2007 
Open Watcom 1.7a (October) released
2009 
Open Watcom 1.8 (February 21) released
2010 
Open Watcom 1.9 (June 2) released
Personal tools