There are differences in evaluating and comparing open source software (OSS) products, as opposed to commercial packages. A number of key OSS criteria are also described. This information is based on years of experience at Redpoint , using OSS and evaluating both open source and commercial software for clients.

Evaluating OSS effectively requires different criteria than the ones used for commercial software. Commercial software selection techniques and tools are inadequate in making informed choices about OSS. Different OSS products will have strengths in different functional and technical areas. Some are growing and vibrant with strong growth in the developer and end-user community, others may be stagnant.

The traditional commercial software evaluation process involves establishing software requirements/functionality evaluation criteria, reading industry analyst reports, meeting with sales and technical pre-sales teams and negotiating licensing and support costs. An in-depth evaluation of commercial software will involve talking to a few reference customers and reviewing the financial stability of the software vendor and the longevity of their product.

Much of the information that is used for the commercial software evaluation is obtained from the software vendor’s sales team. When evaluating OSS, there typically aren’t any sales teams, no on-site demos by the sales team, no glossy sales brochures that highlight product advantages, no advertising and, as a result, the process needs to be different. Also, unlike commercial software, it is not always easy to determine who else is using an OSS product.

Industry analyst reports on OSS are scarce and the information is usually limited. Typically, OSS evaluations focus on the few leading OSS initiatives such as Apache, Linux, MySQL, etc. When evaluating an open source software package, it is important to focus on the following categories:

• Product Capabilities
• Quality
• Integration and Dependencies
• OSS Community
• Support

Product Capabilities

It is usually easy to identify and verify the capabilities of an OSS product. Most groups clearly document their features and exaggerations of their capabilities are not typical as they will generate support email and unfavorable discussions in their community. Also, these groups are typically looking for developers to implement their missing feature so they have an incentive to be honest about the capabilities.

Open source products many times have a different focus because of their origins or because of the influence of the developers that initially developed it. Also, OSS tends to evolve and sometimes take on different focus. It is important to consider the focus of a product and how it has evolved to get an indication of where the product may evolve in the future.

Although all OSS can be modified, it is important to understand what modifications are in-line with the product direction so they are within the upgrade path in the future. Most CRM OSS provides a mechanism to customize the software. However, the technique and range of customization that is allowed varies greatly among the products.

Quality

In addition to inspecting the code, development approach, documentation, build environment, testing strategy and general product literature, inspecting the bug database, which is usually public, is a good way to understand the number of bugs in a system. This will indicate the number of high-visibility bugs outstanding and the length of time it takes for typical bugs to get fixed.

Integration and Dependencies

Most OSS list the other OSS or commercial products with which they integrate. The products with which they integrate and the way in which the OSS integrates often says a lot about the general integration mechanism that is available in the product.

Product dependencies also give an indicator of how the product is intended to be integrated. Many times the dependencies are available explicitly from the OSS literature, other times it must be inferred by reviewing the installation and support documentation.

OSS Community

OSS projects can vary from a single developer to a distributed, world-wide, company-backed team. It is important to understand the project structure, its backing and its motivation. Some products are developed by commercial companies attempting to make their money from the customization, support and training. Other projects are only supported by key alliance partners that are a result of the OSS community.

Ideally the team is more than a single developer, but sometimes a very large, distributed team can hinder the ability for the product to have a clear, concise vision and to make targeted release dates.

The strength of the community is best gauged by the support and activity of the OSS wikis, forums and mailing lists. Typically, there are two mailing lists – one for users that includes help installing, upgrading, etc. and another list for the developers to discuss bugs, the roadmap, new features, etc.

Development Process and Source Code Control

The quality and development process is another important consideration for OSS that is not typically available for commercial software. The code can be reviewed and checked for adherence to general product and industry best practices and coding style.

The testing techniques used are also very important. Even if it is not documented, a quick check of the build script will typically reveal whether any unit testing is incorporated into the project. Overall, the software should have some documentation – both from an architectural and code-level perspective.

Even if you don’t plan to alter the OSS source code, it is important that the system can be built (i.e. compiled). Even if you don’t intend to contribute back to the OSS community, it is important to be able to rebuild the system if a show-stopper bug were encountered.

The release management techniques employed by the OSS development group is important. A quality product does not necessarily have a robust release process and does not necessarily have any easy upgrade path. A feel for the release process can be determined by reviewing the release notes with previous builds and determining whether frequent bug releases are distributed shortly after major product releases.

Support

The first level of support for typical OSS is by the community having access to the source. Many OSS bugs are identified and fixed by developers that use the product. Some fix and contribute back the fix to help ensure it is fixed in a future release, not because they feel they have to contribute to the community.

In order to perform fixes on the OSS source code, it must be logical, easy to understand and proper documentation must exist. Some documentation may be in-line in the source code and other documentation will vary from on-line to PDF format.

Most major OSS products have paid support options. This support can range from first-tier support, where the vendor will act as your first-line help desk, to second- and third-tier support where internal support will first attempt to identify and fix a problem before it is escalated to the paid support. The cost of the support decreases as the company takes on more initial support roles (e.g. help desk).

Paid support can range from typical commercial-style maintenance contracts to per-incident agreements. It is important to understand the typical turn-around time for support under the various support agreements.