The release of netCDF-4 represents a substantial increase in the capabilities of the netCDF C and Fortran APIs.
The netCDF-4.0 release (June, 2008) allows the use of the popular HDF5 data format as a storage layer. The HDF5 format has many features, and only a subset of them are exposed in the netCDF-4 API. This represents a deliberate selection process by netCDF-4 developers to choose the most useful features of the HDF5 model, while retaining the simplicity of the netCDF APIs.
Despite many new features, full backward compatibility is assured (and extensively tested). Existing software and data files will continue to work with netCDF-4.0, just as with previous releases of the netCDF library.
The use of netCDF-4 files allows the use of the expanded data model, including user-defined types, groups, the new unsigned, 64-bit, and string types.
Using netCDF-4 files also allows the use of such features as endianness control, per-variable data compression, chunking, parallel I/O, and checksums. These features fit neatly within the classic netCDF data model.
Although the expanded data model offers many exciting new features, we expect and encourage users to proceed with care - it also allows the creation of needlessly, even horribly complex files. This would decrease interoperability and increase the work of the poor programmers trying to use the data file.
There are many netCDF-4 features which fit comfortably within the classic netCDF model. Existing programs can be very quickly converted to use features such as compression, endianness control, and chunking. This allows users to gain immediate performance pay off, with minimal software development effort.
NetCDF-4 depends on HDF5 to deliver the new features of the expanded data model, as well as the features required to support the classic data model.
NetCDF-4 users must have at least HDF5 version 1.8.1 (and at least zlib-1.2.3) to use HDF5 with netCDF-4.0. If these packages are not found when netCDF is built, then the netCDF library may still be built (without the –enable-netcdf-4 option), but will not allow users to create netCDF-4/HDF5 files, or use the expanded data model. Only classic and 64-bit offset format netCDF files will be created or readable. (see Configure).
The HDF5 files created by netCDF-4 will be readable (and writable) by any HDF5 application. However, netCDF-4.0 cannot read any HDF5 file, only those created by netCDF-4.
In the context of netCDF, backward compatibility has several meanings.
Data Compatibility
Code Compatibility
Model Compatibility
The classic netCDF data model consists of variables, dimensions, and attributes.
The netCDF-4.0 release introduces an expanded data model, which offers many new features. These features will only work on files which have been created with the NC_NETCDF4 flag, and without the NC_CLASSIC_MODEL flag (see nc_create).
New Types
Groups
User Defined Types
Multiple Unlimited Dimensions
Prior to the 4.0 release, two underlying data formats were available for the netCDF user, the classic, and the 64-bit offset format. (The 64-bit offset format was introduced in the 3.6.0 release, and allows the use of larger variables and files).
Software using netCDF, relinked against the netCDF-4.0 library, will continue to work exactly as before. Since the default create mode in nc_create is to create a classic format file, using unmodified netCDF-3 code with the netCDF-4 library will result in the exact same output - a classic netCDF file or 64-bit offset file.
When writing or reading classic and 64-bit offset files, the netCDF-4.0 library relies on the core netCDF-3.x code.
The extra features of netCDF-4 can only be accessed by adding the NC_NETCDF4 flag to the create mode of nc_create. Files created with the NC_NETCDF4 flag can have multiple unlimited dimensions, use the new atomic types, use compound and opaque types, and take advantage of the other features of netCDF-4. (see nc_create).
By changing your nc_create call to create a netCDF-4/HDF5 file you gain access to many new features - perhaps too many! Using groups or user-defined types will make the file unreadable to existing netCDF applications, until they are updated to handle the new netCDF-4 model.
Using the NC_CLASSIC_MODEL flag with the NC_NETCDF4 flag tells the library to create a netCDF-4/HDF5 file which must abide by the rules of the classic netCDF data model. Such a file many not contain groups, user defined types, multiple unlimited dimensions, etc.
But a classic model file is guaranteed to be compatible with existing netCDF software, once relinked to the netCDF 4.0 library.
Some features of netCDF-4 are transparent to the user when the file is read. For example, a netCDF-4/HDF5 file may contain compressed data. When such a file is read, the decompression of the data takes place transparently. This means that data may use the data compression feature, and still conform to the classic netCDF data model, and thus retain compatibility with existing netCDF software (see nc_def_var_deflate). The same applies for control of endianness (see nc_def_var_endian), chunking (see nc_def_var_chunking), checksums (see nc_def_var_fletcher32), and parallel I/O, if netCDF-4 was built on a system with the MPI libraries.
To use these feature, change your nc_create calls to use the NC_NETCDF4 and NC_CLASSIC_MODEL flags. Then call the appropriate nc_dev_var_* function after the variable is defined, but before the next call to nc_enddef.
Using expanded model features impacts portability for Fortran programmers.
Fortran compilers do not always agree as to how data should be laid out in memory. This makes handling compound and variable length array types compiler and platform dependant.
(This is also true for C, but the clever HDF5 configuration has solved this problem for C. Alas, not for Fortran.)
Despite this, Fortran programs can take advantage of the new data model. The portability challenge is no different from that which Fortran programmers already deal with when doing data I/O.
Unfortunately, the C++ API does not support the netCDF-4 expanded data model. A new C++ API is being developed and may be built by adventurous users using the –enable-cxx4 option to configure (see Configure).