Next: , Previous: Summary of C Interface, Up: Top


Appendix B NetCDF 3 to NetCDF 4 Transition Guide

B.1 Introduction

The release of netCDF-4 represents a substantial increase in the capabilities of the netCDF C and Fortran APIs.

The netCDF-4.0 release (June, 2008) allows the use of the popular HDF5 data format as a storage layer. The HDF5 format has many features, and only a subset of them are exposed in the netCDF-4 API. This represents a deliberate selection process by netCDF-4 developers to choose the most useful features of the HDF5 model, while retaining the simplicity of the netCDF APIs.

Despite many new features, full backward compatibility is assured (and extensively tested). Existing software and data files will continue to work with netCDF-4.0, just as with previous releases of the netCDF library.

The use of netCDF-4 files allows the use of the expanded data model, including user-defined types, groups, the new unsigned, 64-bit, and string types.

Using netCDF-4 files also allows the use of such features as endianness control, per-variable data compression, chunking, parallel I/O, and checksums. These features fit neatly within the classic netCDF data model.

Although the expanded data model offers many exciting new features, we expect and encourage users to proceed with care - it also allows the creation of needlessly, even horribly complex files. This would decrease interoperability and increase the work of the poor programmers trying to use the data file.

There are many netCDF-4 features which fit comfortably within the classic netCDF model. Existing programs can be very quickly converted to use features such as compression, endianness control, and chunking. This allows users to gain immediate performance pay off, with minimal software development effort.

B.2 NetCDF-4 and HDF5

NetCDF-4 depends on HDF5 to deliver the new features of the expanded data model, as well as the features required to support the classic data model.

NetCDF-4 users must have at least HDF5 version 1.8.1 (and at least zlib-1.2.3) to use HDF5 with netCDF-4.0. If these packages are not found when netCDF is built, then the netCDF library may still be built (without the –enable-netcdf-4 option), but will not allow users to create netCDF-4/HDF5 files, or use the expanded data model. Only classic and 64-bit offset format netCDF files will be created or readable. (see Configure).

The HDF5 files created by netCDF-4 will be readable (and writable) by any HDF5 application. However, netCDF-4.0 cannot read any HDF5 file, only those created by netCDF-4.

B.3 Backward Compatibility

In the context of netCDF, backward compatibility has several meanings.

Data Compatibility
NetCDF-4 provides backward compatibility for existing data. All netCDF data files remain readable and writable to the netCDF library. When a file is opened, the library detects the underlying format of the file; this is transparent to the programmer and user.
Code Compatibility
NetCDF-4 provides backward compatibility for existing software. Programs using the 4.0 release can use it as a drop-in replacement for netCDF-3.x. Existing programs will continue to create netCDF classic or 64-bit offset files.
Model Compatibility
NetCDF-4 introduces an expanded model of a netCDF data file (include such new elements as groups, user-defined types, multiple-unlimited dimensions, etc.) This expanded model is a super-set of the classic netCDF model. Everything that works in the classic model works in the expanded model as well. (The reverse is not true - code using the expanded data model will fail if run on classic model netCDF files.)

B.4 The Classic and the Expanded NetCDF Data Models

The classic netCDF data model consists of variables, dimensions, and attributes.

The netCDF-4.0 release introduces an expanded data model, which offers many new features. These features will only work on files which have been created with the NC_NETCDF4 flag, and without the NC_CLASSIC_MODEL flag (see nc_create).

New Types
New data types are introduced: NC_UBYTE, NC_USHORT, NC_UINT, NC_INT64, NC_UINT64, and NC_STRING. These types many be used for attributes and variables. See nc_def_var.
Groups
NetCDF objects may now be organizes into a hierarchical set of groups. Groups are organized much line a UNIX file system, with each group capable of containing more groups. Within each group a classic model netCDF “file” exists, with its own dimensions, variables, and attributes. See nc_def_grp.
User Defined Types
NetCDF-4 allows the user to define new data types, including a compound type (see nc_def_compound), a variable length array type (see nc_def_vlen), an enumerated type (see nc_def_enum), and an opaque type (see nc_def_opaque).
Multiple Unlimited Dimensions
NetCDF-4/HDF5 data files may use multiple unlimited dimensions with a file, and even within a variable.

B.5 Using NetCDF-4.0 with the Classic and 64-bit Offset Formats

Prior to the 4.0 release, two underlying data formats were available for the netCDF user, the classic, and the 64-bit offset format. (The 64-bit offset format was introduced in the 3.6.0 release, and allows the use of larger variables and files).

Software using netCDF, relinked against the netCDF-4.0 library, will continue to work exactly as before. Since the default create mode in nc_create is to create a classic format file, using unmodified netCDF-3 code with the netCDF-4 library will result in the exact same output - a classic netCDF file or 64-bit offset file.

When writing or reading classic and 64-bit offset files, the netCDF-4.0 library relies on the core netCDF-3.x code.

B.6 Creating a NetCDF-4/HDF5 File

The extra features of netCDF-4 can only be accessed by adding the NC_NETCDF4 flag to the create mode of nc_create. Files created with the NC_NETCDF4 flag can have multiple unlimited dimensions, use the new atomic types, use compound and opaque types, and take advantage of the other features of netCDF-4. (see nc_create).

B.7 Using NetCDF-4.0 with the Classic Model

By changing your nc_create call to create a netCDF-4/HDF5 file you gain access to many new features - perhaps too many! Using groups or user-defined types will make the file unreadable to existing netCDF applications, until they are updated to handle the new netCDF-4 model.

Using the NC_CLASSIC_MODEL flag with the NC_NETCDF4 flag tells the library to create a netCDF-4/HDF5 file which must abide by the rules of the classic netCDF data model. Such a file many not contain groups, user defined types, multiple unlimited dimensions, etc.

But a classic model file is guaranteed to be compatible with existing netCDF software, once relinked to the netCDF 4.0 library.

Some features of netCDF-4 are transparent to the user when the file is read. For example, a netCDF-4/HDF5 file may contain compressed data. When such a file is read, the decompression of the data takes place transparently. This means that data may use the data compression feature, and still conform to the classic netCDF data model, and thus retain compatibility with existing netCDF software (see nc_def_var_deflate). The same applies for control of endianness (see nc_def_var_endian), chunking (see nc_def_var_chunking), checksums (see nc_def_var_fletcher32), and parallel I/O, if netCDF-4 was built on a system with the MPI libraries.

To use these feature, change your nc_create calls to use the NC_NETCDF4 and NC_CLASSIC_MODEL flags. Then call the appropriate nc_dev_var_* function after the variable is defined, but before the next call to nc_enddef.

B.8 Use of the Expanded Model Impacts Fortran Portability

Using expanded model features impacts portability for Fortran programmers.

Fortran compilers do not always agree as to how data should be laid out in memory. This makes handling compound and variable length array types compiler and platform dependant.

(This is also true for C, but the clever HDF5 configuration has solved this problem for C. Alas, not for Fortran.)

Despite this, Fortran programs can take advantage of the new data model. The portability challenge is no different from that which Fortran programmers already deal with when doing data I/O.

B.9 The C++ API Does Not Handle Expanded Model in this Release

Unfortunately, the C++ API does not support the netCDF-4 expanded data model. A new C++ API is being developed and may be built by adventurous users using the –enable-cxx4 option to configure (see Configure).