So how much should you compress your data in a real-world application? How can you test what your trade-off really is between compression and computing time? But the fine print also acknowledged that this data was for a pretty idealized dataset. That post featured a plot that showed how much you could compress your data when increasing the compression level.

There has been a previous post, courtesy of Greg Garner, on why HDF5/zlib compression matters for NetCDF4. The easiest way to do this is with the following command which will take all the netcdf files in the directory (using the *) and merge them into a file called merged_file.nc: Assuming you have a directory with all the NetCDF files, the first goal is to concatenate the data, since my run was broken into many smaller files. My goal was to ultimately get the 500-hpa geopotential height anomalies across the whole 1200 year period for specifically the Western US.

Some of the NCO commands have size limits of 40 GB, so it’s important to use the right order of operations when processing your files, which will be different depending on your ultimate goal. I will outline how I used some of these functions for my pre-industrial runs. The official documentation and installation instructions for NCO can be found here and CDO here, but if you’re working on a supercomputer, the libraries will already likely be installed. The output was being written on daily timestep, across 1200 years, and for the whole world, so it was absolutely essential that I cut the size of the files down as much as I could before transferring to my own computer. I found these commands to be really nifty when I was working with pre-industrial control runs from a GCM. This post is an introduction to Linux-based climate data and NetCDF operators (CDOs or NCOs) which allow you to perform various operations on netNDF files through the command line.

Working with net cdf files in r