You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
There are a number of places where the CDM library needs to write temporary files to disk. If you end up using the file more than once, its useful to save these files. The CDM uses static methods in <b>_ucar.nc2.util.DiskCache_</b> to manage how the temporary files are managed.
15
+
There are a number of places where the CDM library needs to write temporary files to disk. If you end up using the file more than once, its useful to save these files. The CDM uses static methods in `ucar.nc2.util.DiskCache` to manage how the temporary files are managed.
16
16
17
17
Before the CDM writes the temporary file, it looks to see if it already exists.
18
18
19
-
1. If a filename ends with <b>_".Z", ".zip", ".gzip", ".gz", or ".bz2", NetcdfFile.open_</b> will write an uncompressed file of the same name, but without the suffix.
19
+
1. If a filename ends with `.Z`, `.zip`, `.gzip`, `.gz`, or `.bz2`, `NetcdfFile.open` will write an uncompressed file of the same name, but without the suffix.
20
20
21
-
2.<b>_Nexrad2_</b>, <b>_Cinrad2_</b> files that are compressed will be uncompressed to a file with an <b>_.uncompress_</b> prefix.
22
-
By default, DiskCache prefers to place the temporary file in the same directory as the original file. If it does not have write permission in that directory, by default it will use the directory <b>_${user_home}/.unidata/cache/_</b>. You can change the directory by calling
21
+
2.`Nexrad2`, `Cinrad2` files that are compressed will be uncompressed to a file with an `.uncompress` prefix.
22
+
By default, DiskCache prefers to place the temporary file in the same directory as the original file. If it does not have write permission in that directory, by default it will use the directory `${user_home}/.unidata/cache/`. You can change the directory by calling
You might want to always write temporary files to the cache directory, in order to manage them in a central place. To do so, call
27
27
28
-
<b>_ucar.nc2.util.DiskCache.setCachePolicy( boolean alwaysInCache)_</b> with parameter <b>_alwaysInCache = true_</b>.
28
+
`ucar.nc2.util.DiskCache.setCachePolicy( boolean alwaysInCache)` with parameter `alwaysInCache = true`.
29
29
30
-
You may want to limit the amount of space the disk cache uses (unless you always have data in writeable directories, so that the disk cache is never used). To scour the cache, call <b>_DiskCache.cleanCache()_</b>. There are several variations of the cleanup:
30
+
You may want to limit the amount of space the disk cache uses (unless you always have data in writeable directories, so that the disk cache is never used). To scour the cache, call `DiskCache.cleanCache()`. There are several variations of the cleanup:
31
31
32
-
*<b>_DiskCache.cleanCache(Date cutoff, StringBuilder sbuff)_</b> will delete files older than the cutoff date.
*<b>_DiskCache.cleanCache(long maxBytes, Comparator<File> fileComparator, StringBuilder sbuff)_</b> will retain maxBytes bytes, deleting files in the order defined by your Comparator.
32
+
*`DiskCache.cleanCache(Date cutoff, StringBuilder sbuff)` will delete files older than the cutoff date.
*`DiskCache.cleanCache(long maxBytes, Comparator<File> fileComparator, StringBuilder sbuff)` will retain maxBytes bytes, deleting files in the order defined by your Comparator.
35
35
36
-
For long running application, you might want to do this periodically in a background timer thread, as in the following example.
36
+
For a long-running application, you might want to do this periodically in a background timer thread, as in the following example.
37
37
38
-
~~~
39
-
1) Calendar c = Calendar.getInstance(); // contains current startup time
40
-
c.add( Calendar.MINUTE, 30); // add 30 minutes to current time // run task every 60 minutes, starting 30 minutes from now
// make sure you cancel the timer before your application exits, or else
61
+
// the process will not terminate
62
+
timer.cancel();
63
+
~~~
63
64
64
65
#### Writing temporary files using DiskCache2
65
66
66
-
In a number of places, the <b>_ucar.nc2.util.DiskCache2_</b> class is used to control caching. This does not use static methods, so can be configured for each individual use.
67
+
In a number of places, the `ucar.nc2.util.DiskCache2` class is used to control caching. This does not use static methods, so can be configured for each individual use.
67
68
68
-
The default constructor mimics DiskCache, using <b>_${user_home}/.unidata/cache/</b>_ as the root directory:
69
+
The default constructor mimics DiskCache, using `${user_home}/.unidata/cache/` as the root directory:
69
70
70
-
<b>_DiskCache2 dc2 = new DiskCache2();_</b>
71
+
`DiskCache2 dc2 = new DiskCache2();`
71
72
72
73
You can change the root directory by calling
73
74
74
-
<b>_dc2.setRootDirectory(rootDirectory)._</b>
75
+
`dc2.setRootDirectory(rootDirectory).`
75
76
76
77
You can tell the class to scour itself in a background timer by using the constructor:
77
78
78
-
<b>_DiskCache2 dc2 = new DiskCache2(rootDirectory, false, 24 * 60, 60);_</b>
79
+
`DiskCache2 dc2 = new DiskCache2(rootDirectory, false, 24 * 60, 60);`
79
80
80
-
~~~
81
+
~~~java
81
82
/**
82
83
* Create a cache on disk.
83
84
* @param root the root directory of the cache. Must be writeable.
84
85
* @param relativeToHome if the root directory is relative to the cache home directory.
85
86
* @param persistMinutes a file is deleted if its last modified time is greater than persistMinutes
86
87
* @param scourEveryMinutes how often to run the scour process. If <= 0, dont scour.
87
88
*/
88
-
public DiskCache2(String root, boolean relativeToHome, int persistMinutes, int scourEveryMinutes);
89
+
public DiskCache2(String root, boolean relativeToHome, int persistMinutes, int scourEveryMinutes);
89
90
~~~
90
91
91
92
You can change the cache policy from the default CachePathPolicy.OneDirectory by (eg):
You can ensure that the cache is always used with:
109
110
110
-
<b>_dc2.setCacheAlwaysUsed(true);_</b>
111
+
`dc2.setCacheAlwaysUsed(true);`
111
112
112
113
Otherwise, the cache will try to write the temporary file in the same directory as the data file, and only use the cache if that directory is not writeable.
113
114
114
115
### GRIB Indexing and Caching
115
116
116
-
In 4.3 and above, for each GRIB file the CDM writes a _grib index file_ using the filename plus suffix <b>_.gbx9_</b>. So a file named <b>_filename.grib1_</b> will have an index file <b>_filename.grib1.gbx9_</b> created for it the first time that its read. Usually a _cdm index file_ is also created, using the filename plus suffix <b>_.ncx_</b>. So a file named filename.grib1 will have an index file filename.grib1.ncx created for it the first time. When a GRIB file is only part of a collection of GRIB files, then the ncx file may be created only for the collection.
117
+
In 4.3 and above, for each GRIB file the CDM writes a _grib index file_ using the filename plus suffix `.gbx9`. So a file named `filename.grib1` will have an index file `filename.grib1.gbx9` created for it the first time that its read. Usually a _cdm index file_ is also created, using the filename plus suffix `.ncx`. So a file named filename.grib1 will have an index file filename.grib1.ncx created for it the first time. When a GRIB file is only part of a collection of GRIB files, then the ncx file may be created only for the collection.
117
118
118
119
The location of these index files is controlled by a caching strategy. The default strategy is to try to place the index files in the same directory as the data file. If that directory is not writeable, then the default strategy is to write the index files in the default caching directory. In a client application using the CDM, that default will be
119
120
120
-
<b>_${user_home}/.unidata/cache/._</b>
121
+
`${user_home}/.unidata/cache/.`
121
122
122
123
On the TDS it will be
123
-
<b>_${tomcat_home}/content/thredds/cache/cdm_</b>
124
+
`${tomcat_home}/content/thredds/cache/cdm`
124
125
125
126
Clients of the CDM can change the GRIB caching behavior by configuring a DiskCache2 and calling:
NetcdfFile objects are cached in memory for performance. When acquired, the object is locked so another thread cannot use. When closed, the lock is removed. When the cache is full, older objects are removed from the cache, and all resources released.
133
134
134
-
Note that typically a <b>_java.io.RandomAccessFile_</b> object, holding an OS file handle, is open while its in the cache. You must make sure that your cache size is not so large such that you run out of file handles due to NetcdfFile object caching. Most aggregations do not hold more than one file handle open, no matter how many files are in the aggregation. The exception to that is a Union aggregation, which holds each of the files in the union open for the duration of the NetcdfFile object.
135
+
Note that typically a `java.io.RandomAccessFile` object, holding an OS file handle, is open while its in the cache. You must make sure that your cache size is not so large such that you run out of file handles due to NetcdfFile object caching. Most aggregations do not hold more than one file handle open, no matter how many files are in the aggregation. The exception to that is a Union aggregation, which holds each of the files in the union open for the duration of the NetcdfFile object.
135
136
136
137
Holding a file handle open also creates a read lock on some operating systems, which will prevent the file from being opened in write mode.
137
138
138
139
To enable caching, you must first call
139
140
140
-
~~~
141
-
NetcdfDataset.initNetcdfFileCache(int minElementsInMemory, int maxElementsInMemory, int period);
141
+
~~~java
142
+
NetcdfDataset.initNetcdfFileCache(int minElementsInMemory, int maxElementsInMemory, int period);
142
143
~~~
143
144
144
-
where _minElementsInMemory_ are the number of objects to keep in the cache when cleaning up, maxElementsInMemory triggers a cleanup if the cache size goes over it, and period specifies the time in seconds to do periodic cleanups.
145
+
where `minElementsInMemory` is the number of objects to keep in the cache when cleaning up, `maxElementsInMemory` triggers a cleanup if the cache size goes over it, and `period` specifies the time in seconds to do periodic cleanups.
145
146
146
147
After enabling, you can disable with:
147
148
148
-
~~~
149
+
~~~java
149
150
NetcdfDataset.disableNetcdfFileCache();
150
151
~~~
151
152
152
153
However, you cant reenable after disabling.
153
154
154
-
Setting <b>_minElementsInMemory_</b> to zero will remove all files not currently in use every <b>_period_</b> seconds.
155
+
Setting `minElementsInMemory` to zero will remove all files not currently in use every `period` seconds.
155
156
156
157
Normally the cleanup is done is a background thread to not interferre with your application, and the maximum elements is approximate. When resources such as file handles must be carefully managed, you can set a hard limit with this call:
157
158
158
-
~~~
159
-
NetcdfDataset.initNetcdfFileCache(int minElementsInMemory, int maxElementsInMemory, int hardLimit, int period);
159
+
~~~java
160
+
NetcdfDataset.initNetcdfFileCache(int minElementsInMemory, int maxElementsInMemory, int hardLimit, int period);
160
161
~~~
161
162
162
163
so that as soon as the number of NetcdfFile objects exceeds hardLimit , a cleanup is done immediately in the calling thread.
0 commit comments