Easier access to scientific data
Brought to you by NOAA NMFS SWFSC ERD
Here are the changes associated with each ERDDAP release.
We recommend overwriting the default <startBodyHtml5> to customize your ERDDAP so that ERDDAP's web pages reflect your organization, not NOAA ERD. If you made changes to the original <startBodyHtml> and/or want to customize your ERDDAP now, please copy the new <startBodyHtml5> tag (from below) into your setup.xml and modify it to reflect your organization. Notably, please change the "Brought to you by" to your organization(s). If you need help, please email bob.simons at noaa.gov.
Then delete the 3 tags in your setup.xml which are no longer used.
<startBodyHtml5><![CDATA[ <body> <table class="compact nowrap" style="width:100%; background-color:#128CB5;"> <tr> <td style="text-align:center; width:80px;"><a rel="bookmark" href="http://www.noaa.gov/"><img title="National Oceanic and Atmospheric Administration" src="&erddapUrl;/images/noaab.png" alt="NOAA" style="vertical-align:middle;"></a></td> <td style="text-align:left; font-size:x-large; color:#FFFFFF; "> <strong>ERDDAP</strong> <br><small><small><small>Easier access to scientific data</small></small></small> </td> <td style="text-align:right; font-size:small;"> &loginInfo; <br>Brought to you by <a title="National Oceanic and Atmospheric Administration" rel="bookmark" href="http://www.noaa.gov">NOAA</a> <a title="National Marine Fisheries Service" rel="bookmark" href="http://www.nmfs.noaa.gov">NMFS</a> <a title="Southwest Fisheries Science Center" rel="bookmark" href="https://swfsc.noaa.gov">SWFSC</a> <a title="Environmental Research Division" rel="bookmark" href="https://swfsc.noaa.gov/textblock.aspx?Division=ERD&id=1315&ParentMenuId=200">ERD</a> </td> </tr> </table> ]]></startBodyHtml5>
There are additional ways you can customize ERDDAP so ERDDAP's web pages reflect your organization instead of NOAA ERD.
If you didn't customize those tags, please delete them from your setup.xml file.
Now they all have defaults in messages.xml that refer to datasets
in Bob's ERDDAP at https://coastwatch.pfeg.noaa.gov/erddap/index.html .
So you no longer need to have specific datasets in your ERDDAP.
If you want to override the defaults, copy some or all of those tags
into your setup.xml and change their values.
If you want the examples to point to your ERDDAP, the easiest method is:
<dataset type="EDDGridFromErddap" datasetID="jplMURSST41" active="true"> <sourceUrl>https://coastwatch.pfeg.noaa.gov/erddap/griddap/jplMURSST41</sourceUrl> </dataset> <dataset type="EDDTableFromErddap" datasetID="pmelTaoDySst" active="true"> <sourceUrl>https://coastwatch.pfeg.noaa.gov/erddap/tabledap/pmelTaoDySst</sourceUrl> </dataset>
If you did customize those tags, leave them as is and please add these 2 new tags to your setup.xml to specify the ERDDAP URL for these datasets, but change the URL to your ERDDAP's (https?) URL:
So, please check your datasets: if a variable's values are packed and
if actual_range is specified as packed data values,
please add an <addAttributes> actual_range value to specify the unpacked values.
Otherwise, the dataset will not load in ERDDAP.
A simple and almost perfect way to do this is to search your datasets.xml for
sourceAttributes that have
<att name="actual_range" type="shortList">
or <att name="actual_range" type="intList">
and a scale_factor other than 1.0. Those are the actual_range attributes that you might have to fix.
For axis variables in EDDGrid datasets, ERDDAP always sets the actual_range attribute to be the actual range of the values since it knows those values.
For axis variables with descending values (e.g., some latitude variables), ERDDAP used created actual_range with the ...[last] values, which were high...low. Now it always uses low...high values to make the new CF definition.
The correctness of the actual_range values is particularly important for EDDTable datasets, because ERDDAP will quickly reject user requests for data values which are less than the actual_range minimum value or which are greater than the actual_range maximum value.
Related: the actual_min, actual_max, data_min and data_max attributes are now deprecated. Please convert your datasets to use actual_range instead.
The netcdf-java developers maintain that these vulnerabilities are not relevant because of the way that netcdf code uses these libraries and in any case would only be relevant when accessing Amazon S3. See https://github.com/Unidata/thredds/issues/866 . I believe them. If you still have concerns about this, please contact the netcdf-java developers. (Note that if you don't believe the netcdf-java developers and are contemplating not using ERDDAP because of this, you shouldn't use THREDDS either, because THREDDS uses netcdf-java more fundamentally and more extensively than ERDDAP.)
Details: The troublesome code and the vulnerability warnings are:
See https://nvd.nist.gov/vuln/detail/CVE-2016-7051 - High
See https://nvd.nist.gov/vuln/detail/CVE-2016-7051 - High
See https://nvd.nist.gov/vuln/detail/CVE-2016-7051 - High
See https://nvd.nist.gov/vuln/detail/CVE-2016-3720 - Critical
See https://nvd.nist.gov/vuln/detail/CVE-2016-7051 - High
See https://nvd.nist.gov/vuln/detail/CVE-2016-3720 - Critical
"For version 4.6.10, aws-java-sdk-core pulls in version 2.6.6 of jackson-* artifacts." (email from netcdf-java people).
Thanks to Charles Carleton and NCEI.
<att name="colorBarMaximum" type="double">8000.0</att> <att name="colorBarMinimum" type="double">-8000.0</att> <att name="colorBarPalette">TopographyDepth</att>
<!-- If you want to restrict access to some datasets, you need to specify the method used for logging on (authentication). See the info at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security Currently, the options are: "" (logins not supported, the default), "custom", "email", and "google" (recommended). [No longer supported: "basic", "openid"] -->
In your setup.xml, please change the description for <baseUrl> to be
<!-- baseUrl is the start of the public url, to which "/erddap" is appended. For example: For running/testing on your personal computer: <baseUrl>http://localhost:8080</baseUrl> (127.0.0.1 doesn't work with authentication=google). For ERD releases, we use <baseUrl>http://coastwatch.pfeg.noaa.gov</baseUrl> If you want to encourage all users to use https (not http), make the baseUrl the same as the baseHttpsUrl (see below). -->
<!-- For "custom" authentication, this specifies how you have stored passwords in the roles tags in datasets.xml. If you aren't storing any passwords, this is irrelevant. The options (in order of increasing security) are: "MD5", "UEPMD5" (MD5(UserName:ERDDAP:Password)), "SHA256", "UEPSHA256" (SHA256(UserName:ERDDAP:Password), the default). You should only use "MD5" or "SHA256" if you need to match values stored that way in an external password database. See the info at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security -->
<!-- This is a variant of <baseUrl> which is used when authentication is active and the user is logged in. In general, you take the <baseUrl>, change "http" to "https", and change/add ":8443". This must begin with "https://". If you make a proxy so that ":8443" isn't needed, then don't use ":8443" here. This is relevant even if <authentication> is "". See the instructions at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security For example: For running/testing on your personal computer: <baseHttpsUrl>https://localhost:8443</baseHttpsUrl> For releases at ERD, we use: <baseHttpsUrl>https://coastwatch.pfeg.noaa.gov</baseHttpsUrl> If you want to encourage all users to use https (not http), make the baseUrl (see above) the same as the baseHttpsUrl. -->
<!-- Normally, if you have a EDDGridFromErddap or EDDTableFromErddap dataset in your datasets.xml, it will try to subscribe to the remote ERDDAP dataset so that the local dataset is kept perfectly up-to-date. If this ERDDAP is not publicly accessible (http://localhost), or its IP address will change soon, or you have some other reason, you can tell this ERDDAP to not try to subscribe to the remote ERDDAP datasets by setting this to false. (default=true) This is the overall setting for this ERDDAP. It can be overridden by the same tag (with a different value) in the datasets.xml chunk for a given EDD...FromErddap dataset. For each fromErddap dataset that doesn't subscribe to the remote ERDDAP dataset, you should set <reloadEveryNMinutes> to a smaller number so that the local dataset stays reasonably up-to-date. --> <subscribeToRemoteErddapDataset>true</subscribeToRemoteErddapDataset>
We still recommend: If a dataset has a large number of files (e.g., >1,000), the operating system (and thus EDDGridFromFiles and EDDTableFromFiles) will operate much more efficiently if you store the files in a series of subdirectories (one per year, or one per month for datasets with very frequent files), so that there are never a huge number of files in a given directory.
<!-- If true, when you start up ERDDAP, some types of datasets (e.g., EDDGridFromDap) will used cached information (.dds, .das, etc.) to reload very quickly, without contacting the remote server. The dataset's age will be based on when the dataset was reloaded last. Normally this should be true (the default), but set it to false if you want to bypass the cached information. <quickRestart>true</quickRestart>
<!-- ERDDAP lets you choose between two search engines for full text searches: * original (the default) - is the best choice if your ERDDAP has fewer than about 10,000 datasets. It is very robust and trouble free. * lucene - is the best choice for more than about 10,000 datasets. The advantages are that with any number of datasets it works fast and uses very little memory. But there are many things that might go wrong with individual queries and with the whole system. And although its behaviour (the datasets it finds and the order that it ranks them) is almost identical to the original search engine, it has a few quirky, subtle, small differences. --> <searchEngine>original</searchEngine>
<!-- Tabledap Examples This group of settings is used to make examples for the tabledap documentation that appears at [baseUrl]/erddap/tabledap/documentation.html and elsewhere. If you include the erdGlobecBottle dataset in your datasets.xml (recommended), you don't need to change these. If you don't, you MUST change these before you make your ERDDAP public; otherwise, none of the examples will work! The new settings should be very similar to the defaults. If your ERDDAP won't serve any tabular datasets, use "NOT_APPLICABLE" for all of the enties. In .xml files like this, ampersand, lessThan, and greaterThan have to be HTML encoded as "&", "<", ">". --> <!-- This is the datasetID for an EDDTable dataset that is served by your ERDDAP. This dataset is used as the basis for all of the EDDGrid examples below. Ideally, it is a dataset that has longitude, latitude, and time variables (among others). ('time' allows for making a time series graph. 'latitude' and 'longitude' allow for making a map.) The dataset can have longitude values -180 to 180, or 0 to 360. --> <EDDTableIdExample>erdGlobecBottle</EDDTableIdExample> <!-- This is a comma-separated list of variables from the dataset. It is useful if it is "longitude,latitude,time," plus a data variable name. --> <EDDTableVariablesExample>longitude,latitude,time,bottle_posn,temperature1</EDDTableVariablesExample> <!-- This is the constraints example which is appended to EDDTableVariablesExample. --> <EDDTableConstraintsExample>&time>=2002-08-17T00:00:00Z&time<=2002-08-19T20:18:00Z</EDDTableConstraintsExample> <!-- This is an example data query using an ISO-formatted time. You could generate your example via your dataset's Data Access Form in ERDDAP. --> <EDDTableDataTimeExample>longitude,latitude,time,bottle_posn,temperature1&time>=2002-08-17T00:00:00Z&time<=2002-08-19T20:18:00Z</EDDTableDataTimeExample> <!-- This is an equivalent example data query, but which specifies time as seconds-since-1970-01-01. If you need to convert a date/time to "seconds since 1970-01-01", use https://coastwatch.pfeg.noaa.gov/erddap/convert/time.html --> <EDDTableDataValueExample>longitude,latitude,time,bottle_posn,temperature1&time>=1029542400&time<=1029788280</EDDTableDataValueExample> <!-- This is an example query which generates a graph. You could generate your example via your dataset's Make A Graph form in ERDDAP. --> <EDDTableGraphExample>bottle_posn,temperature1&time=2002-08-19T10:06:00Z&.draw=lines</EDDTableGraphExample> <!-- This is an example query which generates a map. In the default mapExample, temperature1, time, bottle_posn are useful because they appear in GoogleEarth with the .kml example and are ignored by the other image file types. --> <EDDTableMapExample>longitude,latitude,temperature1,time,bottle_posn&time>=2002-08-13T00:00:00Z&time<=2002-08-20T00:00:00Z&bottle_posn=1&.draw=markers&.marker=5|5</EDDTableMapExample> <!-- This is a Matlab example which uses data from the EDDTableGraphExample. Note the Matlab notation datasetName.variableName. --> <EDDTableMatlabPlotExample>plot(erdGlobecBottle.bottle_posn, erdGlobecBottle.temperature1)</EDDTableMatlabPlotExample>
Note that actual_range is unchanged: it may have low,high values or high,low values, since it is intended to indicate the range and the order of storage.
<!-- These default accessConstraints, fees, and keywords are used by the SOS, WCS, and WMS services. They can be overridden by "accessConstraints", "fees", "keywords" attributes in a dataset's global metadata. If a dataset that has an "accessibleTo" tag doesn't override "accessConstraints", then the default for "accessConstraints" is the "accessRequiresAuthorization" value. --> <accessConstraints>NONE</accessConstraints> <accessRequiresAuthorization>only accessible to authorized users</accessRequiresAuthorization> <fees>NONE</fees> <keywords>Earth science, oceans</keywords> <!-- This appears on the erddap/legal.html web page after the General Disclaimer. You can replace any of the [standardParts] with your own HTML. --> <legal><![CDATA[ [standardDisclaimerOfEndorsement] [standardDisclaimerOfExternalLinks] [standardPrivacyPolicy] [standardDataLicenses] ]]></legal> <!-- Specify the default units standard (e.g., "UDUNITS" (the default) or "UCUM") that you (the ERDDAP admin) are using to specify units. The value is case-sensitive. This is used by ERDDAP's SOS server to determine if the units need to be converted to UCUM units for WMS and SOS GetCapabilities responses. --> <units_standard>UDUNITS</units_standard> <!-- For the wms examples, pick one of your grid datasets that has longitude and latitude axes. The sample variable must be a variable in the sample grid dataset. The bounding box values are minx,miny,maxx,maxy. --> <wmsSampleDatasetID>erdBAssta5day</wmsSampleDatasetID> <wmsSampleVariable>sst</wmsSampleVariable> <!-- The bounding box values are minLongitude,minLatitude,maxLongitude,maxLatitude. Longitude values within -180 to 180, or 0 to 360, are now okay. --> <wmsSampleBBox>0,-75,360,75</wmsSampleBBox>
<!-- startHeadHtml has the start of the HTML document and the 'head' tags (starting at "<!DOCTYPE>", but not including "</head>") for all HTML web pages. This may include &erddapUrl;, which is expanded to be [baseUrl]/erddap (or [baseUttpsUrl]/erddap if the user is logged in). If your ERDDAP allows users to log in, all referenced image files, css files, etc. must be in [tomcat]/content/erddap/images or a subdirectory and must be referenced here with &erddapUrl;/images/[fileName]. favicon.ico is the image that browsers associate with your website. More more information, see https://en.wikipedia.org/wiki/Favicon . You can use your own favicon.ico file by putting it in [tomcat]/content/erddap/images. *** Optional: you can change the appearance of all of your ERDDAP's HTML pages by changing the CSS <style> settings below. For an example of a very different style, change the import reference to <tomcat>/content/erddap/images/erddapAlt.css *** If your CSS style includes links to files (e.g., images), that style information must be inline in the style tag below, after the 'import' line, not in the .css file. Put all of the (e.g., image) files in the [tomcat]/content/erddap/images directory (or a subdirectory) and reference them below starting with &erddapUrl;. Why? On ERDDAP https: web pages, *all* links should use "https:" (not "http:"); otherwise, most browsers consider the web page not fully secure. Because ERDDAP would use the same .css file for http: and https: web pages, the links within the .css file wouldn't switch between http: and https:. There doesn't seem to be a way around this other than using inline style information. --> <startHeadHtml><![CDATA[ <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <html> <head> <title>ERDDAP</title> <link rel="shortcut icon" href="&erddapUrl;/images/favicon.ico"> <style type="text/css"> <!-- @import "&erddapUrl;/images/erddap.css"; --> </style> ]]></startHeadHtml> <!-- The tableCommonBGColor MUST be the same color as the table.commonBGColor in erddap.css above. Suggested is #f1ecd8. But if you use erddapAlt.css, change this to #e7dec5. --> <tableCommonBGColor>#f1ecd8</tableCommonBGColor> <!-- This is used, e.g., for the type=variable rows on the metadata info tables. --> <tableHighlightBGColor>#cceecc</tableHighlightBGColor>Thanks to POST, Hans Vedo, and Rick Blair.
<theShortDescriptionHtml><![CDATA[ <h1>ERDDAP</h1> This web site (the Environmental Research Division's Data Access Program) aggregates scientific data from diverse local and remote sources and offers you a simple, consistent way to download subsets of the data in common file formats and make graphs and maps. This particular ERDDAP installation has oceanographic data (for example, data from satellites and buoys). [standardShortDescriptionHtml] ]]></theShortDescriptionHtml>Feel free to change this, particularly the last sentence in the first paragraph.
<!-- If you want to restrict access to some datasets, you need to specify the method used for logging on (authentication). See the info at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security Currently, the options are: "" (logins not supported, the default), "custom", "openid". Note that openid login doesn't work when testing with localhost (https://127.0.0.1:8443). --> <authentication></authentication> <!-- This specifies how you have stored passwords in the roles tags in datasets.xml. If you aren't storing any passwords this is irrelevant. The options (in order of increasing security) are: "plaintext", "MD5", or "UEPMD5" (MD5(UserName:ERDDAP:Password), the default). You should only use "plaintext" or "MD5" if you need to match values stored that way in an external password database. See the info at https://coastwatch.pfeg.noaa.gov/erddap/download/setup.html#security --> <passwordEncoding>UEPMD5</passwordEncoding> <!-- This determines whether datasets that the user doesn't currently have access to (because he isn't logged in or because his roles don't allow access) should be shown on lists of data sets (e.g., from full text search, categorize, view all datasets, ...). The options are: "true", or "false" (the default). If false, no information about the dataset (even its existence) is shown to users who don't have access to it. If true, some information about the dataset (title, summary, etc) is shown to users who don't have access to it. If the user clicks on a link to a dataset he doesn't have access to, he will get an error message and be prompted to log in. --> <listPrivateDatasets>false</listPrivateDatasets> <!-- If the number of requests between two runs of LoadDatasets exceeds unusualActivity, an email is sent to emailEverythingTo. The default is 10000. --> <unusualActivity>10000</unusualActivity>
There is nothing new about making a local copy of a dataset. What is new here is that these classes make it *easy* to create and *maintain* a local copy of data from a *variety* of types of remote data sources and *add metadata* while copying the data.
These dataset types are part of a complete set of features that simplify the creation of grids/clusters/federations of ERDDAPs to handle very heavy loads (e.g., in a data center).
<!-- ERDDAP has a service that lets remote users set a flag to notify ERDDAP to try to reload a dataset. These requests use a key which is generated based on baseUrl/warName, a datasetID, and flagKeyKey. *** Change this once, to any text (a favorite quote? random text? It doesn't matter). Normally, you won't ever change this again. But if you you think someone is abusing the flag system, change this text again, restart ERDDAP, and send all of the users of the flag system the relevant new flagKeys (see the list in the Daily Report). --> <flagKeyKey>A stitch in time saves nine. CHANGE THIS!!!</flagKeyKey> <!-- ERDDAP has an email/URL subscription system which sends a user an email or pings a url whenever a dataset of interest changes. (This is different from the RSS system, which is always active.) The system relies on the server being able to send out emails to people to validate their subscription requests. The emails appear to come from the emailFromAddress below. So if your server can't send out emails, don't make this system active. You may choose (for whatever reason) to make this system active or not, so valid values below are "true" (the default) and "false". Note that if you change this and restart ERDDAP, the list of subscriptions (in [bigParentDirectory]/subscriptionsV1.txt) isn't affected. See also the subscriptionEmailBlacklist in datasets.xml. --> <subscriptionSystemActive>true</subscriptionSystemActive>
<!-- The bounding box values are minLongitude,minLatitude,maxLongitude,maxLatitude. Longitude values within -180 to 180, or 0 to 360, are now okay. --> <wmsSampleBBox>0,-75,360,75</wmsSampleBBox>
<!-- drawLand specifies the default Make A Graph setting for whether the landmask should be drawn "over" (the default) or "under" surface data on maps. "over" is recommended for primarily oceanographic data (so that grid data over land is obscured by the landmask). "under" is recommended for all other data. --> <drawLand>over</drawLand> <!-- Information about the ERDDAP administrator is used for the SOS and WMS servers. You MUST CHANGE these to describe your installation. --> <adminInstitution>NOAA Environmental Research Division</adminInstitution> <adminIndividualName>Your Name</adminIndividualName> <adminPosition>Webmaster</adminPosition> <adminPhone>your-phone-number</adminPhone> <adminAddress>99 Pacific St, Suite 255A</adminAddress> <adminCity>Monterey</adminCity> <adminStateOrProvince>CA</adminStateOrProvince> <adminPostalCode>93940</adminPostalCode> <adminCountry>USA</adminCountry> <adminEmail>yourName@yourSite</adminEmail> <!-- Information about the ERDDAP administrator is used for ERDDAP's SOS server. You MUST CHANGE these to describe your installation. --> <sosTitle>NOAA Environmental Research Division SOS</sosTitle> <sosAbstract>NOAA Environmental Research Division's ERDDAP makes data from multiple sources available via the SOS protocol.</sosAbstract> <sosKeywords>Weather, Ocean Currents, Temperature, Salinity</sosKeywords> <sosAccessConstraints>NONE</sosAccessConstraints> <sosFees>NONE</sosFees> <!-- Information about the ERDDAP administrator is used for ERDDAP's WMS server. You MUST CHANGE these to describe your installation. --> <wmsTitle>NOAA Environmental Research Division WMS</wmsTitle> <wmsAbstract>NOAA Environmental Research Division's ERDDAP makes data from multiple sources available via the WMS protocol.</wmsAbstract> <wmsKeywords>Weather, Ocean Currents, Temperature, Salinity</wmsKeywords> <wmsAccessConstraints>NONE</wmsAccessConstraints> <wmsFees>NONE</wmsFees> <!-- For the wms examples, pick one of your grid datasets that has longitude and latitude axes. The sample variable must be a variable in the sample grid dataset. The bounding box values are minx,miny,maxx,maxy. --> <wmsSampleDatasetID>erdBAssta5day</wmsSampleDatasetID> <wmsSampleVariable>sst</wmsSampleVariable> <wmsSampleBBox>0,-75,180,75</wmsSampleBBox>